The average farm size in Ethiopia is shrinking and the option for expanding the land frontier is also very limited. As a result, increasing farm productivity is critical for achieving higher growth and national food security. Identifying the drivers of productivity and weighing their significance are therefore vital for effective policy making. This paper applied a stochastic input distance function to decompose and test the significance of economic efficiency improvement in boosting the productivity of small-scale farmers. The results show that small-scale farming exhibits scale, technical and scope economies and thus the opportunities for increasing productivity through improving efficiency alone is significant. However, most of the improvement in efficiency in the immediate term is expected to come from the increase in the technical, mix, and scope efficiencies. Farmers that cultivate diverse crops are technically more efficient and are also able to realize economies of scope and scale than farmers with specialized production. While farmer specific factors played some roles, most of the inefficiencies are traced to externally imposed policy and institutional constraints. Addressing market failures and enhancing competition in the goods and factor markets, particular those that led to further land consolidation, will have a significant impact on farm productivity.