Soil erosion and degradation is an important agro-ecological challenge in the highlands of Ethiopia. Conservation agriculture (CA) has a long time been identified as one of the key interventions that could abate the current trend of physical and chemical erosion of soil. This study analyzed adoption of the different components of CA (minimal disturbance of soil, permanent organic soil cover, and crop rotation) and herbicide application in two districts of Ethiopia using a multivariate probit model. The impact of CA on land and labor productivity was also estimated using generalized methods of moments and the control function approach. The initial decision to adopt the different components of CA is influenced by location, family size, access to extension, and formal education. Among the components introduced with CA, herbicide application significantly and strongly influences land productivity. Other factors, which influenced land productivity, were location, sex of head of household, livestock wealth, and human labor endowment. None of the components of CA—including the complementary herbicide application—was found to be influencing labor productivity in the study areas. Generally, access to extension and the main crop under production drive the decision to adopt or not to adopt the different components of CA. Despite its positive impact on land productivity, herbicide application should be further investigated with attention to its effect on sustainable use of cultivable land.