Current use of mineral nitrogen (N) fertilizers is unsustainable because of its high fossil energy requirements and a considerable enrichment of the biosphere with reactive N. Biological nitrogen fixation (BNF) from leguminous crops is the most important renewable primary N source, especially in organic farming. However, it remains unclear to which degree BNF can sustainably replace mineral N, overcome the organic to conventional (O:C) yield gap and contribute to food security. Using an agronomic modelling approach, we show that in high-yielding areas farming systems exclusively based on BNF are unlikely to sustainably reach yield levels of mineral-N based systems. For a high reference wheat yield (7.5 t ha−1) and a realistic proportion of fodder legumes in the rotation (33%) even optimistic levels of BNF (282 kg N ha−1), resulted in an O:C ratio far below parity (0.62). Various constraints limit the agricultural use of BNF, such as arable land available for legumes and highly variable performance under on-farm conditions. Reducing the O:C yield gap through legumes will require BNF performance to be increased and N losses to be minimised, yet our results show that limits to the productivity of legume-based farming systems will still remain inevitable.