The evaluation of rural development policies has improved over the years. The current mid-term evaluation, with all member states applying the sophisticated Common Monitoring and Evaluation Framework (CMEF) underpinned by voluminous Commission guidelines, is another step forward. DG Agri is adding further tools to make evaluations even more productive, such as an Internet platform for all stakeholders involved in the evaluation process.
The philosophy of monitoring, learning and adjusting is correct – but it can lead to an excessive trust in our abilities to manage economic processes, including their social and environmental implications. We must recognize the limits of our ability to glean information about the diverse challenges on the ground, to identify the locally diverse and multifaceted effects of policy interventions, to process this information intelligently and impartially, and to draw adequate conclusions. In other words, we must take the limits of top-down government seriously.
Some problems of evaluation are clear-cut and, theoretically, remedies are within reach. For instance, the data needs of evaluators could be better integrated into routine statistics and evaluators could be made more independent from policymakers. However, the very fact that these problems exist though they are simple to understand and to resolve – and that they exist not only with regard to the CAP but in all policy areas and countries – gives reason to doubt future progress. Sound evaluation tends to hurt; it always focuses on the weaknesses and adopts ever tougher benchmarks. Policymakers and administrators therefore take care that evaluation does not become (too) sound.
In addition, evaluation is beset with dilemma that even the most ambitious reformer cannot resolve. The advantages of control can be offset by a loss of trust: lower-level actors may prefer to have an impeccable formal record rather than to be effective and innovative. The benefits of evaluation must be balanced against their administrative demands on governments and farmers. The system should be sophisticated (and thus yield value even for the most advanced member states) yet also manageable for those member states with the weakest analytic resources. Evaluation reports should be quickly available and incorporate the long-term effects of policy measures. Reports should be adapted to local circumstances – and be comparable across member states. Indicators should be stable, so that evaluators can learn how to handle them and that policymakers can observe how policies work across time, but they should also change in response to new insights and challenges. As a result of such dilemma, improvements in one regard tend to come at hidden costs in other dimensions.
In brief, policy evaluation is not like a new technology whose potential is still hard to grasp; it is an age-old technique whose inherent limits are well known. This matters for policy design.
First, policy objectives should be spelled out clearly and guidance on suitable indicators should be provided directly in the legislation establishing the policy instrument. Currently, objectives are often left vague and actual policies designed in a way that makes a mockery of subsequent evaluation (e.g. when the very intention of a policy is to create windfall gains for farmers but the evaluator is asked to ‘show’ its beneficial effects on competitiveness).
Second, the number of policy objectives and instruments should be limited. DG Agri counts more than 150 indicators to assess rural development – this plethora favors box-ticking over thorough analysis. Interestingly, environmental impacts tend to be easier to measure than socio-economic impacts. It is, for example, impossible to get a solid grip on the net job effects of farm modernization subsidies – especially when cross-sectoral effects are accounted for. This is yet another argument for focusing the CAP on sustainable land use.
Third, difficulties of policy evaluation and learning at the top speak in favor of devolving responsibility to lower levels of governance. Local authorities are best able to identify their needs, to see which solutions work best and to get results at lowest costs. Co-financing is therefore an important tool to create strong incentives for local and national authorities to use their informational advantage for more effective rural development.
This post is based on the personal lessons that I took away from a recent seminar on rural development evaluation. The seminar presentations can be downloaded here.
