One of the innovations in the 2013 revision of the CAP Basic Acts was to extend a formal system of monitoring and evaluation of the CAP’s performance beyond Pillar 2 rural development, where it had been long-established, to also cover Pillar 1 direct payments and market management. This Common Monitoring and Evaluation Framework (CMEF) was set out in Article 110 of the Horizontal Regulation. This requires the Commission to measure and assess the combined impact of all CAP instruments in relation to three common objectives set out in this Article.
The three objectives set out for the CAP (which essentially ‘reinterpret’ the objectives contained in Article 39 of the Lisbon Treaty) are:
• viable food production, with a focus on agricultural income, agricultural productivity and price stability;
• sustainable management of natural resources and climate action, with a focus on greenhouse gas emissions, biodiversity, soil and water;
• balanced territorial development, with a focus on rural employment, growth and poverty in rural areas.
These three objectives, in turn, are meant to feed into the Europe 2020 objectives of smart, sustainable and inclusive growth.
These general objectives are broken down into a number of specific objectives, some of which are common to both Pillars whereas others are linked either to Pillar I or to Pillar II. DG AGRI has prepared a graphical presentation of these general objectives and their breakdown into specific objectives and how the related Pillar I instruments and RD priorities (“specific objectives”) and focus areas feed into them which is presented below. Note how the blue and violet boxes both encompass the specific objectives in the middle purple column (double clicking the diagrams in this post can help to make them more legible if necessary).
When undertaking an evaluation of CAP policies, the key step is to establish the underlying intervention logic (or logic model). This is defined in the Commission’s Better Regulation guidelines as:
The intervention logic is the logical link between the problem that needs to be tackled (or the objective that needs to be pursued), the underlying drivers of the problem, and the available policy options (or the EU actions actually taken) to address the problem or achieve the objective. This intervention logic is used in both prospective Impact Assessments and retrospective evaluations.
Thus the intervention logic sets out the chain of expected effects between the policies implemented (these would be the instruments in CAP Pillar 1 or the various measures in rural development programmes in Pillar 2) and the specific and general objectives to be achieved (set out for the CAP in the previous diagram). These expected effects can be envisaged as links in a chain beginning with the expected output of the policy, which leads to specific results, and which then leads to the expected positive impacts on the policy objectives.
Each link, in turn, is associated with a set of indicators which can be used for the purpose of assessing impact or performance. Measurement of impacts is done at the level of the general objectives (using impact indicators), results at the level of specific objectives (using results indicators) and outputs at the level of instruments/measures (using output indicators). This indicator hierarchy is shown in the following diagram.
In addition, the monitoring and evaluation framework also includes a set of context indicators, to highlight that the relationship between outputs, results and impacts is not a mechanical one but can be influenced by the state of the broader economy, environmental factors, conjunctural factors and so on. Information on the composition of these four groups of indicators is set out on the DG AGRI webpage on CAP monitoring and evaluation indicators.
Intervention logic of CAP policies
The intervention logic for a policy is often set out in the form of a diagram which visually captures the different and overlapping links between policies and objectives. DG AGRI’s Management Plan for 2015 contains four intervention logic diagrams which give a good insight into how it sees the updated CAP working. These diagrams focus on direct payments, the single CMO, horizontal and other measures, and rural development, and I reproduce them below.
In the first three diagrams, each instrument is shown as contributing principally to one specific objective, although it may have indirect effects on one or more other objectives. For example, in the intervention logic for direct payments, the principal link between the basic and redistributive payments is shown as enhancing farm income, but with a secondary impact on maintaining agricultural diversity. The green payment is shown as primarily influencing the provision of environmental public goods, but with a secondary impact on agricultural competitiveness.
For the single CMO intervention logic, market measures are shown as mainly targeted at maintaining market stability, but with an indirect effect on enhancing farm income. Producer organisations are shown as primarily supported for their contribution to agricultural competitiveness, but their secondary contributions to enhancing farm income, maintaining market stability, meeting consumer expectations and maintaining agricultural diversity are also recognised.
The intervention logic for horizontal measures includes farmer innovation as a specific objective, where the main instrument targeting this is the Farm Advisory Service. Promotion policy is seen as geared mainly to promoting agricultural competitiveness, while quality policy is seen as principally geared to maintaining agricultural diversity, but with secondary effects on farm income, agricultural competitiveness and consumer expectations.
Finally, the intervention logic for Pillar 2 measures takes a slightly different form. Overall, there are six priorities (specific objectives) for Rural Development, each broken down into a number of focus areas. Five priorities directly feed into the CAP general objectives. These five priorities are supported by one overall priority, i.e. knowledge transfer and innovation, which contributes to the general CAP objectives via the five other priorities. Pillar 2 measures are also evaluated with respect to their contribution to the thematic objectives for the Europe 2020 strategy for smart, sustainable and inclusive growth.
Measuring CAP performance
At the end of the day, what we are ultimately interested in is what difference has the CAP made to the general and specific policy objectives outlined above, i.e. its impact. Measuring this impact will never be straightforward, of course, because we can never be certain about the counterfactual situation, what would have occurred in the absence of the CAP. The commitment to monitoring and evaluation is praiseworthy, although there is a danger that it becomes a box-ticking exercise.
As the Head of Evaluation in the Regional Policy Directorate wrote in a critical reflection on the use of impact indicators in the context of regional policy:
But we need to ask why we are gathering all this data. Much of the data are not meaningful and certainly do not represent the impact of the Structural Funds. Their purpose seems primarily to provide a figure which will unlock the final payment from the Structural Funds, rather than being an exercise in accountability and learning. There has been little or no public debate on these figures in any Member State. There must be scope to rationalise, streamline and focus, but also to clarify conceptually what we should count and what these figures mean.
The CAP Horizontal Regulation requires that the Commission presents an initial report, including first results on the performance of the CAP, to the European Parliament and the Council by 31 December 2018. A second report including an assessment of the performance of the CAP shall be presented by 31 December 2021.
To this end, DG AGRI envisages offering four framework contracts, three of which to permit analysis of the contribution of the CAP measures towards each of the three general objectives, and a fourth framework contract will analyse the impact of the CAP on cross-thematic issues, as well as provide for synthesis of evaluation/study findings. The tender specification for the first of these has now been published on the DG AGRI website.
In each case, it is planned that the contractors will undertake an initial evaluation of the contribution of the CAP starting in 2016 to be completed in 2017. These evaluations should thus be ready to feed into a possible Mid-Term Review of the CAP at the end of 2017 at the time when the Commission will come forward with its proposal for the next Multi-annual Financial Framework, including the future CAP budget, for the years after 2020.
This post was written by Alan Matthews.