This post is written by Ulrich Koester and Jens-Peter Loy
According to new legislation, the European Commission (EC) is in charge of evaluating Pillar 1 measures of the Common Agricultural Policy (CAP), while Pillar 2 measures have to be evaluated by the Member States (MS). Pillar 1 measures are of utmost importance for EU expenditure, amounting to a share of about 40% of the total expenditure of the EU budget. The request for evaluation is a significant step forward. One may wonder whether this new task indicates that the measures of the CAP have not been evaluated regularly so far. In the following, we focus on one specific measure, direct payments, for two reasons.
First, these payments make up more than 70% of the total CAP expenditure and, second, the EC has already published respective evaluation results.
The structure of the post is as follows: In the first part, we present the evaluation methodology widely used by professional economists. This approach serves as a benchmark for investigating the procedure applied by the EC. In the second part, we present the EC’s approach to identify the impact of direct payments on viable production.
As it is not possible to examine all of the numerated impact indicators by the EC due to lack of space, we concentrate on one specific impact indicator: “entrepreneurial income.” This indicator, also called “farm family income,” is used to challenge the rationale of using impact indicators for policy evaluation. As an evaluator has to know whether the objectives are well defined, we will explore whether the EC’s task is well defined and whether a change in the value of the objectives can be measured. We also examine how the EC deals with multiple and unclearly defined objectives, and investigate whether the EC’s approach deals adequately with the identification problem.
Quantifying changes in the contributions of policy objectives is an important task in the evaluation, as it indicates the effectiveness of the policy intervention. However, the final judgement of any policy intervention must be based on a comparison of the benefits and costs. Hence, we explore whether the EC’s approach includes an adequate assessment of the costs of the intervention. We summarize the main weaknesses of the EC’s methodology in the final section.
A benchmark for the evaluation of the methodology applied by the EC
The following are basic principles of a methodology that is generally accepted and used by professional economists:
1. Define the objectives and measurement of changes. Changes in objective variables should be quantifiable and the ranking of changes must be possible in either a cardinal or ordinal order.
2. Compare the situation at present and in the future with the desired situation considering the desired achievement of objectives. This step in the evaluation method ascertains whether a policy intervention or change might be considered.
3. Ascertain whether the chosen instruments contribute to reducing the gap between the situation with and without policy intervention. The result indicates the effectiveness of the policy intervention.
4. Specify the costs of the policy intervention. Professional economists accept that public expenditure and economic costs are important. Economic costs show the value of resources which are reallocated from alternative uses of resources. Costs also include by-product distortions and administrative costs. By-product distortions may show up if side-effects lead to distortions and administrative costs. The result of this finding informs on the efficiency of the policy measures.
5. Identify measures which would have most likely been more efficient, but only if the need for further policy intervention is supported by the finding of the evaluation.
The EC’s approach
Definition and clarification of objectives
The task of the EC is specified in Article 110 of EU Regulation 1306–2013, which states that a common monitoring and evaluation framework shall be established with a view to measuring the performance of the CAP, particularly with respect to direct payments provided for in Regulation (EU) No 1307/2013. This article also defines the following objectives for measuring performance:
a) Viable food production, with a focus on agricultural income, agricultural productivity and price stability.
b) Sustainable management of natural resources and climate action, with a focus on greenhouse gas emissions, biodiversity, soil, and water.
c) Balanced territorial development, with a focus on rural employment, growth, and poverty in rural areas.
These are just three of the objectives as implied by the enumeration. Take, for example, the first one: “viable production.” The additional description might be considered a specification of “viable production”; however, this interpretation can only be accepted if the additional attributes do not completely, or at least partly, conflict with each other.
Agricultural income is not clearly defined. It could mean the average income of the agricultural sector in the EU or in individual member countries or it could mean the average income of family farms, the income of marginal producers, or the income of a specific group of farmers, e.g., family farms. The specification of the term “agricultural income” determines whether the achievement might be in conflict or harmony with the achievement of agricultural productivity.
Nevertheless, agricultural income of the sector may not be informative for policy decisions based on the evaluation of policy measures.
The term “agricultural income” or the change of it over time is not a reasonable indicator of the agricultural sector’s efficiency at present and in the future. The efficiency of a sector may increase over time, even if some farmers generate a low average income. The setting and change of EU milk production may serve as an example: Small farmers may have low productivity, but low opportunity costs for labor due to a lack of alternative employment opportunities. These farms are economically efficient.
In contrast there may be some highly efficient and productive farms with incentives to grow. The average income may be considered sufficient, but for some, it may only be sufficient for a short period of time. Direct payments might increase the average income of the sector, meaning that less efficient and low productivity farmers, the present marginal producers, will continue their production and reduce the shift of land from low productivity to high productivity farms. Hence, an increase in average farm income may conflict with the objective to increase productivity.
This example supports the well-known insight that an adequate evaluation requires clear definitions of policy variables and their quantification. If policymakers define a set of multiple policy variables, they pose an unsolvable problem for the evaluator. As long as policymakers fail to specify the objectives clearly or inform the evaluator of the trade-off between the achievements of alternative objectives, no policy evaluation will produce a clear assessment.
The EC could reject the evaluation based on the above reasoning. To overcome this situation, the EC developed a specific approach based on impact indicators and their assumed relationship with the stated policy objectives, which will be analyzed below.
Implication of unclear definition, quantification, and the justification of policy intervention—the problem of providing a diagnosis
As mentioned above, a rational design for a policy action must be based on a comparison of the actual situation and the desired situation. The desired situation should achieve or at least contribute to better achievements of the policy objectives. Hence, the need for policy intervention can only be rational if a gap exists between the desired and the actual situation.
Of course, identifying the so-called diagnosis is not easy. As time is needed after the application of the policy intervention to know its impact, the comparison must be based on future situations with and without the policy intervention.
The EC has yet to follow the well-known methodology for providing a diagnosis. This observation is not surprising, as the desired situation cannot be defined due to multiple, and unclearly defined objectives.
The EC use of impact indicators has been recommended in the legislation. “The performance of the CAP in achieving its common objectives shall be measured and assessed on the basis of common impact indicators, and the underlying specific objectives on the basis of result indicators.”
The use of impact indicators by the EC’s policy evaluation
It is suggested by the EC that the following impact indicators reveal the contribution to the objective “viable production.”
a) Agricultural entrepreneurial income
b) Agricultural factor income
c) Agricultural productivity
d) EU commodity price variability
These indicators are calculated for consecutive years. A positive change over time is interpreted as a positive impact of policy interventions.
In the following — due to space limits — we only discuss the calculation of the first impact indicator, which might be most closely related to the objective “viable production,” and the use of these variables for evaluation purposes.
Nevertheless, restricting these impact indicators may allow identification of the main problems using this and other impact indicators for policy evaluation.
Agricultural entrepreneurial income
The method for calculating this variable is well described in the document quoted above. However, the question remains how this variable can be used in the evaluation process. According to the EC,
“The impact indicators thus will be relevant:
1) in structuring the overall assessment of the performance of the CAP
2) as the subject of cross cutting evaluations, e.g. the impact of CAP on biodiversity
3) when assessing the contribution of (individual) first pillar instruments and RDP programmes.”
Agricultural entrepreneurial income is equal to the part of farm income that is available for the remuneration of unpaid labor for the farm family and the capital owned by the farm family. Notably, this income includes a partial or the total amount received by direct payments, depending on whether the farmer cultivates rented land and has to pass over a portion of the direct payments to the landowner. Consequently, the assumption that agricultural entrepreneurial income would be reduced by the same amount as a decrease in the amount of direct payments received by the farmers is highly misleading. A significant difference exists between the recipient of the payments and the beneficiaries. The transfer efficiency is significantly smaller than 1.
Hence, this indicator cannot be used to explain the performance of the CAP. The indicator does not show what the performance of the CAP might have been without direct payments.
The EC also uses this variable to describe the present state of agriculture concerning the income situation: “The indicator uses in its calculation non-salaried AWU in order to show results on the standard of living of self-employed in agriculture per working unit.” Agricultural entrepreneurial income is divided by agricultural work units (AWU); the result is family farm income from unpaid farm labor. This derived variable is compared with “an average of the gross wages and salaries in the whole economy at current prices in cash and in kind.” Figure 1 presents the result of the EC’s comparison.
The EC interprets Figure 1 as follows: “While the EU agricultural sector has displayed a rapid increase in farm size and a significant improvement of productivity, many farms still depend heavily on direct payments due to the low profitability of agricultural activities. Direct payments represented on average 29% of agricultural income in the period 2007–2009 (with total subsidies coming close to 40% of agricultural income)” (p. 11). This interpretation is highly questionable for the following reasons:
First, the income comparison is misleading. The data are taken from Eurostat Economic Accounts for Agriculture, Agricultural Labor Input Statistics, and National Accounts. So-called farm family income does not refer to a specific section of farmers, but includes the entrepreneurial income earned by the whole agricultural sector. Hence, this includes income generated by agricultural activities of part-time farmers, hobby farmers, family farms, and legal entities.
So, first, it is not at all obvious how the variable “family farm income” relates to the variable production. Moreover, this variable does not include farmers’ income from non-farm activities such as remuneration from off-farm work and capital owned but invested in the non-farm sector.
Second, as noticed by the EC, the data in AWU are only rough estimates and are not available for all EU MS.
Third, the qualification of farm labor and non-farm salary and wage earners is different; it is likely on average much higher for the latter.
Fourth, entrepreneurial income does not include non-monetary income from ownership of houses and savings for renting apartments or houses. Many wage and salary earners spend up to 30% of their net income on housing. If this difference were taken into account, the presented income gap would vanish in most years for the EU-15.
Fifth, it is misleading to compare gross income if taxation for farmers differs from that of non-farmers; this is the case for some countries such as Germany.
Sixth, comparing average incomes of a large group of individuals with huge differences in income can hardly be used for income policies, such as direct payments linked to land endowment. It is well known that direct payments increase income disparity among farmers and that recipients are not always beneficiaries. The sentence of the EC, “Direct payments represented on average 29% of agricultural income in the period 2007–2009 (with total subsidies coming close to 40% of agricultural income)” (p. 11) is highly misleading, as it completely neglects the transfer from the recipient to the landowners. Those who transfer part of the payments directly reduce their income by the same amount because payments for rental land are expenses.
This last point restricts the intended message of Figure 2, as it does not show the pure effect of direct payments on agricultural factor income if the beneficiaries of the payments are landowners rather than the active farmers.
The EC used the variable agricultural entrepreneurial income for another income comparison. According to the Treaty of Rome and the following amendments, one objective of the agricultural policy is to contribute to a fair standard of living for the agricultural community. The EC assumes that agricultural entrepreneurial income (“family farm income”) is an indicator of the standard of living of the self-employed in agriculture that can be used to assess the impact of changes in the level of public support, i.e., direct payments, on the standard of living/purchasing power of farmers. Based on the qualifications regarding the entrepreneurial income above, this statement has to be rejected.
In conclusion, the data presented by the EC does not make clear whether the objective “viable production” has been achieved due to past policy interventions.
Moreover, the data provided (see Figure 2) do not support the EC’s conclusion that direct payments have positively contributed to the achievement of the objective “viable production.” As mentioned above, the presented figure overestimates the importance of direct payments for agricultural income, as a share of this payment is transferred to landowners who might not be farmers.
Moreover, the beneficiaries of the payments are often part-time, and in some EU MS, even subsistence farmers. Thus, it is not obvious what the contribution of these types of farmers might be to the “viable production” objective. The graph assumes implicitly that the structure of the agricultural sector with respect to farm sizes and productivity would be very different without direct payments.
We may conclude that the EU’s approach does not clarify whether there is a need for policy actions to improve contributions to the “viable production” objective.
The change of an indicator over time does not indicate the determinants of the change. Any indicator can change due to numerous exogenous variables and not only or mainly due to policy intervention. Thus, one of the most difficult tasks in policy evaluation is identifying the effect of all determinants of the change in a policy objective and specifying the pure effect of policy intervention. The EC has completely neglected these important steps in its policy analysis.
Finally, the EC fails to investigate the costs of direct payments or compare benefits with costs. The quantification of costs first requires the quantification of what could have been produced in the whole economy in monetary terms without direct payments to farmers and, second, what value of resources would have been withdrawn from agriculture and from the administration to implement the direct payment scheme.
It is likely that the direct payment scheme would have been significant, as the structure of the agricultural sector would have been affected by the closure of inefficient farms in favor of more efficient farms. The comparison would most likely show that the costs would be significantly higher than the benefits without affecting the “viable production” objective negatively.
Summary
The legislator of the EU has commissioned the EC to evaluate Pillar 1 measures of the CAP, particularly direct income payments. The legislator has laid down the policy objectives and has instructed the EC to specify the impact indicators for the evaluation.
It is argued in this post that the EC had to accept a task which could not be solved in line with the state-of-the-art methodology generally followed by professional economists. The objectives have not been clearly defined by the legislator, and the EC has failed to suggest a clear definition of objectives to measure expected positive changes resulting from the policy intervention.
Moreover, the approach of the EC did not solve the identification problem; it failed to compare the situation without and with direct payments. Thus, the EC was unable to justify the need for direct payments based on its diagnosis.
The EC used in line with the proposal of the legislator so-called impact indicators. Specific indicators have been selected to relate them to specific objectives; e.g. changes in entrepreneurial income were considered to contribute to the change in the objective ‘variable production’. It is argued that this relationship has to be challenged. The same conclusion holds for the impact indicator ‘factor income’.
None of the impact indicators used by the EC inform on the direct impact on the specific objective variables. Moreover, the approach used by the EU only focuses on potential benefits, but completely neglects economic costs.
This post is written by Ulrich Koester and Jens-Peter Loy
According to new legislation, the European Commission (EC) is in charge of evaluating Pillar 1 measures of the Common Agricultural Policy (CAP), while Pillar 2 measures have to be evaluated by the Member States (MS). Pillar 1 measures are of utmost importance for EU expenditure, amounting to a share of about 40% of the total expenditure of the EU budget. The request for evaluation is a significant step forward. One may wonder whether this new task indicates that the measures of the CAP have not been evaluated regularly so far. In the following, we focus on one specific measure, direct payments, for two reasons.
Photo credit: Derek Harper, licensed under CC licence
