Institutional Learning and Change Initiative

The following text is drawn from the Eurpean Commission's website.

The Logic of Counterfactual Impact Evaluation

The Introduction to Impact Evaluation identified two separate sets of questions, one dealing primarily with quantification of effects, the other with their explanation. The first relying on counterfactual methods, the second on theory-based methods. In this section we deal exclusively with the first set of methods, devoted to quantifying whether a given intervention produces the desired effects on some pre-established dimension of interest.

Questions related to the sign and magnitude of programme impacts arise frequently in the evaluation of socio-economic development programmes. Do R&D subsidies increase the level of R&D expenditure by subsidized firms? Do targeted ERDF funds increase per capita income of the assisted areas? Do urban renewal programmes contribute to the economic development of urban neighbourhoods? Does support to SMEs increase their employment levels? Does investment in new public infrastructure increase housing values?

In other words, the evaluation problem has to do with the “attribution” of the change observed to the intervention that has been implemented. Is the change due to the policy or would it have occurred anyway? Answering these questions is not as straightforward as it might seem. The challenge for quantifying effect is finding a credible approximation to what would have occurred in the absence of the intervention, and to compare it with what actually happened. The difference is the estimated effect, or impact, of the intervention, on the particular outcome of interest (be it per capita GDP, R&D expenditure, housing values or employment levels).

Effects, impacts, and counterfactuals

A notation on terminology is necessary. Unlike in other evaluation settings, here impacts and effects are perfect synonyms. There is truly no meaningful difference between the two terms, they both refer to the notion of “causal effect”, the difference between the outcome occurred after an intervention has taken place and the outcome that would have occurred in the absence of the intervention. The popular distinction between “effects” as immediate results and “impacts” as long-run, or wider, effects, has no meaning on this context.

The counterfactual situation is purely hypothetical, thus can never be directly observed. For the same reason, an effect can never be directly observed, nor can an impact (impact indicators notwithstanding). By contrast, effects and impacts can be inferred, as long as the available data allows a credible way to approximate the counterfactual.

There are two basic ways to approximate the counterfactual: (i) using the outcome observed for non-beneficiaries; or (ii) using the outcome observed for beneficiaries before they are exposed to the intervention. However, caution must be used in interpreting these differences as the “effect” of the intervention.

The use of these basic comparisons is often invoked in the Guide. For example on page 31 “the provision of support to companies to invest in new equipments could be evaluated by tracking the performance of supported companies and comparing this with the performance of an appropriately identified control group of companies not receiving support.”

On page 144 the following example suggests using as counterfactual what is observed before the intervention: “The simplest method is to use the initial situation ("baseline") as the counterfactual. For example, 100 SMEs receive investment support, between them they increase their capital stock from EUR20 million to EUR30 million. In this simple scenario, EUR20m is the baseline and EUR30m-EUR20m = Eur10 million is the estimated impact of assistance."

Please visit the website link for the entire document.

Source: European Commission, Regional Policy website
http://ec.europa.eu/regional_policy/sources/docgener/evaluation/evalsed/...