Institutional Learning and Change Initiative

The following text is drawn from World Bank (2004) http://go.worldbank.org/J35S3J8B60 and  ISNAR (2003)

What is it?
Impact evaluation is the systematic identification of the effects – positive or negative,
intended or not – on individual households, institutions, and the environment caused
by a given development activity such as a program or project. Impact evaluation helps
us better understand the extent to which activities reach the poor and the magnitude of
their effects on people’s welfare. Impact evaluations can range from large scale sample
surveys in which project populations and control groups are compared before and after,
and possibly at several points during program intervention; to small-scale rapid assessment
and participatory appraisals where estimates of impact are obtained from combining
group interviews, key informants, case studies and available secondary data.

What can we use it for?

• Measuring outcomes and impacts of an activity and distinguishing these from the influence of other, external factors.
• Helping to clarify whether costs for an activity are justified.
• Informing decisions on whether to expand, modify or eliminate projects, programs
  or policies.
• Drawing lessons for improving the design and management of future activities.
• Comparing the effectiveness of alternative interventions.
• Strengthening accountability for results.
   

Advantages:

• Provides estimates of the magnitude of outcomes and impacts for different demographic groups, regions or over time.
• Provides answers to some of the most central development questions – to what
  extent are we making a difference? What are the results on the ground? How can we
  do better?
• Systematic analysis and rigor can give managers and policy-makers added confidence
  in decision-making.
   

Disavantages:

• Some approaches are very expensive and time-consuming, although faster and more economical approaches are also used.
• Reduced utility when decision-makers need information quickly.
• Difficulties in identifying an appropriate counter-factual.

Cost:
A number of World Bank impact evaluations have ranged from $200,000 - $900,000
depending on program size, complexity and data collection. Simpler and rapid impact
evaluations can be conducted for significantly less that $100,000 and in some cases for
as little as $10,000 - $20,000.
 

Skills required:
Strong technical skills in social science research design, management, analysis and
reporting. Ideally, a balance of quantitative and qualitative research skills on the part of
the evaluation team.
 

Time required:
Can take up to 2 years or more. Rapid assessment evaluations can often be conducted
in less than 6 months.

Examples of impact evaluation designs
Randomized evaluation designs, involving the collection of information on project and
control groups at two or more points in time, provide the most rigorous statistical
analysis of project impacts and the contribution of other factors. But in practice it is
rarely possible to use these designs for reasons of cost, time, methodological or ethical
constraints. Thus most impact evaluations use less expensive and less rigorous evaluation
designs. The following table describes four approaches to impact evaluation
designs in development evaluation. The first is an example of a randomized evaluation
design; the second is a quasi-experimental design in which a "non-equivalent" control
group is selected to match as closely as possible the characteristics of the project population; in the third example the project population is compared with a non-equivalent
control group after the project has been implemented; and the fourth is a rapid assessment
evaluation which combines group interviews, key informants, case studies and
secondary data. Each successive model sacrifices methodological rigor, in return from
which there are significant reductions in cost and time requirements.

 Image source: World Bank (2004). Monitoring and Evaluation: Some Tools, Methods & Approaches. Washington DC, The World Bank. http://lnweb90.worldbank.org/oed/oeddoclib.nsf/b57456d58aba40e585256ad400736404/a5efbb5d776b67d285256b1e0079c9a3/$FILE/MandE_tools_methods_approaches.pdf page 24.

Text source: World Bank (2004). Monitoring and Evaluation: Some Tools, Methods & Approaches. Washington DC, The World Bank. http://lnweb90.worldbank.org/oed/oeddoclib.nsf/b57456d58aba40e585256ad400736404/a5efbb5d776b67d285256b1e0079c9a3/$FILE/MandE_tools_methods_approaches.pdf

Impact Evaluation (IE) rigorously measures the impact that a project has on beneficiaries. It typically does this by comparing outcomes between beneficiaries and a control group, both before and after a project has been implemented.  Source: The World Bank http://go.worldbank.org/J35S3J8B60

Source: ISNAR (2003) Monitoring, Evaluation, and Impact Assessment of R&D Investments in Agriculture, The Hague: International Service for National Agricultural Research.

References:

Asian Development Bank (2006). Impact Evaluation: Methodological and Operational Issues. Philippines: Asian Development Bank. http://www.adb.org/Documents/Handbooks/Impact-Analysis/Impact-Analysis.pdf

Baker, J. (2000). Evaluating the Impact of Development Projects on Poverty: A Handbook for Practitioners. Washington, DC: World Bank. http://siteresources.worldbank.org/INTISPMA/Resources/handbook.pdf or http://www.worldbank.org/poverty/library/impact.htm

Bloom, H. S. (2006). The Core Analytics of Randomized Experiments for Social Research MDRC Working Papers on Research Methodology. New York: MDRC. http://www.mdrc.org/publications/437/full.pdf

Roche, C. (1999) Impact Assessment for Development Agencies: Learning to Value Change. Oxfam, Oxford.

Website links:

IFAD guide to Project Method and Evaluation http://www.ifad.org/evaluation/guide/index.htm

Impact website of the Consultative Group on International Agricultural Research (CGIAR) - http://impact.cgiar.org/ 

Impact Evaluation Handbook: A guide for Network for a Healthy California Local Incentive Awardees http://ww2.cdph.ca.gov/programs/CPNS/Documents/Research/CPNS-REU-Handbook-2007-2008.pdf

International Initiative for Impact Evaluation (3IE) http://www.cgdev.org/doc/eval%20gap/3IEfound_doc.pdf

Monitoring and Evaluation: some tools, methods and approaches. http://lnweb90.worldbank.org/oed/oeddoclib.nsf/b57456d58aba40e585256ad400736404/a5efbb5d776b67d285256b1e0079c9a3/$FILE/MandE_tools_methods_approaches.pdf

World Bank Web site on impact evaluation: http://www.worldbank.org/poverty/impact/