Institutionalizing impact assessment at CIMMYT

by Roberto La Rovere, John Dixon and Jonathan Hellin

The International Center for Maize and Wheat Improvement (CIMMYT) is a CGIAR centre with a long tradition of impact assessment (IA). Until recently, such work had focused on the rates of adoption of improved germplasm and returns on investment. Limited attention was directed at measuring impacts in terms of livelihood security or poverty reduction. In 2005, therefore, CIMMYT's Impacts Targeting and Assessment Unit (ITAU) began to develop and promote more wide-ranging types of IA.
In order to enhance its IA culture and institutionalize IA within the Center, ITAU organized a series of workshops that brought together different players amongst CIMMYT's staff and partners. Coupled with follow-up activities, these have made it possible for social and biophysical scientists and other partners to share their understanding of IA and their IA experiences. This brief describes the key elements of the process that led to the establishment of a learning platform for a CIMMYT-wide IA culture, as well as ways to consolidate achievements and progress on institutionalizing IA. It also states the lessons learned with regard to institutionalizing IA within the context of an international agricultural research centre.

Introduction

There is an outstanding tradition of impact assessment at CIMMYT (Pingali, 2001) focusing on the assessment of adoption rates and the economic rates of return from investments in crop improvement. Less attention had been directed at assessing impact in terms of poverty reduction and livelihood security.
The 'Seeds of Innovation' vision (CIMMYT, 2004) emphasized people-centred, livelihoods- and poverty-oriented, systems approaches to research. CIMMYT's Business Plan for 2006-10 (CIMMYT, 2006) states that IA should assess a broader range of impacts, by including aspects of poverty, vulnerability, and the distribution of benefits. Direct and indirect impacts arising from linkages within farming systems and agriculture and the non-farm economy should also be recognized. Finally, the new strategy stated that a diversity of stakeholders should be embraced, each having somewhat different expectations from IA. CIMMYT therefore sought to develop, institutionalize, and promote a more wide-ranging type of IA.
Impact assessment is a type of evaluation linked to a range of activities, including ex-ante studies to predict impact, monitoring of accomplishments, and end-of-project evaluations. Its importance is becoming more apparent as funding declines and scientists and management seek to justify its effective use and demonstrate reductions in poverty.
Key issues in effective IA include: (1) identifying a realistic counterfactual, or effective and credible substitutes for it; (2) ensuring optimum timing of the assessment; (3) establishing
appropriate spatial dimensions for the assessment; and (4) properly attributing effects and impacts in complex, multiplayer partnerships.
Emerging challenges for IA include the assessment of research involving natural resource management, policy, participatory approaches, the social sciences, capacity building
and the contributions of diverse stakeholders. CIMMYT also seeks to ensure that IA contributes to effective staff and institutional learning, in order to improve future work.
The institutionalization of IA processes in CIMMYT faces a range of challenges, including (1) the need to develop credible methods to measure a broader array of impacts; (2) the need to identify mechanisms to ensure adequate engagement of staff and partners, in order to achieve the learning goals of IA; and (3) the need to 'package' and communicate results to a range of users.

The elements of CIMMYT's IA learning and operational platform

In May 2005 ITAU led a CIMMYT-wide workshop, co-funded by the Institutional Learning and Change initiative (ILAC), to develop a learning and operational platform for IA and
institutionalize IA practice within CIMMYT and its partners.
Its objectives included the:

• establishment of a multi-disciplinary learning and operational support platform across research
  programmes for relevant, high-quality IA from the
  project to global levels.
• development of a people-centred working framework
  for CIMMYT IA, focusing on systems, livelihoods, and
  poverty reduction.
• strengthening of individual skills and capacity for
  high-performance team research.
   

The workshop (Box 1) was the first of a series of events which raised staff's awareness of the importance of IA and offered tools to undertake IA. It also resulted in a range of
follow-up activities (Table 1).

At the 2005 workshop, it was not as important for participants to reach a consensus as it was for them to learn from and understand each other. The process was facilitated in
order to foster buy-in from participants, and plenary and small-group sessions were held.
An IA framework for use by practitioners (Box 2) was drawn up and discussed in the workshop (for details see La Rovere and Dixon, 2007). Participants then used this to
discuss a set of case studies. These described intended uses and users of information generated; the potential stakeholders involved; topics to be assessed; scope, boundaries, and critical questions; the disciplinary expertise required and ways to mobilize it; and use of results.

These steps are sequential and related. The process involves negotiated planning, as users must be involved to understand the findings and put them to use. Much of an IA's
value, in fact, comes from an active involvement in learning while grappling with real problems.
The case studies included one on potential impacts of the spread of a new strain of wheat stem rust (Box 3). This showed the direct impacts of stem rust on yield, grain quality, and grain prices, and ultimately on the local economy, food security, livelihoods, and in some cases, the national economy and global markets (see La Rovere and Dixon, 2005, for details).

The workshop stimulated a continuous learning process on the effective implementation of IA and integrated work among CIMMYT programmes, which strengthened the IA capacity of scientists from regional and cross-disciplinary programmes. According to participants,
one of the workshop's key benefits was the opportunity to reflect on IA experiences; on individual, programme, and regional capacities for IA; and on their understanding and definitions of 'impact'. The process, outcomes and need for follow up were shared in a workshop report (La Rovere and Dixon, 2005).
The 2005 inception workshop was followed by a series of IArelated activities (Table 1). In 2006, project-based IA training courses were conducted in Africa and Asia. The whole process has been supplemented by e-mail discussions and the sharing of information and documents among participants, as well as the identification of IA focal points.
Follow-up activities included the development of multidisciplinary papers on IA-relevant and non-IA-relevant topics, joint activities for social and biophysical scientists, steps to institutionalize IA in proposals and ensure its inclusion in work plans, and more explicit actions being taken (in partnership with CIMMYT management) to seek funding for IA.
 

Follow-up workshops

Early progress on the actions agreed upon in the inception workshop was assessed during a follow-up workshop in Rome in October 2005.
This brought together social scientists from headquarters and regional programmes, other CGIAR centres (Bioversity International, IFPRI), and a representative from the CGIAR Standing Panel on Impact Assessment (SPIA). The Rome workshop provided a comprehensive overview of ongoing CIMMYT IA activities in the areas of breeding and maintenance research, ex-ante studies, monitoring and evaluation of technology use
and adoption, and ex-post studies.

The workshop resulted in a series of studies that reflect more explicitly the broader focus of IA at CIMMYT, by integrating conventional economic approaches (surveys, surplus analysis, econometrics), qualitative assessment tools, and livelihood approaches.
Take-home messages for CIMMYT included the benefits of looking back and building on past achievements and successes, moving to broader livelihood, systems, and poverty integrated approaches, and integrating much more closely with other disciplines than in the past, while maintaining a strong emphasis on economics. Finally, participants
flagged the need to reduce the use of IA jargon and provide guidelines application of IA. A workshop, held at CIMMYT in late 2006, followed up a CGIAR recommendation that the impact pathways of projects be identified in the Center's Medium-Term Plan (Douthwaite et al., 2003).

What was learned and what changes have occurred?

The process of assessing the learning and change that has occurred as a result of these activities is on-going. A follow-up survey helped to better understand how participants increased their knowledge of IA and how useful the process had been. Progress was also monitored using learning and change indicators. Examples of such include whether or
not those most involved in IA were developing into a coherent team, whether lessons were being applied, whether those facilitating the process were providing sufficient operational support, and whether or not adequate incentives exist to make learning on IA a priority.
Most people felt that they had gained a better understanding and appreciation of the principles of IA and their practical uses.
Participants did, however, feel that they needed guidance (1) on defining relevant datasets and the key variables for use in an IA, (2) on how to prioritize studies that require IA, and (3) on ways to budget for an IA when developing proposals. A major challenge for scientists from all disciplines was that a general lack of time hampered the consolidation and implementation of IA knowledge. The transformation of learning into actual action is still hampered by a generalized lack of time, despite the significant investment and commitment made by CIMMYT in this process in the form of meetings, costs, workshops, staff time and
communication.
Incentives are also important: scientists and partners can be further encouraged to embed their work within an IA lens and livelihoods/poverty-reduction perspective by promoting IA as a key element in resource mobilization. Indeed, research hypotheses in project proposals are now often being formulated by scientists based on the IA framework. This may be the result of a genuine understanding that more IA is needed. However, it may also result from the researchers' recognition that they need to include clear reference to poverty
reduction in their proposals to increase their chances of getting funding.
By the time the mid-term assessment was conducted, it appeared that not everyone had put the lessons learned into practice.
People react differently to experimentation and change; some immediately feel comfortable with a new process, while others need more time to assimilate it. There is also a tendency for some scientists to see IA and the formulation of IA hypotheses as mainly the
responsibility of social scientists.
Behavioural change, particularly for project managers, was often reflected in the clearer attention paid to IA in projects, through the use of more commonly understood language in discussions with colleagues, and during the development of new Medium-Term and Business Plans— in which staff wrote impact statements based on impact pathways
workshop outputs. One concrete change was the institutionalization of templates for project proposals that contain IA budget lines. However, funding is a major obstacle for non-social scientists who want to include IA into their projects, as IA is among the 'desirable' components that often get cut when donors ask for budgets to be reduced.
The follow-up agreed at the workshops was implemented gradually, which is typical because this type of learning takes time and effort. However, there is growing acceptance of the technical and institutional complexity of IA and the key role it plays in supporting
strategic priority-setting. This is testified to by the fact that there is increasing recognition of the need to increase the number of ex-ante IA studies, as well as an increasing demand for ex-ante studies within CIMMYT.
It was important to examine the IA process openly in order to learn from both successful and less successful outcomes. The work done did raise the profile of IA within CIMMYT and initiated an institutionwide learning process. As a result, biophysical and social scientists
within the organization are now more aware of IA and are more interested in what it can mean for them. Moreover, more and more often IA is being developed in an issue-oriented (rather than commodityoriented) fashion. This is broadening what was an earlier focus on crops and adoption to cover different aspects of livelihoods, and different types of impact, while maintaining the elements of robust economic analyses.
Examples of this approach to IA are becoming more frequent and include the following: a study on livelihoods and the economic impact of research on maize in the mid-hills of Nepal; a livelihood impact study of maize diversity projects in Mexico; use of a poverty and
livelihoods explicit baseline and monitoring system for assessing the medium-term impacts of long-term interventions in eastern Africa; and the targeting of poverty-alleviation activities in the Indo-Gangetic Plains of north India.

Conclusions, implications and the way ahead

The process described in this brief generated a wealth of ideas that will take time to be fully implemented. And, though both CIMMYT scientists and the Center's partners have been improving their understanding of IA, there is still a need for:

• Regularly conducted and expanded baseline studies that go beyond maize and wheat simply as commodities and that embrace the livelihood systems associated with them An increased use of qualitative methods, or mix of quantitative and qualitative methods
• More attention to the issue of attribution (Mayne, 2008)
• Greater efforts to measure and account for unintended, positive and negative impacts
• A recognition of the fact that the design and implementation of IA methodologies has become critical and that no unique format exists that will fit all needs or demands
• Early agreement on the role that IA will play in projects, in order to allow for IA to be properly budgeted for Internal evaluators with a formal mandate to support
  institutional learning and change
• Better communication of IA-related outcomes
• New modalities for cooperation among partners on IA.
   

In order to follow up on what has been achieved to date, ITAU plans to implement a comprehensive web-based survey of CIMMYT staff and partners. This will assess the influence of internal and external learning processes, the advancement of an IA culture, and the institutionalization of IA, so summing up achievements and the lessons learned over a five-year period following the Center's strategic shift in IA initiated in 2005.
The survey will investigate what further lessons can be learned, what could have been done differently, and what should be repeated and/or strengthened in the future. It will also assess whether more scientists now recognize the role and importance of IA and the extent
to which this awareness extends to senior management. It will also be used to compare recent research projects with those run in the past, in order to determine whether the former focus on yields, productivity and economic indicators has been enriched as a result of adding into the equation an assessment of the impact on people's livelihoods.
In conclusion, the experiences of CIMMYT in enriching and institutionalizing IA can be relevant and directly useful to other centres that want to undertake a similar process.

Further reading

CIMMYT. 2004. Seeds of Innovation: CIMMYT Strategy for Helping to Reduce Poverty and Hunger by 2020. Mexico, D.F.: CIMMYT. www.cimmyt.org/english/docs/special_publ/splan/contents.htm
CIMMYT. 2006. CIMMYT Business Plan 2006-2010. Translating the Vision of Seeds of Innovation into a Vibrant Work Plan. Mexico, D.F.: CIMMYT.
Douthwaite, B., Kuby, T., van de Fliert, E. and Schulz, S. 2003. Impact pathway evaluation: an approach for achieving and attributing impact in complex systems. Agricultural Systems 78:243-265.
Horton, D. and Mackay, R. 2003. Using evaluation to enhance institutional learning and change: recent experience with agricultural research and development. Agricultural Systems 78: 127-142.
Mayne, J. 2008. Contribution analysis: an approach to exploring cause-effect questions. ILAC Brief No. 16. Rome: Institutional Learning and Change Initiative.
Patton, M.Q. 1995. Utilization-Focused Evaluation: the New Century Text. Thousand Oaks, CA, USA: Sage.
La Rovere, R. and Dixon, J. 2005. Report on the Impact Assessment workshop held at CIMMYT, Mexico, May 3-6th. www.cgiar-ilac.org/downloads/Reports/REPORT_ITA_Workshop_3-6_May_2005.pdf
La Rovere, R. and Dixon, J. 2007. Operational Guidelines for Assessing Impact of Agricultural Research on Livelihoods: Good Practices from CIMMYT. Mexico, D.F.: CIMMYT. www.cimmyt.org/english/docs/manual/ia/contents.htm
Mackay, R. and Horton, D. 2003. Expanding the use of impact assessment and evaluation in agricultural research and development. Agricultural Systems 78: 143-165.
Pingali, P. 2001. Milestones in Impact Assessment Research in the CGIAR, 1970-1999. Mexico, D.F.: CIMMYT.
Watts, J. 2005. Learning-oriented evaluation: a tool for promoting institutional learning and program improvement. ILAC Brief No. 3. www.cgiar-ilac.org/downloads/Brief3Proof2.pdf

About the authors

Roberto La Rovere (r.larovere@cgiar.org) is an Impact Specialist, John Dixon (j.dixon@cgiar.org) is a Director and Jonathan Hellin (j.hellin@cgiar.org) is a Poverty Specialist at the International Maize and Wheat Improvement Center (CIMMYT), El Batan, Mexico.