Home

December 2008 Edition

Thu, 12/11/2008 - 15:55 — Cristina Sette

News from the ILAC Initiative:

  • Follow up Rethinking Impact Workshop
  • ILAC Learning Laboratory Activities
  • 'Group Facilitation Skills for Participatory Decision-Making' Workshop
  • New at the ILAC Website (www.cgiar-ilac.org)
  • New ILAC briefs

News and Events from the Broader Community:

  • European Evaluation Society Conference (Contribution from Douglas Horton, Evaluation Specialist)
  • CGIAR Impact Assessment Focal Point (IAFP) Group and Standing Panel for Impact Assessment (SPIA) Meeting: “Defining and Refining Good Practice in Ex-post Impact Assessment” (Contribution from Jamie Watts)
  • International Workshop on Methodological Innovations in Impact Assessment of Agricultural Research
  • Annual conference of the American Evaluation Association, Denver, Colorado (Contribution from Douglas Horton, Evaluation Specialist)
  • Upcoming Events

 News from the ILAC Initiative:

Follow up Rethinking Impact Workshop

Since the workshop on ‘Rethinking Impact (RI): Understanding the Complexity of Poverty and Change,’ organized by the Systemwide Program on Participatory Research and Gender Analysis for Technology Development and Institutional Innovation (PRGA Program), ILRI's Innovation Works, and the ILAC Initiative, there has been follow-through on a number of initiatives including:

  • Presentation of findings and policy issues at the CGIAR Alliance Deputy Executive (ADE) mid-year meeting
  • Presentation at the American Evaluation Association Conference by Douglas Horton drawing on content from the Workshop
  • Collaboration between ILAC, Research into Use (RIU) and the Royal Melbourne Institute of Technology (RMIT) to develop a project on impact evaluation. Funding was provided by RIU for the scoping phase.
  • Co-organizers are in discussion with international peer-reviewed journals concerning a special issue based on the RI workshop presentations and outcomes.

For more information on these activities and the Rethinking Impact workshop, go to Workshop Newsletter Issue # 5 at http://www.prgaprogram.org/riw/outputs.htm

ILAC Learning Laboratory Activities

  • ILAC sponsored the attendance of 3 participants from the Kenya Smallholder Dairy Project to the workshop on Social Network Analysis, delivered by Dr. Eva Schiffer. Participants learned about the use of software programs designed to display actor-relationships graphically and to illustrate patterns of relationships. Participants were also exposed to NetMap tools for understanding and visualizing the range of actors involved in any given network. More information on those tools at http://netmap.wordpress.com/
  • ILAC sponsored a workshop on Planning Policy Influence, hosted by the Overseas Development Institute, London, UK between 24 and 26 November 2008. The purpose of the workshop was to develop capacity for planning, monitoring and evaluating policy-influencing interventions using the RAPID Outcome Mapping Approach (ROMA). There were 13 participants from four projects. The projects were (i) Cambio Andino Program, (ii) Kenya Smallholder Dairy Project, (iii) Knowledge and Monitoring System of SG2000 Innovations in Africa Project, and (iv) Generating new knowledge on Knowledge Intensive Agricultural Systems (KIA) Project.
  • Xavier Institute of Management Bhubaneswar (XIMB) in association with Dept. of Agriculture (Govt. of Orissa), Centre for World Solidarity (CWS-ORC), ILAC and WWF organized a two day learning alliance workshop of various stakeholders for a mutual enrichment on various aspects related to System of Rice Intensification (SRI) practices. The participants included Govt. officials from agriculture department, extension officers, academicians, NGOS, farmers, management students and private sector organization. These 132 participants brought with them diverse sets of learning, innovative practices and new perspectives to be discussed on a common platform along with many issues that need to be cross examined. More information can be found at http://sri-learning-alliance.blogspot.com/

You can find out more about the Learning Laboratory activities and the participants, by visiting the link http://www.cgiar-ilac.org/content/learning-laboratory

'Group Facilitation Skills for Participatory Decision-Making' Workshop

The 2009 training workshop will take place in Nairobi, Kenya, from 26 to 29 May. The course is being hosted by ICRAF. This important training helps build the essential knowledge, attitudes and skills that project managers and team leaders need to engage in effective participatory decision making. The course will be delivered by Dr. Sarah Fisk, of Community at Work (San Francisco, California). More information at http://www.cgiar-ilac.org/content/next-training-facilitation

New at the ILAC Website (www.cgiar-ilac.org)

ILAC has developed a new website with innovative, user-friendly tools.
In the website you will find several sections, among them are:

  • ILAC publications (free download)
  • Library (with over 1000 references on partnership, collaborative research, PME, impact assessment, and learning
  • Coming events
  • ILAC newsletter subscription
  • ILAC Forum (a blog for discussion related to institutional learning and change)

To access ILAC publications /or to post on the blog, first subscribe to the website following these simple steps:

  1.  Go to www.cgiar-ilac.org
  2. To create a new account choose a username and password at bottom right ‘user login’.
  3. Alternatively go to link http://www.cgiar-ilac.org/user/register
  4. Once your account has been approved, you can download our publications and start posting your comments in our blog http://www.cgiar-ilac.org/forum/8 .

For those who have an account already, simply login and enjoy the site. Comments and suggestions for improving the site are welcome.

New ILAC briefs

 

 

 

All of these and many other publications are available for free download on the ILAC website (www.cgiar-ilac.org). A limited number of paper copies may be obtained upon request. Contact ilac@cgiar.org for more information.

News and Events from the Broader Community

European Evaluation Society Conference (Contribution from Douglas Horton, Evaluation Specialist)

The European Evaluation Society held its biennial conference in September in Lisbon, Portugal. The conference, attended by several hundred evaluators from over 50 countries, included two and one-half days of plenary and concurrent sessions and two days of pre-conference sessions on a broad range of evaluation topics. Some unique features:

  • The participation of greater numbers of evaluators from outside of Europe -- from all regions of the world.
  • "Development evaluation," was highlighted -- previous conferences had focused more on evaluation practices within Europe.
  • Much discussion of methods and evidence for impact evaluation

More information can be found at http://www.europeanevaluation.org/

CGIAR Impact Assessment Focal Point (IAFP) Group and Standing Panel for Impact Assessment (SPIA) Meeting: “Defining and Refining Good Practice in Ex-post Impact Assessment” (Contribution from Jamie Watts)

The IAFP-SPIA meeting took place in Brasilia Brazil, from 10 to 11 November 2008, The meeting was hosted by EMBRAPA and had the following objectives:

  • To foster communication among the CGIAR centers on methods and approaches to ex post impact assessment
  • To contribute to a community of practitioners within and outside the system to exchange knowledge and good practice on epIA for international agricultural research
  • To receive feedback on ongoing and proposed activities of SPIA
  • To make recommendations for future priorities of SPIA & Centers with reference to ex post IA

ILACers Jamie Watts, John Dixon, Roberto La Rovere, Patricia Rogers, among 34 other participants which included CGIAR Centers, Universities, and donor organizations attended the meeting.

The meeting focused on practical problem solving related to impact assessment. Issues raised included attribution in partnership context, methods for assessing capacity development, empowerment, and impact assessment in complex interventions. Participants raised concerns about the way the work of the CGIAR has been and is assessed and showed interest in how the change management initiative in which the CGIAR is currently undergoing may influence practices. Matters of staffing, competency, and funding required to assess a wider range of impacts than those traditionally addressed, were also discussed.

The meeting’s agenda can be found at http://www.sciencecouncil.cgiar.org/fileadmin/user_upload/sciencecouncil/Impact_Assessment/IAFP_Meeting_Agenda_-_7_November_AS.pdf

International Workshop on Methodological Innovations in Impact Assessment of Agricultural Research

The Empresa Brasileira de Pesquisa Agropecuária (Embrapa), the Universidade de Campinas (UNICAMP) and the CGIAR Standing Panel on Impact Assessment (SPIA) organized a workshop on Methodological Innovations in Impact Assessment of Agricultural Research in Brasilia, Brazil from 11 to 14 November 2008.
Around 150 professionals from EMBRAPA, Universities, the CGIAR, and National Research Systems participated in the event.
The initiative is part of a US$ 60 million project called ‘Technological Innovation and New Management Approaches in Agricultural Research AGROFUTURO, funded by the Inter-American Development Bank (http://www.iadb.org/projects/project.cfm?id=BR-L1001&lang=en)

Annual conference of the American Evaluation Association, Denver, Colorado (Contribution from Douglas Horton, Evaluation Specialist)

There were a number of sessions at this year’s AEA conference on performance measurement and issues of “strong evidence” for decision making in governmental bodies. Most of these related to the US government, but there were also presentations on evaluation in other regions. As similar issues are being discussed in the context of international development, some highlights of these sessions will be of interest to readers of the ILAC Newsletter.
The Bush administration introduced the “Program Assessment and Rating Tool” (PART) in an attempt to have standardized measures of program performance that could be used to guide budget decisions in the White House and Congress (http://www.whitehouse.gov/omb/part). The ultimate stated goal was to improve program accountability and performance. An external group led the development of the rating tool. Government agencies have been required to implement the tool for each of their programs and submit the results to the Office of Management and Budget (OMB), which then reviews the submissions and makes recommendations for budget allocations. Presentations at the AEA – some based on personal experience, others on extensive research -- indicate that the PART goals have not been achieved, and that numerous unexpected consequences have resulted. The agencies have found it very costly and difficult to comply with PART and the OMB has found it costly and difficult to review the submissions. There is little evidence of use of the results by the agencies, the OMB or congress. One unexpected, notable result of the PART has been to reduce the use of program evaluation oriented toward program improvement.

Kathryn Newcomer, of George Washington University, presented results of a research study she and others have conducted on PART, which has been issued by the National Association of Public Administration. Some conclusions:

  •  Concerns for "government effectiveness" and performance management date back several years (i.e., are not unique to the Bush administration.
  • The systems implemented to measure and improve performance have not achieved their goals, and have had major unintended consequences.
  • OMB was “saddled” with the PART as a “no-cost add on” and has not had adequate resources to implement it. PART has proved very costly and frustrating for federal agencies to implement, for various reasons (unclear goals, complexity of the programs and the targets, multiple levels of implementation, multiple partners involved, long time lags, inadequate resources for massive data collection, uneven application of the scoring system, political influence ...)
  • Resources in the federal government for evaluation have declined over the past decade or so. As compliance with PART requires considerable evaluation resources, introduction of the system has diverted the use of scarce resources from improvement-oriented program evaluation into accountability-oriented performance measurement.
  • The PART rates the adequacy of evidence of program performance, and specifies randomized controlled trials (RCTs) as the preferred method for generating "strong evidence" of program effectiveness. However, it has proven difficult for many agencies (especially those engaged in R&D) to carry them out. Use of “questionable” or “inadequate” evaluation methods leads to “reduced scores” in the PART system.
  • There's very little evidence of use of results either by the programs, agencies, the OMB or Congress.
  • An "accountability culture" concerned mainly with "compliance" and "legitimization" has crowded out an "evaluation culture" concerned with learning and improvement in governmental agencies.
  • A main recommendation of the analysis is to revive use of program evaluations that promote learning and program improvement

Presentations on performance measurement and evaluation in Denmark and the Netherlands referred to “evaluation machines" and "Frankenstein evaluation monsters," that were characterized as having been designed hurriedly and with little understanding of evaluation and its potential uses/consequences, requiring compliance throughout large governmental systems, absorbing massive amounts of resources, and generating results that are generally not used for decision-making.

Recently, the OMB requested a statement from the AEA on what constitutes strong evidence of program effectiveness. The statement is available at this link:

http://www.eval.org/EPTF.asp#EPTF_Response_to_OMB_on_What_Constitutes_Strong_Evidence_of_a_Programs_Effectiveness

The European Evaluation Society has prepared a similar statement, which is available on the ILAC website (search for “EES statement”).

Upcoming Events

 

We would appreciate your feedback on the newsletter, so that we can better serve your interests and concerns. So please let us know what you like and what you feel we can improve on. If you are not yet on our mailing list but would like to be, please just send us an email. For further information about the ILAC Initiative, please contact:
Cristina Sette,
ILAC Programme Specialist
ilac@cgiar.org
www.cgiar-ilac.org

 

Bookmark/Search this post with: