- Home
- About us
- ILAC Activities
- Resources
- ILAC Publications
- ILAC Briefs
- Working Papers
- ILAC Sourcebook
- Part 2 - Tools and approaches
- Chapter 6: Introduction Part 2
- Chapter 7: Innovation histories
- Chapter 9: Culture study as a tool for change
- Chapter 12: Collaborative agreements
- Chapter 13: Facilitation as a foundation skill for ILAC
- Chapter 14: Learning alliances
- Chapter 15: Institutional histories
- Chapter 16: Engaging Scientists through Institutional Histories
- Chapter 18: Horizontal evaluation
- Chapter 19: Appreciative inquiry
- Part 3 - Cases and experiences
- Part 4 - Challenges and strategies
- About the authors
- Glossary
- References
- Part 2 - Tools and approaches
- Reports
- Journal Articles
- Newsletter
- Presentations by ILAC Team
- ILAC Library
- Tools and methods for M&E
- Appreciative inquiry
- Biophysical measurements
- Case study
- Content analysis
- Contribution analysis
- Cost-Benefit Analysis
- Creative expression
- Diaries, journals and logs
- Dreams realised or visioning
- Expert review
- GIS mapping
- Graphing results
- Group assessment
- Historical trends and timelines
- Horizontal evaluation
- Impact evaluation
- Benefit-cost methods
- Case studies
- Cost-effectiveness analysis
- Counterfactual Impact Evaluation (CIE)
- Difference-in-difference
- Econometric methods
- Ex-post comparison of project beneficiaries with a control group
- Instrumental variables
- Integrated partial indicators
- Mathematical programming
- Modified peer review
- Partial indicators of impact
- Patent analysis
- Pipeline comparison
- Production function approach
- Propensity score matching
- Quasi-experimental design, involving the use of matched control and project groups
- Randomization
- Rapid assessment or review, conducted ex post
- Regression discontinuity design
- Simulation method
- User surveys
- Impact flow diagram
- Innovation histories
- Institutional history
- Institutional linkage diagram
- Interviews
- Learning alliances
- Learning-oriented evaluation
- M&E Frameworks
- M&E wheel (or "spider web")
- Mapping (sketch)
- Matrix scoring
- Most significant change
- Net-Map
- Non-random sampling
- Observation
- Outcome mapping
- Participatory Impact Pathways Analysis
- Participatory methods
- Performance indicators
- Photographs and video
- Problem and objectives trees
- Random sampling
- Ranking and pocket charts
- Rapid appraisal methods
- Relative scales or ladders
- Rich pictures (or mind maps)
- SWOT
- Seasonal calendars
- Semantic differentials
- Social mapping or well-being ranking
- Sociograms
- Stakeholder analysis
- Survey
- Systems (or inputs-outputs) diagram
- Theory-Based Evaluation
- Transects
- Evaluation studies and reports
- Evaluating capacity development
- Partnership
- Seminars
- Video Room
- Links to other sites
- E-learning courses
- ILAC Publications
- Contact
- Blog
What is Institutional Learning?
Wed, 07/30/2008 - 13:12 — Cristina Sette
Dear ILAC Learners:
The Standing Panel on Impact Assessment of the CGIAR Science Council has contracted Leslie Cooksy to do a scoping study on the use of ex-post impact assessment for purposes of learning and program improvement. Leslie is a professional evaluator from the US who has collaborated in the past with some evaluation work of the Science Council. She just sent me this message to me, and I thought it would be a good idea to stimulate some discussion within our group by requesting YOUR inputs on the definition of "institutional learning." The results will be of use not only to Leslie but to us as a "learning group."
Please send me your quick reactions to Leslie's definition below, and I'll prepare a response to Leslie that reflects your thoughts. I'll circulate the consolidated response as well to all of you.
Thanks in advance.
Doug
Begin forwarded message:
From: Leslie Cooksy <ljcooksy@udel.edu>
Date: July 14, 2008 11:14:34 AM EDT
To: Douglas Horton <d.horton@mac.com>
Subject: quick check
Hi Doug,
Feel free to ignore this if you are crunched for time, but I am trying to give a brief definition of institutional learning and I want to make sure I’m capturing the essence. Later in the report, I’ll get to go into more detail about what it looks like, but this is just intended to orient the reader. Let me know if there is anything in here that raises a flag for you. (I know that I am limiting it to “evaluative” information, which is not necessarily the case in general, but that seems appropriate given the focus of the report.)
Thanks for your thoughts. I hope all is well!
~~Leslie
Institutional or organizational learning refers to an ongoing process in which evaluative information on research activities and outcomes feeds into a reflective analysis of what has worked and not worked. In turn, the lessons learned from the reflective analysis inform decisions about future directions (ILAC, 2005; Torres & Preskill, 2001). A learning organization is one with a culture that supports this kind of analysis and change. “Institution” is used instead of “organization” when referring to the learning process that takes place across organizations and among the diverse set of people involved in research (ILAC, 2005). In this report, institutional or organizational “learning” refers to the whole feedback loop of gathering and reflecting on information, and using the information to make changes at a variety of levels, i.e., from the focus of a specific strand of research to the communication mechanisms used to interact with partners.
Leslie J. Cooksy, Ph.D.
Graduate Program in Evaluation
School of Education & DE Education R&D Center
108 Pearson Hall
University of Delaware
Newark, DE 19711
302-831-6872
ljcooksy@udel.edu
- Cristina Sette's blog
- Login or register to post comments
- Email this page
Comments
Jamie's contribution
Dear all:
I am glad to see upon returning from two weeks away, this interesting conversation about institutional learning. Thanks to all of you for contributing and to Doug for leading the effort. Sorry I was not reading my emails to contribute to it in time to send to Leslie, but here are a few thoughts.
The first question would be, can we disconnect organizational from institutional learning? I would agree that we should not treat them as if they are the same thing. But would we agree that we need to be concerned with both “levels”? Can we have learning within the institutions with which we are involved (we meaning agric research for development broadly defined) without having the individual organizations that make up at least part of the institutional package being learning organizations themselves? .
The concept of organizational learning might be more tangible and manageable to some managers that run the organizations we are most directly concerned with (IARCs, NARIs, even NGOs). We need to be careful about how we communicate because for many people, discussion of the intricacies of institution vs organization would definitely be enough to cause a headache (or induce sleep). Thus from a communications stand point, there could be benefits in using the entry point of organizational learning in order to get at issues of the broader institutions. This is how I have been seeing it, but I could well be way off course
Also, I would agree with those that pointed out a cautionary note in terms of starting with definitions in the particular report that Leslie is working on now. We have been following the development of this study for some time now (years?). In commenting on the study we have argued for broadening not narrowing our thinking in terms of what learning potential exists for whatever broad range of actors through evaluation processes broadly defined. Others might prefer to narrow the field of view (either by narrowing in terms of institution or organization, narrowing the range of actors who might learn, narrowing the type of evaluation considered, narrowing the definition of learning etc), which of course result in more limited opportunities for learning. If we start off with strict definitions, we might really miss some important opportunities.
All the best, Jamie
Julius's two cents
Dear All,
I know this is rather late for Leslie’s concerns, and I have found all the contributions made so far quite educational and very illuminating.
I remember someone saying that an institution is more ‘a way of thinking that influences attitude, behavior and practice’. If learning is to make sense about where we are coming from and where and how we ought to go, then that is the level (institutional) at which it must happen.
What we need to keep in mind that ‘organizations’ are institutions that are defined more by a physical sense of belonging (spatially or through some kind of administrative engagement).
My two cents.
Cheers,
Julius
Lack of focus
Dear Jamie and all:
I work for CIAT and the CPWF and hence spend a lot of time being beaten up about lack of focus. Just so you understand my bias.
My preference would be for ILAC to focus on institutional learning and change as a subset of the larger area of organizational learning and change. I would like to see ILAC focus on how evaluation (quite broadly defined) can be used to achieve institutional learning and change (double-loop learning, social learning).
I think this can be described in a straightforward and understandable way. Something along the lines of : "People act based on their mental models of how the world works, so if you want to change the way people behave you need to interact with these mental models, and this is what ILAC does, with respect to changes that make research likely to produce pro-poor outcomes. The main strategy ILAC uses is reflection based on evaluative information" Or something like this.
In realistic evaluation terms, the mechanism ILAC seeks to trigger is institutional learning and change (because this changes peoples mental models and the way they behave). We develop a family of strategies to do this based on evaluation / reflection in different contexts.
Cheers, Boru
Contribution from Lorna
This is a rejoinder to Boru's bit about "institutional" and "organizational". I’d like to make 3 points.
First, I don’t think it’s useful to start the definition with "institutional or organizational learning...". It would be better to use one or the other term.
Second, the phrase "institutional learning" actually carries a tension between:
a) institutional, which connotes something entrenched
b) learning and change, which connotes a flux or a dynamic
However, it should be a good tension. ILAC is a process of improvement from a status quo. "Learning" connotes a change that comes with an updating of information available on which decisions and actions are based. Agents "learn" because information about outcomes of actions is updated as more experience is gained. Learning can also lead to an updating of the "benchmark" or the "standards" against which improvement is gauged.
Finally, I hope the definition doesn't become too "research-centric" or "research-organization-centric". I am looking for something that other partners can relate to; something more general, but operational. What I envision is learning together with our partners (communities, local government units, NGOs, even private citizens). It would be sad if the learning process is experienced or perceived only by the research organization in their (our) own bubble.
Lorna Sister
change mental models
Boru,
You make very good points. Your own draft description of what ILAC is doing is an interesting one and deserves further discussion:
"People act based on their mental models of how the world works, so if you want to change the way people behave you need to interact with these mental models, and this is what ILAC does, with respect to changes that make research likely to produce pro-poor outcomes. The main strategy ILAC uses is reflection based on evaluative information"
My own feeling is that focusing on the terms "organization" and "institution" is not all that necessary when thinking about what ILAC is trying to do. You use the word "world" in the first sentence -- and that may be more to the point. We are pursuing ways of bringing about system change ("world" is fine too but maybe not part of the terminology of the organizational sciences). Learning at both the organizational and the institutional levels may be necessary if we are to succeed at changing systems.
Your comment about mental models and changing these mental models gets at the locus of change -- i.e. the mental models, views of how the world works, or the "constructed realities" of all the actors in the system. But how can this happen? This takes us back to the earlier thread when we were talking about ways of reflecting and ways of learning.
If ILAC's task is to change mental models the issue is "Whose?" and "How can this happen?"
The answer to whose mental models have to change seems to be -- the mental models of all the actors within a given "system". We might take the "system(s)" that ILAC is working with as the "whole" of each of the Learning Lab projects -- and that "whole" has been captured or should be captured in each of the LL's workbooks. The actors whose mental models need to change are represented in the network maps. But what does changing their mental models mean? It would seem to mean (i) first of all teasing out -- you use the term "surfacing" their current view of the system and secondly (ii) using information generated by the individual LL project and from other sources (sucha s all of the LL projects) to construct a common reality -- a mental model that all actors are willing to share because it makes more sense than their earlier individual mental models.
There seem to be (at least) two significant challenges to achieving this. The first is to lay out the mental models of such a wide variety of actors -- these models will vary enormously depending on the disciplinary background of the actor, and the extent to which their models are informed, incomplete, partial, internally consistent, etc.
The second major challenge is to find a way to encourage all of the actors to modify or change or "grow" their various mental models in the direction of a common one that is a better representation of the system that what each possessed before.
We know that mental models are self-sustaining and filter out new information that is inconsistent with or threatens the existing model. We may have an example of that in all the discussion we have engaged in around the CGIAR Impact Assessment guidelines. The dominant IA view (mental model) has shown itself to be largely self-renewing although it has been modified somewhat by "new" relatively benign and non-threatening information about impact pathways and maybe a few other concepts. However, the new mental model is to be constructed by a single author.
What was the process used by the group delegated the task of coming up with new IA guidelines? How effective was it? At least some ILACers seem to think that the group has not moved beyond the original mental model very much at all. So maybe the "learning process" or the reflective processes used by the "actors" has been at fault -- the process involved only a partial group of the significant actors and engaged in learning processes that resulted largely in a reinforcement of what IA is.
Maybe ILAC has to find ways that allow all of the actors in the LL cases to modify their mental models when significant new information becomes available. And that has to involve not only single-loop learning but also double and triple-loop learning.
First loop learning occurs when new information is introduced that is consistent with existing information -- no real change in the actor's level of sophistication to deal with this new information is needed. Knowledge is simply "expanded". The mental model is "confirmed". First loop learning asks "Are we doing what we are doing in the right (best, most efficient, most effective) way?" This -- and maybe a bit of second loop learning -- may be the kind of learning that the IA group were engaged in.
Second loop learning occurs when we question how we are achieving the desired goals. "How are we getting the results we are getting?" "What are the principles involved?" "What theories are emerging?" "Are there other ways of pursuing these goals?"
Third loop learning happens when we ask whether the goals we are pursuing for a given project are the appropriate ones. "Are we addressing the appropriate problem?" "Are there related problems that our current mental model is walling-off from our consideration by its very nature?"
How to get to first, second and third loop learning? Certianly not just by convening a committee and maybe asking a couple of "others" to join it. This brings us back to how can we reflect for learning in the LL cases and not just engage in first loop learning? My feeling is that this is at the crux of ILAC and we have to face the fact that it will involve a willingness of all actors to reconsider their mental models, to dedicate time and energy to doing so, and to making changes -- some even significant changes -- to their mental models if the new information warrants such change. Why does this require effort and time? Because it is very easy to ensure that no "significant" information is introduced by engaging in processes that preclude the generation of significant new information.
Cheers,
Ron
Ron's thoughts
Doug,
Here are my brief thoughts for Leslie and the task she is facing.
While it may be helpful to have one or more acceptable definitions of institutional learning I don't think I would start my report with a definition of that concept. It may "orient the reader" as she hopes, on the other hand it may serve to weaken the report -- especially if a reader intent on diminishing the importance of what she has to say about the CGIAR by focusing on her definition rather than on her findings. I remember being involved in an evaluation, a very long time ago, where the client controlled the definition of the driving concept (capacity building) and so was able also to control what constituted acceptable findings. Focusing on abstract definitions as opposed to creating descriptive inventories can be a serious stumbling block in the process of ILAC in that the former can divert attention from the purpose of a serious exercise towards the denotation of a single term. And diversions from the main purpose are what some readers will always seek.
If I were in Leslie's position, I think I might start off with a broad profile of what characterises learning organizations and what the observable consequences of an organization possessing these characteristics are generally agreed to be. To be credible to the CG it would have to be an agricultural research organization -- but it could equally well apply to the other kinds of organizations that Lorna mentions. Then any reader whose intention is to brush her report to one side, has to argue with how she characterises the qualities and behaviours of a learning organization and not the definition of institutional learning as such. For CGIAR purposes, she could say that she is using the terms organization and institution interchangeably at least in this particular section. That would get rid of any chance that the report would be driven off track by another set of definitions. I take Bru's point (organization vs. institution) and remember the identical point being made and discussed when the term ILAC was emerging in the Washington meeting all these years ago (was it in 2003?). For some unremembered reason (at least unremembered by me) the term "institutional" stuck.
To get back to the characteristics of a learning institution -- there are numerous sources she could draw on to create a summary of characterisitcs and consequences of institutional learning. A useful staring point might be:
Cousins, B., S. Goh, S. Clark and L. Lee (2004). Integrating Evaluative Inquiry into the Organizational Culture: A Review and Synthesis of the Knowledge Base. Canadian Journal of Program Evaluation, 19(2): 99-141.
or a recent -- excellent --draft paper by John Mayne -- Building an Evaluative Culture: The Key to Effective Evaluation and Results Management.
I am sure that Leslie could obtain bot of these from the authors without difficulty.
Best wishes to all, Ron
distinguishing inst learning from org learning
Dear All:
I remember discussions before about distinguishing institutional learning from organizational learning. I've been roughly equating institutional learning with double loop learning. I may well be completely mixed up. My working definition is that institutional learning is about learning about institutions, where institutions are the norms and mental models that govern the behaviour of ourselves and others.
If we are going to say that institutional and organizational learning are the same thing (as the definition implies) then let's drop the term institutional learning and become OLAC.
Cheers, Boru
John's thoughts
Thanks, Doug, for sharing Leslie’s paragraph.
A few quick thoughts, hope they might be of some use:
the definition is about a learning activity; it needs to be augmented by a definition of learning per se
the focus on evaluative information related to activities and outcomes is unhelpfully narrow, given the focus of the scoping study on ex post impact assessment; therefore the entire impact pathway is potentially relevant to learning, and learning takes place from a lot of experiences and from non-evaluative information
a learning culture is necessary but not sufficient condition to classify an organization as a learning organization; it have a learning culture but still fail to learn for a variety of reasons (resources, staff turnover, ..) – and of course an organization which has learned might still fail to act on the implications of what has been learned
there is a risk of confusing “institutions” as networks of actors with institutions as the rules of the game
it would be important for Leslie to relate learning (potential, of different groups in the CGIAR system) to different types of and processes/modalities of ex post impact assessment.
John Dixon