Institutional Learning and Change Initiative

The following text is drawn from ISNAR (2003).

The most common method for assessing the economic and social benefits of R&D involves the use of surveys of the primary intended users of the research results. There are two basic types of such surveys:

Client Surveys: Surveys in which the respondents who are involved with the research organizations in research collaboration, or some other form of active interaction; and
Beneficiary Surveys: Surveys in which beneficiaries are selected to be a representative sample of the broader group of primary intended users of the research results.

These surveys are sometimes structured in such a way that the direct clients are a certain percentage of the total sample of primary intended users.

At times, it is necessary to use experts rather than users to review some of the research activities. This may be the case in situations where the primary intended users are not sufficiently familiar with the research to be able to answer questions.

Advantages of User Surveys

User surveys provide a more systematic review by using standardized interviews and questionnaires and gathering the views of a wider number of people, thereby overcoming the restrictions involved in using a modified peer review procedure with a limited number of participants. Quantitative indices can be formed if the questions are amenable to scoring, thereby providing concurrent method of making comparisons among R&D projects.

Disadvantages of User Surveys

The major disadvantage to user surveys is the problem of "grateful testimony,” i.e., the possibility that the clients and users may be more positive about the relevance and usefulness of the R&D than is warranted. In addition, often there is difficulty in interpreting the results.

Handling "Grateful Testimony

There are several way to handle the problem of “grateful testimony” when implementing user surveys. Several of these are:

• Pose a large number of questions of different types, all dealing with different aspects of client relevance and usefulness. These questions could be related to the interest of the client in the research area, indicators of the value of the R&D to the organization and the indicators of the importance of the R&D to the clients organization;
• Gathering more specific information on associated benefits is to ask the (potential) clients or users to describe, in very specific terms, the details of the use (or potential use) of the R&D and its related social and economic benefits;
• Conduct follow-up interviews with a subset of the (potential) clients/users surveyed to test the veracity of their answers by probing for additional details which generally reveals how honest they have been; and
• Validity also can be assessed in ex-post and on-going reviews by comparing them with other primary and secondary data relating to the relevance and usefulness of the R&D activity.

Issues Addressed in User Surveys

Issues that need to be addressed in user surveys are:

• Which type of user survey to implement;
• How to identify the users;
• How to structure the survey sample;
• When to use experts instead of users;
• How to ensure the validity of the results; and
• How to interpret the results.

Other User Survey Issues

In developing questionnaires and interview guides, one has to be careful to standardize questions in a manner that facilitates analysis, but does not place undue constraints on the amount of information that can be obtained, thereby resulting in the collection of trivial information. The individuals involved in developing the interview guides or questionnaires must have a clear understanding of the nature of the R&D activity being assessed.

Suitability of User Surveys

User surveys are most useful for past R&D and on-going R&D. The method is useful for assessing the impacts associated with R&D activities near the applied and development end of the R&D spectrum. In most cases, user surveys are often combined with some other method.

Source: ISNAR (2003) Monitoring, Evaluation, and Impact Assessment of R&D Investments in Agriculture, The Hague: International Service for National Agricultural Research.