PMCCPMCCPMCC

Search tips
Search criteria 

Advanced

 
Logo of jepicomhJournal of Epidemiology and Community HealthVisit this articleSubmit a manuscriptReceive email alertsContact usBMJ
 
J Epidemiol Community Health. 2007 December; 61(12): 1020.
PMCID: PMC2465665

Are program evaluators judges and/or knowledge brokers?

As Bourdieu1 understood perfectly well, it comes as no surprise that a multidisciplinary field as vast as that of program evaluation, much like public health,2,3 lacks a universally accepted definition. By questioning the way I conduct evaluations in the public health field, a new factor emerged in relation to the traditional definitions: my tendency to carry out evaluations using a process that strives to share knowledge with the stakeholders. Upon reading the preceeding sentence, evaluation specialists will quickly have categorised me as a so‐called “fourth‐generation” evaluator, grouped with those who have a responsive disposition.4 Beyond the rhetoric, I will strive to show how I have been able, in three settings on three different continents, to act as a “knowledge broker”5 by creating a favourable environment for sharing and strengthening capacity. In this paper, the use of the term “knowledge broker” suggests the evaluator is an intermediary between “the worlds of research and action”.6

In a rural area of Quebec, Canada, I took part in evaluating a program for street outreach workers. We used an adaptation of the empowerment evaluation approach; a method not commonly used in the area.7 To help street outreach workers organise their data in a way useful for evaluation, I suggested they produce weekly “portraits” of their interventions with young people. They specifically had to sum up these interventions in a simple, visual, analytical framework of the “determinants” (layers) of health (see appendix at http://jech.bmj.com/supplemental). While use of this framework was indispensable to the evaluation, I was still surprised one day to see one of these street outreach workers pull out of his pocket a laminated graphic representation that helped him support young people in organising their life plans during his interventions. On another continent, in Afghanistan, I discussed in an article8 all the activities favourable to reinforcing individual and organisational capacities over the course of three evaluations conducted between 2001 and 2003. Today, teams from a non‐governmental organisation, following their active participation in successive evaluations, are able to use data‐gathering techniques (household surveys, focus groups and hospital surveys) and describe their program's logic, all of which is useful for effective implementation of their activities. Finally, in Haiti, the experience of evaluating the sustainability of a public nutrition program enabled local teams to assimilate the very recent theoretical proposals on the conceptual aspects of sustainability levels and processes.9 (See also the sustainability tool kit web site based on that example at http://www.cacis.umontreal.ca/perennite/index_en.htm.) Consequently, for their 2006–2009 programming, they integrated activities that promote sustainability into their planning. Similarly, I used an empowering analytical framework10 adapted from Ninacs11 to judge effectiveness of the program against this objective of reinforcing empowerment (fig 11).). The stakeholders therefore understood that implementation of a program favourable to reinforcing empowerment required them to conduct activities related to the aspects of this analytical framework, which they were able to include in their programme.

figure ch65573.f1
Figure 1 Dimensions of empowerment according to Ninacs (2002).11

Consequently, in light of these few empirical examples, I believe—and I imagine this to be a special paradigmatic position—that an evaluator must also be a knowledge broker; a condition I deem essential to improving program quality. The utilisation of conceptual diagrams is central to this approach.12 This new skill should perhaps be added to the list of 60 essential skills for evaluators,13 as “The evaluator is an educator; his success is to be judged by what others learn”, as stated by Lee Cronbach 27 years ago.

The appendix is available at http://jech.bmj.com/supplemental

Supplementary Material

[web only appendix]

Acknowledgements

We thank the two reviewers for their contributions. An earlier version of this paper obtained a prize from the interdisciplinary training program in analysis and evaluation of health interventions (CIHR/AnÉIS Program) and the Chair CHSRF/CIHR on Governance and transformation of health care organisations (GETOS); and was published in their Infoletter. VR is a research fellow (junior 1) from the Fonds pour la Recherche en Santé du Québec (FRSQ).

Footnotes

Competing interests: None.

The appendix is available at http://jech.bmj.com/supplemental

References

1. Bourdieu P. Science de la science et réflexivité. Paris: Raisons d'agir, 2001
2. Fassin D. Faire de la santé publique. Rennes: Editions de l'ENSP, 2005
3. Ridde V. Reducing social inequalities in health: public health, community health or health promotion? Prom Ed 2007. 1463–67.67
4. Stake R E. Standards‐based & responsive evaluation. Thousand Oaks: Sage Publications, 2004
5. Canadian Health Science Research Foundation The theory and practice of knowledge brokering in canada's health system. Ottawa, Canada: Canadian Health Services Research Foundation, 2003
6. Lomas J. The in‐between world of knowledge brokering. BMJ 2007. 334129–132.132 [PMC free article] [PubMed]
7. Ridde V, Baillargeon J, Ouellet P. et al L'évaluation participative de type empowerment: une stratégie pour le travail de rue. Service Social 2003. 50263–279.279
8. Ridde V, Shakir S. Evaluation capacity building and humanitarian organization. J MultiDisciplin Eval 2005. 378–112.112
9. Ridde V, Pluye P, Queuille L. Evaluer la pérennité des programmes de santé publique: un outil et son application en Haïti. Revue d'Epidémiologie et de Santé Publique 2006. 54421–431.431
10. Ridde V, Delormier T, Gaudreau L. Evaluation of empowerment and effectiveness: universal concepts? In: McQueen DV, Jones CM, eds. Global perspectives on health promotion effectiveness, vol. 1. New York: Springer Science & Business Media 2007. 389–403.403
11. Ninacs W A. Types et processus d'empowerment dans les initiatives de développement économique communautaire au Québec [dissertation]. PhD thesis. Québec: Université Laval Ecole de service social 2002
12. Paradies Y, Stevens M. Conceptual diagrams in public health research. J Epidemiol Comm Health 2005. 591012–1013.1013
13. Stevahn L, King J A, Ghere G. et al Establishing essential competencies for program evaluators. Am J Eval 2005. 2643–59.59

Articles from Journal of Epidemiology and Community Health are provided here courtesy of BMJ Publishing Group