“Smallball” evaluation: a prescription for studying community-based information interventions

J Med Libr Assoc. 2005 October; 93(4 Suppl): S43–S48.

PMCID: PMC1255752

“Smallball” evaluation: a prescription for studying community-based information interventions^*

Charles P. Friedman, PhD, Senior Scholar, Extramural Programs Division^†

National Library of Medicine 6705 Rockledge Drive, Suite 301 Rockledge, Maryland 20892

Charles P. Friedman: friedmc1/at/mail.nih.gov

†

On leave from the University of Pittsburgh.

Received April 2005; Accepted May 2005.

This article has been cited by other articles in PMC.

Abstract

Objectives: This paper argues that focused evaluation studies of community-based informational interventions conducted over the life-cycle of the project (“smallball” studies) are more informative and useful than randomized experiments conducted only at the project's conclusion (“powerball” studies).

Method: Based on two contrasting strategies in baseball, smallball and powerball studies are compared and contrasted, emphasizing how the distinctive features of community-based interventions lend advantage to smallball approaches.

Results: Smallball evaluations have several important advantages over powerball evaluations: before system development, they ensure that information resources address real community needs; during deployment, they ensure that the systems are suited to the capabilities of the users and to community constraints; and, after deployment, they enable as much as possible to be learned about the effects of the intervention in environments where randomized studies are usually impossible.

Implications: Many in informatics see powerball studies as the only legitimate form of evaluation and so expect powerball studies to be done. These expectations should be revised in favor of smallball studies.

As I write this essay in the fall of 2004, the baseball playoffs have begun and the media are effervescing with baseball argot and metaphors. So I couldn't resist applying baseball logic as I began to envision the advice I would like to give to health librarians and informaticians in relation to evaluating community-based interventions.

In baseball, “smallball” is a strategy of winning games by scoring single runs in multiple innings. Smallball contrasts with power baseball—a.k.a. “big inning” baseball—designed to produce large numbers of runs in a relatively small number of innings. Teams playing smallball will steal bases, bunt, and sometimes intentionally hit the ball to a particular side of the field to advance a runner to the next base and ultimately score a single run before three outs end the inning. In contrast, in “powerball,” every batter is seen as having the potential to hit a home run. When there is a runner on base, a home run will score two runs no matter what base that runner is on, and the batter is encouraged to “swing for the fences”: to try to hit the ball as hard and as far as possible. So powerball sees no value in smallball strategies to advance the runner with deliberate tactics such as bunting and hitting in particular directions. These tactics are seen as lost opportunities to score multiple runs. In the end, though, there is no universal best strategy for baseball. Wise managers of baseball teams choose a strategy that matches their teams' personnel. Teams with many home run hitters should play powerball, and teams with players who run fast and are adept at hitting the ball to particular spots are best suited to smallball. The worst thing to do is to attempt power baseball with smallball personnel, or vice versa.

Applying the smallball versus powerball distinction to the field of evaluation, as opposed to the baseball field, directs us to think about the challenge of evaluation in constructive ways. I begin with three statements about evaluation that may be considered axiomatic.

Axiom 1: Evaluations are driven by questions. The questions that get on the table for investigation determine the methods that are employed, the data that are collected, and the results that are generated. (When the writer Gertrude Stein was dying of stomach cancer, she supposedly asked her lifelong companion Alice B. Toklas, “What is the answer?” The perplexed Toklas did not reply, whereupon Stein said, “In that case, Alice, what is the question?”) So, as in all processes of inquiry, everything in evaluation begins with questions.
Axiom 2: An infinite number of evaluation questions can be asked about any human endeavor, no matter how large or small. So in any study, only a subset of the potential questions actually gets on the table for systematic study. The process by which some questions get on the table, while others fall by the wayside, is political. While those responsible for conducting the evaluation usually have some influence in shaping the scope of the study, what gets on the table is ultimately determined by what people with power and/or money want to know or what they believe is worth paying to find out.
Axiom 3: Every evaluation is constrained by limited resources of time and money. In any project interesting enough to merit study, there are more questions of substance than can possibly be studied with the resources available. Also, the evaluation study itself cannot extend the timeline of the project unreasonably. Limiting the duration of studies of information resources is particularly important, because the resources themselves are updated so frequently. It is critical that a study, once completed, not apply only to a version of the resource that is three generations old.

Given these axiomatic features of evaluation, the smallball metaphor can be brought into play. The goal of every health-related information intervention is to make people healthier, and it would follow that the key question for evaluating health-related information interventions is “Did the intervention make people healthier?” One might, and people often do, take a radical position, somewhat counter to the axioms listed above, that perhaps the only evaluation question of interest should be the question of whether people are actually healthier at the end of the day. I would argue that this is the power baseball approach to evaluation, analogous to arguing that the only hit of any real importance is the home run. The problem with the powerball approach to evaluation is that it is usually the wrong strategy for the situation nature has presented, analogous to playing powerball with smallball personnel. While I believe this mismatch exists for virtually all health information interventions, I also believe it exists perhaps most profoundly for community-based interventions. The bases for these beliefs will be described below.

SMALLBALL VERSUS POWERBALL APPROACHES TO EVALUATION

In the extreme powerball approach to evaluation, driven monolithically by the question “Is anyone healthier?,” no formal evaluation study would be done until the project was extremely mature. At that point, the study ultimately conducted would ideally be a randomized trial that compared the health of individuals in the target community who used the intervention of interest with that of those who did not. A statistical comparison of appropriate health indexes of the two groups would reveal the probability that differences between the groups, if any, were due to chance alone. The statistical controls built into the design of the experiment would have eliminated any other causal explanation of the differences. From this powerball view of evaluation, any differences favoring the intervention group, with less than a 5% probability of having occurred by chance alone, would then lead to rejoicing throughout the land.

In contrast, a smallball approach to evaluation would consist of a series of more focused studies, conducted across the life cycle of the informational intervention. As illustrated in Figure 1, formal study of the intervention would begin even before deployment, and perhaps even before the design of the proposed information resource begins, to verify whether a need for the resource exists and to understand the character of these needs. Smallball studies conducted at this stage would address the need for the intervention from several perspectives, including the perceptions of the conceivers of the intervention, the perceptions of the end users, and the information that can be gleaned from statistical indicators relevant to the end-user group and their environment. The potential magnitude of differences in these perceptions has been startlingly pointed out by Forsythe and colleagues in studies contrasting what migraine patients really wanted to know about their disease and what their care providers thought they would want to know [1]. Other smallball studies conducted before deployment of the intervention might explore whether the proposed resource design is in fact in line with the validated needs in the population.

Figure 1

Stages of information resource development and evaluative study

During the deployment, Figure 1 suggests a chain of events that must occur if the intervention is going to realize, farther down the road, the desired positive effects on health outcomes. From the outset, the intervention must be properly deployed in a technical sense, meaning that it must function in the field as intended and at least as well as it functioned in more controlled laboratory testing. Then, in the next step, the intended users must actually use the resource and use it appropriately. Even if a resource is used and used in the manner intended, it must then engender health behavior change, invoking the next link in the chain. Depending upon the nature of the informational intervention—for example, if the end users of the intervention are health professionals—behavior change must first occur in these professionals before the behavior of health care consumers can be affected. Other kinds of interventions are directed at health care consumers as end users. In these cases, as shown by the dotted arrow in Figure 1, appropriate use of the resource can lead more directly to health behavior changes in consumers.

As this chain relates to evaluation, the desired effect represented by each link, or each arrow depicted in Figure 1, cannot be assumed. Smallball evaluation studies are required to see if the resource was appropriately deployed, whether and how it was used, and whether the anticipated behavior changes occurred in consumers or health care providers. If, and more often when, the answers to some of these questions turn out to be negative, the smallball evaluation studies can direct the developers and managers of the intervention to alternative strategies that can lead to more favorable results.

Finally, and with attention to the bottom of Figure 1, the posited beneficial effects on health can only be seen sometime after the deployment and use of the intervention. Sometimes, the time lag between a consumer's use of an intervention and the realization of the desired health outcomes is considerable. Consider, for example, the time required between implementation of a successful smoking cessation intervention and the realization of any reduced incidence of lung cancer in the target population. While everyone would argue that reduced rates of illness associated with smoking are the desired end point of the intervention, as a practical matter few audiences for evaluations of such an intervention would be willing to wait until such reductions could be demonstrated (or not). Some might advocate that studies directed at this stage of a project are not the province of informatics at all, but rather fall into the domain of health services research. In this light, informatics evaluation might end with the investigation of health behavior change. If this change is along lines that have, in health services research studies, been shown to engender desirable health outcomes, the project can be termed a success from the perspective of informatics.

THE ARGUMENT FOR SMALLBALL

Seen in this light, the argument for smallball evaluation becomes almost self-evident. Smallball evaluations, examining each step in the process illustrated in Figure 1, can make the design and implementation activities of a project self-correcting. Evaluations conducted each step of the way can suggest how the next step should optimally be conducted, and when activities at a given intermediate step are less than fully successful, the results of smallball evaluations can suggest how to redo that step more successfully before proceeding to the next step. If a powerball approach to evaluation is taken, and all of the finite evaluation resources are reserved for a randomized or quasi-randomized trial conducted at the end, null or negative results in the randomized study will usually not reveal what went wrong or at what stage a break occurred in the chain of effects illustrated in Figure 1. Moreover, powerball evaluations require investigators to hypothesize what the major health outcomes of the intervention will be and how to measure them. At best, these hypotheses are educated guesses. A smallball approach to evaluation can point evaluators to the specific outcomes that are most likely to occur and help them detect effects that may initially have been unanticipated.

The argument for smallball evaluation seems so compelling that it sometimes can be difficult to conceive of anyone arguing the merits of the powerball approach. Yet, very intelligent and experienced people do routinely advocate for powerball evaluation—the exclusive use of randomized trials to assess impact on health outcomes—and it is worth reviewing some of the reasons why they do. First and foremost, this approach has a logical purity. If the primary aim of the project is to make people healthier, that is what the evaluation should address. With reference to axiom 3 introduced earlier, why should time and scarce resources be squandered in the pursuit of any other questions? Second, powerball evaluation studies invoke the most trusted empirical method of biomedical science: the randomized trial that is the foundation of evidence-based practice. Many are concerned that any other approach to evaluation would founder for lack of credibility, spawning further speculation rather than settling matters with “hard” results. Third, to the extent that those who propose evaluation studies believe that the peer-reviewers of their proposals will themselves be powerball advocates, they might see powerball studies as a precondition of receiving funding. Fourth, researchers are often concerned that the results of smallball studies are not publishable because they are not science, whereas the results of powerball studies more clearly are, by the definitions of science that are most commonly held in the informatics and public health communities. To the extent that a visible academic product is a desired outcome of the evaluation, powerball studies are often seen as a necessity.

Against these often powerful arguments, the best defense is that smallball studies conducted across the life cycle of an information intervention have the potential to tell interested parties what they really need to know, in time to maximize the chances that a project can be successful. Often, smallball studies are more informative than definitive [2], but it may be argued that, despite their logical appeal, hardball studies rarely end up being definitive either. (Even the best power hitters in baseball hit a home run less than 10% of the time). Moreover, smallball evaluation does not preclude randomized studies of health outcomes. Rather, it circumscribes their role and allows them to be seen in appropriate perspective. Just as in smallball, successful information interventions advance one step at a time.

SMALLBALL EVALUATION IN COMMUNITY-BASED INTERVENTIONS

To bring the threads of this argument together and to tie them specifically to community-based or community outreach interventions, it is useful to consider the features of community-based interventions that make the case for smallball evaluations incontrovertible. I will consider features operative at each stage of evaluation shown in Figure 1.

Before deployment

Community-based interventions are often characterized by a broad cultural gulf between the end users of the information intervention and the information professionals who build and deploy the intervention. In the earliest informatics projects—for example, the Regenstrief medical record system [3]—the information interventions were built by people who worked in the same health care environment as the end users and shared their culture almost completely. As informatics has matured, commercial software supporting clinical care has become increasingly available, which has created a cultural divide between builders and users [4]. Software companies have sought to bridge this divide by employing physicians, nurses, and other health care professionals, who can communicate between the technology-oriented culture of software vendors and the clinically oriented culture of the end users of their products. While this culture-bridging mechanism has led to successful implementations of the commercial software in health care, it is noteworthy that the most visible successes in health care information technology to date have stemmed from the efforts in Indianapolis, Salt Lake City, Boston, Nashville, and elsewhere, where the technology was developed within the user environment by natives of that environment. In these cases, it can be argued that smallball evaluations to focus the needs for information interventions are largely unnecessary. The technology developers, who are health care practitioners in the same environment or closely allied to them, understand the potential users' needs and problems because, to a large extent, they are users themselves.

In contrast, in community-based interventions, the cultural divide between the developers and end users is usually extremely broad, and the smallball evaluations that identify and sharpen the end users' needs become essential. Powerball studies of health outcomes, which would overlook this developmental stage of the intervention, might reveal null results because the “wrong” intervention was built for the intended audience.

Consider an intervention designed for use by a broad spectrum of health care consumers with diseases that are more prevalent in indigent populations. In this case, the cultural gulf between the highly educated and often affluent developers of the intervention and the eventual end users is profound. If the users' needs and problems are misunderstood, the potential is enormous to build an intervention that for a range of reasons is difficult to use, that provides information different from what users need, or that simply has a “look and feel” foreign to their culture. The only way to bridge the cultures, and allow intervention developers to proceed with confidence, are smallball studies of the end users' information needs, in the domain of the intervention, as the users themselves perceive them. These smallball studies, undertaken before one system flowchart is drawn or one line of code is written, should also address the end users' resources to access the intervention, their reading levels, and an array of other key details that can spell the difference between the intervention's ultimate success and failure.

During deployment

In community-based undertakings, project activities during the initial stages of deployment can encounter many potential problems amenable to careful smallball evaluation. The essential predeployment studies, discussed in the previous paragraphs, can ensure that the intervention addresses needs that are real for the end users and is generally responsive to the users' resources and personal capabilities. However, it is not until a prototype of the information resource exists that the conclusions of these predeployment studies can be validated by putting the resource to a test. Here, smallball once again must come into play. Once end users can actually interact with something, smallball studies can ensure both that the right resource is being built and that it is being built the right way, increasing the probability that the resource will be used.

In community-based interventions, there is little margin for error in this regard because use of an information resource is usually voluntary and discretionary. In more traditional clinical informatics work, where end users are employees of health care or academic organizations, lines of authority over end users may be somewhat ambiguous, but, in the case of community-oriented information resources, they are usually nonexistent. If the first encounter with an information resource is unpleasant or nonproductive, the end user will have almost no incentive or other compelling reason to try it again. If the logistics of use prove formidable—if it takes too much time or effort to locate a workstation to access the intervention, for example—the resource may be used once but never again.

Smallball evaluations are needed to verify that the intervention is used and, if it is not, to discover why not before it is too late to address the problems. Powerball evaluations that overlook this developmental stage of the intervention may yield negative results simply because the intervention largely sat dormant.

After deployment

Smallball studies to explore the effects of a community intervention after deployment are often necessary, because they are the best studies that can be undertaken under the circumstances. Logistical and ethical constraints operating in community settings often preclude randomization, blinding, and other requirements of powerball evaluations designed to demonstrate causal effects of the intervention on health behavior or health outcomes. While patients with extremely serious illness are willing to enter a blinded clinical trial of an experimental drug, accepting a 50% chance of receiving the drug over the certainty of not receiving it at all if they do not enter the study, similar motivations do not exist for information interventions. Powerball evaluations that overlook these key aspects of community-based interventions may yield negative results because the arms of the study (the intervention and “control” groups) become confounded or because the subjects do not participate according to protocol.

Smallball studies undertaken after deployment would seek to make the strongest case possible that the intervention is or is not having the intended effects. This might be accomplished, for example, with “dose–effect” studies. While these are weaker in terms of the logic of causal inference than multi-arm randomized studies, a demonstration that community members who made greater use of the intervention exhibited greater numbers of desirable health behaviors may be very compelling. Such studies may make the strongest possible case for the success of the intervention, particularly if it can be shown that the use of the intervention predated the desired behavior.

Another desirable feature of smallball postdeployment evaluation studies is their sensitivity to unforeseen outcomes, positive or negative. The complexity of community interventions and the new trails they often blaze raise great potential for effects and outcomes that cannot be readily seen in advance. As noted earlier, the scope of powerball evaluations is restricted to outcomes that are predefined, often well in advance of any accumulated experience with the intervention. Powerball studies emphasize outcomes that can be measured with standardized instruments, so what gets measured is often a function of what measurement tools exist, rather than what stakeholders in the evaluation consider important. By contrast, smallball evaluations tend to be more flexible, responsive, and adaptive. Priority in smallball is given to content over method. In smallball, it is better to pursue the issues perceived to be highest on the agenda or of greatest interest, even if pursuit of these comes at some expense of what is canonically considered to be scientific rigor.

CONCLUDING OBSERVATIONS

The case for smallball evaluations is not a case for sloppy, non-rigorous, or casual evaluations. It is case for thinking about evaluation in a particular way, a way that seems particularly well suited to the circumstances of community-based information interventions. Returning to the three axioms introduced at the beginning of this paper, the case for smallball is a case for focusing evaluation on the most important questions, as best as these can be determined, rather than deciding first that a randomized trial is the essential method and then trying to identify questions that such an evaluation can address. It is a case for involving a range of constituencies in determining what questions will be addressed and for allowing new questions to emerge as new experiences with an intervention accrue over time. And it is a case for recognizing that resources for evaluation are always limited. Just as low-budget baseball teams are often better suited to smallball because power hitters tend to carry high salaries, so then are smallball evaluations easier to conduct under the omnipresent circumstances of limited resources.

Footnotes

This paper is based on a presentation at the “Symposium on Community-based Health Information Outreach”; National Library of Medicine, Bethesda, Maryland; December 3, 2004.

REFERENCES

Forsythe DE.. New bottles, old wine: hidden cultural assumptions in a computerized explanation system for migraine sufferers. Med Anthropol Q. 1996;10:551–74. [PubMed]
House ER. Evaluating with validity. Beverly Hills, CA: Sage,1980.
McDonald CJ, Overhage JM, Tierney WM, Dexter PR, Martin DK, Suico JG, Zafar A, Schadow G, Blevins L, Glazener T, Meeks-Johnson J, Lemmon L, Warvel J, Porterfield B, Warvel J, Cassidy P, Lindbergh D, Belsito A, Tucker M, Williams B, and Wodniak C. The Regenstrief medical record system: a quarter century experience. Int J Med Inform. 1999. Jun; 54(3):S225–53.
Friedman CP. Information technology leadership in academic medical centers: a tale of four cultures. Acad Med. 1999. Jul; 74(7):S795–9.

Articles from Journal of the Medical Library Association : JMLA are provided here courtesy of
Medical Library Association