This section will describe the process by which those 18 factors were identified, measured, and agreed upon for the survey. It will also describe how the Bayesian model was developed.
The survey instrument and the accompanying Bayesian model were developed using the Integrative Group Process (Gustafson, Cats-Baril, and Alemi 1992). A panel of experts identified and quantified the factors used in the instrument and the Bayesian model. To identify and select the experts, we employed a snowball nomination process in which six leaders in organizational change were asked to nominate experts who (1) understood the theoretical and practical aspects of organizational change in health care, (2) were respected by their peers, and (3) would function effectively in a group (thoughtful, not domineering). When an expert was nominated twice they were listed as a candidate for our panel. Those receiving the most nominations were sent a letter describing the project and indicating that they would receive a phone call to discuss it further. Ultimately the panel was composed of four theoreticians (e.g., a professor of organizational change specializing in health care) and three healthcare practitioners (e.g., an administrator of a state department of health with a reputation as a “mover and shaker”).
The senior author interviewed each panelist for approximately one hour by phone to determine what “implementation success” meant to the expert, what questions they would want answered before they predicted whether an organizational change would be successful (our strategy to identify factors), and what answers to each question would make them optimistic or pessimistic about success. This provided clues on how to define levels for each factor.
Because our intent was to develop a model that not only predicted chance of success but also provided a tool for improving chances of success, experts were encouraged to choose only factors that were modifiable and causally related to successful implementation.
During interviews, the experts would identify factors by responding to the following question: “Suppose you were asked to predict whether a project would be successfully implemented. You can ask me any question you want about the project and I will find the answer for you. What questions would you ask of me? Also please give me examples of answers that would make you optimistic and pessimistic about the chances of success.” The optimistic and pessimistic responses gave examples upon which the levels of the factors would be determined.
For example, two experts said they would want to know what type of problem exploration occurred. One expert said they would be optimistic if the response was that the team talked with several customers to identify problems. Another said they would be optimistic if the team personally experienced the problem and also had data to demonstrate the severity of the problem. In preparation for the meeting these example answers were combined and the panel, after deciding that problem exploration would be a factor, created the levels. As it turned out, the levels for “Exploration of Problem and Customer Needs” were:
|Highest (strong positive influence) rating:||The team talked to many customers to understand the problem, personally experienced the problem, and had data proving severity of the problem.|
|Middle (minor influence) rating:||The team experienced the problem firsthand and knows it well. They had no data to prove severity of the problem and did not involve customers.|
|Lowest (strong negative influence) rating:||The team had neither experienced the problem firsthand nor talked with customers. They had no data proving problem severity.|
We also reviewed the literature to identify definitions of success, factors, and levels of performance. We considered the literature review to be a secondary source of this information. Our prime source was the interviews with the experts because their suggestions would be based on a specific understanding of how the factors would be used in this particular application. The literature review provided a way to ensure completeness. Prior to a face-to-face meeting of the panelists, the senior author combined the interview and literature data into a nonduplicative taxonomy of more than one hundred factors and possible measures—the “Straw Model.”
Two weeks after the telephone interviews were completed, the panel convened for a face-to-face meeting that lasted from noon one day to noon on the next. During the first day the panelists reviewed the straw model's definitions of success, factors, and associated measures. They were told that before adjourning that day they had to agree on: (1) a definition of success, (2) a small number of factors (<20) that were conditionally independent and would not only predict but also explain whether an improvement project would be successfully implemented, and (3) ways of measuring each factor.
The panel decided that success meant a process improvement that persisted six months after implementation and still had the support of both management and staff.
As a test of conditional independence, the factors that resulted from the discussion were each placed on a 3“×5” card. Panelists were first asked to assume a successful project. They were then asked to sort the cards so that cards would be in the same pile only if knowing the answer to one of them would tell a lot about the answer to the other(s). The task was repeated assuming a failed implementation. Cards sorted into the same pile were then discussed by the panel, and if they agreed that they belonged together then the factors were either rewritten to distinguish the real differences, or all but one factor in a pile were eliminated. For example, “problem exploration” and “understanding of customer needs” were grouped into one common factor.
Because most factors could not be measured quantitatively, the panel created three or four descriptions (which we call “factor levels”) of their potential influence on implementation outcome (strong positive influence, minor influence, strong negative influence). A strong positive influence would increase the chance of implementation; a strong negative influence would decrease it.
By 7:00 PM the panel had completed its first day's work. shows the version of the instrument used to predict the outcome of 221 change efforts in health care organizations that formed the basis for the evaluation reported later.
That evening the project staff created forms to allow the panel to estimate the parameters of a Bayesian model. We also created a set of 60 hypothetical profiles of implementation projects described in terms of one level for each factor. We did so by first creating profiles at the “corner points” (e.g., one profile with all factors at the highest levels and one with all factors at the lowest level). Then a computer program generated the remaining ones using a random number generator. We reviewed the profiles to check whether each factor level occurred with about the same frequency. We also examined whether any profiles did not make sense; one factor level could not be present if another factor was at a certain level. If we found unrealistic combinations it would have suggested that the factors were not independent. None were found.
As discussions were winding down, the panel was asked whether there were any types of changes that would be much more difficult to make than others. It was agreed that changing culture was much more difficult than changing process. The purpose of this question was to decide what issues should be considered in defining Prior Odds, a key element in the Bayesian model and a concept we will discuss next.