Search tips
Search criteria 


Logo of hsresearchLink to Publisher's site
Health Serv Res. Apr 2003; 38(2): 751–776.
PMCID: PMC1360903
Developing and Testing a Model to Predict Outcomes of Organizational Change
David H Gustafson, François Sainfort, Mary Eichler, Laura Adams, Maureen Bisognano, and Harold Steudel
Address correspondence to David H. Gustafson, Ph.D., 1109 WARF Building, 610 Walnut Street, Madison, WI 53704. Dr. Gustafson, is Professor of Industrial Engineering and Preventive Medicine, University of Wisconsin-Madison. François Sainfort, Ph.D., is Professor of Industrial and Systems Engineering, Georgia Institute of Technology, Atlanta. Mary Eichler, M.S., is a Health Care Consultant, Milliman USA, Inc., Chicago. Laura Adams, M.S., is Principal, Laura Adams and Associates LLC, Boston. Maureen Bisognano, R.N., M.S., is Vice President, Institute for Healthcare Improvement, Boston. Harold Steudel, Ph.D., is Professor and Chairman of Industrial Engineering, University of Wisconsin-Madison.
To test the effectiveness of a Bayesian model employing subjective probability estimates for predicting success and failure of health care improvement projects.
Data Sources
Experts' subjective assessment data for model development and independent retrospective data on 221 healthcare improvement projects in the United States, Canada, and the Netherlands collected between 1996 and 2000 for validation.
A panel of theoretical and practical experts and literature in organizational change were used to identify factors predicting the outcome of improvement efforts. A Bayesian model was developed to estimate probability of successful change using subjective estimates of likelihood ratios and prior odds elicited from the panel of experts. A subsequent retrospective empirical analysis of change efforts in 198 health care organizations was performed to validate the model. Logistic regression and ROC analysis were used to evaluate the model's performance using three alternative definitions of success.
Data Collection
For the model development, experts' subjective assessments were elicited using an integrative group process. For the validation study, a staff person intimately involved in each improvement project responded to a written survey asking questions about model factors and project outcomes.
Logistic regression chi-square statistics and areas under the ROC curve demonstrated a high level of model performance in predicting success. Chi-square statistics were significant at the 0.001 level and areas under the ROC curve were greater than 0.84.
A subjective Bayesian model was effective in predicting the outcome of actual improvement projects. Additional prospective evaluations as well as testing the impact of this model as an intervention are warranted.
Keywords: Organizational change, Bayesian model, improvement, empirical evaluation
This paper describes the development of a short survey instrument and the creation of a companion Bayesian model that uses the survey data to predict the potential for successful implementation of a health system change and to explain (identify) the factors that mitigate for and against success in that particular change. The goal of the instrument and model is to help change agents (persons responsible for bringing about a process improvement) detect potential obstacles to and improve chances for successful implementation. The development of the survey instrument and the Bayesian model were conducted with an expert panel. In a subsequent independent empirical study, the performance of the Bayesian model was tested using data from actual hospital and clinic improvement projects. In this paper, we (a) briefly review literature on organizational change, (b) provide details of the expert panel process used to develop the survey instrument and assemble the Bayesian model, (c) provide details of the methods used to collect and analyze data to ascertain the predictive ability of the Bayesian model, and (d) present and discuss the results. When the survey instrument and accompanying model are discussed in combination we refer to them as the “Organizational Change Manager” (OCM).
The literature relating to organizational change is extensive. In 1951 Lewin suggested three stages of change that influenced much future research. Since then many conceptual models have been developed to help understand organizational change (Finstand 1998; Weick 1995; 1976; Starbuck 1976; Orton and Weick 1990; DiMaggio 1988; Zucker 1987). Related research developed theories concerning strategic planning, adaptive learning, decision making, management applications, diffusion of innovations, and social–psychological processes of organizations (Van de Ven and Poole 1995; Chakravarthy and Lorange 1991; Burgelman 1991; March and Olsen 1976; March and Simon 1958; Carroll and Hannan 1989; Weick 1979; Rogers 1995; Gersick 1991). Still others examined how differences in organizational structure, environment, and relationships affect how members manage and achieve change (Kabanoff, Waldersee, and Cohen 1995; Coghlan 1997; Beckhard and Harris 1987; Rashford and Coghlan 1994; McCaughan and Palmer 1994; Senge 1990).
Research and development project selection models have been developed that weight factor importance (Roberto and Pinson 1972). For instance Cooper (1980) used regression analysis on 195 research and development projects to show how his eight-factor model correctly predicted the success or failure in 84 percent of the cases.
However, there is a scarcity of such models for organizational change. Lee and Steinberg (1980) suggested areas to consider in assessing the probability of successful implementation including: characteristics of company management, system users, collaboration between users, the project, the project team and their approach, and the project solution. A model based on Sink (1998) computed an organization's readiness for change using four variables: level of dissatisfaction with the status quo, a desired future state, first steps for achieving that future state, and perceived costs of changing. Models intended to facilitate organizational change have been proposed but have not been thoroughly tested (Hall, Wallace, and Dossett 1973; Snyder 1985; Pearce and Osmond 1996).
The intent of this paper is to describe the development and testing of the OCM, not to provide a comprehensive review of the organizational change field. Hence, the remaining part of this review will concentrate on the 18 factors chosen for inclusion into the OCM. Those factors are summarized in Table 1.
Table 1
Table 1
Factors and Levels Used in the Survey and Bayesian Model
Mandate/Project Launch
Creating and communicating a mandate for change enhances chances of successful implementation. Leonard-Barton and Kraus (1985) found that change was more successful when the need for innovation was defined at a high level within the organization. Lee and Steinberg (1980) found that successful implementation was positively correlated with clear objectives and explicitly identified, tangible, and measurable tasks. Delbecq and Mills (1985) suggest that leaders must energize the organization's members and enable employees to face challenging goals.
Leader Goals, Involvement, and Support
Leadership commitment, involvement, and accountability are key features for successful implementation. Among the most effective ways to engage that support is to have a change effort that is consistent with and contributes to achieving the organizational goals of the leadership (Goodman 1982; Quinn and Cameron 1989; Hurley 1992; Wynne 1979; Lee and Steinberg 1980). Support from leaders is more likely if the leaders are kept involved in and informed about the project (Kanter 1983).
Supporters and Opponents
Support of a proposed change by informal opinion leaders is essential for success. However, leaders are not always innovators. Opinion leaders are more likely to be innovative and to support a change if the organization's norm is to adopt changes (Rogers 1995). The issue then is the relative strength of the supporters and opponents.
Middle Manager Goals, Involvement, and Support
Several researchers address the important role of middle managers in the change process, emphasizing for instance their involvement in networking and negotiating (Beatty and Lee 1992; Rousseau and Tijoriwala 1999; Hosking 1988). Like senior leaders, middle managers are more likely to support a change if they believe that doing so will promote their own organizational goals (Schneider and Goldwasser 1998; Duck 1993), and if they feel involved in discussions about the change (Rousseau and Tijoriwala 1999).
Tension for Change
One of the key predictors of a successful change is whether the affected staff are dissatisfied with the current process. It is difficult to create this tension for change when none exists, suggesting that such tension should be an important consideration in deciding what projects to select. However, effective communication can spread dissatisfaction with status quo as well as announce a change, cultivate commitment, and reduce resistance (Jick 1993; Beckhard and Pritchard 1992; Hunsucker and Loos 1989; Morgan 1988; Kotter and Schlesinger 1979; Ford and Ford 1995; Donnellon 1986).
Staff Needs Assessment, Involvement, and Support
Like senior leaders and middle managers, employees are more likely to support a project that they believe is in their own best interests. The project team should understand employee needs and ensure that employees understand how the project will meet those needs (Hage and Aiken 1970; Ginzberg 1981; Delbecq and Mills 1985; Smith and Carayon 1995; Hurley 1992).
Exploration of Problem and Understanding Customer Needs
Prior to change, the organization's circumstances, problems, and needs must be analyzed. One particularly important key to success is the ability of the change to meet customer needs (Rogers 1983; 1995; Maidique and Zirger 1984). Guimaraes (1981) found incomplete and inaccurate evaluation of user needs results in implementation failure.
Change Agent Prestige and Commitment
A change agent is critical to establishing a climate for creating, implementing, and sustaining change (Meyer and Goes 1988; Stjernberg and Philips 1993). In health care organizations, a physician change agent is often most effective because of stature. Other characteristics improving a change agent's effectiveness include prestige (Freeman 1982), persistence (Schon 1963), political influence, and access to resources (Kanter 1983; Maidique 1980).
Source of Ideas
When studying research and development projects in manufacturing companies, Utterback (1971) found successful projects were more likely to use information from consultants outside the organization. Rogers (1995) also found that innovators look outside their organization for ideas. Amabile (1990) found competition from outside organizations drives creativity.
Several researchers have found change implementation more likely when sufficient money, time, and personnel are allocated (Delbecq and Mills 1985; Damanpour 1991; Hannan and Freeman 1989; Rogers 1995; Amabile 1990).
Relative Advantages
Romano (1995) found that staff will be less likely to resist if they see powerful advantages of the proposed change (Lorenzi and Riley 2000). Demonstrable benefits and valued consequences had a positive impact on implementation success in a study that examined employee opinions regarding reasons for change (Rousseau and Tijoriwala 1999).
Radicalness of Design
Change is more likely to be adopted if perceived as reasonable (Stjernberg and Philips 1993; Delbecq 1975; Kotter and Schlesinger 1979; Rogers 1995; Alavi and Henderson 1981; Guimaraes 1981; Utterback et al. 1976). Changes seen as threatening or incompatible with current conditions will be resisted (Van de Ven, Angle, and Poole 1989; Kotter and Schlesinger 1979; Schneider and Goldwasser 1998).
Flexibility of Design
Successful implementation is more likely if the new design can be easily adapted to fit into existing culture and practices (Rogers 1983; 1995; Schultz and Slevin 1975; Vertinsky, Barth, and Mitchell 1975; Kotter and Schlesinger 1979; Cooper 1980; Dickson 1976; Maidique and Zirger 1984).
Evidence of Effectiveness
Those affected by a change evaluate the change's expected consequences (Rogers 1995). This information is usually obtained from peers whose subjective opinion is often more convincing than empirical evidence, although empirical evidence is not without value. A demonstration of the change on a trial basis will also influence adoption, especially if the demonstrator is an opinion leader (Rogers 1995).
Complexity of Implementation Plan
Many changes are unnecessarily complex. A simple implementation complete with well-defined tasks and a schedule increases likelihood of successful implementation (Dane, Gray, and Woodworth 1979; Rousseau and Tijoriwala 1999; Lonnstedt 1985; Lee and Steinberg 1980).
Work Environment
Too often change is simply dropped into an organization without modifying surrounding organizational systems that influence success. Paolillo and Brown (1978) found rewards and incentives for staff to be positively related to innovativeness of organizations. Others found that clearly defined reporting relationships are needed and that organizational turbulence interferes with chances of success (Coch and French 1948; Hunsucker and Loos 1989; Schneider and Goldwasser 1998; Iskat and Liebowitz 1996; Lawler and Rhode 1976).
Staff Changes Required
Resistance results when those affected fear they cannot develop the new skills required. Hence, the fewer the changes required the greater staff self-efficacy (Bandura 1977; Compeau and Higgins 1995; Lorenzi and Riley 2000). Training that employs peer role modeling minimizes resistance and increases likelihood of successful implementation (Kotter and Schlesinger 1979; Coch and French 1948).
Monitoring and Feedback
Change that can be measured and is tracked tends to be implemented more successfully (Rogers 1983; 1995; Maidique and Zirger 1984; Nadler and Tushman 1990). Furthermore, soliciting employee feedback in a risk-free manner during implementation can shape attitudes favorably toward the change (Hunsucker and Loos 1989; Kanter, Stein, and Jick 1992; Beckhard and Pritchard 1992; Beer, Eisenstat, and Spector 1990).
This section will describe the process by which those 18 factors were identified, measured, and agreed upon for the survey. It will also describe how the Bayesian model was developed.
The survey instrument and the accompanying Bayesian model were developed using the Integrative Group Process (Gustafson, Cats-Baril, and Alemi 1992). A panel of experts identified and quantified the factors used in the instrument and the Bayesian model. To identify and select the experts, we employed a snowball nomination process in which six leaders in organizational change were asked to nominate experts who (1) understood the theoretical and practical aspects of organizational change in health care, (2) were respected by their peers, and (3) would function effectively in a group (thoughtful, not domineering). When an expert was nominated twice they were listed as a candidate for our panel. Those receiving the most nominations were sent a letter describing the project and indicating that they would receive a phone call to discuss it further. Ultimately the panel was composed of four theoreticians (e.g., a professor of organizational change specializing in health care) and three healthcare practitioners (e.g., an administrator of a state department of health with a reputation as a “mover and shaker”).
The senior author interviewed each panelist for approximately one hour by phone to determine what “implementation success” meant to the expert, what questions they would want answered before they predicted whether an organizational change would be successful (our strategy to identify factors), and what answers to each question would make them optimistic or pessimistic about success. This provided clues on how to define levels for each factor.
Because our intent was to develop a model that not only predicted chance of success but also provided a tool for improving chances of success, experts were encouraged to choose only factors that were modifiable and causally related to successful implementation.
During interviews, the experts would identify factors by responding to the following question: “Suppose you were asked to predict whether a project would be successfully implemented. You can ask me any question you want about the project and I will find the answer for you. What questions would you ask of me? Also please give me examples of answers that would make you optimistic and pessimistic about the chances of success.” The optimistic and pessimistic responses gave examples upon which the levels of the factors would be determined.
For example, two experts said they would want to know what type of problem exploration occurred. One expert said they would be optimistic if the response was that the team talked with several customers to identify problems. Another said they would be optimistic if the team personally experienced the problem and also had data to demonstrate the severity of the problem. In preparation for the meeting these example answers were combined and the panel, after deciding that problem exploration would be a factor, created the levels. As it turned out, the levels for “Exploration of Problem and Customer Needs” were:
Highest (strong positive influence) rating:The team talked to many customers to understand the problem, personally experienced the problem, and had data proving severity of the problem.
Middle (minor influence) rating:The team experienced the problem firsthand and knows it well. They had no data to prove severity of the problem and did not involve customers.
Lowest (strong negative influence) rating:The team had neither experienced the problem firsthand nor talked with customers. They had no data proving problem severity.
We also reviewed the literature to identify definitions of success, factors, and levels of performance. We considered the literature review to be a secondary source of this information. Our prime source was the interviews with the experts because their suggestions would be based on a specific understanding of how the factors would be used in this particular application. The literature review provided a way to ensure completeness. Prior to a face-to-face meeting of the panelists, the senior author combined the interview and literature data into a nonduplicative taxonomy of more than one hundred factors and possible measures—the “Straw Model.”
Two weeks after the telephone interviews were completed, the panel convened for a face-to-face meeting that lasted from noon one day to noon on the next. During the first day the panelists reviewed the straw model's definitions of success, factors, and associated measures. They were told that before adjourning that day they had to agree on: (1) a definition of success, (2) a small number of factors (<20) that were conditionally independent and would not only predict but also explain whether an improvement project would be successfully implemented, and (3) ways of measuring each factor.
The panel decided that success meant a process improvement that persisted six months after implementation and still had the support of both management and staff.
As a test of conditional independence, the factors that resulted from the discussion were each placed on a 3“×5” card. Panelists were first asked to assume a successful project. They were then asked to sort the cards so that cards would be in the same pile only if knowing the answer to one of them would tell a lot about the answer to the other(s). The task was repeated assuming a failed implementation. Cards sorted into the same pile were then discussed by the panel, and if they agreed that they belonged together then the factors were either rewritten to distinguish the real differences, or all but one factor in a pile were eliminated. For example, “problem exploration” and “understanding of customer needs” were grouped into one common factor.
Because most factors could not be measured quantitatively, the panel created three or four descriptions (which we call “factor levels”) of their potential influence on implementation outcome (strong positive influence, minor influence, strong negative influence). A strong positive influence would increase the chance of implementation; a strong negative influence would decrease it.
By 7:00 PM the panel had completed its first day's work. Table 1 shows the version of the instrument used to predict the outcome of 221 change efforts in health care organizations that formed the basis for the evaluation reported later.
That evening the project staff created forms to allow the panel to estimate the parameters of a Bayesian model. We also created a set of 60 hypothetical profiles of implementation projects described in terms of one level for each factor. We did so by first creating profiles at the “corner points” (e.g., one profile with all factors at the highest levels and one with all factors at the lowest level). Then a computer program generated the remaining ones using a random number generator. We reviewed the profiles to check whether each factor level occurred with about the same frequency. We also examined whether any profiles did not make sense; one factor level could not be present if another factor was at a certain level. If we found unrealistic combinations it would have suggested that the factors were not independent. None were found.
As discussions were winding down, the panel was asked whether there were any types of changes that would be much more difficult to make than others. It was agreed that changing culture was much more difficult than changing process. The purpose of this question was to decide what issues should be considered in defining Prior Odds, a key element in the Bayesian model and a concept we will discuss next.
Bayesian modeling refers to a wide range of models applied in many areas. In this particular modeling effort, a simple, yet extremely valuable, application of Bayes' rule permitted us to quantify—through expert subjective estimation—the influence of the 18 factors identified and defined by the expert panel to construct a predictive model of implementation success. Such subjective Bayesian models (see for example Gustafson, Cats-Baril, and Alemi 1992, chapter 8; Gustafson et al. 1993) estimate the probability of an event, given certain information about factors that influence it by applying the odds version of Bayes' theorem:
Posterior Odds = Product of Likelihood Ratios × Prior Odds
  • Prior Odds are the initial estimates of how much more likely it is that a change will be a success than a failure. Mathematically, prior odds are the ratio of prior probability of success to prior probability of failure, that is, [P (S) /P (F)], where “S” is a successful change and “F” is a failed change.
  • Likelihood Ratios indicate the diagnostic power of particular information, that is, the extent to which a datum revises the prior odds. Likelihood ratios are the ratio of the conditional probability of observing the level of a particular factor given a successful implementation, to the conditional probability of that same datum given a failed implementation: [P (D|S)/P (D|F)].
  • Posterior Odds are the ratio of the probability of success given the datum to the probability of failure given the same datum, that is, [P (S|D)/P (F|D)]. They represent the revised odds given the datum of information learned.
If there are n data points (D1, D2,…,Dn), and they are conditionally independent, the odds version of Bayes' theorem can be written as:
equation M1
In many situations, likelihood ratios and prior odds can be estimated empirically. However, if necessary data do not exist or are insufficient, behavioral decision theorists assert that likelihood ratios can be estimated subjectively by trained experts (Edwards, Lindman, and Savage 1963; Slovic and Lichtenstein 1971; Hogarth 1975; von Winterfeldt and Edwards 1986).
In the research reported here, no data were available to empirically estimate the prior odds and likelihood ratios for the 18 factors, therefore the expert panel was assembled and guided through a structured process to estimate them. Such process has been used successfully in developing a similar Bayesian model in the context of evaluating the probability of high quality care being delivered in psychiatric emergencies (Gustafson et al. 1993).
While it may appear that experts could directly estimate the probability of success, the cognitive burden in forming an evaluative response to the simultaneous influence of n factors (D1, D2,…, Dn) is much greater than the cognitive burden in forming an evaluative response to one factor at a time (Gustafson et al. 1993; Luke, Stauss, and Gustafson 1977). Thus, experts are better equipped in subjectively estimating n individual likelihood ratios than in subjectively estimating posterior odds given n pieces of data presented simultaneously. To ensure that likelihood ratios can be estimated one by one, it is necessary to define the pieces of data (D1, D2,…, Dn) so that conditional independence can be assumed. As mentioned above, the panel was provided specific tasks to assess conditional independence, and if necessary to redefine factors to achieve it.
Hence, the next morning the panel estimated the prior probability of implementation success. These estimates, like all estimates that morning, involved an estimate-talk-estimate approach (Gustafson et al. 1973) and followed a strategy we have used in other health care applications (Gustafson et al. 1977; 1990). Panelists individually provided numerical estimates, which were posted on a flip chart, and if estimates were widely different (a judgment made by the senior author), the panelists were asked to discuss the rationale behind their scores. After discussion panelists were allowed (but not required) to individually modify their estimates. The revised estimates were then averaged.
The panel estimated each likelihood ratio for the Bayesian model in two different ways.
  • Likelihood ratios were estimated directly by responding to the following two questions:
    • Think about two healthcare improvement projects. One was successfully implemented and the other was not. Which one is more likely to have the following characteristics. (Then a factor level was specified, such as “a staff that hate the current situation and believe that change is essential.”)
    • How much more likely? (a lot, somewhat, barely)
  • Setting these estimates aside, the panel then gave numerical estimates of the individual likelihood ratios for each factor. They were asked to assume that we had dossiers on a random selection of one hundred successful implementations and one hundred failures. They were shown the levels of that factor (from Table 1) and asked to distribute the one hundred successful projects and the one hundred failed projects among the different levels on the factor being considered. For instance they each completed a table similar to Table 2 for the “problem exploration and customer needs understanding” factor. Once these estimates were completed, again using the estimate-talk-estimate approach, the scores in the success row were divided by scores in the failure row to yield likelihood ratios as shown in third column of Table 2.
    Table 2
    Table 2
    Likelihood Ratios for the Factor “Exploration of Problem and Customer Needs”
As a consistency check, these ratios were compared to the judgments obtained in response to question A. If there were differences (e.g., a response of “much more likely” in task A and the numerical estimate was small in task B) then further discussion ensued and, if needed, task B was redone.
A similar process was used to estimate prior odds. The expert panel decided that the probability of successful implementation was 5 percent in projects involving cultural change and 16 percent for changes involving only process changes.
The final task was to conduct a preliminary first test of model performance. The panel (again using the estimate-talk-estimate approach) directly estimated the chance of the implementation success for each of the 60 profiles created the previous evening. The 60 profiles were also each assigned a score calculated using the Bayesian model. The scores assigned by the panelists were averaged for each profile. The correlation between those average scores and the Bayesian model scores was .77. This performance test, however, does not ascertain the validity of the model in predicting success probability because (1) it compares two types of subjective assessments, (2) it involves a small number of hypothetical profiles, (3) it is performed by the same panel that developed the model, and (4) as explained earlier, such holistic judgments are difficult to make. Thus, to investigate the validity and performance of the Bayesian model developed by the expert panel, an independent evaluation study involving actual change projects was performed and is described in the next section.
This section describes how the survey and Bayesian model were evaluated using actual change results. Senior leaders from 198 health care organizations attending seminars on organizational improvement offered by the Institute for Healthcare Improvement ( identified an improvement project with which they were intimately familiar and for which outcome was known. These leaders, who were not part of the expert panel that developed the model, completed a survey instrument, which asked them to indicate:
  • whether the project involved cultural change (to determine which prior odds to use, 5 percent or 16 percent).
  • the project's status in terms of success (don't know, big success, small success, small failure, big failure), time since implementation (less than six months, six months or more, don't know) and current status (ongoing, no longer in place, don't know). Table 3 displays the joint distribution of success, time since implementations, and current status for all 221 projects (big success, modest success, modest failure, big failure).
    Table 3
    Table 3
    Project Outcome, Duration, and Current Status (N=221)
  • which factor levels (as shown in Table 1) best described their project. They could choose level 1, 2, or 3, or they could say their project was somewhere between 1 and 2 or between 2 and 3. Our research team used the appropriate likelihood ratios and prior odds to calculate the posterior odds for the project.
The model has been designed to predict probability of success or failure of a project prior to its implementation. The data at hand ascertains success or failure of 221 actual projects. Hence, using actual success/failure as a dependent variable and Bayesian model scores as a predictor, we used both logistic regression and ROC analysis (Metz 1978; Swets 1988; Sainfort 1991) to evaluate the performance of the model. Specifically, the chi-square statistics from the logistic regression, the overall percentage of correct classification, and the area under the ROC curve are reported as measures of goodness of fit.
To analyze how well the model predicted the outcome, the outcome (success or failure) has to be known, leading to a smaller set (n=192). Three sets of analyses were performed using three definitions of success. In the first set, a very stringent selection criterion was applied to define success. In the second set, a slightly less stringent selection criterion was applied. The third set included all successful projects, independent of their duration.
Selected cases for the first analysis:
  • For successful (small or big success) projects, implementation must be ongoing for six months or more (n=45).
  • All failed projects (n=56).
Selected cases for the second analysis:
  • For successful (small or big success) projects:
  Either project has been implemented for six months or more (n=45),
  Or project is considered successful but has been implemented for less than six months and is still ongoing (n=42).
  • All failed projects (n=56).
 Selection criteria for the third analysis:
  • All successful (small or big success) projects whether ongoing or not (n=136). The rationale here is that a project could have been intended as a temporary, rather than permanent, intervention.
  • All failed projects (n=56)
Analysis 1
With a very stringent definition of success, 45 successful and 56 failed projects were retained. The logistic regression chi-square statistic is 29.62 and significant at the 0.001 level. The area under the ROC curve is 0.849 (standard error=.039) and is significantly greater than 0.50 at the 0.001 significance level.
Analysis 2
With a less stringent definition of success, 87 successful and 56 failed projects were retained. The logistic regression chi-square statistic is 42.24 and significant at the 0.001 level. The area under the ROC curve is 0.842 (standard error=.035) and is significantly greater than 0.50 at the 0.001 significance level.
Analysis 3
Including all successful projects, 136 successful and 56 failed projects were retained. The logistic regression chi-square statistic is 53.98 and significant at the 0.001 level. The area under the ROC curve is 0.846 (standard error=.032) and is significantly greater than 0.50 at the 0.001 significance level.
Table 4 shows summary results (chi-square statistics, area under the curve, standard error, significance level, and 95 percent confidence interval) for the three different analyses performed above. Thus, independent of the success definition used, the model did well in predicting success or failure.
Table 4
Table 4
Logistic Regression and ROC Results for the Three Outcome Definitions
This was a study to determine the extent to which a subjective Bayesian model could predict the outcome of organizational change attempts. The model performed well on all three definitions of success using logistic regression and ROC analysis. While results are encouraging, further research is needed.
First, more tests of reliability and validity of the survey and model are needed. For instance, only one person completed the survey for each of the 221 projects. In addition, the assessment of predisposing factors and the judgment of success were conducted by the same raters. Thus, this could have led to a slight overestimation of the model performance. What interrater reliability would be achieved had several people familiar with the project completed the survey? We subsequently asked the members of one improvement team (ranging in size from three to six people) in each of six hospitals, not part of the research reported above, to independently complete the survey. Average team member agreement on item scoring was 84 percent. On a related issue, our application of Bayes' Theorem assumes conditional independence. While we did not conduct a statistical analysis, the expert panel did engage in a formal process (described earlier) to identify and remove conditional dependencies. Further testing could be needed to ensure the validity of this assumption.
Second, there is no objective measure of success. Success was measured by opinions of the people who completed the survey. The survey was completed by one person who, although being very familiar with the project, may not have had an objective perspective. Ideally, a team of people from each organization would independently complete the survey and then after discussion reach a consensus on the survey as well as the level of success.
The process we employed is intended to produce a survey/model that both predicts and explains reasons for success and failure. During the factor elicitation process experts were encouraged to select factors that were not only correlated with success but were causally related to it and were actionable. While the survey/model was thus intended to predict and explain, the evaluation reported here did not assess the validity of the causal links.
Some factors that are arguably important in predicting and causally linked to organizational change were not included in the OCM; for example the characteristics of the project team and issues related to organizational culture. Both were included in the straw model, but not in the final model. The expert panel felt that the change agent was more important than the team and that culture was not easily modifiable (a key consideration in selecting factors for this model). Still, strong arguments could be made for their inclusion in such models.
Despite the study's limitations, the results are encouraging and the survey is easy to complete (taking approximately 15 minutes). Hence the OCM may be useful in applications such as those described below.
The OCM may help educate those seeking to bring about change. Many in these roles lack formal education in research and theory of successful change. Others may be blind to potential barriers, even though they understand the theory. The model may introduce factors critical to successful change and to current realities. It may also assist in establishing a common language and in preventing future conflict arising from lack of consensus. In theory these people could also learn about change by reading a literature review. However, a simple survey that provides feedback forces the user to think about issues in ways that a journal article might not.
The model may assist in determining whether change is likely to be worth the effort required. Failed change can be costly in resources consumed, opportunity costs, and an organization's loss of confidence in its ability to successfully change. Repeated failures can breed cynicism and increase resistance when the experience has been, “This too shall pass.” Thus, the model's predictive ability may help evaluate the wisdom of pursuing a particular change.
In planning change, the survey and model may lead to preventive action directed toward low scoring factors with high likelihood ratios. Barriers can be removed and critical aspects strengthened before an initiative is even begun. For instance it might be determined that leaders were going to make change desirable but not essential. Discussions prior to project launch might convince the leadership to take a more aggressive position on the importance of change.
In implementing change, the model may assist groups in evaluating an ongoing change process over time. Scoring the change at key checkpoints, and having group members complete the instrument and discuss resulting scores, can strengthen the likelihood of success.
The model may also help quantitatively track the direction of change. Completing the instrument several times during a major initiative may help keep the effort on track and measure the effect of corrective actions taken between evaluations. Using an instrument can help depersonalize issues and legitimatize change agents' requests to strengthen critical factors.
The model may help organizations learn from past efforts. Many organizations experience repeated failure of change efforts due to their inability to learn from experience. An organization's agility may be improved through repeated use of the model. This knowledge may be more systematically applied to future efforts, leading to more successful change efforts, and greater collective confidence in the change process.
However, the model's value in the applications described above requires empirical validation. While the model has been used for such purposes, the only evidence of effectiveness as an intervention (rather than a predictor) is anecdotal. Randomized trials, where improvement teams in one arm receive feedback from analyses of the survey and those in the other arm do not, would provide important insights into the OCM's potential as an agent of effective change. Other useful insight would come from prospective studies in which several people rate the chances of success and score the predisposing factors, and also from studies assessing the reliability of those assessments. These studies would help us determine the number of raters needed to ensure adequate reliability.
This research was funded under grant no. R01 HS10246-43 from the Agency for Healthcare Research and Quality, Principal Investigator, David Gustafson.
  • Alavi M, Henderson JC. “An Evolutionary Strategy for Implementing a Decision Support System.” Management Science. 1981;27(11):1309–23.
  • Amabile T. “A Model of Creativity and Innovation in Organizations.” In: Staw BM, Cummings LL, editors. The Evolution and Adaptation of Organizations. London: JAI Press; 1990.
  • Bandura A. “Self-efficacy: Toward a Unifying Theory of Behavioral Change.” Psychology Review. 1977;84(2):194–215. [PubMed]
  • Beatty CA, Lee GL. “Leadership among Middle Managers—An Exploration in the Context of Technological Change.” Human Relations. 1992;45(9):957–89.
  • Beckhard R, Harris R. Organizational Transitions: Managing Complex Change. Reading, MA: Addison-Wesley; 1987.
  • Beckhard R, Pritchard W. Changing the Essence: The Art of Creating and Leading Fundamental Change in Organizations. San Francisco: Jossey-Bass; 1992.
  • Beer M, Eisenstat R, Spector B. “Why Change Programs Don't Produce Change.” Harvard Business Review. 1990;68(6):158–66. [PubMed]
  • Burgelman RA. “Intraorganizational Ecology of Strategy Making and Organizational Adaptation: Theory and Field Research.” Organization Science. 1991;2:239–62.
  • Carroll G, Hannan MT. “Density Delay in the Evolution of Organizational Populations: A Model and Five Empirical Tests.” Administrative Science Quarterly. 1989;34:411–30.
  • Chakravarthy BS, Lorange P. Managing the Strategy Process. Englewood Cliffs, NJ: Prentice Hall; 1991.
  • Coch L, French J. “Overcoming Resistance to Change.” Human Relations. 1948;34(1):512–32.
  • Coghlan D. “Organizational Learning as a Dynamic Interlevel Process.” In: Rahim RA, Golembiewski R, Pat LE, editors. Current Topics in Management. Vol. 2. Greenwich, CT: JAI Press; 1997. pp. 27–44.
  • Compeau DR, Higgins CA. “Computer Self-efficacy: Development of a Measure and Initial Test.” MIS Quarterly. 1995;19(2):189–211.
  • Cooper RG. Project NEWPROD: What Makes a New Product a Winner? Montreal: Quebec Industrial Innovation Center; 1980.
  • Damanpour F. “Organizational Innovation: A Meta-Analysis of Effects of Determinants and Moderator.” Academy of Management Journal. 1991;34(3):555–90.
  • Dane CW, Gray CF, Woodworth BM. “Factors Affecting the Successful Application of PERT/CPM Systems in Government Organization.” Interfaces. 1979;9(5):94–8.
  • Delbecq AL. 1975 “Contextual Variables Affecting Decision Making in Program Planning.” Discussion paper, School of Social Work, University of Wisconsin.
  • Delbecq A, Mills P. Organizational Dynamics. 1. Vol. 14. 1985. “Managerial Practices that Enhance Innovation.” pp. 24–34.
  • Dickson JW. “The Adoption of Innovative Proposals as Risky Choice: A Model and Some Results.” Academy of Management Journal. 1976;19(2):291–303.
  • DiMaggio PJ. “Interest and Agency in Institutional Theory.” In: Zucker LG, editor. Institutional Patterns and Organizations. Cambridge, MA: Ballinger Publishing Company; 1988. pp. 3–21.
  • Donnellon A. “Language and Communication in Organizations: Bridging Cognition and Behavior.” In: Sims H, Gioia D, editors. The Thinking Organization. San Francisco: Jossey-Bass; 1986. pp. 136–64.
  • Duck JD. Harvard Business Review. 1993. “Managing Change: The Art of Balancing.” pp. 109–18. (November–December):
  • Edwards W, Lindman H, Savage L. “Bayesian Statistical Inference for Psychological Research.” Psychology Review. 1963;70:193–242.
  • Finstad N. “The Rhetoric of Organizational Change.” Human Relations. 1998;51(6):717–40.
  • Ford JD, Ford LW. “The Role of Conversations in Producing Intentional Change in Organizations.” Academy of Management Review. 1995;20(3):541–70.
  • Freeman C. The Economics of Industrial Innovation. Cambridge, MA: MIT Press; 1982.
  • Gersick CJ. “Revolutionary Change Theories: A Multilevel Exploration of the Punctuated Equilibrium Paradigm.” Academy of Management Review. 1991;16:10–36.
  • Ginzberg MJ. “Early Diagnosis of MIS Implementation Failure: Promising Results and Unanswered Questions.” Management Science. 1981;27(4):459–78.
  • Goodman PS. Change in Organizations. San Francisco: Jossey-Bass; 1982.
  • Guimaraes T. “Understanding Implementation Failure.” Journal of Systems Management. 1981;32(3):12–7.
  • Gustafson DH, Cats-Baril WL, Alemi F. Systems to Support Health Policy Analysis. Ann Arbor, MI: Health Administration Press; 1992.
  • Gustafson DH, Greist JH, Strauss FF, Erdman H, Laughren T. “A Probabilistic System for Identifying Suicide Attempts.” Computers and Biomedical Research. 1977;10(2):81–8. [PubMed]
  • Gustafson DH, Sainfort F, Johnson S, Sateia M. “The Construction, Reliability, Validity, Application, and Impact of a Quality Index for Psychiatric Emergencies.” Health Services Research. 1993;28(2):131–58. [PMC free article] [PubMed]
  • Gustafson DH, Sainfort F, Von Konigsveld R, Zimmerman D. “The Quality Assessment Index (QAI) for Measuring Nursing Home Quality.” Health Services Research. 1990;25(1):99–127. [PMC free article] [PubMed]
  • Gustafson DH, Shukla R, Delbecq AL, Walster W. “Comparative Study of Differences in Subjective Likelihood Estimates: Individuals, Interacting, Delphi, and Nominal Groups.” Organizational Behavior and Human Performance. 1973;9(2:):280–91.
  • Hage J, Aiken M. Social Change in Complex Organizations. New York: Random House; 1970.
  • Hall GE, Wallace RC, Jr., Dossett WA. A Developmental Conceptualization of the Adoption Process within Educational Institutions. Austin: University of Texas Research and Development Center for Teacher Education; 1973.
  • Hannan MT, Freeman F. Organizational Ecology. Cambridge, MA: Harvard University Press; 1989.
  • Hogarth RM. “Cognitive Processes and the Assessment of Subjective Probabilities.” Journal of American Statistical Association. 1975;70:271–89. and rejoinder, 294.
  • Hosking DM. “Organizing, Leadership and Skillful Process.” Journal of Management Studies. 1988;25(2):147–66.
  • Hunsucker JL, Loos D. “Transition Management: An Analysis of Strategic Considerations for Effective Implementation.” Engineering Management International. 1989;5:167–78.
  • Hurley JJP. “Towards an Organisational Psychology Model for the Acceptance and Utilization of New Technology Organisations.” Irish Journal of Psychology. 1992;13(1):17–31.
  • Iskat GJ, Liebowitz J. “What to Do When Employees Resist Change.” Supervision. 1996;57(8):3–5.
  • Jick T. “Implementing Change.” In: Jick T, editor. Managing Change. Homewood, IL: Irwin; 1993. pp. 192–201.
  • Kabanoff B, Waldersee R, Cohen M. “Espoused Values and Organizational Change Themes.” Academy of Management Journal. 1995;38(4):1075–104.
  • Kanter RM. The Change Masters: Corporate Entrepreneurs at Work. London: Unwin; 1983.
  • Kanter R, Stein B, Jick T. The Challenge of Organizational Change. New York: Free Press; 1992.
  • Kotter JP, Schlesinger LA. “Choosing Strategies for Change.” Harvard Business Review. 1979;57(2):106–14. [PubMed]
  • Lawler EE, Rhode JG. Information and Control in Organizations. Pacific Palisades, CA: Goodyear; 1976.
  • Lee WB, Steinberg E. “Making Implementation a Success of Failure.” Journal of Systems Management. 1980;3(4):19–25.
  • Leonard-Barton D, Kraus WA. “Implementing New Technology.” Harvard Business Review. 1985;63(6):102–10.
  • Lewin K. Field Theory in Social Science. New York: Harper & Row; 1951.
  • Lonnstedt L. “Factors Related to Implementation of Operations Research Solutions.” Interfaces. 1985;5(2):23–9.
  • Lorenzi NM, Riley RT. “Managing Change: An Overview.” Journal of the American Medical Informatics Association. 2000;7(2):116–24. [PMC free article] [PubMed]
  • Luke R, Stauss F, Gustafson DH. “Comparison of Five Methods of Estimating Subjective Probability Distribution.” Organizational Behavior and Human Performance. 1977;19:162–79. [PubMed]
  • McCaughan N, Palmer B. Systems Thinking for Harassed Managers. London: Karnac; 1994.
  • March JG, Olsen JP. Ambiguity and Choice in Organizations. Bergen, Norway: Universiteforlaget; 1976.
  • March JG, Simon HA. Organizations. New York: Wiley; 1958.
  • Maidique MA. “Entrepreneurs, Champions, and Technological Innovation.” Sloan Management Review. 1980;21(2):59–76.
  • Maidique M, Zirger B. “A Study of Success and Failure in Product Innovation. The Case of the U.S. Electronics Industry.” IEEE Transactions on Engineering Management. 1984;31(4):192–203.
  • Metz CE. “Basic Principles of ROC Analysis.” Seminars in Nuclear Medicine. 1978;8:283–98. [PubMed]
  • Meyer A, Goes J. “Organizational Assimilation of Innovations: A Multilevel Contextual Analysis.” Academy of Management Journal. 1988;31(4):897–923.
  • Morgan G. Riding the Waves of Change: Developing Managerial Competencies for a Turbulent World. San Francisco: Jossey-Bass; 1988.
  • Nadler DA, Tushman ML. “Beyond the Charismatic Leader: Leadership and Organizational Change.” California Management Review. 1990;32(2):77–97.
  • Orton JD, Weick KE. “Loosely Coupled Systems: A Reconceptualization.” Academy of Management Review. 1990;15(2:):203–23.
  • Paolillo J, Brown W. “How Organizational Factors Affect R and D Innovation.” Research Management. 1978;21:12–5.
  • Pearce CL, Osmond CP. “Metaphors for Change: The ALPs Model of Change Management.” Organizational Dynamics. 1996;24(2):23–35.
  • Quinn R, Cameron K. Paradox and Transformation: Toward a Theory of Change in Organizations and Management. Cambridge, MA: Ballinger; 1989.
  • Rashford NS, Coghlan D. The Dynamics of Organizational Levels: A Change Framework for Managers and Consultants. Reading, MA: Addison-Wesley; 1994.
  • Roberto E, Pinson C. “Compatibility Analysis for Screening New Products.” European Journal of Marketing. 1972;6(3:):182–9.
  • Rogers E. Communication of Innovations. 4th ed. New York: Free Press; 1995.
  • Rogers E. Diffusion of innovations. 3d ed. New York: Free Press; 1983.
  • Romano CA. Greenes RA, editor. “Predictor of Nurse Adoption of a Computerized Information System as an Innovation.” MEDINFO 95 Proceedings. 1995. pp. 1335–9. International Medical Informat. [PubMed]
  • Rousseau DM, Tijoriwala SA. “What's a Good Reason to Change? Motivated Reasoning and Social Accounts in Promoting Organizational Change.” Journal of Applied Psychology. 1999;84(4):514–28.
  • Sainfort F. “Evaluation of Medical Technologies: A Generalized ROC Analysis.” Medical Decision Making. 1991;11(3):208–20. [PubMed]
  • Schneider DM, Goldwasser C. “Be a Model Leader of Change.” Management Review. 1998;87(3):41–5.
  • Schon DA. “Champions for Radical New Inventions.” Harvard Business Review. 1963;41(2):77–86.
  • Schultz FL, Slevin DP. Implementing Operations Research/Management Science. New York: American Elsevier; 1975.
  • Senge P. The Fifth Disciple. New York: Doubleday; 1990.
  • Sink S. “The IE as the Change Master.” IIE Solutions. 1998:36–41. (October)
  • Slovic P, Lichtenstein L. “Comparison of Bayesian and Regression Approaches to the Study of Information Processing and Judgment.” Organizational Behavior and Human Performance. 1971;(6):649–744.
  • Smith MJ, Carayon P. “New Technology, Automation, and Work Organization: Stress Problems and Improved Technology Implementation Strategies.” International Journal of Human Factors in Manufacturing. 1995;5(1):99–116.
  • Snyder JR. “Managerial Intervention to Facilitate Organizational Change.” American Journal of Medical Technology. 1985;49(7):513–8. [PubMed]
  • Starbuck WH. “Organizations and Their Environments.” In: Dunnette MD, editor. Handbook of Industrial and Organizational Psychology. Chicago: Rand McNally; 1976. pp. 1069–123.
  • Stjernberg T, Philips A. “Organizational Innovations in a Long-term Perspective: Legitimacy and Souls-of-Fire as Critical Factors of Change and Viability.” Human Relations. 1993;46(10:):1193–219.
  • Swets JA. “Measuring the Accuracy of Diagnostic Systems.” Science. 1988;240:1285–93. [PubMed]
  • Van de Ven A, Angle H, Poole M. Research on the Management of Innovation. New York: Harper & Row; 1989.
  • Van de Ven AH, Poole MS. “Explaining Development and Change in Organizations.” Academy of Management Review. 1995;20(3):510–40.
  • Vertinsky J, Barth R, Mitchell V. “A Study of OR/MS Implementation as a Social Change Process.” In: Schultz RL, Slevin DP, editors. Implementation Operations Research/Management Science. New York: Elsevier; 1975.
  • Von Winterfeldt D, Edwards W. Decision Analysis and Behavioral Research. New York: Cambridge University Press; 1986.
  • Weick KE. The Social Psychology of Organizing. 2d ed. Reading, MA: Addison-Wesley; 1979.
  • Weick KE. “Educational Organizations as Loosely Coupled Systems.” Administrative Science Quarterly. 1976;21:1–19.
  • Weick KE. Sensemaking in Organization. London: Sage; 1995.
  • Wynne BE. “Keys to Successful Management Science Modeling.” Interfaces. 1979;9(4):69–74.
  • Utterback J. “The Process of Innovation: A Study of the Origination and Development of Ideas for New Scientific Instruments.” IEEE Transactions on Engineering Management. 1971;18(4):124–31.
  • Utterback J, Allen T, Holloman J, Sirbus J. “The Process of Innovation in Five Industries in Europe and Japan.” IEEE Transactions on Engineering Management. 1976;23(1):3–9.
  • Zucker LG. “Normal Change or Risky Business: Institutional Effects on the Hazard of Change in Hospital Organization.” Journal of Management Studies. 1987;24(6):1959–74.
Articles from Health Services Research are provided here courtesy of
Health Research & Educational Trust