|Home | About | Journals | Submit | Contact Us | Français|
To examine the association between the scope of quality improvement (QI) implementation in hospitals and hospital performance on selected indicators of clinical quality.
Secondary data from 1997 mailed survey of hospital QI practices, Medicare Inpatient Database, American Hospital Association's Annual Survey of Hospitals, the Bureau of Health Professions' Area Resource File, and two proprietary data sets compiled by Solucient Inc. containing data on managed care penetration and hospital financial performance.
Cross-sectional study of 1,784 community hospitals to assess relationship between QI implementation approach and six hospital-level quality indicators.
Two-stage instrumental variables estimation in which predicted values (instruments) of four QI scope variables and control (exogenous) variables used to estimate hospital-level quality indicators.
Involvement by multiple hospital units in QI effort is associated with worse values on hospital-level quality indicators. Percentage of hospital staff and percentage of senior managers participating in formally organized QI teams are associated with better values on quality indicators. Percentage of physicians participating in QI teams is not associated with better values on the hospital-level quality indicators studied.
Results supported the proposition that the scope of QI implementation in hospitals is significantly associated with hospital-level quality indicators. However, the direction of the association varied across different measures of QI implementation scope.
Significant opportunities exist for improving the quality of care delivered in U.S. hospitals. As many as one-fourth of hospital deaths might be preventable; nearly 180,000 people die each year partly as a result of iatrogenic conditions. Moreover, as much as one-third of some hospital procedures expose patients to risks without improving their health; one-third of drugs prescribed are not indicated; and one-third of laboratory tests showing abnormal results do not get followed up by clinicians (Dubois and Brook 1988; Brook et al. 1990; Leape 1994; Institute of Medicine 2000).
Many believe that quality improvement (QI) represents a promising strategy for improving hospital quality of care. QI is a systemic approach to planning and implementing continuous improvement in performance. QI emphasizes continuous examination and improvement of work processes by teams of organizational members trained in basic statistical techniques and problem solving tools and empowered to make decisions based on their analysis of the data. The systemic focus of QI complements a growing recognition in the field that the quality of the care delivered by clinicians depends substantially on the performance capability of the organizational systems in which they work. While individual clinician competence remains important, many increasingly see the capability of organizational systems to prevent errors, coordinate care among settings and practitioners, and ensure that relevant, accurate information is available when needed as critical elements in providing high quality care (Institute of Medicine 2000). Reflecting the growing emphasis on organizational systems of care, the Joint Commission on Accreditation of Healthcare Organizations, the National Committee for Quality Assurance, and the Peer Review Organizations of the Centers for Medicare and Medicaid have all encouraged hospitals to use QI methods.
Although QI holds promise for improving quality of care, hospitals that adopt QI often struggle with its implementation (Shortell, Bennett, and Byck 1998). By implementation, we refer to the transition period, following a decision to adopt a new idea or practice, when intended users put that new idea or practice into use—for example, when clinical and nonclinical staff begin applying QI principles and practices to improve clinical care processes (Klein and Sorra 1996; Rogers 2003). Successful implementation is critical to the effectiveness of a QI initiative (Blumenthal and Kilo 1998; Shortell, Bennett, and Byck 1998). However, QI implementation is demanding on individuals and organizations. It requires sustained leadership, extensive training and support, robust measurement and data systems, realigned incentives and human resources practices, and cultural receptivity to change (Shortell, Bennett, and Byck 1998; Ferlie and Shortell 2001; Institute of Medicine 2001; Meyer et al. 2004). In addition, the systemic nature of many quality problems implies that the effectiveness of a QI initiative may depend on its implementation across many conditions, disciplines, and departments. This too often proves challenging (Gustofson et al. 1997; Blumenthal and Kilo 1998; Meyer et al. 2004). If successful, though, implementing QI in this manner may create a durable infrastructure for enhancing quality organization-wide.
In the present study, we examine the association of several dimensions of QI implementation in hospitals and hospital performance on selected indicators of clinical quality. To do so, we combine data from a national survey of hospital QI practices with a group of carefully screened and validated measures indicative of patient safety in hospital settings. In taking this approach, we seek to address several problems associated with existing research on hospital QI and quality of care. First, previous studies do not adequately account for differences in how hospitals implement QI. Consequently, the relative advantage of different implementation strategies remains unknown. Second, previous studies of hospital QI typically make use of small samples. This restricts the generalization of study findings to larger populations of hospitals and limits the extent to which study findings could be used to develop managerial or policy recommendations. Finally, data availability has precluded previous studies from examining a broad range of hospital quality indicators. This, in turn, has limited the opportunity to link specific QI structures and practices with a set of quality indicators that broadly reflect quality at the institutional level.
Study results will provide policy makers, accrediting bodies, and consumers with more precise information about how different approaches to QI implementation in hospital settings relate to a range of hospital-level quality indicators. Such information would facilitate the development of QI standards and benchmarks that make use of hospital-level quality indicators that are not only widely available, but also potentially amenable to change through the systematic application of QI practices. For instance, information on QI practices could be useful in the design of financial incentive programs to “make quality pay,” such as the recent CMS program to reward hospitals scoring in the top 10th percentile on various measures. Finally, such information would help hospital managers and clinicians target those approaches to QI implementation that provide the greatest value for resources expended.
QI embraces a philosophy of meeting or exceeding customer expectations through the continuous improvement of the processes of producing a good or service. QI posits that the quality of goods and services depends foremost on the processes by which they are designed and delivered. Hence, QI focuses on understanding, controlling, and improving work processes rather than on correcting individuals' mistakes after the fact. QI also assumes that uncontrolled variance in work processes is the primary cause of quality problems. Hence, QI focuses analyzing the root causes of variability, taking appropriate steps to make work processes predictable, and then continuously improving process performance.
Operationally, QI combines three elements: use of cross-functional teams to identify and solve quality problems, use of scientific methods and statistical tools by these teams to monitor and analyze work processes, and use of process-management tools (e.g., flow charts that graphically depict steps in a clinical process) to help team members use collective knowledge effectively. Cross-functional teams play an integral role in QI because most vital work processes span individuals, disciplines, and departments. Cross-functional teams bring together the many clinical professionals and nonclinical hospital staff members who perform a process to document the process in its entirety, diagnose the causes of quality problems, and develop and test possible solutions to them.
Although many hospitals have implemented QI, the effectiveness of these efforts has not been systematically examined, especially at the organizational level of analysis (Shortell, Bennett, and Byck 1998). Several studies have examined the structures, processes, and relationships common to designing, organizing, and implementing hospital QI efforts (Barsness et al. 1993a, b; Blumenthal and Edwards 1995; Gilman and Lammers 1995; Shortell 1995; Shortell et al. 1995b; Weiner, Alexander, and Shortell 1996; Weiner, Shortell, and Alexander 1997; Westphal, Gulati, and Shortell 1997; Berlowitz et al. 2003). These studies indicate that hospitals vary widely in terms of: (1) their approach to implementing QI; (2) the extent to which QI has “penetrated” core clinical processes; and (3) the degree to which QI practices have been diffused across clinical areas. Few of these studies, however, examined the effectiveness of hospital QI practices. With few exceptions (e.g., Westphal, Gulati, and Shortell 1997; Shortell et al. 2000), most have used perceptual measures of impact or self-reported estimates of cost or clinical impact rather than objectively derived measures of clinical quality (Gilman and Lammers 1995; Shortell et al. 1995b). Other multisite, comparative studies have explicitly examined the impact of hospital QI on clinical practice (Carlin, Carlson, and Nordin 1996; O'Connor et al. 1996; Gordian and Ballard 1997; Goldberg et al. 1998; Ferguson et al. 2003). However, nearly all of these studies focused on a single clinical quality indicator (e.g., risk-adjusted mortality for coronary artery bypass surgery [CABG], adverse drug event) or single clinical practice (e.g., immunization, guideline use) rather than a broad range of measures indicative of quality at an institutional level.
Our thesis is that higher values on multiple hospital-level quality indicators will be associated with the implementation of QI structures and practices that provide a durable infrastructure for continuous improvement. Specifically, we propose that the effectiveness of QI at the organizational level depends in part on the scope of QI implementation—that is, the extent or range of application of QI philosophy and methods. Our thesis is premised on the notion that QI achieves its full potential when it pervasively penetrates organizational routines and becomes a “way of doing business” throughout the organization. Such penetration is critical for sustainable success in enhancing quality across clinical conditions, organizational units, and time. We offer three arguments in support of the proposition that broad implementation scope enhances a hospital's capability to systematically improve work processes and thereby enhance quality organization-wide.
First, as noted earlier, most vital work processes in organizations span individuals, disciplines, and departments (Ishikawa 1985; Deming 1986; Juran 1988; James 1989; Walton 1990). Improving clinical care processes generally requires that clinical professionals and hospital staff from different specialties, functions, or units work together in order to document how the process works in its entirety and identify the key process factors that play a causal role in process performance. Implementing systemic changes also typically requires collaboration across disciplinary, functional, and unit boundaries. For example, improving cardiac surgery outcomes may entail multiple, simultaneous changes in physician offices, inpatient units, and home-health units (Gustofson et al. 1997). Even when implementing “local” changes (i.e., those within a single unit), cross-unit collaboration is often necessary in order to avoid undesirable, unintended consequences to arise in other units because of task interdependencies.
Second, enhancing quality on an organization-wide basis requires mobilizing large numbers of hospital staff, equipping them with technical expertise in QI methods and tools, and empowering them to diagnose and solve patient safety problems. A small number of people working together on a cross-functional team could, with the right support, make systems improvements that address a specific quality problem (e.g., stroke mortality). An organization-wide effort focusing on 5, 10, or even 15 quality problems, however, would require harnessing the knowledge, the energy, and the creativity of many hospital staff members.
Finally, extensive involvement of hospital staff across multiple units may also strengthen the effectiveness of QI efforts by promoting a “quality” culture. That is, pervasive participation in QI promotes shared values about the importance of continuous improvement, using data and scientific methods to identify problems, communicating openly, and collaborating to implement solutions. These shared values, in turn, support the implementation of systemic changes that cross disciplinary, departmental, and organizational boundaries (by reducing turf battles) and increase the likelihood of “holding the gain” (O'Brien et al. 1995).
In addition to these general arguments, we offer specific arguments in support of pervasive participation of two targeted groups: senior managers and physicians. Direct senior management participation in cross-functional QI teams signals to other organizational members that senior management views QI as a top priority. This, in turn, may strengthen the effectiveness of QI efforts by increasing the commitment and contributions of front-line workers. Moreover, senior managers who participate in QI teams may develop a deeper understanding of the root causes of quality problems and feel greater ownership of recommended solutions that such teams generate. As a result, senior managers may be more willing to commit the resources and make the policy changes necessary to ameliorate systemic causes of quality problems.
Widespread physician participation in QI teams may also be critical to QI effectiveness because physicians play a critical role in clinical resource allocation decisions and possess the clinical expertise needed to differentiate appropriate from inappropriate variation in care processes. Pervasive physician participation may not only enhance the quality of analysis and problem solving in QI teams, but also support the implementation of changes recommended by such teams. Research indicates that peer influence can be a powerful lever for provider behavior change. Hence, widespread physician participation in QI teams may facilitate those changes in physician behavior needed to address quality problems.
Data on hospital QI practices were derived from a 1997 national, mailed survey sent to the CEOs of all 6,150 U.S. hospitals. The CEO was asked to complete the survey and seek the assistance of the person responsible for the hospital's QI effort in order to ensure the most accurate data or assessment about the hospital's QI activities. The 26-page survey requested information about all hospital efforts to improve quality and did not assume (nor encourage respondents to make assumptions about) the superiority of any specific approach. The survey provided definitions of terms like “QI,” “quality assurance,” “continuous QI,” “total quality management,” and “QI project” in order to increase the consistency and comparability of respondents' answers. Of the 6,150 hospitals in the sampling frame, 2,300 (or 38 percent) responded to the survey. Multivariate logistic regression analysis showed that survey respondents were less likely than nonrespondents to be investor-owned, smaller, or unaffiliated with a healthcare system (see Appendix A, Tables 1A and 1B).
The Medicare Inpatient Database was the source of data for the risk-adjusted quality indicators used in the study. The Medicare database contains the universe of the inpatient discharge abstracts for Medicare patients, translated into a uniform format to facilitate multistate and multihospital comparisons. The Medicare data contain more than 100 clinical and nonclinical variables included in a hospital discharge abstract such as principal and secondary diagnoses, principal and secondary procedures, admission and discharge status, patient demographics (e.g., gender, age, and, for some states, race), expected payment source (e.g., Medicare, Medicaid, private insurance, self-pay; for some states, additional discrete payer categories, such as managed care), total charges, and length of stay.
In addition to these two principal data sources, the study used data from the 1997 and 1998 American Hospital Association's (AHA) Annual Survey of Hospitals, the 1998 Bureau of Health Professions' Area Resource File, and two proprietary datasets compiled by Solucient Inc. The AHA Annual Survey is administered in the fourth quarter of every year to all AHA registered and nonregistered facilities. The Area Resource File supplies county-specific data on an annual basis for numerous market and demographic factors, making it a rich source of information about the local operating environments of hospitals. Solucient Inc. supplied financial performance ratios for each hospital derived from the 1997 and 1998 Medicare Cost Reports. Solucient Inc. also provided information on county-level insurance coverage for six types of insurance, making possible the construction of managed care penetration measures. Solucient uses multiple sources of insurance coverage data and various models to construct these local estimates. These sources include: Population and Income Demographic Estimates, Claritas Inc.; CMS Medicaid State summaries; County Surveyor Data, Interstudy; health maintenance organization (HMO) county files, Interstudy; State Medicaid agency data, Interstudy; Oregon Department of Human Services; CMS Medicare TEFRA county files, Interstudy; and Current Population Survey (CPS).
Our study focuses on community hospitals located in the U.S. Therefore, from the 2,300 hospitals that responded to the 1997 QI survey, we eliminated federal hospitals, specialty hospitals, and hospitals located in U.S. territories. We also eliminated hospitals that responded to the QI survey but lacked a Medicare (HCFA) identification number or any MEDPAR discharges. These screening procedures generated a useable sample of 1,784 hospitals. Multivariate logistic regression analysis showed that sample hospitals were more likely than nonsample hospitals to be larger and system-affiliated, but less likely to be investor-owned or located in the East South and East North Central census regions (see Appendix A, Tables 1C and 1D).
Scope of a hospital's QI effort was measured by four variables. The extent of organizational deployment was defined as the average level of hospital unit involvement in the QI efforts, measured on a five-point scale that ranged from (1) “not at all” to (5) “very great extent”. The seven possible units included acute inpatient care, outpatient clinics, major physician offices or group practices, home health agencies, owned or affiliated nursing homes, owned or affiliated ambulatory surgery centers, and owned or affiliated hospices. Factor analysis supported the construction of a single scale, which showed reasonably good reliability (α = 0.73). The remaining three variables indicated the percentage of hospital senior managers participating in QI teams, the percentage of hospital staff (total full time equivalents) participating in QI teams, and the percentage of physicians participating in QI teams.
Analysis focused on six hospital-level Quality Indicators developed for the Agency for Healthcare Research and Quality (AHRQ) by the University of California San Francisco–Stanford Evidence-Based Practice Center (UCSF–Stanford EPC). Quality Indicators are measures that use discharge abstracts to identify or signal potential quality problems. AHRQ Quality Indicator development followed the Institute of Medicine's definition of quality, which is “the degree to which health services for individuals and populations increase the likelihood of desired health outcomes and are consistent with current professional knowledge” (Institute of Medicine 2001). AHRQ Quality Indicators include measures of inpatient mortality for medical conditions and procedures; measures of utilization for procedures for which there are questions of overuse, underuse, or misuse; and measures of volume for procedures for which evidence exists that a higher volume is associated with lower mortality.
AHRQ developed the Quality Indicators to address several limitations of the Health Care Utilization Project (HCUP) Quality Indicators, including the lack severity or risk adjustment and the absence of population-based denominators (Agency for Healthcare Research and Quality 2004d). The UCSF–Stanford EPC used an extensive process for identifying, refining, risk-adjusting, and evaluating the AHRQ Quality Indicators (Agency for Healthcare Research and Quality 2004a). Appendix B provides a brief description of this process. Although AHRQ developed the Quality Indicators for the purposes of national tracking and QI, some public and private purchasers and data organizations have used them for public reporting and pay-for-performance initiatives (Agency for Healthcare Research and Quality 2004b).
We selected six AHRQ Quality Indicators based on their favorable performance on four empirical tests of precision and five empirical tests of minimum bias (see Appendix B). Five indicators focused on inpatient procedures or conditions for which mortality rates have been shown to vary substantially across institutions and for which evidence suggests that high mortality, at least in part, may be associated with deficiencies in quality of care. These indicators were inpatient hospital mortality for CABG, acute myocardial infarction, congestive heart failure, stroke, and pneumonia. In addition, one indicator (bilateral catheterization) focused on a procedure whose use varies significantly across hospitals, and for which evidence suggests that a high rate of use represents inappropriate care. Right-sided catheterization coincidental with left-sided catheterization yields little clinically useful information and, hence, is contraindicated in most patients without specific indications (Agency for Healthcare Research and Quality 2004c). We constructed each Quality Indicator as a 2-year average (1997 and 1998) to smooth short-term fluctuations.
We included four categories of covariates expected to relate to QI scope, hospital quality indicators, or both: market characteristics, hospital characteristics, accreditation and quality standards, and length of QI experience. In terms of market characteristics, we included three competition variables: a variable-radius Herfindal index of market concentration, defined as the sum of the squared market shares of the hospitals within a radius capturing 75 percent of a hospital's patient admissions in 1997 (Gresenz, Rogowski, and Escarce 2004); the self-reported number of other hospitals with which the hospital directly competes for patients on either an inpatient or outpatient basis; and the self-reported intensity of competition for patients among hospitals in the market, measured on a seven-point scale that ranged from (1) “not at all intense” to (7) “highly intense”. We also included two managed care variables: managed care penetration, defined as the percentage of the total insured population in a county enrolled in a private risk, Medicare risk, or Medicaid risk insurance product in 1997; and self-reported percentage of patients for which hospital is paid on a capitated, negotiated per case rate, or discounted basis (excluding Medicare and Medicaid).
In terms of hospital characteristics, we included seven variables indicating the hospital's structural complexity. These included a binary variable indicating whether the hospital belonged to the Council of Teaching Hospitals (COTH); a binary variable whether the hospital was owned, leased, or sponsored by a health care system or health network; a binary variable indicating whether the hospital had its own governing board; three binary variables indicating whether a hospital had developed—on its own or through a health system, health network, or joint venture with an insurer—a HMO, preferred provider organization, or indemnity product (FFS); and a variable indicating the number of physician arrangements in which the hospital participates, either on its own or through a health system or health network. Eight physician arrangements were examined (e.g., independent practice association, integrated salary model).
Other hospital characteristics included (1) the number of registered nurse FTEs per inpatient day, divided by 1,000; (2) a measure of hospital volume: number of inpatient surgeries performed in the last 12 months, divided by 1,000; (3) two binary variables to indicate whether the hospital was public (nonfederal) or investor-owned; and (4) a measure of hospital financial health: 2-year average cash flow margin, defined as the net available for debt service (i.e., net patient revenue+total other income−total operating expense−total other expense+depreciation and amortization+interest expense) divided by the sum of net patient revenue and total other income, multiplied by 100; and (4) two measures of hospital service mix variables: total outpatient visits, including emergency room visits and outpatient surgeries, adjusted by hospital bed size and divided by 1,000; and ratio of the number of outpatient services offered by the hospital to the number of inpatient services offered by hospital. For the latter, 25 services listed in AHA Annual Survey were designated outpatient services; 47 were designated inpatient services.
With respect to accreditation and quality standards, we included three variables that indicated the self-reported influence of the Joint Commission on the Accreditation of Healthcare Organizations (JCAHO), Foundation for Accountability, and National Committee of Quality Assurance on the hospital's QI effort. These variables were measured on a five-point scale that ranged from (1) “no influence” to (5) “very great influence”.
Finally, we controlled for number of years since hospital first became involved in QI. We defined “involved” as the first training of organizational members in QI principles and methods or the substantive investment of top management's time in organizing QI. We used a square-root transformation to correct for positive skew.
Table 1 summarizes the descriptive statistics for all study variables.
We address our hypotheses by estimating regression models that relate the hospital-level quality indicators to the QI implementation scores, controlling for hospital organizational and financial characteristics and hospital market attributes. While results from such a model may be informative, a concern with this type of modeling is that an endogenous relationship might exist between hospital-level quality indicators and QI implementation scope. For example, poor performing hospitals may be motivated to correct performance problems by adopting initiatives aimed at improving quality of care. Or, some hospitals may have unobserved attributes that predispose them to higher quality of care, and which also increase the likelihood that they will invest in QI activities. If this happens, results would not reflect the true causal effect of QI implementation scores on quality indicators. To examine this possibility, we performed the Hausman test for endogeneity for each of our dependent variables. The test strongly rejected the hypothesis of no endogeneity (p<.0001) in five of six models (CABG mortality proved the exception). Based on these results, we took an instrumental variables (IV) estimation approach.
Estimation using IV provides a means of determining the extent to which any such bias exists, and, if necessary, correcting for it. To implement the IV technique, we first identified a set of “instruments,” that is, variables that are hypothesized to influence the level of a hospital's QI activity, but not directly influence quality of care. We selected potential instruments by reviewing research on the organizational and environmental factors associated with QI adoption and implementation. Prior studies show that QI implementation varies as a function of organizational receptivity to QI, appropriate and sustained leadership, measurement and information systems, and appropriate levels of funding (Dean and Bowen 1994; Hackman and Wageman 1995; Powell 1995; Weiner, Alexander, and Shortell 1996; Weiner, Shortell, and Alexander 1997). Based on these findings, and the data available to us, we identified 10 potential instruments that we hypothesized would influence the level of a hospital's QI activity, but not directly influence quality of care. These instruments included three measures of leadership for QI, four measures of hospital infrastructure for QI, and three measures of resources for QI (see Appendix C). In order to evaluate whether these instruments presented possible alternative pathways for “management concern for quality” to affect hospital quality of care, we conducted several MEDLINE searches to identify organizational predictors of hospital performance on quality indicators. Based on this review, we included additional variables in our models to control for potential unobserved determinants. For example, we added as a control variable the number of registered nurses per 1,000 patient days in order to statistically control for the possibility that “management concern for quality” could affect hospital quality of care through nursing staff quality.
We tested the predictive power of the instruments by regressing QI implementation scope variables first on the control variables and then on the controls and instruments (Bowden and Turkington 1990). We used these results to evaluate the IV set on three criteria: incremental contribution to variance explained, consistency of statistical significance across models, and consistency of direction of effects. We found that adding the set of IVs resulted in statistically significant increases in the variance explained, with changes in R2 ranging from a low of 0.04 for physician participation in QI teams to a high of 0.06 for hospital staff participation in QI teams. Moreover, the F-statistics for the joint statistical significance tests ranged from 9.71 for physician participation in QI teams (p<.001) to 14.28 for hospital staff participation in QI teams (p<.001). Finally, as reported below under Results, both the second and third criteria were substantially met. These results provide at least partial evidence that our proposed IV set meets the criteria for estimation using IVs.
We then estimated the models using a two-stage IVs approach. The first stage instrumented QI scope variables as a function of leadership for QI, hospital infrastructure and resources for QI. The second-stage model included both the predicted values (instruments) of the QI scope variables and the control (exogenous) variables in predicting each of eight hospital-level quality indicators. Results of both the first- and second-stage model are presented below.
Table 2 shows the results of the first stage model in which we regressed QI implementation scope variables on the control and IVs. Seven of the 10 proposed instruments exhibited statistically significant relationships with at least two measures of QI implementation scope. The four proposed instruments that did not meet this criterion included: board activity in QI, integrated database, and total expenses for QI. With the exception of total expenses for QI activities, these variables were, however, significantly associated with at least one of the QI scope measures. Those proposed instruments that achieved statistical significance generally displayed effects in the predicted direction. Only hospital size and its squared term failed to demonstrate such consistency across models. With respect to the control variables, we see some evidence that market competition is negatively associated with QI implementation scope. Hospitals in more concentrated markets exhibited greater proportional participation of physicians in QI teams and greater involvement of hospital units in QI, although the latter effect achieves only marginal significance (p<.10). Conversely, hospitals in markets with higher managed care penetration exhibited less involvement of hospital units in QI and less proportional physician participation in QI teams. Further, length of hospital involvement in QI was positively related to hospital staff, senior manager, and physician participation in QI teams.
Table 3 presents the results of the two-stage IV models. Hypothesis 1 predicted that greater hospital unit involvement in QI would be positively associated with higher values on quality of care indicators (e.g., lower adjusted mortality). Our results do not support this hypothesis. For five of the six quality indicator models, the parameter estimates for the instrumented version of hospital unit involvement in QI are statistically significant, but in the opposite direction predicted. That is, those hospitals that reported a higher average level of involvement of seven hospital units in hospital QI efforts exhibited poorer values on hospital-level quality indicators. Hospital unit involvement in QI did not exhibit a statistically significant association with CABG mortality, possibly because this model possessed less statistical power than the others.
Hypotheses 2–4 predicted, respectively, that the greater the percentage of hospital staff, senior managers and physicians participating in QI teams, the better the scores on hospital quality indicators. Results of the two-stage IV model are generally supportive of Hypotheses 2 and 3. The percentage of hospital staff participating in QI teams showed statistically significant, positive associations with four of the six hospital-level quality indicators. The percentage of senior managers participating in QI teams showed statistically significant associations with three of the six hospital quality indicators. However, the percentage of physicians participating in QI teams showed no statistically significant, positive associations with any of the six hospital-level quality indicators. Thus, Hypothesis 4 did not receive support.
This study assessed the association between one aspect of hospital QI activity, namely the scope of QI implementation, and several hospital-level indicators of clinical quality. To date, studies making use of hospital-level quality indicators have focused on identifying and describing differences among hospitals that might be indicative of potential quality problems. Relatively little research has focused on the possible determinants of such differences. This study represents one of the first attempts to systematically link management and clinical efforts to improve quality of care in the hospital setting with hospital-level quality indicators.
Study results generally support the proposition that the scope of QI implementation in hospitals is significantly associated with hospital-level quality indicators. However, the direction of the association varies across different measures of QI implementation scope. Consistent with expectations, hospitals that reported a higher percentage of hospital staff and senior managers participating in QI teams also exhibited higher values on hospital-level quality indicators. Contrary to expectations, hospitals that reported more extensive involvement of multiple hospital units in hospital QI efforts exhibited poorer values on hospital-level quality indicators.
Three interrelated interpretations might account for the negative association of hospital unit involvement in QI and hospital-level quality indictors. First, the effectiveness of a hospital's QI effort could suffer from a dilution of focus if extensive involvement by disparate hospital units (e.g., acute inpatient care units, outpatient clinics, and home health agencies) leads to the hospital to engage in a diverse array of potentially unrelated QI projects. Under such conditions, senior leaders and front-line organizational members might find it difficult to give each QI project the attention it deserves. Similarly, extensive involvement by multiple hospital units might spread a hospital's QI resources too thinly. QI efforts often require staffing expertise and financial resources to support the systematic study and continuous improvement of work processes. Hospitals might exhibit little or no improvement in hospital quality indicators if QI projects do not receive adequate technical and financial support. Finally, extensive involvement by multiple units could yield little to no improvement in hospital-level quality indicators if QI projects work at cross purposes because of poor coordination or inappropriate sequencing.
The divergent pattern of findings for our QI implementation scope measures has implications for QI research. Specifically, our findings suggest the need for further theoretical refinement of the construct of implementation scope. QI implementation appears to be a multidimensional construct, and future research would profit from conceptually and empirically distinguishing the diffusion of QI across organizational units from the mobilization of organizational members for QI. Differentiating the construct in this way could stimulate theory development and, by encouraging consistency in measurement, promote greater cumulativeness of empirical research.
Our divergent pattern of findings also has implications for QI practice. As noted earlier, quality experts emphasize that QI requires organization-wide commitment and involvement because most, if not all, vital work processes span many individuals, disciplines, and departments. Debate exists, however, about the best approach for encouraging organization-wide participation in QI. Some favor building a “critical mass” by providing training in QI philosophy and methods to many organizational members at the outset and then initiating a broad array of QI projects across the organization. Others favor providing “just-in-time” training for organizational members and focusing selected groups of organizational members on a few strategically important clinical quality issues. Since studies to date have not adequately accounted for differences in QI implementation, existing research offers little guidance on this issue.
Our study results suggest, however, that a blended approach might prove most useful. That is, hospitals might exhibit higher values on hospital-level quality indicators by encouraging many organizational members to participate in QI activities, yet limiting hospital deployment of QI to a few organizational units. Intriguingly, results indicate that greater participation of hospital staff and senior managers in QI teams is positively associated with higher values on several hospital-level quality indicators, not just one or two. Perhaps intensive mobilization of organizational personnel within organizational units (e.g., acute inpatient care) creates the “critical mass” necessary to overcome the structural, cultural, and technical barriers that often obstruct organization-wide application of QI or otherwise restrict the gains from QI activity to a few clinical outcomes.
The finding of no statistically significant associations between physician participation in QI teams and hospital-level quality indicators speaks to another practical issue: the role of physicians in clinical QI efforts. Many believe that lack of physician involvement represents the single most important obstacle to the success of clinical QI (Berwick, Godfrey, and Roessner 1990; Board 1992; Health Care Advisory Board 1992; McLaughlin and Kaluzny 1994; Blumenthal and Edwards 1995; Shortell 1995). Physicians play a central role in clinical resource allocation decisions and possess the clinical expertise needed to differentiate appropriate from inappropriate variation in care processes. Yet, reports indicate that physicians are reluctant to participate in QI projects because of distrust of hospital motives, lack of time, and fear that reducing variation in clinical processes will compromise their ability to vary care to meet individual needs (Blumenthal and Edwards 1995; Shortell 1995; Shortell et al. 1995a). Study results suggest that widespread physician participation in QI teams, while perhaps desirable, might not be necessary. Widespread participation of hospital staff and senior managers, it seems, is more important, at least for the hospital-level quality indicators examined here. Rather than attempting to mobilize much of the medical staff, hospital leaders could perhaps secure needed physician input by involving selected physicians on an as-needed basis.
Three important study limitations should be noted. First, because our analysis required the merger of several existing databases, our sample is not statistically representative of the population of U.S. community hospitals. Hence, while our sample appears to be formally representative of the population with respect to several organizational and environmental characteristics, we cannot discount the possibility of sampling bias. This suggests that caution should be exercised in generalizing our study findings to a specific hospital population.
Second, dealing with the potential for endogeneity in cross-sectional studies like this can be difficult. We used the IVs approach to address this issue here, but use of IV is not entirely without difficulty. In particular, in this study, as in many studies of this type, it is difficult to find perfect instruments. While we believe that the instruments we used were likely to produce useful results, and our statistical tests were consistent with their validity, if our instruments were imperfect the results from the IV estimation may also be inaccurate. Further work, either with alternate instruments or with alternate study designs that approach endogeneity in other ways (e.g., analysis of changes over time when hospitals adopt QI), could help us better understand these issues.
Finally, we explicitly acknowledge limits of our ability to measure actual quality of care. The literature on quality of care clearly underscores the difficulty of measuring quality in a valid, reliable fashion. Each approach to assessing quality is subject to criticism, regardless of whether measures derive from discharge abstract databases or from patient chart abstracts. We have been careful to not suggest that quality indicators derived from discharge data abstracts are the only measures, or even the best measures, of quality. Moreover, we scrupulously avoided using the term “quality of care” in describing our dependent variables, instead emphasizing the term “quality indicators.” By considering our measures as indicative of possible problems of overuse, underuse or misuse of health services, rather than quality of care per se, we sought to take a conservative position on these issues.
Despite these limitations, we believe that the breadth and depth of our hospital QI data, coupled with reliable, validated quality indicators, represents an advance over previous small-sample studies of hospital QI and provides a solid basis for subsequent research. Multiple stakeholders—from community members and patients to employers and purchasers—are demanding data and evidence from providers regarding the effectiveness of care. Despite a shift in attention toward clinical outcomes, there has not been a commensurate shift in efforts to examine why variations in clinical outcomes exist, and perhaps more importantly, what organizational practices and procedures are associated with improved quality indicators at an institutional level. The present study provides multiple stakeholders with information about the relationship of one aspect of hospital QI activity to hospital-level quality indicators. Given recent changes in JCAHO measurement requirements and Medicare reporting systems, a contemporary replication of this study would be valuable. In addition, more research is needed in order to investigate the association between other aspects of hospital QI activity (e.g., extensive use of outcomes data and statistical tools) and hospital-level quality indicators, and also to identify the organizational and market conditions under which specific QI practices affect quality indicators.
The following supplementary material for this article is available online:
Description of Survey Respondents and Usable Sample Relative to Nonrespondents, Nonsample Hospitals, and Population of U.S. Hospitals.
AHRQ Quality Indicator Development, Risk-Adjustment, and Evaluation.
Instrumental Variables Selection.
This research was supported by grant no. 5 R01 HS11317-02 by the Agency for Healthcare Research and Quality (AHRQ). Reviewed and approved by Institutional Review Board, School of Public Health, University of North Carolina at Chapel Hill (IRB 01-1257; approved 1/29/01). There are no disclosures or disclaimers.