Search tips
Search criteria 


Logo of bmcmrmBioMed Centralsearchsubmit a manuscriptregisterthis articleBMC Medical Research Methodology
BMC Med Res Methodol. 2010; 10: 32.
Published online 2010 April 19. doi:  10.1186/1471-2288-10-32
PMCID: PMC2873511

Global quantitative indices reflecting provider process-of-care: data-base derivation

John L Moran,corresponding author1 Patricia J Solomon,2 and the Adult Database Management Committee (ADMC) of the Australian and New Zealand Intensive Care Society (ANZICS)3



Controversy has attended the relationship between risk-adjusted mortality and process-of-care. There would be advantage in the establishment, at the data-base level, of global quantitative indices subsuming the diversity of process-of-care.


A retrospective, cohort study of patients identified in the Australian and New Zealand Intensive Care Society Adult Patient Database, 1993-2003, at the level of geographic and ICU-level descriptors (n = 35), for both hospital survivors and non-survivors. Process-of-care indices were established by analysis of: (i) the smoothed time-hazard curve of individual patient discharge and determined by pharmaco-kinetic methods as area under the hazard-curve (AUC), reflecting the integrated experience of the discharge process, and time-to-peak-hazard (TMAX, in days), reflecting the time to maximum rate of hospital discharge; and (ii) individual patient ability to optimize output (as length-of-stay) for recorded data-base physiological inputs; estimated as a technical production-efficiency (TE, scaled [0,(maximum)1]), via the econometric technique of stochastic frontier analysis. For each descriptor, multivariate correlation-relationships between indices and summed mortality probability were determined.


The data-set consisted of 223129 patients from 99 ICUs with mean (SD) age and APACHE III score of 59.2(18.9) years and 52.7(30.6) respectively; 41.7% were female and 45.7% were mechanically ventilated within the first 24 hours post-admission. For survivors, AUC was maximal in rural and for-profit ICUs, whereas TMAX (≥ 7.8 days) and TE (≥ 0.74) were maximal in tertiary-ICUs. For non-survivors, AUC was maximal in tertiary-ICUs, but TMAX (≥ 4.2 days) and TE (≥ 0.69) were maximal in for-profit ICUs. Across descriptors, significant differences in indices were demonstrated (analysis-of-variance, P ≤ 0.0001). Total explained variance, for survivors (0.89) and non-survivors (0.89), was maximized by combinations of indices demonstrating a low correlation with mortality probability.


Global indices reflecting process of care may be formally established at the level of national patient data-bases. These indices appear orthogonal to mortality outcome.


The outcomes paradigm is now a dominant influence within medicine [1] and critical care is no exception to this movement [2]. The 1986 paper by Knaus et al [3], evaluating outcomes of a cohort of 13 intensive care units (ICU), established the notion of institutional or provider performance within the critical care discipline by way of the nexus between risk-adjusted mortality and process-of-care, the latter established through questionnaire, on-site visit and case-note review. Similar investigations were concurrently reported in the general medical literature by Dubois and colleagues [4]. A discordant debate has subsequently occurred regarding the relationship between risk-adjusted mortality and process-of-care, the latter being variously assessed [5,6]. On the one hand mortality " unlikely to be a sufficient statistic for quality" [7]; yet, the felicity with which process may be measured is no guarantee that "measuring ...process and reporting performance will improve outcomes" [8]. Contra-wise to the relationship of process-of-care and mortality outcome, a recent study has suggested that the "notion that hospitals with higher risk-adjusted mortality rates have poorer quality care is neither consistent nor reliable" [9]. However, there is a certain circularity in these arguments: reliance on outcome measures (mortality or length of stay [10]) is criticised from the stand point of process-of-care [11] (adherence to checklists [12]), which in turn finds its (ultimate) assessment in terms of the effect on precisely those outcomes which have been rejected in the first place.

As opposed to a piecemeal examination of single indicators or a composite-scores approach [13], there would appear to be advantage in the establishment of global quantitative indices [14,15] which would subsume the diversity of process-of-care and avoid the necessity of direct examination of the modalities of the latter [12]. We sought to establish such indices at the level of a bi-national intensive care patient data-base [16,17], the Adult Patient Database (APD) of the Australian and New Zealand Intensive Care Society (ANZICS); by analysis of:

• the hazard of patient hospital discharge, estimated using time-to-event analysis [18], as reflecting the time-course of process-of-care. The components of the time-hazard curve were determined using pharmaco-kinetic methods.

• individual patient ability to maximize output [19], in this case length of stay, for a given set of physiological inputs, the individual patient component variables of the Acute Physiology and Chronic Health Evaluation (APACHE) III severity of illness algorithm [20]. This ability was conceptualised as one of technical production efficiency ("economic" efficiency [equivalent] output/input, scaled [0,1] [21]), estimated by the econometric technique of stochastic frontier analysis (SFA) [22].

We also determined the degree of correlation, or independence (orthogonality), between these global process-of-care indices and mortality.


Data sources

As previously described [16], the ANZICS APD [17] was interrogated to define an appropriate patient set over the time period 1993-2003. In brief, physiological variables collected, in accordance with the requirements of the APACHE III algorithm [20], were the worst in the first 24 hours after ICU admission, and all first ICU admissions to a particular hospital for the period 1993-2003 were selected. Records were used only when all three components of the Glasgow Coma Score (GCS) were provided; records for which all physiologic variables were missing were excluded, and for the remaining records, missing variables were replaced with the normal range and weighted accordingly. ICU and hospital length-of-stay, initially recorded in hours, were transformed to fractional days. Patients with an ICU length of stay > 60 days and hospital length of stay > 365 days were not considered in formal analysis. Exclusions: unknown hospital vital outcome and date of discharge; patients with an ICU length of stay ≤ 4 hours; and patients aged < 16 years of age. Access to the data was granted by the ANZICS Database Management Committee in accordance with standing protocols; local hospital (The Queen Elizabeth Hospital) Ethics of Research Committee approval was waived.

Data set-up

ICU-year units were formed by ICU-site × calendar-year interaction with a minimum patient number set at 150 to ensure estimation stability. Categorical predictors were parameterized as indicator variables with the reference level (= 0) indicated in parentheses in the following list:

• mechanical ventilation (not ventilated)

• ICU-level, as defined in the ANZICS database, as Rural, Metropolitan, Tertiary and Private (Tertiary)

• Geographical-location; that is New Zealand and the States of the Commonwealth of Australia, excluding Western Australia (New South Wales (NSW), the largest contributor). A reference population-density map is provided in Additional file 1.

• patient surgical status as post-elective surgery, post-emergency surgery and non-surgical (non-surgical)

• annual patient admission number (reflecting ICU "size"), created by the ICU-site × calendar-year interaction, was categorized at the median, the reference category being that denoting the highest number of yearly admissions

Statistical analysis

Variables were reported as mean (SD), except where otherwise indicated. Distributional form of variables of interest was displayed using kernel density estimates and the empirical distribution and parameter 95% confidence intervals (CI) were computed via bootstrapping (1000 repetitions) [23]. In the presence of skewed distribution, median point-estimates were reported. Group differences between continuous variables were estimated using parametric or non-parametric analysis of variance (ANOVA, adjusted (Sidak) for multiple comparisons) where indicated. Stata™ (Version 10 MP, 2007; College Station, Texas) statistical software was used.

Mortality probabilities

Hospital mortality probabilities (MP), summed over descriptors, were estimated using a two-level, patients within ICU-year units, random effects [random-intercept] logistic regression model as previously reported [16]. Descriptor standardised mortality rates (SMR) and 95%CI were estimated using the parametric approach of Rapoport et al [24].

Hazard of discharge

The unit of analysis was patients within descriptors. For hospital survivors, length of stay of non-survivors was defined as >> the maximum length-of-stay of alive discharges, with no censoring. The hazard of hospital (or ICU) discharge was computed via time-to-event analysis with kernel density smoothing [18]. Similarly, for hospital non-survivors, length of stay of survivors was defined >> maximum length-of-stay of those dying, with no censoring [25]. Such an approach obviated the analytical problem of competing risks [26]. The smoothed hazard estimates and 95%CI, truncated at 30 days and adjusted for the mean values of the covariates age and APACHE III score for each descriptor, were output as individual data files and separately processed using standard pharmaco-kinetic techniques (the Stata™ "pk" suite of commands [27]) to estimate parameters of the "hazard" profile; in particular: peak hazard (CMAX) and time to peak hazard (TMAX), reflecting the initial intensity of and time to the maximum (rate of) patient hospital discharge, respectively; area under the hazard curve (AUC), reflecting the integrated experience of the discharge process; and the "elimination rate" (KE), the negative of the parameter estimate for a linear regression of log(time) on hazard, which determines the half-life (t1/2) of the overall hazard-profile (An external file that holds a picture, illustration, etc.
Object name is 1471-2288-10-32-i1.gif). The justification for this approach was that an initial (random effects) first-order compartment model [28] provided a good fit to the data (see Additional file 2). These parameter estimates were understood as global indices reflecting aspects of descriptor process-of-care.

Technical (production) efficiency

Patients were considered as producers, seeking to avoid waste by obtaining maximised outputs from given inputs or, by minimizing input use in the production of given outputs [22]; where, in this context, "maximize" is used in the sense of optimize. The notion of productive efficiency corresponds to technical efficiency, the latter being estimated by SFA. A stochastic production frontier model may be estimated, using the Stata™ module "frontier", as a log-linear function (f): An external file that holds a picture, illustration, etc.
Object name is 1471-2288-10-32-i2.gif; where TEi = exp(-ui) and ui > 0, here assumed exponentially distributed; yi is ICU/hospital length of stay; xijs are (logged) acute physiology score and chronic health evaluation variables (age, GCS, temperature, heart rate, arterial pH, arterial PaCO2, creatinine, mean arterial pressure, white cell count, plasma bilirubin, plasma glucose and total (APACHE III) co-morbidity count); An external file that holds a picture, illustration, etc.
Object name is 1471-2288-10-32-i3.gif, i = 1, ..., N (idiosyncratic patient component). Adjustment for heteroscedasticity of the variance function of both u and v (as provided for by the Stata™ module "frontier") was undertaken in model development with a combination of appropriate patient (gender, patient surgical status), treatment (ventilation status) and provider descriptor variables (calendar year, ICU level, annual patient admission number and geographical locality). Patient efficiencies (TE, scaled [0, 1], where 1 [equivalent] the optimal production frontier) were summed over categorical descriptors of interest.

Data display

The multivariate relationships (joint distribution) between the indices (MP, TE, TMAX, AUC, CMAX and KE), for survivors and non-survivors across descriptors, were displayed using biplots [29]. Biplots consist of lines, reflecting the dataset variables, and "dots" to show the observations. The length of the lines approximates the variances of the variables (the longer the line, the higher is the variance) and the cosine of the angle between the lines approximates the correlation; the closer the angle is to 90, or 270 degrees, the smaller the correlation (orthogonality); an angle of 0 or 180 degrees reflecting a correlation of 1 or -1, respectively. Variable inclusion in the biplots was adjusted to maximize the explained variance.


The data set consisted of 223129 patients from 99 ICUs over an 11 year period. Mean (SD) age and APACHE III score were 59.2 (18.9) years and 52.7 (30.6) respectively; 41.7% were female and 45.7% were mechanically ventilated within the first 24 hours post ICU-admission. Overall ICU and hospital mortalities were 10.4% and 16.1% respectively. ICU length of stay was 3.6 (5.6) (median 1.8, interquartile range 2.9 [0.9-3.8]) days and hospital length of stay was 16.4 (19.5) (median 10.1, interquartile range 14.6 [5.1-19.7]) days. Patient categorization was non-operative (55.2%), elective surgical (28.7%) and emergency surgical (16.1%). Annual patient admission number created by the ICU-site × calendar-year interaction, was categorized at the median (711 patients); this categorization was further used to create "ICU-hospital/geographical area/size" descriptors (n = 35).

Hazard of discharge and technical efficiency

Hazard-of-discharge curves, for geographical areas and hospital-ICU levels, are seen for alive-discharges and those dying in Figures Figures11 and and22 respectively. The curves for those dying were not unexpectedly displaced to the left compared with those surviving. Initial analysis of the smoothed hazard curves used a non-linear mixed effects approach which yielded parameter estimates displaying good between-descriptor discriminative properties (Additional file 2). As this analysis required specialised software [28], standard pharmacokinetic parameters, with bootstrapped 95% CI, were calculated. Descriptor technical efficiencies (Figure (Figure3)3) and mortality probabilities revealed non-normality and median point estimates were reported. Complete summaries at the level of geographical × hospital-ICU level categories × ICU "size", with standardised mortality rates, are given for both survivors and non-survivors in Tables Tables11 and and22 respectively.

Figure 1
Kernel density smoothed hazard curves of alive-discharge for geographic descriptors (New Zealand and States of the Commonwealth of Australia, as indicated) for rural, metropolitan, tertiary and private hospital ICUs.
Figure 2
Kernel density smoothed mortality hazard curves for geographic descriptors (New Zealand and States of the Commonwealth of Australia, as indicated) for rural, metropolitan, tertiary and private hospital ICUs.
Figure 3
Kernel density estimates of technical efficiencies (solid line) for various geographic descriptors, overlaid with a normal density plot.
Table 1
Parameter estimates and 95% CI for survivors
Table 2
Parameter estimates and 95% CI for non-survivors

Summary statistics for process-of-care indices and mortality probabilities, for both survivors and non-survivors, are displayed for descriptors ICU "size" (> 711, < 711 admissions per year); ICU level by "size" and geographical-location by "size" in Tables Tables33 and and4.4. For both survivors and non-survivors, indices demonstrated statistically significant differences between the individual categories of various descriptors: ICU "size" (> 711, < 711 admissions per year); ICU level; ICU level by "size"; geographical-location by "size"; and geographical × hospital-ICU level categories × ICU "size" (ANOVA, P ≤ 0.0001); albeit these differences reflected the large size of both the initial data-set and the empirical distributions of the indices (1000 bootstrap samples). Maximal values of indices were seen (Tables (Tables33 and and4;4; ANOVA, P < 0.001 compared with all other ICU levels) for:

Table 3
Summary statistics for survivors for various descriptors
Table 4
Summary statistics for non-survivors for various descriptors

the descriptor ICU level by "size":

1. for survivors, at the rural and private level (both < 711 and >711 yearly admissions) for AUC; at tertiary levels (both < 711 and >711 yearly admissions) for TMAX and TE; at rural (>711 yearly admissions) and metropolitan (<711 yearly admissions) levels for KE; at rural and private levels for CMAX.

2. for non-survivors, at the tertiary level (both < 711 and >711 yearly admissions) for AUC; at the private level for TMAX; at rural (<711 yearly admissions) and metropolitan (<711 yearly admissions) levels for KE; at the tertiary levels (both < 711 and >711 yearly admissions) for CMAX

the descriptor ICU "size":

1. for survivors: for ICUs with > 711 admissions per year for AUC and TE; for ICUs with < 711 admissions per year for TMAX

2. for non-survivors: for ICUs with > 711 admissions per year for AUC; for ICUs with < 711 admissions per year for TMAX and TE.

Multivariate relationships between indices

The relationships between the global indices are demonstrated in Figure Figure4.4. For survivors (left panel), the total explained variance was 0.886, and the relationships appeared discrete: both technical efficiency (TE) and time to maximal hazard (TMAX) were coincident and tended to be orthogonal to indices reflecting the maximum hazard of alive (hospital) discharge (CMAX) and the total discharge experience of the patient (AUC). When included in the plot, mortality probability (and SMR) was, not surprisingly, directionally opposite to that of AUC and did not increase the explained variance. For non-survivors (right panel), the total explained variance was 0.892. AUC and mortality probability were almost coincident and directionally equivalent to CMAX, but orthogonal to both TMAX and TE. The SMR and mortality probability were coincident (not shown). The "elimination rate" (KE) added no increment to the total explained variation for survivors or non-survivors.

Figure 4
Biplots for survivors and non-survivors demonstrating the multivariate relationships between global performance indices. CMAX: peak hazard; TMAX: time to peak hazard; AUC: area under the hazard curve; KE: "elimination rate", TE: technical efficiency; ...


We have proffered a number of global indices, their uncertainty estimates and multivariate relationships, as reflecting the integrated discharge process for both survivors and non-survivors at the level of various data-base descriptors. For each of the indices, we were able to establish formal statistical difference between, and characteristic clustering within, database descriptor categories for both survivors and non-survivors. These indices would appear to be both internally consistent and plausible proxies for the underlying process(es)-of-care which, in concert with patient characteristics, determine the shape of the particular descriptor hazard of discharge.

Our motivating concepts were two-fold:

1. that of "conditional length of stay", introduced by Silber and colleagues, as the length of stay after a stay is prolonged, reflecting both patient complications and/or co-morbid medical conditions, and provider ability to manage complicated cases [30,31]. As suggested by Silber et al, and based upon empirical analyses: "By studying the CLOS, one can determine when the rate of hospital discharge begins to diminish-without the need to directly observe complications. Policy makers looking for an objective outcome measure may find that CLOS aids in the analysis of a hospital's management of complicated patients..." [31]. The analytic focus was restricted to the prolongation point or day, as estimated by the Hollander and Proschan statistic [32]. Our development of this approach involved modelling the hazard-profile as a "concentration" curve with the estimation of standard pharmaco-kinetic based parameters which would allow a more complete description of the hazard-time curve, in the sense of modelling underlying processes, and allowing formal statistical comparisons.

2. the notion of efficiency. In an innovative study of patients with severe head trauma, Nathanson et al [19] used data envelopment analysis (DEA) to calculate individual patient "efficiency " scores ([0 to 1]) based upon the ability to maximize output (in this case, cerebral perfusion pressure) for a given set of physiological inputs. Patients with high efficiency scores were found to have an improved functional outcome on ICU discharge. At the individual ICU level, Junoy [33] used DEA to establish an efficiency frontier across the bivariate relationship between severity adjusted survival (effectiveness or output) and length of stay (resources or input) and to compute technical efficiency of "output quality". Length of stay, raw, adjusted or observed minus expected, has been used as an indicator of ICU/hospital performance, from both an economic [34] and quality [35] perspective. Traditionally, length of stay has been estimated by ordinary least squares regression (or its variants) by maximising the mean expectation; the residual (difference between observed and predicted) being interpreted, in the current context, as arising from inefficiency. We applied the concept of "efficiency" to individual patient length of stay to model technical production efficiency using a parametric SFA approach. The latter separates the residual into an inefficiency component (ui, positive departures from the (best practice) production frontier) and all other sources of model error (vi), such as random shocks and measurement error [15,36]. To this extent it is less sensitive to outliers than DEA, a deterministic non-parametric technique, assuming no measurement error and requiring a more rigid best-practice production frontier based upon a small subset of efficient peers [36].

That these indices captured aspects of descriptor process-of-care was suggested by the analysis of maximum values. For instance, tertiary level ICUs where more complex and severely ill patients were located [16] and human and material resources were presumably greatest, demonstrated maximum TE for survivors and AUC for non-survivors; whereas, at the rural and for-profit ICU levels, where such patient and resource conditions did not necessarily obtain, AUC was maximal for survivors. With respect to ICU "size", maximal TE was located in tertiary ICUs (> 711 yearly admissions) for survivors; but of interest, for non-survivors, was located in Private ICUs (< 711 admissions per year). These "size" effects must be interpreted against the favourable effect of ICU "size" < 711 yearly admissions on hospital mortality (OR 0.84, P < 0.0001 compared with > 711 yearly admissions) adduced in a previous analysis of the same data-base [16]. The regional-geographic differences in these indices (Tables (Tables1,1, ,2,2, ,3,3, ,4,4, above) presumably reflect both the particular nature of our data-base and determinants such as the distribution of human [37,38] and non-human resources [39] and socio-economic factors.

Analysis of the joint distribution of the global process-of-care indices revealed a weak correlation with, or orthogonality to, mortality outcome (SMR and/or mortality probability). This would appear to be the first formal demonstration of such a (lack-of) relationship and is consistent with the literature reviews of Thomas and Hofer [40] and Pitches et al [9], although both reviews noted considerable and possibly confounding between-study heterogeneity in recorded process-of-care and mortality risk-adjustment measures. The low sensitivity of individual process-of-care measures has also been previously noted [41]. This being said, any such formal "independence" of these measures would have important implications for both health policy and the design and interpretation of trials assessing process-of-care interventions [42].

Critique of methodology

Our postulates are predicated upon the utility of both the pharmaco-kinetic and efficiency analyses. The former would appear to have basis in the good fit of a first-order compartment model; this approach may be formally extended to embrace non-linear mixed effects modelling [28]. Controversy has attended the appropriate form of efficiency analysis [43] and the use of either DEA or SFA; much of this criticism is context dependent; for instance, the efficiency of public services where multiple outputs occur. Our use of SFA was motivated by modelling flexibility and potential extensions to accommodate random effects and different distributions for the inefficiency component [44,45].

Our analysis, at the level of a bi-national database, lacks the empirical grounding of the investigations of Silber and co-workers [30,31] mentioned above and would require such validation. However, at the analytic level, these indices appeared both consistent and intuitively reasonable. In the interests of parsimony the primary descriptors were geographical and hospital-ICU level categories and the data-set was considered as a single cross-sectional unit; that is no analysis by calendar year was undertaken, although this ensured estimation stability. Similarly, we considered only hospital outcomes and truncated analysis time to 30 days, the latter again for estimation stability in the output files from the smoothed hazard estimates. However, our approach is rich with possibilities for various extensions to the individual ICU level and ICU hazard-of-discharge, longitudinal analysis [46], and obviously, different non-ICU patient cohorts.


Global indices reflecting process of care may be formally established at the level of national patient data-bases, thus allowing comparisons between providers/descriptors. These indices appear orthogonal to mortality outcome; such a relationship would have implications for health care policy and the design and interpretation of trials assessing process-of-care interventions.

Competing interests

The authors declare that they have no competing interests.

Authors' contributions

The study was conceived, designed, (data)-analysed, written and critically revised jointly by both authors (JLM, PJS).

Pre-publication history

The pre-publication history for this paper can be accessed here:

Supplementary Material

Additional file 1:

Reference population-density map of Australia and New Zealand. Population density map (inhabitants per square kilometre) of Australia (left figure) and New Zealand (right figure); not drawn to scale. Approximate land masses: Australia, 7706143 km2; New Zealand 270626 km2. Gray-scale legends indicate inhabitant number (range) per km2. WA, Western Australia; NY, Northern Territory; QLD, Queensland; NSW, New South Wales; VIC, Victoria; TAS, Tasmania.

Additional file 2:

Random effects first-order compartment model. A brief explanation of the self-starting first-order compartment model. Figure: Parameter estimates with 95%CI for a first-order compartment non-linear (self-starting "ssfol") mixed effects model for geographical descriptors. TAS, Tasmania; NSW, New South Wales; QLD, Queensland; VIC, Victoria; NT, Northern territory; NZ, New Zealand.


  • Davies HT, Crombie IK. Interpreting health outcomes. J Eval Clin Pract. 1997;3:187–199. doi: 10.1046/j.1365-2753.1997.00003.x. [PubMed] [Cross Ref]
  • Sibbald WJ, Bion JF. Evaluating critical care: using health services research to improve quality. Berlin: Springer; 2001.
  • Knaus WA, Draper EA, Wagner DP, Zimmerman JE. An evaluation of outcome from intensive care in major medical centers. Ann Intern Med. 1986;104:410–418. [PubMed]
  • Dubois RW, Rogers WH, Moxley JH III, Draper D, Brook RH. Hospital inpatient mortality. Is it a predictor of quality? N Engl J Med. 1987;317:1674–1680. [PubMed]
  • Mant J. Process versus outcome indicators in the assessment of quality of health care. Int J Qual Health Care. 2001;13:475–480. doi: 10.1093/intqhc/13.6.475. [PubMed] [Cross Ref]
  • Moran JL, Solomon PJ. Mortality and other event rates: what do they tell us about performance? Critical Care and Resuscitation. 2003;5:292–303. [PubMed]
  • Spiegelhalter DJ. Surgical audit: statistical lessons from Nightingale and Codman. Journal of the Royal Statistical Society A. 1999;162:45–58. doi: 10.1111/1467-985X.00120. [Cross Ref]
  • Allison JJ. Quality assessment tools: incremental advance or paradigm shift? Med Care. 2003;41:575–578. doi: 10.1097/00005650-200305000-00002. [PubMed] [Cross Ref]
  • Pitches D, Mohammed M, Lilford R. What is the empirical evidence that hospitals with higher-risk adjusted mortality rates provide poorer quality care? A systematic review of the literature. BMC Health Services Research. 2007;7:91. doi: 10.1186/1472-6963-7-91. [PMC free article] [PubMed] [Cross Ref]
  • Cleary PD, Greenfield S, Mulley AG, Pauker SG, Schroeder SA, Wexler L. Variations in length of stay and outcomes for six medical and surgical conditions in Massachusetts and California. JAMA. 1991;266:73–79. doi: 10.1001/jama.266.1.73. [PubMed] [Cross Ref]
  • Rubin HR, Pronovost P, Diette GB. Methodology Matters. From a process of care to a measure: the development and testing of a quality indicator. Int J Qual Health Care. 2001;13:489–496. doi: 10.1093/intqhc/13.6.489. [PubMed] [Cross Ref]
  • Duckett SJ, Coory M, Sketcher-Baker K. Identifying variations in quality of care in Queensland hospitals. Med J Aust. 2007;187:571–575. [PubMed]
  • Reeves DP, Campbell SMP, Adams JP, Shekelle PGM, Kontopantelis EP, Roland MOD. Combining Multiple Indicators of Clinical Quality: An Evaluation of Different Analytic Approaches. Med Care. 2007;45:489–496. doi: 10.1097/MLR.0b013e31803bb479. [PubMed] [Cross Ref]
  • Horn SD. Performance Measures and Clinical Outcomes. JAMA: The Journal of the American Medical Association. 2006;296:2731–2732. doi: 10.1001/jama.296.22.2731. [PubMed] [Cross Ref]
  • Smith PC, Street A. Measuring the efficiency of public services: the limits of analysis. Journal of the Royal Statistical Society: Series A (Statistics in Society) 2005;168:401–417. doi: 10.1111/j.1467-985X.2005.00355.x. [Cross Ref]
  • Moran JL, Bristow P, Solomon PJ, George C, Hart GK. for the Australian and New Zealand Intensive Care Society Database Management Committee (ADMC) Mortality and length-of-stay outcomes, 1993-2003, in the binational Australian and New Zealand intensive care adult patient database. Crit Care Med. 2008;36:46–61. doi: 10.1097/01.CCM.0000295313.08084.58. [PubMed] [Cross Ref]
  • Stow PJ, Hart GK, Higlett T, George C, Herkes R, McWilliam D. Development and implementation of a high-quality clinical database: the Australian and New Zealand Intensive Care Society Adult Patient Database. J Crit Care. 2006;21:133–141. doi: 10.1016/j.jcrc.2005.11.010. [PubMed] [Cross Ref]
  • Klein JP, Moeschberger ML. Survival Analysis: Techniques for Censored and Truncated Data. 2. New York: Springer-Verlag; 2003. Topics in univariate estimation: estimating the hazard function; pp. 166–176.
  • Nathanson BH, Higgins TL, Giglio RJ, Munshi IA, Steingrub JS. An Exploratory Study Using Data Envelopment Analysis to Assess Neurotrauma Patients in the Intensive Care Unit. Health Care Management Science. 2003;6:43–55. doi: 10.1023/A:1021912320922. [PubMed] [Cross Ref]
  • Knaus WA, Wagner DP, Draper EA, Zimmerman JE, Bergner M, Bastos PG. The APACHE III prognostic system. Risk prediction of hospital mortality for critically ill hospitalized adults. Chest. 1991;100:1619–1636. doi: 10.1378/chest.100.6.1619. [PubMed] [Cross Ref]
  • Farrell MJ. The measurement of productive efficiency. Journal of the Royal Stataistical Society Series A. 1957;120:253–290. doi: 10.2307/2343100. [Cross Ref]
  • Worthington AC. Frontier efficiency measurement in health care: a review of empirical techniques and selected applications. Med Care Res Rev. 2004;61:135–170. doi: 10.1177/1077558704263796. [PubMed] [Cross Ref]
  • Carpenter J, Bithell J. Bootstrap confidence intervals: when, which, what? A practical guide for medical statisticians. Stat Med. 2000;19:1141–1164. doi: 10.1002/(SICI)1097-0258(20000515)19:9<1141::AID-SIM479>3.0.CO;2-F. [PubMed] [Cross Ref]
  • Rapoport J, Teres D, Lemeshow S, Gehlbach S. A method for assessing the clinical performance and cost-effectiveness of intensive care units: a multicenter inception cohort study. Crit Care Med. 1994;22:1385–1391. doi: 10.1097/00003246-199409000-00006. [PubMed] [Cross Ref]
  • Silber JH, Rosenbaum PR, Even-Shoshan O, Shabbout M, Zhang X, Bradlow ET. Length of stay, conditional length of stay, and prolonged stay in pediatric asthma. Health Serv Res. 2003;38:867–886. doi: 10.1111/1475-6773.00150. [PMC free article] [PubMed] [Cross Ref]
  • Schoenfeld D. Survival methods, including those using competing risk analysis, are not appropriate for intensive care unit outcome studies. Critical Care. 2006;10:103. doi: 10.1186/cc3949. [PMC free article] [PubMed] [Cross Ref]
  • StatCorp.2007. Stata Reference Manual (Release 10): Reference I-P. College Station, TX: StataCorp LP; 2007. pk - Pharmacokinetic (biopharmaceutical) data; pp. 532–575.
  • Pinheiro JC, Bates DM. In: Mixed-effects models in S and S-Plus. Pinheiro JC, Bates DM, editor. Rensselaer, NY: Springer-Verlag New York, Inc; 2000. Fitting nonlinear mixed-effects models; pp. 337–414. full_text.
  • Gabriel KR, Odoroff CL. Biplots in medical research. Stat Med. 1990;9:469–485. doi: 10.1002/sim.4780090502. [PubMed] [Cross Ref]
  • Lorch SA, Zhang X, Rosenbaum PR, Evan-Shoshan O, Silber JH. Equivalent Lengths of Stay of Pediatric Patients Hospitalized in Rural and Nonrural Hospitals. Pediatrics. 2004;114:400–408. doi: 10.1542/peds.2004-0891. [PubMed] [Cross Ref]
  • Silber JH, Rosenbaum PR, Koziol LF, Sutaria N, Marsh RR, Even-Shoshan O. Conditional Length of Stay. Health Serv Res. 1999;34:349–363. [PMC free article] [PubMed]
  • Hollander M, Wolfe DA. In: Nonparametric statistical methods. 2. Hollander M, Wolfe DA, editor. New York: John Wiley & Sons, Inc; 1999. Life distributions and Survival Analysis; pp. 495–562.
  • Junoy JP. Measuring technical efficiency of output quality in intensive care units. Int J Health Care Qual Assur Inc Leadersh Health Serv. 1997;10:117–124. [PubMed]
  • Rapoport J, Teres D, Zhao Y, Lemeshow S. Length of stay data as a guide to hospital economic performance for ICU patients. Med Care. 2003;41:386–397. doi: 10.1097/00005650-200303000-00007. [PubMed] [Cross Ref]
  • Thomas JW, Guire KE, Horvat GG. Is patient length of stay related to quality of care? Hosp Health Serv Adm. 1997;42:489–507. [PubMed]
  • Rosko MD, Mutter RL. Stochastic Frontier Analysis of Hospital Inefficiency: A Review of Empirical Issues and an Assessment of Robustness. Med Care Res Rev. 2008;65:131–166. doi: 10.1177/1077558707307580. [PubMed] [Cross Ref]
  • Needleman J, Buerhaus P, Mattke S, Stewart M, Zelevinsky K. Nurse-staffing levels and the quality of care in hospitals. N Engl J Med. 2002;346:1715–1722. doi: 10.1056/NEJMsa012247. [PubMed] [Cross Ref]
  • Pronovost PJ, Angus DC, Dorman T, Robinson KA, Dremsizov TT, Young TL. Physician staffing patterns and clinical outcomes in critically ill patients: a systematic review. JAMA. 2002;288:2151–2162. doi: 10.1001/jama.288.17.2151. [PubMed] [Cross Ref]
  • Bastos PG, Knaus WA, Zimmerman JE, Magalhaes A Jr, Sun X, Wagner DP. The importance of technology for achieving superior outcomes from intensive care. Brazil APACHE III Study Group. Intensive Care Med. 1996;22:664–669. doi: 10.1007/BF01709743. [PubMed] [Cross Ref]
  • Thomas JW, Hofer TP. Research evidence on the validity of risk-adjusted mortality rate as a measure of hospital quality of care. Medical Care Research & Review. 1998;55:371–404. [PubMed]
  • Werner RM, Bradlow ET. Relationship Between Medicare's Hospital Compare Performance Measures and Mortality Rates. JAMA. 2006;296:2694–2702. doi: 10.1001/jama.296.22.2694. [PubMed] [Cross Ref]
  • Kahn JM, Bates DW. Improving Sepsis Care: The Road Ahead. JAMA. 2008;299:2322–2323. doi: 10.1001/jama.299.19.2322. [PubMed] [Cross Ref]
  • Stone M. How not to measure the efficiency of public services (and how one might) Journal of the Royal Statistical Society: Series A (Statistics in Society) 2002;165:405–434.
  • Greene W. Fixed and Random Effects in Stochastic Frontier Models. Journal of Productivity Analysis. 2005;23:7–32. doi: 10.1007/s11123-004-8545-1. [Cross Ref]
  • Griffin J, Steel M. Flexible mixture modelling of stochastic frontiers. Journal of Productivity Analysis. 2008;29:33–50. doi: 10.1007/s11123-007-0064-4. [Cross Ref]
  • Williams SC, Schmaltz SP, Morton DJ, Koss RG, Loeb JM. Quality of Care in U.S. Hospitals as Reflected by Standardized Measures, 2002-2004. N Engl J Med. 2005;353:255–264. doi: 10.1056/NEJMsa043778. [PubMed] [Cross Ref]

Articles from BMC Medical Research Methodology are provided here courtesy of BioMed Central