The Amsterdam Maastricht Utrecht Study on thrombo-Embolism (AMUSE-2) was a prospective cohort study in primary care, evaluating a diagnostic strategy consisting of the Wells pulmonary embolism rule (table 1) and a point of care D-dimer test (Clearview Simplify; Inverness Medical, Bedford, UK). More than 300 primary care doctors across three different regions of the Netherlands (Amsterdam, Maastricht, and Utrecht) included patients during and outside of practice hours. One researcher (GJG, PE, or WL) contacted all primary care doctors willing to cooperate with the study and explained the logistics of the study and the use of study forms and provided written instruction on how to use the D-dimer test. The study took place between 1 July 2007 and 31 December 2010.
Table 1 Items of Wells pulmonary embolism rule
Patients eligible for inclusion were consecutive adults (≥18 years) with suspected pulmonary embolism, based on the presence of at least one of the following symptoms: unexplained (sudden) dyspnoea, deterioration of existing dyspnoea, pain on inspiration, or unexplained cough. We excluded patients if they received anticoagulant treatment (vitamin K antagonists or heparin) at presentation, they were pregnant, follow-up was not possible, or they were unwilling or unable to provide written informed consent.
Diagnostic strategy under study
After written informed consent had been obtained, the doctors documented information on the patient’s history and physical examination and applied the Wells pulmonary embolism rule using a standard form. A qualitative D-dimer test was subsequently carried out using 35 μL of capillary or venous blood mixed with two drops of test reagent in a disposable device.7
A pink-purple coloured line indicates a positive test result (D-dimer level >80 ng/mL). The test strip can be read at 10 minutes.
Regardless of the results of the Wells rule and D-dimer test, we asked the doctors to refer all patients to secondary care for reference testing. In addition, to avoid interference with our aim to externally validate the Wells rule combined with D-dimer testing, we explicitly provided no guidance on how to use the score (which score thresholds) to guide subsequent management.8
Hence, doctors in secondary care were asked to carry out the diagnostic procedures at their own discretion based on local hospital guidelines, and independent of the results from primary care.
In secondary care, the diagnostic strategy was based on current guidelines and routine care protocols. In the Netherlands, this is mostly a combination of estimated probability and quantitative laboratory based D-dimer testing, followed by computed tomography if indicated. In line with most diagnostic studies in this area5
we used a composite reference standard of spiral computed tomography, ventilation-perfusion scanning, pulmonary angiography, leg ultrasonography, and clinical probability assessment as done in secondary care (with or without D-dimer testing).
We retrieved medical information about the investigations done to establish or refute a diagnosis of pulmonary embolism, including hospital discharge letters. In addition, we followed up all patients for three months. During the follow-up period we asked the primary care doctors to document the occurrence of any potential (recurrent) venous thromboembolic events, and bleeding complications associated with anticoagulant therapy if given. Finally, an independent adjudication committee evaluated all patients with a diagnosis of pulmonary embolism despite a low Wells score and negative D-dimer test result (see supplementary file for a full description of the reference standard strategy).
The primary outcomes of this study were diagnostic accuracy (sensitivity and specificity), proportion of patients at low risk (efficiency), number of missed patients with pulmonary embolism in low risk category (false negative rate), and the presence of symptomatic venous thromboembolism, based on our composite reference standard, including events during the follow-up period of three months.
All statistical analyses were done using the SPSS software package PASW version 17. We quantified the safety and efficiency of ruling out pulmonary embolism on the basis of a low risk score using our diagnostic strategy. Patients considered at low risk were initially defined by a Wells score of ≤4 and a negative D-dimer test result. We defined the failure rate as the proportion of patients with a missed symptomatic and proved venous thromboembolism event during three months’ follow-up in those patients who were initially classified by the strategy to be at low risk, including a 95% confidence interval (using Fischer’s exact test).
In contrast with therapeutic or intervention studies, formal sample size calculations based on power assumptions for diagnostic (or prognostic) modeling cohort studies do not exist and are seldom considered. However, for single dichotomous tests, such calculations can be done for the expected positive or negative predictive values or their complements (false positive or false negative proportion, respectively).14
To obtain some insight a priori of what number of patients and thus doctors needed to be included, we considered the original continuous Wells rule plus D-dimer test result as one overall single test and dichotomised the result. We focused on the exclusion of pulmonary embolism with a minimum of missed of cases (false negative proportion or failure rate). Based on various previous studies in secondary care (notably the Christopher study6
), we assumed that the point estimate of this failure rate would be around 0.5%. We subsequently used this estimate to calculate the number of patients for our study,14
where we selected a stringent upper limit of the confidence interval of this estimate at 2.0%, even though previous studies had higher upper limits (4%).5
Accordingly, expecting a failure rate for detecting pulmonary embolism of 0.5% and being able to exclude (maximally) 2.0% of patients with pulmonary embolism, and using a type I error of 0.05 (one sided, since any value below 0.5% is desired) and type II error of 0.2, we needed to include about 335 patients with a low risk of pulmonary embolism.
Next we calculated the efficiency of the Wells rule in excluding pulmonary embolism. We defined efficiency as the proportion of patients at low risk for pulmonary embolism among all study patients. Subsequently, we similarly estimated the failure rate and efficiency using a Wells threshold score of <2 in combination with a negative D-dimer test result—that is, patients at very low risk.
In addition to the failure rate and efficiency, we calculated the conventional diagnostic accuracy measures (sensitivity, specificity, and predictive values) for the different thresholds on the Wells rule, in combination with D-dimer testing.
Missing values for the Wells rule items or D-dimer test results were observed in 24 patients (4.0% for missing values on any of the Wells rule items or D-dimer test; range 0.5% for heart rate >100 beats/min to 2.7% for results of the D-dimer test). To minimise the effect of the bias associated with selectively ignoring these 24 patients, we imputed these missing values using multiple imputation techniques. Such techniques are based on the correlation between each variable with missing values and all other variables as estimated from the complete set of participants, using available observed data.16