|Home | About | Journals | Submit | Contact Us | Français|
Accurate and reliable blood glucose (BG) measurements require that different test strip lots of the same BG monitoring system provide comparable measurement results. Only a small number of studies addressing this question have been published.
In this study, four test strip lots for each of five different BG systems [Accu-Chek® Aviva (system A), FreeStyle Lite® (system B), GlucoCheck XL (system C), Pura™/mylife™ Pura (system D), and OneTouch® Verio™ Pro (system E)] were evaluated with procedures according to DIN EN ISO 15197:2003. The BG system measurement results were compared with the manufacturer’s measurement procedure (glucose oxidase or hexokinase method). Relative bias according to Bland and Altman and system accuracy according to ISO 15197 were analyzed. A BG system consists of the BG meter itself and the test strips.
The maximum lot-to-lot difference between any two of the four evaluated test strip lots per BG system was 1.0% for system E, 2.1% for system A, 3.1% for system C, 6.9% for system B, and 13.0% for system D. Only two systems (systems A and B) fulfill the criteria of DIN EN ISO 15197:2003 with each test strip lot.
Considerable lot-to-lot variability between test strip lots of the same BG system was found. These variations add to other sources of inaccuracy with the specific BG system. Manufacturers should regularly and effectively check the accuracy of their BG meters and test strips even between different test strip lots to minimize risk of false treatment decisions.
Self-monitoring of blood glucose (SMBG) is a widely accepted instrument in modern diabetes management that allows for tight blood glucose (BG) control in diabetes patients.1–4 Thus SMBG is recommended for all patients with diabetes, especially for patients who adjust their insulin doses based on BG measurement results.2,5–7
The accuracy of a BG measurement, e.g., as defined by the international standard DIN EN ISO 15197:2003,8 is imperative for the result’s reliability and utility for BG control. Compliance with DIN EN ISO 15197:2003 is one step in the Conformité Européenne (CE)-marking process. The CE mark is a prerequisite for all devices, including BG meters, that are distributed in the European Union market. With the application of the CE mark on a BG meter, the manufacturer declares that the device meets the European Union requirements and that all required conformity evaluation procedures were performed in cooperation with a nationally accredited notified body. This approval process for medical devices in Europe is different to the approval process of drugs by the European Medicines Agency or to the approval process for medical devices by the Food and Drug Administration in the United States.
Measurement results obtained with a given BG meter might vary when different test strip lots are used. This lot-to-lot variability is addressed in the current draft revision of International Organization of Standardization 15197 (ISO/DIS 15197, expected to be published in 2012),9 which requires the evaluation of three different test strip lots per BG system; each individual lot has to fulfill the requirements of this standard.
In order to evaluate the impact of lot-to-lot variability for five different BG systems, this study investigated (i) the bias between four test strip lots of each BG system and (ii) the system accuracy for each test strip lot (so, in total, the system accuracy for 20 test strip lots), using data obtained following the standardized procedures of DIN EN ISO 15197:2003, ensuring comparability of results.
This study was performed between 2010 and 2011 at the Institut für Diabetes-Technologie GmbH in Ulm, Germany, in compliance with the German Medical Devices Act. The Ulm University ethics committee reviewed and approved the study protocol, and the competent authority was notified. Signed informed consent forms were provided by all participants. The study procedures are identical to those used in another study performed at the same study site (see Freckmann and colleagues16 in this issue) and described in detail in DIN EN ISO 15197:2003.8 Deviations from these procedures are described here.
In this study’s context, a BG system consists of one type of BG meter and the test strips labeled for use with this meter. The five evaluated BG systems and the four test strip lots used with each BG system are listed in Table 1. They are all current, CE-marked systems. The first test strip lot for each BG system was also assessed in another study at the same study site, which focused on system accuracy of a wide variety of BG systems (see Freckmann and colleagues16 in this issue). The five systems have been selected in order to compare multiple test strip lots of systems from established and new manufacturers. A criterion was market availability of four lots. Also, the availability of a CE mark and the availability of BG meters and test strips in the required quantities as well as the availability of several lots in a defined time frame for an ISO assessment have been the key prerequisites. For a BG system to be included in the evaluation, it had to be labeled for SMBG usage. In deviation to DIN EN ISO 15197:2003, test strips were taken from at least 8 (instead of 10) different packages or vials, which were changed after approximately 10 subjects.
Subjects (≥18 years old) with diabetes mellitus type 1 and type 2 as well as subjects without diabetes were included. Exclusion and study interruption criteria for subjects are identical to those described by Freckmann and colleagues16 (this issue). For each test strip lot of the BG systems evaluated, blood samples of at least 100 subjects were used. Two individual BG meters were used for each test strip lot according to section 7.3.2 of DIN EN ISO 15197:2003. In case of failure, BG meters were replaced. Measurements were performed on at least 10 days for each test strip lot, and suitable control procedures were performed daily prior to the test procedure. The tests were performed by clinical personnel well trained to the handling of the BG systems, the manufacturer’s device labeling, the safety practices, and the test protocol. The tests were performed in a laboratory setting with controlled room temperature (23 ± 5 °C) and humidity (according to the manufacturers’ specifications).
Reference measurements were performed with the following two methods for all BG systems: glucose oxidase (GOx) [YSI 2300 STAT Plus™ glucose analyzer, YSI Life Sciences, Yellow Springs, OH; measurements were performed at the study site] and hexokinase (HK) [Hitachi 917 (from March 2010 to August 2010)/cobas® 6000 c501 (since August 2010), Roche Diagnostics GmbH, Mannheim, Germany; measurements were performed at a Deutsche Akkreditierungsstelle-accredited calibration laboratory of Roche Diagnostics GmbH, Mannheim, Germany]. Internal and external quality control measurements were performed as described by Freckmann and colleagues16 (this issue).
The measurement results of each test strip lot were compared against the measurement results of the manufacturer’s measurement procedure, i.e., the reference method specified in the device labeling.
The BG meters displayed plasma equivalent BG values in mg/dl or mmol/liter. Reference measurements were performed from whole blood, which was hemolyzed and deproteinized for the HK method. Both reference measurement procedures showed whole blood BG values in mg/dl, from which plasma equivalent values were calculated for comparison with the test strip lot results. Measurement results obtained with the GOx method were converted from whole blood BG values to plasma equivalent BG values as follows:
plasma equivalent BG value (in mg/dl) = whole blood BG value (in mg/dl) / [1 - (0.0024 × hematocrit value [in %])].17
Results from the HK method were converted using a constant conversion factor:
plasma equivalent BG value (in mg/dl) = 1.11 × whole blood BG value (in mg/dl).
DIN EN ISO 15197:2003 specifies the distribution of theblood samples into different BG concentration categories (Table 2). The limits of these categories were slightly modified because they are not clearly defined and differ between the English and the German version of the standard. Additionally, the distribution of blood samples is based on the mean BG values obtained with the reference method, thus deviating from the standard, which states that distribution shall be based on the results determined with the BG system. The current draft revision of ISO 15197 demands distribution based on the reference BG values.
For BG concentrations between 50 and 400 mg/dl, only native, unaltered whole blood samples were used.If numbers of unaltered blood samples with BG concentrations <50 and >400 mg/dl were insufficient, modified blood samples were prepared, either by glycolysis or glucose supplementation. A detailed description is available from Freckmann and colleagues.16 These preparation procedures did not ensure constant oxygen concentrations of the blood samples, which might affect BG systems with GOx-based test strip chemistry. The only BG system with GOx test strip chemistry evaluated in this study is the Pura™/mylife™ Pura. No correction or exclusion of modified blood samples was performed because oxygen interference is not mentioned in the manufacturer’s labeling.
Fresh capillary whole blood samples were collected from at least 100 subjects by puncturing the skin at the fingertips. Hematocrit values of these samples had to be within 30% and 55%. For the determination of the hematocrit, capillary whole blood was collected in heparinized capillaries (double test). After centrifugation, the hematocrit was read on an alignment chart.
Sampling sequence steps were as follows (more detailed steps are described by Freckmann and colleagues16):
Before measurements with each test strip lot and before each sample collection for each reference measurement procedure, residual blood was wiped off the fingertip. Normally, the same drop of blood was used for measurements with the two BG meters of the respective BG system.
The drift between the first and second reference measurement result must not exceed 4 mg/dl at BG concentrations ≤100 mg/dl or 4% at BG concentrations >100 mg/dl.
A total of 8 measurement results were excluded due to deviation from test protocol, and 1 sampling sequence was repeated because of a technical error.
The statistical analysis of each test strip lot of each BG system included 200 results from 100 subjects, which were compared with the reference measurement result.
All statistical analyses were performed at the study site. Calculations were performed in mmol/liter with a unit conversion as follows:
BG value in mmol/liter = 1/18.02 × BG value in mg/dl.
The relative bias (%) of the measurement results of each test strip lot was calculated according to Bland and Altman18 using the formula
where BG is a single BG measurement result obtained with one specific test strip lot, reference is the mean value of the reference measurements before and after the BG measurement with this test strip lot, and n is the number of all BG measurement results obtained with this test strip lot. For calculation of each test strip lot’s relative bias, all included measurement results were analyzed. The relative bias is shown with 95% limits of agreement (≈ ±1.96 × standard deviation).
The accuracy of each of the 20 test strip lots’ (four test strip lots from five BG systems) results was evaluated by comparison with the respective mean value of the reference measurement results (GOx method or HK method) obtained immediately before and after the measurements with the test strip lot. According to DIN EN ISO 15197:2003, at BG concentrations <75 mg/dl, the relative number of the test strip lot’s results within ±15, ±10, and ±5 mg/dl and, at BG concentrations ≥75 mg/dl, the relative number of the test strip lot’s results within ±20%, ±15%, ±10% and ±5% of the reference were calculated. In order to assess the overall accuracy of a specific test strip lot, the number of this test strip lot’s results within ±15 mg/dl at BG concentrations <75 mg/dl was added to the number of results within ±20% at BG concentrations ≥75 mg/dl and divided by the number of all measurement results obtained with this test strip lot. In addition, the accuracy of the 20 test strip lots’ results was evaluated using the limits of the current draft revision of ISO 15197. At BG concentrations <100 mg/dl, the relative number of system results within ±15, ±10, and ±5 mg/dl and, at BG concentrations ≥100 mg/dl, the relative number of system results within ±15%, ±10%, and ±5% of the reference measurement were calculated. Similar to the evaluation according to DIN EN ISO 15197:2003, the number of this test strip lot’s results within ±15 mg/dl at BG concentrations <100 mg/dl was added to the number of results within ±15% at BG concentrations ≥100 mg/dl and divided by the number of all measurement results obtained with this test strip lot.
To illustrate the accuracy of each test strip lot according to DIN EN ISO 15197:2003, the agreement between each measurement result and the mean reference result of the manufacturer’s measurement procedure was plotted in a difference plot for each test strip lot separately.
The difference plot shows the deviation of single measure-ment results of the specific test strip lot from the true result (reference value). It shows both random and systematic deviations, which reflect the total measuring error of the test strip lot.
Table 3 shows the relative bias and limits of agreement for each test strip lot separately and the average relative bias of the four test strip lots with each BG system.
The relative bias of the single test strip lots ranged from -17.3% to +9.5% (Figure 1). The maximum difference in relative bias between any two of the four test strip lots of a BG system was 1.0% for system E, 2.1% for system A, 3.1% for system C, 6.9% for system B, and 13.0% for system D. Averaged over all four test strip lots, the smallest relative bias was achieved by system A, with a relative bias of -0.4%, followed by system C (+2.8%), system B (-4.0%), system E (+8.8%), and system D (-9.0%).
The percentage of BG measurement results within different deviation ranges according to DIN EN ISO 15197:2003 and according to the current draft revision of ISO 15197 are shown in Table 4. In addition to these percentages, the overall accuracy assessments, including all BG measurement results, are displayed.
Figures 2–4 show the agreement between the BG measurements results and the mean reference result (manufacturer’s measurement procedure) for each test strip lot of each BG system in difference plots. Considering single test strip lots, only system A and system B fulfilled the requirements stated in DIN EN ISO 15197:2003 with all four test strip lots: 100.0% of system A measurements were within the 2003 limits and 99.0% to 100.0% within the draft limits, and system B had 99.5% to 100.0% of the measurements within the 2003 limits and 89.5% to 100.0% within the draft limits. For system C, two of four test strip lots fulfilled the 2003 requirements (93.5% to 95.5%) and 88.5% to 91.5% of the measurements were within the draft limits. System D test strip lots showed 87.0% to 100.0% of measurements within the accuracy limits of the 2003 standard, thus three of four test strip lots fulfilled the requirements; 52.5% to 100.0% of the measurements were within the draft limits. For system E, three of four test strip lots fulfilled the 2003 requirements (94.0% to 96.5%), with 83.5% to 93.0% of measurements within the limits of the current draft revision of ISO 15197.
An important aspect of the measurement accuracy is the lot-to-lot variability between multiple test strip lots used with one BG system. Therefore, data generated in an accuracy analysis according to EN ISO 15197 and thus including measurement results ranging from <50 to >400 mg/dl was used in order to investigate lot-to-lot variability of five BG systems with four test strip lots each. The requirements of DIN EN ISO 15197:2003 were fulfilled by all evaluated test strip lots separately for only two of the tested BG systems. Only one system had at least 95% of the measurements within the stricter accuracy limits of the current draft revision of ISO 15197 with each test strip lot.
In this study, for the evaluation of the measurement accuracy, besides analysis of the system accuracy according to ISO 15197, the relative bias according to Bland and Altman was taken into account. Considerably varying bias (>5%) between the evaluated test strip lots was found in two out of five BG systems.
This result is comparable to results from other studies that investigated between-lot variations and also found marked lot-to-lot variability in bias for some BG systems.13–15 Constant bias over multiple test strip lots may not necessarily be an issue, as patients may, over time, accommodate to a stable given bias, whereas strongly varying bias might have a negative effect on BG control, as accommodation is not possible.
This investigation considered a BG concentration interval from <50 to >400 mg/dl, without separating into the different clinically relevant concentration intervals, i.e., hypoglycemic, euglycemic, and hyperglycemic ranges as well as intervals specific to certain types of diabetes therapy. Therefore, the clinical impact of the direction of a measurement bias in different intervals, e.g., negative bias at hypoglycemic concentrations versus positive bias at hypoglycemic concentrations, was not examined.
The lot-to-lot variability adds to other sources of inaccuracy of a specific BG system. These sources include differences in the manufacturer’s measurement procedure,19 manufacturer-specific plasma conversion factors, but also handling errors by the users.20–23 As another possible source of inaccuracy, we observed variability between different test strip vials inside one lot.
Several studies showed that accuracy of BG measure-ments vary largely between different BG systems.13,24–26 It is worth mentioning that the extent of some lot-to-lot variability observed in this study was comparable to the variations observed between different BG systems. These variations hamper the achievement of strict glycemic goals, e.g., as required in patients with gestational diabetes or type 1 diabetes, because neither the patients nor the health care professionals are aware of the magnitude or even the existence of these variations.
In summary, this study showed that there are considerable differences in the measurement quality of different test strip lots of the same BG system. These differences probably have an impact on the reliability of the BG measurements and, subsequently, on therapeutic decisions. The clinical impact of lot-to-lot differences of test strips has not been studied thoroughly so far. Before being introduced to the market, BG systems have to be tested in a premarket approval, i.e., if they fulfill certain requirements. However, once being commercially available, there are no mandatory tests, e.g., for accuracy evaluation of test strip lots from routine production.
As inaccurate BG systems bear the risk of erroneous therapeutic decisions by the patient and/or health care professional and subsequent possible severe health injury, lot-to-lot variability of test strips should be evaluated by manufacturers or by an independent specialized institution.
We thank Prof. Dr. Theodor Koschinsky for his valuable input in data evaluation and discussion. We also thank Prof. Dr. Lutz Heinemann for contributing his scientific expertise to the discussion.
This study was funded by a grant from Roche Diagnostics GmbH, Mannheim, Germany.
Guido Freckmann received speaker’s honoraria and refunding of traveling expenses for congress participation from Roche Diagnostics GmbH, Mannheim, Germany, the sponsor of this study, and from Berlin-Chemie AG, Berlin, Germany.