|Home | About | Journals | Submit | Contact Us | Français|
Promoting access to medicines requires concurrent efforts to strengthen quality assurance for sustained impact. Although problems of substandard and falsified medicines have been documented in low- and middle-income countries, reliable information on quality is rarely available.
The aim of this study was to validate an alternative post-market surveillance model to complement existing models.
The study used standardized patients or mystery clients (people recruited from the local community and trained to pose as real patients) to collect medicine samples after presenting a pre-specified condition. The patients presented four standardized conditions to 42 blinded facilities in Nairobi, Kenya, resulting in 166 patient–clinician interactions and dispensing of 300 medicines at facilities or nearby retail pharmacies. The medicine samples obtained thus resemble those that would be given to real patients.
Sixty samples were selected from the 300, and sent for analysis at the Kenya National Quality Control Laboratory. Of these, ten (17%) did not comply with monograph specifications (three ibuprofen, two cetirizine, two amoxicillin/clavulanic acid combinations, and one each for prednisone, salbutamol and zinc). Five of the ten samples that failed had been inappropriately prescribed to patients who had presented symptoms of unstable angina. There was no association between medicine quality and ownership, size or location of the facilities.
The study shows that the standardized patient model can provide insights into multiple dimensions of care, thus helping to link primary care encounters with medicine quality. Furthermore, it makes it possible to obtain medicines from blinded sellers, thus minimizing the risk of obtaining biased samples.
There is widespread recognition that promoting access to medicines is not sufficient on its own, and that mechanisms must be put in place to guarantee compliance to acceptable quality standards. However, such mechanisms can only exist in markets where information on the quality of medicines exists, with proper mechanisms for enforcing compliance to standards [1, 2]. This information is not as widely available as it should be in low-income countries, where complaints of substandard and falsified medicines are common, and where regulatory enforcement may be weak [3, 4]. More recent evidence suggests that the problem may be worsening in middle- and higher-income countries as well . Worse still, there has been little effort to describe the causes and broader impact of poor quality medicines .
The risk of poor quality medicines in the market is higher in lower-income countries [7, 8]. A pilot study by Bate et al., for instance, found that whereas substandard medicines exist across the globe, unscrupulous manufacturers have a particular preference for the African market, where regulatory efforts are absent or minimal . Low-quality medicines have been linked to adverse public health outcomes; for instance, a 2013 analysis linked 122,350 child deaths in 39 African countries to consumption of poor-quality antimalarials . These statistics underscore the value of understanding and strengthening the quality of medicines in the region.
Past studies in Kenya have established the presence of poor quality medicines, although the severity has varied across studies, regions, and disease components [4, 11]. Post-market surveillance studies have mainly focused on malaria, HIV, and TB.
These studies have relied on medicine samples collected openly from pharmacies for purposes of analysis. While this has proved to be a powerful tool to determine the quality of specific batches of medicines, it is less informative about the quality of medicines that patients receive for given symptoms or diagnoses. The divergence comes from two sources. First, patients often receive medicines in the clinic itself as opposed to pharmacies, a practice that is especially common in public facilities. And second, it is possible for a pharmacy provider to offer samples for analysis that they deem to be of superior quality, and conceal those acquired through dubious means, or those that are beyond their expiry date.
The World Bank Group has supported various reforms aimed at strengthening patient safety and quality of healthcare in Kenya. As part of this mandate, a validation study was conducted using standardized patients (SPs) or mystery clients to describe the quality of medicines at Kenyan health facilities.
SPs are people recruited from the local community and extensively trained to present the same set of symptoms to multiple providers. They are increasingly used to assess the quality of healthcare and are widely regarded as the gold standard in such assessments. On the other hand, mystery clients or shoppers have been used to understand pharmacy/drug seller dispensing practices, usually in a retail setting [12–15]. Although conceptually both methods use blinded patients to assess provider practices, mystery shoppers typically ask for a specific medicine/treatment, while SPs present a symptom and are given prescriptions and dispensed drugs in accordance with the diagnosis and treatment choice selected by the healthcare provider. Blending the two approaches, as we attempt in this study, can provide valuable information as SPs are treated just like any other patient and the quality of the drugs they receive can be closely tied to the illness they present with.
This study was part of a pilot for the larger Kenya Patient Safety Impact Evaluation (or KePSIE), which sought to evaluate the effectiveness of different ways of enforcing regulations on patient safety. This component sought to identity medicines that fail quality tests, describe the proportion of SPs who received medicines that failed the tests, and describe patterns of association between the presence of medicines of questionable quality and selected facility and patient characteristics.
As this is a validation study, our primary objective was to assess whether the methodology could be used successfully in larger samples, and whether doing so would yield new insights on the quality of medicines in the market. The study specifically sought to contribute to the knowledge on how drug quality studies can move beyond assessing the pharmaceutical quality of the product, to answering the broader public health questions such as the proportion of patients with certain disease conditions that get the right products prescribed, and of those, how many go on to receive a product of acceptable quality. Table 1 provides the working definition of key terms used in the article.
The study employed a cross-sectional survey design using SPs to collect samples from blinded facilities and pharmacies. The collection of samples for the analysis was anchored on the broader pilot of the KePSIE study, which used SPs to investigate processes of care, including diagnosis and management of pre-set symptoms specific to four diseases, namely, diarrhea, asthma, tuberculosis, and unstable angina. The four were selected following wide consultations with clinicians in Kenya. A more detailed description of the KePSIE methodology is provided elsewhere . This component of the study focuses specifically on the medicines given to SPs at the end of their respective interactions with healthcare providers. The samples analyzed were medicines dispensed to SPs posing as actual patients.
SPs either received the drugs in the facilities they visited (most often in public facilities) or received prescriptions from providers in sampled facilities, and then purchased the medicines from the nearest pharmacy, all within Nairobi. Health facilities were approached in a convenience sample designed to include low-, middle-, and high-income neighborhoods in various parts of Nairobi. Care was taken to ensure a fair representation of relatively poorer and wealthier neighborhoods in the selection of facilities. The SP visits were conducted in six sub-counties (divisions) in Nairobi: Dagoretti, Kamkunji, Kasarani, Langata, Starehe, and Westlands.
Of 49 health facilities approached, 46 agreed to participate. SP interactions were completed in 42 facilities, with four randomly held as reserves in case a sampled facility was closed or otherwise inaccessible. Of 168 potential interactions in these 42 facilities, we completed 166 (98.5%). Of those facilities, 14 were public and 28 were privately owned and operated. Among the private sector, five were operated by faith-based organizations (FBOs), four by social franchise operations (SFOs) and one was a community clinic. In analysis, facilities are classified only as ‘public’ or ‘private’ (which includes the not-for-profit facilities) due to the small sample size. A total of 300 medicines were prescribed in the 166 interactions. The SPs presented the prescriptions to the facility pharmacy or a nearby retail pharmacy. Of the 300 medicines, 19 were dispensed in unlabeled packaging (dosage written, but name of medicine missing). This typically happens in cases where medicines are purchased in bulk containers; for instance, loose tablets bought in tins of 1000s.
Table 2 shows the medicines given for the four conditions presented by the SPs, and the frequencies with which each medicine was given for each condition. The sources of the medicines varied. Of the 91 medicines offered in 37 public facility interactions, 61 (67%) were obtained from the facility pharmacy and 30 (33%) were purchased from an outside chemist. Of the 209 medications offered in 82 private facility interactions, 179 (86%) were obtained from the facility pharmacy and 30 (14%) were purchased from an outside chemist. SPs presenting with the same set of symptoms could receive different treatments depending on the clinician’s diagnosis and treatment preference. These variations underscore the importance of doing a study of this nature, where questions answered are not just about the pharmaceutical quality of the product, but the proportion of patients who received the correct treatment, and received a product of the right quality.
A chain of custody protocol was developed to guide the handling and transportation of medicine samples from the dispensing point to the analysis laboratory. The SPs were instructed to leave medicines in their original packaging and not handle medicines directly or tamper with the labeling. The SPs were debriefed by supervisors following each interaction, and facility information entered into a questionnaire. Medicines for each interaction were put in a sealable container bearing a unique facility and patient number, before being given to the study managers, who submitted them to the Kenya National Quality Control Laboratory (NQCL). A formal contract was signed between the NQCL and the World Bank for analysis of selected samples, and the remainder of the medicines were destroyed according to the NQCL protocols.
One major difference between this validation study and previous analysis is that drug quality testing typically entails batch sampling, which refers to selection of samples based on batch characteristics. A ‘sample’ will typically refer to an item of a given dosage form and strength, collected from the same packaging (same batch number). Even where medicines bear the same active pharmaceutical ingredients (APIs), dosage form, strength, manufacturer, and batch number, they would still be considered as different samples if they were collected from different sites. However, the SP study design precludes the application of this approach. In this case, samples were collected by the SPs, who could therefore not insist on getting medicines from the same batch, or even ask for the original packaging with batch details. For purposes of this study, and recognizing the peculiarity of using SPs, we defined a sample as “a molecule of a specific active pharmaceutical ingredient (API), dosage form and strength, given to a SP at a facility, regardless of batch details”.
Analysis was limited to describing the pharmaceutical quality, rather than making a statement about the performance of different brands. While this varies from the traditional analysis, the approach addressed the overall objective of describing the proportion of SPs who received medicines of questionable quality.
Actual sampling of dispensed medicines sought to balance scientific rigor with budgetary constraints. Three sets of factors guided the sampling: the need to include representative medicines from major pharmacological classes; the need to apply some kind of proportionate sampling; and the need to keep the analysis costs at below US$18,000. Sampling was done in two stages. First, medicines were put into 11 pharmacological classes (with all unlabeled medicines being grouped together into a twelfth class). Secondly, medicines were sampled in each of the 11 classes, with the number selected varying depending on the overall number of prescriptions carrying the medicine. A total of 60 unique medicine samples were selected from 11 pharmacological classes. From each class, a simple algorithm was used to decide the number of samples to be analyzed for each medicine, based on the popularity of the medicine (Table 3).
Figure 1 gives an overview of the selection process.
Two pharmacological classes (cardiovascular and central nervous system medicines) were excluded from analysis because they each only had one medicine prescribed. Unlabeled medicines were also excluded from the analysis. Table 4 shows the final sample included in the analysis, including the tests performed, and costs per medicine. The tests are defined briefly in Table 5. These included a range of qualitative and assay tests defined by various monographs, including the International Pharmacopoeia, the British Pharmacopoeia (BP) and the US Pharmacopoeia (USP). The decisions on which tests to perform for the different medicines were reached following discussions with the analysis experts at the NQCL. The decisions were guided by three key factors: first, the fact that the quantities available for analysis were relatively small compared with the traditional batch sampling approach, where sufficient samples are collected with the knowledge of the seller, not by standardized clients; secondly, the relatively low budget available for this study component; and thirdly, the availability of reference standards for the different tests (the NQCL observed that not all tests can be done for all samples, based on local availability of reference standards). The lab analysts were blinded to the identity of facilities from which samples were obtained.
Each analysis was reported as ‘pass’ or ‘out of specification (OOS, or failed)’. Typically, analysts would repeat the test for failed samples to confirm. However, this was not possible here, given the way the samples were collected. The NQCL experts nonetheless revealed that from past experiences, over ninety percent of medicines that fail the first test go on to fail subsequent tests.
The proportion of samples that passed the tests were reported. Pearson’s Chi-square tests were also done to check for any association between samples passing the test and three key facility characteristics: facility size, facility ownership, and facility location.
The study received ethics clearance by the African Medical and Research Foundation’s Ethics and Scientific Review Committee (AMREF-ESRC, approval P94/2013).
A total of 60 samples were analyzed (Table 6). Of these, ten did not comply with monograph specifications (17%), with four samples (all salbutamol samples) complying, but showing an unidentified peak in the high-performance liquid chromatography (HPLC) output. In the absence of more detailed tests, these were not characterized as ‘failed’ samples. The peaks could have been anything, ranging from impurities to excipients such as carmoisine or erythrosine (used for color), which are not typically included in monographs.
Of the ten samples that did not comply, three were ibuprofen. Two samples each of cetirizine and the amoxicillin/clavulanic acid combination failed, with prednisone, salbutamol, and zinc tablets having one sample failing each. Five of the 10 samples that did not comply were inappropriately prescribed to SPs presenting symptoms of unstable angina (three ibuprofen, one cetirizine, and one amoxicillin/clavulanic acid, see Table 6). Similarly, four medicines prescribed for symptoms of asthma failed, yet only one prescription was correct, based on the treatment protocols (salbutamol).
Pearson’s Chi-square tests were done to describe associations between compendial compliance and facility characteristics. Apart from ownership and location, we also classified facilities by type/size where dispensaries were the smallest type of health facility, usually manned by two or three providers and offering basic curative and preventive services only, while health centers were larger, offering a slightly broader set of services, including basic laboratory testing. The Chi-square test reported p values that had no statistical significance at the 5% level (p values considerably greater than 0.05 in all cases, Table 7), suggesting that any observed differences in compendial compliance across facilities of different ownership, type/size, and location were likely to have been a result of chance.
Recent years have seen a shift in emphasis, from promoting access to commodities, to promoting access to high-quality and effective healthcare services and commodities. In Kenya, issues of quality have become increasingly salient in the policy agenda following devolution of health services delivery to counties. Under the new Constitutional dispensation, the national government retains the stewardship role, which includes coordinating policy development, defining standards, and enforcing regulation, all aimed at ensuring that every Kenyan has access to high-quality services and commodities .
While issues of medicine quality are gaining prominence, there is a general dearth of knowledge on the quality of medicines, and especially, how the problems with drug quality impact broadly on health . The problem of inadequate information is particularly important in Kenya, where over 10,000 registered medicines are in circulation. The Pharmacy and Poisons Board’s pharmacovigilance mechanisms are relatively new, and are yet to achieve a meaningful impact, particularly for commodities that are not funded by the Global Fund and other development agencies. Independent assessments such as this add to the existing knowledge stock.
Standard pharmacovigilance entails collecting samples directly from sellers identified through convenience sampling, random sampling, sentinel-site sampling, or lot quality assurance sampling . Some of these approaches, however, may carry inherent biases, as sampled providers may conceal expired batches and those they deem to be of inferior quality. The behavior of medicine sellers concealing certain medicines deemed inappropriate has been documented previously in a study in Tanzania . In addition, using the typical pharmacy-based sampling would have missed the majority of clients who received their medicines directly from the healthcare facilities. Approximately 67 and 86% of medicines given to SPs came directly from public and private health facilities, respectively.
Overall, selecting facilities using random sampling is not directly compatible with the model used here, as it would require that (i) the researcher predict the diagnoses to be made and treatments to be prescribed to the patients, (ii) that the researcher know whether the prescribed medicine is available at the facility or whether patients would have to go to a nearby pharmacy, and (iii) that the patient only go to the sampled pharmacy, regardless of whether or not it was the nearest or most preferred pharmacy in the area. This would have made it impossible to meet the study’s primary purpose of assessing the entire continuum of care by following a typical patient’s journey from diagnosis to treatment, and checking the pharmaceutical quality of medicines given.
Using SPs is therefore a novel approach, as it minimizes the bias and assesses medicines ‘as they are given’ to real patients. In addition, it allows the analysis of the quality of the medicines to be linked to the quality and appropriateness of treatment given to patients with various ailments. The model complements the traditional sampling-based models, as it links the pharmaceutical quality of medicines to the broader impact on health outcomes. Globally, research and innovation are increasingly targeted at availing novel and cost-effective detection technologies that can allow field testing of medicines in low-resource settings, as part of pharmacovigilance efforts . However, these new technologies will still require innovative ways of deployment for maximum impact.
In this study, the SPs were able to successfully obtain both the medicines that were dispensed in the facility as well as those that were prescribed but not dispensed at the respective facilities. For the latter, they purchased the medicines from the nearest pharmacy to the sampled facility. Further, we were able to maintain a sterile chain of custody and successfully sample key drug samples from those obtained by the SPs. Finally, the samples were successfully analyzed in all 60 cases.
This analysis found that over four-fifths of all medicines complied with compendial standards, with ten products failing to meet the specifications (three ibuprofen, two amoxicillin/clavulanic acid, two cetirizine, and one each for salbutamol, zinc, and prednisone). While a single test is not considered conclusive, the findings raise important concerns over the quality of these medicines. It is also telling that some of the samples that did not meet the compendial specifications had been given inappropriately. In our analysis, for instance, we found that five SPs presenting symptoms of unstable angina were inappropriately treated, and still went on to receive medicines that did not meet compendial specifications. Similarly, three SPs with asthma symptoms got the wrong diagnosis, had the wrong treatment prescribed, and received medicines that failed the compendial tests. These findings underscore the value of understanding the quality management challenges in full, rather than examining aspects of poor care at specific points only. The findings also point at a worrying problem of low-quality medicines in the Kenyan market.
Different studies have reported variations in failure rates across medicines in Kenya. Three rounds of post-market surveillance of antimalarials, for instance, revealed failure rates of between 3.0 and 8.0% (8.0, 3.0, and 5.5% of samples failing for surveillance rounds one, two, and three, respectively) [19–21]. All three surveillance rounds, conducted between 2011 and 2013, had samples averaging 500 per round. A separate WHO-supported surveillance study reported a failure of 4% for antimalarials . Higher failure rates were reported in earlier studies. Bate et al., for instance, reported a failure rate of 38% across six types of antimalarials obtained from pharmacies in the Kenyan market in 2007 .
Studies conducted in other countries have shown varied results. Nayyar et al. reviewed seven studies on medicine quality, and found that failure rates varied from 9 to 41% . In Cambodia, 31% of 291 artemisinin derivatives failed quality tests, while in Afghanistan, one quarter of antimalarial medicines collected from 60 randomly selected facilities failed the disintegration test [23, 24].
In this study, no associations were reported between the medicine quality and health facility characteristics, although this is likely due to the small sample size. Different studies have looked at patterns of association between medicine quality and facility characteristics, with varying results. The 2013 antimalarial post-market surveillance in Kenya, for instance, reported failure rates of 2 and 8% for public and private sector-obtained samples, respectively, concluding that private sector providers were more likely to have substandard products in stock . Elsewhere, a WHO study found higher failure rates from private sector samples in Uganda, but did not find any associations in Madagascar . Similarly, no association was found between medicine quality and facility location in Cambodia .
The registration status of a health facility and prior drug approval by a stringent regulatory agency have also been associated with the quality of medicines. A past review, for instance, found medicines from unlicensed facilities to have a higher likelihood of failing the quality tests . Another study found that medicines that had approval from a stringent regulatory agency had lower failure rates than those that had not been subjected to similar approval processes .
This analysis gave a snapshot of the quality of medicines dispensed around Nairobi. The use of SPs helped to minimize bias and allow assessment of the entire process of care, from clinical diagnosis to the quality of medicines given.
The pilot shows that the alternative model is viable, but requires modifications based on certain limitations. The main limitation of the study was insufficient samples for re-analyses. Medicines filled as prescriptions are rarely sufficient for compendial analysis and re-analysis when required. Although the bulk of medicines that fail the initial compendial tests go on to fail subsequent confirmatory tests, one cannot discount the possibility of false negatives. Ideally, repeat testing is recommended for samples that fail. Achieving sufficient sample quantities is difficult when using SPs; one would have to have prior information on prescribing frequencies and medicine choices for the different disease scenarios, and estimated failure rates for the different medicines prescribed.
Another limitation is lack of geographic representativeness. All dispensing facilities were located within Nairobi, meaning that the findings are unlikely to be representative. Nairobi hosts nearly all pharmaceutical manufacturers and distributors. Rural facilities have a higher risk of selling degraded products (long transport times) and/or falsified products (relatively porous borders, particularly in the Northern parts). On the other hand, facilities in more urban locations face much higher competition, which may result in their resorting to inappropriate behavior. Regardless of the directionality of the effect, one would expect variations in quality for medicines collected further away from Nairobi.
Finally, analysis checking for association between poverty levels in the neighborhood and compendial compliance were not done, as poverty data were not available beyond the sub-county (location) level. It is fairly common to find slums and relatively wealthier suburbs located within one sub-county in Nairobi.
The study shows that in addition to providing useful data for post-market surveillance, the standardized patient method can provide insights into other dimensions of care, thus helping to link primary care encounters with medicine quality. Furthermore, SPs make it possible to obtain medicines from blinded sellers, thus minimizing the risk of biased samples if providers know that medicines are being used for quality assurance testing. However, more effort should go towards defining more objective sampling methods that work for studies that use SPs. More thinking should also go towards understanding how medicine samples that are large enough for reanalysis can be collected without revealing the true identities of SPs.
We thank the following for support at various stages: Mr. Stephen Mawa, Dr. Edward Abwao, Dr. Serah Muteru, Dr. Kefa Bota, Dr. Hezekiah Chepkwony, Dr. Redemptar Kimeu, Prof. Ruth Nduati, and Drs. Fatmah Abdallah, Sarah Chuchu, Rahab Maina, Izaq Odongo, Rachel Nyamai, Charles Kandie, John Kabanya, Pamela Kuya, Purity Kimuru, Leah Adero, Salome Omondi and Ipsos Synovate staff. The findings, interpretations, and conclusions expressed in this article are those of the authors and do not necessarily reflect the views of the World Bank, its executive directors, or the governments they represent.
This work is part of the Kenya Patient Safety Impact Evaluation Project, a joint project of the Kenyan Ministry of Health and the World Bank Group. The study was funded by the Health in Africa Initiative (HIA), the Development Economics Impact Evaluation (DIME) unit, and the Knowledge for Change Program at The World Bank Group.
Francis Wafula declares no conflict of interest. Amy Dolinger declares no conflict of interest. Benjamin Daniels declares no conflict of interest. Njeri Mwaura declares no conflict of interest. Guadalupe Bedoya declares no conflict of interest. Khama Rogo declares no conflict of interest. Ana Goicoechea declares no conflict of interest. Jishnu Das declares no conflict of interest. Bernard Olayo declares no conflict of interest
The study received written approval from the Ministry of Health and the county health management for the respective counties where SPs visited facilities. Ethics clearance was granted by the African Medical and Research Foundation’s Ethics and Scientific Review Committee (AMREF-ESRC, approval P94/2013). The study was funded by Development Economics Vice Presidency and the Health in Africa Initiative (HIA), both at The World Bank Group.