|Home | About | Journals | Submit | Contact Us | Français|
To validate trained community health workers' (CHWs') recognition of signs and symptoms of newborn illnesses and classification of illnesses using a clinical algorithm during routine home visits in rural Bangladesh.
Between August 2005 and May 2006, 288 newborns were assessed independently by a CHW and a study physician. Based on a 20-sign algorithm, sick neonates were classified as having very severe disease (VSD), possible very severe disease (PVSD) or no disease. Physician's assessment was considered as the gold standard.
CHWs correctly classified VSD in newborns with a sensitivity of 91%, specificity of 95%, and kappa value of 0.85 (p<0.001) indicating almost perfect agreement with physicians' classification of VSD. CHWs' recognition showed a sensitivity of more than 60% and a specificity of 97–100% for almost all signs and symptoms.
CHWs with minimal training can use a diagnostic algorithm to identify severely ill newborns with high validity.
Every year, an estimated 4 million newborns die globally within the first month of life; 99% of those deaths occur in the developing world (Lawn et al. 2005; World Health Organization 2006). Serious infections, including sepsis and pneumonia, account for up to 50% of neonatal deaths in high mortality settings (Lawn et al. 2005; Bryce et al. 2005; Black et al. 2003). Many of these deaths could be prevented by improving early recognition of newborn illnesses and access to appropriate and timely treatment (Lawn et al. 2005; Darmstadt et al. 2005; Bhutta et al. 2005). In response to the need for improved illness recognition in the postnatal period, the World Health Organization expanded its Integrated Management of Childhood Illness (IMCI) strategy to include diagnosis and management of severe illnesses in infants under 2 months of age (World Health Organization and UNICEF 2005). Both the original IMCI guidelines and newer young infant guidelines were designed and tested at first-level health facilities by professional health workers (Kalter et al. 1997a; Kalter et al. 1997b; Kolstad et al. 1997; Perkins et al. 1997; Simoes et al. 1997; Weber et al. 1997a; Weber et al. 1997b; 2008; Gupta et al. 2000; Narang et al. 2007). In limited resource settings where most births occur in the home attended by untrained providers and care-seeking is low, minimally trained community health workers (CHWs) can play a vital role in reducing neonatal mortality by improving illness recognition and access to medical treatment (Bang et al. 1999; Kumar et al. 2008; Bhutta et al. 2005; Haines et al. 2007; Baqui et al. 2008b).
Limited evidence from developing countries suggests CHWs working outside formal health facilities can implement IMCI-type algorithms to identify serious illnesses in children (Kallander et al. 2006; Simoes and McGrath 1992; Winch et al. 2005; Zeitz et al. 1993; Kelly et al. 2001; Kahigwa et al. 2002; Hadi 2001). However, few studies have validated the ability of CHWs to recognize clinical signs and classify diseases in newborns in the first month of life in the home or at the community level (Darmstadt et al. 2009; Bang et al. 2005; Mullany et al. 2006). We report the findings of a study conducted to compare the performance of CHWs with six weeks of training in assessing neonates using an IMCI-type algorithm, with that of physicians using the same clinical algorithm.
The CHW validation study was nested within a community-based cluster randomized controlled trial in rural Bangladesh with 3 arms: Home Care, Community Care and comparison (Baqui et al. 2008b). The trial was implemented in a population of about 480 000 in three sub-districts of Sylhet district. Sylhet has the highest neonatal mortality rate of Bangladesh's six divisions and at baseline in 2002, 91% of births occurred at home and only 22% of newborns received a checkup within the first 30 days of life (National Institute of Population Research and Training (NIPORT) et al. 2001; Baqui et al. 2008a). In the Home Care arm, a female CHW provided home-based preventive and curative maternal-neonatal health care to a catchment population of about 4,000. In the Community Care arm, community mobilizers provided the same information on maternal and newborn care during group education sessions rather than home visits. We previously reported a 34% neonatal mortality reduction in the Home Care arm, from 46.9 deaths per 1000 at baseline to 29.2 during the last six months of the 30 month intervention (Baqui et al. 2008b).
CHWs in the Home Care arm made two antenatal home visits to promote birth and newborn care preparedness and three postnatal home visits to check the health of the newborn on day one (day of birth), day three, and day seven of life using a standardized newborn assessment form. CHWs' used a 20-sign clinical algorithm adapted from Bangladesh's IMCI algorithm to classify sick neonates with very severe disease (VSD) or possible very severe disease (PVSD) (Figure 1). Neonates were classified with VSD if one or more of the following eight signs or symptoms were present: (1) observed convulsions, (2) unconsciousness, (3) fast breathing (respiratory rate of ≥ 60 breaths per minute), (4) severe chest in-drawing, (5) fever (temperature ≥ 38.3°C), (6) hypothermia (temperature ≤ 35.3°C), (7) many or severe skin pustules or blisters on single large area, or pus or redness with swelling, or (8) umbilical redness extending to skin. Neonates were classified with PVSD if one or more of the following twelve signs or symptoms were present: (1) history of convulsion, (2) bulging fontanelle, (3) vomiting after every feed, (4) mild fever (temperature 37.8–38.4°C), (5) mild hypothermia (temperature 35.3–36.4°C), (6) weak, abnormal or absent cry, (7) lethargic, less than normal movement, (8) not able to feed, suck, or attach to breast, (9) umbilicus discharging pus, (10) umbilical redness not extending to skin, (11) some skin pustules, or (12) jaundiced palm and sole after one day of life. Newborns with VSD or PVSD were referred for free treatment to government sub-district hospitals. CHWs offered home antibiotic treatment to VSD cases and PVSD cases with more than one sign if the family was unable to take the sick newborn to the hospital and made a follow-up visit within the next 24 hours to monitor the infant and reinforce referral. We previously reported that CHW home treatment of severe illness in neonates was as effective as treatment by medically qualified providers.(Baqui et al. 2009) Adjusted hazard ratio for death during the neonatal period was 0.22 (0.07–0.71) for CHW treatment and 0.60 (0.37–0.99) for treatment by qualified providers, compared to newborns who received no treatment or treatment by untrained providers.
Women with at least a tenth grade education were recruited as CHWs via advertisements in local newspapers, community gathering places, and educational institutions. A preliminary group of 41 CHWs was selected to undergo training after a formal interview and final placements were made based on successful completion of testing at the end of training. Mean age of CHWs at time of recruitment was 23 (standard deviation ±4 years, range 18–46) and over 60% of CHWs were single, divorced, or separated. CHWs received six weeks of training, including skills development for behaviour change communication, clinical assessment of neonates, treatment of newborns with injectable antibiotics, and record-keeping necessary for the trial. The training also included hands-on clinical training under supervision in a tertiary care hospital and in households. A three-day refresher training was conducted midway through the 30-month intervention period. CHWs received ongoing training and support from five female field services supervisors (one for every eight CHWs). The supervisors accompanied CHWs during home visits to support their work and evaluate performance using a structured checklist at least two days a month. Supervisors provided feedback to CHWs at fortnightly meetings and discussed job responsibilities, difficulties encountered in the field, and solutions to those problems. About half of the CHWs originally recruited for the trial worked for the entire 30 months of the intervention, and replacement CHWs were recruited and trained following a similar process as needed. CHW attrition and replacement were primarily due to CHWs' leaving to get married or to take another job; additional analysis and discussion of attrition will be published separately.
The CHW validation study was conducted in the Home Care arm of the trial from August 2005–June 2006. Sample size was calculated to provide estimates of sensitivity and specificity of CHWs' assessment assuming project physicians' assessment as the gold standard. Physicians had MBBS degree and received training to standardize their use of the algorithm for newborn assessment. To estimate assumed sensitivity and specificity of 80% with precision of +/− 15%, 95% significance level and 80% power, 150 well and 150 sick babies (i.e. those having VSD or PVSD) were required. Since the prevalence of VSD and PVSD was about 10% in the community, we needed to oversample sick newborns. To recruit neonates randomly and yet to recruit about equal numbers of well and sick newborns, we used the following sampling method. We randomly selected a CHW supervisor every day (n=5) and asked the supervisor to prepare a list of well and sick newborns in the eight CHW areas under her supervision by mid day. If there was one or more sick newborns identified, then the first sick newborn identified was selected. If there was no sick newborn in her area, then the first well newborn from her area was selected. The CHWs were not informed of the selection. After the CHW completed her scheduled visit, the supervisor notified a study physician using cell phone who reassessed the selected newborn within 3–6 hours using the same clinical algorithm, either in the home or at hospital if newborn was successfully referred. The CHW who assessed the selected newborn was not present during the physician's assessment. CHWs were given feedback on the results of the two assessments at their regularly scheduled fortnightly meetings by supervisors.
A total of 288 neonates participated in the validation study and mean age at assessment was 9 days (standard deviation ± 6.9, range 1–28); 53% of assessments were performed during the first week of life. Forty-one CHWs active during the validation study participated, 31 with at least one year of work experience and the others with less experience. Each CHW completed an average of 7 assessments (standard deviation ± 4.4, range 1–17). One of the two physicians conducted 60% of the re-assessments
All 288 neonates with consecutive assessments by CHW and physician were analyzed. Agreement between CHWs' and physicians' assessments was evaluated using kappa statistic, with values of <0, 0.00–0.20, 0.21–0.40, 0.41–0.60, 0.61–0.80, and 0.81–1.00 considered as poor, slight, fair, moderate, substantial, and almost perfect agreement, respectively (Landis and Koch 1977). All validity measures were calculated using physicians' assessment as the gold standard. Sensitivity, specificity, and 95% confidence intervals (CI) were calculated for CHWs' identification of signs and symptoms and classification of VSD and PVSD. To estimate the predictive value of CHWs' assessments in the entire Home Care arm population, we calculated positive predictive value (PPV) and negative predictive value (NPV) using sensitivity and specificity from the validation study, and prevalence of signs and symptoms in the Home Care arm, since the predictive value of a test varies based on prevalence of the condition in the population (Altman and Bland 1994; Gordis 2004; Hussaini et al. 1999; Verbeek et al. 2000). We estimated prevalence data from postnatal visit records of 8474 newborns assessed by CHWs from January 2004– December 2005. Newborn were counted as having VSD or PVSD if CHWs classified the newborn at least once with VSD or PVSD during any visit. Data were analyzed using Stata Version 9.2 (StataCorp, College Station, TX, USA).
Study supervisors and investigators checked the accuracy of data collection by CHWs in the field and routinely reviewed data forms for accuracy, consistency and completeness. All data quality problems were addressed promptly. Data were entered in custom-designed databases with necessary range and consistency checks and periodically checked by reviewing frequency distributions and cross-tabulations.
The study received ethical approval from the Johns Hopkins Bloomberg School of Public Health Committee on Human Research and the Ethical Review Committee of the International Centre for Diarrhoeal Disease Research, Bangladesh. Informed verbal consent was obtained from all individual study participants by study staff.
Sensitivity of CHWs' assessment as compared to physicians' assessment was above 60% for all signs and symptoms in the algorithm except mild fever (temperature 37.8–38.4°C) which was 28.6% and history of convulsion (50.0%) (Table 1). Specificity was 97–100% for all signs and symptoms. Sensitivity for the most prevalent sign of illness in the validation sample, rapid breathing, was 87.5% and specificity was 98.0%. The kappa values were highly significant (p<0.001) and indicated substantial (0.61–0.80) or almost perfect agreement (0.81–1.00) between the two sets of assessments on 16 of the 20 signs and symptoms of the algorithm. CHWs' identification of mild fever (temperature 37.8–38.4°C) achieved only fair agreement (kappa 0.39, p<0.001). A negative value (kappa −0.01) for the sign umbilical redness not extending to the skin indicates less than chance agreement, although the result was not significant (p=0.54). Kappa values were not calculated for the remaining two signs, severe chest in-drawing and unconsciousness, because they were not observed in any newborns.
CHWs' classification of VSD had a 90.5% sensitivity, 95.3% specificity, and kappa value of 0.85 (p<0.001) which corresponds to almost perfect agreement between CHWs' and physicians' assessments. CHWs' classification of PVSD had a kappa value of 0.78 (p<0.001) indicating substantial agreement as well as high sensitivity and specificity (81.4% and 95.6%, respectively).
For individual signs and symptoms in the algorithm, PPV ranged from a low of 29.6% for mild hypothermia (temperature 35.3–36.4°C) to a high of 100.0% for five other signs (history or observed convulsions, fever ≥ 38.3°C, bulging fontanelle, and vomiting) (Table 2). We calculated PPV of 51.0% for VSD and 66.4% for PVSD. NPV was high (99–100%) for all signs and symptoms, and classifications.
We evaluated the ability of trained CHWs in a rural area of Bangladesh to assess newborns in their homes and correctly identify sick newborns, in comparison with assessments of physicians. We demonstrated that CHWs with no prior background in health were able to recognize signs of illness and classify illness in neonates using a simple clinical algorithm after six weeks of training. CHWs correctly identified newborns with VSD requiring immediate referral with a sensitivity of 91%, specificity of 95%, and kappa value of 0.85. Based on their assessment of clinical signs, CHWs missed 7 out of 74 cases of VSD and incorrectly classified 10 neonates out of 214 neonates as having VSD.
Compared to previous validation studies of CHWs' assessments of neonates in their homes, CHWs in this study detected illnesses with equal or higher sensitivity and similar specificity. In a seven year study in rural India, the ability of village health workers (VHWs) to diagnose neonatal sepsis using a shorter algorithm of seven signs was compared to the results generated by computer algorithm using visit records (Bang et al. 2005). VHWs diagnosed sepsis cases with 89% sensitivity and 99.5% specificity in comparison to computer algorithm. Our group recently conducted another validation study of CHWs with similar training in Mirzapur, Bangladesh which also compared their assessment of neonates to that of a study physician (Darmstadt et al. 2009). This study used an 11-sign algorithm to diagnose VSD and observed 73% sensitivity, 98% specificity and kappa value 0.63, indicating substantial agreement between CHWs and physicians. Lower sensitivity in the Mirzapur study may be due to inclusion of fewer ill newborns (3% VSD compared to 47% in this study) and longer interval between CHW's and physician's assessments.
Our results also compare favourably to validation studies in older children; our sensitivity for severe illness was higher and specificity was equivalent. In a hospital-based study in Kenya, CHWs with three weeks of initial training and a 1 week refresher course used a 21-sign algorithm to identify very severe disease (VSD) in children < 5 years old with sensitivity 57–65% compared to physician's assessment (VSD was indicated by presence of one of 4 danger signs or fever in children <2 months) (Kelly et al. 2001). In a study in northern and central Bangladesh, female community health volunteers (CHVs) received three days of training on identification and treatment of acute respiratory infections (ARIs) in children 3–59 months of age and monthly refresher trainings (Hadi 2003). They diagnosed ARIs with 68% sensitivity and 95% specificity compared to study paediatrician's assessment.
Ability to recognize fast breathing is a necessary skill for diagnosis of severe illnesses in most algorithms for assessing ill infants and children, including current WHO IMCI guidelines (World Health Organization and UNICEF 2005). Our analysis indicates CHWs in Sylhet identified fast breathing accurately with sensitivity of 88% and specificity of 98%. Since fast breathing was the most commonly observed symptom in our study (14%), the high sensitivity and specificity with which CHWs identified fast breathing was an important factor in overall accuracy of their diagnoses. Previous CHW evaluations of sensitivity and specificity in detection of fast breathing reported lower sensitivity and specificity. In the Mirzapur study, two levels of fast breathing were included in the algorithm and sensitivity of both signs was very low (25% for ≥70 breaths per minute and 7% for 60–69 breaths per minute) (Darmstadt et al. 2009). In the Kenya study mentioned above, CHWs recognized tachypnea in children <5 years of age with moderate sensitivity (62–66%) (Kelly et al. 2001). Another study of children <5 years in Uganda reported that community drug distributors with two days of training on detection of ARIs correctly counted and classified fast breathing with 75% sensitivity and 83% specificity (Kallander et al. 2006).
The strong initial training and continuing supervision and support received by CHWs in our study are likely reasons why we observed such strong agreement between assessments. Seventy-eight percent of CHWs working at the time of the validations study received 6 weeks of training and had worked for the project for a year or more. CHWs in most validation studies discussed here had shorter training periods, sometimes only a few days, and little ongoing supervision or feedback on performance. The Bangladesh study mentioned previously compared assessments of community health volunteers (CHVs) who received routine supervision by para-professionals to those of CHVs who did not (Hadi 2003). Regular supervision in that study included monthly meetings to discuss performance and re-examination of selected patients by supervisors to confirm diagnosis. Supervised CHVs were significantly more likely to make a correct ARI diagnosis in children 3–59 months of age compared to those who were supervised irregularly (93% vs. 78%, p<0.01). Logistic regression analysis to control for factors such as child's age and gender and CHVs' experience and training indicated odds of correct ARI diagnosis were more than four times higher for regularly supervised CHVs (OR 4.21, p<0.001).
Sensitivities and specificities for CHWs' assessments reported here are based on oversampling of sick newborns. There were 133 newborns (47%) with VSD or PVSD in the validation sample while only 17% of neonates in the parent trial population were classified with illnesses. To predict the probability of a correct assessment and diagnosis by CHWs, we applied prevalence data from all neonates assessed in the parent trial (Hussaini et al. 1999; Verbeek et al. 2000). Therefore, our reported PPV of 51% for VSD indicates an estimated 51% of newborns classified with VSD by CHWs during the entire trial were correctly diagnosed (Altman and Bland 1994). Based on these calculations, we estimate between one-third and one-half of newborns would be inaccurately classified as ill by CHWs in routine use of this algorithm in Sylhet. This could lead to considerable over-referral and subsequent burden on first-level health facilities, as well as unnecessary treatment. However, actual compliance with referral was low in the parent study.
Rates of successful referral to qualified medical providers were 34% for VSD, 25% for PVSD with multiple signs and 10% for PVSD with one sign (Baqui et al. 2008b). High NPV for VSD and PVSD indicate that an estimated 2% of newborns classified by CHWs in the parent trial as healthy would be false negatives. Thus, risk of failure to treat ill newborns would be low. Considering the high neonatal mortality rate in Bangladesh, we prefer a low risk of missed diagnosis and higher potential referral rather than the reverse situation. Further CHW training targeted to improve assessment skills for signs with low sensitivity and refinement and simplification of the current algorithm would likely decrease the number of newborns unnecessarily referred. In addition, while we would hope to see higher successful referral in the future, low compliance during the trial suggest there is little risk expansion of CHW surveillance for newborn illnesses would generate an unreasonable burden on the local health system.
Strengths of this study include the design which ensured a sample with high percentage of ill babies and the ability to use prevalence data from the parent trial to predict accuracy of CHWs' assessments under routine conditions. In addition, we present validation results from neonates assessed in their homes, a population infrequently studied in this context. A potential limitation of all validation studies is the lag time between consecutive assessments. Although efforts were made to time the physicians' visits to follow as soon as possible after CHWs', there may have been changes in the condition of the newborn between assessments.
In conclusion, this study corroborates limited evidence that CHWs can recognize signs of illness and classify illnesses in newborns during the first month of life using a clinical algorithm. Our finding that CHWs can assess newborns with very high sensitivity and specificity provides strong support for expansion of community-based neonatal health care in settings where the burden of newborn illness is high and care seeking is low.
The validation study presented here was a sub-study within a larger study known as Projahnmo (the Project for Advancing the Health of Newborns and Mothers.) Projahnmo was a partnership of the ICDDR,B; the Bangladesh government's Ministry of Health and Family Welfare; Bangladeshi nongovernmental organizations, including Shimantik, Save the Children-US, Dhaka Shishu Hospital and the Institute of Child and Mother Health; and the Johns Hopkins Bloomberg School of Public Health. We thank the members of the Projahnmo Technical Review Committee, the members of the Shimantik Executive Committee, and colleagues at the Bangladesh Ministry of Health and Family Welfare at the sub-district, district and national levels for their valuable help and advice. We thank Renata Pharmaceuticals Ltd, Bangladesh, for preparing the penicillin for study purposes. We thank Sylhet MAG Osmani Medical College for serving as our training venue as well as providing support for management of sick babies referred from the project area. We thank the many individuals in Sylhet district who gave their time generously as well as Projahnmo field and data management staff who worked tirelessly. The critical innovative inputs of Projahnmo study group members are acknowledged.
Funding Sources This research was funded by the United States Agency for International Development (USAID), through cooperative agreements with the Johns Hopkins Bloomberg School of Public Health and the International Centre for Diarrhoeal Disease Research, Bangladesh (ICDDR,B), and by the Saving Newborn Lives program of Save the Children-US through a grant from the Bill & Melinda Gates Foundation. Study sponsors had no role in study design; the collection, analysis, and interpretation of data; writing of the report; or in the decision to submit this article for publication.
Conflict of Interest We declare that we have no conflict of interest.