PMCCPMCCPMCC

Search tips
Search criteria 

Advanced

 
Logo of f1000resSubmitAuthor GuidelinesAboutAdvisory PanelF1000ResearchView this article
 
Version 3. F1000Res. 2016; 5: 1477.
Published online 2017 February 23. doi:  10.12688/f1000research.8287.3
PMCID: PMC5357025
Other versions

Electronic medical records in humanitarian emergencies – the development of an Ebola clinical information and patient management system

Abstract

By November 2015, the West Africa Ebola epidemic had caused 28598 infections and 11299 deaths in the three countries most affected. The outbreak required rapid innovation and adaptation. Médecins sans Frontières (MSF) scaled up its usual 20-30 bed Ebola management centres (EMCs) to 100-300 beds with over 300 workers in some settings. This brought challenges in patient and clinical data management resulting from the difficulties of working safely with high numbers of Ebola patients. We describe a project MSF established with software developers and the Google Social Impact Team to develop context-adapted tools to address the challenges of recording Ebola clinical information. We share the outcomes and key lessons learned in innovating rapidly under pressure in difficult environmental conditions. Information on adoption, maintenance, and data quality was gathered through review of project documentation, discussions with field staff and key project stakeholders, and analysis of tablet data. In March 2015, a full prototype was deployed in Magburaka EMC, Sierra Leone. Inpatient data were captured on 204 clinical interactions with 34 patients from 5 March until 10 April 2015. Data continued to also be recorded on paper charts, creating theoretically identical record “pairs” on paper and tablet. 83 record pairs for 33 patients with 22 data items (temperature and symptoms) per pair were analysed. The overall Kappa coefficient for agreement between sources was 0.62, but reduced to 0.59 when rare bleeding symptoms were excluded, indicating moderate to good agreement. The time taken to deliver the product was more than that anticipated by MSF (7 months versus 6 weeks). Deployment of the tablet coincided with a dramatic drop in patient numbers and thus had little impact on patient care. We have identified lessons specific to humanitarian-technology collaborative projects and propose a framework for emergency humanitarian innovation. Time and effort is required to bridge differences in organisational culture between the technology and humanitarian worlds. This investment is essential for establishing a shared vision on deliverables, urgency, and ownership of product.

Keywords: Ebola, electronic medical records, EMR, innovation, humanitarian

Background

By November 2015, the West Africa Ebola epidemic had caused 28598 infections and 11299 deaths in the three countries most affected 1. In response, Médecins sans Frontières (MSF), drawing on over 20 years of experience in responding to Ebola outbreaks, had treated more than 7500 suspected Ebola cases, including more than 4700 confirmed cases. The outbreak, unprecedented in size, required rapid innovation and adaptation. To cope with the number of cases, MSF scaled up its usual 20–30 bed Ebola management centres (EMCs) to 100–300 beds with over 300 workers in some settings. This increased scale brought challenges in patient and clinical data management resulting from the difficulties of working safely with high numbers of Ebola patients ( Panel 1). Little had been published on managing clinical documentation and data transfer from filovirus wards, and there were no established, standardized, or widely used approaches 2. Here we describe how MSF developed context-adapted tools to address the challenges of recording Ebola clinical information for in-patient management. We share the key issues and lessons learned in innovating rapidly under pressure in difficult environmental conditions and propose a framework for emergency humanitarian innovation.

Panel 1.

Challenges in patient and clinical data management in Ebola management centres.
• Staff working in the high-risk zone of an EMC (the area where there is high risk of Ebola infection) must wear PPE, which is heavy,
limits vision severely, adversely affects manual dexterity, and is extremely hot. Clinicians have a strictly limited time in which they
can safely remain in PPE without overheating and increasing infection risk during PPE removal. A lot of time was lost while clinicians
in PPE tried to find patients inside the high risk zone of the EMC. Patients were moved through tents in the EMC representing
different levels of risk during their treatment and could also move around within the EMC of their own volition (patients can become
confused and delirious during Ebola infection), which made locating a specific patient in a large EMC challenging.
• Clinicians lacked an efficient means to document patient history and record patients’ progress through the EMC. In some EMCs,
clinicians in full PPE shouted data over the fence for someone outside the high-risk zone to record on ‘clean’ paper, or data were
recorded in digital photographs, since nothing could be taken out of the high-risk zone without being disinfected in chlorine. The
original paper records made in the high-risk zone were incinerated. As a result, adequate data to support good clinical care were
rarely available.
• Data collection processes varied widely between EMCs, resulting in little data sharing between them and limiting the potential for
learning and improving care. There was no centralized organizing authority for data collection forms or formats.
• Despite the efforts of the Emergency Telecommunications Cluster *, most EMCs had spotty and intermittent internet connectivity at best.

EMC=Ebola management centre. PPE=personal protective equipment.

*Information on the ETC is at: http://www.etcluster.org/about-etc

Methods

The project

In September 2014, MSF medical staff established a collaborative project with a small group of software developers and the Google Social Impact Team. The aim was to develop a tool to enable the collection, visualisation, and sharing of standardised information on Ebola patients and treatment programmes, and thereby make the most efficient use of the limited time that staff are able to remain in personal protective equipment (PPE) to treat patients in the high-risk zone of an EMC.

Informal discussions with current and returning MSF field staff enabled a first set of requirements to be drawn up. Initial scoping, conducted by Google over 2 days, showed that several data collection devices were already under development for Ebola, some of which promised to be rapidly deployable 3, 4. However, it was determined that none of these would sufficiently meet the requirements that had been identified. These requirements were further refined through extensive user consultation including current MSF national and international field staff, operations managers, and medical specialists with experience in Ebola, as well as public health specialists from Harvard Medical School and The London School of Hygiene and Tropical Medicine. A product specification was generated to meet these requirements ( Panel 2) based on the following principles: the solution should be available quickly (within 6 weeks); it should be “just enough” to meet the requirements; it should be useable without programming knowledge; and both the product and its intellectual property (IP) should be freely or cheaply available.

Panel 2.

Core requirements and the initial proposed solution (the minimum viable product).
RequirementsSolution (minimum viable product)
Clinicians within the high-risk zone wearing PPE
are able to quickly, easily, and safely record and
visualise patient data, to enable appropriate
clinical decision making.
Waterproof tablets with wireless charging, disinfected by dipping in 0.5%
chlorine. A tablet app enabling rapid data entry in PPE (large buttons,
no swipe-screen features), providing instant access to patient historical
records, a current patient list which staff can fill during clinical rounds, and
an interface for the creation of more fields as
required.
Data transfer out of the high-risk zone is easy,
safe, and secure, enabling aggregation and
analysis of outcomes (programme monitoring).
A local server-side application to synchronise data between tablets, store
it securely on a server outside the high-risk zone, and export anonymised
patient data (to the field office and headquarters in Europe) when an
internet connection is available.
System functions with unreliable power supply
(up to 12 hours without electricity), poor or no
internet connectivity, and no in-country IT support
desk or readily available electronics supply.
Lightweight, plug-and-play, battery-powered server infrastructure,
automatically backed up on every device in case some fail.
Clinicians within the high-risk zone able to locate
patients quickly, easily, and safely.
Radio-frequency identification tags on wristbands to rapidly locate
patients within the centre and confirm their identity; readers for the tags
located on each tablet device.

PPE=personal protective equipment.

To speed up development and ensure that the final tools would be available free of charge to Ministries of Health or other humanitarian organisations, the development team used pre-existing open-source platforms where possible, rather than developing from scratch. The data model and database from OpenMRS (an open-source Medical Record System; platform v 1.10.1 on an Edison server running Yocto Linux + Debian GNU/Linux 7) were combined with data entry elements from OpenDataKit (open-source mobile data collection tools), together with a bespoke user interface (made with code forked from ODK Collect v1.4.4, running on Android 4.4.2 [KitKat]), designed for a high-risk zone. Panel 3 outlines the tool that was finally developed. A full, open-source, technical specification of the product is available 5.

Panel 3.

Core components of the final Ebola data management solution.
Hardware
Sony Android tablets (Sony Experia Z2 running Android 4.4.2 [KitKat]) in a custom-built polycarbonate casing containing a radio-
frequency identification reader and rechargeable batteries, custom-built by Thinkify.
The server (Edison server running Yocto Linux + Debian GNU/Linux) and local area network were based on a federated Edison
server structure (housed in individual waterproof polycarbonate boxes developed by Thinkify) powered by rechargeable batteries
linked to mains electricity.
Software
The tablet software is a custom-built Android app. The server-side application (back end) is OpenMRS (platform v 1.10.1) built
on a Linux Operating System running on a low-energy server (Intel Edison), with a custom-built module for data import from the
Android app (imported data is anonymised but linkable to an individual if required, using a patient ID code and approximate date
of birth based on current age).
Data content
The following data fields were hard-coded:
       • Identifiers: patient identification number, approximate date of birth, encounter time and date, date of EMC admission.
       • Bleeding: blood in stool (black/red), red eyes, bleeding from eyes, nosebleed, haemoptysis, haematemesis, vaginal
bleeding, haematuria, oral bleeding, bleeding NOS.
       • Other symptoms: chest pain, muscle/joint pain, difficulty breathing, hiccups, heartburn, headache, cough, back pain, sore
throat, abdominal pain (upper/lower), nausea.
       • Signs: consciousness, hydration, diarrhoea (episodes/24 hours), mobility, vomiting (episodes/24 hours), blood pressure,
heart rate, temperature (°C), weight (kg), weakness, respiratory rate, pregnant, diet, IV access present, pain level.
       • Laboratory test results: Ebola NP gene PCR CT value, Ebola L gene PCR CT value.

EMC=Ebola management centre. NOS=not otherwise specified. IV=intravenous.

The client-server architecture developed for this project involves a low-energy server implemented on a 36 × 25 × 4 mm Intel Edison computer, built into an enclosure full of batteries and a custom charging circuit to ensure all-hours reliability. Rather than relying on the internet for updates, backups, and maintenance, the backup and updating system relies solely on USB sticks. The client software installed on the tablets can be replaced with any other application, which can then benefit from the hardware and server capacity of the system. Servers can be deployed inside and outside the high risk zone, enabling safe and effortless transfer of data between locations.

Approximately US$1.9 million was spent on development, of which $1.8 million was provided by Google. This included a team of five full-time engineers for 6 months, as well as contractors and consultants. Approximately $500,000 was spent on custom manufacturing of hardware, often at above-market costs due to the urgency of the project. For instance, a factory in California was commissioned to run for 72 hours straight during the Christmas of 2014 to manufacture tablet enclosures. In all, the project took 7 months from concept to deployment.

Piloting and evaluation

In January 2015, an alpha version was field-piloted in the MSF EMC in Magburaka, Sierra Leone; feedback from the field team was incorporated and bugs were fixed. In March 2015, a full prototype was deployed in Magburaka. Inpatient data were captured on 204 clinical interactions with 34 patients from 5 March until 10 April 2015 (equating to 95% of those admitted in this relatively quiet period). In this initial deployment, for each clinician-patient interaction the routine paper record system was maintained in parallel; clinical observations and treatments were recorded on paper in the high risk zone by the clinician wearing PPE, then shouted over the fence to a colleague standing in the low-risk zone (an area of an EMC where there is low risk of Ebola transmission and staff do not wear full PPE) who transcribed them to ‘clean’ paper patient charts, and later entered into an Excel database by a data encoder.

Information on adoption, maintenance, and data quality was gathered through review of project documentation, discussions with field staff and key project stakeholders, and analysis of data from the tablets. We carried out a rapid informal mixed methods evaluation to look at adoption/acceptance by health workers, implementation and maintenance challenges, and data quality and usefulness. Observation of staff and six unstructured staff interviews were carried out by a member of the implementation team with experience of implementing and evaluating e-health innovations, and were recorded in note form. Six semi-structured interviews with key project stakeholders (Google, MSF, Harvard) were carried out using a topic guide by an MSF administrator with experience in qualitative research, who recorded and transcribed the interviews. Project documentation from September 2014 to April 2015, including situation reports, vision and requirements documents, were included as an additional data source. Thematic analysis of project documentation, observations and field interview notes, and stakeholder interview transcripts was performed by the administrator supported by a senior team member with experience in health-care evaluation.

Data from the tablets were analysed via a semi-automated match of record “pairs” (clinician-patient interactions where a record for the same interaction existed in both the tablet dataset and the paper→Excel dataset). For data items in both tablet and Excel datasets (temperature and symptoms), the simple data (raw data rather than OpenMRS codes) were extracted from the tablet dataset and multiple records for interactions within 30 minutes were manually merged. If a symptom was recorded as present in any record in a set of multiple records within 30 minutes, the symptom was marked as present. The Excel dataset was checked and corrected against scanned paper records. The aligned tablet and cleaned Excel datasets were combined into a single list, sorted by patient and date/time of interaction, such that the single daily record in Excel was paired with the first tablet record that day. This is a potential limitation, as it assumes that no observations would be recorded on the tablet prior to the principle morning clinical encounter, at which the single daily paper record was made. The record pairs were then recoded to collapse graded symptoms in the tablet record (e.g. 24 hour count of diarrhoea or vomiting episodes, extent of weakness) to only symptom present or absent, and some symptoms combined into a single symptom variable (e.g. abdominal pain was a single variable on paper but recorded separately for upper and lower on the tablet, so a combined variable was used for comparison). Temperature was recoded to <37.5°C or >=37.5°C, and also matched using precise value. Kappa coefficients were calculated using STATA v14.2 (StataCorp, College Station, Texas USA) for each symptom and for all records with an entry in both sources. Observations of vital signs were not part of the Excel dataset, but the presence of these items in the scanned paper records was documented and compared to whether these signs were also recorded in the tablet record. The quality of the scanned paper charts was subjectively assessed.

Ethics

This evaluation met the criteria of the MSF Ethics Review Board for exemption from ethics review. The product was discussed with and demonstrated to health authorities prior to implementation, who were supportive of this innovation and did not suggest additional ethical scrutiny.

Results/outcomes

The time taken to deliver the product was substantially more than that anticipated by MSF (7 months, as opposed to 6 weeks). The initial specification was largely but not fully delivered; a patient localising feature (radio-frequency identification tags for patients and a network of readers) was developed but never completed, as the declining patient numbers reduced the potential value of this feature. In addition, exported data require significant work to clean and analyse outside of the application interface. The database was not configured to create a single record for a patient encounter if the user moved in and out of the record, resulting in up to five partial records within a 10 minute period which all related to a single encounter, and thus required to be merged into a single record of that encounter. Records were then exported with content and order based on OpenMRS coding, resulting in 3 data fields per data element and an order that was non-intuitive and would require further development for the product to be usable in field conditions.

Due to the hasty decommissioning of the Google team at the end of the project, there was insufficient time for a comprehensive handover (from Google to MSF) of information and understanding required to operate and maintain the software and hardware that had been developed. In the case of the software and key hardware, MSF was able to hire an ex-Google engineer who worked on this project to further develop the software and complete the process of handover to MSF. However some parts of the hardware (e.g. polycarbonate casing) were deemed too expensive to warrant further investment by MSF, so little attempt was made to obtain the knowledge necessary to produce these.

Deployment of the tablet coincided with a dramatic drop in patient numbers. As a result, the full prototype had little impact on patient care.

Adoption. The final product deployed in Magburaka was well received by medical staff, some of whom had no previous experience with computers or touch-screen devices. Staff described the system as intuitive and reliable for data entry and visualisation, even when power and internet connections were interrupted. Medical staff valued the ability to access more detailed, real-time, clinical data.

Implementation and maintenance. Tablets remained functional after frequent dipping in chlorine disinfectant, and the system remained active for periods of up to 24 hours without electricity.

Data quality and usefulness. 83 record “pairs” for 33 patients with 22 data items (temperature and symptoms) per pair were available to be analysed.

  • The overall Kappa coefficient for agreement between both sources was 0.62, but reduced to 0.59 when rare bleeding symptoms were excluded, indicating moderate to good agreement. For individual symptoms it the Kappa coefficient ranged from 0.29 to 0.86, with chest pain, diarrhoea and sore throat having only fair agreement, but precise temperature very good agreement. The number of pairs with data available in both sources per symptom ranged from 78–81 of the 83 record pairs. Most patients were admitted for a short period and symptoms were recorded on paper only once per day, so 30/33 (91%) had 1–3 record pairs. One patient had 19 record pairs, but if all record pairs beyond 4 were excluded, the Kappa coefficient was still 0.59.
  • The tablet contained more unique patient encounters than the paper records – the paper chart usually showed one set of observations per day, while the tablet was used to record additional encounters that were (at best) otherwise written only in patient progress notes (which were not standardised and thus difficult for a data encoder to enter into a standardised database format). However, it is unknown if this recording would still occur if the EMC were very busy.
  • The tablet contained data fields for vital signs (BP, HR, RR) that were not always recorded on the paper vital signs charts, but rather only on the free-text clinician progress notes (which had not been entered into the Excel database at the time of matching). At least one of these vital signs was entered in the tablet for about 40% of 204 interactions, usually matching the paper record, and sometimes providing data that did not exist in the paper record. In a few instances (11/85 interaction pairs), one or more vital signs in the paper record were not in the tablet dataset.
  • The paper charts were sometimes hard to read, leading to errors in the Excel database such as a temperature recorded in the paper record that could be read as either 39.2 or 34.2 (the tablet record showed 34.2). In addition, there was variable use in the symptoms chart of symbols such as yes, y, no, n, [check], ✘, -, |, ?. About 20% of paper-recorded encounters had some ambiguity, including no or only partial zero reporting (that is, specifying that symptoms were absent, rather than an assumption that any symptom not marked as present is absent as opposed to not assessed).

Interviews with field staff and key stakeholders revealed these common themes around challenges faced in the project.

  • Lack of agreed vision from start: The team’s haste to deploy something quickly that was “just enough” meant that insufficient time was taken to establish internal (MSF) project vision. Instead, the project team adopted a more perfection-driven Google vision (which extended the timeframe), and the user (MSF) questioned only late why something so complex had been made that was potentially useful in the long term, but not in this outbreak.
  • No explicit approach to project management: MSF project management structure, including governance and an organogram, was defined only when the project started to falter. As a result, the software developers were never sure who was the MSF lead, or who to go to when things went wrong, so at times asked the wrong question of the wrong person and lost focus on the minimum viable product requirements. At a late stage, efforts were made to develop an evaluation plan, but the numbers of cases dropped very substantially soon after.
  • Business pressures: the pressure to start meant that reflection on team composition happened too late. For example, the team approached numerous medical people with different backgrounds, experiences, and expectations for user input. Since the epidemic evolved rapidly, this focus on “the last medic returning” as the optimal source of information meant that the user stories became rapidly out of date. A consistent medical focal person was appointed to the team to provide ongoing guidance, but this was done too late. Likewise, Google’s need to move its developers onto other projects meant that there was insufficient time for effective knowledge transfer to a technologically-naïve partner.
  • Lack of understanding between the humanitarian and technology fields: MSF expected to be able to describe what was needed quickly and then leave the software developers to deliver, without realising the time commitment required from them to enable the developers to make the right product. In parallel, the developers expected that health programmes could be measured in terms of number of patients accessed, and struggled to incorporate a health outcome perspective in their objectives and notion of impact.

Discussion

The tablet facilitated more frequent and slightly more detailed data than the fence→ paper→Excel database routine, and there was high consistency between the results from the two systems. Given that the tablet data record is essentially real-time (in terms of data entry and opportunity for correction), it is likely to be the more accurate record. This assumption requires validation, but there is anecdotal evidence of the errors and uncertainties of the non-electronic data system from several EMCs.

To our knowledge, this is the first time a portable real-time electronic medical record (EMR) has been developed that specifically meets the needs of an extreme biohazard environment in a resource-limited setting with erratic power and internet supply. A client-server architecture normally necessitates a reliable, high-bandwidth internet connection, or a local server with very reliable electrical energy, IT support, and the availability of specialized parts. These requirements make a client-server architecture difficult to implement in rural sub-Saharan Africa.

MSF has gained experience and understanding of the process of technological innovation, including the limits and challenges of deploying new technology and working with technology partners 6. The positive collaboration between MSF and Google has sparked interest in the potential for humanitarian-technology collaborations 7. Successful deployment of reliable client-server architecture in a rural sub-Saharan African environment represents a proof of concept, such that it is already being deployed by other agencies (The Wellbody Alliance, in collaboration with Harvard Medical School).

However, there were major challenges with the project. Due to the length of time to deliver the product, in the context of an evolving Ebola outbreak, a product was delivered too late to have an impact on patient care or efficiency of EMC management. Lack of knowledge transfer meant that MSF had a product that they did not fully know how to support or adapt for future use.

The challenges we describe are echoed throughout the literature on humanitarian and technological innovation, such as best practice principles for humanitarian innovation defined by ALNAP and lessons learnt from IT innovation as outlined in the Chaos Manifesto 8, 9. Our experience has led us to identify some concrete lessons specific to humanitarian-technology collaborative projects. Most importantly, time and effort is required to bridge differences in organisational culture between the technology and humanitarian worlds. This investment is essential for establishing a shared vision on deliverables, urgency, and ownership of product. To guide this process, there is a need for a framework for innovation in humanitarian emergencies. The overwhelming priority for medical teams is immediate delivery of care, and innovation projects must fit around this reality; careful consideration must be given to whether there are sufficient resources to deliver the project. Therefore, emergency innovation projects need to be agile, iterative, and free from heavy project management processes. Yet if we are serious about innovation, we need to push the boundaries to ensure it has impact not only on our current patients, but also for future patients. We have outlined the key components of such a framework in Panel 4. This framework should be used as a flexible and practical tool; we caution against adapting systems that could lead to stifling of innovation, especially in complex and challenging environments where fast solutions are needed.

Panel 4.

A framework for humanitarian-technology innovation projects
• Problem statement and project vision, with deliverables.
• High-level requirements for the solution with early agreement on the minimum viable product. Users should be involved in
defining requirements, and for each subsequent step.
• Implementation requirements: clarifying who is involved and what they are responsible for; ensure they all have a common
understanding of terms such as impact, and that the right parties are involved in decisions.
• Implementation planning with timeline for key milestones (even if this is likely to evolve): planning, prototyping, piloting, roll
out. Ensuring the time factored in for input and delivery is feasible.
• Evaluation plan with expected impact and key performance indicators against which to measure success.
• Communication and change management plan: how and who to handle internal updates and external communication; who
needs to be consulted and who needs to be informed.
• Governance: expert steering committee (or equivalent) which validates each of the above milestones; a mechanism for
ethical oversight; clarity about who owns the IP that results.

An MSF team is in the process of adapting the software code-base of the Ebola data management solution to develop a generic emergency EMR that can be rapidly modified for different types of emergency. Our vision is to build a modifiable Open-MRS backbone for which new disease-specific apps can be developed within 5 days by a non-coder. The first ‘test-case’ is nutrition, for which we are working on apps for inpatient and ambulatory therapeutic feeding centres. In this project, having learnt from our experience with the Ebola EMR, we have taken the time to apply our framework ( Panel 4) from the start.

A complete kit of open-source software, hardware, and documentation for the Ebola EMR is available on-line and can be downloaded for free; the code is also freely available for developers to use 5. Several of the developers involved have formed an open-source community that is supporting this code, and can help develop the software for new-use cases 10. Harvard Medical School, MSF, and Google are all represented in this consortium. Harvard Medical School have carried out successful deployments of the hardware, which they have combined with new software for community surveillance, triage, and inpatient care. The consortium is focused upon new use-cases, the economics of the solution and driving economies of scale, the incorporation of additional software and hardware toolsets, and the enabling of clinical research during outbreak settings.

Conclusion

The fundamental innovation of our project in the end, was not the new technology that was developed, but rather the adaptation of existing technology so that it would work in an environment that is generally hostile for complex systems. We hope that the lessons learned and the tools developed in this collaboration will help others involved in innovation in humanitarian crises to gain the balance between speed, impact, and sustainability.

Data availability

The data referenced by this article are under copyright with the following copyright statement: Copyright: © 2017 Jobanputra K et al.

Data are deposited in a secure server in MSF Operational Centre Amsterdam and are available via a managed access process under the terms of the MSF Data Sharing Policy 11. For contact details and explanation of process please visit http://fieldresearch.msf.org/msf/handle/10144/306501 or email gro.fsm@gnirahs.atad.

Acknowledgments

The authors would like to acknowledge Jesse Berns (MSF Switzerland) and Karl Blanchet (London School of Hygiene and Tropical Medicine) for their valuable input into the initial development of the Ebola Clinical Information System. We thank Sarah Venis (MSF UK) for editing assistance; Isobel Aiken for key informant interviews; and Philipp du Cros for project oversight and input into the article.

Notes

[version 3; referees: 2 approved]

Funding Statement

The author(s) declared that no grants were involved in supporting this work.

Notes

Revised. Amendments from Version 2

This article has been updated in response to reviewers' comments as follows: - Clarifications have been added as to how data was transferred out of the high risk zone - Kappa coefficients have been used to show the degree of agreement between electronic and paper data collection - Further cleaning of the raw data to put it in an appropriate format for calculating the kappa statistic revealed a minor error in record allocation.  This correction has been made in the text, with ‘85 record pairs for 32 patients with 26 data items’ being changed to ’83 record pairs for 33 patients with 22 data items’

References

1. WHO: Ebola Situation Report - 18 November 2015. (accessed 18/11/15). Reference Source
2. Bühler S, Roddy P, Nolte E, et al. : Clinical documentation and data transfer from Ebola and Marburg virus disease wards in outbreak settings: health care workers’ experiences and preferences. Viruses. 2014;6(2):927–937. 10.3390/v6020927 [PMC free article] [PubMed] [Cross Ref]
3. Gallego MS.: Project ELEOS: a barcode/handheld computer based solution for Ebola Management Centres.Presentation at: MSF Scientific Day 2015, London (accessed 11/11/15). Reference Source
4. Jazayeri D, Oza S, Ramos G, et al. : Design and development of an EMR for Ebola Treatment Centers in Sierra Leone using OpenMRS. Stud Health Technol Inform. 2015;216:916. 10.3233/978-1-61499-564-7-916 [PubMed] [Cross Ref]
5. Project Buendia. (accessed 11/11/15). Reference Source
6. Gayton I, Achar J, Greig J, et al. : Tablet-based clinical management tool: building technology for Ebola management centres.Presentation at: MSF Scientific Day 2015, London. (accessed 11/11/15). Reference Source
7. Metz C.: Google Builds a New Tablet for the Fight Against Ebola. Wired. 2015; (accessed 11/11/15). Reference Source
8. Ramalingham B, Scriven K, Foley C.: Innovations in International Humanitarian Action. In: ALNAP Review of Humanitarian Action (ch 3)2014. Reference Source
9. The Standish Group: Chaos Manifesto 2013: Think big, act small.(Accessed 11/11/15). Reference Source
10. Project Buendia. (accessed 21/09/2016). Reference Source
11. Karunakara U.: Data sharing in a humanitarian organization: the experience of Médecins Sans Frontières. PLoS Med. 2013;10(12):e1001562. 10.1371/journal.pmed.1001562 [PMC free article] [PubMed] [Cross Ref]

Review Summary Section

Review dateReviewer name(s)Version reviewedReview status
2017 March 10James Whitworth and Hilary BowerVersion 3Approved
2016 October 25James WhitworthVersion 2Approved with Reservations
2016 August 23Benjamin O. BlackVersion 1Approved
2016 July 11James Whitworth and Hilary BowerVersion 1Approved with Reservations

Approved

James Whitworth, Referee1 and Hilary Bower, Co-referee2
1Department of Infectious Disease Epidemiology, Faculty of Epidemiology and Population Health, London School of Hygiene and Tropical Medicine, London, UK
2Department of Infectious Disease Epidemiology, Faculty of Epidemiology and Population Health, London School of Hygiene & Tropical Medicine, London, UK
Competing interests: A colleague from LSHTM is acknowledged for his input into the development of this information system. We have had no direct interaction about this project. HB worked for MSF in Sierra Leone in the same period as the trial but had no involvement in it. She has worked with some of the authors in other locations and contexts.
Review date: 2017 March 10. Status: Approved

We thank the authors for their clarification of all the points that we have queried and for their revisions of the text. We are content to approve the paper fully now and congratulate the authors on an interesting contribution to the scientific literature.

We note that the authors have used and interpreted the Kappa statistic as 'moderate to good' in the abstract and results, but called it 'high' consistency in the discussion. We think this apparent discrepancy is acceptable as for some elements, e.g. temperature the kappa statistic was very good. 

We provide below a reference for the interpretation of the Kappa statistic for those who might require it. 1

We have read this submission. We believe that we have an appropriate level of expertise to confirm that it is of an acceptable scientific standard.

References

1. Viera AJ, Garrett JM.: Understanding interobserver agreement: the kappa statistic. Fam Med.2005;37(5) :360-3 [PubMed]

Approved with Reservations

James Whitworth, Referee1
1Department of Infectious Disease Epidemiology, Faculty of Epidemiology and Population Health, London School of Hygiene and Tropical Medicine, London, UK
Competing interests: A colleague from LSHTM is acknowledged for his input into the development of this information system. We have had no direct interaction about this project.   HB worked for MSF in Sierra Leone in the same period as the trial but had no involvement in it. She has worked with some of the authors in other locations and contexts.
Review date: 2016 October 25. Status: Approved with Reservations

Thank you for giving us the opportunity to review the revised article and explanations from the authors. In general they have responded and given clarifications either in the revised article, or the covering letter, and sometimes both. However, there are a few outstanding points that have not been addressed and we would like to see responses to these before we would be ready to amend the current status of approved with reservations.

Our outstanding three comments are detailed below.

  1. The authors have added a para about exporting data from the red zone, but it would still be good to have some simple text explaining exactly what staff needed to do to transfer the data, to help those readers who find ‘servers’ a technical term. We are still a bit puzzled about the reference to USB sticks for backup and updating and whether that refers only to outside the red zone back up, or internally.
  2. As mentioned in our previous referee report:
    “The results are generally clearly described, and suggest that the tablet may be at least as good as paper records, though the analysis currently does not appear to take into account the role of chance, and a proportion of records (17% ?) had to be excluded. The qualitative study is quite weak. It appears that different data were collected systematically using the two different systems, so only some of the data were collected using both methods.
    Further comments on the quantitative study are:
    If we understand correctly the agreement between the two methods was calculated by simple percentage agreement: would it be possible to perform a Kappa test to take into account chance?
    What was the range of interaction pairs per patient? Was any difference seen in the recording by either method related to the severity of the patient’s illness, or the number of people admitted at the time?
    Is it possible to state the percentage of missing data overall and/or by recording method and comment on the implications for the agreement results? On rough calculation overall it seems to be 17% (204-(85x2) = 34, 34/204) missing data in one or other system.
    The agreement between data sources ranged between 69 and 95%. This is rated as ‘high consistency’ but I was not sure on what basis. It was not clear to me, which collection method was felt to be the gold standard, or even if this was determined before the comparison. I sense from the discussion, the tablet was thought to perform better than the paper method, but I am not convinced about this from the results.” 1
    These points are not addressed either in response or text – and the reference to high consistency remains (1 st para discussion)
  3. Similarly, this point has also not been addressed:
    “The authors claim that this is the first such device to be developed. How did they search for other evidence of similar projects? I heard anecdotally of other projects, but am not aware whether they reached fruition, or have ever been published in any form. It would be good to have a thorough systematic search to establish what else has been developed.” 1

I have read this submission. I believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard, however I have significant reservations, as outlined above.

References

1. Bower H, Whitworth J.: Referee Report For:Electronic medical records in humanitarian emergencies – the development of an Ebola clinical information and patient management system [version 2; referees: 1 approved, 1 approved with reservations]. F1000Research.2016;5(1477) : 10.5256/f1000research.8913.r14552 10.5256/f1000research.8913.r14552 [PMC free article] [PubMed] [Cross Ref]

Approved

1The Whittington Hospital, London, UK
Competing interests: BOB worked for Médecins Sans Frontières during and after the Ebola outbreak 2014-2016, including in Magburaka, Sierra Leone. However he was not directly involved in the design, implementation or use of the Electronic Medical Record, and was not present in Sierra Leone during the time of the trial. He has worked with some of the authors in other projects and contexts.
Review date: 2016 August 23. Status: Approved

Research and development of technology in the humanitarian arena is a rapidly growing field that encompasses many specialities. There are international working groups, peer reviewed journals and conferences dedicated to this field. This paper is a valuable addition to the body of evidence openly available to the academic, technician and aid worker. My review of this paper relies on my field experience of working in the humanitarian setting, particularly though the West African Ebola outbreak of 2014 till 2016.

This paper describes the process, implementation and challenges of introducing a novel way of recording patient data electronically in a high biohazard environment. The aim being to improve data collection and the efficiency of patient interactions and communication.  

It is bold and brave of the authors to openly state the errors made through the design and implementation of this project. While there is substantial discussion around the lessons identified from their experience, there could have also been more highlighting some of the possible successes, not least that this project was rolled out at a time of unprecedented pressures on the organisation responsible, Médecins Sans Frontières (MSF).

In their peer review Whitworth and Bower (2016) have critically appraised the methodology and data analysis described in this paper. I am therefore not going to repeat what has already been expertly written, focussing instead on the patient care and field worker practicalities of this tool.

Involvement from the field is raised on several occasions in the article, however it remains unclear to what lengths the designers and implementers used the returnees’ knowledge on the field realities to make this tool suitable for the task in-hand, beyond “informal discussions”. All volunteers returning from Ebola missions were routinely interviewed in the European headquarters by the organisation, to what extent this was used as an opportunity by the developing team is unclear, if not utilised was this a missed opportunity that could be added to their list of “lessons identified”. It would be useful to know how many returnees were interviewed, the style of interview used and how the information gained guided the designers. Furthermore, given that all returnees were debriefed at head office it would be worth stating if the returnees were all systematically questioned in a standardised manner, or if specific persons were chosen and how this choice was made.

The field staff who used both the alpha version and full prototype were also interviewed, it would be informative to the reader to understand how many staff were involved in these interviews, whether they were expatriate or local staff and what their specific job role was. The experience of a European doctor may be different to that of a local nurse or health care assistant.

The authors rightly discuss the difficulty of working in the full Personal Protective Equipment (PPE) that is necessary to remain safe inside the “high risk” area of the Ebola Management Centre (EMC), it would be interesting to gain a deeper insight into how much this was factored into the overall program design. For example, visual difficulties and clumsiness were often described as part of the challenge of working in PPE. Were large buttons or colour coding on the tablet part of the design to assist the healthcare worker overcome these difficulties?

During very busy times at the peak of the outbreak a numbered system was used to quickly identify the patients with the most immediate needs, a score from 1 (very well) to 5 (critically ill) was assigned to each patient. It would have been informative to know if this or other field-designed approaches were considered or incorporated into the program. Whilst the focus of the tablet appeared to be for transfer of medical information, there is much communication that must also be made between the EMC workers. Identification and location of deceased patients must be efficiently passed to hygiene teams, or safety concerns to logistical teams, was inter-disciplinary communication considered as part of this program?

Similarly, whilst communication on patient symptomatology was traditionally shouted across the fence and then transcribed into folders outside the “high-risk” area, treatment decisions also needed to be remembered or shouted over. It is not entirely clear how much the tablet was utilised for recording treatments given, for example intravenous fluid administration, hygiene procedures or palliative care medications. This would be valuable from a patient care perspective and likely reduce prescribing errors, missing or duplication of doses. Retrospectively this could also have added to data on what treatment and care was being given and how often. Commonly a generic recipe of medications was administered to all patients from admission into the EMC, this could have therefore been pre-programmed into the tablets as an electronic prescription.

The method of shouting information across the fence in the EMC, whilst this was often a part of high to low risk communication, was not exclusive. Digital photography that could later be transcribed was often used, particularly during times of high activity or when transferring patients from one location to another. Similarly, only those patients that were bedbound or too unwell to speak with healthcare workers from across the fence required examination and questioning in full PPE. Hence for many inpatient interactions the “high-risk” tablet would not be required. Explanation of how these other interactions were incorporated into the data gathering would be useful, particularly for the completeness of the information generated. The benefits of being able to talk with a patient across the fence, where the clinician was recognisable (no PPE needed) and there were no time constraints due to heat stress, meant that this was a desirable way of working whenever possible. However, it would not negate the use of the tablet as information could still be recorded on the hand-held device.

The stated aim of making the most effective use of the limited time in PPE to treat patients has not been quantified. Did the tablet assist the healthcare worker in treating the patients, did the number of patients receiving clinical review or treatment change as a result of the tablet or were the health workers able to achieve the same quality of work in a shorter period of time? I think these would have been important questions to ask, rather than the focus being on the ease of data gathering and transfer, it could have been on quality and availability of patient care, perhaps including patient outcomes.

I found the concept of using radio-frequency identification tags particularly interesting, this would have had great value both for finding and identifying patients for examination, but also importantly for identification and tracking of the deceased. Perhaps because this intervention came during the tail-end of the outbreak there is little discussion of how the tablet could be used for the management of deceased patients, though this was a major issue particularly during the peak of the outbreak when there was a high mortality rate. Unfortunately, the radio-frequency tag aspect of this project did not come to fruition, however a discussion on how this could be moved forward would be worthwhile.

The potential benefits in the efficiency and quality of data collection gained through the use of the electronic tablet system are clear. How this translated into improving patient care seems less apparent. The balance between patient benefit and data for the sake of research and publication remained controversial throughout the Ebola outbreak. Whilst I don’t question the motives of the team behind this innovation, it would be interesting to know how they managed the balance of this innovation being primarily for patient rather than organisational benefit.

The honesty of the authors and implementing team is to be applauded, though deeper discussion on how to move forward would have been of value. This venture came at a high price, and though it was not the success they had hoped for it is certainly not an entirely failed project either. Whilst the authors have identified lessons, it is not clear how they have been learnt. The conclusion would benefit a wider discussion of what next, where this technology can continue to be of use and developed to meet its ultimate aim: saving lives.

The technology is presented as being developed for the specific situation of the EMC, however it would be easily transferable to many humanitarian settings where large numbers of patients or biohazard risks limit the accuracy and efficiency of traditional clinical record keeping. Measles, yellow fever, hepatitis E, cholera and Lassa fever outbreaks would all be well suited to the continued use and development of a potentially very useful field tool. The authors do refer to their vision of the future, and the plan to test a modifiable version of this technology in a nutrition project, a greater emphasis on this vison and how the lessons identified will be learned for the future could have been of use.

As previously stated, humanitarian technology is a rapidly growing and developing phenomenon. The challenges, lessons and future plans identified by the authors are a useful reminder of how best intentions can go askew without the correct planning and inter-agency communication. These lessons are unlikely be unique to this situation, being broadly adaptable. The honesty of the authors on the difficulties in collaborating with Google is commendable, the next step though is who and how they will work with next. MSF has an opportunity to work within and alongside others in the humanitarian technology field, perhaps partners with the experience to assist them in avoiding these common pitfalls again.

I have read this submission. I believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard.

Approved with Reservations

James Whitworth, Referee1 and Hilary Bower, Co-referee2
1Department of Infectious Disease Epidemiology, Faculty of Epidemiology and Population Health, London School of Hygiene and Tropical Medicine, London, UK
2Department of Infectious Disease Epidemiology, Faculty of Epidemiology and Population Health, London School of Hygiene & Tropical Medicine, London, UK
Competing interests: A colleague from LSHTM is acknowledged for his input into the development of this information system. We have had no direct interaction about this project.   HB worked for MSF in Sierra Leone in the same period as the trial but had no involvement in it. She has worked with some of the authors in other locations and contexts.
Review date: 2016 July 11. Status: Approved with Reservations

The authors present a honest evaluation of an operational research project to innovate a new tool for recording, visualising and sharing medical data in response to the challenging needs of a high risk infectious situation in a humanitarian emergency, namely the outbreak of Ebola in West Africa 2014-15.  This is an interesting and valuable paper about developing a clinical information and patient management system based on tablet computers that can be used in Ebola Treatment Centres and similar high risk environments. It is mainly a narrative paper with little in the way of quantitative results. The lessons learned are fairly generic to any project management challenge, and go beyond those of innovation in humanitarian settings.

They describe the process of developing a tablet and new software to respond to the complex requirements of high bio-hazard infection control, and compare this tool with standard practice in this setting. They note the urgency of the project, which was intended to improve clinical care, and by implication outcomes, for patients by helping medical staff make more efficient use of the limited time they could spend in the red zone due to the constraints of personal protective equipment.

The title is appropriate and the abstract is generally clear, although I felt that the description of the comparison of tablet and paper-based data was not clear. It was not just tablet data that were analysed, and what is meant by ‘record “pairs”’ is not clear just from reading the abstract.

The background and methods are mostly clear and appropriate. I am not an expert on computer hardware or software, so cannot comment on the appropriateness of the systems selected. The researchers used a mixed methods approach appropriate for the different aspects of the project they wished to evaluate: namely need identification, buy-in, implementation, tool functionality and maintenance, and a measurement of data quality. They have identified key issues that delayed the process of development and the limited testing of the tool they were able to achieve.

I am astonished at the cost of development of the system, which seems to have been driven largely by the salary costs and person-hours involved, as well as producing customised hardware.

It was not clear to me quite how the data collected in the high risk zone were exported for further analysis and storage. Was this by disinfecting the tablets in chlorine and physically removing them from the high risk zone, or was it through connection some form of local area network that allowed the data collected in the high risk zone to be electronically transmitted outside? This could be made clearer.

I wondered about issues of confidentiality of patient data being transferred from Sierra Leone to the MSF headquarters in Europe (panel 2). How were these addressed, approved and overseen?

I was surprised that no data were collected on clinical management according to panel 3. This seems like a missed opportunity, as further research into optimal management of cases of ebola is still required. There is comment about this at the end of the discussion (page7), but was this an oversight when the system was being developed?

Did this study undergo any ethical scrutiny in Sierra Leone? I would imagine that it should have done, but this is not mentioned.

Methods para 6:

  • Suggest rephrasing the second sentence to make it clear that this part of the paragraph refers to data extraction for the tablet, and the latter to the paper/excel database.
  • Is it a correct understanding that only the first tablet entry of the day was taken into account? If so, could you mention why it was not possible to take account of subsequent entries given the relatively small number of records under analysis
  • Did practitioners entering data in the high risk zone review their own entries once out? Our question refers to the comment in the first para of the discussion about opportunity for correction. It might be useful to include a paragraph on the procedure for using the tablet.

 

The results are very interesting, and it is good to see an honest description of the problems encountered with this project. Their results highlight areas of tension in the process and detail the successes (a functioning tool with good data capture) and failures (implementation too late to have impact on patient care, and lacking one of the initial requirement – a way to find patients in a large facility) of the project. From both they draw out lessons for future innovation between humanitarian and technological enterprises.  They also detail the technical specifications and overview costings for the tool.

I was not clear what was meant by ‘transfer of knowledge of the hardware from Google to MSF did not occur’ (page 5). Further on in the discussion section of the paper there is a comment about the transfer of knowledge, but this appears to be related to the software rather than the hardware. Indeed, it seems that this problem has been overcome and the software has been adapted for use in other emergencies (page 6).

I would have appreciated more information about the ‘significant work needed to clean and analyse the exported data’ from the tablet. Why was that necessary?

The results are generally clearly described, and suggest that the tablet may be at least as good as paper records, though the analysis currently does not appear to take into account the role of chance, and a proportion of records (17% ?) had to be excluded. The qualitative study is quite weak. It appears that different data were collected systematically using the two different systems, so only some of the data were collected using both methods.  Further comments on the quantitative study are:

  • If we understand correctly the agreement between the two methods was calculated by simple percentage agreement: would it be possible to perform a Kappa test to take into account chance?
  • What was the range of interaction pairs per patient?  Was any difference seen in the recording by either method related to the severity of the patient’s illness, or the number of people admitted at the time?
  • Is it possible to state the percentage of missing data overall and/or by recording method and comment on the implications for the agreement results? On rough calculation overall it seems to be 17% (204-(85x2) = 34, 34/204) missing data in one or other system.

 

The agreement between data sources ranged between 69 and 95%. This is rated as ‘high consistency’ but I was not sure on what basis. It was not clear to me, which collection method was felt to be the gold standard, or even if this was determined before the comparison. I sense from the discussion, the tablet was thought to perform better than the paper method, but I am not convinced about this from the results.

There was clearly a tension between MSF wanting a ‘good-enough‘ product, and Google being perfection-driven. I suspect this was not just a question of communication, there is likely to be some underlying differences in corporate culture, which might be hard to bridge even with all the time in the world.

What is meant by ’Google’s haste to move onto the next project’ (page 6). Was this ebola-related, or simply reflecting that this department at Google was busy with a range of different projects?

The discussion was well written and interesting. The authors claim that this is the first such device to be developed. How did they search for other evidence of similar projects? I heard anecdotally of other projects, but am not aware whether they reached fruition, or have ever been published in any form. It would be good to have a thorough systematic search to establish what else has been developed.

Is it correct to assume that the more frequent and detailed data available with the tablet related to the recording of vital signs, which were not entered from the paper record at the time of analysis?

Could you comment on the value gained from the additional encounters recorded in the tablet and whether users found recording real-time information was clinically useful? 

It is not clear to what extent this system can be made available to others. The reference to the consortium being established is to a website that is being prepared.

What data are available from MSF in Amsterdam? Are they data related to the development of this tablet computer based system or the patient data from this Ebola Treatment Centre? This is not clear to me.

In our view, however, the authors present a well-targeted piece of operational research undertaken in challenging circumstances with transparent insights into where the development process fell short. They make it clear that their results are limited, but provide a foundation for moving forward.

We have read this submission. We believe that we have an appropriate level of expertise to confirm that it is of an acceptable scientific standard, however we have significant reservations, as outlined above.


Articles from F1000Research are provided here courtesy of F1000 Research Ltd