Site capability analysis
As part of our process, we needed to analyze the current electronic data capabilities of each hospital. In particular, we needed to identify the particular sources of the lab, medical record, and data warehouse systems from our six contributing institutions. The site capabilities are shown in . The site capabilities include vendor and in-house developed systems.
Electronic data sources for the PHIS+ hospitals. The “PHIS+ Lab Datasource” column refers to the source of information for lab data extracted for PHIS+.
One of the first steps in our process was to select the lab tests to include in the PHIS+ repository. Hypothesizing that there would be wide variation between the sites in lab tests performed, we decided to utilize lab order information already stored in PHIS to determine the overall prevalence of specific tests across the sites and then use this to prioritize labs for PHIS+. CHCA created an ordered frequency list of PHIS inpatient (including ED and ambulatory surgery) lab orders for 2009 (the most recent complete year of PHIS data). Microbiology, pathology and cytogenetics tests were excluded from this phase. The resulting list contained 497 lab orders. (Items were grouped by test order type and did not differentiate by specimen type or other attributes that might normally differentiate lab orders.) Note that PHIS stores lab information by order while the goal of PHIS+ is to obtain the results of these orders. Since a set of results may come from one order (for example a CBC panel test will produce multiple individual results such as WBC, Hemoglobin, Hematocrit, etc) there is not a 1:1 relationship between orders and results. The OC debated how to handle panel tests in the list, knowing that the panel content could be quite different between the sites. It was agreed that we would investigate the top ten individual (vs. panel) test orders on the list as an initial “proving ground” for the PHIS+ data federation process. The list of the top ten tests is shown in . Each test corresponds to a single analyte that can be measured in a variety of ways. Because of this, the OC further suggested that our initial data collection should be for all results on these particular analytes, regardless of ordering method, lab method or specimen, in order to ensure that we have near-complete coverage for a given analyte. For example, tests for Sodium could include serum/plasma, blood, urine or other body fluid specimens, and might also include point-in-time and 24-hour collections. The hospitals were directed to find all possible tests performed at their sites for each analyte and report these as separate test result items.
Top ten PHIS individual lab test orders across the 6 hospitals.
The OC also wanted to include two panel tests, CBC with differential and Complete Urinalysis, in order to examine variability among the sites in panel content and to further test the data federation capabilities of our system. Many of the analyte tests that are commonly included in these panels also appeared high in our lab order frequency list. The hospitals were asked to find all local examples of how they ordered these panels and then provide the complete list of individual analyte tests that made up these panels, again regardless of test methodology or specimen. The combination of the top ten tests in and the additional tests from the two panels was denoted as “Lab Sample 1”. Selection of a larger set of tests for the first operational phase of PHIS+ was deferred until after analysis of the Lab Sample 1 results.
The process of mapping local terminology and data models to FURTHeR’s terminology and data model requires knowledge about how local systems store their data. This local metadata must be discovered and shared in order to create the mapping content necessary to semantically and syntactically integrate/federate data. There are two types of metadata we required for the lab data: metadata to map local test result codes, and metadata to map local lab result instance formats.
As part of FURTHeR’s regular process for lab test result code mapping, BMIC uses the Regenstrief LOINC Mapping Assistant (RELMA)8,9
. RELMA is particularly useful for automated batch processing of large test code files. It operates best when a specific set of information (metadata) is included with each test code, and when this information is presented in a standard format. Following published directions for RELMA, BMIC developed a metadata specification for the lab result test codes in Lab Sample 1. A partial list of the metadata fields and their descriptions is shown in . The list also included metadata that could be used for interpreting certain result instance field formats, such as date field formats, but most of these were later removed for reasons explained below. The metadata fields were reviewed in phone meetings with the OC and ITC. Each of the sites confirmed the local availability of required fields and most of the optional fields.
A subset of the Lab Sample 1 metadata fields and their descriptions.
For the lab result instance formats from the sites, we originally proposed that the sites would be able to provide their lab sample data in a format that was most suitable to their data export capabilities. FURTHeR is able to process input information in a variety of formats, and our goal was to lessen the burden on the sites of configuring data in an unfamiliar manner. But, due to design changes that required that data flow through CHCA first and then to BMIC, one common data format was agreed upon. The common format is discussed in more detail in the Lab Data Collection section below.
A Microsoft Excel spreadsheet template was created with columns for each of the metadata fields. This template was given to each of the hospitals as a metadata collection tool, along with field description information, a metadata example file using the template, and an instruction sheet for the collection process.
Over a period of several weeks, the hospitals collected and began to send their metadata spreadsheets via email to BMIC. (Email was considered safe for this task since no patient identifying information or hospital sensitive information was included.) Some sites struggled to some degree with discovering the metadata due to a lack of available technical personnel or a more complicated IT infrastructure that made metadata discovery more difficult because the information was spread over more than one location (e.g., separation of specimen information from test results), making it harder to join information into the metadata spreadsheet. This led more than one site to submit information in a non-standard format.
We initially estimated two weeks for the sites to complete their metadata collection, but some sites needed more time. Initial review of metadata spreadsheets also revealed some problems with the data (besides the template format changes mentioned previously), including misinterpretation of fields, incorrect formatting of data, and missing data. An iterative process with the hospitals corrected the errors and the initial metadata collection was completed in approximately one month.
BMIC began processing each of the metadata files as they were received. Any non-standard reporting formats (as reported above) in the metadata spreadsheets were corrected first. One of the authors (RG), a Master’s trained informaticist with a medical background, then used the RELMA tool to map the local test codes for each site to a corresponding LOINC code. RELMA provided an output file with the local and LOINC code pairs, which were then loaded into the FURTHeR terminology server. Each site received its own namespace in the terminology server in order to separate the local codes and provide more efficient code maintenance. In addition, the LOINC mappings, including LOINC code and descriptive LOINC name, were combined with the original metadata spreadsheets from the sites, along with comments/questions about the metadata and LOINC interpretations where appropriate, creating a “return file” for each site. From these return files, a master file was created that listed, side-by-side, all of the local test codes and corresponding LOINC codes, grouped and sorted by LOINC code. Our terminology mapping expert (RG) was able to process approximately 200 test codes per day using this process.
The site codes for the Test Value (when coded), Units of Measure, and Interpretation Code fields (see ) were also added to the terminology server in the site namespaces and mapped to appropriate standard terminologies.
Some of the local tests could not be mapped to LOINC due to ambiguities and missing data in the metadata. We used the return file spreadsheets in an iterative process with the sites in order to address these inconsistencies. For example, for some test metadata the specimen or method was not clear, and multiple LOINC code mappings were possible. We also found potential errors with reported units and reference ranges. In these cases, we asked the sites to provide additional metadata information about the tests. This iterative process continued for approximately 2–3 months, including extending into the lab data collection phase when actual lab results indicated missing codes and erroneous mappings. In the end, we were able to map each local test to an unambiguous LOINC code. (Some local tests were discarded because they were not part of the Lab Sample 1 list.)
An area that required special attention was where tests were mapped to a non-specific body fluid type, which might have consequences on research where more specific specimens are expected. The original LOINC mappings were re-examined at the request of the OC to ensure accurate mappings. Several tests were changed to more specific specimen tests by this process, but others remained with a generic specimen because the sites allowed this.
A beneficial consequence of the PHIS+ mapping was in some cases to confirm local LOINC mappings and in other cases to correct erroneous maps. One of the sites provided its own LOINC mappings for all its local test codes, and BMIC’s mapping of the site’s metadata largely agreed with these LOINC mappings. Discrepancies were primarily due to ambiguous metadata that were resolved during the iterative cleanup process. Three other sites provided some LOINC mappings for their local codes. Discrepancies found with these mappings were again often due to ambiguous metadata. Others were actual erroneous mappings by the sites. We also discovered some instances where a site’s LOINC mapping was accurate when it was originally made, but the test had changed over time and the LOINC mapping was no longer correct. Finally, for those sites that had not yet implemented LOINC coding, this process gave them valuable information to begin the transition to LOINC usage. In the future, we anticipate that the sites will assume the LOINC mapping process, and a simpler QA process will be performed for PHIS+.
shows the results of the metadata processing, with the sites anonymized. The number of mapped, unique tests for each site is shown. A single LOINC code may map to more than one test at a site. Across all the sites, 435 unique LOINC codes were identified, which mapped to 959 total site tests. The vast majority of LOINC codes (58%) are used at only one site; 19% are used at two sites, and 8% at three sites. But 15% (68) of the LOINC codes are used at four or more of the sites. Nine of the top ten tests in were covered by at least five of the sites for serum/plasma specimens: lack of total agreement was due to a site reporting a different specimen for its test. The anomaly in the top ten list was Urea Nitrogen: more variability exists in specimen and method among the sites for this test. In addition to the top ten tests, all six sites perform common Hematocrit, Urine Leukocyte Esterase, Urine pH, Urine Reducing Substance, and Hemoglobin tests.
Metadata results from the six hospitals showing the number of unique local tests from each site and corresponding number of linked LOINC codes, as well as other mapped codes. (Sites have been anonymized.)
The differences in the number of local tests between the sites performing the most (Sites A & B) vs. least (Site F) tests are partly due to Sites A & B more often reporting multiple tests that map to the same LOINC code (i.e. multiple local methods to obtain the same result). But Sites A & B still listed far more tests than the other sites, mostly due to unique tests reported as part of CBC and Complete Urinalysis panels.
In addition to the LOINC Codes, 34 SNOMED codes for Units of Measure were linked to 101 local Units; 9 HL7 Interpretation Codes were linked to 35 local codes; and 11 SNOMED Specimen codes were linked to 25 local codes.
Lab Data Collection
Once the metadata issues were largely resolved, we began the task of collecting lab data from the sites. As mentioned earlier, the project participants decided to use a common format for reporting the lab result data for Lab Sample 1. For simplicity, we developed a field-delimited text file format that had characteristics similar to HL7 v2.x message syntax10
, including the use of a pipe delimiter between data fields. The required fields were derived from the metadata in as well as additional fields necessary for the lab result instance data. For example, we added Patient ID and Billing Number fields in order to join the lab data with the existing administrative data in PHIS. We also specified a sequence number on each row of data so that we could QA our results with the sites and resolve issues. The format and fields for the lab sample file are shown in , along with an example lab result. (The Hospital_Number and Campus_ID fields correspond to specific site identifiers issued previously by CHCA for PHIS.) Each row in the file ends with a carriage return. Descriptions of the file format, field contents, and example lab results were provided to the sites. The sites were instructed to compile results from the year 2009 for all the tests mapped in their corresponding metadata return files.
Lab data format for Lab Sample 1. The example shows an abnormal Hemoglobin result from a CBC panel. In the example, the End_Date_Time, Ref_Range_Low, Ref_Range_Hi, and Comments fields are empty.
Because of the issues discussed previously with data use agreements and IRB protocols, the sites were instructed to create a de-identified copy of their lab sample files. Patient ID and Billing Number fields were left blank, and all date/time fields were set to the same value (“January 1, 2009, 9:00 am”). The de-identified files could then be sent to BMIC. (The scrubbed fields were not necessary for the data translation process.) Once the data use agreements and IRB protocols were agreed to, the translated data from the de-identified files could be joined with the information in the original (identified) files via the Sequence_Number field.
We had previously established a secure FTP transmission process for sending the lab sample files from the sites to CHCA. We retained this process for the de-identified files so that the sites would have to implement only one communication method for PHIS+. CHCA received the de-identified files, performed a simple QA to verify that identifying fields were scrubbed, and then forwarded the files to a secure FTP server at BMIC. BMIC then moved the files to a secure local server for data translation.
Lab Data Processing
As discussed previously, BMIC used a modified version of FURTHeR to process the lab sample files. One of the authors (OEL), a senior software engineer on the FURTHeR project, created a data file adapter that could read the pipe-delimited results from the sample files and feed the lab records to the FURTHeR translation engine. A simple command line interface initiated the process by pointing the data file adapter to the correct sample file and configuration file, and invoking the FURTHeR application. The translation engine marshaled the raw lab data into the FURTHeR lab object and translated all local codes to the standard terminologies (using the code associations in the terminology server, described earlier). Unrecognized codes and malformed input data were flagged to a log file for manual review. An output adapter, also developed by OEL for PHIS+, took each translated lab result and inserted it into a MySQL database via a Java Hibernate object11
. (The Hibernate persistence layer can easily be reconfigured to support any other JDBC compliant database.) The fields in the database mimicked those in the lab sample data file (). We included columns for the original field values as well as for the translated values for QA purposes. We also added fields for auditing purposes (insertion date/time, status).
Initial translation performance of the lab files was poor, taking up to several hours to process an entire file on a desktop workstation. FURTHeR does not typically handle input data sets as large as the lab sample files. Tuning enhancements were made by OEL and PM, including local caching of common terminology mappings, additional CPU RAM and database tuning. These enhancements significantly improved performance of the system: approximately 4,000 results per second could be translated on a desktop workstation. The enhancements were also added to the standard FURTHeR code base, providing a beneficial speed-up in FURTHeR queries.
Some issues with the sample file formats were identified during processing. Some sites had inserted quotation marks around field values or around entire row results. While it would have been relatively easy to correct for this in the data file adapter, we asked the sites to correct these errors and retransmit their files in order to maintain a consistent file format across the sites. One of the sites inserted additional, unanticipated fields into its sample file. The site was asked to remove these fields and retransmit its file.
Several of the sites sent unknown test codes, units of measure, and interpretation codes that were caught during the terminology translation process. Consultation with the sites resolved these anomalies, usually resulting in modifications/additions to a site’s metadata and/or namespace terminology. In certain cases, the errors were traced to errors in the source systems or in the data collection at the site.
Lab Data Results
shows the results from the lab data processing, with the sites anonymized as in . The number of results from each site for all the tests identified from a site’s metadata for the year 2009 is shown. The most common lab tests across the sites are measurements normally performed as part of CBC, electrolyte and urinalysis panels.
Lab data result counts from the six hospitals. (Sites have been anonymized.)
Of the 435 total LOINC test codes originally returned by the sites in their metadata files, results were returned for only 392 LOINC codes. 24 LOINC tests covered 50% of all the tests reports; 141 tests covered 95% of all the results reported. Comparing the LOINC results from and , three sites had fewer actual LOINC tests in their data files than were specified in their metadata. This is because the metadata they originally supplied represented possible tests the site could perform during the sample period even though these tests were not performed in 2009. Site E reported far fewer tests than the other sites because they transitioned to a new EMR in 2009 and did not have the entire year’s labs converted to the new EMR format at the time of the data extract.