Our analysis of 1000 eligibility criteria randomly drawn from ClinicalTrials.gov
demonstrates significant semantic and clinical variability across criteria. This variability presents challenges to informaticians, researchers, and clinicians. The good news for informaticians designing expression languages is that 23% of C&S criteria are simple criteria, or can be reduced to simple criteria through Boolean, exclusion, and if-then decomposition. On the other hand, 77% of C&S criteria remain complex to evaluate as they contain one or more of the following patterns: 9% of complex criteria involve the use of semantic connectors which are not captured by the current representation languages or coded data, 40% require the definition of temporal constraints (many of which are only loosely specified), 19% require clinical judgment, and 24% require linkage to study metadata. Researchers trying to determine patient eligibility for studies face incomprehensible and ambiguous criteria as well as under-specified criteria requiring clinical judgment or assessments. Furthermore, 7 % of criteria require radiographic, histologic, or EKG data that may not be available in coded format in the EHR. Automation of screening based on these criteria may require natural language processing of narrative documents, with attendant problems of sensitivity and specificity. Clinicians seeking to determine if a study’s population is similar to their own patients are equally challenged to understand just what clinical phenotype was studied in a given trial.
In this work, we conceptualize the clinical domain referenced in eligibility criteria to be orthogonal to the criteria’s form (e.g., Boolean combination) and data sources (e.g., laboratory test results). We aimed to classify criteria in a manner that would be useful across clinical domains, to drive development of domain-independent eligibility criteria representations that will allow for tools and discoveries made in one domain to be applied to all areas of medicine, including facilitating the automation of matching potential subjects to trials, designing trials to expand on findings from prior studies, and pooling data across studies for meta-analysis. A disease-specific representation format based on data elements and their values, such as that of the ASPIRE project, cannot satisfy the expressivity requirements revealed in this study. Domain-independent languages with composition capability, such as Arden Syntax, SAGE and GELLO, do not have this limitation.
Additionally, criteria requiring referencing external sources, and those based on ill-defined criteria provide obstacles to the automation of clinical tasks across medical domains. Such criteria have to be disambiguated, de-abstracted, and operationalized in terms of available data. Work on these issues will benefit from an interdisciplinary approach in designing ontologies and data structures that allow access not only to current sources of quantitative laboratory data, but also to high-throughput and imaging data that will one day be used regularly in clinical practice.
One strength of our study is that we analyzed eligibility criteria without restriction to clinical domain. This method likely produced a wider range of variance in criteria than if we had restricted our analysis to specific domains, but is more reflective of the true range of complexity in eligibility criteria. Researchers in a specific field may use criterion patterns that are not used in other fields, but we abstracted common semantic patterns and our categories covered all patterns seen in our sample of 1000 criteria.
A limitation of this study is that all of the criteria we analyzed were taken from ClinicalTrials.gov
, which may produce a bias towards criteria from the types of trials found in this database, e.g., more quantitative than qualitative. Other biases may include more well formed eligibility criteria because trials registered in ClinicalTrials.gov
may be of higher study design quality than unregistered trials. Finally, eligibility criteria reported to ClinicalTrials.gov
may be simplified versions of more detailed eligibility criteria in actual study protocols. Our categorization of complexity and clinical and semantic patterns will need to be tested against criteria extracted directly from study protocols to demonstrate its full applicability.
Given the diversity of these criteria, including incomprehensible and ill defined criteria, it may well be advantageous to consider the formation of clear standards for clinical researchers to follow when writing eligibility criteria. Our work provides an initial framework for considering best practices for expressing the clinical and semantic content of criteria. While best practice criteria may be viewed as initially burdensome, we believe that the benefits of having more clearly written criteria will far outweigh the costs, and will facilitate formalization and optimal use of these criteria as specific phenotypes throughout clinical research and care across all domains of medicine.