|Home | About | Journals | Submit | Contact Us | Français|
Clinical questions are often studied by randomized clinical trials (RCTs) of heterogeneous design. Systematic reviewers and trial designers need to compare the design and results across these trials. If trial information is available in computer processable form, computer-based visualization techniques can provide cognitive support for such comparisons. CTeXplorer offers systematic reviewers and trial designers a tool to better and more quickly understand design heterogeneity in RCTs. CTeXplorer supports dynamic queries on eligibility criteria, interventions, and outcomes in three linked views. We tested CTeXplorer for displaying 12 RCTs on prevention of mother-to-child transmission of HIV. Three target users found the representation and organization of information intuitive and easy to learn. They were able to use CTeXplorer to achieve a quick cognitive overview of a heterogeneous group of RCTs. This work shows the benefit of capturing trial information in computable form. Future work includes leveraging ontologies to enhance CTeXplorer visualizations.
As a consequence of recently enacted laws regulating trial registration and results reporting,1 a high volume of data pertaining to trial design and results will be available. Users will need cognitive support to help them explore and summarize the large amount of data that will be added to the already sizable amount of information published in journal articles. Computer-based visualization techniques from the field of human-computer interaction can provide cognitive support for exploring clinical data. The use of these techniques, however, requires that trial information be available in machine-understandable form.
In the Trial Bank Project, we implemented RCT Bank, a machine understandable repository of RCT design, execution and results information sufficiently detailed to support critical appraisal and meta-analysis.2 We were able to capture a wide diversity of RCTs from different clinical domains into RCT Bank, and we showed how trial-bank reporting can be integrated with traditional journal publishing. 3
We partnered with the National Center for Biomedical Ontology (NCBO)4 as a Driving Biological Project (DBP) to drive the development of semantic technologies for dynamic visualization of data from clinical trials. We focused our research on visualization techniques that would support systematic reviewers, trial designers and clinicians in their analysis of clinical trial data. In the process, we realized that different visualization tools are needed for visualizing the key features of trials (main condition studied, intervention names, and outcome names) and for visualizing detailed trial design features (eligibility criteria, regimen of interventions, and time points of outcome assessment).
Although there are some visualization applications that address the problem of comparing homogeneous collections of clinical data, such as The Cube,5 Simvis,6 and InfoZoom,7 they do not visualize data on trial design. In this paper, we describe the design and evaluation of an interactive visualization tool for exploring clinical trials, called CTeXplorer.
The comparison of trials is a cognitively demanding task that has traditionally required the manual extraction and manipulation of data from journal articles, which complicates cross-trial comparison. CTeXplorer is geared towards satisfying the needs of systematic reviewers and trial designers, who compare the detailed design from multiple clinical trials that are often highly heterogeneous in their eligibility criteria, intervention administration, study outcomes and time points of outcome assessment. We also discuss the implications of our work for semantic technologies that are needed to make customizable and semantically richer visualizations.
We used a convenience sample of 10 RCTs included in a Cochrane review on the use of antiretrovirals for the prevention of mother-to-child transmission (MTCT) of HIV8 and two additional RCTs on the same topic, a set of trials that are particularly heterogeneous in design. We retrieved reports of those 12 RCTs published between 1994 and 2006. We extracted the following information: 1) eligibility criteria and subject characteristics, 2) intervention details, 3) outcomes and results, and 4) methodological features. We defined “use cases”9 that described detailed information needs of systematic reviewers and trial designers.
In the first phase, we focused on views of trial design elements: eligibility criteria, interventions, outcomes and sample size. To expedite prototype development, we extracted the relevant data into comma-separated text files. Not all of the data were extracted and some of the data were adapted. For example, eligibility criteria were reworded into inclusion criteria and, in the case of continuous variables, we standardized the corresponding units of measure. As a result, the data shown are to be considered for demonstration purposes only.
We followed a participatory design10 model and iteratively created the views for effective cognitive support. Participatory design is a model to gather user requirements via iterative collaboration between designer and user. In this research, two of the authors acted as designers and two of the authors played the user role. We implemented different views to support trials from any clinical domain. We then conducted a qualitative evaluation with three target users to assess usability and to brainstorm on tool design.
The participants used CTeXplorer to individually complete a set of pilot-tested tasks and a questionnaire. A post-study focus group session took place with the participants and the design team.
The findings from the usability evaluation came from analyzing our observations, the answers to the tasks in the usability testing, the answers from the questionnaire and the input received through the focus group.
CTeXplorer is a Java standalone implementation that organizes information from multiple trials into three views on the screen (Figure 1): eligibility rules at A, intervention details at C, and study outcomes at D. Users can see at a glance: relative trial sizes, which drugs were administered in which trials and during which trial period (pregnancy, labor, postpartum in the mother and shortly after birth in the infant), and the time points when maternal and infant safety and infant HIV infection were assessed.
The Eligibility Criteria view (see A) displays eight criteria by default, which constitute a partial list of the available set. Eligibility criteria involving continuous variables are represented using glyphs that describe open or closed intervals with end-points included or excluded. Eligibility criteria involving dichotomous variables are represented using either a filled or an empty circle, indicating that the criterion must be satisfied or must not be satisfied, respectively. All criteria are represented as inclusion criteria. One way the Eligibility Criteria view can be changed is by accessing a list that allows the user to control which criteria are displayed (Figure 2).
Similarly, a list allows the user to select the trials to be displayed. By default, all the trials are listed in order of publication date of the article where the main results were published. The circles at B are representing the trial size and hovering over them displays a tooltip with the actual sample size.
Users can dynamically query information in one view and automatic updating occurs in the other views. For example, appropriately positioning the double-ended slider over the age criterion, allows the user to select trials that could enroll women less than 18 years old. The set of selected trials includes those in which the criterion was not reported. The user can choose to have the excluded trials grayed out (i.e., filter disabled) or removed from the display (as in Figure 3, where the setting is filter enabled). The same trials are removed from the other two views.
On the Interventions view (at C in Figure 1), check boxes allow the user to control which trials to display based on the interventions administered in them. Trial arms in which more than one drug was administered are displayed using a diagonally shaded bar. A double asterisk next to the arm label indicates that the arm was stopped early.
The Outcomes view uses a scatter plot to show which outcome was assessed at which time point. Each outcome is represented by a colored glyph. The user can toggle between viewing efficacy and safety outcomes (Figure 1 at D and Figure 4).
The user can also toggle between viewing eligibility criteria and the geographic distribution of trial sites across the top (Figure 4). The Geographical Distribution view is static, i.e., it is not dynamically updated based on the user's selection of trials.
We designed CTeXplorer for clinical trials’ researchers with minimal experience with computer applications. We conducted an exploratory study in an early stage of the development to confirm the usefulness of such a tool and to anticipate potential usability issues. We successfully recruited 3 users with different backgrounds: an HIV trialist, a general internist/systematic reviewer, and a Cochrane editor.
The participants were assigned to individual sessions, and portable usability lab equipment was used to video record the participant’s face, voice and interactions with the tool. Handwritten notes were also taken by a second interviewer. Participants were given a brief orientation on the functionality of CTeXplorer and a brief description of the trials. They were then asked to carry out a set of 13 tasks (defined in terms of our ‘use cases’) and were encouraged to ‘think aloud’ while performing these tasks. The set of tasks included finding the range of the actual sample size across trials, the most common pregnancy stage of participants, and the two trials most similar with respect to treatment. The tasks were designed so that they would require a participant to search for specific information, to compare data across trials, to find relations among trials, and to find gaps in the information.
After completing the tasks, each participant filled out a questionnaire. Six questions were asked to identify the level of motivation, satisfaction and confidence of the participant using CTeXplorer. A Likert scale with seven levels, ranging from “strongly disagree” to “strongly agree,” was used for the answers. For example: 1) My experience using the tool was very satisfying, and 2) I found the organization of data appropriate for the domain. Eight open-ended style questions were used to identify insights that our visualizations might have provided to the user, such as: 1) Could you describe a new trial that may fill some gap in the existing trials? and 2) Did the tool allow you to detect similarities and differences?
The users also participated in a focus group with the design team. Questions were asked to determine if the participants’ requirements were covered by CTeXplorer. Examples of questions asked included: 1) How do you currently explore information on clinical trials? 2) Did you find the tasks you were asked to perform representative of the kinds of tasks you would normally do when exploring clinical trials data? and 3) Describe your experience of understanding the trials by using CTeXplorer.
The findings from our case study came from analyzing our observations, the answers to the tasks in the usability testing, the answers from the questionnaire and the input received through the focus group. The users found some minor usability issues with all three views, such as choice of colour, in addition to the following observations grouped by categories:
One user suggested displaying just one criterion at a time with the option to cycle through all of them by clicking <next> rather than presenting a list for selecting/deselecting criteria.
It was suggested by one user that another view is needed to represent the total number of people that received each drug (How many people took drug A?). Two users required that the size of the arm from each trial should be also available.
The questionnaire revealed that the three users were motivated to use the tool, (e.g., “I am very impressed and see practical applications”, “This tool can revolutionize how we look at things”). They said they felt confident when using the tool and felt the system was somewhat easy to use and learn (e.g. “I felt surprisingly comfortable”). The three users thought that the organization of data was appropriated for the domain.
During the individual sessions, two participants completed the 13 tasks correctly. The third participant completed 10 out of 13 tasks, seven tasks correctly and three tasks incorrectly. Why this participant did not achieve a clear understanding of the displayed data was not clear. One reason may be that the participant was rushed due to a pending appointment. Another possible reason for misinterpretation of the visualizations might be the screen size of some of the view elements as the participant’s body language suggested difficulties in viewing data on the screen.
Suggested improvements included: the visualization of trial results closely with trial design, the ability to share their understanding of the trials with other users, the ability to rank and annotate trials on trial quality, the ability to save views, the ability to export and import data to and from other tools (especially Excel), the ability to cluster trials by similarities, and the possibility of highlighting hypotheses not tested by the trials.
Our user evaluation results show that CTeXplorer is a good start for providing cognitive support for systematic reviewers and trial designers who are performing detailed evaluation and comparison of related but heterogeneous trials.
We acknowledge that the generalizability of our evaluation results is limited due to the small number of participants in our user study and the potential bias we have introduced from our interpretation of their feedback. This limitation will be mitigated in future studies by making the tool available to a larger group of participants. A future deployment will also include some of the suggested enhancements.
We have migrated the standalone version of CTeXplorer to a Web-based version implemented in Flex11 (an Adobe platform for the development of Rich Internet Applications). This version of the tool takes input data conforming to an XML DTD. Users can upload their own set of trials in XML files for visualization in the Web-based CTeXplorer.12
To complement CTeXplorer for summarized comparisons of trial design and results, we are extending the current visualizations to include a different interface called CTSearch for browsing, filtering and pruning search results comprising large numbers of trials on main condition, interventions, and outcomes.
This architecture will allow us to use NCBO technology to provide semantically richer visualizations. For example, we plan to call NCBO Web services to annotate terms (e.g., condition name) to a UMLS concept unique identifier (CUI). Clinical trial queries can be executed and the search results can be dynamically displayed conveying the semantic level added by the annotated terms. CTeXplorer provides us with a user interface of demonstrated acceptability in which to introduce semantically-driven customization.
We have shown that machine understandable trial information can drive an effective tool for the visualization of heterogeneity in RCT designs. Target users were able to easily learn CTeXplorer and use it to achieve a rapid cognitive overview of a heterogeneous group of RCTs on MTCT prevention of HIV. Current work includes incorporating ontologies and NCBO semantic Web services to provide more clinically sophisticated dynamic visualization of both generic and detailed clinical trial design information.
This research was funded by the National Institutes of Health through the NIH Roadmap for Medical Research, Grant U54 HG004028.