Data Source: We used a convenience sample of 10 RCTs included in a Cochrane review on the use of antiretrovirals for the prevention of mother-to-child transmission (MTCT) of HIV
8 and two additional RCTs on the same topic, a set of trials that are particularly heterogeneous in design. We retrieved reports of those 12 RCTs published between 1994 and 2006. We extracted the following information: 1) eligibility criteria and subject characteristics, 2) intervention details, 3) outcomes and results, and 4) methodological features. We defined “use cases”
9 that described detailed information needs of systematic reviewers and trial designers.
In the first phase, we focused on views of trial design elements: eligibility criteria, interventions, outcomes and sample size. To expedite prototype development, we extracted the relevant data into comma-separated text files. Not all of the data were extracted and some of the data were adapted. For example, eligibility criteria were reworded into inclusion criteria and, in the case of continuous variables, we standardized the corresponding units of measure. As a result, the data shown are to be considered for demonstration purposes only.
We followed a participatory design
10 model and iteratively created the views for effective cognitive support. Participatory design is a model to gather user requirements via iterative collaboration between designer and user. In this research, two of the authors acted as designers and two of the authors played the user role. We implemented different views to support trials from any clinical domain. We then conducted a qualitative evaluation with three target users to assess usability and to brainstorm on tool design.
The participants used CTeXplorer to individually complete a set of pilot-tested tasks and a questionnaire. A post-study focus group session took place with the participants and the design team.
The findings from the usability evaluation came from analyzing our observations, the answers to the tasks in the usability testing, the answers from the questionnaire and the input received through the focus group.
Tool Description: CTeXplorer is a Java standalone implementation that organizes information from multiple trials into three views on the screen (): eligibility rules at A, intervention details at C, and study outcomes at D. Users can see at a glance: relative trial sizes, which drugs were administered in which trials and during which trial period (pregnancy, labor, postpartum in the mother and shortly after birth in the infant), and the time points when maternal and infant safety and infant HIV infection were assessed.
The Eligibility Criteria view (see A) displays eight criteria by default, which constitute a partial list of the available set. Eligibility criteria involving continuous variables are represented using glyphs that describe open or closed intervals with end-points included or excluded. Eligibility criteria involving dichotomous variables are represented using either a filled or an empty circle, indicating that the criterion must be satisfied or must not be satisfied, respectively. All criteria are represented as inclusion criteria. One way the Eligibility Criteria view can be changed is by accessing a list that allows the user to control which criteria are displayed ().
Similarly, a list allows the user to select the trials to be displayed. By default, all the trials are listed in order of publication date of the article where the main results were published. The circles at B are representing the trial size and hovering over them displays a tooltip with the actual sample size.
Users can dynamically query information in one view and automatic updating occurs in the other views. For example, appropriately positioning the double-ended slider over the age criterion, allows the user to select trials that could enroll women less than 18 years old. The set of selected trials includes those in which the criterion was not reported. The user can choose to have the excluded trials grayed out (i.e., filter disabled) or removed from the display (as in , where the setting is filter enabled). The same trials are removed from the other two views.
On the Interventions view (at C in ), check boxes allow the user to control which trials to display based on the interventions administered in them. Trial arms in which more than one drug was administered are displayed using a diagonally shaded bar. A double asterisk next to the arm label indicates that the arm was stopped early.
The Outcomes view uses a scatter plot to show which outcome was assessed at which time point. Each outcome is represented by a colored glyph. The user can toggle between viewing efficacy and safety outcomes ( at D and ).
The user can also toggle between viewing eligibility criteria and the geographic distribution of trial sites across the top (). The Geographical Distribution view is static, i.e., it is not dynamically updated based on the user's selection of trials.
User Study: We designed CTeXplorer for clinical trials’ researchers with minimal experience with computer applications. We conducted an exploratory study in an early stage of the development to confirm the usefulness of such a tool and to anticipate potential usability issues. We successfully recruited 3 users with different backgrounds: an HIV trialist, a general internist/systematic reviewer, and a Cochrane editor.
The participants were assigned to individual sessions, and portable usability lab equipment was used to video record the participant’s face, voice and interactions with the tool. Handwritten notes were also taken by a second interviewer. Participants were given a brief orientation on the functionality of CTeXplorer and a brief description of the trials. They were then asked to carry out a set of 13 tasks (defined in terms of our ‘use cases’) and were encouraged to ‘think aloud’ while performing these tasks. The set of tasks included finding the range of the actual sample size across trials, the most common pregnancy stage of participants, and the two trials most similar with respect to treatment. The tasks were designed so that they would require a participant to search for specific information, to compare data across trials, to find relations among trials, and to find gaps in the information.
After completing the tasks, each participant filled out a questionnaire. Six questions were asked to identify the level of motivation, satisfaction and confidence of the participant using CTeXplorer. A Likert scale with seven levels, ranging from “strongly disagree” to “strongly agree,” was used for the answers. For example: 1) My experience using the tool was very satisfying, and 2) I found the organization of data appropriate for the domain. Eight open-ended style questions were used to identify insights that our visualizations might have provided to the user, such as: 1) Could you describe a new trial that may fill some gap in the existing trials? and 2) Did the tool allow you to detect similarities and differences?
The users also participated in a focus group with the design team. Questions were asked to determine if the participants’ requirements were covered by CTeXplorer. Examples of questions asked included: 1) How do you currently explore information on clinical trials? 2) Did you find the tasks you were asked to perform representative of the kinds of tasks you would normally do when exploring clinical trials data? and 3) Describe your experience of understanding the trials by using CTeXplorer.
Results: The findings from our case study came from analyzing our observations, the answers to the tasks in the usability testing, the answers from the questionnaire and the input received through the focus group. The users found some minor usability issues with all three views, such as choice of colour, in addition to the following observations grouped by categories:
Eligibility Criteria View One user suggested displaying just one criterion at a time with the option to cycle through all of them by clicking <next> rather than presenting a list for selecting/deselecting criteria.
Interventions View It was suggested by one user that another view is needed to represent the total number of people that received each drug (How many people took drug A?). Two users required that the size of the arm from each trial should be also available.
User Satisfaction The questionnaire revealed that the three users were motivated to use the tool, (e.g., “I am very impressed and see practical applications”, “This tool can revolutionize how we look at things”). They said they felt confident when using the tool and felt the system was somewhat easy to use and learn (e.g. “I felt surprisingly comfortable”). The three users thought that the organization of data was appropriated for the domain.
During the individual sessions, two participants completed the 13 tasks correctly. The third participant completed 10 out of 13 tasks, seven tasks correctly and three tasks incorrectly. Why this participant did not achieve a clear understanding of the displayed data was not clear. One reason may be that the participant was rushed due to a pending appointment. Another possible reason for misinterpretation of the visualizations might be the screen size of some of the view elements as the participant’s body language suggested difficulties in viewing data on the screen.
Suggested improvements included: the visualization of trial results closely with trial design, the ability to share their understanding of the trials with other users, the ability to rank and annotate trials on trial quality, the ability to save views, the ability to export and import data to and from other tools (especially Excel), the ability to cluster trials by similarities, and the possibility of highlighting hypotheses not tested by the trials.
This article has been 
