Unlike other dedicated meta-analysis packages, Meta-Analyst integrates the capabilities to perform meta-analyses of binary or continuous outcomes and diagnostic or prognostic tests, combining the functionality of software such as MIX and Meta-DiSc. For each of these types of outcomes, we have implemented standard meta-analysis routines, as well as some more advanced ones. Table summarizes the analyses Meta-Analyst can currently perform.
Methods available in Meta-Analyst
Currently (as of version Beta 3.1) Meta-Analyst implements only one Bayesian model for each type of data (binary, continuous and diagnostic; for model details see http://tuftscaes.org/meta_analyst/AppendixA.html
, last accessed 11/12/2009). Because of the way we have interfaced Meta-Analyst with OpenBUGS (Figure ) we can easily add additional models.
Exploratory and sensitivity analyses
Meta-Analyst automates cumulative meta-analysis
, leave-one-out sensitivity analysis
and subgroup analysis
. In cumulative meta-analysis one reorders the studies according to a covariate (e.g., increasing year of publication) and re-estimates the summary effect at each step, i.e., each time a new study is added. It is typically a graphical analysis that plots the aggregate overall estimate at each step [17
]. This elucidates the evolution, or pattern, of evidence over time. Leave-one-out analyses explore the influence of individual studies as follows: If there are n
studies in the meta-analysis at hand, plot n
summary estimates, each corresponding to leaving one of the n
studies out of the calculation. This plot illuminates influential studies, as when they are left out of the analysis, the overall estimate will be notably perturbed. Subgroup analysis is a tool for exploring the effects of a treatment on population subgroups, e.g., females older than 50 years old versus younger women. This is done by conducting separate meta-analyses on the respective subgroups and plotting overall estimates for both.
Meta-Analyst generates different graphical output suitable to the data at hand. Figures and summarize the plots available.
Plots available in Meta-Analyst.
Plots available in Meta-Analyst.
The GUI comprises two tabs; one for data entry and editing and the other for displaying the results of analyses (see Figure ). The help panel on the bottom of the tab is always available and provides context-specific explanations and instructions for the user. The main data manipulation tool is a spreadsheet with a standard data-entry interface. Data can either be entered by hand, or imported from Excel (xls) or Comma Separated Value (csv) files via an import 'wizard'. Meta-Analyst uses its own custom data file format to save data, which bundles comma-separated study data with some meta-data (for example, data type, covariate names, etc.) about the meta-analysis.
Screenshot of the Data Entry screen in Meta-Analyst.
While editing data, the outcome and corresponding confidence intervals are updated dynamically. The outcome metric can be changed via a right-click menu (e.g., from odds ratio to risk difference) and the outcomes will be re-computed automatically. While Figure shows binary data being manipulated, the interface is analogous for continuous and diagnostic data.
The user can also provide additional numeric and string variables that describe characteristics of the analyzed studies. By convention, user-added numeric variables are termed covariates. Typically, covariates are used as explanatory variables in meta-regression analyses. To use covariates in the analyses a user has to activate them (by right clicking on the corresponding covariate name and choosing the respective option). When at least one covariate is activated, fixed and random effects meta-regression becomes available as an analytic option in the program's menus. If the covariate is excluded from the analyses, meta-regression is not available as an option. To perform meta-regression with several explanatory variables, the user simply activates the corresponding covariates.
User-added string variables are termed labels and are typically used to provide textual descriptions, or to specify subgroups for subgroup analyses. When a dataset contains at least one label, the program allows the user to perform subgroup analyses according to the categories defined by the selected label. The subgroups are automatically named according to the contents of the label. Labels are ignored in meta-regression analyses (though displayed in plots when pertinent). For example, suppose the user adds a label "country". Further suppose that studies 1 and 2 are labelled "United States" while studies 3 and 4 are labelled "India". Then a subgroup analysis performed using the "country" label will automatically plot the overall (pooled) effects for the studies that were labelled as being conducted in "India" (1 and 2) and the pooled estimate will also be plotted for those labelled as being conducted in the "United States" (3 and 4).
Studies can be included and excluded from a particular analysis by selecting/deselecting the corresponding checkbox in the first column. Once the data is entered, the outcome metric set and the studies and covariates desired to be excluded from the analysis (if any) are deselected, users can perform an analysis via the drop down 'Analysis' menu, at which point they will be prompted with the dialogue shown in Figure . Here users can pick the model to be used in the analysis, and specify the parameters for the selected model. The program provides editable default values for many of the options.
After the analysis runs to completion, the results will be displayed in the results tab (Figure ).
The left-hand side of the results tab shows a tree populated with collapsible parent nodes for each analysis that has been run ("Analysis 1" and "Analysis 2", in the figure). Each of these parent nodes have child nodes corresponding to the various tabular and graphical outputs associated with the analysis. Clicking on one of these child nodes, e.g., "Forest Plot", scrolls the corresponding graphic into view.
All of the tables and graphics can be copied to the user's clipboard via a right-click menu (and subsequently pasted into other programs). Additionally, the tables and any text therein can be edited and formatted via an embedded editor. Moreover, the user can edit forest plots using the forest plot editor tool, as seen in Figure . Using this interface, the user can change which columns are displayed in the forest plot (e. g., study sizes), and in which order, as well as the symbols used for point and overall estimates, the scale of the plot (minimum and maximum) values and more.
In addition to the interactive output shown in Figures and , Adobe© PDF and Microsoft© Word -ready Rich Text Format report files are generated and saved. These include all of the tabular and graphical output. Finally, the graphics themselves are automatically output to separate image (.PNG) files, for use in other programs. Our aim was to provide the output in as many formats as possible, in order to provide flexibility to the user.
To validate the computational results, we systematically tested Meta-Analyst vs. the results from metan
version 1.86 in Stata. We compared the output of the programs in 11,803 meta-analyses of binary outcomes and 6,881 meta-analyses of continuous outcomes from issue 4 of the Cochrane Library of Systematic Reviews, 2005. This database of meta-analyses was described elsewhere [19
], and it includes meta-analyses that have very different characteristics.
Over the 11,803 analyses (over all methods and all metrics) for binary data and 6,881 analyses over continuous data, we recorded the minimum of the absolute and normalized differences between the outputs from Stata and Meta-Analyst, where the normalized difference is defined as Δrel = | ΘStata - ΘMA|/ΘStata. Θ is any of the numerical output of the program such as a summary effect size for each meta-analysis metric and method, its variance, and the Q, τ2 and I2 statistics (for random effects models). We aimed to identify differences that are beyond those introduced by machine (im)precision. If there are such differences, both the absolute and normalized difference between the two numbers will be relatively large. When the numbers in question are very large, the absolute difference might be relatively large (merely because of machine imprecision) whereas their normalized difference will be very small. Reciprocally, for small magnitudes the normalized difference can be relatively large (in the absence of computational errors), while the absolute difference is very small.
Over the binary set of meta-analyses, the maximum discrepancy was 2.9 × 10-6. For continuous data analyses, the maximum discrepancy was 7.4 × 10-5. These maximum discrepancies appeared in meta-analyses with extreme between study heterogeneity, and are ascribed to rounding errors (version 1.86 of metan does not use double precision for all internal calculations as Meta-Analyst does).
As previously discussed, Bayesian analyses are run through OpenBUGS, and so the output is as thoroughly tested as OpenBUGS.
Testing for diagnostic test accuracy analyses is not as extensive, because we have not found a suitable reference scripting environment to test Meta-Analyst output against. However, the simple diagnostic test methods are based on weighted proportions (sensitivity, specificity), relative risks (likelihood ratios), odds ratios and regression (SROC, meta-regression). These methods use the same computational algorithms as those for binary data and so have been tested. Two diagnostic methods remain to be sufficiently tested; bivariate meta-analysis of sensitivity and specificity for diagnostic tests and random effects SROC. These analyses are flagged as not thoroughly checked when they are requested from the user. However, these will soon be reconfigured to run in OpenBUGS so that they will be validated as well.