Based on our review, there is no specific tool that is adequately designed for the robust evaluation of pharmacoepidemiological studies of drug safety. No single tool considered all the selected domains and elements and most tools failed to address critical evaluation elements. Making a distinction between RE and QAA is important as even if an element of the study is mentioned in the final report, one must then determine if this was appropriate for the specific study in the context of the drug safety question. Only a few instruments specifically made this distinction as we did in our a priori framework. Additionally, important RE and QAA were lacking in most of the checklists and scales we reviewed which highlights the need for a tool focused on the evaluation of epidemiological studies designed to evaluate drug-related harms; this need has been previously identified by others.88
Quality attributes related to exposure definition and ascertainment were considered in less than half of the assessment tools, with less than 15% including RE and QAA pertaining to the comparator group, despite the fact that the selection of a comparator is critical for drug safety and effectiveness trials and epidemiological studies, as the choice of suboptimal comparators can provide misleading results.89
Only 30% of the instrument included quality assessment elements pertaining to the validity and appropriateness of the operational aspects of exposure ascertainment, and only 36% addressed quality attributes of validation of outcome ascertainment approaches. These are important facets of pharmacoepidemiological safety studies their misclassification may lead to false-negative findings regarding the association between a drug and adverse event.
Only about 40% of the checklists and scales included QAA on approaches to handle confounding and biases. As observational studies are not randomised, the approaches to handle confounding and bias are of paramount importance.7
This is an important limitation of most tools because there are often uncertainties regarding results from pharmacoepidemiological studies due to the limitations of electronic healthcare data and complex nature of the practice of medicine.92
Only a small percentage of tools (28% RE; 18% QAA) included elements on the consideration of study findings in the context of the design, conduct, limitations and statistical power despite the fact that these elements are essential in assessing implications of study findings.
Some of the tools we reviewed were designed as ‘all-purpose’ assessment instruments for evaluation of clinical trials and observational studies, while others focused on a particular study design (eg, case−control, cohort). It may be useful to create one consolidated, validated tool for evaluating observational pharmacoepidemiological safety studies focused on general reporting and quality attributes; tools for the specific study designs, that is, case−control and cohort studies, may also be useful due to some of the unique aspects of these designs. By creating such a tool, regulatory agencies, clinical guideline developers, and clinicians could consistently evaluate studies and for decision making. The creation of this instrument could be led by an independent group of expert methodologists, perhaps with input from multiple stakeholders, including regulators and professional organisations.
Although we did not address weighing of importance of different domains and elements based on their relative impact on study contribution to the available streams of evidence, this may be an important consideration in the formulation of an assessment tool. Also, it is not clear if numerical scores are helpful in assessing the quality of epidemiological studies, as when numerical scores were used to evaluate systematic reviews or meta-analyses of such studies, they did not produce valid results.93
The appropriate tradeoff between the utility of a checklist or scale for review and the comprehensiveness of the evaluation elements has yet to be determined. This is complicated by the lack of validation of most of the available tools. Before these issues can be addressed, it is first necessary to engage in a broader discussion of the utility of such assessment tools in the evaluation of pharmacoepidemiological safety studies. It is worth noting that critical assessment elements of pharmacoepidemiological studies focused on effectiveness may be different than those focused on safety; however, pharmacoepidemiological comparative effectiveness studies focus on both comparative safety and benefits associated with drugs and thus such elements are not mutually exclusive.94
Thus, it is important to consider the potential to leverage current efforts to create a validated assessment tool (GRACE checklist95
) for observational comparative effectiveness pharmacoepidemiological studies.
Our review has some limitations. The purpose and scope of the checklists and scales we reviewed, varied greatly. Although we conducted a comprehensive review, there may be tools that we were unable to access or that were published after our search. If a reporting or quality assessment element contained some aspects of the element, we counted this as full representation, even if not all the important sub-elements were included. Each checklist or scale was reviewed by one study team member; repeating the evaluations via a second reviewer was deemed unnecessary at this stage as the primary goal of the review was to obtain a broad understanding of the utility of available assessment tools in evaluating pharmacoepidemiological safety studies based on a preliminary assessment framework. Some factors that may be increasingly relevant in future studies, such as electronic health records with linkages to other data sources like outpatient claims, health information exchanges or personal health records, were not included in our framework but may be included in a future validated instrument. Guidelines and checklists published after the time period of our review have included some elements that may be important for future linked studies which may leverage the increasing availability of these data sources.96
In the evaluation of many emerging safety issues, pharmacoepidemiological safety studies are discussed and may influence safety-related decision making. However, often the quality-driven contribution of such studies is not discussed in a consistent way. The development of an assessment tool based on expert input may facilitate consistent, evidence-based quality assessment of such studies and the subsequent determination of their value based on evaluating the impact of bias on the robustness of a study results, and the interpretation of its findings, within the context of the specific drug safety issue. The framework we developed may serve as a foundation for future development of such an instrument. Efforts to improve the evaluation of the contribution of pharmacoepidemiological safety studies would be consistent with the FDA's focus on strengthening regulatory safety science.97
If after further consideration and discussions with stakeholders development of a tool to evaluate epidemiological data for drug safety is pursued, it would be necessary to first determine the scope of the assessment tool as well as steps for its comprehensive validation. Further, relevant aspects of the design and analysis of pharmacoepidemiology studies should be considered (we refer the reader to some helpful references).98–100
Importantly, such a tool would be intended to complement, and not replace, expert clinical, methodological and statistical expertise necessary to complete a robust evaluation and determination of the contribution of a specific pharmacoepidemiological safety study to the available evidence for regulatory decision making.