|Home | About | Journals | Submit | Contact Us | Français|
To review the transparency of reports of behavioral interventions for pathological gambling and other gambling-related disorders.
We used the Transparent Reporting of Evaluations with Nonrandomized Designs (TREND) Statement to develop the 59-question Adapted TREND Questionnaire (ATQ). Each ATQ question corresponds to a transparency guideline and asks how clearly a study reports its objectives, research design, analytic methods, and conclusions. A subset of 23 ATQ questions is considered particularly important. We searched PubMed, PsychINFO, and Web of Science to identify experimental evaluations published between 2000 and 2011 aiming to reduce problem gambling behaviors or decrease problems caused by gambling. Twenty-six English-language reports met the inclusion criteria and were reviewed by three abstractors using the ATQ.
The average report adhered to 38.4 (65%) of the 59 ATQ transparency guidelines. Each of the 59 ATQ questions received positive responses from an average of 16.9 (63.8%) of the reports. The subset of 23 particularly relevant questions received an average of 15.3 (66.5%) positive responses. Thirty of 59 (50.8%) ATQ questions were answered positively by 75% or more of the study reports, while 12 (20.3%) received positive responses by 25% or fewer. Publication year did not affect these findings.
Gambling intervention reports need to improve their transparency by adhering to currently neglected and particularly relevant guidelines. Among them are recommendations for comparing study participants who are lost to follow-up and those who are retained, comparing study participants with the target population, describing methods used to minimize potential bias due to group assignment, and reporting adverse events or unintended effects.
Pathological gambling and other gambling-related disorders are a public health concern, with prevalence rates commensurate with many other mental illnesses [1, 2]. Because pathological gamblers often suffer significant adverse consequences [3, 4], there is a critical need to identify and develop effective interventions for these individuals [5-7].
Although the literature on gambling disorders is relatively new compared to the literature for other psychiatric or addiction problems [8, 9], gambling intervention studies are readily accessible to today’s researchers and clinicians [10, 11]. Transparent reporting of these studies is essential in assessing their validity and building a body of evidence-based treatment. To be transparent, a report must provide a clear and comprehensive description of the intervention and comparison condition, setting, participants, and outcomes. Most importantly, transparency requires that researchers report all information related to the study’s outcomes, especially the information that readers will need to assess possible biases . These biases commonly result from flaws in the research design and unanticipated events associated with the intervention’s implementation.
Fortunately, reporting guidelines are available to facilitate transparency. Reporting guidelines are statements that provide advice on how to present research methods and findings. They usually take the form of a checklist, flow diagram or explicit text, and they specify a minimum set of items required for a clear account of what was done and what was found in a research study. The Transparent Reporting of Evaluations of Nonrandomized Designs (TREND) Statement is designed to improve the reporting standards of evaluations of behavioral and public health interventions . TREND is intended only for reports that have a defined intervention and a research design that provides for an evaluation of efficacy or effectiveness. It is not designed for studies that do not have comparison groups in which at least one group receives an experimental or new intervention or treatment.
Most recognized guidelines, like TREND, are based on the available evidence about the characteristics of high-quality scientific research and reflect the consensus of experts in a variety of disciplines and include the support of research methodologists and journal editors. TREND was developed by an international forum of over 50 researchers in public health and drug abuse who published it in the American Journal of Public Health in 2004. It complements the widely adopted Consolidated Standards Of Reporting Trials (CONSORT) developed for randomized controlled trials  as well as other reporting quality guides .
TREND consists of a 22-item checklist pertaining to the content of an evaluation report’s introduction, methods, results, and discussion. The checklist’s purpose is to enable readers to better understand a study’s design, analysis and interpretation, and, as a result, have the information they need to assess the validity and appropriateness of its results. The emerging literature on the effects of CONSORT and other guidelines confirms that recent studies tend to be more transparent than those of the past, with beneficial effects on the quality of the literature and potentially on health and health care [15-18]. Transparency is an indication of reporting quality and not necessarily of an intervention’s effectiveness. A study that finds no difference between interventions may still be transparent, while those claiming large effects may not be.
Evaluations of behavioral programs for gambling-related disorders have not yet been assessed for transparency or reporting quality despite the need that clinicians, researchers and policy-makers have for a reliable database of effective interventions. In this study, we systematically retrieved and reviewed published reports of behavioral interventions for pathological gambling and other gambling-related disorders using the Adapted TREND Questionnaire (ATQ). The ATQ consists of 59 questions, of which 23 are based on TREND standards its proponents regard as essential to transparent reporting. Our study objective was to identify strengths and weakness in reporting quality. We hypothesized, that similar to the literature in other fields, the introduction of TREND in 2004 would be accompanied by an improvement in transparency.
This systematic review of the transparency of gambling intervention reports followed the principles described in the Center for Reviews and Dissemination on undertaking reviews in health care  and the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analysis) guidelines . UCLA’s Review Board granted the study an exemption because we did not involve human subjects in the research.
We chose the TREND statement because of its applicability to the research designs that are most commonly used to evaluate the effectiveness of behavioral interventions . For practical reasons, we organized the 22-item TREND checklist into The Adapted TREND Questionnaire (ATQ) questionnaire (Appendix). The ATQ consists of 59 questions so that each TREND category (e.g. baseline data) and subcategory has a separate question (e.g., Does the report describe the baseline demographic and clinical characteristics of participants in each study condition?). Each ATQ question corresponds to a guideline for transparent reporting.
Eight of TREND’s 22 items and 23 of ATQ’s corresponding questions are presented in boldface type, indicating that, they are particularly relevant to behavioral and public health intervention studies . As in the TREND statement, especially relevant ATQ questions are bolded (Appendix). The ATQ also includes three questions about funding source, public registration, and involvement of an ethical review board. These questions were added to the ATQ to address possible biases associated with ambiguous reporting, and they are consistent with other reporting guidelines [21-23]. The ATQ’s response choices are “yes,” “no” and “not applicable.” A “yes” or a “not applicable” response means adherence to a transparency guideline.
TREND does not have a formal scoring system. Like CONSORT, it is considered to be an evidence-based, minimum set of recommendations for transparent reporting . In this study, we assumed that a “yes” response to all 59 ATQ questions means transparency, and we placed special emphasis on the 23 questions that correspond to the 8 TREND items considered essential. We did not fault reports for neglecting to report information if they provided an explanation for the omission.
Eligible reports were published in peer-reviewed journals between January 1, 2000 and April 1, 2011 and were limited to experimental evaluations of behavioral interventions aimed at reducing problem gambling behaviors or decreasing problems caused by gambling . We defined an experimental intervention as a controlled trial in which an experimental group receiving a new or innovative treatment is compared with one or more groups. All behavioral interventions for pathological or other problem gamblers were included regardless of the investigator’s use of terms (e.g., pathological gambler, problem gambler) or inclusion criteria (e.g. when gambling last occurred). Typical behavioral interventions include cognitive behavioral therapy, brief motivational enhancement therapy, and use of self-help workbooks. We excluded reports that were not in English; focused on pharmacotherapy; relied on physiological rather than behavioral measurements; or were reviews, case studies, or commentaries (Figure 1).
We retrieved eligible study reports from three electronic databases (PubMed, PsychINFO, and the Web of SCIENCE) on April 1, 2011. We used search strategies that were specifically applicable to each database. For PubMed we used MeSH terms; for PsychINFO, Descriptors; and Web of Science, Topic Terms. Our MeSH terms were “gambling” AND “intervention studies” OR “therapeutics” or “treatment outcome” OR “psychotherapy.” We limited the search by these terms: humans, journal article, 2000-2012. The descriptors we used to search PsycINFO were “gambling AND (“treatment” OR “intervention” OR “effectiveness evaluation”) NOT “drug therapy.” The search was limited by these terms: humans, peer-reviewed journal articles, 2000-2012. The Web of Science topic terms we used were: “gambling,” AND (“therapy” OR “intervention” OR “treatment” OR “treatment outcome”). The limitations were: NOT “Drug Therapy.” We restricted our review to articles in peer-reviewed journals published between 2000 and 2011.
We compiled the reports that were retrieved from the three databases and deleted duplicates in EndNote Version X3. We then reviewed titles and abstracts for potential eligibility and excluded ineligible reports. Finally, we assessed and reviewed full-text reports for eligibility. To ensure that all searches were consistently performed, two trained researchers (IP and MC) conducted independent searches on all eligible reports and compared results. Two unaffiliated gambling research and treatment experts reviewed the list of remaining studies for comprehensiveness and representativeness.
Three reviewers (IP, MC, AS) used the ATQ to answer each question for each study report. All reviewers were trained to do the abstractions and supervised throughout the process. The online Cochrane Collaboration’s glossary of terms and the CONSORT (Consolidated Standards of Reporting Trials) website were used to help standardize interpretation of the ATQ terms . After the three reviewers independently completed the ATQ for each study report, their findings were compared, and differences were resolved by consensus. A fourth reviewer (AF) resolved any differences that could not be negotiated and also evaluated the accuracy of a sample of 50% of the reviews.
We used STATA Version 18.0 to enter and calculate statistics. We calculated the total proportion of positive responses to the ATQ by each study report and computed the average across all reports. We also calculated the total proportion of positive responses for each of the 59 individual ATQ questions across all 26 reports and calculated the average across all questions. We then compared the proportions and characteristics of the ATQ questions receiving positive responses from 75% or more of the study reports with those receiving positive responses from 25% or fewer.
To examine the relationship between publication year and transparency, we examined if the proportion of positive responses to the ATQ increased between 2000 and 2011. We plotted each study’s publication year and the proportion of its positive responses to the ATQ. Then we conducted a regression analysis using a Generalized Linear Model (GLM) to determine whether the year of publication predicted the proportion of positive responses. GLM can be used for small data sets with proportional values to examine the relationship between the total proportion of positive responses and time . We also examined if the proportion of positive responses to each ATQ question changed from 2000 to 2011. We created a binomial variable for each ATQ question based on whether or not a report received a positive response (1 versus 0), with the binomial variable as the dependent variable and the year the report was published as the predictive independent variable. In this binomial regression, we excluded questions that received positive responses from all or none of the reports (n = 13) because the variance of the residuals were constant. We used a p-value of less than 0.05 to estimate the significance of the prediction.
The initial search produced 437 reports including 40 intervention studies, 243 cross sectional studies, 14 case studies, 16 pharmacological studies, 53 non-English language studies, and 71 reviews and commentaries (Figure 1).
Of the 40 interventions, 14 were excluded because they did not include a comparison group (N = 10), or used physiological rather than behavioral measures (N = 4). We excluded the non-English language studies (n =53) because we could not evaluate them. This exclusion process resulted in 26 eligible studies, which are described in Table 1.
Table 2 shows the frequency of positive responses received by each study to all 59 ATQ questions. The 26 studies received a range of positive responses from 26 to 49 (44.1% to 83.1%) of the 59 questions, with an average of 38.4 (65.1%) positive responses, and from 8 to 18 (34.8% to 78.3%) of the 23-question subset of particularly important guidelines, with an average of 13.5 (58.7%).
Each of the 59 ATQ questions received a positive response from an average of 16.9 (63.8%) of the reports and from an average of 15.3 (66.5%) of particularly important questions (Table 2).
Thirty-two out of 59 (54.2%) ATQ questions were answered positively by 75% or more of the study reports (Table 3). This includes 10 (43.5%) of the particularly important questions. For instance, 100% of studies described theories used in designing behavioral interventions (question #1), described the delivery method (#7), described specific objectives and hypotheses (#14), clearly defined the primary and secondary outcome measures (#15), described the unit of assignment (#20), reported the smallest unit that was analyzed (#24), described the statistical methods used to compare study groups (#25), summarized results for each condition for each primary and secondary outcome (#44), summarized other analyses performed (#49), interpreted the results (#51), and described the general interpretation of the results in the context of current evidence and current theory (#56).
Table 3 also shows that twelve of the 59 ATQ (20.3%), questions including 7 of 23 (30.4%) of the particularly relevant ones, received positive responses from 25% or fewer of the study reports. Among the questions receiving these relatively low responses are those asking if the studies described their methods of minimizing potential bias due to non-randomization or randomization (question #22), described the baseline characteristics of participants who were lost to follow-up (#38), and presented a comparison between the study population at baseline and the target population of interest (#40).
We found no relationship between the number of positive responses to the ATQ and the year of a report’s publication (Table 4), nor did we find a change over time in the number of positive responses to an individual ATQ question (Table 5). The introduction of the TREND statement in 2004 did not influence the transparency of individual reports or the number of positive responses to ATQ questions.
This review of 26 English-language reports of behavioral intervention research on gambling published between 2000 and 2011 suggests a need for improved reporting transparency. The average report received positive responses to just over 61.5% of 59 ATQ questions, and the average ATQ question received a positive response from about 64% of studies. Further, about 30% of the particularly important questions were answered positively in just 25% of the reviewed studies.
The gambling behavioral intervention literature’s need for improvement is most evident in its methodological deficits. For instance, fewer than 25% of the reviewed reports described their sampling methods and how they minimized potential bias due to non-randomization or randomization. Without sufficient information about the methods that are used to assign participants to groups, readers cannot fairly evaluate the likelihood of bias in group assignment . In keeping with current standards, reports should describe who generated the allocation sequence, who enrolled participants, and who assigned participants to interventions.
Users of effectiveness research must be able to evaluate whether an intervention is likely to be effective in their patients. This may not be possible for consumers of much of the gambling intervention literature because fewer than 25% of study reports described the baseline characteristics of study participants lost to follow-up and retained overall or by study condition. Further, the reports did not compare whether the study participants were likely to be similar in important characteristics (such as age, gender, severity of gambling disorder) from those in the target population.
Gambling behavioral intervention reports need to be more transparent in their descriptions of who participated in the research, who dropped out, and if the drop out rates differed between experimental and control groups. TREND advocates using a flow diagram that can assist investigators in reporting the study patients’ progress throughout the course of the research. The diagram enables readers to more accurately evaluate the potential impact of the intervention on their patients. The gambling literature also needs to be explicit about unanticipated consequences of participation and provide an explanation for the interventions effects in patients who have the selected characteristics.
Reporting quality did not change over time unlike other fields in which reporting quality has improved in recent years with the introduction of guidelines [15-18]. It is important to note, however, that although reporting in other fields has improved, uniform standards for comparing reporting quality across fields are unavailable , and the gambling literature’s place on the quality spectrum cannot be known.
Adherence to reporting quality is essential if the gambling field is to progress, especially within the context of the growth of evidence-based health care  and its reliance on meta-analyses. A meta-analysis is the use of statistical techniques in a systematic literature review to integrate the results of included studies. Its application requires that investigators report their sampling strategies and statistical methods in great detail. Unfortunately, many guidelines for statistical transparency, such as describing the sampling methods, are neglected in the gambling literature, inhibiting its usefulness for meta-analysis.
Reporting guidelines are important to a field’s growth because they can have a beneficial effect on the way research is conducted. For instance, as the American Psychological Association has noted , the TREND guideline suggesting that dropout rates be reported may encourage researchers to consider what levels of attrition are acceptable, and thus they may come to employ more effective procedures for maximizing the number of participants who complete their study.
Reporting guidelines are not without their critics. Some researchers have raised questions about the criteria for assessing reporting quality . They point out that reviewers tend to use heterogeneous criteria and do not rely on uniform definitions, thereby possibly limiting the relevance of their reviews . Another concern raised by researchers is that strict adherence to guidelines may lead to excessive standardization. For example, compliance with reporting standards may fill articles with details of methods and results that are inconsequential to interpretation, resulting in a loss of critical facts to an excess of minutiae . TREND, however, is a minimum set of standards, so its use does not preclude providing other information that investigators believe will maximize the reader’s understanding of their study’s objectives, methods, and findings.
Despite some concerns about reporting guidelines, almost all major biomedical and public health journals have accepted and even promoted their use. The CONSORT Statement, for example, has been adopted by over 450 journals, and TREND is amassing an increasing number of supporters including Addiction, the American Journal of Public Health and the Journal of Alcohol and Drug Addiction .
Behavioral gambling intervention reports can improve their transparency by focusing on particularly important guidelines currently neglected in the literature. These include describing the baseline characteristics of study participants who are lost to follow-up and those who remain throughout the study, comparing participants at baseline with the target population; describing the methods used to minimize potential bias due to group assignment, and reporting adverse events or unintended effects that occur in the experimental and control groups.
This review examined published reports in English. It is possible that there are unpublished studies in other languages whose reporting quality may have elevated the gambling literature’s average transparency. However, we had outside experts review our list of eligible study reports, and they found it to be complete and representative.
We reviewed 26 reports. This number represents only a segment of the gambling literature. Other reviewers may have selected different search terms, scoring criteria, or reporting standards. We did not review case or cross sectional studies primarily because checklists like TREND are not currently available for these research designs, which are not typically used to evaluate the effectiveness of interventions.
The number of reports included in this study, however, is consistent with reviews in other fields. For example, a review of the effectiveness of brief alcohol interventions in primary care included 21 studies . The Cochrane Collaboration’s review of psychosocial smoking cessation interventions to help people with coronary heart disease stop smoking included 16 studies , and its review of combined-pharmacotherapy and psychological therapies for-posttraumatic stress disorder relies on four studies .
Another potential limitation of this review is that we may have penalized studies for neglecting to report negative events and deviations from protocol as required by the ATQ. It is conceivable that the reports did not contain this information because the investigators did not observe any adverse events and implemented their studies as planned. Without explanatory information from the authors, however, we were unable to distinguish between intentional reporting omissions and lack of transparency. Additionally, standardized transparency scores are not available, and our criteria for evaluating reporting quality may be considered arbitrary or set too high. Finally, individual reports may have had unique characteristics that the TREND statement is not designed to uncover, and if so, our review necessarily failed to identify and discuss them.
We thank Drs. Jeremiah Weinstock and David Ledgerwood for reviewing our list of reported studies for completeness and accuracy. Margarit Davtian reviewed and edited previous drafts. Lorna Kwan Herbert and Qiaolin Chen offered invaluable statistical advice.
This project was partially funded by the National Institute on Drug Abuse (Grant #: K23DA 19522-2) and the California Office of Problem Gambling.
Declaration of interest: There are no affiliations, members, funding, or financial holdings that might be perceived as affecting the objectivity of this project or providing a conflict of interest. There are no contractual constrains on publishing required by the funder.