|Home | About | Journals | Submit | Contact Us | Français|
Opportunities for community members to actively participate in policy development are increasing. Community/citizen's juries (CJs) are a deliberative democratic process aimed to illicit informed community perspectives on difficult topics. But how comprehensive these processes are reported in peer‐reviewed literature is unknown. Adequate reporting of methodology enables others to judge process quality, compare outcomes, facilitate critical reflection and potentially repeat a process. We aimed to identify important elements for reporting CJs, to develop an initial checklist and to review published health and health policy CJs to examine reporting standards.
Using the literature and expertise from CJ researchers and policy advisors, a list of important CJ reporting items was suggested and further refined. We then reviewed published CJs within the health literature and used the checklist to assess the comprehensiveness of reporting.
CJCheck was developed and examined reporting of CJ planning, juror information, procedures and scheduling. We screened 1711 studies and extracted data from 38. No studies fully reported the checklist items. The item most consistently reported was juror numbers (92%, 35/38), while least reported was the availability of expert presentations (5%, 2/38). Recruitment strategies were described in 66% of studies (25/38); however, the frequency and timing of deliberations was inadequately described (29%, 11/38).
Currently CJ publications in health and health policy literature are inadequately reported, hampering their use in policy making. We propose broadening the CJCheck by creating a reporting standards template in collaboration with international CJ researchers, policy advisors and consumer representatives to ensure standardized, systematic and transparent reporting.
It is incumbent on researchers to provide adequate descriptions of their research methodology and methods when reporting their findings in the peer‐reviewed literature. Where research methods are well described, we better understand the process by which the findings have been generated and have more confidence in appraising the credibility of these outcomes. Reporting standards also create an incentive for researchers to be meticulous in designing studies. Reporting guidelines have been developed for a range of methods and disciplines for these reasons (e.g. CONSORT, PRISMA, STARD,1 CHEERS2) and some journals now require these checklists to be completed prior to publication. More recently, templates have been developed to provide guidance for describing the interventions tested in research (TIDieR3). When used, these checklists allow comparisons between studies, enhance transparency, support trust in the process and provide robust foundations for future research.
In Western liberal democracies, there has been an increased openness in government public policy processes to include public voices in policy debate and formulation.4, 5, 6 Public engagement processes are gaining popularity, but their methods vary.4, 7 Three broad methods of public engagement include communicative (e.g. public meetings, website information), consultative (e.g. opinion poll, focus groups) and participatory (deliberative poll, citizen jury).8 This article is focused on a group of participatory, deliberative democratic processes with methodologies similar to those described by the Jefferson Center.9 Some have been called citizens’ panels, citizens’ fora and community/citizens’ jury. We refer to these, collectively, as community juries (CJs).
As a participatory form of public engagement, CJs aim to illicit an informed community perspective on controversial or difficult topics where the values and preferences of community members are sought. For this reason, CJ participants are recruited from the general population and deliberate on questions that require an ethically sensitive or value‐based decision. Common CJ elements include extensive provision of information from expert presentations, questioning opportunities, substantial time for deliberation and formulation of a consensus or majority “verdict” by CJ participants.9
CJs have been used to explore community perspectives on several health areas including screening,10, 11, 12 resource priority setting,13, 14 informed consent processes for screening decisions15 and pandemic planning.16 These particular examples were conducted for research purposes and none formally linked CJ outcomes to policy decisions. But in Australia, the extent to which CJs are embedded in policy processes is changing. Recently, CJs have been used by local and provincial governments to garner public input to policy making.17, 18, 19 However, despite the increased use in policy decision making, some hesitancy exists around the trustworthiness of CJ processes.20 If the process methods of CJs were reported comprehensively, consistently and transparently, their use in shaping health policy may increase.
Over a decade has passed since Abelson et al.4 suggested design and evaluation principles for public deliberations. Four key components were identified: representation of citizenry, documentation of deliberative procedures, descriptions of information provided to community members and reporting of outcomes. However, inconsistencies in reporting of these components still occur. A recent systematic review explored the adaptations in CJ methodologies and reported wide variations in how CJs were conducted and reported.21 Also, in a review of public deliberation in health policy, Abelson et al.5 continued to find inconsistencies and ambiguity in reporting method descriptions. Different CJs could legitimately have different outcomes even if they followed the same methodology (random recruitment, same expert information, etc.), but poor and/or inconsistent reporting means it is difficult to compare CJ findings or consider potential reasons for differing results, and it is not yet empirically clear how methodological variation may alter outcomes.
We cannot assess quality until we are clear about methodology. Implementers of CJs know little about what might enhance or detract from the legitimacy of claims about the representation of different types of public; quality of the processes of information provision and public deliberation; or the authenticity of the outcomes. We sought to determine the adequacy of process reporting in CJs published in the peer‐reviewed health and health policy literature. We used two processes. First, we conducted a Delphi survey with a group of published CJ researchers to generate an agreed checklist of CJ characteristics and research methods that are important to report in publications. Second, using this checklist, we then reviewed published CJs to establish the utility of the checklist and determine the most frequently reported and missing descriptions of the CJ methods.
Fourteen published Australian CJ researchers and selected government policy advisors attended a CJ Research Symposium at Bond University, Australia, in July 2015. Participants were invited if they were active Australian CJ researchers (13 invited, 10 attended); active policy advisors/government workers (four invited, all attended or were represented); and/or consumer representatives (two invited and neither able to attend). Symposium goals were broad with an aim to discuss several key research and policy questions. Outcomes included the identification of several research agenda and collaborative links. One research question arising from the symposium was to identify potentially important CJ characteristics and processes and determine whether these were reported in published CJ studies. Methodological elements of CJs were brainstormed and iteratively developed by all symposium participants in small group, round robin exercises. A smaller self‐selected working party (from the larger symposium group) participated in a two‐stage Delphi method22 to refine these CJ elements.
The first Delphi round was conducted in September 2015. The survey consisted of the 17 reportable methods suggested by the larger CJ symposium group. These criteria were divided into four groups (planning the CJ, characteristics of the jurors, CJ procedures and CJ schedule). Respondents were asked to rate each criterion on a five‐point Likert scale (1=“important” to 5=“unimportant”). Changes to item wording and new items could be suggested for ranking in the second round. Items that were scored by six or more respondents as “important” or “somewhat important” were retained. Items that had fewer than five respondents scoring in these top two categories were re‐ranked in the second round.
The second round was conducted in October 2015. Questions were again ranked on a five‐point Likert scale. As before, there was one optional, free response question, for further suggestions. On this occasion, respondents were also asked whether the item was important but not necessary to report.
The final checklist (CJCheck) was used in data extraction. Seven items were rated as either “yes” or “no,” while the remainder as “yes,” “no” or “unclear.” Specific checklist criteria are reported in Table S1.
We used two strategies to search for included studies. First, included articles were drawn from two recent systematic reviews pertaining to deliberative democracy.21,23 These systematic reviews included a broad range of deliberative democratic techniques; therefore, we only included articles from these reviews that nominated the deliberative democratic technique most resembling CJs (e.g. random or representative selection of jurors, provision of balanced information to jurors, sufficient time allocated to deliberation).8 Included studies from these two reviews described the deliberative process as “citizens’ panels,” “community or citizens’ juries,” “citizens’ council” or “citizens’ fora”).
In addition to including relevant studies from the two previous reviews, we replicated the search strategy of Degeling et al.23 and modified the search dates to encompass 2013 to 2015 to include studies published since the initial search. We conducted the new search in Medline, Web of Science, Current Contents Connect and Scopus databases in September 2015. Therefore by combining the two search strategies, we included published CJ studies in the health and health policy literature between 1996 and September 2015.
Remaining consistent with others,9, 21, 23 we included studies that described the deliberative process as having characteristics similar to CJs. Our inclusion criteria required studies to describe any or some characteristics resembling those outlined by the Jefferson Centre9 regardless of whether they were reported in detail. For example, studies were required to define themselves as conducting a citizen jury, community jury, citizen panel or fora, providing information to participants from different “experts” and having a deliberative component.9 Protocols were excluded. All studies were screened independently against eligibility criteria by two authors (R.T. and R.S.). Conflicts were checked periodically and resolved through discussion. Criteria definitions were clarified where necessary.
Data extraction was conducted by the same two authors using the checklist developed in the Delphi rounds. Data were analysed descriptively using Excel.
In the first Delphi round, 13 CJ methodology reporting criteria were identified as either important or somewhat important and retained. In the second Delphi round, nine items were presented: four from round one that required further ranking and five new items. Following the second round, a further three questions were indicated to be important in the reporting of CJs and were retained, two were combined with other retained items, one item was reworded and retained and three were excluded. The Delphi round outcomes are reported in Table 1. No items were consistently ranked as “somewhat unimportant” or “unimportant.”
Ranking outcomes of Delphi rounds 1 and 2
Six checklist items were rated higher in importance than others. To be designated as an important item, it had to be rated either important or somewhat important by a clear majority of respondents (8/8 respondents in the first round and 8/9 respondents in the second round).
The final checklist contained 17 items grouped into four areas: planning of the CJ; juror information and characteristics; procedural information; and the scheduling of CJs.
From the two systematic reviews,21, 23 45 studies were excluded because they were not described as a CJ and 22 studies were reported in both reviews, leaving a total of 46 studies for full‐text review. In addition to these, our focused literature search modified from Degeling et al.23 found 1598 de‐duplicated articles for further analysis, of which only 18 met the inclusion criteria and were included in full‐text review and data extraction. We excluded 26 full‐text articles from data extraction, leaving 38 studies in our final analyses (see Fig. 1). Included studies and their characteristics are listed in Table 2, with data extraction details of included studies available in Table S2 and reason for study exclusion from full‐text review provided in Table S3.
PRISMA flow chart
Studies included in data extraction
Studies varied widely in their reporting of the methodological processes necessary for reliable CJ reproduction. No study reported all 17 checklist items. On average, only 53% of checklist items (nine items from a possible 17) were reported in any given article (SD=19.5%) and this proportion ranged from 12% (2/17 items13) to 88% (15/17 items24). Overall, the checklist item reported most consistently was “Was the number of jurors reported” (92%, 35/38 studies). In contrast, the least reported item was “Are the expert presentations available,” which was only reported in 5% (2/38 studies) of articles. Figure 2 shows the proportion of articles reporting checklist items.
The number of included studies with rating descriptions of checklist items. *Indicates items identified by Delphi respondents as important
When reporting about planning a CJ, 21% (8/38) of studies described whether they had a stakeholder group or committee and if so, the role of this group in CJ planning. Less than half (45%, 17/38) stated how and why the experts were chosen; however, more (58%, 22/38) defined the roles of the experts and most (89%, 34/38) gave a clear description of the jury “charge.”
The majority of studies reported some type of juror information. The recruitment strategy was described in 66% (25/38) of studies and inclusion and exclusion criteria were described in 58% (22/38). Whether the juror was an affected member of the public, unaffected member of the public, advocate or invited via random sampling was reported in 79% (30/38) of studies. However, only just over half (58%, 22/38) reported the demographic characteristics of the jurors. The most consistently reported juror detail was the number of jurors (92%, 35/38).
There was a wide variation in what procedural information was provided in the published studies. The role and experience of the facilitator was described in less than half (45%, 17/38) of the examined studies. There was an absence of description and accessibility of materials provided to the jurors, with this reported in only 26% (10/38) of studies. Approximately 68% (26/38) described the expert cross‐examination opportunity provided to the jurors. The outcome of juries was the most consistently reported criterion in this section with 84% (32/38) reporting this outcome. But only 47% (18/38) described how the jurors were instructed to deliberate.
Overall, CJ scheduling information was poorly reported. The frequency and timing of deliberations was reported in only 29% (11/38) of examined studies. The number of presenters and their topics was reported less than half the time (45%, 17/38) and the availability of expert presentations was reported in only 5% of studies (2/38).
Overall, our findings indicate considerable opportunity for improvement in the reporting of CJs. No published CJs we assessed fully described CJCheck items identified by CJ researchers as important to report. Less than half of the studies in our analyses reported the role of the stakeholder committee; how and why experts were selected; the role of the facilitator; a description of the materials provided to the jurors; how the deliberations were framed; the scheduling of events; the number of presenters and their topics; and whether the expert presentations were available. The most reported checklist items were the number of jurors, a description of the jury charge and the jury outcome. Although these are essential reporting items, they are not sufficient.
Although in this review we have focused on peer‐reviewed publications, we believe our suggested checklist items apply to all reported CJs. CJ organizers need to be sufficiently transparent and rigorous in reporting their methods for several reasons: leaving open the possibility of attempting to approximate and repeat a CJ elsewhere; enabling judgements about the quality of the CJ process; allowing comparison of juries; and facilitating critical reflection on CJ design and its potential effect on CJ outcomes. We cannot do this if authors inconsistently report methodology and methods.
It is not our contention that if a particular CJ was repeated that the CJ “verdict” and reasoning would necessarily be the same. It is feasible that it might not be, given different contexts and participants and therefore different values and preferences. But it is also feasible that repeating a CJ, with different participants but the same content and processes, may produce similar outcomes. Because CJ methods are not currently consistently reported, we cannot assess these questions. Perhaps more importantly, comprehensive reporting of CJs supports transparency and would permit the improved evaluation of CJ methodologies. How representative, procedurally fair and accountable CJs are in the health sector has not been fully evaluated.4, 21 This hampers the utility of CJs to contribute to important health policy debates.
Our study has numerous strengths and some limitations. We conducted a focused review of published CJ studies within the health sector and mapped descriptions of their methods to criteria developed by CJ researchers. The criteria checklist had two rounds of iterations by published CJ authors to ensure that important methodology and processes were captured. Studies were then independently rated using these criteria for completeness of reporting. However, the criteria may be somewhat limited as they were developed by a group of Australian CJ researchers and policy advisors and may not reflect the wider view of other CJ organizers. We consider this an important future development. Additionally, we rated the adequacy of reporting on the basis of the written publication and did not contact the authors for further information. This was a deliberate choice, as our purpose was to assess the adequacy of reporting of published CJs.
Other CJ research has reported the variations in methodology.5, 21, 23 This was the first study to quantify those variations by mapping researcher‐identified methodological reporting criteria. Our process was similar to the initial development of the TIDieR checklist,25 the uptake of which seems likely to strengthen the utility of intervention research. We intend to expand our consortium of CJ researchers, policy advisors and consumer representatives to develop an internationally agreed CJ reporting template. We expect that the legitimization of CJs’ input into public health policies will be assisted by the use of standards for reporting of methodology and methods.
To fully report processes within a CJ, it is important that the planning of the event, information about the jurors and their recruitment, and procedural and technical information be available and clearly documented. Our findings suggest that many current CJ publications in health and health policy literature are inadequately reported. Therefore, methods are not transparent and readers cannot compare CJ processes and outcomes. This may not be solely a result of inadequate reporting, but also due to the absence of specific reporting standards for CJs.
Street et al.21 suggested that strict adherence to CJ methodology as described by the originators might limit the generation of new knowledge. We agree. We advocate that CJ organizers and authors be committed to rigour and openness, but also innovation. It is through testing and adapting methodologies that new ideas are developed and understanding expands. As researchers who utilize CJ methodology, it is not our intention to restrict methodological development, but to enhance understanding. In this study, we present an empirically developed and trialled, proposed checklist for reporting CJ methodology and methods. We advocate broadening CJCheck by creating a reporting standards template developed by a consortium of international CJ researchers, health policy advisors and consumer representatives to use when designing and reporting CJs.
The authors report no conflicts of interest.
RT was supported by a NHMRC Screening and Test Evaluation Program (STEP) Grant (#633033). RS was supported by a Bond University Vice Chancellor's Research Grant Scheme. CD, SMC and LR received funding support from NHMRC Project Grant (#1023197). CD received funding support from a NHMRC Project Grant (#1083079). SMC is funded through NHMRC Career Development Fellowship (#1032963). JMS was funded by an Australian National Preventive Health Agency Fellowship (20STR2013F) and an NHMRC Capacity Building Grant (565501).