|Home | About | Journals | Submit | Contact Us | Français|
Conceived and designed the experiments: DK MC HD JI. Performed the experiments: DK AG. Analyzed the data: DK CL. Wrote the paper: DK. Critical revision of manuscript: DK MC HD JI CL AG. Obtained funding: DK MC HD JI.
Patient reported outcomes (PROs) are increasingly assessed in clinical trials, and guidelines are available to inform the design and reporting of such trials. However, researchers involved in PRO data collection report that specific guidance on ‘in-trial’ activity (recruitment, data collection and data inputting) and the management of ‘concerning’ PRO data (i.e., data which raises concern for the well-being of the trial participant) appears to be lacking. The purpose of this review was to determine the extent and nature of published guidelines addressing these areas.
Systematic review of 1,362 articles identified 18 eligible papers containing ‘in-trial’ guidelines. Two independent authors undertook a qualitative content analysis of the selected papers. Guidelines presented in each of the articles were coded according to an a priori defined coding frame, which demonstrated reliability (pooled Kappa 0.86–0.97), and validity (<2% residual category coding). The majority of guidelines present were concerned with ‘pre-trial’ activities (72%), for example, outcome measure selection and study design issues, or ‘post-trial’ activities (16%) such as data analysis, reporting and interpretation. ‘In-trial’ guidelines represented 9.2% of all guidance across the papers reviewed, with content primarily focused on compliance, quality control, proxy assessment and reporting of data collection. There were no guidelines surrounding the management of concerning PRO data.
The findings highlight there are minimal in-trial guidelines in publication regarding PRO data collection and management in clinical trials. No guidance appears to exist for researchers involved with the handling of concerning PRO data. Guidelines are needed, which support researchers to manage all PRO data appropriately and which facilitate unbiased data collection.
Patient reported outcomes (PROs) such as health-related quality of life (HRQL) are increasingly assessed in clinical trials.– PROs provide researchers, clinicians and patients with important information regarding the effect of a disease and its treatment: on symptoms (for example, pain or fatigue) and on HRQL or satisfaction with care. In general, patients participating in a trial do not directly benefit from completing a PRO questionnaire. This approach is adopted to ensure trial participants are not tempted to tailor their answers in order to influence the treatment they receive within a study, which is a potential source of bias.,  PRO results are therefore used to inform the care of future patients, who, with their clinicians, may use PRO data to inform significant health-care decisions. For example, between interventions offering similar survival or progression-free survival rates, or those that have differing trade-offs between therapeutic benefit and undesirable side-effects. Thus, it is crucial that PROs are administered and processed in an un-biased way.
In order to ensure high quality PRO trial data, consistent and rigorous standardised data collection methods should be used throughout a trial. The use of standardised methods should serve to minimise errors, measurement variability, missing data and systematic bias, thus contributing to the validity of trial results. Local site staff require access to ‘in-trial’ (i.e. recruitment, data collection and data inputting, see Box S1) guidelines that clearly outline the standardised methods in-use, so that all study personnel may fully incorporate them into practice. Such guidelines should be contained within the trial protocol, supported by standard operating procedures (SOPs) where appropriate.
It is of concern, therefore, that anecdotal evidence - obtained during national quality of life training days run by the MRC Midland Hub for Trials Methodology in the UK - suggests that in-trial PRO guidelines are not routinely included within trial documentation and that, as a result, unstandardised PRO data collection may be common. Researchers also report feeling particularly uncomfortable that they receive no specific guidance on how to manage ‘concerning’ PRO data, i.e. data that might raise concern for the wellbeing of the trial participant in some way. Staff encountering such data - commonly represented by markedly low HRQL scores, or unexpected unprompted additional information recorded on the back of questionnaires - were therefore unsure where their responsibility should lie, or whether they should be viewing this information in the first place. In this situation, some described experiencing a ‘dual-role’ tension between their concurrent responsibilities as a clinician and researcher: the duty to act upon the information to benefit the patient verses that of protecting trial integrity by not intervening. In some instances, reports indicated that off-protocol concomitant interventions had been administered, some of which may not have been captured by standard trial reporting mechanisms. Such interventions have the potential to bias trial results. These anecdotal reports have since been supported by a recently completed qualitative study, in which we used semi-structured interviews to explore the experiences of 26 research nurses, research facilitators, trial coordinators and data managers across three NHS sites and two clinical trials units in the UK (under review). This study confirmed a potential for bias associated with concerning PRO data, during both postal or clinic-based and self-reported or researcher/research nurse-assisted data collection.
These reports suggest a lack of in-trial PRO guidance, with a subsequent absence of systematic monitoring of potentially concerning PRO data and a resulting risk of bias. It is uncertain, however, whether they also reflect a deficiency in the published literature in this area. There are recent publications concerning the design of trials with a PRO outcome,  and, with the development of the CONSORT PRO extension, there is now guidance to improve PRO reporting: it remains unclear if the literature provides adequate coverage of in-trial issues.
The purpose of this study was to systematically review the current published in-trial PRO guidance, as no review of this kind had been previously undertaken. The objectives for our review were:
The MEDLINE (Ovid), EMBASE, AMED and CINHAL+ databases were searched from inception to March 2012 (electronic search strategies are presented in full in Appendix S1). We also searched; the US Food and Drug Administration, European Medicines Agency, General Medical Council, Medical Research Council and Royal College of Nursing websites; PROQUEST (Thesis repository); Google; and made use of expert communication in an attempt to find additional potentially eligible papers not returned during the electronic database search. Records were first screened by title/abstract before full-text articles were retrieved for eligibility evaluation. Remaining articles were then subject to a citation search before a final hand-search of all reference lists.
Papers were deemed eligible if they included any form of in-trial guideline focused on PRO assessment during clinical trials. We defined the term ‘in-trial’ as relating to recruitment, data collection and data inputting activity, occurring from the first participant recruitment, through to inputting the final participant's data. The reviewers used the Oxford English Dictionary definition of the word ‘guideline’ during eligibility screening; “a general rule, principle, or piece of advice”. Non-English papers were excluded. There were no other restrictions. All citations were downloaded into Endnote® software version 14, and duplicates deleted. DK screened all articles by title/abstract to determine their eligibility and AG reviewed a random sample of 10% in order to evaluate the reliability of the selection process. Agreement was high (Kappa=0.903) and any discrepancies were resolved through discussion. Full text articles were retrieved following first round exclusions and were also subject to two independent eligibility reviews (DK 100%, AG 10%), this time with perfect agreement.
Data extraction occurred following the final selection of included articles.
DK and CL independently searched each paper to identify all sentences that provided any type of ‘guideline statement’ (which we defined as ‘an expression in words of a general rule, principle, or piece of advice') regarding PRO measurement (in-trial or otherwise). A consensus meeting was then held, to resolve any disagreements and finalise the selection. Each sentence, representing one ‘guideline statement’, was then extracted, as a text excerpt, into a mixed-method data analysis software package (Dedoose © 2011 SCRC) and tagged with its source data (Article title, Journal, Year of publication).
DK and CL undertook a qualitative content analysis of the excerpts extracted from the included papers. All text excerpts were categorised according to an a priori coding frame, which was developed using a concept-driven strategy (i.e. codes were assigned based on the authors' prior knowledge of the literature and the study research questions). DK and CL piloted the coding framework, each independently applying the first draft to a random selection of the included papers, ,  (n=3 (17%)). Following the pilot, a meeting was held to discuss issues requiring clarification and to reach consensus regarding the data-driven changes that would improve the validity of the framework. Three of the co-authors (MC, HD and JI), who possess expertise in PRO design, implementation, reporting and ethics, checked and approved the face validity of the final coding frame. The definitive coding frame is presented in Figure 1. During the main analysis, DK and CL independently categorised each guideline statement according to the phase of trial activity to which it pertained, using a major dimension within the coding frame. These major dimensions were as follows; ‘Pre-Trial’, which included all content relating to the trial inception (including training logistics), up to the start of recruitment; ‘In-Trial’, denoting content directly related to the act of trial recruitment, data collection and inputting; ‘Post-Trial’, including activity taking place following data collection, for example, data analysis/reporting; ‘Future Research’, representing statements addressing the future direction of PRO research activity; and ‘Other’, used to identify guideline statements not captured in the main coding categories. Each individual guideline was also sub-categorised, as appropriate, in order to further identify its role within a given area.
Throughout both the pilot and the main analysis phase, the reviewers met frequently to determine coding reliability for each paper and to seek consensus regarding coding disagreements. The reliability of coding application was determined using Cohen's kappa statistic. Specifically, pooled kappa was employed, as it is the preferred method of calculating inter-rater agreement across a large number of coding items. Face validity of the coding frame was further evaluated by determining the proportion of codes applied to the residuals (i.e., the ‘Other’ major- and sub-categories). A high level of residual coding may indicate that the main categories of the coding frame do not adequately describe the concept under study. Whilst there are no firm guidelines regarding the desirable level of residual coding, we theorised that a figure of less than 5% would support the validity of our coding frame.
A protocol was not published or registered for this study. However, all reviewers followed a protocol detailing a priori determined search strategies, data extraction and data analysis methods.
The search strategy yielded 1273 citations from MEDLINE, EMBASE, AMED and CINHAL+, 89 citations were returned using other sources (PROQUEST, professional bodies, Google, expert communication) (PRISMA flow diagram, Figure 2). In total, 41 full text articles were retrieved for review. 25 articles were excluded at this stage, as they contained no in-trial guideline statements. An additional 2 papers were included following the reference list and citation searches. A final total of 18 relevant articles were included in the analysis.
The characteristics of the 18 included papers are summarised in Table 1. The majority of papers were concerned with the incorporation of PRO/HRQL measures into cancer trial design., , , – Several considered PRO issues relating to pharmaceutical prescribing/labelling., , – Two papers presented generalised guidance on using PRO/HRQL measures in clinical trials.,  Finally, one paper presented recommendations for PRO/HRQL assessment in allergy-related clinical trials. The included articles were drawn from 16 different sources and the mean number of excerpts extracted from each paper was 58 (range 16–127).
Over 1,110 guideline statements were extracted and coded following review of the 18 papers. The coding frame demonstrated reliability, with pooled kappa ranging from 0.86 to 0.97 across articles, and face validity, with overall residual coding at 1.2%. A summary of the final coding breakdown is presented in Table 2.
‘In-trial’ guidance, whilst present in all papers, did not represent the major focus of any, accounting for 9.2% of guideline content across the articles reviewed. ‘Pre-trial’ guidelines were predominant throughout (72.2%), again present in all papers. ‘Post-trial’ guidance was the next most prevalent category (15.8%), presented across 13 articles., , , , , , , – Statements pertaining to ‘future research’ represented 1.8% of guidelines (9 papers), , , , –, ,  and the major category ‘Other’ was attributed to 1% of content (8 papers)., , , , , –
There were no guideline statements addressing the management of concerning PRO data, or related questions including how additional information recorded on the back of questionnaires should be handled and who should have routine access to PRO data in the first instance. The majority of in-trial guidelines (61.2%) tackled notions surrounding quality control, compliance and the correct use of PROs., –, , , –, ,  Authors highlighted the importance of minimising missing items during data collection.,  A number of papers presented guidance aimed at improving compliance within a trial in order to maximise data quality: examples included the proposed education of local site staff, training of patients and use of real-time adherence monitoring , , , , , , –, . Other guidelines were concerned with piloting and standardisation,  of data collection. Examples of suggested methods of standardisation included the following:
Where a trial participant is unable to complete their PRO questionnaire, a proxy (commonly a partner or close relative) may be asked to complete the form on their behalf. Discussion surrounding the role of proxies represented 16.5% of in-trial guidelines., , , , , ,  Authors mainly highlighted the situations in which proxy assessment was justified., , , , ,  The use of a proxy was generally promoted as a last resort , , , however it was acknowledged that proxy data was better than no data at all.,  The ideal identity of the proxy was discussed by two authors, who concluded that, if possible, the same person should be used throughout the trial and they should be close enough to the patient to provide valid data. Guidelines for the reporting of data collection represented 9.7% of in-trial content, , , , ,  and were primarily concerned with the need to document reasons for non-compliance, , ,  and the need to report whether or not a proxy was used, . A small number of in-trial guideline statements (7.8%) focused on patient information, endorsing the use of a supplementary leaflet for patients to take home, and highlighting the importance of the investigator in ensuring the patient fully understands the role of PRO measurement. Two papers by the same author,  presented guidelines suggesting that PRO data should not be used to influence management during a trial and one paper suggested that trial participants ought to be informed when data would be used for the benefit of future patients only.
The majority of pre-trial guidelines (87%) were focused on study design, procedural issues (including training logistics) and the evaluation/selection of appropriate PRO measures., –, , , – Others (12.8%) were concerned with questionnaire development and validation, or with issues arising from questionnaire modification., , , , , , –
Most post-trial guidelines (66.7%) concentrated on data analysis, reporting and presentation issues., , , , , , , – The remaining guidance in this area (33.3%) surrounded the interpretation of PRO data and related labeling claims., , , , , –
The purpose of this review was to investigate whether anecdotal claims (subsequently confirmed by data under review), highlighting a lack of in-trial PRO guidance, reflect a deficiency in the published literature in this area. Our main findings suggest there a minimal guidelines in publication focused on in-trial PRO activity and there are a complete lack of guidelines addressing the management of concerning PRO data.
Of the small number of in-trial guidelines that are in circulation, the majority appear to deal with the procedural issues associated with the prevention of missing data. This focus may be understandable given the detrimental effect missing data may have on a trial. Trial reports indicate that PRO questionnaires are commonly returned with incomplete entries and some may not be returned at all. This data may not be missing at random and it represents a serious potential bias when present. Therefore, it is encouraging there is some consensus in the guidelines reviewed. To reduce missing PRO data, authors recommended that:
The guidance surrounding missing data is therefore comprehensive. In contrast, no guidelines appear to adequately address aspects surrounding the management of concerning PRO data. This may be a problem given this issue has been identified as key by those involved in PRO data collection, as it can result in dual-role tension and may risk the potential introduction of bias into a trial.
A PRO questionnaire may be the only outcome within a trial capable of identifying ‘tolerable’ symptoms such as participant anxiety or depression; and the research nurse checking the form may be the only individual to whom participants have disclosed how they feel. Understandably, nurses may feel it is their duty to intervene when faced with PRO data that raises concern for the participant. A problem arises if the intervention is non-medicinal; for example, words of comfort, or advice to visit one's general practitioner, or if the advice results in the participant self-medicating. Direct medicinal interventions are far more easily controlled-for during data analysis. Non-medicinal or self-directed interventions, that are selectively delivered in response to concerning PRO data, may influence patient well-being but remain unrecorded in the trial documentation: this may represent a hitherto unforeseen source of bias.
Research nurses have reported experiencing dual-role tension when handling PRO data. Dual-role tension arises when an individual's values and responsibilities as a researcher conflict with those associated with being a clinical practitioner. Assuming ethical norms have been followed and participant ‘risk and burden’ does not outweigh the potential benefit of trial participation , the nurse researcher may justifiably choose not to intervene when concerning PRO data is disclosed, in order to protect trial integrity. This decision may be driven by consequentialist values, geared toward achieving the greatest benefit at the lowest cost, and reasoning that the benefits of producing unbiased trial results outweigh the personal costs experienced by the ‘few’ participants who continue to (tolerably) suffer. Conversely, nurse practitioners are obliged to make the care of their patients their first concern, as outlined in the Nursing and Midwifery Council code of conduct, which compels them to take steps to address any evident suffering. This conflict between the two professional duties has been recognized elsewhere–. However, what sets PRO data collection apart from the management of other trial outcomes is the current lack of published, and trial-based, guidance in this area. In our experience, the trial protocol often contains clear guidelines surrounding the levels at which some clinical outcomes, blood pressure for example, need to reach before the data collector should become concerned. There is usually also a clear system in place to manage participants whose clinical measurements exceed agreed limits. Equivalent guidance is not always provided for PROs. Thus, the researcher collecting/inputting PRO data may be left to determine independently, on a case-by-case basis, whether PRO results signal a risk to the participant that outweighs the benefit of trial involvement. We believe this situation places unreasonable demands upon the researcher and promotes inconsistency, as there is unlikely to be uniformity in decision-making across trial sites; this may adversely affect data quality. Our findings highlight the need to develop and publish specific guidelines that clearly outline how concerning PRO data should be handled, as there are none currently in circulation. PRO in-trial guidelines should be brought in line with those covering traditional clinical outcomes and should define the conditions under which the researcher may take remedial action, and the form this intervention might take.
Non-English language papers were excluded from the review, which potentially lessens the generalisability of the results presented. However, this decision was taken as a key element of qualitative content analysis involved determining the implied or latent meaning of the material. We questioned the validity of such analysis using material translated from the original language by a third party, as some latent meaning may be lost during the translation process. Our search strategy dictated that we carefully reviewed papers for their guideline content only if their title/abstract gave an indication that some aspect of in-trial activity might be discussed. It is possible that papers providing ‘in-trial’ guidance exist, which make no reference to in-trial activity in their title or abstract.
In-trial guidelines aimed at PRO recruitment, data collection and data inputting within clinical trials are lacking. No guidance appears to exist for researchers involved with the handling of concerning PRO data. This is a worry as this activity may be associated with considerable personal and professional anxiety and may risk the introduction of bias when the ethical tension generated, is resolved in favour of responding to the needs of the patient over the expectations of the trial. Further research is needed to produce guidelines aimed at supporting researchers so they can deal effectively with dual-role tensions, manage PRO data appropriately and facilitate unbiased data collection.
Definition of terms.
We would like to thank the National School of Primary Care Research and the MRC Midland Hub for Trials Methodology Research for assistance with the development of this review.