|Home | About | Journals | Submit | Contact Us | Français|
Health service accreditation programmes are a regulatory mechanism adopted to drive improvements inpatient safety and quality. Research investigating the benefits or limitations, of consumer involvement in accreditation programmes is negligible. To develop our knowledge in this area the ACCREDIT collaboration (Accreditation Collaborative for the Conduct of Research, Evaluation and Designated Investigations through Teamwork) has developed a research plan, known as the ACCREDIT-SCI (Standards of Consumer Involvement) study protocol. Two complementary studies have been designed: one, to examine the effectiveness of a standard for consumer participation and two, to explore how patient experiences vary across a range of settings with differing accreditation results.
The research setting is the Australian healthcare system, and the two studies focus on three accreditation programmes in the primary, acute and aged care domains. The studies will use multimethods: document analysis; interviews and surveys. Participants will be stakeholders across the three domains including: policy officers; frontline healthcare professionals; accreditation agency personnel, including surveyors and healthcare consumers. Drawing on previous experience, the research team has developed purpose-designed tools. Data will be analysed using thematic, narrative and statistical (descriptive and inferential) procedures.
The University of New South Wales Human Research Ethics Committee has approved the two studies (HREC 10274). Findings will be disseminated through seminars, conference presentations, academic publications and research partner websites. The findings will be formulated to facilitate uptake by policy and accreditation agency professionals, researchers and academics, and consumers, nationally and internationally.
Health service accreditation programmes are a regulatory mechanism adopted globally to drive improvements in patient safety and quality.1–3 They are complex organisational interventions that seek to reassure external stakeholders, including consumers, that healthcare industry safety and quality standards are being achieved and improved. Accreditation programmes are used across the healthcare continuum of primary, acute and aged care services.4 While there are variations between countries, and the health systems within them, in the regulatory frameworks that contextualise and operationalise accreditation programmes, a common model has evolved.5 Healthcare organisations (HCOs) enrol with an accrediting agency and self-assess against their standards. Accreditation standards comprise a set of performance statements and associated criteria and outcomes or actions. HCOs produce reports that document their quality maintenance and improvement activities against the accreditation standards. Reports are reviewed by the accrediting agency which sends teams of peer-reviewers, usually known as surveyors, to conduct onsite visits to validate the claims. When visiting HCOs, surveyors conduct observations of facilities, interviews with staff and document analysis of organisational reports, meeting minutes, quality improvement projects and policies and procedures.6 Surveyors provide verbal feedback to the HCO on their assessment of their achievement against the accreditation standards, including, where appropriate, remedial actions recommended. Written survey reports are provided to the accrediting agency that decides on conferring accreditation status, or not. Accreditation status is the declaration by the accreditation agency, an external authority that HCOs have demonstrated competency against recognised industry-based safety and quality standards.5 Accreditation status is typically for a period of 3–5 years.
Consumer participation in healthcare decision-making and health services governance has become an internationally accepted target.7 Studies have shown benefits from consumer participation in both the clinical8 9 and organisational10 domains. However, research investigating the benefits, or limitations, of consumer participation in accreditation programmes is negligible;1 4 only two studies have been identified.11 12 In one study, an evaluation of consumers as surveyors in an accreditation programme highlighted stakeholder perceptions of an increase in survey teams’ objectivity and credibility. The role of the consumer surveyor was identified as needing to be clarified and more consistently applied. In addition, their capacity to contribute to rating of criteria and writing of the survey report was noted as problematical.11 A second study examined accreditation performance against the consumer involvement in health services, reporting no relationship between the two.12
Beyond this research, there are a limited number of studies that relate consumer views about care received or patient satisfaction with services, with accreditation outcomes. Where they have been done, studies show no clear or consistent findings.4 No relationship between hospital accreditation outcomes and patient satisfaction have been identified.13–15 Accredited health services have not been rated higher than non-accredited organisations against patient-reported quality measures.16 Conversely, another study demonstrated that accredited health units performed better than non-accredited units on patient satisfaction measures.17 Similarities and differences in patient and health professionals’ assessment of accreditation standards of care have been identified.18 In the related area of accreditation of medical practices in primary care, patient experience data have been shown to assist in improving medical practice quality.19 Nevertheless, the study concluded that knowing how and to what extent patient experience data are to be included in accreditation decisions has yet to be defined.19 In short, the existing empirical evidence is both limited and ambiguous as to the value of consumer involvement in health accreditation programmes.5 6 15–17 The motivation for the present study is to address this knowledge deficit.
In Australia, to investigate healthcare accreditation a collaborative research partnership has been formed involving academic healthcare researchers, accreditation agencies from across the care continuum of primary, acute and aged care and quality improvement bodies at national and state levels: the Accreditation Collaborative for the Conduct of Research, Evaluation and Designated Investigations through Teamwork (ACCREDIT).1 The organisational partners are: the Centre for Clinical Governance Research and Centre for Health Systems and Safety Research, in the Australian Institute of Health Innovation at the University of New South Wales (UNSW); Australian General Practice Accreditation Limited (AGPAL); the Australian Council on Healthcare Standards (ACHS); Aged Care and Standards Accreditation Agency (ACSAA); the Australian Commission on Safety and Quality in Health Care (ACSQHC); and the Clinical Excellence Commission (CEC) from the state of New South Wales. The ACCREDIT collaboration also has an international advisory group (IAG) providing expert advice and critique to their work. The IAG members are prominent healthcare quality and safety and health services researchers based in Europe.
The ACCREDIT collaboration have scoped a research project with four aims: evaluate current accreditation processes; analyse the costs and benefits of accreditation; improve future accreditation via evidence and develop and apply new standards of consumer involvement in accreditation. These aims emerged from the melding of ideas from: previous research into healthcare accreditation;2 4 6 12 20–26 literature reviews;4 27 28 a workshop consultation with the partners and other accreditation stakeholders;29 and extended negotiations to clarify and focus the research agenda. In 2010, the ACCREDIT collaboration became the recipient of an Australian Research Council linkage grant (LP100200586). These grants are awarded to research projects that have a significant academic merit and the ability to generate findings to benefit Australian industry and society,30 and international parties with interest in the area.
This research protocol details the investigation of consumer involvement in Australian health service accreditation programmes within the ACCREDIT project; known as the ACCREDIT-SCI (Standards of Consumer Involvement) study protocol. To achieve this aim, two studies have been conceptualised with multiple components. The studies correspond to studies 4 and 5 outlined in the overarching ACCREDIT design.1 The first examines the effectiveness of a standard for consumer participation, and the second explores how patient experiences vary across a range of settings with differing accreditation results. To counteract potential limitations of any single research method, a multi-method approach will be used. This strategy enables triangulation of data sources and promotes credibility of findings.31
In the intervening period since the key questions and details were agreed to by the partners and encoded into a funded proposal,1 the ACSQHC developed a new Australian national standard for consumer participation. The standard Partnering with Consumers is one of the new 10 Australian National Safety and Quality Health Service Standards (NSQHSS).32 The work of the ACSQHC has superseded the first three aspects of the consumer involvement study as proposed in the overarching ACCREDIT project design.1 Consequently, the study has been revised to accommodate the changed contextual circumstances and is now comprised of three parts: conduct a retrospective analysis of the development process for the NSQHSS consumer standard; apply the standard in the field (n=30) and evaluate its use and efficacy with survey and qualitative methods.
The research team will retrospectively analyse the activities and processes used by the ACSQHC to develop the NSQHSS Partnering with Consumers standard. Two evaluation methods will be used to triangulate findings, including documentary analysis33 of ACSQHC Standards Development Committee deliberations and workshop reports. Additionally, semistructured interviews with members of the ACSQHC Standards Development Committee will be conducted. Recruitment and access to key participants and relevant reports will be facilitated by ACSQHC, who will email potential participants inviting them to take part in the study. Interviews will be recorded, transcribed and thematically analysed. Drawing on the research team's knowledge from prior work,23–25 the issues focusing the evaluation, either by document analysis or interview method, will include: what evidence was drawn upon for the standard; how was the evidence assessed; what was the decision making process to include or exclude evidence; to what extent were the stakeholders engaged and how does the standard integrate with the other standards of the NSQHCS?
Study information and consent forms, approved by the UNSW Human Research Ethics Committee (HREC),34 will be provided to potential participants electronically via email. The potential informants will be followed up, via telephone calls, after 2 weeks if there has not been a response to the invitation. Interviews will be conducted face-to-face where possible, or alternatively via telephone, in locations suitable for those involved. To promote participant responses, face-to-face interviews will be the first option. To enhance reliability and internal comparison of data35 the researchers involved in interviews will complete a training session together and use a purpose designed, standardised semi-interview schedule. Furthermore, the schedule will provide structure for uniformity while allowing scope for respondents to expand on issues important to them. Interviews will be digitally recorded and professionally transcribed.
A two-step process will be used to analyse the data. First, the issues employed to direct the evaluation will be used to collate and thematically group the data from the documentation and interview transcriptions. Second, a joint approach of a narrative strategy32 and temporal bracketing32 will be used to interpret the data. This strategy involves constructing an account of the standard development process with time used as the framework to structure the narrative. This approach is suitable for ordering32 and examining change events36 and capturing the complexity of proceedings.37
Thirty accreditation surveys, comprising 10 from each healthcare domain, will be chosen using stratified randomised sampling38 to examine the implementation and assessment of the NSQHSS Partnering with Consumers standard. Based on prior research experience, including accreditation research studies,6 12 22 24 this cohort is expected to be appropriate to provide both depth and breadth of data to assess the standard. A multimethod approach will be used, involving document analysis, interviews and a survey questionnaire.
The research team, in collaboration with the study partners, will map the NSQHSS Partnering with Consumers standard to the accreditation standards used in each domain. That is, the NSQHSS to the ACSAA Accreditation Standards, The Royal College of General Practitioners Standards for General Practice (4th edition), and the ACHS accreditation programme, Evaluation and Quality Improvement Program (EQuIP) V.5, respectively, to identify corresponding standards and criteria related to consumer participation. These details will be used to focus the document analysis of the accreditation outcomes and reports, and provide topics and suitable phrasing of language for the interviews and survey questionnaire.
The HCOs accreditation outcomes and survey reports will be collected. Using purposeful sampling38 accreditation agencies’ representatives and surveyors, and key informants from the HCOs involved with each survey, will be invited to participate in the study. They will be offered an individual or group interview and asked to complete a questionnaire. Informed by their previous work6 21 22 in this field, the research team will examine the following issues with participants: what do you understand the standard is aiming to achieve; is your understanding similar or different to your colleagues and other stakeholders; is the standard easy to survey against or provide evidence for; what criteria or actions do HCOs implement well and which others are difficult to implement; what criteria or actions stimulated most discussion between survey participants; and what, if any, resources are required to implement the Partnering with Consumers standard? The same processes for ethics approval, study information and data collection outlined above will be applied to the interview data. Analysis will follow accepted norms for systematic classification of interview data.39 It will be directed by the principles of addressing the significant points in the data while incorporating key interpretations, with analytical decisions documented by an audit trail and shaped by previous research experience and findings.6 24 25 40–42 An inductive process will guide the analysis and the textual grouping software, NVivo V.943 will be employed.
A web-based survey, using KeySurvey44 software, which will take approximately 15 min to complete, will complement the interviews. This technology enables complex question routing and ease of data collection at a reasonable cost.45 Focusing on similar topics as the interviews, the survey will employ a five-point Likert scale46 with ratings from ‘strongly agree’ to ‘strongly disagree’. Demographic details on participants will be also collected including: occupational position; professional background; highest qualification and level of experience with accreditation surveys. Piloting of the survey will be undertaken. Feedback regarding comprehensiveness of instructions and phrasing of questions and time for completion will be reviewed and the survey amended as necessary. Survey data will be analysed using descriptive and inferential statistics, including generalised linear modelling applied to Likert-type outcomes,47 assisted by other categorical data analysis techniques.48 To identify participant variations regarding views of key themes, responses will be compared both within and between stakeholder groups and accreditation domains.
Study two aims to examine how the patient experience varies across a range of settings with different accreditation results. The study will be undertaken using the approach detailed below.
The research team will review and update or adapt the partners’ existing patient journey tools22 for application in the acute, aged care and general practice settings. Three new purpose designed research tools, detailed below, will be produced: a ‘patient experience questionnaire’; an ‘HCO patient trajectory tool’ and a ‘health professional patient experience questionnaire’. The content and structure of the tools will also be shaped by experience from the research team's previous accreditation research work.3 5 12 A three-stage recruitment and data collection process, indicating participants and research tools, is outlined below and represented in figure 1.
Stage one aims to identify HCOs to participate in the study. To recruit subjects the research team will send an electronic request, that is, an invitation to participate in the study, to the accreditation partners to forward on to the HCOs they accredit. The request will contain the UNSW HREC approved study information and consent forms.34
Identifying potential subjects within the nominated HCOs is the goal of stage two. The HCOs nominating to participate will be approached by the research team to identify potential subjects to track at the time of, or immediately following, the accreditation survey. Potential patients will be those with healthcare journeys characterised as a complex case involving multiple organisations, services, departments and health professionals. The research team and an HCO representative using purposeful sampling38 will together review potential subjects for the study. Once a potential subject is identified, an HCO representative will forward an electronic or written request from the research team to the potential subject to participate in the study. The request will contain the UNSW HREC approved study information and consent forms.34 Using this process, the research team will seek to identify 20 patient journeys to investigate in each domain. Based on prior research experience, including accreditation research studies,22–25 this cohort is expected to be appropriate to provide both depth and breadth of data to assess the patient experience.
Stage three involves the collection of data and construction of individual case studies. After confirmation of suitable subjects, the research team will initiate four actions. First, to capture the patients’ experience of the care and services they received, a ‘patient experience questionnaire’ will be administered. The research team will speak with each patient to give them the choice to complete the questionnaire on paper or electronically. The patients, in addition to providing basic demographic data and reasons for attending the HCO, will be asked to report upon their experience of: making appointments; arrival; waiting for appointments; assessments; interactions with health professionals, including communication and understanding of issues; referral information; discharge; timeliness and accessibility and negotiation of the physical environment. The questionnaire will use a five-point Likert scale and is expected to take 20 min to complete.
Second, an ‘HCO patient trajectory audit tool’ will be used to map the interorganisational and intraorganisational trajectory of individual patients. This tool will enable documentation of the HCOs, and the various services or departments within them, and health professionals or teams who provided significant assessment, intervention or advice to the patient or the primary health professionals or team caring for them. A researcher and HCO representative will examine the medical record to document these issues; it is anticipated that this task will take between 1 and 4 h, depending on the domain.
Third, the health professionals or teams identified will be approached, in person or via telephone, by the research team or HCO representative and asked to participate. They will be surveyed using a ‘health professional patient experience questionnaire’. The health professionals or team will be asked to comment upon: services and care provided, including assessments, interventions and referrals undertaken; information provided; length and quality of interactions with the patient and timeliness in provision of care. The questionnaire will use a five-point Likert scale and is expected to take 20 min to complete. Fourth, the research team will collect, from the accrediting bodies, each participating HCO's accreditation surveyor report and outcomes.
A case study of each patient experience will be constructed using the data collected from the three tools and documentary analysis33 of the associated accreditation report and outcomes. Case study methodology, framed by complexity theory, will be used to seek to understand the system as an integrated whole.49 Within the case study framework, the analysis activity will be the same as that defined for the retrospective analysis of the development of the consumer standard.
The ACCREDIT-SCI study has been approved by the UNSW HREC (HREC 10274). Study information sheets and consent forms explaining activities, processes and participant roles have been developed. A research team contact document has also been finalised for distribution to enable participants to raise questions or concerns. In accordance with UNSWHREC guidelines: complaints will be systematically recorded and actioned; prior to publication or presentation participant information will be de-identified; findings will be made available to participants and research data will be stored in a secure location, accessible only to the research team, and deleted after a minimum of 7 years.
Dissemination of study findings by the research team will occur through a variety of forms. Seminars will be conducted, with targeted invitations made to partners and stakeholders. Presentations will be made at national and international conferences, and journal articles developed for academic and industry publications. Additionally, information, updates and outcomes will be made via UNSW and partner websites.
The investigation of consumer involvement in health service accreditation programmes is an important task that addresses a significant gap in the knowledge base. The ACCREDIT-SCI study protocol details two studies—one examining the effectiveness of a standard for consumer participation and the other exploring how patient experiences vary across a range of settings with differing accreditation results—that are designed to complement each other and meet this need. The ACCREDIT research collaboration engages and uses the skills and experience of a diverse range of academics, accrediting agency personnel and policy makers. In this way, the collaboration provides capacity to implement the ACCREDIT-SCI study protocol and distribute findings across the continuum of the healthcare industry.
We acknowledge the staff of the industry partners (ACHS, AGPAL and ACSAA) and the quality improvement agencies (ACSQHC and CEC) that are providing support for the project.
Contributors: All authors contributed to the writing of the ACCREDIT-SCI study protocol and will assist implementation of the research activities.
Funding: This research is supported under Australian Research Council's Linkage Projects scheme (LP100200586). The ARC has peer-reviewed and funded the proposal but has no role in the implementation of the studies. The industry partners may contribute to the implementation of the studies, analysis and interpretation of findings and presentation of results. However, the final responsibility and decision-making for all research matters, including to publish papers in journals, resides with the UNSW.
Ethics approval: The UNSW Human Research Ethics Committee provided approval for the studies (HREC 10274).
Competing interests: None.
Provenance and peer review: Not commissioned; internally peer reviewed.