Search tips
Search criteria

Results 1-24 (24)

Clipboard (0)

Select a Filter Below

Year of Publication
Document Types
2.  Development of the AGREE II, part 1: performance, usefulness and areas for improvement 
We undertook research to improve the AGREE instrument, a tool used to evaluate guidelines. We tested a new seven-point scale, evaluated the usefulness of the original items in the instrument, investigated evidence to support shorter, tailored versions of the tool, and identified areas for improvement.
We report on one component of a larger study that used a mixed design with four factors (user type, clinical topic, guideline and condition). For the analysis reported in this article, we asked participants to read a guideline and use the AGREE items to evaluate it based on a seven-point scale, to complete three outcome measures related to adoption of the guideline, and to provide feedback on the instrument’s usefulness and how to improve it.
Guideline developers gave lower-quality ratings than did clinicians or policy-makers. Five of six domains were significant predictors of participants’ outcome measures (p < 0.05). All domains and items were rated as useful by stakeholders (mean scores > 4.0) with no significant differences by user type (p > 0.05). Internal consistency ranged between 0.64 and 0.89. Inter-rater reliability was satisfactory. We received feedback on how to improve the instrument.
Quality ratings of the AGREE domains were significant predictors of outcome measures associated with guideline adoption: guideline endorsements, overall intentions to use guidelines, and overall quality of guidelines. All AGREE items were assessed as useful in determining whether a participant would use a guideline. No clusters of items were found more useful by some users than others. The measurement properties of the seven-point scale were promising. These data contributed to the refinements and release of the AGREE II.
PMCID: PMC2900328  PMID: 20513780
3.  Development of the AGREE II, part 2: assessment of validity of items and tools to support application 
We established a program of research to improve the development, reporting and evaluation of practice guidelines. We assessed the construct validity of the items and user’s manual in the β version of the AGREE II.
We designed guideline excerpts reflecting high-and low-quality guideline content for 21 of the 23 items in the tool. We designed two study packages so that one low-quality and one high-quality version of each item were randomly assigned to each package. We randomly assigned 30 participants to one of the two packages. Participants reviewed and rated the guideline content according to the instructions of the user’s manual and completed a survey assessing the manual.
In all cases, content designed to be of high quality was rated higher than low-quality content; in 18 of 21 cases, the differences were significant (p < 0.05). The manual was rated by participants as appropriate, easy to use, and helpful in differentiating guidelines of varying quality, with all scores above the mid-point of the seven-point scale. Considerable feedback was offered on how the items and manual of the β-AGREE II could be improved.
The validity of the items was established and the user’s manual was rated as highly useful by users. We used these results and those of our study presented in part 1 to modify the items and user’s manual. We recommend AGREE II (available at as the revised standard for guideline development, reporting and evaluation.
PMCID: PMC2900368  PMID: 20513779
4.  How does context influence collaborative decision-making for health services planning, delivery and evaluation? 
Collaboration among researchers (clinician, non-clinician) and decision makers (managers, policy-makers, clinicians), referred to as integrated knowledge translation (IKT), enhances the relevance and use of research, leading to improved decision-making, policies, practice, and health care outcomes. However IKT is not widely practiced due to numerous challenges. This research explored how context influenced IKT as a means of identifying how IKT could be strengthened.
This research investigated IKT in three health services programs for colon cancer screening, prostate cancer diagnosis, and the treatment of pancreatic cancer. Qualitative methods were used to explore contextual factors that influenced how IKT occurred, and its impact. Data were collected between September 1, 2012 and May 15, 2013 from relevant documents, observation of meetings, and interviews with researchers and decision-makers, analyzed using qualitative methods, and integrated.
Data were analyzed from 39 documents, observation of 6 meetings, and 36 interviews. IKT included interaction at meetings, joint undertaking of research, and development of guidelines. IKT was most prevalent in one program with leadership, clear goals, dedicated funding and other infrastructural resources, and an embedded researcher responsible for, and actively engaged in IKT. This program achieved a variety of social, research and health service outcomes despite mixed individual views about the value of IKT and the absence of a programmatic culture of IKT. Participants noted numerous challenges including lack of time and incentives, and recommendations to support IKT. A conceptual framework of factors that influence IKT and associated outcomes was generated, and can be used by others to plan or evaluate IKT.
The findings can be applied by researchers, clinicians, managers or policy-makers to plan or improve collaborative decision-making for health services planning, delivery, evaluation or quality improvement. Further research is needed to explore whether these findings are widespread, and further understand how IKT can be optimized.
Electronic supplementary material
The online version of this article (doi:10.1186/s12913-014-0545-x) contains supplementary material, which is available to authorized users.
PMCID: PMC4239386  PMID: 25407487
Integrated knowledge translation; Health services research; Collaboration; Qualitative methods
5.  The AGREE Enterprise: a decade of advancing clinical practice guidelines 
The original AGREE (Appraisal of Guidelines for REsearch and Evaluation) Instrument was published in 2003, and its revision, the AGREE II, in 2009. Together, they filled an important gap in the guideline and quality of care fields. Ten years later, the AGREE Enterprise reflects on a trajectory of projects and international collaboration that have contributed to advancing the science and quality of practice guidelines and the uptake of AGREE/AGREE II.
The AGREE Enterprise has undertaken activities to improve the tool and to develop resources to support its use. Since 2003, the uptake and adoption of AGREE by the international community has been swift and broad. A total of 33 language translations of the original AGREE Instrument and the current AGREE II are available and were initiated by the international community. A recent scan of the published literature identified over 600 articles that referenced the AGREE tools. The AGREE tools have been widely received and applied, with several organizations having incorporated the AGREE as part of their formal practice guideline programs. Since its redevelopment in 2010, the AGREE Enterprise website ( continues to experience steady increases in visitors per month and currently has over 10,000 registered users.
The AGREE Enterprise has contributed to the advancements of guidelines through research activities and international participation by scientific and user communities. As we enter a new decade, we look forward to ongoing collaborations and contributing to further advancements to improve quality of care and health care systems.
PMCID: PMC4148001  PMID: 25123781
AGREE tools; AGREE II; AGREE Instrument; Clinical practice guidelines; Quality of care
6.  A framework of the desirable features of guideline implementation tools (GItools): Delphi survey and assessment of GItools 
Guidelines are the foundation for healthcare planning, delivery and quality improvement but are not consistently implemented. Few guidelines are accompanied by guideline implementation tools (GItools). Users have requested GItools, and developers have requested guidance on how to develop GItools. First it is necessary to characterize GItools. The purpose of this research was to generate a framework of desirable features of GItools.
Items representing desirable GItool features were generated by a cross-sectional survey of the international guideline community. Items were confirmed by 31 guideline developers, implementers and researchers in a two-round Delphi survey administered on the Internet. The resulting GItool framework was applied with a sample of GItools accompanying guidelines identified in the National Guideline Clearinghouse.
The cross-sectional survey was completed by 96 respondents from Australia, Canada, the United Kingdom, the United States, The Netherlands, and various other countries. Seven of nine items were rated by the majority as desirable. A total of 31 panelists from 10 countries including Australia, Canada, Germany, New Zealand, Peru, Saudi Arabia, Spain, the United Kingdom, and the United States took part in a two-round Delphi survey. Ten items achieved consensus as desirable GItool features in round #1, and two additional items in round #2. A total of 13 GItools for Resource Planning, Implementation and Evaluation were identified among 149 guidelines on a variety of clinical topics (8.7%). Many GItools did not possess features considered desirable.
Inclusion of higher quality GItools in guidelines is needed to support user adoption of guidelines. The GItool framework can serve as the basis for evaluating and adapting existing GItools, or developing new GItools. Further research is needed to validate the framework, develop and implement instruments by which developers can apply the framework, and specify which guidelines should be accompanied by GItools.
PMCID: PMC4244063  PMID: 25091091
Guidelines; Implementation
8.  A mixed methods approach to understand variation in lung cancer practice and the role of guidelines 
Practice pattern data demonstrate regional variation and lower than expected rates of adherence to practice guideline (PG) recommendations for the treatment of stage II/IIIA resected and stage IIIA/IIIB unresected non-small cell lung cancer (NSCLC) patients in Ontario, Canada. This study sought to understand how clinical decisions are made for the treatment of these patients and the role of PGs.
Surveys and key informant interviews were undertaken with clinicians and administrators.
Participants reported favorable ratings for PGs and the evidentiary bases underpinning them. The majority of participants agreed more patients should have received treatment and that regional variation is problematic. Participants estimated that up to 30% of patients are not good candidates for treatment and up to 20% of patients refuse treatment. The most common barrier to implementing PGs was the lack of organizational support by clinical administrative leadership. There was concern that the trial results underpinning the PG recommendations were not generalizable to the typical patients seen in clinic. The qualitative analysis yielded five themes related to physicians’ decision making: the unique patient, the unique physician, the family, the clinical team, and the clinical evidence. A dynamic interplay between these factors exists.
Our study demonstrates the challenges inherent in (i) the complexity of clinical decision making; (ii) how quality of care problems are perceived and operationalized; and (iii) the clinical appropriateness and utility of PG recommendations. We argue that systematic and rigorous methodologies to help decision makers mitigate or negotiate these challenges are warranted.
PMCID: PMC3998045  PMID: 24655753
Practice guidelines; Clinical decision making; Practice variation; Non-small cell lung cancer; Quality of care
9.  How can diagnostic assessment programs be implemented to enhance inter-professional collaborative care for cancer? 
Inter-professional collaborative care (ICC) for cancer leads to multiple system, organizational, professional, and patient benefits, but is limited by numerous challenges. Empirical research on interventions that promote or enable ICC is sparse so guidance on how to achieve ICC is lacking. Research shows that ICC for diagnosis could be improved. Diagnostic assessment programs (DAPs) appear to be a promising model for enabling ICC. The purpose of this study was to explore how DAP structure and function enable ICC, and whether that may be associated with organizational and clinical outcomes.
A case study approach will be used to explore ICC among eight DAPs that vary by type of cancer (lung, breast), academic status, and geographic region. To describe DAP function and outcomes, and gather information that will enable costing, recommendations expressed in DAP standards and clinical guidelines will be assessed through retrospective observational study. Data will be acquired from databases maintained by participating DAPs and the provincial cancer agency, and confirmed by and supplemented with review of medical records. We will conduct a pilot study to explore the feasibility of estimating the incremental cost-effectiveness ratio using person-level data from medical records and other sources. Interviews will be conducted with health professionals, staff, and referring physicians from each DAP to learn about barriers and facilitators of ICC. Qualitative methods based on a grounded approach will be used to guide sampling, data collection and analysis.
Findings may reveal opportunities for unique structures, interventions or tools that enable ICC that could be developed, implemented, and evaluated through future research. This information will serve as a formative needs assessment to identify the nature of ongoing or required improvements, which can be directly used by our decision maker collaborators, and as a framework by policy makers, cancer system managers, and DAP managers elsewhere to strategically plan for and implement diagnostic cancer services.
PMCID: PMC3884012  PMID: 24383742
Inter-professional collaborative care; Multidisciplinary care team; Inter-professional relations; Communication; Cooperative behavior; Diagnostic assessment program; Breast cancer; Lung cancer
10.  Integrating guideline development and implementation: analysis of guideline development manual instructions for generating implementation advice 
Guidelines are important tools that inform healthcare delivery based on best available research evidence. Guideline use is in part based on quality of the guidelines, which includes advice for implementation and has been shown to vary. Others hypothesized this is due to limited instructions in guideline development manuals. The purpose of this study was to examine manual instructions for implementation advice.
We used a directed and summative content analysis approach based on an established framework of guideline implementability. Six manuals identified by another research group were examined to enumerate implementability domains and elements.
Manuals were similar in content but lacked sufficient detail in particular domains. Most frequently this was Accomodation, which includes information that would help guideline users anticipate and/or overcome organizational and system level barriers. In more than one manual, information was also lacking for Communicability, information that would educate patients or facilitate their involvement in shared decision making, and Applicability, or clinical parameters to help clinicians tailor recommendations for individual patients.
Most manuals that direct guideline development lack complete information about incorporating implementation advice. These findings can be used by those who developed the manuals to consider expanding their content in these domains. It can also be used by guideline developers as they plan the content and implementation of their guidelines so that the two are integrated. New approaches for guideline development and implementation may need to be developed. Use of guidelines might be improved if they included implementation advice, but this must be evaluated through ongoing research.
PMCID: PMC3457906  PMID: 22824094
Guideline development; Guideline implementation; Implementability
11.  The guideline implementability research and application network (GIRAnet): an international collaborative to support knowledge exchange: study protocol 
Modifying the format and content of guidelines may facilitate their use and lead to improved quality of care. We reviewed the medical literature to identify features desired by different users and associated with guideline use to develop a framework of implementability and found that most guidelines do not contain these elements. Further research is needed to develop and evaluate implementability tools.
We are launching the Guideline Implementability Research and Application Network (GIRAnet) to enable the development and testing of implementability tools in three domains: Resource Implications, Implementation, and Evaluation. Partners include the Guidelines International Network (G-I-N) and its member guideline developers, implementers, and researchers. In phase one, international guidelines will be examined to identify and describe exemplar tools. Indication-specific and generic tools will populate a searchable repository. In phase two, qualitative analysis of cognitive interviews will be used to understand how developers can best integrate implementability tools in guidelines and how health professionals use them for interpreting and applying guidelines. In phase three, a small-scale pilot test will assess the impact of implementability tools based on quantitative analysis of chart-based behavioural outcomes and qualitative analysis of interviews with participants. The findings will be used to plan a more comprehensive future evaluation of implementability tools.
Infrastructure funding to establish GIRAnet will be leveraged with the in-kind contributions of collaborating national and international guideline developers to advance our knowledge of implementation practice and science. Needs assessment and evaluation of GIRAnet will provide a greater understanding of how to develop and sustain such knowledge-exchange networks. Ultimately, by facilitating use of guidelines, this research may lead to improved delivery and outcomes of patient care.
PMCID: PMC3338081  PMID: 22471937
Guidelines; Guideline development; Guideline implementation; Research networks; Knowledge exchange
12.  The landscape of knowledge translation interventions in cancer control: What do we know and where to next? A review of systematic reviews 
Effective implementation strategies are needed to optimize advancements in the fields of cancer diagnosis, treatment, survivorship, and end-of-life care. We conducted a review of systematic reviews to better understand the evidentiary base of implementation strategies in cancer control.
Using three databases, we conducted a search and identified English-language systematic reviews published between 2005 and 2010 that targeted consumer, professional, organizational, regulatory, or financial interventions, tested exclusively or partially in a cancer context (primary focus); generic or non-cancer-specific reviews were also considered. Data were extracted, appraised, and analyzed by members of the research team, and research ideas to advance the field were proposed.
Thirty-four systematic reviews providing 41 summaries of evidence on 19 unique interventions comprised the evidence base. AMSTAR quality ratings ranged between 2 and 10. Team members rated most of the interventions as promising and in need of further research, and 64 research ideas were identified.
While many interventions show promise of effectiveness in the cancer-control context, few reviews were able to conclude definitively in favor of or against a specific intervention. We discuss the complexity of implementation research and offer suggestions to advance the science in this area.
PMCID: PMC3284444  PMID: 22185329
13.  What implementation interventions increase cancer screening rates? a systematic review 
Appropriate screening may reduce the mortality and morbidity of colorectal, breast, and cervical cancers. However, effective implementation strategies are warranted if the full benefits of screening are to be realized. As part of a larger agenda to create an implementation guideline, we conducted a systematic review to evaluate interventions designed to increase the rate of breast, cervical, and colorectal cancer (CRC) screening. The interventions considered were: client reminders, client incentives, mass media, small media, group education, one-on-one education, reduction in structural barriers, reduction in out-of-pocket costs, provider assessment and feedback interventions, and provider incentives. Our primary outcome, screening completion, was calculated as the overall median post-intervention absolute percentage point (PP) change in completed screening tests.
Our first step was to conduct an iterative scoping review in the research area. This yielded three relevant high-quality systematic reviews. Serving as our evidentiary foundation, we conducted a formal update. Randomized controlled trials and cluster randomized controlled trials, published between 2004 and 2010, were searched in MEDLINE, EMBASE and PSYCHinfo.
The update yielded 66 studies new eligible studies with 74 comparisons. The new studies ranged considerably in quality. Client reminders, small media, and provider audit and feedback appear to be effective interventions to increase the uptake of screening for three cancers. One-on-one education and reduction of structural barriers also appears effective, but their roles with CRC and cervical screening, respectively, are less established. More study is required to assess client incentives, mass media, group education, reduction of out-of-pocket costs, and provider incentive interventions.
The new evidence generally aligns with the evidence and conclusions from the original systematic reviews. This review served as the evidentiary foundation for an implementation guideline. Poor reporting, lack of precision and consistency in defining operational elements, and insufficient consideration of context and differences among populations are areas for additional research.
PMCID: PMC3197548  PMID: 21958556
14.  Effective interventions to facilitate the uptake of breast, cervical and colorectal cancer screening: an implementation guideline 
Appropriate screening may reduce the mortality and morbidity of colorectal, breast, and cervical cancers. Several high-quality systematic reviews and practice guidelines exist to inform the most effective screening options. However, effective implementation strategies are warranted if the full benefits of screening are to be realized. We developed an implementation guideline to answer the question: What interventions have been shown to increase the uptake of cancer screening by individuals, specifically for breast, cervical, and colorectal cancers?
A guideline panel was established as part of Cancer Care Ontario's Program in Evidence-based Care, and a systematic review of the published literature was conducted. It yielded three foundational systematic reviews and an existing guidance document. We conducted updates of these reviews and searched the literature published between 2004 and 2010. A draft guideline was written that went through two rounds of review. Revisions were made resulting in a final set of guideline recommendations.
Sixty-six new studies reflecting 74 comparisons met eligibility criteria. They were generally of poor to moderate quality. Using these and the foundational documents, the panel developed a draft guideline. The draft report was well received in the two rounds of review with mean quality scores above four (on a five-point scale) for each of the items. For most of the interventions considered, there was insufficient evidence to support or refute their effectiveness. However, client reminders, reduction of structural barriers, and provision of provider assessment and feedback were recommended interventions to increase screening for at least two of three cancer sites studied. The final guidelines also provide advice on how the recommendations can be used and future areas for research.
Using established guideline development methodologies and the AGREE II as our methodological frameworks, we developed an implementation guideline to advise on interventions to increase the rate of breast, cervical and colorectal cancer screening. While advancements have been made in these areas of implementation science, more investigations are warranted.
PMCID: PMC3222606  PMID: 21958602
15.  E-learning interventions are comparable to user's manual in a randomized trial of training strategies for the AGREE II 
Practice guidelines (PGs) are systematically developed statements intended to assist in patient and practitioner decisions. The AGREE II is the revised tool for PG development, reporting, and evaluation, comprised of 23 items, two global rating scores, and a new User's Manual. In this study, we sought to develop, execute, and evaluate the impact of two internet interventions designed to accelerate the capacity of stakeholders to use the AGREE II.
Participants were randomized to one of three training conditions. 'Tutorial'--participants proceeded through the online tutorial with a virtual coach and reviewed a PDF copy of the AGREE II. 'Tutorial + Practice Exercise'--in addition to the Tutorial, participants also appraised a 'practice' PG. For the practice PG appraisal, participants received feedback on how their scores compared to expert norms and formative feedback if scores fell outside the predefined range. 'AGREE II User's Manual PDF (control condition)'--participants reviewed a PDF copy of the AGREE II only. All participants evaluated a test PG using the AGREE II. Outcomes of interest were learners' performance, satisfaction, self-efficacy, mental effort, time-on-task, and perceptions of AGREE II.
No differences emerged between training conditions on any of the outcome measures.
We believe these results can be explained by better than anticipated performance of the AGREE II PDF materials (control condition) or the participants' level of health methodology and PG experience rather than the failure of the online training interventions. Some data suggest the online tools may be useful for trainees new to this field; however, this requires further study.
PMCID: PMC3162563  PMID: 21791080
16.  How can we improve guideline use? A conceptual framework of implementability 
Guidelines continue to be underutilized, and a variety of strategies to improve their use have been suboptimal. Modifying guideline features represents an alternative, but untested way to promote their use. The purpose of this study was to identify and define features that facilitate guideline use, and examine whether and how they are included in current guidelines.
A guideline implementability framework was developed by reviewing the implementation science literature. We then examined whether guidelines included these, or additional implementability elements. Data were extracted from publicly available high quality guidelines reflecting primary and institutional care, reviewed independently by two individuals, who through discussion resolved conflicts, then by the research team.
The final implementability framework included 22 elements organized in the domains of adaptability, usability, validity, applicability, communicability, accommodation, implementation, and evaluation. Data were extracted from 20 guidelines on the management of diabetes, hypertension, leg ulcer, and heart failure. Most contained a large volume of graded, narrative evidence, and tables featuring complementary clinical information. Few contained additional features that could improve guideline use. These included alternate versions for different users and purposes, summaries of evidence and recommendations, information to facilitate interaction with and involvement of patients, details of resource implications, and instructions on how to locally promote and monitor guideline use. There were no consistent trends by guideline topic.
Numerous opportunities were identified by which guidelines could be modified to support various types of decision making by different users. New governance structures may be required to accommodate development of guidelines with these features. Further research is needed to validate the proposed framework of guideline implementability, develop methods for preparing this information, and evaluate how inclusion of this information influences guideline use.
PMCID: PMC3072935  PMID: 21426574
17.  Protocol: developing a conceptual framework of patient mediated knowledge translation, systematic review using a realist approach 
Patient involvement in healthcare represents the means by which to achieve a healthcare system that is responsive to patient needs and values. Characterization and evaluation of strategies for involving patients in their healthcare may benefit from a knowledge translation (KT) approach. The purpose of this knowledge synthesis is to develop a conceptual framework for patient-mediated KT interventions.
A preliminary conceptual framework for patient-mediated KT interventions was compiled to describe intended purpose, recipients, delivery context, intervention, and outcomes. A realist review will be conducted in consultation with stakeholders from the arthritis and cancer fields to explore how these interventions work, for whom, and in what contexts. To identify patient-mediated KT interventions in these fields, we will search MEDLINE, the Cochrane Library, and EMBASE from 1995 to 2010; scan references of all eligible studies; and examine five years of tables of contents for journals likely to publish quantitative or qualitative studies that focus on developing, implementing, or evaluating patient-mediated KT interventions. Screening and data collection will be performed independently by two individuals.
The conceptual framework of patient-mediated KT options and outcomes could be used by healthcare providers, managers, educationalists, patient advocates, and policy makers to guide program planning, service delivery, and quality improvement and by us and other researchers to evaluate existing interventions or develop new interventions. By raising awareness of options for involving patients in improving their own care, outcomes based on using a KT approach may lead to greater patient-centred care delivery and improved healthcare outcomes.
PMCID: PMC3076239  PMID: 21426573
18.  Canadian Thoracic Society: Presenting a new process for clinical practice guideline production 
A key mandate of the Canadian Thoracic Society (CTS) is to promote evidence-based respiratory care through clinical practice guidelines (CPGs). To improve the quality and validity of the production, dissemination and implementation of its CPGs, the CTS has revised its guideline process and has created the Canadian Respiratory Guidelines Committee to oversee this process. The present document outlines the basic methodological tools and principles of the new CTS guideline production process. Important features include standard methods for choosing and formulating optimal questions and for finding, appraising, and summarizing the evidence; use of the Grading of Recommendations Assessment, Development and Evaluation system for rating the quality of evidence and strength of recommendations; use of the Appraisal of Guidelines for Research and Evaluation instrument for quality control during and after guideline development and for appraisal of other guidelines; use of the ADAPTE process for adaptation of existing guidelines to the local context; and use of the GuideLine Implementability Appraisal tool to augment implementability of guidelines. The CTS has also committed to develop guidelines in new areas, an annual guideline review cycle, and a new formal process for dissemination and implementation. Ultimately, it is anticipated that these changes will have a significant impact on the quality of care and clinical outcomes of individuals suffering from respiratory diseases across Canada.
PMCID: PMC2807796  PMID: 20011719
Clinical practice guideline; Evidence-based medicine; Guideline adherence; Practice guidelines
19.  Recruitment of multiple stakeholders to health services research: Lessons from the front lines 
Self-administered surveys are an essential methodological tool for health services and knowledge translation research, and engaging end-users of the research is critical. However, few documented accounts of the efforts invested in recruitment of multiple different stakeholders to one health services research study exist. Here, we highlight the challenges of recruiting key stakeholders (policy-makers, clinicians, guideline developers) to a Canadian Institutes of Health Research (CIHR) funded health services research (HSR) study aimed to develop an updated and refined version of a guideline appraisal tool, the AGREE.
Using evidence-based methods of recruitment, our goal was to recruit 192 individuals: 80 international guideline developers, 80 Canadian clinicians and 32 Canadian policy/decision-makers. We calculated the participation rate and the recruitment efficiency.
We mailed 873 invitation letters. Of 838 approached, our participation rate was 29%(240) and recruitment efficiency, 19%(156). One policy-maker manager did not allow policy staff to participate in the study.
Based on the results from this study, we suggest that future studies aiming to engage similar stakeholders in HSR over sample by at least 5 times to achieve their target sample size and allow for participant withdrawals. We need continued efforts to communicate the value of research between researchers and end-users of research (policy-makers, clinicians, and other researchers), integration of participatory research strategies, and promotion of the value of end-user involvement in research. Future research to understand methods of improving recruitment efficiency and engaging key stakeholders in HSR is warranted.
PMCID: PMC2880290  PMID: 20465795
20.  A randomized trial to evaluate e-learning interventions designed to improve learner's performance, satisfaction, and self-efficacy with the AGREE II 
Practice guidelines (PGs) are systematically developed statements intended to assist in patient, practitioner, and policy decisions. The AGREE II is the revised and updated standard tool for guideline development, reporting and evaluation. It is comprised of 23 items and a user's Manual. The AGREE II is ready for use.
To develop, execute, and evaluate the impact of two internet-based educational interventions designed to accelerate the capacity of stakeholders to use the AGREE II: a multimedia didactic tutorial with a virtual coach, and a higher intensity training program including both the didactic tutorial and an interactive practice exercise component.
Participants (clinicians, developers, and policy makers) will be randomly assigned to one of three conditions. Condition one, didactic tutorial -- participants will go through the on-line AGREE II tutorial supported by a virtual coach and review of the AGREE II prior to appraising the test PG. Condition two, tutorial + practice -- following the multimedia didactic tutorial with a virtual coach, participants will review the on-line AGREE II independently and use it to appraise a practice PG. Upon entering their AGREE II score for the practice PG, participants will be given immediate feedback on how their score compares to expert norms. If their score falls outside a predefined range, the participant will receive a series of hints to guide the appraisal process. Participants will receive an overall summary of their performance appraising the PG compared to expert norms. Condition three, control arm -- participants will receive a PDF copy of the AGREE II for review and to appraise the test PG on-line. All participants will then rate one of ten test PGs with the AGREE II. The outcomes of interest are learners' performance, satisfaction, self-efficacy, mental effort, and time-on-task; comparisons will be made across each of the test groups.
Our research will test innovative educational interventions of various intensities and instructional design to promote the adoption of AGREE II and to identify those strategies that are most effective for training. The results will facilitate international capacity to apply the AGREE II accurately and with confidence and to enhance the overall guideline enterprise.
PMCID: PMC2868048  PMID: 20403188
21.  An exploration of how guideline developer capacity and guideline implementability influence implementation and adoption: study protocol 
Practice guidelines can improve health care delivery and outcomes but several issues challenge guideline adoption, including their intrinsic attributes, and whether and how they are implemented. It appears that guideline format may influence accessibility and ease of use, which may overcome attitudinal barriers of guideline adoption, and appear to be important to all stakeholders. Guideline content may facilitate various forms of decision making about guideline adoption relevant to different stakeholders. Knowledge and attitudes about, and incentives and capacity for implementation on the part of guideline sponsors may influence whether and how they develop guidelines containing these features, and undertake implementation. Examination of these issues may yield opportunities to improve guideline adoption.
The attributes hypothesized to facilitate adoption will be expanded by thematic analysis, and quantitative and qualitative summary of the content of international guidelines for two primary care (diabetes, hypertension) and institutional care (chronic ulcer, chronic heart failure) topics. Factors that influence whether and how guidelines are implemented will be explored by qualitative analysis of interviews with individuals affiliated with guideline sponsoring agencies.
Previous research examined guideline implementation by measuring rates of compliance with recommendations or associated outcomes, but this produced little insight on how the products themselves, or their implementation, could be improved. This research will establish a theoretical basis upon which to conduct experimental studies to compare the cost-effectiveness of interventions that enhance guideline development and implementation capacity. Such studies could first examine short-term outcomes predictive of guideline utilization, such as recall, attitude toward, confidence in, and adoption intention. If successful, then long-term objective outcomes reflecting the adoption of processes and associated patient care outcomes could be evaluated.
PMCID: PMC3224968  PMID: 19573246
22.  Written informed consent and selection bias in observational studies using medical records: systematic review 
Objectives To determine whether informed consent introduces selection bias in prospective observational studies using data from medical records, and consent rates for such studies.
Design Systematic review.
Data sources Embase, Medline, and the Cochrane Library up to March 2008, reference lists from pertinent articles, and searches of electronic citations.
Study selection Prospective observational studies reporting characteristics of participants and non-participants approached for informed consent to use their medical records. Studies were selected independently in duplicate; a third reviewer resolved disagreements.
Data extraction Age, sex, race, education, income, or health status of participants and non-participants, the participation rate in each study, and susceptibility of these calculations to threats of selection and reporting bias.
Results Of 1650 citations 17 unique studies met inclusion criteria and had analysable data. Across all outcomes there were differences between participants and non-participants; however, there was a lack of consistency in the direction and the magnitude of effect. Of 161?604 eligible patients, 66.9% consented to use of data from their medical records.
Conclusions Significant differences between participants and non-participants may threaten the validity of results from observational studies that require consent for use of data from medical records. To ensure that legislation on privacy does not unduly bias observational studies using medical records, thoughtful decision making by research ethics boards on the need for mandatory consent is necessary.
PMCID: PMC2769263  PMID: 19282440
23.  Evaluating the role of quality assessment of primary studies in systematic reviews of cancer practice guidelines 
The purpose of this study was to evaluate the role of study quality assessment of primary studies in cancer practice guidelines.
Reliable and valid study quality assessment scales were sought and applied to published reports of trials included in systematic reviews of cancer guidelines. Sensitivity analyses were performed to evaluate the relationship between quality scores and pooled odds ratios (OR) for mortality and need for blood transfusion.
Results found that that whether trials were classified as high or low quality depended on the scale used to assess them. Although the results of the sensitivity analyses found some variation in the ORs observed, the confidence intervals (CIs) of the pooled effects from each of the analyses of high quality trials overlapped with the CI of the pooled odds of all trials. Quality score was not predictive of pooled ORs studied here.
Had sensitivity analyses based on study quality been conducted prospectively, it is highly unlikely that different conclusions would have been found or that different clinical recommendations would have emerged in the guidelines.
PMCID: PMC553981  PMID: 15715916
24.  Can surveying practitioners about their practices help identify priority clinical practice guideline topics? 
Clinical practice guidelines are systematically developed statements designed to assist in patient and physician clinical decision making for specific clinical circumstances. In order to establish which guideline topics are priorities, practitioners were surveyed regarding their current practice.
One hundred ninety-seven practitioners in Ontario, Canada were mailed a survey exploring their current practice or opinion regarding the prophylactic use of anticonvulsant drugs in patients with malignant glioma who had never had a seizure. The survey consisted of seven questions regarding the relevance of a guideline on the subject to the practitioner's practice, the proportion of clinical cases involving anticonvulsant use, knowledge of existing guidelines on this topic, interest in reviewing a completed practice guideline and three clinical scenarios.
There were 122 respondents who returned the survey (62% rate of return). Eighty percent of the practitioners who responded indicated that less than 25% of their clinical cases involved the use of anticonvulsants; however, only 16% of respondents indicated that a practice guideline would be irrelevant to their practice. Eighty percent of respondents volunteered to review a draft version of a practice guideline on the use of anticonvulsants. The survey presented the practitioners with three scenarios where anticonvulsants in patients with brain tumours may be appropriate: peri-operatively in patients without seizures, postoperatively in patients currently using anticonvulsants, and thirdly in patients not currently using anticonvulsants or undergoing surgery. In contrast to the third situation, the first two situations yielded considerable variation in practitioner response.
The survey established that there is some variation present in the current practice of anticonvulsant use in the patients with brain tumours. Whether there is an optimal treatment practice has yet to be determined. Practitioners do seem to feel that a guideline on anticonvulsant use in warranted, and most practitioners would be interested in being part of the guideline development process.
PMCID: PMC317341  PMID: 14687426

Results 1-24 (24)