Search tips
Search criteria

Results 1-25 (1138106)

Clipboard (0)

Related Articles

1.  An exploration of how guideline developer capacity and guideline implementability influence implementation and adoption: study protocol 
Practice guidelines can improve health care delivery and outcomes but several issues challenge guideline adoption, including their intrinsic attributes, and whether and how they are implemented. It appears that guideline format may influence accessibility and ease of use, which may overcome attitudinal barriers of guideline adoption, and appear to be important to all stakeholders. Guideline content may facilitate various forms of decision making about guideline adoption relevant to different stakeholders. Knowledge and attitudes about, and incentives and capacity for implementation on the part of guideline sponsors may influence whether and how they develop guidelines containing these features, and undertake implementation. Examination of these issues may yield opportunities to improve guideline adoption.
The attributes hypothesized to facilitate adoption will be expanded by thematic analysis, and quantitative and qualitative summary of the content of international guidelines for two primary care (diabetes, hypertension) and institutional care (chronic ulcer, chronic heart failure) topics. Factors that influence whether and how guidelines are implemented will be explored by qualitative analysis of interviews with individuals affiliated with guideline sponsoring agencies.
Previous research examined guideline implementation by measuring rates of compliance with recommendations or associated outcomes, but this produced little insight on how the products themselves, or their implementation, could be improved. This research will establish a theoretical basis upon which to conduct experimental studies to compare the cost-effectiveness of interventions that enhance guideline development and implementation capacity. Such studies could first examine short-term outcomes predictive of guideline utilization, such as recall, attitude toward, confidence in, and adoption intention. If successful, then long-term objective outcomes reflecting the adoption of processes and associated patient care outcomes could be evaluated.
PMCID: PMC3224968  PMID: 19573246
2.  Addressing implementation challenges during guideline development – a case study of Swedish national guidelines for methods of preventing disease 
Many of the world’s life threatening diseases (e.g. cancer, heart disease, stroke) could be prevented by eliminating life-style habits such as tobacco use, unhealthy diet, physical inactivity and excessive alcohol use. Incorporating evidence-based research on methods to change unhealthy lifestyle habits in clinical practice would be equally valuable. However gaps between guideline development and implementation are well documented, with implications for health care quality, safety and effectiveness. The development phase of guidelines has been shown to be important both for the quality in guideline content and for the success of implementation. There are, however, indications that guidelines related to general disease prevention methods encounter specific barriers compared to guidelines that are diagnosis-specific. In 2011 the Swedish National board for Health and Welfare launched guidelines with a preventive scope. The aim of this study was to investigate how implementation challenges were addressed during the development process of these disease preventive guidelines.
Seven semi-structured interviews were conducted with members of the guideline development management group. Archival data detailing the guideline development process were also collected and used in the analysis. Qualitative data were analysed using content analysis as the analytical framework.
The study identified several strategies and approaches that were used to address implementation challenges during guideline development. Four themes emerged from the analysis: broad agreements and consensus about scope and purpose; a formalized and structured development procedure; systematic and active involvement of stakeholders; and openness and transparency in the specific guideline development procedure. Additional factors concerning the scope of prevention and the work environment of guideline developers were perceived to influence the possibilities to address implementation issues.
This case study provides examples of how guideline developers perceive and approach the issue of implementation during the development and early launch of prevention guidelines. Models for guideline development could benefit from an initial assessment of how the guideline topic, its target context and stakeholders will affect the upcoming implementation.
PMCID: PMC4308005  PMID: 25608684
Clinical practice guidelines; Development process; Evidence-based public health; Implementation; Disease prevention; Lifestyle change
3.  Inclusion of Ethical Issues in Dementia Guidelines: A Thematic Text Analysis 
PLoS Medicine  2013;10(8):e1001498.
Clinical practice guidelines (CPGs) aim to improve professionalism in health care. However, current CPG development manuals fail to address how to include ethical issues in a systematic and transparent manner. The objective of this study was to assess the representation of ethical issues in general CPGs on dementia care.
Methods and Findings
To identify national CPGs on dementia care, five databases of guidelines were searched and national psychiatric associations were contacted in August 2011 and in June 2013. A framework for the assessment of the identified CPGs' ethical content was developed on the basis of a prior systematic review of ethical issues in dementia care. Thematic text analysis and a 4-point rating score were employed to assess how ethical issues were addressed in the identified CPGs. Twelve national CPGs were included. Thirty-one ethical issues in dementia care were identified by the prior systematic review. The proportion of these 31 ethical issues that were explicitly addressed by each CPG ranged from 22% to 77%, with a median of 49.5%. National guidelines differed substantially with respect to (a) which ethical issues were represented, (b) whether ethical recommendations were included, (c) whether justifications or citations were provided to support recommendations, and (d) to what extent the ethical issues were explained.
Ethical issues were inconsistently addressed in national dementia guidelines, with some guidelines including most and some including few ethical issues. Guidelines should address ethical issues and how to deal with them to help the medical profession understand how to approach care of patients with dementia, and for patients, their relatives, and the general public, all of whom might seek information and advice in national guidelines. There is a need for further research to specify how detailed ethical issues and their respective recommendations can and should be addressed in dementia guidelines.
Please see later in the article for the Editors' Summary
Editors’ Summary
In the past, doctors tended to rely on their own experience to choose the best treatment for their patients. Faced with a patient with dementia (a brain disorder that affects short-term memory and the ability tocarry out normal daily activities), for example, a doctor would use his/her own experience to help decide whether the patient should remain at home or would be better cared for in a nursing home. Similarly, the doctor might have to decide whether antipsychotic drugs might be necessary to reduce behavioral or psychological symptoms such as restlessness or shouting. However, over the past two decades, numerous evidence-based clinical practice guidelines (CPGs) have been produced by governmental bodies and medical associations that aim to improve standards of clinical competence and professionalism in health care. During the development of each guideline, experts search the medical literature for the current evidence about the diagnosis and treatment of a disease, evaluate the quality of that evidence, and then make recommendations based on the best evidence available.
Why Was This Study Done?
Currently, CPG development manuals do not address how to include ethical issues in CPGs. A health-care professional is ethical if he/she behaves in accordance with the accepted principles of right and wrong that govern the medical profession. More specifically, medical professionalism is based on a set of binding ethical principles—respect for patient autonomy, beneficence, non-malfeasance (the “do no harm” principle), and justice. In particular, CPG development manuals do not address disease-specific ethical issues (DSEIs), clinical ethical situations that are relevant to the management of a specific disease. So, for example, a DSEI that arises in dementia care is the conflict between the ethical principles of non-malfeasance and patient autonomy (freedom-to-move-at-will). Thus, healthcare professionals may have to decide to physically restrain a patient with dementia to prevent the patient doing harm to him- or herself or to someone else. Given the lack of guidance on how to address ethical issues in CPG development manuals, in this thematic text analysis, the researchers assess the representation of ethical issues in CPGs on general dementia care. Thematic text analysis uses a framework for the assessment of qualitative data (information that is word-based rather than number-based) that involves pinpointing, examining, and recording patterns (themes) among the available data.
What Did the Researchers Do and Find?
The researchers identified 12 national CPGs on dementia care by searching guideline databases and by contacting national psychiatric associations. They developed a framework for the assessment of the ethical content in these CPGs based on a previous systematic review of ethical issues in dementia care. Of the 31 DSEIs included by the researchers in their analysis, the proportion that were explicitly addressed by each CPG ranged from 22% (Switzerland) to 77% (USA); on average the CPGs explicitly addressed half of the DSEIs. Four DSEIs—adequate consideration of advanced directives in decision making, usage of GPS and other monitoring techniques, covert medication, and dealing with suicidal thinking—were not addressed in at least 11 of the CPGs. The inclusion of recommendations on how to deal with DSEIs ranged from 10% of DSEIs covered in the Swiss CPG to 71% covered in the US CPG. Overall, national guidelines differed substantially with respect to which ethical issues were included, whether ethical recommendations were included, whether justifications or citations were provided to support recommendations, and to what extent the ethical issues were clearly explained.
What Do These Findings Mean?
These findings show that national CPGs on dementia care already address clinical ethical issues but that the extent to which the spectrum of DSEIs is considered varies widely within and between CPGs. They also indicate that recommendations on how to deal with DSEIs often lack the evidence that health-care professionals use to justify their clinical decisions. The researchers suggest that this situation can and should be improved, although more research is needed to determine how ethical issues and recommendations should be addressed in dementia guidelines. A more systematic and transparent inclusion of DSEIs in CPGs for dementia (and for other conditions) would further support the concept of medical professionalism as a core element of CPGs, note the researchers, but is also important for patients and their relatives who might turn to national CPGs for information and guidance at a stressful time of life.
Additional Information
Please access these Web sites via the online version of this summary at
Wikipedia contains a page on clinical practice guidelines (note: Wikipedia is a free online encyclopedia that anyone can edit; available in several languages)
The US National Guideline Clearinghouse provides information on national guidelines, including CPGs for dementia
The Guidelines International Network promotes the systematic development and application of clinical practice guidelines
The American Medical Association provides information about medical ethics; the British Medical Association provides information on all aspects of ethics and includes an essential tool kit that introduces common ethical problems and practical ways to deal with them
The UK National Health Service Choices website provides information about dementia, including a personal story about dealing with dementia
MedlinePlus provides links to additional resources about dementia and about Alzheimers disease, a specific type of dementia (in English and Spanish)
The UK Nuffield Council on Bioethics provides the report Dementia: ethical issues and additional information on the public consultation on ethical issues in dementia care
PMCID: PMC3742442  PMID: 23966839
4.  Do guidelines influence the implementation of health programs? — Uganda’s experience 
A guideline contains processes and procedures intended to guide health service delivery. However, the presence of guidelines may not guarantee their implementation, which may be a result of weaknesses in the development process. This study was undertaken to describe the processes of developing health planning, services management, and clinical guidelines within the health sector in Uganda, with the goal of understanding how these processes facilitate or abate the utility of guidelines.
Qualitative and quantitative research methods were used to collect and analyze data. Data collection was undertaken at the levels of the central Ministry of Health, the district, and service delivery. Qualitative methods included review of documents, observations, and key informant interviews, as well as quantitative aspects included counting guidelines. Quantitative data were analyzed with Microsoft Excel, and qualitative data were analyzed using deductive content thematic analysis.
There were 137 guidelines in the health sector, with programs related to Millennium Development Goals having the highest number (n = 83). The impetus for guideline development was stated in 78% of cases. Several guidelines duplicated content, and some conflicted with each other. The level of consultation varied, and some guidelines did not consider government-wide policies and circumstances at the service delivery level. Booklets were the main format of presentation, which was not tailored to the service delivery level. There was no framework for systematic dissemination, and target users were defined broadly in most cases. Over 60% of guidelines available at the central level were not available at the service delivery level, but there were good examples in isolated cases. There was no framework for systematic monitoring of use, evaluation, and review of guidelines. Suboptimal performance of the supervision framework that would encourage the use of guidelines, assess their utilization, and provide feedback was noted.
Guideline effectiveness is compromised by the development process. To ensure the production of high-quality guidelines, efforts must be employed at the country and regional levels. The regional level can facilitate pooling resources and expertise in knowledge generation, methodology development, guideline repositories, and capacity building. Countries should establish and enforce systems and guidance on guideline development.
PMCID: PMC3534441  PMID: 23068082
Guidelines; Implementation; Health services; Planning; Management; Uganda
5.  Implementing change in primary care practices using electronic medical records: a conceptual framework 
Implementing change in primary care is difficult, and little practical guidance is available to assist small primary care practices. Methods to structure care and develop new roles are often needed to implement an evidence-based practice that improves care. This study explored the process of change used to implement clinical guidelines for primary and secondary prevention of cardiovascular disease in primary care practices that used a common electronic medical record (EMR).
Multiple conceptual frameworks informed the design of this study designed to explain the complex phenomena of implementing change in primary care practice. Qualitative methods were used to examine the processes of change that practice members used to implement the guidelines. Purposive sampling in eight primary care practices within the Practice Partner Research Network-Translating Researching into Practice (PPRNet-TRIP II) clinical trial yielded 28 staff members and clinicians who were interviewed regarding how change in practice occurred while implementing clinical guidelines for primary and secondary prevention of cardiovascular disease and strokes.
A conceptual framework for implementing clinical guidelines into primary care practice was developed through this research. Seven concepts and their relationships were modelled within this framework: leaders setting a vision with clear goals for staff to embrace; involving the team to enable the goals and vision for the practice to be achieved; enhancing communication systems to reinforce goals for patient care; developing the team to enable the staff to contribute toward practice improvement; taking small steps, encouraging practices' tests of small changes in practice; assimilating the electronic medical record to maximize clinical effectiveness, enhancing practices' use of the electronic tool they have invested in for patient care improvement; and providing feedback within a culture of improvement, leading to an iterative cycle of goal setting by leaders.
This conceptual framework provides a mental model which can serve as a guide for practice leaders implementing clinical guidelines in primary care practice using electronic medical records. Using the concepts as implementation and evaluation criteria, program developers and teams can stimulate improvements in their practice settings. Investing in collaborative team development of clinicians and staff may enable the practice environment to be more adaptive to change and improvement.
PMCID: PMC2254645  PMID: 18199330
6.  How Evidence-Based Are the Recommendations in Evidence-Based Guidelines? 
PLoS Medicine  2007;4(8):e250.
Treatment recommendations for the same condition from different guideline bodies often disagree, even when the same randomized controlled trial (RCT) evidence is cited. Guideline appraisal tools focus on methodology and quality of reporting, but not on the nature of the supporting evidence. This study was done to evaluate the quality of the evidence (based on consideration of its internal validity, clinical relevance, and applicability) underlying therapy recommendations in evidence-based clinical practice guidelines.
Methods and Findings
A cross-sectional analysis of cardiovascular risk management recommendations was performed for three different conditions (diabetes mellitus, dyslipidemia, and hypertension) from three pan-national guideline panels (from the United States, Canada, and Europe). Of the 338 treatment recommendations in these nine guidelines, 231 (68%) cited RCT evidence but only 105 (45%) of these RCT-based recommendations were based on high-quality evidence. RCT-based evidence was downgraded most often because of reservations about the applicability of the RCT to the populations specified in the guideline recommendation (64/126 cases, 51%) or because the RCT reported surrogate outcomes (59/126 cases, 47%).
The results of internally valid RCTs may not be applicable to the populations, interventions, or outcomes specified in a guideline recommendation and therefore should not always be assumed to provide high-quality evidence for therapy recommendations.
From an analysis of cardiovascular risk-management recommendations in guidelines produced by pan-national panels, McAlister and colleagues concluded that fewer than half were based on high-quality evidence.
Editors' Summary
Until recently, doctors largely relied on their own experience to choose the best treatment for their patients. Faced with a patient with high blood pressure (hypertension), for example, the doctor had to decide whether to recommend lifestyle changes or to prescribe drugs to reduce the blood pressure. If he or she chose the latter, he or she then had to decide which drug to prescribe, set a target blood pressure, and decide how long to wait before changing the prescription if this target was not reached. But, over the past decade, numerous clinical practice guidelines have been produced by governmental bodies and medical associations to help doctors make treatment decisions like these. For each guideline, experts have searched the medical literature for the current evidence about the diagnosis and treatment of a disease, evaluated the quality of that evidence, and then made recommendations based on the best evidence available.
Why Was This Study Done?
The recommendations made in different clinical practice guidelines vary, in part because they are based on evidence of varying quality. To help clinicians decide which recommendations to follow, some guidelines indicate the strength of their recommendations by grading them, based on the methods used to collect the underlying evidence. Thus, a randomized clinical trial (RCT)—one in which patients are randomly allocated to different treatments without the patient or clinician knowing the allocation—provides higher-quality evidence than a nonrandomized trial. Similarly, internally valid trials—in which the differences between patient groups are solely due to their different treatments and not to other aspects of the trial—provide high-quality evidence. However, grading schemes rarely consider the size of studies and whether they have focused on clinical or so-called “surrogate” measures. (For example, an RCT of a treatment to reduce heart or circulation [“cardiovascular”] problems caused by high blood pressure might have death rate as a clinical measure; a surrogate endpoint would be blood pressure reduction.) Most guidelines also do not consider how generalizable (applicable) the results of a trial are to the populations, interventions, and outcomes specified in the guideline recommendation. In this study, the researchers have investigated the quality of the evidence underlying recommendations for cardiovascular risk management in nine evidence-based clinical practice guides using these additional criteria.
What Did the Researchers Do and Find?
The researchers extracted the recommendations for managing cardiovascular risk from the current US, Canadian, and European guidelines for the management of diabetes, abnormal blood lipid levels (dyslipidemia), and hypertension. They graded the quality of evidence for each recommendation using the Canadian Hypertension Education Program (CHEP) grading scheme, which considers the type of study, its internal validity, its clinical relevance, and how generally applicable the evidence is considered to be. Of 338 evidence-based recommendations, two-thirds were based on evidence collected in internally valid RCTs, but only half of these RCT-based recommendations were based on high-quality evidence. The evidence underlying 64 of the guideline recommendations failed to achieve a high CHEP grade because the RCT data were collected in a population of people with different characteristics to those covered by the guideline. For example, a recommendation to use spironolactone to reduce blood pressure in people with hypertension was based on an RCT in which the participants initially had congestive heart failure with normal blood pressure. Another 59 recommendations were downgraded because they were based on evidence from RCTs that had not focused on clinical measures of effectiveness.
What Do These Findings Mean?
These findings indicate that although most of the recommendations for cardiovascular risk management therapies in the selected guidelines were based on evidence collected in internally valid RCTs, less than one-third were based on high-quality evidence applicable to the populations, treatments, and outcomes specified in guideline recommendations. A limitation of this study is that it analyzed a subset of recommendations in only a few guidelines. Nevertheless, the findings serve to warn clinicians that evidence-based guidelines are not necessarily based on high-quality evidence. In addition, they emphasize the need to make the evidence base underlying guideline recommendations more transparent by using an extended grading system like the CHEP scheme. If this were done, the researchers suggest, it would help clinicians apply guideline recommendations appropriately to their individual patients.
Additional Information.
Please access these Web sites via the online version of this summary at
• Wikipedia contains pages on evidence-based medicine and on clinical practice guidelines (note: Wikipedia is a free online encyclopedia that anyone can edit; available in several languages)
• The National Guideline Clearinghouse provides information on US national guidelines
• The Guidelines International Network promotes the systematic development and application of clinical practice guidelines
• Information is available on the Canadian Hypertension Education Program (CHEP) (in French and English)
• See information on the Grading of Recommendations Assessment, Development and Evaluation (GRADE) working group, an organization that has developed an grading scheme similar to the CHEP scheme (in English, Spanish, French, German, and Italian)
PMCID: PMC1939859  PMID: 17683197
7.  The development of a guideline implementability tool (GUIDE-IT): a qualitative study of family physician perspectives 
BMC Family Practice  2014;15:19.
The potential of clinical practice guidelines has not been realized due to inconsistent adoption in clinical practice. Optimising intrinsic characteristics of guidelines (e.g., its wording and format) that are associated with uptake (as perceived by their end users) may have potential. Using findings from a realist review on guideline uptake and consultation with experts in guideline development, we designed a conceptual version of a future tool called Guideline Implementability Tool (GUIDE-IT). The tool will aim to involve family physicians in the guideline development process by providing a process to assess draft guideline recommendations. This feedback will then be given back to developers to consider when finalizing the recommendations. As guideline characteristics are best assessed by end-users, the objectives of the current study were to explore how family physicians perceive guideline implementability, and to determine what components should comprise the final GUIDE-IT prototype.
We conducted a qualitative study with family physicians inToronto, Ontario. Two experienced investigators conducted one-hour interviews with family physicians using a semi-structured interview guide to 1) elicit feedback on perceptions on guideline implementability; 2) to generate a discussion in response to three draft recommendations; and 3) to provide feedback on the conceptual GUIDE-IT. Sessions were audio taped and transcribed verbatim. Data collection and analysis were guided by content analyses.
20 family physicians participated. They perceived guideline uptake according to facilitators and barriers across 6 categories of guideline implementability (format, content, language, usability, development, and the practice environment). Participants’ feedback on 3 draft guideline recommendations were grouped according to guideline perception, cognition, and agreement. When asked to comment on GUIDE-IT, most respondents believed that the tool would be useful, but urged to involve “regular” or community family physicians in the process, and suggested that an online system would be the most efficient way to deliver it.
Our study identified facilitators and barriers of guideline implementability from the perspective of community and academic family physicians that will be used to build our GUIDE-IT prototype. Our findings build on current knowledge by showing that family physicians perceive guideline uptake mostly according to factors that are in the control of guideline developers.
PMCID: PMC4016596  PMID: 24476491
Guideline implementability; Family practice; Knowledge translation; Qualitative
8.  Developing a checklist for guideline implementation planning: review and synthesis of guideline development and implementation advice 
Developers, users and others have requested or advocated for guidance on how to plan for, and implement guidelines concurrent to their development given that existing resources are lacking such information. The purpose of this research was to develop a guideline implementation planning checklist.
Documents that described or evaluated the processes of planning or undertaking implementation were identified in several publications that had systematically identified such resources, and by searching medical literature databases (MEDLINE, EMBASE). Data that described implementation planning; how to develop guideline versions or tools that would support user implementation; and options and mechanisms for disseminating or implementing guidelines were independently extracted from eligible documents by the principal investigator and a trained research assistant. Data were integrated to create a unique list of guideline implementation planning processes and considerations.
Thirty-five documents were eligible. Of these, 16 (45.7%) provided sparse information on implementation planning, 25 (71.4%) mentioned different versions or tools for implementation, and 30 (85.7%) listed options for dissemination or implementation. None provided instructions for operationalizing implementation strategies. Data were integrated into a multi-item Guideline Implementation Planning Checklist including considerations for implementation planning (12), development of implementation tools (8), types of implementation tools (12), and options for dissemination (11) and implementation (12).
Developers or users can apply the Guideline Implementation Planning Checklist to prepare for and/or undertake guideline implementation. Further development of the checklist is warranted to elaborate on all components. In ongoing research, we will consult with the international guideline community to do so. At the same time, guideline implementation is complex, so developers and users would benefit from training, and by including knowledge translation experts and brokers on implementation planning committees.
Electronic supplementary material
The online version of this article (doi:10.1186/s13012-015-0205-5) contains supplementary material, which is available to authorized users.
PMCID: PMC4329197
Guidelines; Implementation
9.  A self-evaluation tool for integrated care services: the Development Model for Integrated Care applied in practice 
The purpose of the workshop is to show the applications of the Development Model for Integrated Care (DMIC) in practice. This relatively new and validated model, can be used by integrated care practices to evaluate their integrated care, to assess their phase of development and reveal improvement areas. In the workshop the results of the use of the model in three types of integrated care settings in the Netherlands will be presented. Participants are offered practical instruments based on the validated DMIC to use in their own setting and will be introduced to the webbased tool.
To integrate care from multiple providers into a coherent and streamlined client-focused service, a large number of activities and agreements have to be implemented like streamlining information flows and adequate transfers of clients. In the large range of possible activities it is often not clear what essential activities are and where to start or continue. Also, knowledge about how to further develop integrated care services is needed. The Development Model for Integrated Care (DMIC), based on PhD research of Mirella Minkman, describes nine clusters containing in total 89 elements that contribute to the integration of care. The clusters are named: ‘client-centeredness’, ‘delivery system’, ‘performance management’, ‘quality of care’, ‘result-focused learning’, ‘interprofessional teamwork’, ‘roles and tasks’, ‘commitment’, and ‘transparant entrepreneurship’ [1–3]. In 2011 a new digital webbased self-evolution tool which contains the 89 elements grouped in nine clusters was developed. The DMIC also describes four phases of development [4]. The model is empirically validated in practice by assessing the relevance and implementation of the elements and development phases in 84 integrated care services in The Netherlands: in stroke, acute myocardial infarct (AMI), and dementia services. The validation studies are recently published [5, 6]. In 2011 also other integrated care services started using the model [7]. Vilans developed a digital web-based self-evaluation tool for integrated care services based on the DMIC. A palliative care network, four diabetes services, a youth care service and a network for autism used the self-evaluation tool to evaluate the development of their integrated care service. Because of its generic character, the model and tool are believed to be also interesting internationally.
Data sources
In the workshop we will present the results of three studies in integrated diabetes, youth and palliative care. The three projects consist of multiple steps, see below. Workshop participants could also work with the DMIC following these steps.
One: Preparation of the digital self-evolution tool for integrated care services
Although they are very different, the three integrated care services all wanted to gain insight in their development and improvement opportunities. We tailored the digital self-evaluation tool for each specific integrated care services, but for all the basis was the DMIC. Personal accounts for the digital DMIC self-evalution survey were sent to multiple partners working in each integrated care service (4–16 partners).
Two: Use of the online self-evaluation tool each partner of the local integrated care setting evaluated the integrated care by filling in the web-based questionnaire. The tool consists of three parts (A-C) named: general information about the integrated care practice (A); the clusters and elements of the DMIC (B); and the four phases of development (C). The respondents rated the relevance and presence of each element in their integrated care practice. Respondents were asked to estimate in which phase of development their thought their service was.
Three: Analysing the results
Advisers from Vilans, the Centre of excellence for long-term care in the Netherlands, analysed the self-evolution results in cooperation with the integrated care coordinators. The results show the total amount of implemented integrated care elements per cluster in spider graphs and the development phase as calculated by the DMIC model. Suggestions for further development of the integrated care services were analysed and reported.
Four: Discussing the implications for further development
In a workshop with the local integrated care partners the results of the self-evaluation were presented and discussed. We noticed remarkable results and highlight elements for further development. In addition, we gave advice for further development appropriate to the development phase of the integrated care service. Furthermore, the professionals prioritized the elements and decided which elements to start working on. This resulted in a (quality improvement) plan for the further development of the integrated care service.
Five: Reporting results
In a report all the results of the survey (including consensus scores) and the workshops came together. The integrated care coordinators stated that the reports really helped them to assess their improvement strategy. Also, there was insight in the development phase of their service which gave tools for further development.
Case description
The three cases presented are a palliative network, an integrated diabetes services and an integrated care network for youth in the Netherlands. The palliative care network wanted to reflect on their current development, to build a guiding framework for further development of the network. About sixteen professionals within the network worked with the digital self-evaluation tool and the DMIC: home care organisations, welfare organizations, hospice centres, health care organisations, community organizations.
For diabetes care, a Dutch health care insurance company wished to gain insight in the development of the contracted integrated care services to stimulate further development of the services. Professionals of three diabetes integrated care services were invited to fill in the digital self-evaluation tool. Of each integrated care service professionals like a general practitioner, a diabetes nurse, a medical specialist, a dietician and a podiatrist were invited. In youth care, a local health organisation wondered whether the DMIC could be helpful to visualize the results of youth integrated care services at process- and organisational level. The goal of the project was to define indicators at a process- and organisational level for youth care services based on the DMIC. In the future, these indicators might be used to evaluate youth care integrated care services and improve the quality of youth care within the Netherlands.
Conclusions and discussion
It is important for the quality of integrated care services that the involved coordinators, managers and professionals are aware of the development process of the integrated care service and that they focus on elements which can further develop and improve their integrated care. However, we noticed that integrated care services in the Netherlands experience difficulties in developing their integrated care service. It is often not clear what essential activities are to work on and how to further develop the integrated care service. A guiding framework for the development of integrated care was missing. The DMIC model has been developed for that reason and offers a useful tool for assessment, self-evaluation or improvement of integrated care services in practice. The model has been validated for AMI, dementia and stroke services. The latest new studies in diabetes, palliative care and youth care gave further insight in the generic character of the DMIC. Based on these studies it can be assumed that the DMIC can be used for multiple types of integrated care services. The model is assumed to be interesting for an international audience. Improving integrated care is a complex topic in a large number of countries; the DMIC is also based on the international literature. Dutch integrated care coordinators stated that the DMIC helped them to assess their integrated care development in practice and supported them in obtaining ideas for expanding and improving their integrated care activities.
The web-based self-evaluation tool focuses on a process- and organisational level of integrated care. Also, the self assessed development phase can be compared to the development phase as calculated by the DMIC tool. The cases showed this is fruitful input for discussions. When using the tool, the results can also be used in quality policy reports and improvement plans. The web-based tool is being tested at this moment in practice, but in San Marino we can present the latest webversion and demonstrate with a short video how to use the tool and model. During practical exercises in the workshop the participants will experience how the application of the DMIC can work for them in practice or in research. For integrated care researchers and policy makers, the DMIC questionnaire and tool is a promising method for further research and policy plans in integrated care.
PMCID: PMC3617779
development model for integrated care; development of integrated care services; implementation and improvement of integrated care; self evaluation
10.  Hypertension guidelines and their effects on the health system 
Hypertension guidelines, which have existed for many years and primarily used in the USA, Canada and Great Britain, are now becoming an issue in Germany. Strong efforts are presently underway for a German version comparable to the guidelines developed for the mentioned countries. The development of guidelines is a part of the implementation system of guidelines in Germany. It covers the mode of operation of the AWMF (work community of the scientific medical subject companies) with the clearinghouse for guidelines (CLA) and the cooperation with the centre for medical quality (ÄZQ).
In the HTA report the real use of the hypertension guidelines shall be investigated for Germany from the development trends and further possibilities of use according to a medical applicability. Economic issues and an optimisation of use are also discussed.
The following questions shall be answered in particular:
How much are the guidelines used concerning hypertension? Can effects (or their influence) be established on the medical procedures? Are there statements available about costs and cost effectiveness? Are there recommendations for further use?
To answer these questions, a comprehensive literature search was done. No empirical investigation was carried out. From this enquiry 206 articles were checked in detail but not all of them were available in full text.
Only those publications which directly dealt with high blood pressure guidelines or articles with a direct reference to the topic have been considered in the HTA report.
Publications concerning screening or methods of prevention, medical studies of the hypertension syndrome without a direct reference to guidelines and publications concerned with putting guidelines into action were excluded.
After an analysis of the selected literature addressing the topic of hypertension guidelines, it was evident that the use of these guidelines cannot be gathered from existing literature at the present time. One can assume from international studies with analogical reasoning that these are confessed and have a high level of acceptance in the medical community. Unfortunately the actual usage is not represented satisfactorily in the scientific literature.
The effects of the guidelines on the medical procedures seem to be very strongly individual and the analyses to the compliance show at least an observable effect within the last few years. No publications could be found for the cost effectiveness of the guidelines.
The actual compliance with guidelines seems to be in relation with the duration of the professional practice. It seems the shorter the professional practice takes place, the stronger the guidelines are adhered.
At present, there are only a few notes for the German health service regarding the actual effect of the hypertonus guidelines. However, the reason is not that the effect would not be possibly strong but at the methodical challenge to evaluate the sustaining effects of the application of the hypertonus guidelines. For this reason the literature is very rare regarding this topic.
For Germany it can be derived by analogical reasoning from foreign studies that guidelines will facilitate a more and more essential contribution to the design of the health system. Considering that primarily younger physicians accepted guidelines mode, the further construction, update and implementation of guidelines are essential, particularly with regard to the quality assurance. Straight guidelines can express a standard of the quality of a health system as a benchmark. The existence of guidelines or the lack thereof is considered also as a quality indicator of a health system at the organisation for economic cooperation and development (OECD).
Guidelines should be evaluated - especially the hypertonus guideline. Also further development and implementation should be emphasised. Methodically oriented work to the approach is pretty recent.
It is undeniable that guidelines represent a very essential and important contribution for the successful dealing with significant morbidity problems in a health system.
The fact that primarily younger doctors more frequently adopt, employ and adhere to guidelines leads to the assumption that expected sustainability for practical use will increase. Furthermore intensified use of guidelines can be considered in the "mainstream" of the development of the public health system also in an international perspective.
Not one single publication contradicts that a further acquirement, update and distribution of guidelines for the use of practices is necessary. The importance of the guideline is also not questioned in any article.
PMCID: PMC3011314  PMID: 21289932
11.  Tailoring intervention procedures to routine primary health care practice; an ethnographic process evaluation 
Tailor-made approaches enable the uptake of interventions as they are seen as a way to overcome the incompatibility of general interventions with local knowledge about the organisation of routine medical practice and the relationship between the patients and the professionals in practice. Our case is the Quattro project which is a prevention programme for cardiovascular diseases in high-risk patients in primary health care centres in deprived neighbourhoods. This programme was implemented as a pragmatic trial and foresaw the importance of local knowledge in primary health care and internal, or locally made, guidelines. The aim of this paper is to show how this prevention programme, which could be tailored to routine care, was implemented in primary care.
An ethnographic design was used for this study. We observed and interviewed the researchers and the practice nurses. All the research documents, observations and transcribed interviews were analysed thematically.
Our ethnographic process evaluation showed that the opportunity of tailoring intervention procedures to routine care in a pragmatic trial setting did not result in a well-organised and well-implemented prevention programme. In fact, the lack of standard protocols hindered the implementation of the intervention. Although it was not the purpose of this trial, a guideline was developed. Despite the fact that the developed guideline functioned as a tool, it did not result in the intervention being organised accordingly. However, the guideline did make tailoring the intervention possible. It provided the professionals with the key or the instructions needed to achieve organisational change and transform the existing interprofessional relations.
As tailor-made approaches are developed to enable the uptake of interventions in routine practice, they are facilitated by the brokering of tools such as guidelines. In our study, guidelines facilitated organisational change and enabled the transformation of existing interprofessional relations, and thus made tailoring possible. The attractive flexibility of pragmatic trial design in taking account of local practice variations may often be overestimated.
PMCID: PMC1959520  PMID: 17683627
12.  Characteristics of communication guidelines that facilitate or impede guideline use: a focus group study 
BMC Family Practice  2007;8:31.
The quality of doctor-patient communication has a major impact on the quality of medical care. Communication guidelines define best practices for doctor patient communication and are therefore an important tool for improving communication. However, adherence to communication guidelines remains low, despite doctors participating in intensive communication skill training. Implementation research shows that adherence is higher for guidelines in general that are user centred and feasible, which implies that they are consistent with users' opinions, tap into users' existing skills and fit into existing routines. Developers of communication guidelines seem to have been somewhat negligent with regard to user preferences and guideline feasibility. In order to promote the development of user centred and practicable communication guidelines, we elicited user preferences and identified which guideline characteristics facilitate or impede guideline use.
Seven focus group interviews were conducted with experienced GPs, communication trainers (GPs and behavioural scientists) and communication learners (GP trainees and medical students) and three focus group interviews with groups of GP trainees only. All interviews were transcribed and analysed qualitatively.
The participants identified more impeding guideline characteristics than facilitating ones. The most important impeding characteristic was that guidelines do not easily fit into GPs' day-to-day practice. This is due to rigidity and inefficiency of communication guidelines and erroneous assumptions underpinning guideline development. The most important facilitating characteristic was guideline structure. Guidelines that were structured in distinct phases helped users to remain in control of consultations, which was especially useful in complicated consultations.
Although communication guidelines are generally considered useful, especially for structuring consultations, their usefulness is impaired by lack of flexibility and applicability to practice routines. User centred and feasible guidelines should combine the advantages of helping doctors to structure consultations with flexibility to tailor communication strategies to specific contexts and situations.
PMCID: PMC1885263  PMID: 17506878
13.  Understanding integrated care pathways in palliative care using realist evaluation: a mixed methods study protocol 
BMJ Open  2012;2(4):e001533.
Policy- and evidence-based guidelines have highlighted the need for improved palliative and end-of-life care. However, there is still evidence of individuals dying undignified deaths with little pain control, therefore inflicting unnecessary suffering. New commissioning powers have enabled a 2-year pilot of an innovative integrated care pathway (ICP) designed to improve arrangements for individuals with life-limiting illnesses requiring palliative care. A novel feature of the ICP is its focus on palliative care over the last 6 months of life, aiming to intervene early to prepare for and ensure a good death. What is not known is if this pathway works, how it works and who it works for.
Methods and analysis
A realist evaluation and a complex analytical framework will investigate and discover context, mechanism and outcome conjectures and configurations of the ICP and thus facilitate exploration of how it works and who it works for. A mixed methods approach will be used with small sample sizes to capture the breadth of the ICP. Phase 1 will identify if the pathway works through analysis of NHS Morbidity Information Query and Export Syntax data, locality Death Audit data and the Quality of Dying and Death Questionnaire. Phase 2 employs soft systems methodology with data from focus groups with health professionals to identify how the pathway works. Phase 3 uses the Miller Behavioural Style Scale and interviews with palliative care patients and bereaved relatives to analyse communication in palliative care.
Ethics and dissemination
Ethical approval has been granted from the NHS local ethics committee (REC reference number: 11/NE/0318). Research & Development approval has been gained from four different trusts, and relevant voluntary organisations and the local council have been informed about the research. This protocol illustrates the complexity inherent in evaluating a palliative care ICP. Identification of whether the pathway works, how it works and who it works for will be beneficial to all practices and other care providers involved as it will give objective data on the impact of the ICP. Results will be disseminated throughout the study for continuous quality improvement of the ICP. Outcomes from each data collection phase will be disseminated separately if analysis warrants it; all data collection will be utilised in the realist evaluation. The research provides a potential for the dissemination of the pathway to other localities through the transferable knowledge it will generate, from its focus on the contexts that are crucial for successful implementation, the mechanisms that facilitate implementation and the outcomes achieved.
Article summary
Article focus
This article is a protocol of a realist evaluation of a palliative care ICP, which was developed in Primary Care by health practitioners. The ICP itself uses elements of long-term chronic illness care in order to provide holistic, supportive, high-quality palliative care. The focus of the article is to detail how the ICP will be evaluated, using a variety of data collection tools, which will identify contexts and mechanisms that lead to improved outcomes, thus taking the main focus away from just the outcomes alone. The identification of contexts and mechanisms for improved outcomes is known as realist evaluation and will provide a better knowledge of the essential conditions of effectiveness when the ICP is implemented in other localities.
Key messages
The key aim of this article is to detail the creation of a complex realist evaluation, which utilises a unique and varied methodological framework. It is hoped that through this article, others will understand the groundwork needed to set up and execute a realist evaluation.
Strengths and limitations of this study
The protocol details a complex evaluation of a unique palliative care ICP using a new and innovative methodology: realist evaluation.
Some may perceive the small sample sizes in the qualitative sections of the study as a weakness. However, the aim of the study is not to find a robust causal mechanism; this would be premature with an ICP in its infancy. The aim is to unpack the contexts and mechanisms that work in certain circumstances, from this conditions crucial for effectiveness can be highlighted, which are essential for implementation of the ICP in other localities.
The ICP involves 15 general practitioner practices, which collectively care for 80 300 patients. The study described will use Morbidity Information Query and Export Syntax and Death Audit data from all 15 practices and will conduct the other sections of research within selected practices, both rural and suburban.
Finally, palliative care is commonly misunderstood in the literature and in the field. This paper addresses this confusion and fills a gap in the literature.
PMCID: PMC3391371  PMID: 22761292
14.  Guideline adaptation and implementation planning: a prospective observational study 
Adaptation of high-quality practice guidelines for local use has been advanced as an efficient means to improve acceptability and applicability of evidence-informed care. In a pan-Canadian study, we examined how cancer care groups adapted pre-existing guidelines to their unique context and began implementation planning.
Using a mixed-methods, case-study design, five cases were purposefully sampled from self-identified groups and followed as they used a structured method and resources for guideline adaptation. Cases received the ADAPTE Collaboration toolkit, facilitation, methodological and logistical support, resources and assistance as required. Documentary and primary data collection methods captured individual case experience, including monthly summaries of meeting and field notes, email/telephone correspondence, and project records. Site visits, process audits, interviews, and a final evaluation forum with all cases contributed to a comprehensive account of participant experience.
Study cases took 12 to >24 months to complete guideline adaptation. Although participants appreciated the structure, most found the ADAPTE method complex and lacking practical aspects. They needed assistance establishing individual guideline mandate and infrastructure, articulating health questions, executing search strategies, appraising evidence, and achieving consensus. Facilitation was described as a multi-faceted process, a team effort, and an essential ingredient for guideline adaptation. While front-line care providers implicitly identified implementation issues during adaptation, they identified a need to add an explicit implementation planning component.
Guideline adaptation is a positive initial step toward evidence-informed care, but adaptation (vs. ‘de novo’ development) did not meet expectations for reducing time or resource commitments. Undertaking adaptation is as much about the process (engagement and capacity building) as it is about the product (adapted guideline). To adequately address local concerns, cases found it necessary to also search and appraise primary studies, resulting in hybrid (adaptation plus de novo) guideline development strategies that required advanced methodological skills.
Adaptation was found to be an action element in the knowledge translation continuum that required integration of an implementation perspective. Accordingly, the adaptation methodology and resources were reformulated and substantially augmented to provide practical assistance to groups not supported by a dedicated guideline panel and to provide more implementation planning support. The resulting framework is called CAN-IMPLEMENT.
PMCID: PMC3668213  PMID: 23656884
Knowledge to action; Practice guidelines; Evidence-informed practice; Knowledge activation; Guideline adaptation; Implementation planning
15.  Threats to Validity in the Design and Conduct of Preclinical Efficacy Studies: A Systematic Review of Guidelines for In Vivo Animal Experiments 
PLoS Medicine  2013;10(7):e1001489.
The vast majority of medical interventions introduced into clinical development prove unsafe or ineffective. One prominent explanation for the dismal success rate is flawed preclinical research. We conducted a systematic review of preclinical research guidelines and organized recommendations according to the type of validity threat (internal, construct, or external) or programmatic research activity they primarily address.
Methods and Findings
We searched MEDLINE, Google Scholar, Google, and the EQUATOR Network website for all preclinical guideline documents published up to April 9, 2013 that addressed the design and conduct of in vivo animal experiments aimed at supporting clinical translation. To be eligible, documents had to provide guidance on the design or execution of preclinical animal experiments and represent the aggregated consensus of four or more investigators. Data from included guidelines were independently extracted by two individuals for discrete recommendations on the design and implementation of preclinical efficacy studies. These recommendations were then organized according to the type of validity threat they addressed. A total of 2,029 citations were identified through our search strategy. From these, we identified 26 guidelines that met our eligibility criteria—most of which were directed at neurological or cerebrovascular drug development. Together, these guidelines offered 55 different recommendations. Some of the most common recommendations included performance of a power calculation to determine sample size, randomized treatment allocation, and characterization of disease phenotype in the animal model prior to experimentation.
By identifying the most recurrent recommendations among preclinical guidelines, we provide a starting point for developing preclinical guidelines in other disease domains. We also provide a basis for the study and evaluation of preclinical research practice.
Please see later in the article for the Editors' Summary
Editors' Summary
The development process for new drugs is lengthy and complex. It begins in the laboratory, where scientists investigate the causes of diseases and identify potential new treatments. Next, promising interventions undergo preclinical research in cells and in animals (in vivo animal experiments) to test whether the intervention has the expected effect and to support the generalization (extension) of this treatment–effect relationship to patients. Drugs that pass these tests then enter clinical trials, where their safety and efficacy is tested in selected groups of patients under strictly controlled conditions. Finally, the government bodies responsible for drug approval review the results of the clinical trials, and successful drugs receive a marketing license, usually a decade or more after the initial laboratory work. Notably, only 11% of agents that enter clinical testing (investigational drugs) are ultimately licensed.
Why Was This Study Done?
The frequent failure of investigational drugs during clinical translation is potentially harmful to trial participants. Moreover, the costs of these failures are passed onto healthcare systems in the form of higher drug prices. It would be good, therefore, to reduce the attrition rate of investigational drugs. One possible explanation for the dismal success rate of clinical translation is that preclinical research, the key resource for justifying clinical development, is flawed. To address this possibility, several groups of preclinical researchers have issued guidelines intended to improve the design and execution of in vivo animal studies. In this systematic review (a study that uses predefined criteria to identify all the research on a given topic), the authors identify the experimental practices that are commonly recommended in these guidelines and organize these recommendations according to the type of threat to validity (internal, construct, or external) that they address. Internal threats to validity are factors that confound reliable inferences about treatment–effect relationships in preclinical research. For example, experimenter expectation may bias outcome assessment. Construct threats to validity arise when researchers mischaracterize the relationship between an experimental system and the clinical disease it is intended to represent. For example, researchers may use an animal model for a complex multifaceted clinical disease that only includes one characteristic of the disease. External threats to validity are unseen factors that frustrate the transfer of treatment–effect relationships from animal models to patients.
What Did the Researchers Do and Find?
The researchers identified 26 preclinical guidelines that met their predefined eligibility criteria. Twelve guidelines addressed preclinical research for neurological and cerebrovascular drug development; other disorders covered by guidelines included cardiac and circulatory disorders, sepsis, pain, and arthritis. Together, the guidelines offered 55 different recommendations for the design and execution of preclinical in vivo animal studies. Nineteen recommendations addressed threats to internal validity. The most commonly included recommendations of this type called for the use of power calculations to ensure that sample sizes are large enough to yield statistically meaningful results, random allocation of animals to treatment groups, and “blinding” of researchers who assess outcomes to treatment allocation. Among the 25 recommendations that addressed threats to construct validity, the most commonly included recommendations called for characterization of the properties of the animal model before experimentation and matching of the animal model to the human manifestation of the disease. Finally, six recommendations addressed threats to external validity. The most commonly included of these recommendations suggested that preclinical research should be replicated in different models of the same disease and in different species, and should also be replicated independently.
What Do These Findings Mean?
This systematic review identifies a range of investigational recommendations that preclinical researchers believe address threats to the validity of preclinical efficacy studies. Many of these recommendations are not widely implemented in preclinical research at present. Whether the failure to implement them explains the frequent discordance between the results on drug safety and efficacy obtained in preclinical research and in clinical trials is currently unclear. These findings provide a starting point, however, for the improvement of existing preclinical research guidelines for specific diseases, and for the development of similar guidelines for other diseases. They also provide an evidence-based platform for the analysis of preclinical evidence and for the study and evaluation of preclinical research practice. These findings should, therefore, be considered by investigators, institutional review bodies, journals, and funding agents when designing, evaluating, and sponsoring translational research.
Additional Information
Please access these websites via the online version of this summary at
The US Food and Drug Administration provides information about drug approval in the US for consumers and for health professionals; its Patient Network provides a step-by-step description of the drug development process that includes information on preclinical research
The UK Medicines and Healthcare Products Regulatory Agency (MHRA) provides information about all aspects of the scientific evaluation and approval of new medicines in the UK; its My Medicine: From Laboratory to Pharmacy Shelf web pages describe the drug development process from scientific discovery, through preclinical and clinical research, to licensing and ongoing monitoring
The STREAM website provides ongoing information about policy, ethics, and practices used in clinical translation of new drugs
The CAMARADES collaboration offers a “supporting framework for groups involved in the systematic review of animal studies” in stroke and other neurological diseases
PMCID: PMC3720257  PMID: 23935460
16.  Improving the quality of health care: using international collaboration to inform guideline programmes by founding the Guidelines International Network (G-I-N)* 
Quality & safety in health care  2004;13(6):455-460.

 Clinical practice guidelines are regarded as powerful tools to achieve effective health care. Although many countries have built up experience in the development, appraisal, and implementation of guidelines, until recently there has been no established forum for collaboration at an international level. As a result, in different countries seeking similar goals and using similar strategies, efforts have been unnecessarily duplicated and opportunities for harmonisation lost because of the lack of a supporting organisational framework. This triggered a proposal in 2001 for an international guidelines network built on existing partnerships. A baseline survey confirmed a strong demand for such an entity. A multinational group of guideline experts initiated the development of a non-profit organisation aimed at promotion of systematic guideline development and implementation. The Guidelines International Network (G-I-N) was founded in November 2002. One year later the Network released the International Guideline Library, a searchable database which now contains more than 2000 guideline resources including published guidelines, guidelines under development, "guidelines for guidelines", training materials, and patient information tools. By June 2004, 52 organisations from 27 countries had joined the network including institutions from Oceania, North America, and Europe, and WHO. This paper describes the process that led to the foundation of the G-I-N, its characteristics, prime activities, and ideas on future projects and collaboration.
PMCID: PMC1743909  PMID: 15576708
17.  Improving the use of research evidence in guideline development: 1. Guidelines for guidelines 
The World Health Organization (WHO), like many other organisations around the world, has recognised the need to use more rigorous processes to ensure that health care recommendations are informed by the best available research evidence. This is the first of a series of 16 reviews that have been prepared as background for advice from the WHO Advisory Committee on Health Research to WHO on how to achieve this.
We reviewed the literature on guidelines for the development of guidelines.
We searched PubMed and three databases of methodological studies for existing systematic reviews and relevant methodological research. We did not conduct systematic reviews ourselves. Our conclusions are based on the available evidence, consideration of what WHO and other organisations are doing and logical arguments.
Key questions and answers
We found no experimental research that compared different formats of guidelines for guidelines or studies that compared different components of guidelines for guidelines. However, there are many examples, surveys and other observational studies that compared the impact of different guideline development documents on guideline quality.
What have other organizations done to develop guidelines for guidelines from which WHO can learn?
• Establish a credible, independent committee that evaluates existing methods for developing guidelines or that updates existing ones.
• Obtain feedback and approval from various stakeholders during the development process of guidelines for guidelines.
• Develop a detailed source document (manual) that guideline developers can use as reference material.
What should be the key components of WHO guidelines for guidelines?
• Guidelines for guidelines should include information and instructions about the following components: 1) Priority setting; 2) Group composition and consultations; 3) Declaration and avoidance of conflicts of interest; 4) Group processes; 5) Identification of important outcomes; 6) Explicit definition of the questions and eligibility criteria ; 7) Type of study designs for different questions; 8) Identification of evidence; 9) Synthesis and presentation of evidence; 10) Specification and integration of values; 11) Making judgments about desirable and undesirable effects; 12) Taking account of equity; 13) Grading evidence and recommendations; 14) Taking account of costs; 15) Adaptation, applicability, transferability of guidelines; 16) Structure of reports; 17) Methods of peer review; 18) Planned methods of dissemination & implementation; 19) Evaluation of the guidelines.
What have other organizations done to implement guidelines for guidelines from which WHO can learn?
• Obtain buy-in from regions and country level representatives for guidelines for guidelines before dissemination of a revised version.
• Disseminate the guidelines for guidelines widely and make them available (e.g. on the Internet).
• Develop examples of guidelines that guideline developers can use as models when applying the guidelines for guidelines.
• Ensure training sessions for those responsible for developing guidelines.
• Continue to monitor the methodological literature on guideline development.
PMCID: PMC1665445  PMID: 17118181
18.  Promoting best gynecologic oncology practice: a role for the Society of Gynecologic Oncologists of Canada 
Current Oncology  2006;13(3):94-98.
During March 30–April 1, 2005, the Society of Gynecologic Oncologists of Canada (goc) and the Canadian Strategy for Cancer Control (cscc) Clinical Practice Guidelines Action Group (cpg-ag) met to
determine how goc would like to influence practice in the care of women with gynecologic cancer.explore a collaborative model for developing and implementing evidence-based practice guidelines.investigate the utility of the cpg evaluation and adaptation cycle as a tool for selecting, adapting, and adopting guidelines.
At the workshop meeting, 21 members of the goc and the cpg-ag heard presentations from various Canadian guideline initiatives. As an example of adaptation and adoption processes, the agree (Appraisal of Guidelines for Research and Evaluation) tool was applied to guidelines in recurrent ovarian cancer, and the group explored their opportunity to use knowledge translation to influence the care of women with gynecologic cancer.
The themes influencing practice are consistent with goc’s mandate. The future is expected to involve partnering with other groups to maximize scarce resources. Resources should be directed to facilitating implementation of existing guidelines rather than to developing new documents. The full spectrum of cancer care includes prevention, screening, diagnosis, primary treatment, follow-up, treatment of recurrent disease, and palliation. High-quality evidence is available in some areas, but gaps exist where guideline panels could provide guidance. Development of a pan-Canadian gynecologic oncology process could provide an opportunity to influence access to care at the political and policy levels.
The goc will develop linkages such that the toolbox available through cscc-cpg-ag can be incorporated into future collaboration.
PMCID: PMC1891179  PMID: 17576448
Practice guidelines; gynecologic oncology
19.  Improving the use of research evidence in guideline development: 16. Evaluation 
The World Health Organization (WHO), like many other organisations around the world, has recognised the need to use more rigorous processes to ensure that health care recommendations are informed by the best available research evidence. This is the last of a series of 16 reviews that have been prepared as background for advice from the WHO Advisory Committee on Health Research to WHO on how to achieve this.
We reviewed the literature on evaluating guidelines and recommendations, including their quality, whether they are likely to be up-to-date, and their implementation. We also considered the role of guideline developers in undertaking evaluations that are needed to inform recommendations.
We searched PubMed and three databases of methodological studies for existing systematic reviews and relevant methodological research. We did not conduct systematic reviews ourselves. Our conclusions are based on the available evidence, consideration of what WHO and other organisations are doing and logical arguments.
Key questions and answers
Our answers to these questions were informed by a review of instruments for evaluating guidelines, several studies of the need for updating guidelines, discussions of the pros and cons of different research designs for evaluating the implementation of guidelines, and consideration of the use of uncertainties identified in systematic reviews to set research priorities.
How should the quality of guidelines or recommendations be appraised?
• WHO should put into place processes to ensure that both internal and external review of guidelines is undertaken routinely.
• A checklist, such as the AGREE instrument, should be used.
• The checklist should be adapted and tested to ensure that it is suitable to the broad range of recommendations that WHO produces, including public health and health policy recommendations, and that it includes questions about equity and other items that are particularly important for WHO guidelines.
When should guidelines or recommendations be updated?
• Processes should be put into place to ensure that guidelines are monitored routinely to determine if they are in need of updating.
• People who are familiar with the topic, such as Cochrane review groups, should do focused, routine searches for new research that would require revision of the guideline.
• Periodic review of guidelines by experts not involved in developing the guidelines should also be considered.
• Consideration should be given to establishing guideline panels that are ongoing, to facilitate routine updating, with members serving fixed periods with a rotating membership.
How should the impact of guidelines or recommendations be evaluated?
• WHO headquarters and regional offices should support member states and those responsible for policy decisions and implementation to evaluate the impact of their decisions and actions by providing advice regarding impact assessment, practical support and coordination of efforts.
• Before-after evaluations should be used cautiously and when there are important uncertainties regarding the effects of a policy or its implementation, randomised evaluations should be used when possible.
What responsibility should WHO take for ensuring that important uncertainties are addressed by future research when the evidence needed to inform recommendations is lacking?
• Guideline panels should routinely identify important uncertainties and research priorities. This source of potential priorities for research should be used systematically to inform priority-setting processes for global research.
PMCID: PMC1702533  PMID: 17156460
20.  Introducing the Canadian Thoracic Society Framework for Guideline Dissemination and Implementation, with Concurrent Evaluation 
The Canadian Thoracic Society (CTS) is leveraging its strengths in guideline production to enable respiratory guideline implementation in Canada. The authors describe the new CTS Framework for Guideline Dissemination and Implementation, with Concurrent Evaluation, which has three spheres of action: guideline production, implementation infrastructure and knowledge translation (KT) methodological support. The Canadian Institutes of Health Research ‘Knowledge-to-Action’ process was adopted as the model of choice for conceptualizing KT interventions. Within the framework, new evidence for formatting guideline recommendations to enhance the intrinsic implementability of future guidelines were applied. Clinical assemblies will consider implementability early in the guideline production cycle when selecting clinical questions, and new practice guidelines will include a section dedicated to KT. The framework describes the development of a web-based repository and communication forum to inventory existing KT resources and to facilitate collaboration and communication among implementation stakeholders through an online discussion board. A national forum for presentation and peer-review of proposed KT projects is described. The framework outlines expert methodological support for KT planning, development and evaluation including a practical guide for implementers and a novel ‘Clinical Assembly – KT Action Team’, and in-kind logistical support and assistance in securing peer-reviewed funding.
PMCID: PMC3956335  PMID: 23936883
Implementation; Guidelines; Knowledge translation
21.  Report on activities and attitudes of organizations active in the clinical practice guidelines field. 
The organizing committee of a workshop on clinical practice guidelines (CPGs) surveyed invited organizations on their attitudes and activities related to five topics to be covered during the workshop sessions: organizational roles, priority setting, guidelines implementation, guidelines evaluation and development of a network of those active in the CPG field. Organizational roles: The national specialty societies were felt to have the largest role to play; the smallest roles were assigned to consumers, who were seen to have a role mainly in priority setting, and to industry and government, both of which were seen to have primarily a funding role. Many barriers to collaboration were identified, the solutions to all of which appeared to be better communication, establishment of common principles and clear role definitions. Priority setting: There was considerable agreement on the criteria that should be used to set priorities for CPG activities: the burden of disease on population health, the state of scientific knowledge, the cost of treatment and the economic burden of disease on society were seen as important factors, whereas the costs of guidelines development and practitioner interest in guidelines development were seen as less important. Organizations were unable to give much information on how they set priorities. Guidelines implementation: Most of the organizations surveyed did not actively try to ensure the implementation of guidelines, although a considerable minority devoted resources to implementation. The 38% of organizations that implemented guidelines actively listed a wide variety of activities, including training, use of local opinion leaders, information technology, local consensus processes and counter detailing. Guidelines evaluation: Formal evaluation of guidelines was undertaken by fewer than 13% of the responding organizations. All the evaluations incorporated assessments before and after guideline implementation, and some used primary patient data. Barriers to evaluation included lack of money, time, data or expertise. CPG Network: Most of the respondents felt that all organizations and individuals interested or involved in guidelines should form the membership of the network. The three most important functions of such a network were deemed to be (a) to facilitate collaboration among those involved in the CPG process, (b) to maintain an information centre on CPGs and (c) to provide expertise to the CPG process. It was felt that the network should have some formal structure and communicate through e-mail and print media.
PMCID: PMC1487370  PMID: 7553491
22.  Implementing clinical guidelines in psychiatry: a qualitative study of perceived facilitators and barriers 
BMC Psychiatry  2010;10:8.
Translating scientific evidence into daily practice is complex. Clinical guidelines can improve health care delivery, but there are a number of challenges in guideline adoption and implementation. Factors influencing the effective implementation of guidelines remain poorly understood. Understanding of barriers and facilitators is important for development of effective implementation strategies. The aim of this study was to determine perceived facilitators and barriers to guideline implementation and clinical compliance to guidelines for depression in psychiatric care.
This qualitative study was conducted at two psychiatric clinics in Stockholm, Sweden. The implementation activities at one of the clinics included local implementation teams, seminars, regular feedback and academic detailing. The other clinic served as a control and only received guidelines by post. Data were collected from three focus groups and 28 individual, semi-structured interviews. Content analysis was used to identify themes emerging from the interview data.
The identified barriers to, and facilitators of, the implementation of guidelines could be classified into three major categories: (1) organizational resources, (2) health care professionals' individual characteristics and (3) perception of guidelines and implementation strategies. The practitioners in the implementation team and at control clinics differed in three main areas: (1) concerns about control over professional practice, (2) beliefs about evidence-based practice and (3) suspicions about financial motives for guideline introduction.
Identifying the barriers to, and facilitators of, the adoption of recommendations is an important way of achieving efficient implementation strategies. The findings of this study suggest that the adoption of guidelines may be improved if local health professionals actively participate in an ongoing implementation process and identify efficient strategies to overcome barriers on an organizational and individual level. Getting evidence into practice and implementing clinical guidelines are dependent upon more than practitioners' motivation. There are factors in the local context, e.g. culture and leadership, evaluation, feedback on performance and facilitation, -that are likely to be equally influential.
PMCID: PMC2822755  PMID: 20089141
23.  WHO Essential Medicines Policies and Use in Developing and Transitional Countries: An Analysis of Reported Policy Implementation and Medicines Use Surveys 
PLoS Medicine  2014;11(9):e1001724.
Kathleen Holloway and David Henry evaluate whether countries that report having implemented WHO essential medicines policies have higher quality use of medicines.
Please see later in the article for the Editors' Summary
Suboptimal medicine use is a global public health problem. For 35 years the World Health Organization (WHO) has promoted essential medicines policies to improve quality use of medicines (QUM), but evidence of their effectiveness is lacking, and uptake by countries remains low. Our objective was to determine whether WHO essential medicines policies are associated with better QUM.
Methods and Findings
We compared results from independently conducted medicines use surveys in countries that did versus did not report implementation of WHO essential medicines policies. We extracted survey data on ten validated QUM indicators and 36 self-reported policy implementation variables from WHO databases for 2002–2008. We calculated the average difference (as percent) for the QUM indicators between countries reporting versus not reporting implementation of specific policies. Policies associated with positive effects were included in a regression of a composite QUM score on total numbers of implemented policies. Data were available for 56 countries. Twenty-seven policies were associated with better use of at least two percentage points. Eighteen policies were associated with significantly better use (unadjusted p<0.05), of which four were associated with positive differences of 10% or more: undergraduate training of doctors in standard treatment guidelines, undergraduate training of nurses in standard treatment guidelines, the ministry of health having a unit promoting rational use of medicines, and provision of essential medicines free at point of care to all patients. In regression analyses national wealth was positively associated with the composite QUM score and the number of policies reported as being implemented in that country. There was a positive correlation between the number of policies (out of the 27 policies with an effect size of 2% or more) that countries reported implementing and the composite QUM score (r = 0.39, 95% CI 0.14 to 0.59, p = 0.003). This correlation weakened but remained significant after inclusion of national wealth in multiple linear regression analyses. Multiple policies were more strongly associated with the QUM score in the 28 countries with gross national income per capita below the median value (US$2,333) (r = 0.43, 95% CI 0.06 to 0.69, p = 0.023) than in the 28 countries with values above the median (r = 0.22, 95% CI −0.15 to 0.56, p = 0.261). The main limitations of the study are the reliance on self-report of policy implementation and measures of medicine use from small surveys. While the data can be used to explore the association of essential medicines policies with medicine use, they cannot be used to compare or benchmark individual country performance.
WHO essential medicines policies are associated with improved QUM, particularly in low-income countries.
Please see later in the article for the Editors' Summary
Editors' Summary
The widespread availability of effective medicines, particularly those used to treat infectious diseases, has been largely responsible for a doubling in the average global life expectancy over the past century. However, the suboptimal use (overuse and underuse) of medicines is an ongoing global public health problem. The unnecessary use of medicines (for example, the use of antibiotics for sore throats caused by viruses) needlessly consumes scarce resources and has undesirable effects such as encouraging the emergence of antibiotic resistance. Conversely, underuse deprives people of the undisputed benefits of many medicines. Since 1977, to help optimize medicine use, the World Health Organization (WHO) has advocated the concept of “essential medicines” and has developed policies to promote the quality use of medicines (QUM). Essential medicines are drugs that satisfy the priority needs of the human population and that should always be available to communities in adequate amounts of assured quality, in the appropriate dosage forms, and at an affordable price. Policies designed to promote QUM include recommendations that medicines should be free at the point of care and that all health care professionals should be educated about the WHO list of essential medicines (which is revised every two years) throughout their careers.
Why Was This Study Done?
Surveys of WHO member countries undertaken in 2003 and 2007 suggest that the implementation of WHO policies designed to promote QUM is patchy. Moreover, little is known about whether these policies are effective, particularly in middle- and low-income countries. For most of these countries, it is not known whether any of the policies affect validated QUM indicators such as the percentage of patients prescribed antibiotics (a lower percentage indicates better use of medicines) or the percentage of patients treated in compliance with national treatment guidelines (a higher percentage indicates better use of medicines). Here, the researchers analyze data from policy implementation questionnaires and medicine use surveys to determine whether implementation of WHO essential medicines policies is associated with improved QUM in low- and middle-income countries.
What Did the Researchers Do and Find?
The researchers extracted data on ten validated QUM indicators and on implementation of 36 policy variables from WHO databases for 2002–2008 and compared the average differences for the QUM indicators between low- and middle-income countries that did versus did not report implementation of specific WHO policies for QUM. Among 56 countries for which data were available, 27 policies were associated with improved QUM. Four policies were particularly effective, namely, doctors' undergraduate training in standard treatment guidelines, nurses' undergraduate training in standard treatment guidelines, the existence of a ministry of health department promoting the rational use of medicines, and the provision of essential medicines free to all patients at point of care. The researchers also analyzed correlations between how many of the 27 effective policies were implemented in a country and a composite QUM score. As national wealth increased, both the composite QUM score of a country and the reported number of policies implemented by the country increased. There was also a positive correlation between the numbers of policies that countries reported implementing and their composite QUM score. Finally, the implementation of multiple policies was more strongly associated with the composite QUM score in countries with a gross national income per capita below the average for the study countries than in countries with a gross national income above the average.
What Do These Findings Mean?
These findings show that between 2002 and 2008, the reported implementation of WHO essential medicines policies was associated with better QUM across low- and middle-income countries. These findings also reveal a positive correlation between the number of policies that countries report implementing and their QUM. Notably, this correlation was strongest in the countries with the lowest per capita national wealth levels, which underscores the importance of essential medicines policies in low-income countries. Because of the nature of the data available to the researchers, these findings do not show that the implementation of WHO policies actually causes improvements in QUM. Moreover, the age of the data, the reliance on self-report of policy implementation, and the small sample sizes of the medicine use surveys may all have introduced some inaccuracies into these findings. Nevertheless, overall, these findings suggest that WHO should continue to develop its medicine policies and to collect data on medicine use as part of its core functions.
Additional Information
Please access these websites via the online version of this summary at
The World Health Organization provides information about essential medicines; its latest lists of essential medicines are available on the Internet; information about WHO policies to improve the quality use of medicines is also available (in several languages)
The International Network for the Rational Use of Drugs designs, tests, and disseminates effective strategies to improve the way drugs are prescribed, dispensed, and used, particularly in resource-poor countries
The website helps health care professionals, researchers, and policy makers obtain and discuss current information on essential drugs, policy, program activities, education, and training (available in several languages); the website is run by Satellife, which aims to use technology to connect health workers in resource-limited countries to each other and to up-to-date clinical and public health content
PMCID: PMC4165598  PMID: 25226527
24.  A mixed methods pilot study with a cluster randomized control trial to evaluate the impact of a leadership intervention on guideline implementation in home care nursing 
Foot ulcers are a significant problem for people with diabetes. Comprehensive assessments of risk factors associated with diabetic foot ulcer are recommended in clinical guidelines to decrease complications such as prolonged healing, gangrene and amputations, and to promote effective management. However, the translation of clinical guidelines into nursing practice remains fragmented and inconsistent, and a recent homecare chart audit showed less than half the recommended risk factors for diabetic foot ulcers were assessed, and peripheral neuropathy (the most significant predictor of complications) was not assessed at all.
Strong leadership is consistently described as significant to successfully transfer guidelines into practice. Limited research exists however regarding which leadership behaviours facilitate and support implementation in nursing.
The purpose of this pilot study is to evaluate the impact of a leadership intervention in community nursing on implementing recommendations from a clinical guideline on the nursing assessment and management of diabetic foot ulcers.
Two phase mixed methods design is proposed (ISRCTN 12345678). Phase I: Descriptive qualitative to understand barriers to implementing the guideline recommendations, and to inform the intervention. Phase II: Matched pair cluster randomized controlled trial (n = 4 centers) will evaluate differences in outcomes between two implementation strategies. Primary outcome: Nursing assessments of client risk factors, a composite score of 8 items based on Diabetes/Foot Ulcer guideline recommendations.
Intervention: In addition to the organization's 'usual' implementation strategy, a 12 week leadership strategy will be offered to managerial and clinical leaders consisting of: a) printed materials, b) one day interactive workshop to develop a leadership action plan tailored to barriers to support implementation; c) three post-workshop teleconferences.
This study will provide vital information on which leadership strategies are well received to facilitate and support guideline implementation. The anticipated outcomes will provide information to assist with effective management of foot ulcers for people with diabetes.
By tracking clinical outcomes associated with guideline implementation, health care administrators will be better informed to influence organizational and policy decision-making to support evidence-based quality care. Findings will be useful to inform the design of future multi-centered trials on various clinical topics to enhance knowledge translation for positive outcomes.
Trial Registration
Current Control Trials ISRCTN06910890
PMCID: PMC2631597  PMID: 19077199
25.  The Impact of eHealth on the Quality and Safety of Health Care: A Systematic Overview 
PLoS Medicine  2011;8(1):e1000387.
Aziz Sheikh and colleagues report the findings of their systematic overview that assessed the impact of eHealth solutions on the quality and safety of health care.
There is considerable international interest in exploiting the potential of digital solutions to enhance the quality and safety of health care. Implementations of transformative eHealth technologies are underway globally, often at very considerable cost. In order to assess the impact of eHealth solutions on the quality and safety of health care, and to inform policy decisions on eHealth deployments, we undertook a systematic review of systematic reviews assessing the effectiveness and consequences of various eHealth technologies on the quality and safety of care.
Methods and Findings
We developed novel search strategies, conceptual maps of health care quality, safety, and eHealth interventions, and then systematically identified, scrutinised, and synthesised the systematic review literature. Major biomedical databases were searched to identify systematic reviews published between 1997 and 2010. Related theoretical, methodological, and technical material was also reviewed. We identified 53 systematic reviews that focused on assessing the impact of eHealth interventions on the quality and/or safety of health care and 55 supplementary systematic reviews providing relevant supportive information. This systematic review literature was found to be generally of substandard quality with regards to methodology, reporting, and utility. We thematically categorised eHealth technologies into three main areas: (1) storing, managing, and transmission of data; (2) clinical decision support; and (3) facilitating care from a distance. We found that despite support from policymakers, there was relatively little empirical evidence to substantiate many of the claims made in relation to these technologies. Whether the success of those relatively few solutions identified to improve quality and safety would continue if these were deployed beyond the contexts in which they were originally developed, has yet to be established. Importantly, best practice guidelines in effective development and deployment strategies are lacking.
There is a large gap between the postulated and empirically demonstrated benefits of eHealth technologies. In addition, there is a lack of robust research on the risks of implementing these technologies and their cost-effectiveness has yet to be demonstrated, despite being frequently promoted by policymakers and “techno-enthusiasts” as if this was a given. In the light of the paucity of evidence in relation to improvements in patient outcomes, as well as the lack of evidence on their cost-effectiveness, it is vital that future eHealth technologies are evaluated against a comprehensive set of measures, ideally throughout all stages of the technology's life cycle. Such evaluation should be characterised by careful attention to socio-technical factors to maximise the likelihood of successful implementation and adoption.
Please see later in the article for the Editors' Summary
Editors' Summary
There is considerable international interest in exploiting the potential of digital health care solutions, often referred to as eHealth—the use of information and communication technologies—to enhance the quality and safety of health care. Often accompanied by large costs, any large-scale expenditure on eHealth—such as electronic health records, picture archiving and communication systems, ePrescribing, associated computerized provider order entry systems, and computerized decision support systems—has tended to be justified on the grounds that these are efficient and cost-effective means for improving health care. In 2005, the World Health Assembly passed an eHealth resolution (WHA 58.28) that acknowledged, “eHealth is the cost-effective and secure use of information and communications technologies in support of health and health-related fields, including health-care services, health surveillance, health literature, and health education, knowledge and research,” and urged member states to develop and implement eHealth technologies. Since then, implementing eHealth technologies has become a main priority for many countries. For example, England has invested at least £12.8 billion in a National Programme for Information Technology for the National Health Service, and the Obama administration in the United States has committed to a US$38 billion eHealth investment in health care.
Why Was This Study Done?
Despite the wide endorsement of and support for eHealth, the scientific basis of its benefits—which are repeatedly made and often uncritically accepted—remains to be firmly established. A robust evidence-based perspective on the advantages on eHealth could help to suggest priority areas that have the greatest potential for benefit to patients and also to inform international eHealth deliberations on costs. Therefore, in order to better inform the international community, the authors systematically reviewed the published systematic review literature on eHealth technologies and evaluated the impact of these technologies on the quality and safety of health care delivery.
What Did the Researchers Do and Find?
The researchers divided eHealth technologies into three main categories: (1) storing, managing, and transmission of data; (2) clinical decision support; and (3) facilitating care from a distance. Then, implementing methods based on those developed by the Cochrane Collaboration and the NHS Service Delivery and Organisation Programme, the researchers used detailed search strategies and maps of health care quality, safety, and eHealth interventions to identify relevant systematic reviews (and related theoretical, methodological, and technical material) published between 1997 and 2010. Using these techniques, the researchers retrieved a total of 46,349 references from which they identified 108 reviews. The 53 reviews that the researchers finally selected (and critically reviewed) provided the main evidence base for assessing the impact of eHealth technologies in the three categories selected.
In their systematic review of systematic reviews, the researchers included electronic health records and picture archiving communications systems in their evaluation of category 1, computerized provider (or physician) order entry and e-prescribing in category 2, and all clinical information systems that, when used in the context of eHealth technologies, integrate clinical and demographic patient information to support clinician decision making in category 3.
The researchers found that many of the clinical claims made about the most commonly used eHealth technologies were not substantiated by empirical evidence. The evidence base in support of eHealth technologies was weak and inconsistent and importantly, there was insubstantial evidence to support the cost-effectiveness of these technologies. For example, the researchers only found limited evidence that some of the many presumed benefits could be realized; importantly, they also found some evidence that introducing these new technologies may on occasions also generate new risks such as prescribers becoming over-reliant on clinical decision support for e-prescribing, or overestimate its functionality, resulting in decreased practitioner performance.
What Do These Findings Mean?
The researchers found that despite the wide support for eHealth technologies and the frequently made claims by policy makers when constructing business cases to raise funds for large-scale eHealth projects, there is as yet relatively little empirical evidence to substantiate many of the claims made about eHealth technologies. In addition, even for the eHealth technology tools that have proven to be successful, there is little evidence to show that such tools would continue to be successful beyond the contexts in which they were originally developed. Therefore, in light of the lack of evidence in relation to improvements in patient outcomes, as well as the lack of evidence on their cost-effectiveness, the authors say that future eHealth technologies should be evaluated against a comprehensive set of measures, ideally throughout all stages of the technology's life cycle, and include socio-technical factors to maximize the likelihood of successful implementation and adoption in a given context. Furthermore, it is equally important that eHealth projects that have already been commissioned are subject to rigorous, multidisciplinary, and independent evaluation.
Additional Information
Please access these websites via the online version of this summary at
The authors' broader study is: Car J, Black A, Anandan C, Cresswell K, Pagliari C, McKinstry B, et al. (2008) The Impact of eHealth on the Quality and Safety of Healthcare. Available at:
More information is available on the World Health Assembly eHealth resolution
The World Health Organization provides information at the Global Observatory on eHealth, as well as a global insight into eHealth developments
The European Commission provides Information on eHealth in Europe and some examples of good eHealth practice
More information is provided on NHS Connecting for Health
PMCID: PMC3022523  PMID: 21267058

Results 1-25 (1138106)