Search tips
Search criteria

Results 1-25 (1677521)

Clipboard (0)

Related Articles

1.  User-Centered Design and Usability Testing of an Innovative Health-Related Quality of Life Module 
Applied Clinical Informatics  2014;5(4):958-970.
Various computerized health risk appraisals (HRAs) are available, but few of them assess health-related quality of life (HRQoL) in a goal-directed framework. This study describes the user-centered development and usability testing of an innovative HRQoL module that extends a validated HRA tool in primary care settings.
Systematic user-centered design, usability testing, and qualitative methods were used to develop the HRQoL module in primary care practices. Twenty two patients and 5 clinicians participated in two rounds of interactive technology think-out-loud sessions (TOLs) and semi-structured interviews (SSIs) to iteratively develop a four-step, computerized process that collects information on patient goals for meaningful life activities and current level of disability and presents a personalized and prioritized list of preventive recommendations linked to online resources.
Analysis of TOLs and SSIs generated 5 categories and 11 sub-categories related to facilitators and barriers to usability and human-technology interaction. The categories included: Understanding the Purpose, Usability, Perceived Value, Literacy, and Participant Motivation. Some categories were inter-connected. The technology was continually and iteratively improved between sessions until saturation of positive feedback was achieved in 4 categories (addressing motivation will require more research). Usability of all screen units of the module was improved substantially. Clinician feedback emphasized the importance of the module’s ability to translate the patient-centered HRQoL Report into actionable items for clinicians to facilitate shared decision-making. Complete integration of the HRQoL module into the existing HRA will require further development and testing.
Systematic application of user-centered design and human factors principles in technology development and testing may significantly improve the usability and clinical value of health information systems. This more sophisticated approach helped us translate complex clinical concepts, goal-setting steps, and decision-support processes into an accepted and value-added technology.
PMCID: PMC4287674  PMID: 25589910
Health risk appraisal; user-centered design; quality of life; goal-directed care; prioritization
2.  Designing, Implementing, and Evaluating Mobile Health Technologies for Managing Chronic Conditions in Older Adults: A Scoping Review 
JMIR mHealth and uHealth  2016;4(2):e29.
The current landscape of a rapidly aging population accompanied by multiple chronic conditions presents numerous challenges to optimally support the complex needs of this group. Mobile health (mHealth) technologies have shown promise in supporting older persons to manage chronic conditions; however, there remains a dearth of evidence-informed guidance to develop such innovations.
The purpose of this study was to conduct a scoping review of current practices and recommendations for designing, implementing, and evaluating mHealth technologies to support the management of chronic conditions in community-dwelling older adults.
A 5-stage scoping review methodology was used to map the relevant literature published between January 2005 and March 2015 as follows: (1) identified the research question, (2) identified relevant studies, (3) selected relevant studies for review, (4) charted data from selected literature, and (5) summarized and reported results. Electronic searches were conducted in 5 databases. In addition, hand searches of reference lists and a key journal were completed. Inclusion criteria were research and nonresearch papers focused on mHealth technologies designed for use by community-living older adults with at least one chronic condition, or health care providers or informal caregivers providing care in the home and community setting. Two reviewers independently identified articles for review and extracted data.
We identified 42 articles that met the inclusion criteria. Of these, described innovations focused on older adults with specific chronic conditions (n=17), chronic conditions in general (n=6), or older adults in general or those receiving homecare services (n=18). Most of the mHealth solutions described were designed for use by both patients and health care providers or health care providers only. Thematic categories identified included the following: (1) practices and considerations when designing mHealth technologies; (2) factors that support/hinder feasibility, acceptability, and usability of mHealth technologies; and (3) approaches or methods for evaluating mHealth technologies.
There is limited yet increasing use of mHealth technologies in home health care for older adults. A user-centered, collaborative, interdisciplinary approach to enhance feasibility, acceptability, and usability of mHealth innovations is imperative. Creating teams with the required pools of expertise and insight regarding needs is critical. The cyclical, iterative process of developing mHealth innovations needs to be viewed as a whole with supportive theoretical frameworks. Many barriers to implementation and sustainability have limited the number of successful, evidence-based mHealth solutions beyond the pilot or feasibility stage. The science of implementation of mHealth technologies in home-based care for older adults and self-management of chronic conditions are important areas for further research. Additionally, changing needs as cohorts and technologies advance are important considerations. Lessons learned from the data and important implications for practice, policy, and research are discussed to inform the future development of innovations.
PMCID: PMC4919548  PMID: 27282195
Telemedicine; Mobile health; Health Plan Implementations; Evaluation Studies as Topic; Design; mHealth Innovations; Frail Elderly; Older Adults; Multiple Chronic Conditions; Home Care Services; Scoping Review; Communication; Information Communication Technologies
3.  Enabling Access to Medical and Health Education in Rwanda Using Mobile Technology: Needs Assessment for the Development of Mobile Medical Educator Apps 
JMIR Medical Education  2016;2(1):e7.
Lack of access to health and medical education resources for doctors in the developing world is a serious global health problem. In Rwanda, with a population of 11 million, there is only one medical school, hence a shortage in well-trained medical staff. The growth of interactive health technologies has played a role in the improvement of health care in developed countries and has offered alternative ways to offer continuous medical education while improving patient's care. However, low and middle-income countries (LMIC) like Rwanda have struggled to implement medical education technologies adapted to local settings in medical practice and continuing education. Developing a user-centered mobile computing approach for medical and health education programs has potential to bring continuous medical education to doctors in rural and urban areas of Rwanda and influence patient care outcomes.
The aim of this study is to determine user requirements, currently available resources, and perspectives for potential medical education technologies in Rwanda.
Information baseline and needs assessments data collection were conducted in all 44 district hospitals (DHs) throughout Rwanda. The research team collected qualitative data through interviews with 16 general practitioners working across Rwanda and 97 self-administered online questionnaires for rural areas. Data were collected and analyzed to address two key questions: (1) what are the currently available tools for the use of mobile-based technology for medical education in Rwanda, and (2) what are user's requirements for the creation of a mobile medical education technology in Rwanda?
General practitioners from different hospitals highlighted that none of the available technologies avail local resources such as the Ministry of Health (MOH) clinical treatment guidelines. Considering the number of patients that doctors see in Rwanda, an average of 32 patients per day, there is need for a locally adapted mobile education app that utilizes specific Rwandan medical education resources. Based on our results, we propose a mobile medical education app that could provide many benefits such as rapid decision making with lower error rates, increasing the quality of data management and accessibility, and improving practice efficiency and knowledge. In areas where Internet access is limited, the proposed mobile medical education app would need to run on a mobile device without Internet access.
A user-centered design approach was adopted, starting with a needs assessment with representative end users, which provided recommendations for the development of a mobile medical education app specific to Rwanda. Specific app features were identified through the needs assessment and it was evident that there will be future benefits to ongoing incorporation of user-centered design methods to better inform the software development and improve its usability. Results of the user-centered design reported here can inform other medical education technology developments in LMIC to ensure that technologies developed are usable by all stakeholders.
PMCID: PMC5041362  PMID: 27731861
mobile medical education; technology; user-centered design
4.  Managing symptoms during cancer treatments: evaluating the implementation of evidence-informed remote support protocols 
Management of cancer treatment-related symptoms is an important safety issue given that symptoms can become life-threatening and often occur when patients are at home. With funding from the Canadian Partnership Against Cancer, a pan-Canadian steering committee was established with representation from eight provinces to develop symptom protocols using a rigorous methodology (CAN-IMPLEMENT©). Each protocol is based on a systematic review of the literature to identify relevant clinical practice guidelines. Protocols were validated by cancer nurses from across Canada. The aim of this study is to build an effective and sustainable approach for implementing evidence-informed protocols for nurses to use when providing remote symptom assessment, triage, and guidance in self-management for patients experiencing symptoms while undergoing cancer treatments.
A prospective mixed-methods study design will be used. Guided by the Knowledge to Action Framework, the study will involve (a) establishing an advisory knowledge user team in each of three targeted settings; (b) assessing factors influencing nurses’ use of protocols using interviews/focus groups and a standardized survey instrument; (c) adapting protocols for local use, ensuring fidelity of the content; (d) selecting intervention strategies to overcome known barriers and implementing the protocols; (e) conducting think-aloud usability testing; (f) evaluating protocol use and outcomes by conducting an audit of 100 randomly selected charts at each of the three settings; and (g) assessing satisfaction with remote support using symptom protocols and change in nurses’ barriers to use using survey instruments. The primary outcome is sustained use of the protocols, defined as use in 75% of the calls. Descriptive analysis will be conducted for the barriers, use of protocols, and chart audit outcomes. Content analysis will be conducted on interviews/focus groups and usability testing with comparisons across settings.
Given the importance of patient safety, patient-centered care, and delivery of quality services, learning how to effectively implement evidence-informed symptom protocols in oncology healthcare services is essential for ensuring safe, consistent, and effective care for individuals with cancer. This study is likely to have a significant contribution to the delivery of remote oncology services, as well as influence symptom management by patients at home.
PMCID: PMC3527220  PMID: 23164244
Cancer; Symptom management; Implementation science; Mixed-methods study; Guidelines based
5.  Applying Human Factors Principles to Mitigate Usability Issues Related to Embedded Assumptions in Health Information Technology Design 
JMIR Human Factors  2014;1(1):e3.
There is growing recognition that design flaws in health information technology (HIT) lead to increased cognitive work, impact workflows, and produce other undesirable user experiences that contribute to usability issues and, in some cases, patient harm. These usability issues may in turn contribute to HIT utilization disparities and patient safety concerns, particularly among “non-typical” HIT users and their health care providers. Health care disparities are associated with poor health outcomes, premature death, and increased health care costs. HIT has the potential to reduce these disparate outcomes. In the computer science field, it has long been recognized that embedded cultural assumptions can reduce the usability, usefulness, and safety of HIT systems for populations whose characteristics differ from “stereotypical” users. Among these non-typical users, inappropriate embedded design assumptions may contribute to health care disparities. It is unclear how to address potentially inappropriate embedded HIT design assumptions once detected.
The objective of this paper is to explain HIT universal design principles derived from the human factors engineering literature that can help to overcome potential usability and/or patient safety issues that are associated with unrecognized, embedded assumptions about cultural groups when designing HIT systems.
Existing best practices, guidance, and standards in software usability and accessibility were subjected to a 5-step expert review process to identify and summarize those best practices, guidance, and standards that could help identify and/or address embedded design assumptions in HIT that could negatively impact patient safety, particularly for non-majority HIT user populations. An iterative consensus-based process was then used to derive evidence-based design principles from the data to address potentially inappropriate embedded cultural assumptions.
Design principles that may help identify and address embedded HIT design assumptions are available in the existing literature.
Evidence-based HIT design principles derived from existing human factors and informatics literature can help HIT developers identify and address embedded cultural assumptions that may underlie HIT-associated usability and patient safety concerns as well as health care disparities.
PMCID: PMC4797669  PMID: 27025349
cultural ergonomics; culturally informed design; EHR; health care disparities; health information technology; human factors; patient portal; patient safety; usability; workflow
6.  The Electronic Patient Reported Outcome Tool: Testing Usability and Feasibility of a Mobile App and Portal to Support Care for Patients With Complex Chronic Disease and Disability in Primary Care Settings 
JMIR mHealth and uHealth  2016;4(2):e58.
People experiencing complex chronic disease and disability (CCDD) face some of the greatest challenges of any patient population. Primary care providers find it difficult to manage multiple discordant conditions and symptoms and often complex social challenges experienced by these patients. The electronic Patient Reported Outcome (ePRO) tool is designed to overcome some of these challenges by supporting goal-oriented primary care delivery. Using the tool, patients and providers collaboratively develop health care goals on a portal linked to a mobile device to help patients and providers track progress between visits.
This study tested the usability and feasibility of adopting the ePRO tool into a single interdisciplinary primary health care practice in Toronto, Canada. The Fit between Individuals, Fask, and Technology (FITT) framework was used to guide our assessment and explore whether the ePRO tool is: (1) feasible for adoption in interdisciplinary primary health care practices and (2) usable from both the patient and provider perspectives. This usability pilot is part of a broader user-centered design development strategy.
A 4-week pilot study was conducted in which patients and providers used the ePRO tool to develop health-related goals, which patients then monitored using a mobile device. Patients and providers collaboratively set goals using the system during an initial visit and had at least 1 follow-up visit at the end of the pilot to discuss progress. Focus groups and interviews were conducted with patients and providers to capture usability and feasibility measures. Data from the ePRO system were extracted to provide information regarding tool usage.
Six providers and 11 patients participated in the study; 3 patients dropped out mainly owing to health issues. The remaining 8 patients completed 210 monitoring protocols, equal to over 1300 questions, with patients often answering questions daily. Providers and patients accessed the portal on an average of 10 and 1.5 times, respectively. Users found the system easy to use, some patients reporting that the tool helped in their ability to self-manage, catalyzed a sense of responsibility over their care, and improved patient-centered care delivery. Some providers found that the tool helped focus conversations on goal setting. However, the tool did not fit well with provider workflows, monitoring questions were not adequately tailored to individual patient needs, and daily reporting became tedious and time-consuming for patients.
Although our study suggests relatively low usability and feasibility of the ePRO tool, we are encouraged by the early impact on patient outcomes and generally positive responses from both user groups regarding the potential of the tool to improve care for patients with CCDD. As is consistent with our user-centered design development approach, we have modified the tool based on user feedback, and are now testing the redeveloped tool through an exploratory trial.
PMCID: PMC4911509  PMID: 27256035
eHealth; mHealth; multimorbidity; primary care; usability; feasibility; pilot
7.  The Development of a Mobile Monitoring and Feedback Tool to Stimulate Physical Activity of People With a Chronic Disease in Primary Care: A User-Centered Design 
JMIR mHealth and uHealth  2013;1(2):e8.
Physical activity is an important aspect in the treatment of patients with chronic obstructive pulmonary disease or type-2 diabetes. A monitoring and feedback tool combined with guidance by a primary care provider might be a successful method to enhance the level of physical activity in these patients. As a prerequisite for useful technology, it is important to involve the end-users in the design process from an early stage.
The aim of this study was to investigate the user requirements for a tool to stimulate physical activity, embedded in primary care practice. The leading principle of this tool is to change behavior by self-monitoring, goal-setting, and feedback.
The research team collected qualitative data among 15 patients, 16 care professionals, and several experts. A prototype was developed in three stages. In stage 1, the literature was searched to identify end-users and context. In stage 2, the literature, experts and patient representatives were consulted to set up a use case with the general idea of the innovation. In stage 3, individual interviews and focus groups were held to identify the end-user requirements. Based on these requirements a prototype was built by the engineering team.
The development process has led to a tool that generally meets the requirements of the end-users. A tri-axial activity sensor, worn on the hip, is connected by Bluetooth to a smartphone. In an app, quantitative feedback is given about the amount of activity and goals reached by means of graphical visualization, and an image shows a sun when the goal is reached. Overviews about activity per half an hour, per day, week, and month are provided. In the menu of the app and on a secured website, patients can enter information in individual sessions or read feedback messages generated by the system. The practice nurse can see the results of all patients on a secure webpage and can then discuss the results and set personalized goals in consultation with the patient.
This study demonstrates that a user-centered approach brings in valuable details (such as the requirements for feedback in activity minutes per day) to improve the fit between the user, technology, and the organization of care, which is important for the usability and acceptability of the tool. The tool embedded in primary care will be evaluated in a randomized controlled trial.
PMCID: PMC4114510  PMID: 25099556
user-centered design; self-management; physical activity; accelerometry; remote sensing technology; primary health care
8.  Development and usability testing of a web-based cancer symptom and quality-of-life support intervention 
Health informatics journal  2014;21(1):10-23.
The feasibility and acceptability of computerized screening and patient-reported outcome measures have been demonstrated in the literature. However, patient-centered management of health information entails two challenges: gathering and presenting data using “patient-tailored” methods and supporting “patient-control” of health information. The design and development of many symptom and quality-of-life information systems have not included opportunities for systematically collecting and analyzing user input. As part of a larger clinical trial, the Electronic Self-Report Assessment for Cancer–II project, participatory design approaches were used to build and test new features and interfaces for patient/caregiver users. The research questions centered on patient/caregiver preferences with regard to the following: (a) content, (b) user interface needs, (c) patient-oriented summary, and (d) patient-controlled sharing of information with family, caregivers, and clinicians. Mixed methods were used with an emphasis on qualitative approaches; focus groups and individual usability tests were the primary research methods. Focus group data were content analyzed, while individual usability sessions were assessed with both qualitative and quantitative methods. We identified 12 key patient/caregiver preferences through focus groups with 6 participants. We implemented seven of these preferences during the iterative design process. We deferred development for some of the preferences due to resource constraints. During individual usability testing (n = 8), we were able to identify 65 usability issues ranging from minor user confusion to critical errors that blocked task completion. The participatory development model that we used led to features and design revisions that were patient centered. We are currently evaluating new approaches for the application interface and for future research pathways. We encourage other researchers to adopt user-centered design approaches when building patient-centered technologies.
PMCID: PMC4747103  PMID: 24406906
consumer health information; decision-support systems; ehealth; evidence-based practice; IT design and development methodologies
9.  Initial Usability and Feasibility Evaluation of a Personal Health Record-Based Self-Management System for Older Adults 
eGEMs  2015;3(2):1152.
Electronic personal health record-based (ePHR-based) self-management systems can improve patient engagement and have an impact on health outcomes. In order to realize the benefits of these systems, there is a need to develop and evaluate heath information technology from the same theoretical underpinnings.
Using an innovative usability approach based in human-centered distributed information design (HCDID), we tested an ePHR-based falls-prevention self-management system—Self-Assessment via a Personal Health Record (i.e., SAPHeR)—designed using HCDID principles in a laboratory. And we later evaluated SAPHeR’s use by community-dwelling older adults at home.
The innovative approach used in this study supported the analysis of four components: tasks, users, representations, and functions. Tasks were easily learned and features such as text-associated images facilitated task completion. Task performance times were slow, however user satisfaction was high. Nearly seven out of every ten features desired by design participants were evaluated in our usability testing of the SAPHeR system. The in vivo evaluation suggests that older adults could improve their confidence in performing indoor and outdoor activities after using the SAPHeR system.
We have applied an innovative consumer-usability evaluation. Our approach addresses the limitations of other usability testing methods that do not utilize consistent theoretically based methods for designing and testing technology. We have successfully demonstrated the utility of testing consumer technology use across multiple components (i.e., task, user, representational, functional) to evaluate the usefulness, usability, and satisfaction of an ePHR-based self-management system.
PMCID: PMC4537150  PMID: 26290889
Usability evaluation; personal health record; self-management
10.  Designing and evaluating an interprofessional shared decision-making and goal-setting decision aid for patients with diabetes in clinical care - systematic decision aid development and study protocol 
Care of patients with diabetes often occurs in the context of other chronic illness. Competing disease priorities and competing patient-physician priorities present challenges in the provision of care for the complex patient. Guideline implementation interventions to date do not acknowledge these intricacies of clinical practice. As a result, patients and providers are left overwhelmed and paralyzed by the sheer volume of recommendations and tasks. An individualized approach to the patient with diabetes and multiple comorbid conditions using shared decision-making (SDM) and goal setting has been advocated as a patient-centred approach that may facilitate prioritization of treatment options. Furthermore, incorporating interprofessional integration into practice may overcome barriers to implementation. However, these strategies have not been taken up extensively in clinical practice.
To systematically develop and test an interprofessional SDM and goal-setting toolkit for patients with diabetes and other chronic diseases, following the Knowledge to Action framework.
1. Feasibility study: Individual interviews with primary care physicians, nurses, dietitians, pharmacists, and patients with diabetes will be conducted, exploring their experiences with shared decision-making and priority-setting, including facilitators and barriers, the relevance of a decision aid and toolkit for priority-setting, and how best to integrate it into practice.
2. Toolkit development: Based on this data, an evidence-based multi-component SDM toolkit will be developed. The toolkit will be reviewed by content experts (primary care, endocrinology, geriatricians, nurses, dietitians, pharmacists, patients) for accuracy and comprehensiveness.
3. Heuristic evaluation: A human factors engineer will review the toolkit and identify, list and categorize usability issues by severity.
4. Usability testing: This will be done using cognitive task analysis.
5. Iterative refinement: Throughout the development process, the toolkit will be refined through several iterative cycles of feedback and redesign.
Interprofessional shared decision-making regarding priority-setting with the use of a decision aid toolkit may help prioritize care of individuals with multiple comorbid conditions. Adhering to principles of user-centered design, we will develop and refine a toolkit to assess the feasibility of this approach.
PMCID: PMC3937124  PMID: 24450385
Shared decision-making; Priority setting; Patient decision aid; Interprofessional care; Diabetes mellitus; Patient education; Medical informatics; Toolkit development; Study protocol; User-centred design; Qualitative methods
11.  Designing a patient-centered personal health record to promote preventive care 
Evidence-based preventive services offer profound health benefits, yet Americans receive only half of indicated care. A variety of government and specialty society policy initiatives are promoting the adoption of information technologies to engage patients in their care, such as personal health records, but current systems may not utilize the technology's full potential.
Using a previously described model to make information technology more patient-centered, we developed an interactive preventive health record (IPHR) designed to more deeply engage patients in preventive care and health promotion. We recruited 14 primary care practices to promote the IPHR to all adult patients and sought practice and patient input in designing the IPHR to ensure its usability, salience, and generalizability. The input involved patient usability tests, practice workflow observations, learning collaboratives, and patient feedback. Use of the IPHR was measured using practice appointment and IPHR databases.
The IPHR that emerged from this process generates tailored patient recommendations based on guidelines from the U.S. Preventive Services Task Force and other organizations. It extracts clinical data from the practices' electronic medical record and obtains health risk assessment information from patients. Clinical content is translated and explained in lay language. Recommendations review the benefits and uncertainties of services and possible actions for patients and clinicians. Embedded in recommendations are self management tools, risk calculators, decision aids, and community resources - selected to match patient's clinical circumstances. Within six months, practices had encouraged 14.4% of patients to use the IPHR (ranging from 1.5% to 28.3% across the 14 practices). Practices successfully incorporated the IPHR into workflow, using it to prepare patients for visits, augment health behavior counseling, explain test results, automatically issue patient reminders for overdue services, prompt clinicians about needed services, and formulate personalized prevention plans.
The IPHR demonstrates that a patient-centered personal health record that interfaces with the electronic medical record can give patients a high level of individualized guidance and be successfully adopted by busy primary care practices. Further study and refinement are necessary to make information systems even more patient-centered and to demonstrate their impact on care.
Trial Registration identifier: NCT00589173
PMCID: PMC3250934  PMID: 22115059
12.  Usability Testing of a Digital Pen and Paper System in Nursing Documentation 
Usability testing was used to evaluate whether a new technology, a digital pen and paper system, would be usable for hospital nurses. Twenty-one nurses in a Labor and Delivery unit were randomly assigned into two groups, and a crossover design was used to compare the digital pen and paper system to conventional pens. Data collection included observations, interviews, and a questionnaire. Results showed that nurses had a positive attitude toward the system and could foresee its potential benefits, but they found that in its current design the system had poor usability and interfered with nurses’ work practices. Usability testing provided important insight into the needs of nurses and the suitability of this technology. This study is an example of how a user-centered approach can improve our understanding of the real needs of nurses and contribute to the design of useful and usable technologies for healthcare.
PMCID: PMC1560675  PMID: 16779159
13.  Developing an efficient scheduling template of a chemotherapy treatment unit 
The Australasian Medical Journal  2011;4(10):575-588.
This study was undertaken to improve the performance of a Chemotherapy Treatment Unit by increasing the throughput and reducing the average patient’s waiting time. In order to achieve this objective, a scheduling template has been built. The scheduling template is a simple tool that can be used to schedule patients' arrival to the clinic. A simulation model of this system was built and several scenarios, that target match the arrival pattern of the patients and resources availability, were designed and evaluated. After performing detailed analysis, one scenario provide the best system’s performance. A scheduling template has been developed based on this scenario. After implementing the new scheduling template, 22.5% more patients can be served.
CancerCare Manitoba is a provincially mandated cancer care agency. It is dedicated to provide quality care to those who have been diagnosed and are living with cancer. MacCharles Chemotherapy unit is specially built to provide chemotherapy treatment to the cancer patients of Winnipeg. In order to maintain an excellent service, it tries to ensure that patients get their treatment in a timely manner. It is challenging to maintain that goal because of the lack of a proper roster, the workload distribution and inefficient resource allotment. In order to maintain the satisfaction of the patients and the healthcare providers, by serving the maximum number of patients in a timely manner, it is necessary to develop an efficient scheduling template that matches the required demand with the availability of resources. This goal can be reached using simulation modelling. Simulation has proven to be an excellent modelling tool. It can be defined as building computer models that represent real world or hypothetical systems, and hence experimenting with these models to study system behaviour under different scenarios.1, 2
A study was undertaken at the Children's Hospital of Eastern Ontario to identify the issues behind the long waiting time of a emergency room.3 A 20-­‐day field observation revealed that the availability of the staff physician and interaction affects the patient wait time. Jyväskylä et al.4 used simulation to test different process scenarios, allocate resources and perform activity-­‐based cost analysis in the Emergency Department (ED) at the Central Hospital. The simulation also supported the study of a new operational method, named "triage-team" method without interrupting the main system. The proposed triage team method categorises the entire patient according to the urgency to see the doctor and allows the patient to complete the necessary test before being seen by the doctor for the first time. The simulation study showed that it will decrease the throughput time of the patient and reduce the utilisation of the specialist and enable the ordering all the tests the patient needs right after arrival, thus quickening the referral to treatment.
Santibáñez et al.5 developed a discrete event simulation model of British Columbia Cancer Agency"s ambulatory care unit which was used to study the impact of scenarios considering different operational factors (delay in starting clinic), appointment schedule (appointment order, appointment adjustment, add-­‐ons to the schedule) and resource allocation. It was found that the best outcomes were obtained when not one but multiple changes were implemented simultaneously. Sepúlveda et al.6 studied the M. D. Anderson Cancer Centre Orlando, which is a cancer treatment facility and built a simulation model to analyse and improve flow process and increase capacity in the main facility. Different scenarios were considered like, transferring laboratory and pharmacy areas, adding an extra blood draw room and applying different scheduling techniques of patients. The study shows that by increasing the number of short-­‐term (four hours or less) patients in the morning could increase chair utilisation.
Discrete event simulation also helps improve a service where staff are ignorant about the behaviour of the system as a whole; which can also be described as a real professional system. Niranjon et al.7 used simulation successfully where they had to face such constraints and lack of accessible data. Carlos et al. 8 used Total quality management and simulation – animation to improve the quality of the emergency room. Simulation was used to cover the key point of the emergency room and animation was used to indicate the areas of opportunity required. This study revealed that a long waiting time, overload personnel and increasing withdrawal rate of patients are caused by the lack of capacity in the emergency room.
Baesler et al.9 developed a methodology for a cancer treatment facility to find stochastically a global optimum point for the control variables. A simulation model generated the output using a goal programming framework for all the objectives involved in the analysis. Later a genetic algorithm was responsible for performing the search for an improved solution. The control variables that were considered in this research are number of treatment chairs, number of drawing blood nurses, laboratory personnel, and pharmacy personnel. Guo et al. 10 presented a simulation framework considering demand for appointment, patient flow logic, distribution of resources, scheduling rules followed by the scheduler. The objective of the study was to develop a scheduling rule which will ensure that 95% of all the appointment requests should be seen within one week after the request is made to increase the level of patient satisfaction and balance the schedule of each doctor to maintain a fine harmony between "busy clinic" and "quiet clinic".
Huschka et al.11 studied a healthcare system which was about to change their facility layout. In this case a simulation model study helped them to design a new healthcare practice by evaluating the change in layout before implementation. Historical data like the arrival rate of the patients, number of patients visited each day, patient flow logic, was used to build the current system model. Later, different scenarios were designed which measured the changes in the current layout and performance.
Wijewickrama et al.12 developed a simulation model to evaluate appointment schedule (AS) for second time consultations and patient appointment sequence (PSEQ) in a multi-­‐facility system. Five different appointment rule (ARULE) were considered: i) Baily; ii) 3Baily; iii) Individual (Ind); iv) two patients at a time (2AtaTime); v) Variable Interval and (V-­‐I) rule. PSEQ is based on type of patients: Appointment patients (APs) and new patients (NPs). The different PSEQ that were studied in this study were: i) first-­‐ come first-­‐serve; ii) appointment patient at the beginning of the clinic (APBEG); iii) new patient at the beginning of the clinic (NPBEG); iv) assigning appointed and new patients in an alternating manner (ALTER); v) assigning a new patient after every five-­‐appointment patients. Also patient no show (0% and 5%) and patient punctuality (PUNCT) (on-­‐time and 10 minutes early) were also considered. The study found that ALTER-­‐Ind. and ALTER5-­‐Ind. performed best on 0% NOSHOW, on-­‐time PUNCT and 5% NOSHOW, on-­‐time PUNCT situation to reduce WT and IT per patient. As NOSHOW created slack time for waiting patients, their WT tends to reduce while IT increases due to unexpected cancellation. Earliness increases congestion whichin turn increases waiting time.
Ramis et al.13 conducted a study of a Medical Imaging Center (MIC) to build a simulation model which was used to improve the patient journey through an imaging centre by reducing the wait time and making better use of the resources. The simulation model also used a Graphic User Interface (GUI) to provide the parameters of the centre, such as arrival rates, distances, processing times, resources and schedule. The simulation was used to measure the waiting time of the patients in different case scenarios. The study found that assigning a common function to the resource personnel could improve the waiting time of the patients.
The objective of this study is to develop an efficient scheduling template that maximises the number of served patients and minimises the average patient's waiting time at the given resources availability. To accomplish this objective, we will build a simulation model which mimics the working conditions of the clinic. Then we will suggest different scenarios of matching the arrival pattern of the patients with the availability of the resources. Full experiments will be performed to evaluate these scenarios. Hence, a simple and practical scheduling template will be built based on the indentified best scenario. The developed simulation model is described in section 2, which consists of a description of the treatment room, and a description of the types of patients and treatment durations. In section 3, different improvement scenarios are described and their analysis is presented in section 4. Section 5 illustrates a scheduling template based on one of the improvement scenarios. Finally, the conclusion and future direction of our work is exhibited in section 6.
Simulation Model
A simulation model represents the actual system and assists in visualising and evaluating the performance of the system under different scenarios without interrupting the actual system. Building a proper simulation model of a system consists of the following steps.
Observing the system to understand the flow of the entities, key players, availability of resources and overall generic framework.
Collecting the data on the number and type of entities, time consumed by the entities at each step of their journey, and availability of resources.
After building the simulation model it is necessary to confirm that the model is valid. This can be done by confirming that each entity flows as it is supposed to and the statistical data generated by the simulation model is similar to the collected data.
Figure 1 shows the patient flow process in the treatment room. On the patient's first appointment, the oncologist comes up with the treatment plan. The treatment time varies according to the patient’s condition, which may be 1 hour to 10 hours. Based on the type of the treatment, the physician or the clinical clerk books an available treatment chair for that time period.
On the day of the appointment, the patient will wait until the booked chair is free. When the chair is free a nurse from that station comes to the patient, verifies the name and date of birth and takes the patient to a treatment chair. Afterwards, the nurse flushes the chemotherapy drug line to the patient's body which takes about five minutes and sets up the treatment. Then the nurse leaves to serve another patient. Chemotherapy treatment lengths vary from less than an hour to 10 hour infusions. At the end of the treatment, the nurse returns, removes the line and notifies the patient about the next appointment date and time which also takes about five minutes. Most of the patients visit the clinic to take care of their PICC line (a peripherally inserted central catheter). A PICC is a line that is used to inject the patient with the chemical. This PICC line should be regularly cleaned, flushed to maintain patency and the insertion site checked for signs of infection. It takes approximately 10–15 minutes to take care of a PICC line by a nurse.
Cancer Care Manitoba provided access to the electronic scheduling system, also known as "ARIA" which is comprehensive information and image management system that aggregates patient data into a fully-­‐electronic medical chart, provided by VARIAN Medical System. This system was used to find out how many patients are booked in every clinic day. It also reveals which chair is used for how many hours. It was necessary to search a patient's history to find out how long the patient spends on which chair. Collecting the snapshot of each patient gives the complete picture of a one day clinic schedule.
The treatment room consists of the following two main limited resources:
Treatment Chairs: Chairs that are used to seat the patients during the treatment.
Nurses: Nurses are required to inject the treatment line into the patient and remove it at the end of the treatment. They also take care of the patients when they feel uncomfortable.
Mc Charles Chemotherapy unit consists of 11 nurses, and 5 stations with the following description:
Station 1: Station 1 has six chairs (numbered 1 to 6) and two nurses. The two nurses work from 8:00 to 16:00.
Station 2: Station 2 has six chairs (7 to 12) and three nurses. Two nurses work from 8:00 to 16:00 and one nurse works from 12:00 to 20:00.
Station 3: Station 4 has six chairs (13 to 18) and two nurses. The two nurses work from 8:00 to 16:00.
Station 4: Station 4 has six chairs (19 to 24) and three nurses. One nurse works from 8:00 to 16:00. Another nurse works from 10:00 to 18:00.
Solarium Station: Solarium Station has six chairs (Solarium Stretcher 1, Solarium Stretcher 2, Isolation, Isolation emergency, Fire Place 1, Fire Place 2). There is only one nurse assigned to this station that works from 12:00 to 20:00. The nurses from other stations can help when need arises.
There is one more nurse known as the "float nurse" who works from 11:00 to 19:00. This nurse can work at any station. Table 1 summarises the working hours of chairs and nurses. All treatment stations start at 8:00 and continue until the assigned nurse for that station completes her shift.
Currently, the clinic uses a scheduling template to assign the patients' appointments. But due to high demand of patient appointment it is not followed any more. We believe that this template can be improved based on the availability of nurses and chairs. Clinic workload was collected from 21 days of field observation. The current scheduling template has 10 types of appointment time slot: 15-­‐minute, 1-­‐hour, 1.5-­‐hour, 2-­‐hour, 3-­‐hour, 4-­‐hour, 5-­‐hour, 6-­‐hour, 8-­‐hour and 10-­‐hour and it is designed to serve 95 patients. But when the scheduling template was compared with the 21 days observations, it was found that the clinic is serving more patients than it is designed for. Therefore, the providers do not usually follow the scheduling template. Indeed they very often break the time slots to accommodate slots that do not exist in the template. Hence, we find that some of the stations are very busy (mostly station 2) and others are underused. If the scheduling template can be improved, it will be possible to bring more patients to the clinic and reduce their waiting time without adding more resources.
In order to build or develop a simulation model of the existing system, it is necessary to collect the following data:
Types of treatment durations.
Numbers of patients in each treatment type.
Arrival pattern of the patients.
Steps that the patients have to go through in their treatment journey and required time of each step.
Using the observations of 2,155 patients over 21 days of historical data, the types of treatment durations and the number of patients in each type were estimated. This data also assisted in determining the arrival rate and the frequency distribution of the patients. The patients were categorised into six types. The percentage of these types and their associated service times distributions are determined too.
ARENA Rockwell Simulation Software (v13) was used to build the simulation model. Entities of the model were tracked to verify that the patients move as intended. The model was run for 30 replications and statistical data was collected to validate the model. The total number of patients that go though the model was compared with the actual number of served patients during the 21 days of observations.
Improvement Scenarios
After verifying and validating the simulation model, different scenarios were designed and analysed to identify the best scenario that can handle more patients and reduces the average patient's waiting time. Based on the clinic observation and discussion with the healthcare providers, the following constraints have been stated:
The stations are filled up with treatment chairs. Therefore, it is literally impossible to fit any more chairs in the clinic. Moreover, the stakeholders are not interested in adding extra chairs.
The stakeholders and the caregivers are not interested in changing the layout of the treatment room.
Given these constraints the options that can be considered to design alternative scenarios are:
Changing the arrival pattern of the patients: that will fit over the nurses' availability.
Changing the nurses' schedule.
Adding one full time nurse at different starting times of the day.
Figure 2 compares the available number of nurses and the number of patients' arrival during different hours of a day. It can be noticed that there is a rapid growth in the arrival of patients (from 13 to 17) between 8:00 to 10:00 even though the clinic has the equal number of nurses during this time period. At 12:00 there is a sudden drop of patient arrival even though there are more available nurses. It is clear that there is an imbalance in the number of available nurses and the number of patient arrivals over different hours of the day. Consequently, balancing the demand (arrival rate of patients) and resources (available number of nurses) will reduce the patients' waiting time and increases the number of served patients. The alternative scenarios that satisfy the above three constraints are listed in Table 2. These scenarios respect the following rules:
Long treatments (between 4hr to 11hr) have to be scheduled early in the morning to avoid working overtime.
Patients of type 1 (15 minutes to 1hr treatment) are the most common. They can be fitted in at any time of the day because they take short treatment time. Hence, it is recommended to bring these patients in at the middle of the day when there are more nurses.
Nurses get tired at the end of the clinic day. Therefore, fewer patients should be scheduled at the late hours of the day.
In Scenario 1, the arrival pattern of the patient was changed so that it can fit with the nurse schedule. This arrival pattern is shown Table 3. Figure 3 shows the new patients' arrival pattern compared with the current arrival pattern. Similar patterns can be developed for the remaining scenarios too.
Analysis of Results
ARENA Rockwell Simulation software (v13) was used to develop the simulation model. There is no warm-­‐up period because the model simulates day-­‐to-­‐day scenarios. The patients of any day are supposed to be served in the same day. The model was run for 30 days (replications) and statistical data was collected to evaluate each scenario. Tables 4 and 5 show the detailed comparison of the system performance between the current scenario and Scenario 1. The results are quite interesting. The average throughput rate of the system has increased from 103 to 125 patients per day. The maximum throughput rate can reach 135 patients. Although the average waiting time has increased, the utilisation of the treatment station has increased by 15.6%. Similar analysis has been performed for the rest of the other scenarios. Due to the space limitation the detailed results are not given. However, Table 6 exhibits a summary of the results and comparison between the different scenarios. Scenario 1 was able to significantly increase the throughput of the system (by 21%) while it still results in an acceptable low average waiting time (13.4 minutes). In addition, it is worth noting that adding a nurse (Scenarios 3, 4, and 5) does not significantly reduce the average wait time or increase the system's throughput. The reason behind this is that when all the chairs are busy, the nurses have to wait until some patients finish the treatment. As a consequence, the other patients have to wait for the commencement of their treatment too. Therefore, hiring a nurse, without adding more chairs, will not reduce the waiting time or increase the throughput of the system. In this case, the only way to increase the throughput of the system is by adjusting the arrival pattern of patients over the nurses' schedule.
Developing a Scheduling Template based on Scenario 1
Scenario 1 provides the best performance. However a scheduling template is necessary for the care provider to book the patients. Therefore, a brief description is provided below on how scheduling the template is developed based on this scenario.
Table 3 gives the number of patients that arrive hourly, following Scenario 1. The distribution of each type of patient is shown in Table 7. This distribution is based on the percentage of each type of patient from the collected data. For example, in between 8:00-­‐9:00, 12 patients will come where 54.85% are of Type 1, 34.55% are of Type 2, 15.163% are of Type 3, 4.32% are of Type 4, 2.58% are of Type 5 and the rest are of Type 6. It is worth noting that, we assume that the patients of each type arrive as a group at the beginning of the hourly time slot. For example, all of the six patients of Type 1 from 8:00 to 9:00 time slot arrive at 8:00.
The numbers of patients from each type is distributed in such a way that it respects all the constraints described in Section 1.3. Most of the patients of the clinic are from type 1, 2 and 3 and they take less amount of treatment time compared with the patients of other types. Therefore, they are distributed all over the day. Patients of type 4, 5 and 6 take a longer treatment time. Hence, they are scheduled at the beginning of the day to avoid overtime. Because patients of type 4, 5 and 6 come at the beginning of the day, most of type 1 and 2 patients come at mid-­‐day (12:00 to 16:00). Another reason to make the treatment room more crowded in between 12:00 to 16:00 is because the clinic has the maximum number of nurses during this time period. Nurses become tired at the end of the clinic which is a reason not to schedule any patient after 19:00.
Based on the patient arrival schedule and nurse availability a scheduling template is built and shown in Figure 4. In order to build the template, if a nurse is available and there are patients waiting for service, a priority list of these patients will be developed. They are prioritised in a descending order based on their estimated slack time and secondarily based on the shortest service time. The secondary rule is used to break the tie if two patients have the same slack. The slack time is calculated using the following equation:
Slack time = Due time - (Arrival time + Treatment time)
Due time is the clinic closing time. To explain how the process works, assume at hour 8:00 (in between 8:00 to 8:15) two patients in station 1 (one 8-­‐hour and one 15-­‐ minute patient), two patients in station 2 (two 12-­‐hour patients), two patients in station 3 (one 2-­‐hour and one 15-­‐ minute patient) and one patient in station 4 (one 3-­‐hour patient) in total seven patients are scheduled. According to Figure 2, there are seven nurses who are available at 8:00 and it takes 15 minutes to set-­‐up a patient. Therefore, it is not possible to schedule more than seven patients in between 8:00 to 8:15 and the current scheduling is also serving seven patients by this time. The rest of the template can be justified similarly.
PMCID: PMC3562880  PMID: 23386870
14.  Implementing the 2009 Institute of Medicine recommendations on resident physician work hours, supervision, and safety 
Long working hours and sleep deprivation have been a facet of physician training in the US since the advent of the modern residency system. However, the scientific evidence linking fatigue with deficits in human performance, accidents and errors in industries from aeronautics to medicine, nuclear power, and transportation has mounted over the last 40 years. This evidence has also spawned regulations to help ensure public safety across safety-sensitive industries, with the notable exception of medicine.
In late 2007, at the behest of the US Congress, the Institute of Medicine embarked on a year-long examination of the scientific evidence linking resident physician sleep deprivation with clinical performance deficits and medical errors. The Institute of Medicine’s report, entitled “Resident duty hours: Enhancing sleep, supervision and safety”, published in January 2009, recommended new limits on resident physician work hours and workload, increased supervision, a heightened focus on resident physician safety, training in structured handovers and quality improvement, more rigorous external oversight of work hours and other aspects of residency training, and the identification of expanded funding sources necessary to implement the recommended reforms successfully and protect the public and resident physicians themselves from preventable harm.
Given that resident physicians comprise almost a quarter of all physicians who work in hospitals, and that taxpayers, through Medicare and Medicaid, fund graduate medical education, the public has a deep investment in physician training. Patients expect to receive safe, high-quality care in the nation’s teaching hospitals. Because it is their safety that is at issue, their voices should be central in policy decisions affecting patient safety. It is likewise important to integrate the perspectives of resident physicians, policy makers, and other constituencies in designing new policies. However, since its release, discussion of the Institute of Medicine report has been largely confined to the medical education community, led by the Accreditation Council for Graduate Medical Education (ACGME).
To begin gathering these perspectives and developing a plan to implement safer work hours for resident physicians, a conference entitled “Enhancing sleep, supervision and safety: What will it take to implement the Institute of Medicine recommendations?” was held at Harvard Medical School on June 17–18, 2010. This White Paper is a product of a diverse group of 26 representative stakeholders bringing relevant new information and innovative practices to bear on a critical patient safety problem. Given that our conference included experts from across disciplines with diverse perspectives and interests, not every recommendation was endorsed by each invited conference participant. However, every recommendation made here was endorsed by the majority of the group, and many were endorsed unanimously. Conference members participated in the process, reviewed the final product, and provided input before publication. Participants provided their individual perspectives, which do not necessarily represent the formal views of any organization.
In September 2010 the ACGME issued new rules to go into effect on July 1, 2011. Unfortunately, they stop considerably short of the Institute of Medicine’s recommendations and those endorsed by this conference. In particular, the ACGME only applied the limitation of 16 hours to first-year resident physicans. Thus, it is clear that policymakers, hospital administrators, and residency program directors who wish to implement safer health care systems must go far beyond what the ACGME will require. We hope this White Paper will serve as a guide and provide encouragement for that effort.
Resident physician workload and supervision
By the end of training, a resident physician should be able to practice independently. Yet much of resident physicians’ time is dominated by tasks with little educational value. The caseload can be so great that inadequate reflective time is left for learning based on clinical experiences. In addition, supervision is often vaguely defined and discontinuous. Medical malpractice data indicate that resident physicians are frequently named in lawsuits, most often for lack of supervision. The recommendations are: The ACGME should adjust resident physicians workload requirements to optimize educational value. Resident physicians as well as faculty should be involved in work redesign that eliminates nonessential and noneducational activity from resident physician dutiesMechanisms should be developed for identifying in real time when a resident physician’s workload is excessive, and processes developed to activate additional providersTeamwork should be actively encouraged in delivery of patient care. Historically, much of medical training has focused on individual knowledge, skills, and responsibility. As health care delivery has become more complex, it will be essential to train resident and attending physicians in effective teamwork that emphasizes collective responsibility for patient care and recognizes the signs, both individual and systemic, of a schedule and working conditions that are too demanding to be safeHospitals should embrace the opportunities that resident physician training redesign offers. Hospitals should recognize and act on the potential benefits of work redesign, eg, increased efficiency, reduced costs, improved quality of care, and resident physician and attending job satisfactionAttending physicians should supervise all hospital admissions. Resident physicians should directly discuss all admissions with attending physicians. Attending physicians should be both cognizant of and have input into the care patients are to receive upon admission to the hospitalInhouse supervision should be required for all critical care services, including emergency rooms, intensive care units, and trauma services. Resident physicians should not be left unsupervised to care for critically ill patients. In settings in which the acuity is high, physicians who have completed residency should provide direct supervision for resident physicians. Supervising physicians should always be physically in the hospital for supervision of resident physicians who care for critically ill patientsThe ACGME should explicitly define “good” supervision by specialty and by year of training. Explicit requirements for intensity and level of training for supervision of specific clinical scenarios should be providedCenters for Medicare and Medicaid Services (CMS) should use graduate medical education funding to provide incentives to programs with proven, effective levels of supervision. Although this action would require federal legislation, reimbursement rules would help to ensure that hospitals pay attention to the importance of good supervision and require it from their training programs
Resident physician work hours
Although the IOM “Sleep, supervision and safety” report provides a comprehensive review and discussion of all aspects of graduate medical education training, the report’s focal point is its recommendations regarding the hours that resident physicians are currently required to work. A considerable body of scientific evidence, much of it cited by the Institute of Medicine report, describes deteriorating performance in fatigued humans, as well as specific studies on resident physician fatigue and preventable medical errors.
The question before this conference was what work redesign and cultural changes are needed to reform work hours as recommended by the Institute of Medicine’s evidence-based report? Extensive scientific data demonstrate that shifts exceeding 12–16 hours without sleep are unsafe. Several principles should be followed in efforts to reduce consecutive hours below this level and achieve safer work schedules. The recommendations are: Limit resident physician work hours to 12–16 hour maximum shiftsA minimum of 10 hours off duty should be scheduled between shiftsResident physician input into work redesign should be actively solicitedSchedules should be designed that adhere to principles of sleep and circadian science; this includes careful consideration of the effects of multiple consecutive night shifts, and provision of adequate time off after night work, as specified in the IOM reportResident physicians should not be scheduled up to the maximum permissible limits; emergencies frequently occur that require resident physicians to stay longer than their scheduled shifts, and this should be anticipated in scheduling resident physicians’ work shiftsHospitals should anticipate the need for iterative improvement as new schedules are initiated; be prepared to learn from the initial phase-in, and change the plan as neededAs resident physician work hours are redesigned, attending physicians should also be considered; a potential consequence of resident physician work hour reduction and increased supervisory requirements may be an increase in work for attending physicians; this should be carefully monitored, and adjustments to attending physician work schedules made as needed to prevent unsafe work hours or working conditions for this group“Home call” should be brought under the overall limits of working hours; work load and hours should be monitored in each residency program to ensure that resident physicians and fellows on home call are getting sufficient sleepMedicare funding for graduate medical education in each hospital should be linked with adherence to the Institute of Medicine limits on resident physician work hours
Moonlighting by resident physicians
The Institute of Medicine report recommended including external as well as internal moonlighting in working hour limits. The recommendation is: All moonlighting work hours should be included in the ACGME working hour limits and actively monitored. Hospitals should formalize a moonlighting policy and establish systems for actively monitoring resident physician moonlighting
Safety of resident physicians
The “Sleep, supervision and safety” report also addresses fatigue-related harm done to resident physicians themselves. The report focuses on two main sources of physical injury to resident physicians impaired by fatigue, ie, needle-stick exposure to blood-borne pathogens and motor vehicle crashes. Providing safe transportation home for resident physicians is a logistical and financial challenge for hospitals. Educating physicians at all levels on the dangers of fatigue is clearly required to change driving behavior so that safe hospital-funded transport home is used effectively. Fatigue-related injury prevention (including not driving while drowsy) should be taught in medical school and during residency, and reinforced with attending physicians; hospitals and residency programs must be informed that resident physicians’ ability to judge their own level of impairment is impaired when they are sleep deprived; hence, leaving decisions about the capacity to drive to impaired resident physicians is not recommendedHospitals should provide transportation to all resident physicians who report feeling too tired to drive safely; in addition, although consecutive work should not exceed 16 hours, hospitals should provide transportation for all resident physicians who, because of unforeseen reasons or emergencies, work for longer than consecutive 24 hours; transportation under these circumstances should be automatically provided to house staff, and should not rely on self-identification or request
Training in effective handovers and quality improvement
Handover practice for resident physicians, attendings, and other health care providers has long been identified as a weak link in patient safety throughout health care settings. Policies to improve handovers of care must be tailored to fit the appropriate clinical scenario, recognizing that information overload can also be a problem. At the heart of improving handovers is the organizational effort to improve quality, an effort in which resident physicians have typically been insufficiently engaged. The recommendations are: Hospitals should train attending and resident physicians in effective handovers of careHospitals should create uniform processes for handovers that are tailored to meet each clinical setting; all handovers should be done verbally and face-to-face, but should also utilize written toolsWhen possible, hospitals should integrate hand-over tools into their electronic medical records (EMR) systems; these systems should be standardized to the extent possible across residency programs in a hospital, but may be tailored to the needs of specific programs and services; federal government should help subsidize adoption of electronic medical records by hospitals to improve signoutWhen feasible, handovers should be a team effort including nurses, patients, and familiesHospitals should include residents in their quality improvement and patient safety efforts; the ACGME should specify in their core competency requirements that resident physicians work on quality improvement projects; likewise, the Joint Commission should require that resident physicians be included in quality improvement and patient safety programs at teaching hospitals; hospital administrators and residency program directors should create opportunities for resident physicians to become involved in ongoing quality improvement projects and root cause analysis teams; feedback on successful quality improvement interventions should be shared with resident physicians and broadly disseminatedQuality improvement/patient safety concepts should be integral to the medical school curriculum; medical school deans should elevate the topics of patient safety, quality improvement, and teamwork; these concepts should be integrated throughout the medical school curriculum and reinforced throughout residency; mastery of these concepts by medical students should be tested on the United States Medical Licensing Examination (USMLE) stepsFederal government should support involvement of resident physicians in quality improvement efforts; initiatives to improve quality by including resident physicians in quality improvement projects should be financially supported by the Department of Health and Human Services
Monitoring and oversight of the ACGME
While the ACGME is a key stakeholder in residency training, external voices are essential to ensure that public interests are heard in the development and monitoring of standards. Consequently, the Institute of Medicine report recommended external oversight and monitoring through the Joint Commission and Centers for Medicare and Medicaid Services (CMS). The recommendations are: Make comprehensive fatigue management a Joint Commission National Patient Safety Goal; fatigue is a safety concern not only for resident physicians, but also for nurses, attending physicians, and other health care workers; the Joint Commission should seek to ensure that all health care workers, not just resident physicians, are working as safely as possibleFederal government, including the Centers for Medicare and Medicaid Services and the Agency for Healthcare Research and Quality, should encourage development of comprehensive fatigue management programs which all health systems would eventually be required to implementMake ACGME compliance with working hours a “ condition of participation” for reimbursement of direct and indirect graduate medical education costs; financial incentives will greatly increase the adoption of and compliance with ACGME standards
Future financial support for implementation
The Institute of Medicine’s report estimates that $1.7 billion (in 2008 dollars) would be needed to implement its recommendations. Twenty-five percent of that amount ($376 million) will be required just to bring hospitals into compliance with the existing 2003 ACGME rules. Downstream savings to the health care system could potentially result from safer care, but these benefits typically do not accrue to hospitals and residency programs, who have been asked historically to bear the burden of residency reform costs. The recommendations are: The Institute of Medicine should convene a panel of stakeholders, including private and public funders of health care and graduate medical education, to lay down the concrete steps necessary to identify and allocate the resources needed to implement the recommendations contained in the IOM “Resident duty hours: Enhancing sleep, supervision and safety” report. Conference participants suggested several approaches to engage public and private support for this initiativeEfforts to find additional funding to implement the Institute of Medicine recommendations should focus more broadly on patient safety and health care delivery reform; policy efforts focused narrowly upon resident physician work hours are less likely to succeed than broad patient safety initiatives that include residency redesign as a key componentHospitals should view the Institute of Medicine recommendations as an opportunity to begin resident physician work redesign projects as the core of a business model that embraces safety and ultimately saves resourcesBoth the Secretary of Health and Human Services and the Director of the Centers for Medicare and Medicaid Services should take the Institute of Medicine recommendations into consideration when promulgating rules for innovation grantsThe National Health Care Workforce Commission should consider the Institute of Medicine recommendations when analyzing the nation’s physician workforce needs
Recommendations for future research
Conference participants concurred that convening the stakeholders and agreeing on a research agenda was key. Some observed that some sectors within the medical education community have been reluctant to act on the data. Several logical funders for future research were identified. But above all agencies, Centers for Medicare and Medicaid Services is the only stakeholder that funds graduate medical education upstream and will reap savings downstream if preventable medical errors are reduced as a result of reform of resident physician work hours.
PMCID: PMC3630963  PMID: 23616719
resident; hospital; working hours; safety
15.  Efficiency and Usability of a Near Field Communication-Enabled Tablet for Medication Administration 
JMIR mHealth and uHealth  2014;2(2):e26.
Barcode-based technology coupled with the electronic medication administration record (e-MAR) reduces medication errors and potential adverse drug events (ADEs). However, many current barcode-enabled medication administration (BCMA) systems are difficult to maneuver and often require multiple barcode scans. We developed a prototype, next generation near field communication-enabled medication administration (NFCMA) system using a tablet.
We compared the efficiency and usability of the prototype NFCMA system with the traditional BCMA system.
We used a mixed-methods design using a randomized observational cross-over study, a survey, and one-on-one interviews to compare the prototype NFCMA system with a traditional BCMA system. The study took place at an academic medical simulation center. Twenty nurses with BCMA experience participated in two simulated patient medication administration scenarios: one using the BCMA system, and the other using the prototype NFCMA system. We collected overall scenario completion time and number of medication scanning attempts per scenario, and compared those using paired t tests. We also collected participant feedback on the prototype NFCMA system using the modified International Business Machines (IBM) Post-Study System Usability Questionnaire (PSSUQ) and a semistructured interview. We performed descriptive statistics on participant characteristics and responses to the IBM PSSUQ. Interview data was analyzed using content analysis with a qualitative description approach to review and categorize feedback from participants.
Mean total time to complete the scenarios using the NFCMA and the BCMA systems was 202 seconds and 182 seconds, respectively (P=.09). Mean scan attempts with the NFCMA was 7.6 attempts compared with 6.5 attempts with the BCMA system (P=.12). In the usability survey, 95% (19/20) of participants agreed that the prototype NFCMA system was easy to use and easy to learn, with a pleasant interface. Participants expressed interest in using the NFCMA tablet in the hospital; suggestions focused on implementation issues, such as storage of the mobile devices and infection control methods.
The NFCMA system had similar efficiency to the BCMA system in a simulated scenario. The prototype NFCMA system was well received by nurses and offers promise to improve nurse medication administration efficiency.
PMCID: PMC4114445  PMID: 25100043
medication systems; medication errors; mobile applications; automatic data processing; nursing
16.  Usability Study of a Computer-Based Self-Management System for Older Adults with Chronic Diseases 
JMIR Research Protocols  2012;1(2):e13.
Usability can influence patients’ acceptance and adoption of a health information technology. However, little research has been conducted to study the usability of a self-management health care system, especially one geared toward elderly patients.
This usability study evaluated a new computer-based self-management system interface for older adults with chronic diseases, using a paper prototype approach.
Fifty older adults with different chronic diseases participated. Two usability evaluation methods were involved: (1) a heuristics evaluation and (2) end-user testing with a think-aloud testing method, audio recording, videotaping, and interviewing. A set of usability metrics was employed to determine the overall system usability, including task incompletion rate, task completion time, frequency of error, frequency of help, satisfaction, perceived usefulness, and perceived ease of use. Interviews were used to elicit participants’ comments on the system design. The quantitative data were analyzed using descriptive statistics and the qualitative data were analyzed for content.
The participants were able to perform the predesigned self-management tasks with the current system design and they expressed mostly positive responses about the perceived usability measures regarding the system interface. However, the heuristics evaluation, performance measures, and interviews revealed a number of usability problems related to system navigation, information search and interpretation, information presentation, and readability. Design recommendations for further system interface modifications were discussed.
This study verified the usability of the self-management system developed for older adults with chronic diseases. Also, we demonstrated that our usability evaluation approach could be used to quickly and effectively identify usability problems in a health care information system at an early stage of the system development process using a paper prototype. Conducting a usability evaluation is an essential step in system development to ensure that the system features match the users’ true needs, expectations, and characteristics, and also to minimize the likelihood of the users committing user errors and having difficulties using the system.
PMCID: PMC3626148  PMID: 23612015
Usability evaluation; self-management; patient participation; chronic disease
17.  User-Centered Design and Interactive Health Technologies for Patients 
Despite recommendations that patients be involved in the design and testing of health technologies, few reports describe how to involve patients in systematic and meaningful ways to ensure that applications are customized to meet their needs. User-centered design (UCD) is an approach that involves end-users throughout the development process so that technology support tasks, are easy to operate, and are of value to users. In this paper we provide an overview of UCD and use the development of Pocket Personal Assistant for Tracking Health (Pocket PATH), to illustrate how these principles and techniques were applied to involve patients in the development of this interactive health technology. Involving patient-users in the design and testing ensured functionality and usability, therefore increasing the likelihood of promoting the intended health outcomes.
PMCID: PMC2818536  PMID: 19411947
user-centered design; interactive health technologies; lung transplantation; self-monitoring; handheld computers
18.  Rationale, design, and implementation protocol of an electronic health record integrated clinical prediction rule (iCPR) randomized trial in primary care 
Clinical prediction rules (CPRs) represent well-validated but underutilized evidence-based medicine tools at the point-of-care. To date, an inability to integrate these rules into an electronic health record (EHR) has been a major limitation and we are not aware of a study demonstrating the use of CPR's in an ambulatory EHR setting. The integrated clinical prediction rule (iCPR) trial integrates two CPR's in an EHR and assesses both the usability and the effect on evidence-based practice in the primary care setting.
A multi-disciplinary design team was assembled to develop a prototype iCPR for validated streptococcal pharyngitis and bacterial pneumonia CPRs. The iCPR tool was built as an active Clinical Decision Support (CDS) tool that can be triggered by user action during typical workflow. Using the EHR CDS toolkit, the iCPR risk score calculator was linked to tailored ordered sets, documentation, and patient instructions. The team subsequently conducted two levels of 'real world' usability testing with eight providers per group. Usability data were used to refine and create a production tool. Participating primary care providers (n = 149) were randomized and intervention providers were trained in the use of the new iCPR tool. Rates of iCPR tool triggering in the intervention and control (simulated) groups are monitored and subsequent use of the various components of the iCPR tool among intervention encounters is also tracked. The primary outcome is the difference in antibiotic prescribing rates (strep and pneumonia iCPR's encounters) and chest x-rays (pneumonia iCPR only) between intervention and control providers.
Using iterative usability testing and development paired with provider training, the iCPR CDS tool leverages user-centered design principles to overcome pervasive underutilization of EBM and support evidence-based practice at the point-of-care. The ongoing trial will determine if this collaborative process will lead to higher rates of utilization and EBM guided use of antibiotics and chest x-ray's in primary care.
Trial Registration Identifier NCT01386047
PMCID: PMC3184082  PMID: 21929769
19.  Usability and Feasibility of a Tablet-Based Decision-Support and Integrated Record-Keeping (DESIRE) Tool in the Nurse Management of Hypertension in Rural Western Kenya 
Mobile health (mHealth) applications have recently proliferated, especially in low- and middle-income countries, complementing task-redistribution strategies with clinical decision support. Relatively few studies address usability and feasibility issues that may impact success or failure of implementation, and few have been conducted for non-communicable diseases such as hypertension.
To conduct iterative usability and feasibility testing of a tablet-based Decision Support and Integrated Record-keeping (DESIRE) tool, a technology intended to assist rural clinicians taking care of hypertension patients at the community level in a resource-limited setting in western Kenya.
Usability testing consisted of “think aloud” exercises and “mock patient encounters” with five nurses, as well as one focus group discussion. Feasibility testing consisted of semi-structured interviews of five nurses and two members of the implementation team, and one focus group discussion with nurses. Content analysis was performed using both deductive codes and significant inductive codes. Critical incidents were identified and ranked according to severity. A cause-of-error analysis was used to develop corresponding design change suggestions.
Fifty-seven critical incidents were identified in usability testing, 21 of which were unique. The cause-of-error analysis yielded 23 design change suggestions. Feasibility themes included barriers to implementation along both human and technical axes, facilitators to implementation, provider issues, patient issues and feature requests.
This participatory, iterative human-centered design process revealed previously unaddressed usability and feasibility issues affecting the implementation of the DESIRE tool in western Kenya. In addition to well-known technical issues, we highlight the importance of human factors that can impact implementation of mHealth interventions.
PMCID: PMC4314432  PMID: 25612791
Feasibility Studies; Electronic Health Records; Clinical Decision Support Systems; Hypertension; Nurses; World Health
20.  The PAediatric Risk Assessment (PARA) Mobile App to Reduce Postdischarge Child Mortality: Design, Usability, and Feasibility for Health Care Workers in Uganda 
JMIR mHealth and uHealth  2016;4(1):e16.
Postdischarge death in children is increasingly being recognized as a major contributor to overall child mortality. The PAediatric Risk Assessment (PARA) app is an mHealth tool developed to aid health care workers in resource-limited settings such as Sub-Saharan Africa to identify pediatric patients at high risk of both in-hospital and postdischarge mortality. The intended users of the PARA app are health care workers (ie, nurses, doctors, and clinical officers) with varying levels of education and technological exposure, making testing of this clinical tool critical to successful implementation.
Our aim was to summarize the usability evaluation of the PARA app among target users, which consists of assessing the ease of use, functionality, and navigation of the interfaces and then iteratively improving the design of this clinical tool.
Health care workers (N=30) were recruited to participate at Mbarara Regional Referral Hospital and Holy Innocents Children’s Hospital in Mbarara, Southwestern Uganda. This usability study was conducted in two phases to allow for iterative improvement and testing of the interfaces. The PARA app was evaluated using quantitative and qualitative measures, which were compared between Phases 1 and 2 of the study. Participants were given two patient scenarios that listed hypothetical information (ie, demographic, social, and clinical data) to be entered into the app and to determine the patient’s risk of in-hospital and postdischarge mortality. Time-to-completion and user errors were recorded for each participant while using the app. A modified computer system usability questionnaire was utilized at the end of each session to elicit user satisfaction with the PARA app and obtain suggestions for future improvements.
The average time to complete the PARA app decreased by 30% from Phase 1 to Phase 2, following user feedback and modifications. Participants spent the longest amount of time on the oxygen saturation interface, but modifications following Phase 1 cut this time by half. The average time-to-completion (during Phase 2) for doctors/medical students was 3 minutes 56 seconds. All participants agreed they would use the PARA app if available at their health facility. Given a high PARA risk score, participants suggested several interventions that would be appropriate for the sociocultural context in southwestern Uganda, which involved strengthening discharge and referral procedures within the current health care system.
Through feedback and modifications made during this usability study, the PARA app was developed into a user-friendly app, encompassing user expectations and culturally intuitive interfaces for users with a range of technological exposure. Doctors and medical students had shorter task completion times, though all participants reported the usefulness of this tool to improve postdischarge outcomes.
PMCID: PMC4771927  PMID: 26879041
infectious disease; postdischarge mortality; mHealth; prediction model; risk assessment; usability; Africa; resource-limited settings
21.  The added value of user involvement during the development of a feedback system regarding physical functioning for community-dwelling elderly people 
The number of frail elderly people is increasing. Unfortunately, the number of caregivers is not increasing at the same pace, which affects older people, caregivers and healthcare systems. Because of these developments, self-management is becoming more important in healthcare. To support community-dwelling elderly people in their self-management, a system was developed that monitors their physical functioning. This system provides feedback to elderly people and their caregivers regarding physical indicators of frailty. The feedback is provided to elderly people via the screen of a mobile phone. It is important that elderly people understand the content of the feedback and are able to use the mobile phone properly. If not, it is unlikely that the system can support self-management. Many interactive health technologies that have been developed do not fulfil their promises. An important reason for this is that human and other non-technology issues are not sufficiently taken into consideration during the development process.
To collaborate with elderly people during the development and evaluation of a feedback system for community-dwelling elderly people regarding physical functioning.
An iterative user-centered design that consists of five phases was used to develop and evaluate the feedback system. These five phases were: 1) Selection of users, 2) Analysis of users and their context, 3) Identification of user needs, 4) Development of a prototype, and 5) Evaluation of the prototype. Three representatives of a target group panel for elderly people were selected in phase 1. They shared their needs and preferences during three expert group meetings that took place during the development process. This resulted in the development of a prototype which was first evaluated in a heuristic evaluation. Once adjustments were made, 11 elderly people evaluated the adjusted prototype using a think aloud procedure. They rated the usability and acceptability of the developed interface on a scale from 1 till 7 using an adapted version of the Post-Study System Usability Questionnaire (PSSUQ).
A feedback system was developed that provides feedback regarding physical indicators of frailty via a touch screen mobile phone. The interface uses colours, smiley’s, and spoken/written messages to provide feedback that is easy to understand. The heuristic evaluation revealed that there were some problems with consistency and the use of user language. The think aloud evaluation showed that the 11 elderly people were able to navigate through the interface without much difficulty despite some small problems related to the lay-out of the interface. The mean score on an adapted version of the PSSUQ was 5.90 (SD 1.09) which indicates high user satisfaction and good usability.
The involvement of end-users significantly influenced the lay-out of the interface that was developed. This resulted in an interface that was accepted by the target group. Evaluation of the prototype revealed that the usability of the interface was good. The feedback system will only succeed in supporting self-management when elderly people are able to use the interface and understand the feedback. The input of elderly people during the development process contributed to this.
PMCID: PMC3571197
user-centered design; elderly people; self management; physical functioning, usability
22.  A Framework to Assist Health Professionals in Recommending High-Quality Apps for Supporting Chronic Disease Self-Management: Illustrative Assessment of Type 2 Diabetes Apps 
JMIR mHealth and uHealth  2015;3(3):e87.
This paper presents an approach to assist health professionals in recommending high quality apps for supporting chronic disease self-management. Most app reviews focus on popularity, aesthetics, functionality, usability, and information quality. There is no doubt these factors are important in selecting trustworthy apps which are appealing to users, but behavioral theory may be also be useful in matching the apps to user needs.
The framework developed aims to be methodologically sound, capable of selecting popular apps which include content covered by evidence-based programs, consistent with behavioral theory, as well as a patient-centered approach for matching apps to patients’ individual needs.
A single disease—type 2 diabetes—was selected to illustrate how the framework can be applied as this was deemed to represent the types of strategies used in many chronic diseases. A systematic approach based on behavioral theory and recommendations from best practice guidelines was developed for matching apps to patients’ needs. In March 2014, a series of search strategies was used to identify top-rated iPhone and Android health apps, representing 29 topics from five categories of type 2 diabetes self-management strategies. The topics were chosen from published international guidelines for the management of diabetes. The senior author (KH) assessed the most popular apps found that addressed these topics using the Behavioral Theory Content Survey (BTS), which is based on traditional behavioral theory. A tool to assist decision making when using apps was developed and trialed with health professionals for ease of use and understanding.
A total of 14 apps were assessed representing all five topic categories of self-management. Total theoretical scores (BTS scores) were less than 50 on a 100-point scale for all apps. Each app scored less than 50% of the total possible BTS score for all four behavioral theories and for most of the 20 behavioral strategies; however, apps scored higher than 50% of the total possible BTS score for specific strategies related to their primary focus. Our findings suggest that the apps studied would be more effective when used in conjunction with therapy than as stand-alone apps. Apps were categorized according to topic and core intervention strategies. A framework for matching apps to identified patient needs was developed based on app categorization and principles of patient-centered care. The approach was well accepted and understood by a convenience sample of health practitioners.
The framework presented can be used by health practitioners to better match apps with client needs. Some apps incorporate highly interactive strategies of behavioral theory, and when used as an adjunct may increase patient participation and the effectiveness of therapy.
PMCID: PMC4704910  PMID: 26369346
mobile apps; chronic disease; patient-centered care; technology
23.  Redesigning printed educational materials for primary care physicians: design improvements increase usability  
Printed educational materials (PEMs) are a frequently used tool to disseminate clinical information and attempt to change behavior within primary care. However, their effect on clinician behavior is limited. In this study, we explored how PEMs can be redesigned to better meet the needs of primary care physicians (PCPs) and whether usability and selection can be increased when design principles and user preferences are used.
We redesigned a publicly available PEM using physician preferences, design principles, and graphic designer support. We invited PCPs to select their preferred document between the redesigned and original versions in a discrete choice experiment, followed by an assessment of usability with the System Usability Scale and a think aloud process. We conducted this study in both a controlled and opportunistic setting to determine whether usability testing results vary by study location. Think aloud data was thematically analyzed, and results were interpreted using the Technology Acceptance Model.
One hundred and eighty four PCPs participated in the discrete choice experiment at the 2014 Family Medicine Forum, a large Canadian conference for family physicians. Of these, 87.7 % preferred the redesigned version. Follow-up interviews were held with a randomly selected group of seven participants. We repeated this in a controlled setting in Toronto, Canada, with a set of 14 participants. Using the System Usability Scale, we found that usability scores were significantly increased with the redesign (p < 0.001). We also found that when PCPs were given the choice between the two versions, they selected the redesigned version as their preferred PEM more often than the original (p < 0.001). Results did not appear to differ between the opportunistic and controlled setting. We used the results of the think aloud process to add to a list of end user preferences developed in a previous study.
We found that redesigning a PEM with user preferences and design principles can improve its usability and result in the PEM being selected more often than the original. We feel this finding supports the involvement of the user, application of design principles, and the assistance of a graphic designer in the development of PEMs.
Electronic supplementary material
The online version of this article (doi:10.1186/s13012-015-0339-5) contains supplementary material, which is available to authorized users.
PMCID: PMC4634785  PMID: 26537589
Primary care; Educational materials; Usability; Design
24.  Evaluation of an integrated graphical display to promote acute change detection in ICU patients 
The purpose of this study was to evaluate ICU nurses’ ability to detect patient change using an integrated graphical information display (IGID) versus a conventional tabular ICU patient information display (i.e. electronic chart).
Using participants from two different sites, we conducted a repeated measures simulator-based experiment to assess ICU nurses’ ability to detect abnormal patient variables using a novel IGID versus a conventional tabular information display. Patient scenarios and display presentations were fully counterbalanced.
We measured percent correct detection of abnormal patient variables, nurses’ perceived workload (NASA-TLX), and display usability ratings.
32 ICU nurses (87% female, median age of 29 years, and median ICU experience of 2.5 years) using the IGID detected more abnormal variables compared to the tabular display [F (1,119)=13.0, p < 0.05]. There was a significant main effect of site [F (1, 119)=14.2], with development site participants doing better. There were no significant differences in nurses’ perceived workload. The IGID display was rated as more usable than the conventional display, [F (1, 60)=31.7].
Overall, nurses reported more important physiological information with the novel IGID than tabular display. Moreover, the finding of site differences may reflect local influences in work practice and involvement in iterative display design methodology. Information displays developed using user-centered design should accommodate the full diversity of the intended user population across use sites.
PMCID: PMC3414670  PMID: 22534099
medical informatics; human engineering; man-machine systems; intensive care
25.  User-centered design and the development of patient decision aids: protocol for a systematic review 
Systematic Reviews  2015;4(1):11.
Providing patient-centered care requires that patients partner in their personal health-care decisions to the full extent desired. Patient decision aids facilitate processes of shared decision-making between patients and their clinicians by presenting relevant scientific information in balanced, understandable ways, helping clarify patients’ goals, and guiding decision-making processes. Although international standards stipulate that patients and clinicians should be involved in decision aid development, little is known about how such involvement currently occurs, let alone best practices. This systematic review consisting of three interlinked subreviews seeks to describe current practices of user involvement in the development of patient decision aids, compare these to practices of user-centered design, and identify promising strategies.
A research team that includes patient and clinician representatives, decision aid developers, and systematic review method experts will guide this review according to the Cochrane Handbook and PRISMA reporting guidelines. A medical librarian will hand search key references and use a peer-reviewed search strategy to search MEDLINE, EMBASE, PubMed, Web of Science, the Cochrane Library, the ACM library, IEEE Xplore, and Google Scholar. We will identify articles across all languages and years describing the development or evaluation of a patient decision aid, or the application of user-centered design or human-centered design to tools intended for patient use. Two independent reviewers will assess article eligibility and extract data into a matrix using a structured pilot-tested form based on a conceptual framework of user-centered design. We will synthesize evidence to describe how research teams have included users in their development process and compare these practices to user-centered design methods. If data permit, we will develop a measure of the user-centeredness of development processes and identify practices that are likely to be optimal.
This systematic review will provide evidence of current practices to inform approaches for involving patients and other stakeholders in the development of patient decision aids. We anticipate that the results will help move towards the establishment of best practices for the development of patient-centered tools and, in turn, help improve the experiences of people who face difficult health decisions.
Systematic review registration
PROSPERO CRD42014013241
Electronic supplementary material
The online version of this article (doi:10.1186/2046-4053-4-11) contains supplementary material, which is available to authorized users.
PMCID: PMC4328638  PMID: 25623074
Patient decision aids; Decision support; Shared decision-making; Patient education; Counseling; User-centered design; Human-centered design; Patient partnership; Stakeholder engagement; Implementation; Knowledge translation; Patient-centered care

Results 1-25 (1677521)