Informatics Systems to Promote Improved Care for Chronic Illness: A Literature Review

doi:10.1197/jamia.M2255

J Am Med Inform Assoc. 2007 Mar-Apr; 14(2): 156–163.

PMCID: PMC2213468

Informatics Systems to Promote Improved Care for Chronic Illness: A Literature Review

David Dorr, MD, MS,^a Laura M. Bonner, PhD,^b Amy N. Cohen, PhD,^c Rebecca S. Shoai, MPH, MSW,^c Ruth Perrin, MA,^d Edmund Chaney, PhD,^b and Alexander S. Young, MD, MSHS^c^e

^aOregon Health & Science University, Department of Medical Informatics & Clinical Epidemiology, Portland, OR

^bNorthwest HSR&D Center of Excellence, VA Puget Sound Healthcare System, Seattle, WA

^cVA Desert Pacific Mental Illness Research, Education and Clinical Centers (MIRECC), Los Angeles, CA

^dVA Information Resource Center (VIReC), Hines, IL

^eUniversity of California Department of Psychiatry, Los Angeles, CA.

Correspondence and reprints: David A. Dorr, 3181 SW Sam Jackson Park Rd, Mailcode:BICC, Portland, OR 97239. (Email: dorrd/at/ohsu.edu).

Received August 24, 2006; Accepted December 12, 2006.

This article has been cited by other articles in PMC.

Abstract

Objective

To understand information systems components important in supporting team-based care of chronic illness through a literature search.

Design

Systematic search of literature from 1996-2005 for evaluations of information systems used in the care of chronic illness.

Measurements

The relationship of design, quality, information systems components, setting, and other factors with process, quality outcomes, and health care costs was evaluated.

Results

In all, 109 articles were reviewed involving 112 information system descriptions. Chronic diseases targeted included diabetes (42.9% of reviewed articles), heart disease (36.6%), and mental illness (23.2%), among others. System users were primarily physicians, nurses, and patients. Sixty-seven percent of reviewed experiments had positive outcomes; 94% of uncontrolled, observational studies claimed positive results. Components closely correlated with positive experimental results were connection to an electronic medical record, computerized prompts, population management (including reports and feedback), specialized decision support, electronic scheduling, and personal health records. Barriers identified included costs, data privacy and security concerns, and failure to consider workflow.

Conclusion

The majority of published studies revealed a positive impact of specific health information technology components on chronic illness care. Implications for future research and system designs are discussed.

Introduction

Over 100 million persons in the United States have one or more chronic illnesses and more than 30 million are disabled from their illness.¹ Costs for chronic illness care are approximately 75% of total health care costs, and rising. Senior patients are hardest hit, with 65% of Medicare recipients having 2 or more chronic illnesses and accounting for 96% of all Medicare expenditures.² Mental health care is especially pertinent to discussions of chronic illness care, since mental illnesses are growing in incidence (and yet still significantly under-diagnosed) and represent large costs.³ Despite heavy expenditures, care for chronic illness is poor—treatments known to be beneficial are provided about 50% of the time and ineffective treatments may be given 20%–30% of the time.^{4, 5}

Improving care for chronically ill patients requires a longitudinal, team-based approach. Models and change packages exist for reorganizing care to improve collaboration and quality of care, such as the Chronic Care Model⁶ and two Veterans Administration (VA) Quality Enhancement Research Initiative (QUERI) projects, Enhancing Quality-of-care in Psychosis (EQUIP)⁷ and Translating Best Practices for Depression Care into VA Care Solutions (TIDES).⁸ Many models specify the use of health information technology (HIT) as a key component in managing populations and complex communications.⁹ As part of the Creating HealtheVet Informatics Applications for Collaborative Care (CHIACC) study, a VA project to design and evaluate software that supports chronic illness care, we sought to understand the specific information technology components important for success.

HIT can provide knowledge about guidelines and safety, information about patient conditions, treatments and other pertinent characteristics, and reminders to providers at the point-of-care of important quality steps. While important, these are not sufficient to ensure effective chronic illness care. According to chronic care models, special information-based tasks and processes are needed, including facilitating population-based care, tracking measures of health over time (such as depression scores), involving the care team, including the patient, and giving feedback about progress. Most systems fall short of providing the necessary support. Little is known about how to create and successfully implement a comprehensive system which incorporates all these components and will positively impact health care for patients with complex chronic illness.

Previous systematic reviews of chronic disease management and HIT provide insight but no clear answer. First, Hillestad et al. highlighted the promise of systems, basing an estimate of near and long term savings of over $150 billion on studies of successful implementations of HIT.⁹ In a broad review of HIT, de Keiser and Ammenwerth reviewed studies for content (983 studies)¹⁰ and outcomes (64 studies),¹¹ and found that 70% of process studies showed positive results and half of the outcome or efficiency reports were positive. Similarly, Chaudhry et al. reviewed information systems components in 257 studies and found improved adherence to protocols, reduced errors, and improved surveillance; however, most of these papers were from the same institutions and the generalizability was uncertain.¹² Weingarten et al., in a large meta-analysis of disease management studies, showed positive effect sizes for process and efficiency of provider reminders (0.52) and feedback (0.61) and for outcomes of patient (0.27) and physician reminders (0.22), although a large proportion of the studies showed no effect.¹³

Looking at individual system components, several studies of computerized decision support systems measured positive effects on process (58%–68%). Fewer (13%) reported positive outcomes for specific diseases.^14–18 Factors important for success in these studies were timeliness of alerts, automatic generation of alerts, and integration into workflow. Computerized patient education has shown mixed results; one study showed very successful knowledge transfer (21/22 articles)¹⁹ while another showed little improvement.²⁰ Reviews of other system components (self-management, population-based care) did not specify attributes of HIT that contributed to success or failure. Thus, although HIT systems generally have high potential benefits and some demonstrated effect, specific components in chronic disease management associated with success are unclear.

To better understand how to build information systems to support collaborative, team-based, chronic illness care, we performed a novel literature review. Our primary goal was to understand which elements are necessary for software to facilitate best practices and which bring the highest likelihood of successful implementation in a broad network. The literature review targeted functions of health care information systems and improvements in processes and outcomes attributable to HIT and it sought to understand lessons learned from failures.

Methods

Search Strategy and Article Selection

Studies were identified by searching MEDLINE, PREMEDLINE, Business Source Premier, ABI, and the Cochrane Library for January 1, 1996–February 28, 2005 using search concepts (with appropriate synonyms): 1) informatics/information systems; 2) patient care management/collaborative care; and 3) chronic illness. This strategy was supplemented by articles identified as key in the reference sections of the studies received and from experts in the field.

Abstracts from identified articles were each rated separately by two reviewers into A (exact topic match), B (doubtful match), and C (no match) categories. Agreement was high (kappa=0.78, p<0.01) and consensus was used to resolve disagreements. All A articles were read and key information was abstracted from them unless: 1) information systems were not a major part of the study; 2) the study was wholly theoretical; or 3) the article did not describe what was done in enough detail to understand important components according to two or more reviewers. Review articles were analyzed separately. Only their general purpose and results were abstracted for comparison with current efforts.

Abstraction Method and Template Development

The primary units of analysis were the components of the system being studied, categorized using the eight Institute of Medicine (IOM) domains (Health Information and Data, Decision Support, Results Management, Communication/Connectivity, Population Health Management, Order Entry, Patient Support, Administrative Processes).²¹ When articles compared two or more systems, each system was abstracted individually. Multiple articles discussing the same system were analyzed independently if the components under study were different.

An abstraction template and a data dictionary were developed by consensus and iterative review of the team. The abstraction template included study design, setting (clinic, hospital, etc.), disease addressed, information system components and quality of care components. Study quality was evaluated based on study design, with experiments defined as the highest quality. To be classified as an experiment, a study had to have a control or comparison group, measure outcomes, and make appropriate statistical analyses.²² Other study designs included non-experiment observation, description, and case study.

Statistical Analyses

At least one reviewer of seven independently abstracted each study. The first author abstracted all review articles and independently re-reviewed 10% of all the articles as a verification step. Key verification variables included the study type, a summary variable for presence of each IOM domain (8 total), presence of quality, accuracy, and usability evaluations (3 total), and presence of cost-effectiveness analysis. A weighted kappa measurement was used to compare these 13 key indicators of consistency.

Several limitations precluded formal meta-analysis. These limitations included a paucity of randomized, controlled trials (RCTs) among the articles reviewed, varying definitions of positive outcomes, inconsistent outcomes measurement, and lack of complete information system descriptions. Therefore, article counting methods were used where appropriate and a weighted sign test used to detect positivity of each study’s results. Correlations were used to understand single components most associated with improvements in process and outcomes. Multivariable logistic regression was used to predict a positive effect on quality of care scores (outcomes) with information system components, study designs, and setting/disease treated.

Results

Articles Selected

Of the 704 abstracts yielded by the search strategy (

), 158 were chosen as highly relevant (A) by two reviewers. In all, 16 were reviews (considered for overlap with current efforts only) and 33 did not meet minimum criteria, leaving 109 articles with 112 system descriptions. In fact, 81 systems were named in the articles but 31 of them were presented in more than one paper with descriptions of markedly different aspects of the systems or implementations in different settings; hence each of the 112 system descriptions is included in this analysis.

Figure 1

Article selection process.

summarizes the 109 articles, their study designs, user types, the diseases/conditions targeted, and the information systems under consideration. In all, 50 (46%) were experiments, 55 were observational (with evaluation) or descriptive studies (without), and the remaining were case studies. Agreement among reviewers about study type was moderate (κ=0.40–0.82; weighted κ =0.63). Differences predominantly involved articles assessing accuracy and usability (rather than assessing changes in quality of patient care) and were resolved through consensus. All details of the 109 articles covering 112 systems, including references, are given in Appendix 1, available at www.jamia.org as an on-line data supplement.

Table 1

Table 1 Scope of Articles Addressing Comprehensive Care for Chronic Disease

Targeted Disorder

Diabetes was the disorder most frequently targeted (44%, 48 studies). Heart disease was next most frequent (38%, 41 studies). Depression was included in 17% (18 studies) and schizophrenia in 7% (eight studies). About one third (34%) addressed multiple disorders.

Settings and Users

The majority of studies (60%) covered systems implemented in outpatient settings including 59% in primary care and 28% also in specialty care. (Informatics systems could be implemented in more than one setting simultaneously.) Forty-three percent studied hospital implementations and 42% studied multi-site systems. Seventeen percent looked at informatics systems implemented within the VA. Physicians were most frequently the intended users (39%). Nurses (37%) and patients (17%) were also commonly identified as end-users.

Components

The most commonly included IOM component of the systems studied was Health Information and Data (

). All systems contained some health information and data; 73% included some linkage to a larger electronic medical record and/or some attempt to systematically capture outcomes data. The majority also included Decision Support (67%), usually computerized prompts (54%) and/or access to guidelines (42%). Results Management (57%) and Communication/Connectivity (52%) were also included in a majority of systems. The most common Results Management features were the ability to generate patient summary sheets and longitudinal displays of data. The most frequently included subcomponents of Communication/Connectivity were electronic messaging (28%), health information exchange (24%), and telemedicine/telemonitoring (13%). Other components frequently reported were Population Management (46%), Order Entry (43%), Patient Support (32%), and Administrative Processes (21%). Under Population Management, subcomponents included disease management (19%), audit/feedback (19%), disease registries (18%), population reporting (17%), and public health reporting (5%).

Table 2

Table 2 Components in IOM Domains (Ranked by Most Frequent) Used in Information Systems for Chronic Disease Care and Correlation with Positive Quality of Care Outcomes

Measures of Success

We examined results reported in 30 observational studies; overall, non-experiments were found far more likely than experiments to be associated with positive results (non-experiments 94%; experiments 67%; Fisher’s exact p<0.0001). Due to this positivity bias, we limit remaining results reported here to experimental studies.

Results of the 50 experimental studies were divided into measures associated with technical aspects of the software, process-of-care measures, and changes in clinical outcomes attributable to implementation of HIT.

displays these measures using a count of positive, negative, and neutral studies.

Figure 2

Results of technological, process, and clinical outcomes measures in 50 experimental studies of information technology used for care of chronic disease.

Process of care: Guideline adherence (for example, screening for target disorders, conducting lab tests on a recommended schedule) was the most frequently evaluated process outcome; it was assessed in 19 studies with 79% (15) positive and 21% (4) neutral assessments. Change in visit frequency (e.g., decrease in emergency visits) was assessed in ten studies with 50% in the positive direction and 50% neutral. Documentation (e.g., provider documentation of diagnostic criteria for specified disorders) was assessed in six studies with 83% (five) positive and 17% (one) neutral assessment. Treatment adherence (primarily adherence to medications) was assessed in three studies with 67% (two) positive and 33% (one) neutral assessment. Change in referral rate was assessed in two studies; both had neutral results. Screening and testing was assessed in two studies; both had positive results. Eleven studies assessed cost (typically involving some analysis of informatics system costs and savings to the organization) with 91% (ten) positive and 9% (one) neutral outcomes.

Clinical outcomes: Changes in laboratory values were assessed in ten studies with 50% positive and 50% neutral outcomes. Changes in scores on standardized instruments were assessed in ten studies with 30% positive, 60% neutral and 10% negative outcomes. Number of hospitalizations was assessed in seven studies with 43% (three) positive and 57% (four) neutral outcomes. Quality of life was assessed four times with 75% (three) positive and 25% (one) neutral outcome. Disease complications were assessed in only two studies with one positive and one neutral outcome.

Relationship between informatics system components and outcomes:

displays the relationship between system components and improvement in study measures. One subcomponent of Health Information and Data, the ability to exchange data with an electronic medical record, was positively related to improvements in process outcomes (r=0.28, p=0.05). Decision support was moderately helpful in the form of computerized prompts (r=0.20, p=0.08), but were related to failure when only electronic access to guidelines were provided through the system (r=−0.37, p=0.02). Population management in general (r=0.25, p=0.06) and especially features such as generating reports of traditional (disease state, adherence) and non-traditional (unfinished care plan elements, telephone calls) measures (r=0.32, p=0.02) and auditing/providing feedback to providers (r=0.31, p=0.02) were positively associated with process and outcomes improvement. Advanced, specialized Order Entry systems, such as those including disease specific checks and corollary orders templates, those facilitating ordering of care plan elements like referral to a specialist or nurse care manager, and those wherein team members other than the primary care provider can create role-specific orders, were also related to improvements in process outcomes (r=0.41, p=0.02). Patient Support/Patient Portals were modestly associated with success (r=0.20, p=0.10). Electronic scheduling (an Administrative task) was associated with success (r=.19, p=0.08).

Figure 3

Correlation between presence of information system components and process and outcome measure changes (Base = 112 system descriptions). n.s. = not significant.

Multivariate analysis: Due to the small number of experimental studies, we could not create a comprehensive mathematical model of the relationship between informatics components and clinical or care process outcomes. A multivariable logistic regression demonstrated strong concordance with the above results, especially the combination of advanced Order Entry features (for instance, information about drug interactions) with Decision Support (like care plan elements reminders) and key Population Management and Administrative Processes features (such as follow-up of care plan, scheduling, and referrals) (c=0.86; Hosmer-Lemeshow p=0.28). Access to Population Management and Health Information and Data features remained strongly identified with positive results but insufficient variation existed to determine the strength of the association in the multivariate models.

Sociotechnical Assessment: Thirty studies described some usability assessment, using methodologies such as user interviews, surveys, number of encounters with system, etc. Of these studies, 80% (24) had mostly positive assessments of informatics system usability; 13% (four) were neutral and 7% (two) were negative. Most were not formal usability studies, making comparisons difficult. Accuracy was assessed in only four studies using methods such as comparison of informatics-generated advice to expert-clinician advice; all four studies found positive results. In addition, fifty-one studies included qualitative descriptions of implementation issues, usability suggestions, and recommended content. Suggestions for success in implementing HIT systems included involving end-users in the development process, responsiveness to end-user feedback, and thorough training. Having a physician buy-in to teach peers about the software was helpful. Barriers to adoption of HIT systems included concerns about the impact of HIT use on the clinical encounter, security issues, and concerns about resources. Failure to consider increased time to use the system (performance usability) or alteration in workflow were also barriers. Barriers to building informatics systems included resource-related challenges, technological difficulties, security concerns, and social barriers related to the availability of particular technologies in some areas. Usability recommendations ranged from concrete details of the user interface to more global suggestions about workflow (designating one person to handle all on-line messages, enabling real-time data entry for synchronous decision support). Improvements to decision support tools were the most frequently requested content modifications; specifically requested were support for medication and dosage decisions and additional features to support adherence to guidelines. Reported unintended outcomes were mostly positive and included improved communication and more efficient workflow.

Twenty-five studies contained discussions about positives and negative aspects of their software systems (see Appendix 2, available at www.jamia.org as an on-line data supplement). These “lessons learned” can be grouped into themes as follows: cost-benefit for providers and other staff; alerts and reminders; usability; decision support; and other issues. Cost-benefit issues included initial time for training,²³ and other start-up endeavors,²⁴ staff anxiety about using the software,²⁵ and integration into the facility workflow,²⁶ among other concerns. Alerts and reminders raised some concerns about the ability to ignore them²⁷ and providers’ fears of being overwhelmed by too many messages.²⁸ Usability concerns included the need to make the system easy to install,²⁹ learn,²³ and use.^30,31 Decision support was used infrequently in one study;³² in another study, real time data entry to improve recommendations was a suggested enhancement.³³ Additional suggestions included adding videoconferencing³⁴ and providing feedback to residents to support guideline adherence.³⁵

Discussion

These studies of information systems that address longitudinal care for chronic illness generally had positive results (67%). Descriptions of systems, while spare, did allow us to identify core components related to success: connection to a broad electronic health record (EHR) system; order entry, especially when focused on the care team, specific to disease, and allowing longitudinal care planning (such as specialist or care manager referrals); and population-based reporting and feedback (such as reporting back unfinished care plan elements). Decision support in the form of computerized prompts was found to be important but simply providing guidelines, even as an electronic resource, was significantly less likely than other technologies to bring success; this concurs with the conclusion of other, broader reviews.¹⁸

For information systems to be successful, an appropriate non-IT system of care must be in place, and the use of specialized IT components must fit with systems of care. Usability is essential to successful implementation of a software system. However, in the literature reviewed, formal usability assessment was rare. A handful of studies contained comments or lessons learned about usability, highlighting the importance of making things easier for the user but not offering concrete information about optimal usability evaluation methods.

As quality of care and patient safety concerns for patients with chronic illness increase, so do the requirements for accurate measurement and well-specified change hypotheses. The most effective utilization of HIT for chronic disease care has been unclear, especially when addressing important but hard to track diseases such as schizophrenia and depression. The devil may indeed be in the details; creating and using these components has been challenging for most practices.³⁶ Learning from previously implemented systems supports efforts to leverage current knowledge into optimal improvement.

This study has several limitations. First, a significant bias was found in uncontrolled studies, where process and clinical outcomes measures had 41% higher odds to be positive than did experiments. The nature of informatics experiments frequently makes RCTs difficult (e.g., blinding is not possible, randomization can be technically challenging, and the unit of study is not clear in a complex, team-based system). While observational studies with no controls can be discounted, quasi-experimental designs often must provide measures of success.

Meta-analysis was not possible due to heterogeneity of the articles. The quality of the studies varied, and there were few RCTs; however, we did not find a positive bias when comparing RCTs to controlled experiments (OR=1.01). Descriptions of the systems were poor; standardized reporting methods for informatics systems do not yet exist and space limitations may have added to this issue. However, it is likely that core innovative components were mentioned in the articles, while the authors omitted “usual” EHR components. Finally, the different process and outcomes measures, even within the same categories, make the definition of success more difficult. Although strict standards were used, the core differences in the measures may have led to failure to detect improvement.

One particular bias noted by Chaudhry et al.—including multiple articles reviewing a single institution’s HIT—was not seen here.¹² Although the VA CPRS had the most articles (six) reviewing its function, the vast majority of these articles came from different settings and reviewed different components.

Future research should focus on a few key areas. First, trends were detected in the less frequent studies on electronic scheduling, patient portals, and telemedicine, highlighting their potential importance. Next, the perceived lack of use of otherwise successful components in standard EHR implementations should be quantified and barriers for use identified. For instance, others have found focusing on the aspects of an information system related to quality care was a risk for small practices due to the reimbursement system.³⁶ Third, both the recent review by Chaudhry and this one identify the need for standardized reporting methods for both outcomes and information systems. Multiple frameworks, such as the IOM information systems domains (used here) and the HL7 Electronic Health Record System function model should be adapted into core categories and specific areas for reporting (akin to CONSORT or MOOSE).^37–39 Efforts around the certification of electronic health records by Certification Commission for Healthcare Information Technology (CCHIT) should speed the ability to assess functionality as experiments are performed.⁴⁰ Finally, this framework can be used as innovators create systems of care, as we plan to do with the CHIACC project, or simply used as a checklist when considering adding HIT functionality to programs to support care for chronic illness.

Conclusion

In our review of information systems to support care for chronic illness, we found many were successfully implemented. Studies assessed impact of informatics systems on process of care variables including guideline adherence, visit frequency, documentation, treatment adherence, referral rate, appropriate screening/testing, and cost; studies reported mostly positive and some neutral results. Several specific subcomponents of informatics systems were associated with positive change in process outcomes. Only one subcomponent (access to guidelines) was related to failure to improve the target outcome.

Although few studies presented a formal usability assessment, many included a discussion of the importance of usability for system adoption and sustained use. Ease of use and respect for providers’ time constraints are important usability considerations.

Footnotes

This work was supported by the Department of Veterans Affairs through the Health Services Research & Development Service Quality Enhancement Research Initiative (MHS-03-218), and the Desert Pacific Mental Illness Research, Education and Clinical Centers (MIRECC); and by the John A. Hartford Foundation and National Library of Medicine (K22 LM 8427-01).

The funders did not have any role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; and preparation, review, or approval of the manuscript. David A. Dorr had full access to all of the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis. Any opinions expressed are only those of the authors and do not necessarily represent the views of any affiliated institutions.

References

1. Wolff JL, Boult C, Boyd C, Anderson G. Newly reported chronic conditions and onset of functional dependency J Am Geriatr Soc 2005;53:851-855. [PubMed]

2. Wolff JL, Starfield B, Anderson G. Prevalence, expenditures, and complications of multiple chronic conditions in the elderly Arch Intern Med 2002;162:2269-2276. [PubMed]

3. Neumeyer-Gromen A, Lampert T, Stark K, Kallischnigg G. Disease management programs for depression: a systematic review and meta-analysis of randomized controlled trials Med Care 2004;42:1211-1221. [PubMed]

4. McGlynn EA, Asch SM, Adams J, Keesey J, Hicks J, DeCristofaro A, et al. The quality of health care delivered to adults in the United States N Engl J Med 2003;348:2635-2645. [PubMed]

5. Wennberg JE. Practice variation: implications for our health care system Manag Care 2004;13(9 Suppl):3-7. [PubMed]

6. Bodenheimer T. Interventions to improve chronic illness care: evaluating their effectiveness Dis Manag 2003;6:63-71. [PubMed]

7. Young AS, Mintz J, Cohen AN, Chinman MJ. A network-based system to improve care for schizophrenia: the medical informatics network tool (MINT) J Am Med Inform Assoc 2004;11:358-367. [PMC free article] [PubMed]

8. Felker BL, Chaney E, Rubenstein LV, Bonner LM, Yano EM, Parker LE, et al. Developing effective collaboration between primary care and mental health providers Prim Care Companion J Clin Psychiatry 2006;8:12-16. [PubMed]

9. Hillestad R, Bigelow J, Bower A, Girosi F, Meili R, Scoville R, et al. Can electronic medical record systems transform health care?Potential health benefits, savings, and costs. Health Aff (Millwood) 2005;24:1103-1117. [PubMed]

10. Ammenwerth E, de Keizer N. An inventory of evaluation studies of information technology in health care trends in evaluation research 1982-2002 Methods Inf Med 2005;44:44-56. [PubMed]

11. de Keizer N AE. The effects and quality of medical IT evaluation studies: Trends in 1982–2002. AMIA 2005 Symposium Proceedings. 2005.

12. Chaudhry B, Wang J, Wu S, Maglione M, Mojica W, Roth E, et al. Systematic Review: Impact of health information technology on quality, efficiency, and costs of medical care Ann Intern Med 2006;144:E2-E22.

13. Weingarten SR, Henning JM, Badamgarav E, Knight K, Hasselblad V, Gano Jr. A, et al. Interventions used in disease management programmes for patients with chronic illness—which ones work?Meta-analysis of published reports. BMJ 2002;325:925. [PMC free article] [PubMed]

14. Balas EA, Austin SM, Mitchell JA, Ewigman BG, Bopp KD, Brown GD. The clinical value of computerized information servicesA review of 98 randomized clinical trials. Arch Fam Med 1996;5:271-278. [PubMed]

15. Balas EA, Jaffrey F, Kuperman GJ, Boren SA, Brown GD, Pinciroli F, et al. Electronic communication with patientsEvaluation of distance medicine technology [see comments]. JAMA 1997;278:152-159. [PubMed]

16. Bennett JW, Glasziou PP. Computerised reminders and feedback in medication management: a systematic review of randomised controlled trials Med J Aust 2003;178:217-222. [PubMed]

17. Garg AX, Adhikari NK, McDonald H, Rosas-Arellano MP, Devereaux PJ, Beyene J, et al. Effects of computerized clinical decision support systems on practitioner performance and patient outcomes: a systematic review JAMA 2005;293:1223-1238. [PubMed]

18. Kawamoto K, Houlihan CA, Balas EA, Lobach DF. Improving clinical practice using clinical decision support systems: a systematic review of trials to identify features critical to success BMJ 2005;330:765. [PMC free article] [PubMed]

19. Krishna S, Balas EA, Spencer DC, Griffin JZ, Boren SA. Clinical trials of interactive computerized patient education: implications for family practice J Fam Pract 1997;45:25-33. [PubMed]

20. Ferris TG, Dougherty D, Blumenthal D, Perrin JM. A report card on quality improvement for children’s health care Pediatrics 2001;107:143-155. [PubMed]

21. Key Capabilities of an Electronic Health Record System. Washington, DC: Institute of Medicine: Committee on Data Standards for Patient Safety Board on Health Care Services; 2003.

22. Harris RP, Helfand M, Woolf SH, Lohr KN, Mulrow CD, Teutsch SM, et al. Current methods of the US Preventive Services Task Force: a review of the process Am J Prev Med 2001;20(3 Suppl):21-35. [PubMed]

23. Murff HJ, Kannry J. Physician satisfaction with two order entry systems J Am Med Inform Assoc 2001;8:499-509. [PMC free article] [PubMed]

24. Gomez EJ, Hernando ME, Garcia A, Del Pozo F, Cermeno J, Corcoy R, et al. Telemedicine as a tool for intensive management of diabetes: the DIABTel experience Comput Methods Programs Biomed 2002;69:163-177. [PubMed]

25. Modai I, Walevski A, Mark M, Rabinowitz J, Munitz H. Successful replacement of traditional psychiatric records with a multi-user computer system Med Care 1991;29:1057-1060. [PubMed]

26. Meigs JB, Cagliero E, Dubey A, Murphy-Sheehy P, Gildesgame C, Chueh H, et al. A controlled trial of web-based diabetes disease management: the MGH diabetes primary care improvement project Diabetes Care 2003;26:750-757. [PubMed]

27. Murray MD, Harris LE, Overhage JM, Zhou XH, Eckert GJ, Smith FE, et al. Failure of computerized treatment suggestions to improve health outcomes of outpatients with uncomplicated hypertension: results of a randomized controlled trial Pharmacother 2004;24:324-337.

28. Lin C-T WL, Moore L, Beaty BL, Ross SE. An Internet-based patient-provider communication system: randomized controlled trial J Med Internet Res 2005;7(4).

29. McCowan C, Neville RG, Ricketts IW, Warner FC, Hoskins G, Thomas GE. Lessons from a randomized controlled trial designed to evaluate computer decision support software to improve the management of asthma Med Inform Internet Med 2001;26:191-201. [PubMed]

30. Martins SB, Shahar Y, Galperin M, Kaizer H, Goren-Bar D, McNaughton D, et al. Evaluation of KNAVE-II: a tool for intelligent query and exploration of patient data Medinfo 2004;11(Pt 1):648-652.

31. Weiner M, Gress T, Thiemann DR, Jenckes M, Reel SL, Mandell SF, et al. Contrasting views of physicians and nurses about an inpatient computer-based provider order-entry system J Am Med Inform Assoc 1999;6:234-244. [PMC free article] [PubMed]

32. Eccles M, Hawthorne G, Whitty P, Steen N, Vanoli A, Grimshaw J, et al. A randomised controlled trial of a patient based Diabetes Recall and Management System: the DREAM trial: a study protocol [ISRCTN32042030] BMC Health Serv Res 2002;2:5. [PMC free article] [PubMed]

33. Lobach DF, Hammond WE. Computerized decision support based on a clinical practice guideline improves compliance with care standards Am J Med 1997;102:89-98. [PubMed]

34. Noel HC, Vogel DC, Erdos JJ, Cornwall D, Levin F. Home telehealth reduces healthcare costs Telemed J E Health 2004;10:170-183. [PubMed]

35. Demakis JG, Beauchamp C, Cull WL, Denwood R, Eisen SA, Lofgren R, et al. Improving residents’ compliance with standards of ambulatory care: results from the VA Cooperative Study on Computerized Reminders JAMA 2000;284:1411-1416. [PubMed]

36. Miller RH, West C, Brown TM, Sim I, Ganchoff C. The value of electronic health records in solo or small group practicesPhysicians’ EHR adoption is slowed by a reimbursement system that rewards the volume of services more than it does their quality. Health Aff (Millwood) 2005;24:1127-1137. [PubMed]

37. HL7 EHR group. Electronic health record functional descriptors. 2004 Available at: http://www.hl7.org/ehr/documents/Documents.asp. Accessed January 30, 2005.

38. Begg C, Cho M, Eastwood S, Horton R, Moher D, Olkin I, et al. Improving the quality of reporting of randomized controlled trialsThe CONSORT statement. JAMA 1996;276:637-639. [PubMed]

39. Stroup DF, Berlin JA, Morton SC, Olkin I, Williamson GD, Rennie D, et al. Meta-analysis Of Observational Studies in Epidemiology (MOOSE) group Meta-analysis of observational studies in epidemiology: a proposal for reporting JAMA 2000;283:2008-2012. [PubMed]

40. Leavitt M, Gallagher L. The EHR seal of approval: CCHIT introduces product certification to spur EHR adoption [Quiz 3–4] J Ahima 2006;77:26-30. [PubMed]

Articles from Journal of the American Medical Informatics Association : JAMIA are provided here courtesy of
American Medical Informatics Association