PMCCPMCCPMCC

Search tips
Search criteria 

Advanced

 
Logo of meoMedical Education Online
 
Med Educ Online. 2010; 15: 10.3402/meo.v15i0.5350.
Published online 2010 October 19. doi:  10.3402/meo.v15i0.5350
PMCID: PMC2958708

Development and evaluation of a Health Record Online Submission Tool (HOST)

Abstract

Introduction

Health records (HRs) are crucial to quality patient care. The Michigan State University College of Human Medicine begins teaching health record (HR) writing during the second-year clinical skills courses. Prior to this project, we used a cumbersome paper system to allow graduate assistants to grade and give feedback on students' HRs. This study discusses the development and evaluates the effectiveness of the new Health Record Online Submission Tool (HOST).

Methods

We developed an electronic submission system with the goals of decreasing the logistical demands of the paper-based system; improving the effectiveness, consistency, and oversight of HR instruction and evaluation; expanding the number of students who could serve as written record graduate assistants (WRGAs); and to begin preparing students for the use of electronic health records (EHRs). We developed the initial web-based system in 2003 and upgraded it to its present form, HOST, in 2007. We evaluated the system using course evaluations, surveys of WRGAs and clinical students, and queries of course faculty and staff.

Results

Course evaluation by 1,106 students during years 2001 through 2008 revealed that the students' self-assessment of ability to write HRs improved briefly with the introduction of HOST but then returned to baseline. The initial change to electronic submission was well received, though with continued use its rating dropped. A survey of 65 (response rate 61.3%) clinical students indicated that HOST did not completely prepare them for EHRs. The WRGAs (n=14; response rate 58%) found the system easy to use to give feedback to students. Faculty (n=3) and staff (n=2) found that it saved time and made the review of students' HRs and WRGAs grading simpler. Student perception of grading consistency did not improve.

Conclusions

HOST is the first published online method of in-depth HR training for preclinical students using information gathered in clinical encounters. With it we were able to maintain effective instruction, streamline course management, and significantly decrease staff time. HOST did not improve student perception of grading consistency and did not prepare students for specific EHR use. Within the context of our class size expansion and our community-based educational program, HOST bridges geography and can support future improvements in HR instruction and faculty development. Medical educators at other institutions could use a similar system to accomplish these goals.

Keywords: medical education, competencies, health records, assessment, course management, evaluation, computer-aided instruction, electronic health record

Health records (HRs) are a critical ingredient in quality patient care. Not only do they serve as a means of communication between members of the health care team, they also provide important information for billing, epidemiological and health systems research, peer review, quality improvement, and malpractice litigation (1). Evaluating students' HRs provides an important vehicle for teaching and evaluating critical thinking skills (2). Reading the contents of a health record (HR) enables the evaluator to assess the correspondence between the ‘thinking’ and ‘doing’ of a doctor (3) or of a medical student. According to the Association of American Medical Colleges (AAMC) Recommendations for Clinical Skills Curricula for Undergraduate Medical Education (4), medical students entering their clinical clerkships should be able to record clinical information accurately in standard format and also use an electronic medical record at an advanced beginner level. Writing satisfactory progress notes is now required for licensure, as evidenced by the patient note portion of the USMLE Step 2 Clinical Skills examination (5). For these reasons, learning to compose high quality HRs on an electronic system is an important part of students' medical education.

Weed's introduction of the problem-oriented medical record (6) has led to the standardization of the HR format. A survey of US medical schools found that the most important medical writing skills for students to master are the history and physical examination, the progress note, and the discharge summary (7). However, the survey also revealed that medical schools were not providing extensive teaching of these skills. There continues to be little in the medical education literature about standardized teaching and assessment methods for trainee HRs. Most of the literature that does exist relates to teaching students to write HRs in the clinical years, and the methods used often require a great deal of attending physicians’ time (8). There are relatively few examples of methods for teaching preclinical students to write HRs described in the literature. Those that have been described include teaching students the subjective and objective portions related to cases in basic science coursework (9), or asking students to convert a source-oriented HR to a problem-oriented HR (10), rather than asking students to write HRs based on a clinical encounter. Systems for recording student HRs that result from standardized patient encounters exist, such as WebSP (Lionis software). These systems serve as reservoirs for trainee-created HRs but do not by design facilitate teaching or iterative practice with feedback.

An additional challenge to written record pedagogy is the increasing use of electronic health records (EHRs), which are likely the future conveyance of health information. As far back as the mid-1990s, medical educators developed a web-based, EHR-like system for teaching preclinical students to compose written records (11). This particular system included a prompting vehicle to help students remember all the pertinent questions and physical examination steps, required minimal typing, and produced a comprehensive history and physical examination document. Additionally, students are increasingly likely to encounter and benefit from using EHRs during clinical clerkships. A recent survey of third-year medical students found that using an EHR led to better student performance on several parameters, including asking historical questions and ordering clinical preventive services. These students also reported receiving more feedback from faculty on their EHR notes compared to paper chart notes (12). Other medical educators have successfully used a web-based, EHR-like system to evaluate progress note writing in the clinical years (13).

The Michigan State University College of Human Medicine is a community-based medical school. Prior to 2008, there were 106 students per class and all students completed their preclinical years at our East Lansing campus. In 2008, the college began expanding to 200 students per class, split between two preclinical campuses. At the end of the second year of medical school, students move to one of eight clinical campuses to complete the last 2 years of training.

We teach students the foundations of HR writing in our fall and spring semester second-year Clinical Skills courses. The curriculum includes lectures, a written tutorial, and the writing of 17 graded HRs all generated from encounters with simulated or real patients. These records include several complete ‘history and physicals,’ numerous progress notes, and other portions of the written record, such as the Master Problem List. Third- and fourth-year medical students are paid to serve as written record graduate assistants (WRGAs). These WRGAs receive an initial training session as well as ongoing oversight and feedback on their grading. They grade students' HR assignments based on an explicit competency rubric that consists of a checklist. The WRGAs ask students to rewrite unacceptable HRs until they are acceptable and give them written comments to help them improve. We require students to complete all portions of all assignments to an acceptable standard in order to pass the course, but allow several attempts with feedback. Each WRGA grades the HRs of approximately 10–15 students for one semester and then 10–15 different students for the second semester.

Prior to the project described, we used a paper-based system for HR writing, evaluation, and feedback. Students brought two copies of completed paper records to the course secretary. The secretary logged these in, bundled them into WRGA groups, and took them to a locker in the basement of an administrative building. The WRGAs were chosen by necessity from only the Lansing campus, where the preclinical students were also located. These WRGAs periodically picked up the bundles, graded them, provided handwritten feedback, and gave due dates for the rewritten portions (‘rewrites’). The course secretary then notified second-year students to come and pick up their written records, with grading and feedback, and the cycle repeated. This system incorporated several pedagogical principles including the use of rubrics to both teach and evaluate the HRs of students, the provision of iterative feedback to students as they crafted their HRs, and a competency-based grading system where each record was rewritten until all portions adhered to the rubric.

However, this system had a number of limitations. It required a significant amount of secretarial time, repetitive student and WRGA travel and was difficult for course staff and faculty to monitor. It also greatly limited the pool of third- and fourth-year students eligible to be WRGAs, as only those who were doing their clinical work in the Lansing community could serve. It was very difficult for the course director to oversee either student or WRGA performance, and it was challenging for course staff to monitor timeliness of submissions.

We sought to maintain our competency-based approach to the teaching and grading of HRs, efficiently expose our students to the rubric underlying the creation of their HRs, and provide the same amount of practice and feedback. At the same time we sought to decrease the time, travel, and secretarial demands of the paper-based system. Additional goals were to improve the effectiveness, consistency, and oversight of HR instruction and evaluation by course staff and faculty; to expand the number of students who could serve as WRGAs; and to begin to familiarize students with the use of an electronic recording system for written records. Toward these ends, we created an electronic submission system that was later improved to become HOST. This paper details the development of our online submission tool, describes how the tool was implemented, and provides multisource evaluation data about its impact.

Methods

The conversion to the initial web-based system

In 2003, we converted our ‘Written Record Protocol (WRP) Tutorial’ from paper format to an online, menu-driven application that students accessed via a unique login. In order to maintain patient confidentiality, written records were stored on a dedicated server. When students logged in, the system presented them with a menu of written record assignments. After choosing an assignment, the student would see each section of the assignment as a blank text box (Fig. 1). The system differed significantly from actual EHRs, as it did not include drop-down menus or other cues so that students would be required to develop skills in de novo HR creation. Grading rubrics for each section of the relevant written record assignment appeared below each text box, reinforcing the guidelines in the Written Record Protocol. Students could access links to sections of the WRP from the submission application if desired. The WRGAs could access the system via the Internet in the same way to grade assignments.

Fig. 1
Screen showing portion of completed HR, ready for submission.

The system included an assignment development application so that when a new HR assignment was needed, the course director could access a template of items to be included (e.g., History of Present Illness, Social History, Assessment) as well as the grading criteria for each element (Fig. 2). It also included a structured Master Problem List, allowing students to list the problem number, onset date, current status (active, resolved, etc.), and resolved date, if applicable, for each patient problem identified (Fig. 1). The web-based interface allowed students to update and save these sections at any time, allowing them to save work in progress and return to it later. When the entire assignment was completed, the student clicked a button to submit the HR.

Fig. 2
Screen showing addition of grading criteria (‘attributes’) for a section of the health record.

This initial web-based system provided a number of enhancements over the paper-based one. The grading rubric for each section appeared at the ‘point of need.’ Legibility of both the student HR and the WRGA feedback was no longer a problem. The system presented the same requirement for students to create their own correct HR components, as in a paper HR. The use of the Internet allowed us to recruit from a broader pool of third- and fourth-year students to serve as WRGAs. Students from any of the clinical campuses across the state could easily access the system to grade assignments. It also eliminated the time that WRGAs spent picking up and delivering the written HRs. Secretarial time requirements for this aspect of the curriculum decreased from approximately 0.25 full-time equivalent (FTE) with the pen and paper system to less than 1 hour per week, resulting in a major improvement in course management. The use of paper was eliminated. This ‘green’ component was less of an explicit goal in 2003, but was as helpful then as it is in 2010.

Upgrade to HOST

In 2007 the Clinical Skills course directors decided to embark upon a major revision of the 2003 electronic submission system. The updated system was named ‘HOST’: a Health Record Online Submission Tool. As with the previous version, all users access the system through a secure website using a unique user ID and password. The basic mechanics of the system are the same as the first system. Grading rubrics are made explicit and made available for each section via a weblink.

This upgrade provided several improvements over the previous electronic system. First, the new system included an automated notification system. When a student's assignment was completed, the system notified the WRGA via a computer-generated email. Similarly, when the WRGA completed grading the HR, the system generated an email to the student if a rewrite was required and indicated the due date for the rewrite.

Second, the new system improved tracking and organization of submissions. Course faculty or administrators could select a list of submitted HRs, late HRs, all the HRs assigned to a particular WRGA, or those submitted by a particular student (Fig. 3). This feature allowed the administrator to easily track and respond to late submissions. It also allowed faculty to easily see if a grader was requesting more or fewer rewrites than typical, and review the HRs submitted by any student or graded by any WRGA. The WRGAs could notify faculty of students that were having problems and faculty could view those students' HRs and give individualized feedback.

Fig. 3
Screen showing menu of options for course faculty and administrative staff to manage student assignments and review submissions or grading.

Evaluation

The Institutional Review Board of Michigan State University approved this research. We evaluated the impact of the initial electronic submission system as well as the improved HOST by combining a variety of data sources. We reviewed course evaluations for the years 2000 through 2008. Students were required to complete these evaluations anonymously at the end of each course through an online system. These evaluations consisted of multiple statements that students rated using a 5-point Likert scale with the following choices and point assignments: Not Applicable, Strongly Disagree (1 point), Disagree (2 points), Neutral (3 points), Agree (4 points), and Strongly Agree (5 points). The evaluation system also invited anonymous student comments. We reviewed student ratings of the following statements: ‘I feel I can write an acceptable History and Physical and Subjective/Objective/Assessment/Plan (SOAP) note,’ ‘I feel that the electronic written record submission system was valuable,’ as well as any comments that related to either HR submission system.

We used Survey Monkey™ to anonymously survey clinical students in their third year of medical school who had used HOST during their second year with the question: ‘How well did the electronic health record submission tool (HOST) used in Block II prepare you to use an EHR?’ and the following choices: ‘Not at all prepared, ‘A little prepared,’ ‘Somewhat prepared,’ ‘Fairly prepared,’ ‘Very Prepared,’ and ‘Completely Prepared.’ We also gave them the opportunity to submit anonymous comments.

We used Survey Monkey™ to anonymously survey current WRGAs using these four statements: (1) Once I got used to it, HOST was easy to use. (2) I received adequate training to use HOST. (3) I enjoyed using HOST. (4) HOST made it easy to give feedback to students. The WRGAs rated each statement using this 5-point Likert scale: Strongly Disagree (1 point), Disagree (2 points), Unsure (3 points), Agree (4 points), and Strongly Agree (5 points). A final question solicited ‘Any other comments you have regarding the use of HOST.’

Lastly, we contacted current and previous course directors and course coordinators (administrative staff) by email to solicit their opinions using the following query: ‘Can you make any comments on the following:

  1. Positive aspects of the paper system prior to 2003.
  2. Negative aspects of the paper system prior to 2003.
  3. Positive aspects of the electronic system since 2003/any thoughts on changes in the system since then.
  4. Negative aspects of the electronic system since 2003/any thoughts on changes in the system since then.’

Results

Student course evaluations

Students completed course evaluations from 2001 through 2008. Student ratings of statements about the written record teaching and assignments during their second-year Clinical Skills courses were in the 3.3–4.1 range (out of five possible). There was a brief improvement in ratings of the students' self-evaluation of their ability to write an acceptable history and physical (H&P) and SOAP note following the introduction of the electronic system, with ratings of 4.1 and 4.2 in the fall of 2004 and spring of 2005, respectively, but this was not maintained, with evaluations remaining in the 3.8–4.1 range. The most favorable ratings of the electronic system occurred from 2003 through 2005 (Table 1), the years right after the change from the paper to the electronic system, when the Likert rating for ‘I felt that the electronic submission system for written records was valuable’ ranged from 4.1–4.2. Some of the student comments about the functioning of the application included:

  1. And what can I say about the written record system. It still needs improvement. It is very frustrating …. Good idea but more of the glitches should have been worked out ….
  2. After the kinks were worked out, the system worked beautifully, and I cannot imagine what it was like before the electronic submission system was available!
  3. I thought the online written records, once ready, was rather convenient. Additionally, it was very helpful to get detailed feedback upon submission from our GAs.
  4. The online submission system is heaven-sent and much easier than writing by hand.

In 2005 and forward, Likert scale responses continued to average in the high threes, but comments tended to reference problems with speed:

  1. The written records submission was far too slow.
  2. The electronic records submission website is slow and difficult to use.

Students also commented on grading consistency:

  1. I felt the WRGAs were really inconsistent in the way they graded write-ups … some WRGAs were really adamant about including unnecessary details … whereas other WRGAs were too lenient on their students.

Table 1
Results of student course evaluations relating to health record training and HOST

Survey of clinical students

Out of approximately 106 third-year students, 65 (response rate = 61.3%) answered the question ‘How well did the electronic health record submission tool (HOST) used in Block 2 prepare you to use an EHR?’ 31% of respondents chose ‘not at all prepared,’ 32% chose ‘a little prepared,’ 23% chose ‘somewhat prepared’, and 14% chose ‘fairly prepared.’ No student chose ‘very prepared’ or ‘completely prepared.’

Survey of WRGAs

Out of 24 WRGAs surveyed, 14 (58% response rate) responded to four statements about using HOST on a 5-point Likert scale. They rated ‘Once I got used to it, HOST was easy to use’ − 4.2, ‘HOST made it easy to give feedback to students' − 4.1, ‘I received adequate training to use HOST’ − 3.9, and ‘I enjoyed using HOST’ − 3.6.

Query of faculty

All three course directors for the period from 2003 through 2008 responded to the email query. The course director during the initial transition to a web-based system commented that the course management and WRGA feedback improvements:

  1. Were very important, having endured an incredibly clumsy and limiting paper submission system for several years.

The next course director was most impressed with the improvements in teaching, training of WRGAs, and course evaluation that occurred in the transition to the HOST system:

  1. HOST allowed faculty to monitor health record submissions efficiently, which enabled important improvements in teaching.
  2. Previously such monitoring was too difficult to make it a routine faculty task, and until it became a routine faculty task, we really didn't know just how productive a task it was.

The current course director believes the system:

  1. Provides the students critical practice with specific feedback, and allows tracking of the student and WRGA performance over the course of two semesters.

Query of staff

The clinical skills administrator and secretary responded to the email query. Their comments included:

  1. With the students submitting their papers one by one to our office, there were lots of interruptions. We had to sort submissions by WRGA, affix evaluation forms, etc.
  2. Papers were due by a certain time, and students would argue that our clocks were different.
  3. We had to make arrangements for WRGAs to pick up papers after hours and on weekends. We needed to keep checking the lockers to see if the GAs had picked things up or dropped them off.
  4. We could only hire WRGAs from the Lansing community.
  5. Positive aspects of the electronic system included using WRGAs from all campuses, easier faculty review of WRGA comments and student submissions, and easy tracking.

Staff also made comments that the main problems with the system were ‘technical glitches’ such as student loss of data and system ‘crashes.’

Discussion

Our HOST project accomplished many of our pedagogical and course management goals but left us disappointed with others. We maintained our competency-based teaching and grading system and preserved the amount of practice and feedback given to students.

We successfully decreased the time, travel, and secretarial demands of the curriculum compared to when we were using a paper-based system. We expanded the number of students who could serve as WRGAs. We were able to maintain the effectiveness of HR instruction, evidenced by students' self-evaluation of HR writing skills. We significantly improved the oversight of HR instruction and evaluation, as the electronic system made it much simpler to monitor the performance of students and WRGAs.

We did not attempt to specifically measure the consistency of grading by various WRGAs using either the paper or electronic systems. Despite the increased pool of students to serve as WRGAs, clear grading criteria, and WRGA training and oversight, students still complained about inconsistency in grading. We suspect this is primarily due to two factors. First, we have not yet arrived at an entirely satisfactory method of training WRGAs. Second, faculty resources limit our ability to give ongoing oversight of grading. Although HRs with grading and comments are easily available for review and feedback, faculty found it difficult to make the time to review them, often only responding to student complaints.

The use of HOST did not result in students feeling well prepared to use an EHR. We designed HOST using very different principles than the EHRs employed in clinical settings. Like others, we believe that novice learners are best served when they use internalized rubrics to create de novo HRs (14, 15). The EHRs make use of drop-down menus and check boxes, while HOST used exclusively free text. Clinicians use EHRs to review patient information, a process that is often more complex than recording it, and that is not a part of our HR teaching at the preclinical student level. Consequently, HOST did not provide any opportunity to practice reviewing a patient's past history or any other part of a patient's database. In addition, many of our students still do not have the opportunity to use an EHR regularly in their clinical clerkships. It will be challenging to train students in the use of EHRs to an advanced beginner level prior to entering their clinical clerkships, as recommended by the AAMC (4). There is tension between the needs of the novice HRs writer and the constraints of most EHRs, which were not designed for teaching.

There are some limitations of our project. Resources and the long time over which we developed HOST limited our evaluation methods. The multisource evaluations were subjective, not objective, assessments of students' skills. We did not perform qualitative analysis of the subjective data.

Medical educators continue to struggle to address inadequacies in trainee HRs across the continuum of preclinical, clinical, and residency education. Some of the reasons include differing expectations and inadequate faculty training and time (16). Educators at any medical school could use electronic tools like HOST to provide a consistent structure for HR instruction and evaluation rubrics for preclinical students. This structure solves geographic challenges and could be expanded to use with clinical students and residents. HOST could support faculty development across this continuum because of its standardized format and evaluation rubrics. HOST, and other methods of online instruction and evaluation, are especially useful to community-based medical schools like our own, with students spread out over a wide geographic area.

There are several opportunities for further curriculum development and evaluation. What is the standard for a clinical clerk's HR? Do students completing our preclinical curriculum, or any preclinical curriculum, meet that standard? Would a simpler grading system, or more practice grading with feedback, result in more consistent grading of HRs? What is the best way to prepare preclinical students for use of the EHR? We are currently pursuing the answers to some of these questions.

Conclusions

HOST is the first published online method of in-depth HR training for preclinical students using information gathered in clinical encounters. With it, we were able to maintain effective instruction in HR writing, decrease logistical demands of the curriculum, and improve oversight of HR writing and evaluation. Student perception of grading consistency did not change. The differing principles upon which HOST and actual EHRs are based resulted in a failure to facilitate students feeling prepared for the use of an EHR.

This system provides advantages in our educational program, supporting the key pedagogical principles underlying instruction, enhancing administrative efficiency, and is scalable to the needs of HR instruction across the curriculum.

Conflict of interest and funding

The authors have not received any funding or benefits from industry to conduct this study.

References

1. Szauter KM, Ainsworth MA, Holden MD, Mercado AC. Do students do what they write and write what they do? The match between the patient encounter and patient note. Acad Med. 2006;81:S44–7. [PubMed]
2. Condon W, Kelly-Riley D. Assessing and teaching what we value: the relationship between college-level writing and critical thinking abilities. Assess Writing. 2004;9:56–75.
3. Malavasi A. Logical structure of the problem oriented medical record. Ann Ital Med Int. 2002;17:130–1. [PubMed]
4. American Association of Medical Colleges. Washington, DC: American Association of Medical Colleges; 2008. Recommendations for clinical skills curricula for undergraduate medical education; p. 30. Appendix 8.
5. USMLE Orientation Materials. 2009. Available from: http://www.usmle.org/orientation/2009/patientnote/pnmain/htm [cited 6 March 2009]
6. Weed LL. Medical records that guide and teach. New Eng J Med. 1968;278:593–600. [PubMed]
7. Yanoff KL, Burg FD. Types of medical writing and teaching of writing in U.S. medical schools. J Med Educ. 1988;63:30–7. [PubMed]
8. Miller MD, Albritton A, Rindt K. Teaching medical documentation according to the HCFA Guidelines. Acad Med. 1999;74:596. [PubMed]
9. Kibble J, Hansen PA, Nelson L. Use of modified SOAP notes and peer-led small-group discussion in a medical physiology course: addressing the hidden curriculum. Adv Physiol Educ. 2006;30:230–6. [PubMed]
10. Bashook PG, Sandflow LJ, Hammett WH. Teaching problem-oriented recording: a systematic approach. J Med Educ. 1975;50:959–64. [PubMed]
11. Naeymi-Rad F, Trace D, Moidu K, Carmony L, Booden T. Education review: applied medical informatics – informatics in medical education. Top Health Inform Manage. 1994;14:44–50. [PubMed]
12. Rouf E, Chumley HS, Dobbie AE. Electronic health records in outpatient clinics: perspectives of third year medical students. BMC Med Educ. 2008;8:13. [PMC free article] [PubMed]
13. McCarty T, Parkes MV, Anderson TT, Mines J, Skipper BJ, Grebosky J. Improved patient notes from medical students during web-based teaching using faculty-calibrated peer review and self-assessment. Acad Med. 2005;80:S67–70. [PubMed]
14. Peled J, Sagher O, Morrow J, Dobbie A. Do electronic health records help or hinder medical education? PLoS Med. 2009:6. [PMC free article] [PubMed]
15. Hartzband P, Groopman J. Off the record – avoiding the pitfalls of going electronic. New Eng J Med. 2008;358:1656–8. [PubMed]
16. Wagner D, Radford J, Mavis B. Trainee written records: what's right? what's wrong? it depends! Med Teach. 2010;32:184. [PubMed]

Articles from Medical Education Online are provided here courtesy of Taylor & Francis