Although we specifically targeted primary care in our evaluation, the 3 health care systems participating in the study differed in many ways: Beth Israel Deaconess Medical Center (BIDMC) in Boston is a Harvard teaching hospital with approximately 113 hospital-based and community PCPs, serving approximately 90,000 patients, including over 50,000 patients registered to use PatientSite, a secure patient Internet portal based on a home-grown electronic medical record (EMR). Geisinger Health System (GHS) in rural northeastern Pennsylvania has approximately 175 PCPs serving about 300,000 primary care patients, with about 163,000 patients registered to use MyGeisinger, a patient portal based on the Epic EMR (Epic Systems Corp., Verona, WI). Harborview Medical Center (HMC) is a county-owned hospital in Seattle, WA managed by the University of Washington; HMC has a specific mission to care for the community's most vulnerable patients and is also the Disaster Control Hospital for Seattle and King County. HMC's two participating primary care clinics are staffed by approximately 65 faculty and fellows serving approximately 7,000 inner city and indigent patients, including about 2,000 patients with HIV. The internally-developed HMC patient portal, HealthReach, had been used only for research studies and not offered to the general HMC primary care patient population at the time we began this project. HealthReach was not integrated with HMC's EMR but could display information from ORCA, their clinical information system was based on the Cerner Millenium EMR platform (Cerner Corp., North Kansas City, MO) and other ancillary systems.
In preparation for our OpenNotes demonstration and evaluation, we convened a team of investigators from across the U.S., including three (SR, JR, and TD) who had expertise in patient access to medical records and, specifically, doctors' visit notes. Informed by the literature and our own scientific experience related to patient and doctor attitudes toward open access to medical records [1
], we developed a set of hypotheses about the potential impact of OpenNotes on primary care doctors and their patients, described in detail elsewhere [4
]. In designing our test of the intervention, we specified 3 key areas where we hypothesized the greatest impacts would occur: beliefs and attitudes about patient accessible notes, use of the patient internet portals, and patient-doctor communication. We developed a quasi-experimental design that would allow us to perform a number of comparisons, portrayed in Figure , comparing doctors and patients, participants and non-participants, before, during, and after the one-year experience with OpenNotes.
Conceptual model showing primary planned comparisons among doctors and patients in their attitudes, experiences, and portal use before and after the implementation of OpenNotes. Primary planned comparisons are indicated by red arrows.
We employed mixed methods, using qualitative data from semi-structured interviews and focus groups to inform survey instrument development and, at the end of the study, to provide a more in-depth assessment of patients' and doctors' experiences with OpenNotes. Using Creswell's typology, this approach is described as a sequential study involving qualitative data collection for instrument development, followed by quantitative data collection and subsequent qualitative data collection for triangulation purposes [5
]. To structure our evaluation, we applied the RE-AIM framework [6
] in order to capture as comprehensive a picture as possible of the impact of OpenNotes on patients, doctors, and health care systems. A scientific approach to assessing both the internal and external validity of real-world interventions, RE-AIM, is an acronym for: Reach, Effectiveness, Adoption, Implementation, and Maintenance. To address each of these elements of the intervention, we drew upon multiple data sources, including PCP and patient surveys, administrative and billing records, portal utilization data and electronic health records. The framework for our data collection and analysis is summarized in Table . Analyses directed to our primary and secondary study hypotheses will involve largely quantitative data from pre- and post-surveys of both doctors and patients, and also patient and doctor use of the Internet portals; subsequent qualitative analyses for secondary aims will utilize data obtained from patients' narrative survey information and patient and PCP focus groups.
Table 1 Summary of RE-AIM framework  for Evaluation of OpenNotes
We considered several study designs, but given the scope of the study and the involvement of 3 health systems, we identified a quasi-experimental, nonequivalent group, pretest-posttest approach as the one that would be most informative and efficient [7
]. It was not feasible to use a traditional experimental approach because we knew that many providers would hesitate to be part of the first wave of participants for an intervention that would potentially impose further burdens on their already stretched schedules. Randomized allocation on the patient or provider level would have limited the study to only those PCPs who agreed to have their visit notes posted online and their patients and would have resulted in a much smaller number of patients who had access to their notes during the intervention period. Maximizing participation by allowing all willing PCPs to offer OpenNotes to their patient panels provided for the first large-scale effort to assess the impact of sharing visit notes electronically with patients.
Survey instrument development
After developing our own list of expectations about the impact of OpenNotes [4
] and before the one-year study began, we turned to patients and doctors in a series of focus groups to further explore attitudes and expectations about the intervention. The PCP focus group guide addressed communication between patients and PCPs, expected impact on PCP practice and workload, and impact on patients. Similarly, the patient focus group guide dealt with current communication and means of understanding regarding plan of care, and experiences and expectations about reading doctor's visit notes. Following 2 PCP focus groups (6 PCPs at GHS, and 8 at BIDMC) and 5 patient focus groups (11 patients at BIDMC, and 30 patients at HMC), three investigators (TD, EV, JW) reviewed transcripts of the recorded group sessions and identified common themes.
We developed survey questions based on the themes of PCP expectations about the impact on their practice and PCP and patient attitudes about benefits and risks of OpenNotes. Other than the set of questions specifically for PCPs about potential practice effects of OpenNotes, the PCP and patient surveys were designed in parallel to address similar questions about the anticipated benefits and risks of OpenNotes. In planning ahead for the one-year follow-up survey, we decided we would also include a set of questions about attitudes toward possible future modifications to OpenNotes that might further engage patients and their families in their care. We performed a series of tests of the pre-intervention surveys, first using paper questionnaires to ascertain the intent and clarity of the questions, then incorporating changes based on feedback from PCPs and patients, and we conducted additional testing in the online format to assess content and potential technical issues.
Concordant with the study's aims regarding the impact on attitudes, we developed 3 items to directly query patient respondents about whether or not they: a) thought OpenNotes was a good idea, b) would like to read their notes, and c) would be likely to share their notes with others. To explore these attitudes further, we developed three item sets, Perceptions of Benefits and Perceptions of Risks of OpenNotes on Patients, comprising 7 and 4 items, respectively, and a set of 4 items about Risks to PCP practices.
The survey questions regarding perceived patient benefits were as follows:
1. I would better understand my health and medical conditions.
2. I would better remember the plan for my care.
3. I would take better care of myself.
4. I would be more likely to take my medications as prescribed.
5. I would feel more in control of my health care.
6. I would be better prepared for visits.
7. Patients will trust me more as their doctor. (PCP survey only)
The questions addressing perceived patient risks were as follows:
1. I would worry more
2. I would be concerned about my privacy.
3. The notes would be more confusing than helpful.
4. It could make my doctor's job more difficult. (Patient survey only)
The PCP survey questions about the impact on their practice were as follows:
1. Patients will disagree with what I write in their visit notes
2. Patients will request changes to the content of visit notes
3. Patients will find significant errors in the notes
4. Patients will contact me or my practice with questions about their notes
The patient benefits and risk items were designed as parallel items on both patient and PCP surveys except where otherwise indicated. Response options were as follows: agree, somewhat agree, somewhat disagree, disagree; we added a "don't know" response option for patients. To compare PCP perceptions about anticipated and experienced impact of OpenNotes on their practices in the pre- and post-intervention surveys, respectively, we designed a set of questions addressing PCP documentation (content and process), communications with patients, potential impact on medical liability, and effects on patient care. In the baseline survey, we also included items about doctor and practice characteristics. For the post-intervention survey, to address our primary hypothesis, doctors and their patients in the intervention were specifically asked whether or not they would like OpenNotes to continue.
To measure patients' perceptions about patient-doctor communication, we used 2 standardized instruments. The Ambulatory Care Experiences Survey (ACES) was developed to survey patients about their experiences with health care providers, as a complement to existing health plan level strategies for assessing health care quality [8
]. We used the 6-item ACES subscale assessing the quality of patient-provider interactions. Patients' self-confidence for communicating with doctors was measured using the validated Perceived Efficacy in Patient-Physician Interactions (PEPPI) scale [9
]. This instrument was based on the previous work of Bandura regarding the assessment of self-efficacy, a motivator for health behavior [10
]. The survey also included a set of questions regarding Internet use and sociodemographic and health characteristics (self-rated general health, education, race/ethnicity, and employment status). Lastly, in the follow-up survey, we included several open-ended questions, giving respondents opportunities to describe their experiences with OpenNotes and, if they did not read their notes, their reasons for not doing so.
For both patients and doctors, surveys were conducted online using SurveyGizmo (v2.0 & v3.0. Widgix, LLC dba SurveyGizmo, Boulder, Colorado). We required a response for all items except for demographic items, free text questions and questions within skip patterns. Based on our pilot tests, the PCP surveys were designed to be completed in less than 10 minutes, and the patient surveys in less than 20 minutes. The instruments are available from the authors upon request. Results regarding PCP and patients attitudes from the baseline survey were published elsewhere [12
Human subjects protections
Before the start of the study, all study protocols, data collection procedures and the intervention methods were approved by the institutional review boards of the 3 participating institutions, the BIDMC, the GHS and the University of Washington.
Recruitment of primary care doctors
We began the study with the recruitment and enrollment of primary care doctors who would be willing to make their visit notes available to their patients through the Internet portal. OpenNotes doctors signed an informed consent prior to completing the baseline study surveys that preceded the OpenNotes intervention.
Invitations to join the intervention went to all primary care doctors (excluding trainees) practicing in BIDMC and affiliated primary care practices who were using both electronic health records (EHR) and the patient portal, PatientSite. PCPs were informed about the study through email announcements, presentations by study investigators, and informal discussions. We obtained the support of key clinical leaders within the institution who announced their endorsements on websites and through email communications with the PCPs. Similar recruitment strategies were employed at HMC and GHS, but because the GHS primary care practices are distributed geographically across a large area of rural central and northeastern Pennsylvania, the bulk of the recruitment contacts at GHS took place via email.
PCPs from the BIDMC and GHS primary care practices who chose not to offer any of their patients access to their visit notes online, referred to here as "non-intervention PCPs," were included in the study as the PCP comparison group. Their patients who were registered portal users comprised the comparison group referred to as "non-intervention patients." As no patient portal was available previously to patients at HMC, we did not have PCP or patient comparison groups at HMC.
Recruitment of patients
After PCPs agreed to join the study, we sent them a list of their eligible patients and allowed them to exclude individual patients from the intervention with no requirement that they explain the exclusions. At BIDMC and GHS, patient eligibility was defined as having been registered on the patient Internet portal for at least 1 year before the start of the study. Following PCPs' exclusions, all patients registered on the patient Internet portals at BIDMC and GHS were included automatically as OpenNotes patients for the study, regardless of their portal use history or participation in study surveys. Prior to the start of the intervention, OpenNotes patients were sent a message through the portal informing them that their doctor was participating in OpenNotes, with an explanation of the study and a link to the OpenNotes website http://www.myopennotes.org
that included detailed information about the study and the research team. The message included an invitation and a Web link to the pre-intervention survey.
We included two comparison groups of patients in our evaluation from BIDMC and GHS. The first group comprised patients of non-intervention PCPs who were registered portal users, referred to as "non-intervention patients." The second group included all patients of OpenNotes PCPs who were not registered portal users. This latter group was not approached to participate in surveys; their health care utilization and administrative data will be used in the evaluation. As noted above, there was no comparison group of HMC patients.
Prior to the OpenNotes Study, patients at HMC did not have access to the patient internet portal which was developed and implemented previously for research studies within the University of Washington system [2
]. Therefore, the research team actively recruited HMC patients, many of whom were indigent or homeless but used the Internet through community venues. Eligibility required that a patient could communicate in English and had a current e-mail address. Some patients obtained new e-mail addresses in order to join the study. During the 4-month recruitment period, HMC staff used a multi-faceted approach that included letters of invitation mailed to all eligible patients with known addresses, active recruitment of a subset of eligible patients by telephone and in person recruitment of all eligible patients who came into the primary care clinics for a regularly scheduled clinic visit. Eligible patients who were approached about the study in the clinics were given an information flyer about the study and were invited to enroll after their clinic appointment or to join a bi-weekly group enrollment session. To enroll, HMC patients provided their email addresses, e-signed the study consent form, and then completed the online pre-intervention survey.
OpenNotes: the intervention
Prior to OpenNotes, patients registered on the multifunctional GHS or BIDMC patient portals already had online access to their problem lists, medication records, and laboratory and radiology reports. In addition, the portals provided appointment scheduling and secure e-mail messaging among patients and their doctors and health care teams. As participants in the OpenNotes project, the PCP's at BIDMC or GHS could selectively exclude their patients, as described above, but they would be allowing all of their other eligible patients access to their visit notes if they were registered portal users at the time the study began in Spring, 2010.
At BIDMC and GHS, the OpenNotes intervention consisted of a simple addition to the existing menu of accessible records. Following a scheduled office visit, PCPs would record and sign their visit notes, at which point their patient would receive an automatically-generated e-mail invitation to read the visit note. Two weeks prior to a next scheduled visit to their PCP, patients also received an e-mail message suggesting that they review prior "OpenNotes" before coming to see their doctor. At HMC, patients did not have prior online access to their health records; therefore the intervention included new access not only to the visit notes, but also to other sections of their EHR including laboratory and radiology reports. Unlike the portals at BIDMC and GHS, the HMC portal was not interactive; patients could only view their records but could not schedule appointments or send messages to the healthcare team.
Surveys of doctors and patients
We are using several data sources in the multi-faceted evaluation of OpenNotes, but the principal sources of information used to assess the study's primary outcomes are PCP and patient surveys conducted prior to, and then immediately following the one-year OpenNotes intervention. All surveys were conducted online using the web-based program, SurveyGizmo. Both intervention and non-intervention PCP's and their patients from the BIDMC and GHS were provided links to the surveys via survey invitation messages sent through the portals. Each respondent was given a unique study ID number assigned by the information technology (IT) staff of each health system. Survey data were downloaded from the password-protected SurveyGizmo databases into password protected files on the computer network at the study coordinating center at the BIDMC. At HMC, patients completed their surveys at the time of enrollment which took place in their HMC clinic.
In addition to the surveys, IT staff of each site assembled electronic data from multiple existing sources at the 3 sites, including the portal tracking systems, administrative databases and health care utilization databases. Using the unique study ID assigned to both intervention and non-intervention PCP's and patients at each site, we collected demographic and health characteristics, information about portal use, and health care utilization data for the year prior to the start of the intervention and for the one-year during which the visit notes were made available to intervention patients.
We used administrative data sources to obtain demographic information (age and sex) about PCPs and patients at each of the 3 sites.
Portal use data
We assessed portal use at each site through their electronic tracking systems. At BIDMC, HMC and GHS, the systems recorded the time and date of each patient portal login, each click into each section of the portal, including the OpenNotes section, and, with the exception of HMC, each message sent or received from within the portal. During the year of the intervention, we determined the total number of times patients entered the new OpenNotes sections of their portal records.
Health care utilization
Patient utilization measures, determined from billing records, included counts of visits to PCP's and their practices, hospitalizations, and emergency room visits.