Pre-study Internet use survey
The initial survey was given to 63 physicians and completed by 60; five faculty attendings and all 55 residents rotating in the internal medicine program during July and August 2005, the first two months of the training year. They were divided in two groups based on the specialty level of training and practice, twenty-four were new first year residents (Interns group) and 36 were senior residents or faculty attendings (PostPGY1 group). Although not all the questions were answered, the entire group of participants (100%) reported habitual use of the Internet, accessing the Web usually from desktop computers [Table ]. Access from the hospital was more frequent in the PostPGY1 group (55% of the times) while it was more frequent from home in the Interns group (68%). Fifty-seven of the 60 physicians interviewed (95%) used the Internet on a daily basis [Table ]. Thirty-one doctors (52%) reported a daily usage of less than five times per day whereas five physicians (8%) used the Internet more than 10 times a day at that time. The reported time spent on the Internet was between one to two hours a day for the majority of the group (64%) [Table ]. Only three of the physicians in the Interns group were not using the Internet on a daily basis – their reported usage was three to four times per week.
Pre-study survey. Proportion of the Internet access by location.
Pre-study survey. Daily Internet usage.
Pre-study survey. Approximate number of hours on the Internet.
We found differences between the two groups in the usage of the Internet [Table ]. The Interns group spent more time on personal information activities, including e-mail (43%) whereas the PostPGY1 group spent more time on patient clinical information (30%). The second most common use by both groups was searching for general medical or scientific information. The groups used fairly similar Websites to search for general scientific information but their pattern of use was slightly different. The five sites most commonly mentioned were UpToDate, Google, eMedicine, PubMed and MDConsult [Table ]. If the questions related to specific issues concerning patient management, the preferred Websites were UpToDate, PubMed, MDConsult, Google and New England Journal of Medicine (NEJM) [Table ]. When asked about the resources they accessed for evidence-based medicine, both groups mentioned similar choices, the four most common were: UpToDate, NEJM, PubMed and MDConsult [Table ] When asked about their interest in getting more education and training in the use of Internet for medical applications, 83% of the PostPGY1 and 95% of the Interns were interested on medical knowledge resources, tools for clinical practice and EBM resources training [Table ].
Pre-study survey. Reported proportion of the tasks done on the Internet.
Pre-study survey. Websites used to search for general medical and scientific information.
Pre-study survey. Websites accessed for specific questions on patient management.
Pre-study survey. Most common "EBM" resources cited.
Pre-study survey. Areas of interest on medical Internet training.
Pre-study handhelds use survey
Not all questions were answered by the group. Twenty of the 60 physicians interviewed (33%) owned a PDA at the time of the survey, sixteen in the PostPGY1 group and four in the Interns group. Ten of them had PDAs for one to three years and nine for less than a year [Table ]. Seven of the handhelds were Palm devices and the others were Pocket-PC, including Dell (6), Sony (3), and HP (1), or were unidentified. Although 13 of the PDAs were wireless-enabled they were not being used for Internet access to obtain real-time medical information. None of the physicians had previous formal training on the use of these devices and 75% of them reported self learning from manuals or learned from peers. Fifteen physicians who had PDAs (75%) reported using it daily [Table ]. They used the handhelds on average seven times a day (range: 1–20). Their main usage was for patient care resources, including drug information programs (pharmacopoeias), medical references and clinical tools [Table ]. Most of the 40 physicians who did not own a handheld at the time of the survey were planning to buy one in the near future – twelve in the PostPGY1 group (60%) and 14 in the Interns group (70%).
Pre-study PDA survey. Ownership and usage.
Pre-study PDA survey. Reported uses of PDAs.
Evaluation of smart phones usage
From the group of 55 residents that answered the pre-study survey, thirty-one used the smart phones during the seven-month trial period – thirteen in the PostPGY1 group and 18 in the Interns group. All of them filled the post-study evaluation and reported that this was their first experience with real-time mobile Internet access for clinical activities. During the time of the study, twenty-five physicians (80%) reported accessing the Web from the smart phones between 1 to 5 times a day and four of them between 5 to 10 times, only two physicians reported accessing the Internet more than 10 times a day.
We monitored and measured the usage of NLM resources but the participants also used the smart phones to access other Websites for their searches. The most common reported were UpToDate, eMedicine, MDConsult, New England Journal of Medicine, Google and Medscape. The smart phones were found "very easy" or "easy" to use by twenty two of the physicians (71%), whereas nine of them considered their usage as "fair". None of them considered that their use was difficult [Figure ]. Thirteen physicians rated the speed of the wireless connection as "fast" and a similar number considered that it was "average". On the other hand, only three of the participants (10%) rated the speed of the Internet connection as slow or very slow [Figure ]. Residents reported a slow transmission from some Websites that were not handheld-friendly, making their access a time-consuming process.
Ease of use and perceived rate of speed. Perceived usability of the smart phones and rate of speed (n = 31)
Eighteen of the 31 physicians (58%) reported that they "frequently" found the information they were looking for, ten of them found it "sometimes" (32%), and three doctors in the Interns group reported that they "always" found the proper information during their searches [Table ]. Sixteen physicians (52%) considered that the information obtained at real-time "frequently" had an impact in the diagnostic or management process of patients, whereas for 13 participants it happened "sometimes" (42%) and "rarely" or "never" for only two physicians from the Interns group [Table ]. Twenty-nine of the participants said that they were "satisfied" or "very satisfied" with the experience (94%), and the same number stated that they were "likely" or "very likely" to use this technology in the future. Moreover, twenty-three of them (74%) were planning to buy a PDA with wireless Internet access for personal use in the future. All the participants would recommend the daily use of these devices to colleagues in their practices.
Smart phones usage evaluation. How often the information was found on the Internet?
Smart phones usage evaluation. How often did the information have an impact in the diagnostic or management process?
The final evaluation included a section for free comments. We did not measure the actual time spent doing the searches during the rounds, but the work of the team was not affected by the use of the smart phones because usually only one of the team members was in charge of the literature search while the others continued assessing the patient. In general, house staff commented that they saved time using the phones because of the "immediate availability of information for discussion of patients' medical problems". Physicians considered the small size and mobility as main advantages of these devices. This availability and easy access to medical, scientific information saved time during the daily activities "when it is difficult to find a desktop available" or "from any place in the hospital". The smart phones were easy to carry and allowed fast and ubiquitous access to the Internet. They also commented on better results "when proper questions were made" and there was not "impatience for developing analysis". Residents and faculty participating in this study reported that the information retrieved from the Internet was used not only for discussions about specific cases but also to review topics with attendings, update individual knowledge and prepare academic activities such as morning reports, journal clubs and noon conferences.
The lack of familiarity with smart phones and the small keyboard and screen were reported as negative factors for usability. Other barriers or disadvantages mentioned were: "cost of the equipment", "phone company charges", "large amount of information needed every day", and "physician's impatience". There were no reports of interference of the cellular phones with medical devices during the study period.
NLM server logs analysis
The analysis of NLM's Web server logs from August 2005 to February 2006 showed a cyclic pattern of usage, with peak usage during the months of December and January. On the other hand, the access dropped between September and November [Figure ]. A total of 546 searches were performed using NLM tools during the seven-month period of study. Table shows a monthly breakdown of specific NLM resources accessed by the participants. Eighty eight questions were sent to askMEDLINE. The four most common questions were on cocaine and acute renal failure, tinnitus, hypernatremia, and arrhythmias in anemia. Two hundred and fifty five searches were carried out using PICO. The ten most frequently searched terms in PICO were colon cancer, mast cells, Crohn's disease, splenomegaly, pancytopenia, pancreatitis, systemic lupus, renal abscess, rhabdomyolysis and hypercalcemia. Disease Associations (DA) was used to perform two hundred and three searches. The ten more frequent searched associations were: pulmonary embolism and arthroscopy of the knee, stomatitis and recurrent herpes, asthma and magnesium, spirochetes in sputum, AIDS and Crohn's disease, vasculitis and purpura, adrenal insufficiency and eosinophilia, hepatomegaly and sarcoidosis, transudates and ovarian cancer, and kidney infarction and cocaine abuse. The weekly use of NLM tools showed a decreased use on Thursdays and Fridays from an initial three-day average of one hundred and fifty hits [Figure ]. The time of major activity in the hospital wards correlated with the analysis of hourly access observed at the NLM server, which showed a maximum usage during the mornings, with a peak between 8 am and 10 am and progressive declining after 2 pm [Figure ]. The devices were not available at night or weekends.
Monthly Access to NLM Server. Number of hits to NLM server by month from August 2005 to February 2006
Monthly access to NLM server by resource from August 2005 to February 2006.
Daily Access to NLM Server. Number of hits to NLM server from Monday to Friday during the study period
Hourly Access to NLM Server. Number of hits to NLM server hourly from 8 am to 4 pm during the study period