Search tips
Search criteria

Results 1-25 (25)

Clipboard (0)

Select a Filter Below

Year of Publication
Document Types
1.  Effects of a decision support system on the diagnostic accuracy of users: a preliminary report. 
OBJECTIVES: To assess the effects of incomplete data upon the output of a computerized diagnostic decision support system (DSS), to assess the effects of using the system upon the diagnostic opinions of users, and to explore if these effects vary as a function of clinical experience. DESIGN: Experimental pilot study. Four clusters of nine cases each were constructed and equated for case difficulty. Definitive findings were omitted from the case abstracts. Subjects were randomly assigned to one of four clusters and were trained on the DSS prior to use. SUBJECTS: The study involved 16 physicians at three levels of clinical experience (six general internists, four residents in internal medicine, and six fourth-year medical students), from three academic medical centers. PROCEDURE: Each subject worked up nine cases, first without and then with ILIAD consultation. They were asked to offer up to six potential diagnoses and to list up to three steps that should be the next items in the diagnostic workup. Effects of DSS consultation were measured by changes in the position of the correct diagnosis in the lists of differential diagnoses, pre- and post-consultation. RESULTS: The DSS lists of diagnostic possibilities contained the correct diagnosis in 38% of cases, about midway between the levels of accuracy of residents and attending general internists. In over 70% of cases, the DSS output had no effect on the position of the correct diagnosis in the subjects' lists. The system's diagnostic accuracy was unaffected by the clinical experience of the users.
PMCID: PMC116326  PMID: 8930858
4.  A continuous-speech interface to a decision support system: II. An evaluation using a Wizard-of-Oz experimental paradigm. 
OBJECTIVE: Evaluate the performance of a continuous-speech interface to a decision support system. DESIGN: The authors performed a prospective evaluation of a speech interface that matches unconstrained utterances of physicians with controlled-vocabulary terms from Quick Medical Reference (QMR). The performance of the speech interface was assessed in two stages: in the real-time experiment, physician subjects viewed audiovisual stimuli intended to evoke clinical findings, spoke a description of each finding into the speech interface, and then chose from a list generated by the interface the QMR term that most closely matched the finding. Subjects believed that the speech recognizer decoded their utterances; in reality, a hidden experimenter typed utterances into the interface (Wizard-of-Oz experimental design). Later, the authors replayed the same utterances through the speech recognizer and measured how accurately utterances matched with appropriate QMR terms using the results of the real-time experiment as the "gold standard." MEASUREMENTS: The authors measured how accurately the speech-recognition system converted input utterances to text strings (recognition accuracy) and how accurately the speech interface matched input utterances to appropriate QMR terms (semantic accuracy). RESULTS: Overall recognition accuracy was less than 50%. However, using language-processing techniques that match keywords in recognized utterances to keywords in QMR terms, the semantic accuracy of the system was 81%. CONCLUSIONS: Reasonable semantic accuracy was attained when language-processing techniques were used to accommodate for speech misrecognition. In addition, the Wizard-of-Oz experimental design offered many advantages for this evaluation. The authors believe that this technique may be useful to future evaluators of speech-input systems.
PMCID: PMC116236  PMID: 7895136
5.  Information retrieved from a database and the augmentation of personal knowledge. 
OBJECTIVE: To assess the degree to which information retrieved from a biomedical database can augment personal knowledge in addressing novel problems, and how the ability to retrieve information evolves over time. DESIGN: This longitudinal study comprised three assessments of two cohorts of medical students. The first assessment occurred just before student course experience in bacteriology, the second occurred just after the course, and the third occurred five months later. At each assessment, the students were initially given a set of bacteriology problems to solve using their personal knowledge only. Each student was then reassigned a sample of problems he or she had answered incorrectly, to work again with assistance from a database containing information about bacteria and bacteriologic concepts. The initial pass through the problems generated a "personal knowledge" score; the second pass generated a "database-assisted" score for each student at each assessment. RESULTS: Over two cohorts, students' personal knowledge scores were very low (approximately 12%) at the first assessment. They rose substantially at the second assessment (approximately 48%) but decreased six months later (approximately 25%). By contrast, database-assisted scores rose linearly: from approximately 44% at the first assessment to approximately 57% at the second assessment, to approximately 75% at the third assessment. CONCLUSION: The persistent increase in database-assisted scores, even when personal knowledge had attenuated, was the most remarkable finding of this study. While some of the increase may be attributed to artifacts of the design, the pattern seems to result from the retained ability to recognize problem-relevant information in a database even when it cannot be recalled.
PMCID: PMC116215  PMID: 7719819
6.  Designing medical informatics research and library--resource projects to increase what is learned. 
Careful study of medical informatics research and library-resource projects is necessary to increase the productivity of the research and development enterprise. Medical informatics research projects can present unique problems with respect to evaluation. It is not always possible to adapt directly the evaluation methods that are commonly employed in the natural and social sciences. Problems in evaluating medical informatics projects may be overcome by formulating system development work in terms of a testable hypothesis; subdividing complex projects into modules, each of which can be developed, tested and evaluated rigorously; and utilizing qualitative studies in situations where more definitive quantitative studies are impractical.
PMCID: PMC116182  PMID: 7719785
7.  Development of visual diagnostic expertise in pathology. 
In this study, we use methods from information-processing to explore the visual diagnostic processes of novice, intermediate, and expert pathologists. Subjects were asked to examine and interpret a set of slides while we collected think-aloud verbal protocols and captured on digital video the actual visual data they examined from the microscope. We performed an in-depth combined video and protocol-based analysis of processes and errors occurring as the task was performed. Additionally, we collected measures of accuracy, certainty, and difficulty for all cases. Our preliminary analysis identified significant differences between groups in all three major aspects of this task: searching skills, perceptual skills and cognitive/reasoning skills. We describe the implications of our preliminary cognitive task analysis on the design of a developing intelligent educational system in Pathology.
PMCID: PMC2243368  PMID: 11825167
8.  Development of a MeSH-based index of faculty research interests. 
An index of faculty research interests terms has many uses for an institution's researchers and administrators. This paper describes the Faculty Research Interests Project (FRIP), which addresses vocabulary and compliance problems inherent in research interests index development. FRIP creates an index using Medical Subject Headings (MeSH) associated with the MEDLINE-indexed publications of faculty authors. Following a preliminary study, a Web-based term selection component was developed that allows faculty users not only to choose MeSH terms but also to add both additional author names under which they have published and original terms in real time. In a study involving 136 medical school faculty, users successfully navigated the term selection component, and more than 90 percent of the terms they selected were MeSH terms, confirming MeSH's usefulness for indexing research interests.
PMCID: PMC2244091  PMID: 11079886
9.  Information resources assessment of a healthcare integrated delivery system. 
While clinical healthcare systems may have lagged behind computer applications in other fields in the shift from mainframes to client-server architectures, the rapid deployment of newer applications is closing that gap. Organizations considering the transition to client-server must identify and position themselves to provide the resources necessary to implement and support the infrastructure requirements of client-server architectures and to manage the accelerated complexity at the desktop, including hardware and software deployment, training, and maintenance needs. This paper describes an information resources assessment of the recently aligned Pennsylvania regional Veterans Administration Stars and Stripes Health Network (VISN4), in anticipation of the shift from a predominantly mainframe to a client-server information systems architecture in its well-established VistA clinical information system. The multimethod assessment study is described here to demonstrate this approach and its value to regional healthcare networks undergoing organizational integration and/or significant information technology transformations.
PMCID: PMC2232856  PMID: 10566414
10.  Scoring performance on computer-based patient simulations: beyond value of information. 
As computer based clinical case simulations become increasingly popular for training and evaluating clinicians, approaches are needed to evaluate a trainee's or examinee's solution of the simulated cases. In 1997 we developed a decision analytic approach to scoring performance on computerized patient case simulations, using expected value of information (VOI) to generate a score each time the user requested clinical information from the simulation. Although this measure has many desirable characteristics, we found that the VOI was zero for the majority of information requests. We enhanced our original algorithm to measure potential decrements in expected utility that could result from using results of information requests that have zero VOI. Like the original algorithm, the new approach uses decision models, represented as influence diagrams, to represent the diagnostic problem. The process of solving computer based patient simulations involves repeated cycles of requesting and receiving these data from the simulations. Each time the user requests clinical data from the simulation, the influence diagram is evaluated to determine the expected VOI of the requested clinical datum. The VOI is non-zero only it the requested datum has the potential to change the leading diagnosis. The VOI is zero when the data item requested does not map to any node in the influence diagram or when the item maps to a node but does not change the leading diagnosis regardless of it's value. Our new algorithm generates a score for each of these situations by modeling what would happen to the expected utility of the model if the user changes the leading diagnosis based on the results. The resulting algorithm produces a non-zero score for all information requests. The score is the VOI when the VOI is non-zero It is a negative number when the VOI is zero.
PMCID: PMC2232774  PMID: 10566413
11.  Student and faculty performance in clinical simulations with access to a searchable information resource. 
In this study we explore how students' use of an easily accessible and searchable database affects their performance in clinical simulations. We do this by comparing performance of students with and without database access and compare these to a sample of faculty members. The literature supports the fact that interactive information resources can augment a clinician's problem solving ability in small clinical vignettes. We have taken the INQUIRER bacteriological database, containing detailed information on 63 medically important bacteria in 33 structured fields, and incorporated it into a computer-based clinical simulation. Subjects worked through the case-based clinical simulations with some having access to the INQUIRER information resource. Performance metrics were based on correct determination of the etiologic agent in the simulation and crosstabulated with student access of the information resource; more specifically it was determined whether the student displayed the database record describing the etiologic agent. Chi-square tests show statistical significance for this relationship (chi 2 = 3.922; p = 0.048). Results support the idea that students with database access in a clinical simulation environment can perform at a higher level than their counterparts who lack access to such information, reflecting favorably on the use of information resources in training environments.
PMCID: PMC2232500  PMID: 10566439
12.  Feasibility and marketing studies of health sciences librarianship education programs. 
The University of North Carolina at Chapel Hill evaluated five curricular models designed to improve education for health sciences librarianship. Three of the models enhanced existing degree and certificate programs, and two were new programs for working information professionals. Models were developed with input from experts and a Delphi study; the marketability of the models was tested through surveys of potential students and employers; and recommendations were made as a guide to implementation. The results demonstrated a demand for more specialized curricula and for retraining opportunities. Marketing data showed a strong interest from potential students in a specialized master's degree, and mid-career professionals indicated an interest in postmaster's programs that provided the ability to maintain employment. The study pointed to the opportunity for a center of excellence in health sciences information education to enable health sciences librarians to respond to their evolving roles.
PMCID: PMC226522  PMID: 9934529
13.  Changes in diagnostic decision-making after a computerized decision support consultation based on perceptions of need and helpfulness: a preliminary report. 
We examined the degree to which attending physicians, residents, and medical students' stated desire for a consultation on difficult-to-diagnose patient cases is related to changes in their diagnostic judgments after a computer consultation, and whether, in fact, their perceptions of the usefulness of these consultations are related to these changes. The decision support system (DSS) used in this study was ILIAD (v4.2). Preliminary findings based on 16 subjects' (6 general internists, 4 second-year residents in internal medicine, and 6 fourth-year medical students) workup of 136 patient cases indicated no significant main effects for 1) level of experience, 2) whether or not subjects indicated they would seek a diagnostic consultation before using the DSS, or 3) whether or not they found the DSS consultation in fact to be helpful in arriving at a diagnosis (p > .49 in all instances). Nor were there any significant interactions. Findings were similar using subjects or cases as the unit of analysis. It is possible that what may appear to be counter-intuitive, and perhaps irrational, may not necessarily be so. We are currently examining potential explanatory hypotheses in our ongoing current, larger study.
PMCID: PMC2233524  PMID: 9357629
14.  Acceptability and usage patterns of an image analysis workstation. 
Critical to the successful deployment and use of new computer systems is the acceptance of the system by the users, i.e., the clinicians. We describe a study which evaluated, in an experimental setting, the potential acceptability of an image analysis workstation for radiation therapy. The acceptability and usage patterns were measured using semi-structured questionnaires and maintaining logs of user interactions. The results of the study showed that the radiation oncologists, who were the subjects for the study, perceived the workstation as acceptable. The results also suggested several areas for improvement of workstation that could increase its acceptance in the clinical setting.
PMCID: PMC2233455  PMID: 9357665
15.  A decision analytic method for scoring performance on computer-based patient simulations. 
As computer based clinical case simulations become increasingly popular for training and evaluating clinicians, approaches are needed to evaluate a trainee's or examinee's solution of the simulated cases. We developed a decision analytic approach to scoring performance on computerized patient case simulations. We developed decision models for computerized patient case simulations in four specific domains in the field of infectious disease. The decision models were represented as influence diagrams. A single decision node represents the possible diagnoses the user may make. One chance node represents a probability distribution over the set of competing diagnoses in the simulations. The value node contains the utilities associated with all possible combinations of diagnosis and disease. All relevant data that the user may request from the simulation are represented as chance nodes with arcs to or from the diagnosis node and/or each other. Probabilities in the decision model were derived from the literature, where available, or expert opinion. Utilities were assessed by standard gamble from clinical experts. The process of solving computer based patient simulations involves repeated cycles of requesting data (history, physical examination or laboratory) and receiving these data from the simulations. Each time the user requests clinical data from the simulation, the influence diagram is evaluated with and without an arc from the corresponding chance node to the decision node. The difference in expected utility between the two solutions of the influence diagram represents the expected value of information (VOI) from the requested clinical datum. The ratio of the expected VOI from the data requested and the expected value of perfect information about the diagnosis is a normative measure of the quality of each of the user's data requests. This approach provides a continuous measure of the quality of the user's data requests in a way that is sensitive to the previous data collected. The score distinguishes serious from minor misdiagnoses. And the same influence diagram can be used to evaluate performance on multiple simulations in the same clinical domain.
PMCID: PMC2233377  PMID: 9357709
16.  Preparing tomorrow's health sciences librarians: feasibility and marketing studies. 
The University of North Carolina at Chapel Hill is devising and evaluating five curricular models designed to improve education for health sciences librarianship. These models fit into a continual learning process from the initial professional preparation to lifelong learning opportunities. Three of them enhance existing degree and certificate programs in the School of Information and Library Science (SILS) with a health sciences specialization, and two are new programs for working information professionals. The approaches involve partnerships among SILS, the Health Sciences Library, and the program in Medical Informatics. The planning process will study the feasibility of the proposed programs, test the marketability of the models to potential students and employers, and make recommendations about implementation.
PMCID: PMC226194  PMID: 8913557
17.  The influence of a decision support system on the differential diagnosis of medical practitioners at three levels of training. 
As computer-based diagnostic consultation systems become, available, their influence and usefulness need to be evaluated. This report, based on partial data from a larger study, examines the influence of Iliad, a diagnostic consultation system, on the differential diagnosis of fourth year medical students, residents in medicine, and attendings in general internal medicine. Our results show that when faced with difficult diagnostic cases, medical students add significantly more diagnoses from Iliad's differential than do residents or attendings. However, the quality of Iliad's diagnostic advice in terms of the presence of the correct diagnosis, is no better for consultations done by students or residents compared to attendings.
PMCID: PMC2233132  PMID: 8947660
18.  Verifying radiotherapy treatment setup by interactive image registration. 
Digital image analysis techniques can be used to assist the physician in diagnostic or therapeutic decision making. In radiation oncology, portal image registration can improve the accuracy of detection of errors during radiation treatment. Following a discussion of the general paradigm of interactive image registration, we describe PortFolio, a workstation for portal image analysis.
PMCID: PMC2233020  PMID: 8947672
19.  A comparison of hypertext and Boolean access to biomedical information. 
This study explored which of two modes of access to a biomedical database better supported problem solving in bacteriology. Boolean access, which allowed subjects to frame their queries as combinations of keywords, was compared to hypertext access, which allowed subjects to navigate from one database node to another. The accessible biomedical data were identical across systems. Data were collected from 42 first year medical students, each randomized to the Boolean or hypertext system, before and after their bacteriology course. Subjects worked eight clinical case problems, first using only their personal knowledge and, subsequently, with aid from the database. Database retrievals enabled students to answer questions they could not answer based on personal knowledge only. This effect was greater when personal knowledge of bacteriology was lower. The results also suggest that hypertext was superior to Boolean access in helping subjects identify possible infectious agents in these clinical case problems.
PMCID: PMC2232979  PMID: 8947616
20.  Information-seeking behaviors of medical students: a classification of questions asked of librarians and physicians. 
To solve a problem, a person often asks questions of someone with more expertise. This paper reports on a study of the types of questions asked and how the experts are chosen. In the study, sixty-three first-year medical students responded to clinical scenarios, each describing a patient affected by a toxin and asking questions concerning the identity of the toxin and its characteristics. After answering those questions, the students were asked to imagine that they had access to a medical reference librarian and an internist specializing in toxicology. The students then generated two questions for each expert about each clinical scenario. Each question was categorized according to the type of information requested, and the frequency of each type of question was calculated. The study found that students most often asked for the identification of the toxin(s), references about the scenario, or the effects of the toxin; an explanation of the patient's symptoms; or a description of the appropriate treatment. Students were more likely to address questions on the identity of the toxin and references to the hypothetical librarian; they were more likely to ask the internist for explanations of the symptoms and descriptions of the treatment. The implications of these results for the design of information and educational systems are discussed.
PMCID: PMC225929  PMID: 7920340
21.  Academic physicians' assessment of the effects of computers on health care. 
We assessed the attitudes of academic physicians towards computers in health care at two academic medical centers that are in the early stages of clinical information-system deployment. We distributed a 4-page questionnaire to 470 subjects, and a total of 272 physicians (58%) responded. Our results show that respondents use computers frequently, primarily to perform academic-oriented tasks as opposed to clinical tasks. Overall, respondents viewed computers as being slightly beneficial to health care. They perceive self-education and access to up-to-date information as the most beneficial aspects of computers and are most concerned about privacy issues and the effect of computers on the doctor-patient relationship. Physicians with prior computer training and greater knowledge of informatics concepts had more favorable attitudes towards computers in health care. We suggest that negative attitudes towards computers can be addressed by careful system design as well as targeted educational activities.
PMCID: PMC2247834  PMID: 7949990
22.  Automated medical knowledge acquisition: a study of consistency. 
Knowledge bases are more representative of the population of medical experts if they are constructed by a group of individuals, rather than one practitioner. However, one runs into problems with consistency when information is elicited from a group without a consistent format and terminology. This study examines the consistency of relatively unconstrained computer-elicited medical knowledge using the computer program, KSSO. The results of this study show that the group of ten general internists were somewhat consistent in the diagnoses they listed for a patient presenting with chest pain. They were much less consistent in the findings they listed to differentiate between the diagnoses they had listed. The mean number of subjects listing each diagnosis was 3.3 +/- 2.7 while the mean for findings was 2.0 +/- 1.5. The implications of these data are discussed.
PMCID: PMC2247733  PMID: 7950020
23.  Database access and problem solving in the basic sciences. 
This study examined the potential contribution that access to a database of biomedical information may offer in support of problem-solving exercises when personal knowledge is inadequate. Thirty-six medical students were assessed over four occasions and three domains in the basic sciences: bacteriology, pharmacology, and toxicology. Each assessment consisted of a two-pass protocol in which students were first assessed for their personal knowledge of a domain with a short-answer problem set. Then, for a sample of problems they had missed, they were asked to use a database, INQUIRER, to respond to questions which they had been unable to address with their personal knowledge. Results indicate that for a domain in which the database is well-integrated in course activities, useful retrieval of information which augmented personal knowledge increased over three assessment occasions, even continuing to increase several months after course exposure and experience with the database. For all domains, even at assessments prior to course exposure, students were able to moderately extend their ability to solve problems through access to the INQUIRER database.
PMCID: PMC2850661  PMID: 8130561
24.  Initial validation of a test of spatial knowledge in anatomy. 
The authors have developed HERCULES, a computer-based test designed to assess the spatial, non-verbal components of knowledge in anatomy. The test consists of two tasks, each requiring subjects to estimate the vertical level in the body of a set of color, cross-sectional images. In Task 1, subjects make the estimate based on a limited number of clues, where each clue is an anatomical structure that appears in the cross-section. In Task 2, subjects estimate the level based on a view of the cross-section with all structures shown. A validation study of this test using six images for each task was performed with preclinical medical students, fourth year medical students, and experienced teachers of anatomy as subjects. Results indicate that the exercise is at an appropriate level of difficulty and that a somewhat longer test than used in this study would be adequately reliable for use in actual assessment. The test appears to discriminate the expert faculty from more novice students and thus exhibits an aspect of validity that is very important in assessment exercises of this type.
PMCID: PMC3203562  PMID: 8130586
25.  A collaborative institutional model for integrating computer applications in the medical curriculum. 
The introduction and promotion of information technology in an established medical curriculum with existing academic and technical support structures poses a number of challenges. The UNC School of Medicine has developed the Taskforce on Educational Applications in Medicine (TEAM), to coordinate this effort. TEAM works as a confederation of existing research and support units with interests in computers and education, along with a core of interested faculty with curricular responsibilities. Constituent units of the TEAM confederation include the medical center library, medical television studios, basic science teaching laboratories, educational development office, microcomputer and network support groups, academic affairs administration, and a subset of course directors and teaching faculty. Among our efforts have been the establishment of (1) a mini-grant program to support faculty initiated development and implementation of computer applications in the curriculum, (2) a symposium series with visiting speakers to acquaint faculty with current developments in medical informatics and related curricular efforts at other institution, (3) 20 computer workstations located in the multipurpose teaching labs where first and second year students do much of their academic work, (4) a demonstration center for evaluation of courseware and technologically advanced delivery systems. The student workstations provide convenient access to electronic mail, University schedules and calendars, the CoSy computer conferencing system, and several software applications integral to their courses in pathology, histology, microbiology, biochemistry, and neurobiology. The progress achieved toward the primary goal has modestly exceeded our initial expectations, while the collegiality and interest expressed toward TEAM activities in the local environment stand as empirical measures of the success of the concept.
PMCID: PMC2247631  PMID: 1807705

Results 1-25 (25)