Evaluation Methods in Medical Informatics

323 Pages·1997·6.582 MB·English

by  Charles P. Friedman Ph.D., Jeremy C. Wyatt M.B., B.S., D.M., M.R.C.P. (auth.)

Computers and Medicine
Helmuth F. Orthner, Series Editor
Springer Science+Business Media, LLC
Computers and Medicine
Use and Impact of Computers in Clinical Medicine
James G. Anderson and Stephen J. Jay (Editors)
Information Systems for Patient Care
Bruce I. Blum (Editor)
A Clinical Information System for Oncology
John P. Enterline, Raymond E. Lenhard, Jr., and Bruce I. Blum
(Editors)
Evaluation Methods in Medical Informatics
Charles P. Friedman and Jeremy C. Wyatt (Editors)
HELP: A Dynamic Hospital Information System
Gilad J. Kuperman, Reed M. Gardner, and T. Allan Pryor
Expert Critiquing Systems
Perry L. Miller
Selected Topics in Medical Artificial Intelligence
Perry L. Miller (Editor)
Implementing Health Care Information Systems
Helmuth F. Orthner and Bruce I. Blum (Editors)
Computer-Assisted Medical Decision Making, Volume 1
James A. Reggia and Stanley Tuhrim (Editors)
Computer-Assisted Medical Decision Making, Volume 2
James A. Reggia and Stanley Tuhrim (Editors)
Nursing and Computers: An Anthology
Virginia K. Saba, Karen A. Rieder, and Dorothy B. Pocklington
(Editors)
Knowledge Engineering in Health Informatics
Homer R. Warner, Dean K. Sorenson, and Omar Bouhaddou
Charles P. Friedman
Jeremy C. Wyatt
Evaluation Methods
in Medical Informatics
Foreword by Edward H. Shortliffe
With 40 Illustrations
Springer
Charles P. Friedman, Ph.D. Jeremy C. Wyatt, M.B., B.S., D.M., M.R.C.P.
Formerly Assistant Dean for Medical Consultant, Medical Informatics
Education and Informatics, University of Imperial Cancer Research Fund
North Carolina P.O. Box 123
Professor and Director Lincoln's Inn Fields
Center for Biomedical Informatics London WC2A 3PX, UK
University of Pittsburgh
8074 Forbes Tower
Pittsburgh, PA 15213, USA
Contributors:
Bonnie Kaplan, Ph.D. Allen C. Smith, m, Ph.D.
Associate Professor, Computer Science! Assistant Professor and Associate Director
Information Systems Office of Educational Development
Director, Medical Information Systems Program CB 7530-322 MacNider Building
School of Business University of North Carolina School of
Quinnipiac College Medicine
Hamden, cr 06518, USA Chapel Hill, NC 27599, USA
Series Editor:
Helmuth F. Orthner, Ph.D.
Professor of Medical Informatics
University of Utah Health Sciences Center
Salt Lake City, UT 84132, USA
Library of Congress Cataloging-in-Publication Data
Evaluation methods in medical informatics/Charles P. Friedman,
Jeremy C. Wyatt, with contributions by Bonnie Kaplan, Allen C. Smith m
p. cm.-(Computers and medicine)
Includes bibliographical references and index.
1. Medical informatics--Research--Methodology. 2. Medicine-Data
processing-Evaluation. I. Friedman, Charles P. II. Wyatt, J.
(Jeremy) m. Series: Computers and medicine (New York, N.Y.)
[DNLM: 1. Medical informatics. 2. Technology, Medical.
3. Decision Support Techniques. W 26.55.A 7 E92 1996]
R858.E985 1996
610.285--dc20 96-18411
Printed on acid-free paper.
© 1997 Springer Science+Business Media New York
Originally Published by Springer-Verlag New York, Inc. in 1997
Softcover reprint of the hardcover 1st edition 1997
All rights reserved. This work may not be translated or copied in whole or in part without the written
permission of the publisher (Springer-Verlag New York, Inc., 175 Fifth Avenue, New York, NY
10010, USA), except for brief excerpts in connection with reviews or scholarly analysis. Use in con
nection with any form of information storage and retrieval, electronic adaptation, computer software,
or by similar or dissimilar methodology now known or hereafter developed is forbidden.
The use of general descriptive names, trade names, trademarks, etc., in this publication, even if the for
mer are not especially identified, is not to be taken as a sign that such names, as understood by the
Trade Marks and Merchandise Marks Act, may accordingly be used freely by anyone.
While the advice and information in this book are believed to be true and accurate at the date of going
to press, neither the authors, nor the editors, nor the publisher can accept any legal responsibility for
any errors or omissions that may be made. The publisher makes no warranty, express or implied, with
respect to the material contained herein.
Production coordinated by Carlson Co. and managed by Natalie Johnson; manufacturing supervised
by Jeffrey Taub.
Typeset by Carlson Co., Yellow Springs, OH, from the authors' electronic files.
9 8 765 432 1
ISBN 978-1-4757-2687-9 ISBN 978-1-4757-2685-5 (eBook)
DOI 10.1007/978-1-4757-2685-5
To Pat and Sylvia
Foreword
As director of a training program in medical informatics, I have found that one of
the most frequent inquiries from graduate students is, "Although I am happy with
my research focus and the work I have done, how can I design and carry out a
practical evaluation that proves the value of my contribution?" Informatics is a
multifaceted, interdisciplinary field with research that ranges from theoretical
developments to projects that are highly applied and intended for near-term use in
clinical settings. The implications of "proving" a research claim accordingly vary
greatly depending on the details of an individual student's goals and thesis state
ment. Furthermore, the dissertation work leading up to an evaluation plan is often
so time-consuming and arduous that attempting the "perfect" evaluation is fre
quently seen as impractical or as diverting students from central programming or
implementation issues that are their primary areas of interest. They often ask what
compromises are possible so they can provide persuasive data in support of their
claims without adding another two to three years to their graduate student life.
Our students clearly needed help in dealing more effectively with such dilem
mas, and it was therefore fortuitous when, in the autumn of 1991, we welcomed
two superb visiting professors to our laboratories. We had known both Chuck
Friedman and Jeremy Wyatt from earlier visits and professional encounters, but it
was coincidence that offered them sabbatical breaks in our laboratory during the
same academic year. Knowing that each had strong interests and skills in the areas
of evaluation and clinical trial design, I hoped they would enjoy getting to know
one another and would find that their scholarly pursuits were both complementary
and synergistic. To help stir the pot, we even assigned them to a shared office that
we try to set aside for visitors, and within a few weeks they were putting their
heads together as they learned about the evaluation issues that were rampant in
our laboratory.
The contributions by Drs. Friedman and Wyatt during that year were mar
velous, and they continue to have ripple effects today. They served as local con
sultants as we devised evaluation plans for existing projects, new proposals, and
student research. By the spring they had identified the topics and themes that
needed to be understood better by those in our laboratory, and they offered a well
received seminar on evaluation methods for medical information systems. It was
out of the class notes formulated for that course that the present volume evolved.
vii
viii Foreword
Its availability will allow us to rejuvenate and refine the laboratory's knowledge
and skills in the area of evaluating medical information systems, so we have
eagerly anticipated its publication.
This book fills an important niche that is not effectively covered by other med
ical informatics textbooks or by the standard volumes on evaluation and clinical
trial design. I know of no other writers who have the requisite knowledge of sta
tistics coupled with intensive study of medical informatics and an involvement
with creation of applied systems as well. Drs. Friedman and Wyatt are scholars
and educators, but they are also practical in their understanding of the world of
clinical medicine and the realities of system implementation and validation in set
tings that defy formal controlled trials. Thus the book is not only of value to stu
dents of medical informatics but will be a key reference for all individuals
involved in the implementation and evaluation of basic and applied systems in
medical informatics.
EDWARD H. SHORTLIFFE, M.D., PH.D
Section of Medical Informatics
Stanford University School of Medicine
Series Preface
This monograph series intends to provide medical information scientists, health
care administrators, physicians, nurses, other health care providers, and computer
science professionals with successful examples and experiences of computer appli
cations in health care settings. Through these computer applications, we attempt to
show what is effective and efficient, and hope to provide guidance on the acquisi
tion or design of medical information systems so that costly mistakes can be
avoided.
The health care provider organizations such as hospitals and clinics are experi
encing large demands for clinical information because of a transition from a "fee
for-service" to a "capitation-based" health care economy. This transition changes
the way health care services are being paid for. Previously, nearly all heath care
services were paid for by insurance companies after the services were performed.
Today, many procedures need to be pre-approved, and many charges for clinical
services must be justified to the insurance plans. Ultimately, in a totally capitated
system, the more patient care services are provided per patient, the less profitable
the health care provider organization will be. Clearly, the financial risks have
shifted from the insurance carriers to the health care provider organizations. In
order for hospitals and clinics to assess these financial risks, management needs to
know what services are to be provided and how to reduce them without impacting
the quality of care. The balancing act of reducing costs but maintaining health care
quality and patient satisfaction requires accurate information of the clinical ser
vices. The only way this information can be collected cost-effectively is through
the automation of the health care process itself. Unfortunately, current health infor
mation systems are not comprehensive enough, and their level of integration is low
and primitive at best. There are too many "islands" even within single health care
provider organizations.
With the rapid advance of digital communications technologies and the accep
tance of standard interfaces, these "islands" can be bridged to satisfy most infor
mation needs of health care professionals and management. In addition, the
migration of health information systems to client/server computer architectures
allows us to re-engineer the user interface to become more functional, pleasant, and
also responsive. Eventually, we hope, the clinical workstation will become the tool
that health care providers use interactively without intermediary data entry support.
ix
x Series Preface
Computer-based information systems provide more timely and legible informa
tion than traditional paper-based systems. In addition, medical information systems
can monitor the process of health care and improve quality of patient care by pro
viding decision support for diagnosis or therapy, clinical reminders for follow-up
care, warnings about adverse drug interactions, alerts to questionable treatment or
deviations from clinical protocols, and more. The complexity of the health care
workplace requires a rich set of requirements for health information systems. Fur
ther, the systems must respond quickly to user interactions and queries in order to
facilitate and not impede the work of health care professionals. Because of this and
the requirement for a high level of security, these systems can be classified as very
complex and, from a developer's perspective, also as "risky" systems.
Information technology is advancing at an accelerated pace. Instead of waiting
for three years for a new generation of computer hardware, we are now confronted
with new computing hardware every 18 months. The forthcoming changes in the
telecommunications industry will be revolutionary. Within the next five years,
and certainly before the end of this century, new digital communications tech
nologies, such as the Integrated Services Digital Network (ISDN), Asynchronous
Data Subscriber Loop (ADSL) technologies, and very high speed local area net
works using efficient cell switching protocols (e.g., ATM), will not only change
the architecture of our information systems, but also the way we work and manage
health care institutions.
The software industry constantly tries to provide tools and productive develop
ment environments for the design, implementation, and maintenance of informa
tion systems. Still, the development of information systems in medicine is an art,
and the tools we use are often self-made and crude. One area that needs desperate
attention is the interaction of health care providers with the computer. While the
user interface needs improvement and the emerging graphical user-interfaces form
the basis for such improvements, the most important criterion is to provide relevant
and accurate information without drowning the physician in too much (irrelevant)
data.
To develop an effective clinical system requires an understanding of what is to
be done and how to do it, as well as an understanding on how to integrate informa
tion systems into an operational health care environment. Such knowledge is rarely
found in anyone individual; all systems described in this monograph series are the
work ofteams. The size of these teams is usually small, and the composition is het
erogeneous, i.e., health professionals, computer and communications scientists and
engineers, statisticians, epidemiologists, and so on. The team members are usually
dedicated to working together over long periods of time, sometimes spanning
decades.
Clinical information systems are dynamic systems, their functionality con
stantly changing because of external pressures and administrative changes in
health care institutions. Good clinical information systems will and should change
the operational mode of patient care which, in tum, should affect the functional
requirements of the information systems. This interplay requires that medical
information systems be based on architectures that allow them to be adapted
Series Preface xi
rapidly and with minimal expense. It also requires a willingness by management
of the health care institution to adjust its operational procedures, and, most of all,
to provide end-user education in the use of information technology. While med
ical information systems should be functionally integrated, these systems should
also be modular so that incremental upgrades, additions, and deletions of modules
can be done in order to match the pattern of capital resources and investments
available to an institution.
We are building medical information systems just as automobiles were built
early in this century, i.e., in an ad-hoc manner that disregarded even existent stan
dards. Although technical standards addressing computer and communications
technologies are necessary, they are insufficient. We still need to develop conven
tions and agreements, and perhaps a few regulations that address the principal use
of medical information in computer and communications systems. Standardization
allows the mass production of low cost parts which can be used to build more com
plex structures. What exactly are these parts in medical information systems? We
need to identify them, classify them, describe them, publish their specifications,
and, most importantly, use them in real health care settings. We must be sure that
these parts are useful and cost effective even before we standardize them.
Clinical research, health service research, and medical education will benefit
greatly when controlled vocabularies are used more widely in the practice of med
icine. For practical reasons, the medical profession has developed numerous classi
fications, nomenclatures, dictionary codes, and thesauri (e.g., lCD, CPT, DSM-III,
SNOMED, COSTAR dictionary codes, BAlK thesaurus terms, and MESH terms).
The collection of these terms represents a considerable amount of clinical activi
ties, a large portion of the health care business, and access to our recorded knowl
edge. These terms and codes form the glue that links the practice of medicine with
the business of medicine. They also link the practice of medicine with the literature
of medicine, with further links to medical research and education. Since informa
tion systems are more efficient in retrieving information when controlled vocabu
laries are used in large databases, the attempt to unify and build bridges between
these coding systems is a great example of unifying the field of medicine and
health care by providing and using medical informatics tools. The Unified Medical
Language System (UMLS) project of the National Library of Medicine, NIH, in
Bethesda, Maryland, is an example of such an effort.
The purpose of this series is to capture the experience of medical informatics
teams that have successfully implemented and operated medical information sys
tems. We hope the individual books in this series will contribute to the evolution of
medical informatics as a recognized professional discipline. We are at the threshold
where there is not just the need but already the momentum and interest in the health
care and computer science communities to identify and recognize the new disci
pline called Medical Informatics.
I would like to thank the editors of Springer-Verlag New York for the opportu
nity to edit this series. Also, many thanks to the present and past departmental
chairmen who allowed me to spend time on this activity: William S. Yamamoto,
M.D., and Thomas E. Piemme, M.D., of the Department of Computer Medicine at