Human Factors Testing and Evaluation

437 Pages·2015·44.777 MB·English

by  Meister, D

Advances in Human Factors/Ergonomics, 5
Human Factors
Testing and Evaluation
Advances in Human Factors/Ergonomics
Series Editor: Gavriel Salvendy, Purdue University, West Lafayette, IN 47907, U.S.A.
Vol. 1. Human—Computer Interaction (G. Salvendy, Editor)
Vol. 2. Human-Computer Dialogue Design (R.W. Ehrich and R.C. Williges, Editors)
Vol. 3. Expertise Transfer for Expert System Design (J.H. Boose)
Vol. 4. Engineering Physiology (K.H.E. Kroemer, H.J. Kroemer and
K.E. Kroemer-Elbert)
Vol. 5. Human Factors Testing and Evaluation (D. Meister)
Advances in Human Factors/Ergonomics, 5
Human Factors
Testing and Evaluation
David Meister
U.S. Navy Personnel Research and Development Center, San Diego, CA, U.S.A.
ELSEVIER
Amsterdam Oxford - New York - Tokyo 1986
ELSEVIER SCIENCE PUBLISHERS B.V.
Sara Burgerhartstraat 25
P.O. Box 211, 1000 AE Amsterdam, The Netherlands
Distributors for the United States and Canada:
ELSEVIER SCIENCE PUBLISHING COMPANY INC.
52, Vanderbilt Avenue
New York, NY 10017, U.S.A.
ISBN 0-444-42701-5 (Vol. 5)
ISBN 0-444-42396-6 (Series)
© Elsevier Science Publishers B.V., 1986
All rights reserved. No part of this publication may be reproduced, stored in a retrieval system or
transmitted in any form or by any means, electronic, mechanical, photocopying, recording or other-
wise, without the prior written permission of the publisher, Elsevier Science Publishers B.V./Science &
Technology Division, P.O. Box 330, 1000 AH Amsterdam, The Netherlands.
Special regulations for readers in the USA — This publication has been registered with the Copyright
Clearance Center Inc. (CCC), Salem, Massachusetts. Information can be obtained from the CCC about
conditons under which photocopies of parts of this publication may be made in the USA. All other
copyright questions, including photocopying outside of the USA, should be referred to the publisher.
Printed in The Netherlands
ν
THIS BOOK IS DEDICATED TO THE MEMORY
OF
MY PARENTS, IDA AND JULIUS MEISTER
AND
MY WIFE, SHIRLEY DAVIS MEISTER
This page intentionally left blank
vii
PREFACE
Measurement, which is the heart of science, is the heart of Human Factors.
Just as Human Factors itself is different from Psychology and Engineering, so
Human Factors measurement has characteristics that set it apart from psycholo-
gical or engineering measurement.
For that reason Human Factors testing and evaluation deserve special treat-
ment. Unfortunately, the many excellent texts available in the behavioral area
do not present an adequate picture of this topic.
Because of its unique characteristics, it is necessary to approach Human
Factors measurement in a scholarly manner. By this I mean that Human Factors
testing and evaluation can help to answer rather profound questions: for
example, how does performance of the part (i.e., the personnel subsystem)
affect the whole (i.e., the entire system); how does one interpret behavioral
effects in terms of physical processes and vice versa? Beyond such questions,
there is a great intellectual challenge in attempting to measure human perfor-
mance in an operational environment in which close control of variables is
ordinarily not possible, where many multidimensional factors affect the
phenomena to be measured and it is difficult to separate their individual
influences. Measurement in the "real" world demands capabilities above and
beyond those required by traditional experimental designs.
A few caveats. Although my orientation, and that of most test specialists,
is pragmatic, any description of the Human Factors test process must be some-
what idealistic. Despite all the cautions and qualifications I have introduced
into the following chapters, this is a book about how human performance testing
should be performed. The reality of any individual test may be subtly or
grossly different. Each test has certain unique qualities which may not be
adequately expressed in general principles. Moreover, there are many questions
in behavioral measurement for which there are no answers currently; in those
cases I have had to rely on experience and intuition, neither of which may be
completely error-free.
Several colleagues have been extremely helpful in the preparation of this
text. I have difficulty expressing the deep appreciation I feel for the
efforts of a long time friend, Dr. Frederick A. Muckler, whose review of and
suggestions for the draft manuscript were of inestimable value. The editor of
this series in which this book is published, Professor Gavriel Salvendy, made a
number of useful suggestions. As always, I owe a debt of gratitude to the
Chief Librarian, Ms. Marie McDowell, and her assistant, Ms. Maria Alves, of
the Navy Personnel Research and Development Center. Their invaluable help was
always willingly forthcoming. Finally, but not least, Mrs. Joanne Newton drew
the illustrations with aplomb.
San Diego, 1986
This page intentionally left blank
TABLE OF CONTENTS PAGE NUMBER
Chapter 1 INTRODUCTION TO HUMAN FACTORS TEST AND EVALUATION
1.1 INTRODUCTION 1
1.2 DEFINITIONS 1
1.3 WHAT DO WE ATTEMPT TO MEASURE? 8
1.4 TESTING AND EVALUATION PURPOSES 10
1.5 SYSTEM DEVELOPMENT AS A TESTING CONTEXT 15
1.6 LABORATORY RESEARCH COMPARED WITH PERFORMANCE TESTING 22
1.7 TEST SITUATIONS 25
1.8 ASSUMPTIONS UNDERLYING HUMAN PERFORMANCE TESTING AND
28
EVALUATION
1.9 FACTORS AFFECTING TESTING ADEQUACY 29
1.10 THE REMAINDER OF THE BOOK 45
Chapter 2 INTRODUCTION TO MOCKUP, DEVELOPMENTAL AND OPERATIONAL TESTING
2.1 INTRODUCTION 50
2.2 MOCKUPS 50
2.3 DEVELOPMENTAL TESTS 60
2.4 OPERATIONAL TESTING 62
2.5 AN ILLUSTRATIVE SYSTEM 65
Chapter 3 DEVELOPMENT OF THE TEST PLAN
3.1 INTRODUCTION 85
3.2 SECTION 1.0, PURPOSE OF THE TEST 91
3.3 SECTION 2.0, DESCRIPTION OF THE SYSTEM 96
3.4 SECTION 3.0, EXPERIMENTAL COMPARISONS 98
3.5 SECTION 4.0, DIMENSIONS, CRITERIA, MEASURES, AND SYSTEMS 101
3.6 SECTION 5.0, DATA COLLECTION METHODS 118
3.7 SECTION 6.0, SUBJECT CHARACTERISTICS 122
3.8 SECTION 7.0, CONSTRAINTS 125
3.9 SECTION 8.0, DATA ANALYSIS 129
3.10 SECTION 9.0, TESTING SCHEDULE 133
3.11 CONDUCTING THE TEST 133
3.12 TEST REPORTING 135
Chapter 4 MEASUREMENT METHODS
4.1 INTRODUCTION 141
4.2 JOB PERFORMANCE OBSERVATION 141
4.3 INTRODUCTION TO SELF-REPORT TECHNIQUES 155
4.4 THE INTERVIEW 156
4.5 QUESTIONNAIRES 166
4.6 RATING METHODS 183
4.7 MISCELLANEOUS SUBJECTIVE METHODS 194
4.8 ACTIVITY ANALYSIS 199
4.9 OBJECTIVE MEASURES 201
4.10 CONCLUDING COMMENTS 210
Chapter 5 ENVIRONMENTAL TESTING
5.1 INTRODUCTION 215
5.2 INSTRUMENTATION 216
5.3 LIGHTING MEASUREMENTS 218
5.4 NOISE MEASUREMENT 221
5.5 VIBRATION MEASUREMENT 231
5.6 ATMOSPHERIC COMPOSITION MEASUREMENT 236
5.7 TEMPERATURE, HUMIDITY, AND VENTILATION MEASUREMENT 242
5.8 VISIBILITY MEASUREMENT 244
5.9 SUBJECTIVE METHODS OF ENVIRONMENTAL TESTING 246