Table Of Content

DOCUMENT RESUME IR 017 659 ED 391 506 Gibbs, William J.; Cheng, He Ping AUTHOR Formative Evaluation and World-Wide-Web TITLE Hypermedia. PUB DATE [95] 8p.; In: Eyes on the Future: Converging Images, NOTE Ideas, and Instruction. Selected Readings from the Annual Conference of the International Visual Literacy Association (27th, Chicago, IL, October 18-22, 1995); see IR 017 629. Research/Technical (143) Reports PUB TYPE Speeches/Conference Papers (150) MFOI/PC01 Plus Postage. EDRS PRICE *Formative Evaluation; Graduate Students; Higher DESCRIPTORS Education; Information Networks; Information Retrieval; Interaction; *Online Searching; Pilot Projects; Undergraduate Students; Users (Information); Use Studies Help Systems; *World Wide Web IDENTIFIERS ABSTRACT This paper discusses a formative evaluation method by which to assess World-Wide-Web (WWW) documents. Also presented are the results from a pilot evaluation which examined users' interactions with Web documents and the effects of providing "document maps" to guide users to specific information. A video-split screen technique was used to record users interacting with the WWW and to assess the design of associated documents. Three female and two male subjects participated in the study; four were undergraduate students and one was a graduate student. Subjects were given two (1) to browse through the WWW and locate specific tasks: to use and assess Eastern Illinois infnrmation/documents; and (2) University's Department of Media Services Web site. At the completicn of the session, subjects filled out a survey which collected information on their reactions toward the usefulness of the "document maps" and data related to their computer experience and knowledge of the WWW. Results indicated that the video-split screen method can be an effective means to assess WWW documents. Subjects need to be shown that a "document map" exists for their use and they must be explicitly informed as to its purpose and function. As WWW is used for the delivery of instruction, it is important to provide learners with a means of instructional support. Three figures present sample transcripts from the evaluation sessions and depict the time spent searching (by trials) and the number of times maps were used (by trial). (Contains 18 references.) (AEF) *********************************************************************** Reproductions supplied by EDRS are the best that can be made from the original document. ********************************************************************** U S DEPARTMENT OF EDUCATION 011.:e 01 Eu.,c.0,oniv Hosea,. ard "PERMISSION TO EDUCATIONAL RESOURCES INFORMATION REPRODUCE THIS MATERIAL HAS CENTER (ERIC) BEEN GRANTED BY O This document has been reproduced a.; received horn the person or oiganliation Alice D. Walker originahng it O Minor changes have been made to Improve reproduCt:on quahty Points of mew or opinions stated in this TO THE EDUCATIONAL document do not necessarily represent RESOURCES official OERI position or policy INFORMATION CENTER (ERIC) Formative Evaluation and World- Wide-Web Hypermedia William J. Gibbs He Ping Cheng which information to study and thus can Introduction organize it to match their cognitive learning programs are Hypermedia capabilities (Park, 1991). While hyper- computer-based software systems for media programs provide greater learner organizing and storing information. They control, a well-cited problem with these are structured as an arrangement of nodes environments is that users frequently "get (concepts in text or graphical form) and lost" or disoriented (see Conklin, 1987; links (semantic relationships between Neilson. 1989; Edwards and Hardman concepts), and allow users to obtain 1989). As the number of nodes and the mediated experiences non-sequentially complexity of their linkages the in based on individual need and interest is not hypermedia program increase, it (Kumar, Helgeson & White, 1994). unusual for the users of a complex Users can create unique lesson structures hypermedia program to get lost in the learning individual reflective of middle of the program (Park, 1991). As requirements (Park & Hannafin, 1993). Stanton and Baber (1994) point out, the Early computer-based learning programs phenomenon of getting "lost" can take often forced users to page sequentially several forms: users not being able to find through information one screen at a time or becoming information required from beginning to end. Today's hypertext or being disoriented, losing their way and hypermedia programs, on the other unable to formulate appropriate actions. hand, typically provide clickable objects This problem is fundamental to the design (e.g.. text, pictures, etc.) as a means to It may be of hypermedia programs. related anthor additional access compounded in networked environments For educational purposes. information. such as the World-Wide-Web (WWW these environments have the potential to which have the capability to expand access represent knowledge (Bower & Hilgard. world-wide presenting users with a Accordingly. they may help to 1981). seemingly infinite number of nodes and information encoding and facilitate links. retrieval (Park & Hannafin, 1993) and serve appropriately human more While world-wide hypermedia information processing needs. systems are having tremendous impact on information access and retrieval. their Proponents assert that hypermedia development, it seems, is often influenced programs are ideal for adaptive instruction. more by the technological capabilities of They contend that students can discern f ) 259 4-0 PUT COPY fVAILAMF holistic view (Winograd & Flores, 1986). computers rathei than by visual and Many approaches advocate qualitative principles. Such design interface methodologies (Card, Moran & Newell, environments are frequently designed as 1983) using multiple data collection scrambled, content-free information or instruments (Marchionini, 1990) in an gibberish and more complex systems attempt to develop an understanding of being developed appear to compound this how computer-based systems are used. problem (Stanton & Baber, 1994). The video-split screen (Gibbs & Shapiro, Technologically, developers have the for technique such one 1994) is capability to include pictures, text, audio monitoring the interactions between user and video in documents and to almost and computer. This method video records instantaneously link to sites throughout the subjects' actions and decisions in a world. However, design protocols often program, their verbal commentary and differ from one site to the next which can their observations, and elaboration, confuse users and increase the possibility nonverbal and attitudinal reactions. For of them getting disoriented or "lost". Through a video effects generator, two example, browsers of the World-Wide- images (subject and computer screen) are Web are likely to find many dissimilar integrated so that the subject's behavior home page While designs. "web Netscape, Mosaic) can be observed simultaneously with the (e.g., browsers" computer screen. Subject and computer provide navigational options, home page screen are each simultaneously video- developers often add navigational elements recorded. This technique provides a which confuse and mislead users and are permanent visual and auditory account of not consistent from one site to the next. for subsequent analysis. evaluation Developers should attempt to understand Among other things, this data helps to .,how users browse information and how reactions, needs, use identify users the design of documents influences user patterns, and interactions and portrays, to navigation. some extent, a more fuller representation of them in the learning environment. Purpose of the Paper From it, researchers can potentially design Complex interactive hypermedia more powerful environments. pose significant challenges for those researching its effect on users (Gay & In recent years, there has been is no Mazur, 1993). For example, it in methods that help much interest simple task to design a world-wide access researchers understand cognition and the hypermedia system which all users find interplay between learners and computer- intuitive and easy to use. Understanding based learning environments. With how users interact with these systems is techniques such as the video-split screen, a important because their success is often learner's interactions with a computer can contingent on an appropriate human- be collected using multiple data collection However, while computer interface. instruments and analyzed to obtain a fuller powerful from an information processing awareness of their thought processes. perspective, flexible, non-sequential, user- centered designs make it more complex to This paper discusses a formative examine how users interact with a system evaluation method by which to assess (Gay & Mazur, 1993). Moreover, as the World-Wide-Web documents. Also utilization of media objects such as video, results from a presented are pilot audio, and animation increases, so does which examined evaluation users' the difficulty of designing an effective user interactions with Web documents and the i nterface (Wadlow, 1990, P. 181). effects of providing "document maps" to guide users to specific information. To better understand the human- computer interface, recent formative evaluation approaches have taken a more 260 revisions, discuss misunderstandings or Methods provide any commentary that would A video-split screen technique was contribute to improving the document. used to record users interacting with the They could spend as much time or as little World-Wide-Web and to assess the design time as they wanted on this task. of associated documents. Three female and two male subjects participated in the At the completion of the session, Four of the evaluation sessions. subjects filled out a survey which collected participants were undergraduate students information on their reactions toward the and one was a graduate student. Most of usefulness of the "document maps". The them had used the World-Wide-Web no survey also collected data related to more than three to four times prior to the subjects' computer experience and evaluation sessions. knowledge of the World-Wide-Web. Obser vation Technique Results Subjects were given two tasks: 1) This was a pilot study with a small to browse the World-Wide-Web and locate number of subjects. Given this constraint. specific information/documents and 2) use no attempt is made to generalize the and assess Eastern Illinois University's findings discussed below to other groups. Department of Media Services Web site. yielded Nevertheless, study the On each task, subjects were asked to think informative data worthy of additional aloud. They took as much or as little time Periodically, the needed. research. they as researchers prompted them to speak up or It appears that the video-split tell what they were thinking. screen method can be an effective means to The video assess WWW documents. Task 1 recordings allowed the researchers to On this task, subjects had three visually and aurally reconstruct the actions trials. For each trial they had to find one taken by subjects. This is beneficial for document located within a web site. First, it enables one to several reasons. Printouts of the documents to be found simultaneously monitor, among other were given to the subject. The site chosen things, how users navigate in a system, for this task was the White House; the types of options selected and the order however, many other sites could have in which they are selected. It also shows been substituted for it. where users appear confused or where they get lost, what they like and dislike. and 3, "document On trials 1 and their observations about the program. maps" were provided to assist with Second, the technique allows for learners' These maps information searches. immediate feedback about their interactions consisted of a hierarchical breakdown (in the program. Other forms of with text form only) of topics that directed assessment may not fully reflect a subjects to the document for which they subject's instantaneous processing of were searching. The map was positioned stimuli or the way in which one explores in a window behind the Web page and and links conceptual events presented by would display when clicked with the Third, assessing the hypermedia system. computer's mouse. No map was available a system can be a laborious task when for the second trial. users are asked to record/write their comments and reactions. This method of Task 2 assessment removes the burden of For this task, subjects used Eastern evaluation from subjects and allows them Illinois University's Department of Media to more easily provide feedback. Lastl Services' home page and talked aloud the technique permits the review and Subjects were while interacting with it. validation of observations by several asked to critique the home page, suggest 261 than on the second (no map available). reviewers. Sample transcripts, resulting Three subjects had 4 or more years of from the evaluation sessions, are presented this Despite computer experience. Transcripts made from the in Figure 1. computer familiarity, the search tasks, video recordings provide a useful resource especially trial 2, posed difficulty for for analyzing subject-computer interaction. It should be pointed out, however, them. that experience with computers does not Figure 1 necessarily mean that one will be able to Structured Observation effectively search the Web having little Transcript Sample prior exposure to it. "All this extra space and all these little pictures. Figure 2 They don't really draw your eye ..." Time Spent Searching (By Trials) "I was just kind of wondering where I am. Uh, it looks like I started something major just by all the data going back and forth." Total Time Minutes in Search Trials "I like the purple and blue, the colors are really si bright, they catch your eye, it looks neat. It S5 S4 S2 S'3 seems like it gets kind of repetitive, everywhere you go you end up just seeing the staffs names, 05 09 41 08 07 12 1st the name of the staff personnel." 07 25 71 10 15 14 2nd "I don't know what else is supposed to be on here, the interactive learning, I thought that would be a 02 32 06 03 08 13 3ni little game or something to play -- to learn how to use the computer, but it wasn't." Subjects stated that the "document map" was very useful when seizching for On three occasions, subjects were documents. From observations, it asked to find one document located within appeared that subjects attempted to first the White House WWW site. They were find the documents without the aid of the given printouts of the three documents and map. However, once it was clear that the asked to locate them on the web. On the map provided a visual guide to the to-be- first and third trials a "document map" was found documents, their use increased. provided. No map was available for the Three subjects use of the "document second trial. maps" increased from the first to the third trial. Use was determined by counting the There appeared to be differences in number of times subjects viewed the map the way subjects reacted during search on each trial. On trial 1, subjects 3, 4 and trials depending on the availability of the 5 used the maps a total of three times. "documents map". In the second trial, for Their use of the map increased to a total of example, there was no map and most of 16 times on ,he third trial (see Figure 3). the subjects showed what appeared to be Subjects 1 and 2 used the map less on the impatience soon after their first few third trial than on the first. attempts to find the dccument failed. Three of the subjects showed uneasiness after two minutes. They looked for the information aimlessly, proceeding forward and backward through many screens and quitting the task before finding the document. Figure 2 indicates that most subjects took less time to find documents on the first and third trials (maps available) 262 Figure 3 path is established to reach a particular Number of Times Maps Were Used topic, a simple hierarchical breakdown of the topics will be of much value to (By Trial) learners. It provides them with a guide and appears to prevent them from getting side- tracked by distracting information. Number of Times Map was Used Trial Total From this study, it was observed that subjects need to be shown that a 00 05 02 20 "document map" exists for their use. They 1st 12 01 also must be explicitly informed as to its 03 06 3rd 07 03 30 purpose and function. In some cases, it 11 was not until several failed attempts and much frustration that subjects realized that they had the map. Therefore, for instructional purposes, such resources During trials, subjects initially should be made identifiable and easily for context and seemed to search It would have accessible to learners. informational clues rather than visual clues helped if subjects had a "Read Me First" and some resorted to aimlessly clicking button which informed them about the navigation options only after all other purpose of the map. approaches failed. This has implications for designers in that the informational In this study, a map was provided content presented should be supported by a separate application and was not and/or coincide with visual cues and clickable. Nevertheless, subjects made navigational elements. For example, while repeated attempts to click the words on the certain visual cues attracted attention, map in attempts to go to a destination. subjects also read the text on-screen and When no action resulted from their clicks, from this made inferences as to where they some appeared confused and experienced might find the documents for which they difficulty getting back to the search. Thus, it was not a were searching. Therefore, whenever possible, designers haphazard random process by which should provide navigational elements or subjects searched for information -- the clickable objects on maps so that when meaning of the presented information was users click them, they are taken to the setiously considered and directed their desired location. sea:ch efforts. Summary Discussion This paper has discussed an As the World-Wide-Web is used approach for assessing World-Wide-Web for the delivery of instruction, is it hypermedia systems. Recent assessment important to provide learners with a means approaches have become multifaceted and instructional of support. This is holistic in an attempt to more fully particularly important, given that users understand .)e human-computer interface. frequently experience disorientation within Data collection instruments, such as the hypermedia systems. For instructional one discussed in this paper, have the purposes, learners need information about potential provide direct more to how to find topics and concepts under comprehensive evaluations of complex study. Creating "document maps" for use systems. with the World-Wide-Web can be a time consuming process considering the A current problem with hyper- seemingly infinite number of nodes and media programs is that users frequently links that could be established. However, lost" when using them. Developers from this pilot study its seems even the must consider methods to guide users to simplest of maps are useful. When a finite 263 Edwards, D.M. & Hardman, L. (1989). information and to reduce the potential of Lost in hyperspace: cognitive mapping From the them becoming disoriented. and navigation in a hypertext analysis presented, informational maps environment. In R. McAleese (Ed.), will likely increase the efficiency with Hypertext: Theory into practice. which learners search for information. Oxford: Blackwell Scientific. When designed effectively, they appear to impact on students. have positive Ericsson, K., & Simon, H. (1984). Moreover, these maps can present the Protocol analysis: Verbal reports as structure of to-be-learned information and data. Cambridge, MA: MIT press. thus have positive learning benefits. Gay. G., & Mazur, J. (1993). The Further Research Utility of Computer Tracking Tools for From this pilot, a more in-depth User-Centered Design. Educational study is planned to examine the effects of Technology, 33(4) 45-59. maps on users' browsing behavior and cognitive processing. Using the techniques Gibbs, W.J. & Shapiro, A.F. (1994). described in this paper, it may be possible Video split-screen technology: A data to determine the effects of various types of Journal of collection instrument. maps and WWW interface designs on Computing in Higher Education, 5(2), Our processing. cognitive users' 113-121. knowledge of learners' thought processes various types of computer-based in Hammond, N., & Thompson, R. (1989). learning environments is limited (Nakhleh Extending hypertext for learning: An & Krajcik. 1991). Using techniques like investigation of access and guidance the video-split screen with methods to tools. In A. Sutcliffe & L. McCauley analyze subjects' verbal commentary such (Eds.), People and computers V. as protocol analysis (Ericsson & Simon, Cambridge University Press. 1984) may prove better useful in cognitive learners' understanding Kumar, D.D., Helgeson. S.L., & White, processes while engaged in a hypermedia Computer technology- A.L. (1994). program. From this knowledge, more cognitive psychology interface and effective design may result. assessment. performance science Education Technology Research and References Development, 42(4), 6-16. Bower. G.H., & Hilgard, E.R. (1981). Kumar, D.D., Helgeson. S.L., & White, Theories of learning. Englewood Cliffs, A.L. (1994). Computer technology- NJ: Prentice Hall. cognitive psychology interface and science performance assessment. Card, S., Moran, T., & Newell, A. Education Technology Research and The psychology of human- (1983). Development, 42(4), 6-16. Hilllsdale, NJ: computer interaction. Lawarence Erlbaum Associates. Marchionini, G. (1990). Evaluating hypermedia systems. In Proceedings of Conklin, J. (1987). Hypertext: An the CH '90 Conference on Computer introduction and survey, Computer, and Human Interaction. (pp. 387-390). 20(9), 17-41. April 1990, Seattle. New York: ACM. 264 Nakhleh, M. B., & Krajcik, J. S. (1991). The use of videotape to analyze the correspondence between the verbal commentary of students and their actions when using different levels of instrumentation during laboratory activities. Paper presented at the annual meeting of the National Association for Research in Science Teaching, Lake Geneva, WI. Neilson, J. (1989). The art of navigating through hypertext. Communications of the ACM, 33 (3), 296-310. I., & Hannafin, M. J. (1993). Park, Empirically-based guidelines for the of interactive multimedia. design Education Technology Research and Development, 41(3), 63-85. Park, O.C. (1991). Functional Features and Research Issues. Educational Technology, August, 24-30. Stanton, N. A., & Baber. C. (1994). The myth of navigating in hypertext: How a "Bandwagon" has lost its courseJournal of Educational Multimedia and Hyper- media, 3(3/4), 235-249. Wadlow, M.G. (1990). The andrew system: The roles of human interface guidelines in the design of multimedia applications. Current Psychology: Research & Reviews, 9(2), 181-191. Winograd, T., & Flores, F. (1986). Understanding computer and cognition: A new foundation for design. Reading. MA: Addison-Wesley. 265