Table Of Content

DOCUMENT RESUME IR 056 296 ED 405 867 McClure, Charles R.; Lopata, Cynthia AUTHOR Performance Measures for the Academic Networked TITLE Environment. Department of Education, Washington, DC. SPONS AGENCY PUB DATE [95] R197D40019-94A CONTRACT 15p.; Paper presented at the Northumbria NOTE International Conference on Performance Measurement in Libraries and Information Services (1st, Northumberland, England, August 30-September 4, 1995). Evaluative/Feasibility (142) PUB TYPE Reports Speeches /Conference Papers (150) MF01/PC01 Plus Postage. EDRS PRICE *Computer Networks; Computer Uses in Education; Cost DESCRIPTORS Effectiveness; Educational Planning; Efficiency; *Evaluation Methods; Higher Education; Information Networks; Information Technology; Online Systems; Organizational Effectiveness; *Performance Factors; Resource Allocation; Technological Advancement; User Needs (Information); Users (Information) *Academic Computing; *Impact Evaluation; Performance IDENTIFIERS Indicators; User Characteristics ABSTRACT As more academic institutions spend larger sums of money to network their campuses and provide network connections to sources outside their campuses, questions emerge concerning the selection and configuration of appropriate network technologies and the appropriate types and levels of services to provide. Increasingly, academic administrators are asking questions about the benefits and impacts of networking. This paper reports on a study to develop and operationalize performance measures and indicators of the impacts of networking on the academic institution. The study examines information resources and services provided, organizational structures within which they are provided, the various classes of users, and users' activities on the network. Measurement and evaluation of networked information services is essential for administrators to justify such services and better meet user information needs. Approaches for evaluating networked information (1) extensiveness--how much of the service has services are based on: (2) efficiency--the use of resources in providing or been provided; accessing networked information services; (3) effectiveness--how well the networked information service met the objectives of the provider or user; and (4) impact--how a service made a difference in some other activity or situation. Performance measures and evaluation; key issues; attitudes, problems, and perceptions; possible performance measures; study progress; and the importance of the project are described. A brief discussion with the author is provided. (Contains 16 references.) (Author/SWC) U.S. DEPARTMENT OF EDUCATION Office of Educational Research and Improvement EDUCATIONAL RESOURCES INFORMATION CENTER (ERIC) This document has been reproduced as received from the person or organization originating it. Minor changes have been made to improve reproduction quality. Points of view or opinions stated in this document do not necessarily represent official OERI position or policy. Performance Measures for the Academic Networked Environment by Charles R. McClure and Cynthia Lopata "PERMISSION TO REPRODUCE THIS MATERIAL HAS BEEN GRANTED BY Charles R. McClure BEST COPY AVAILABLE TO THE EDUCATIONAL RESOURCES INFORMATION CENTER (ERIC)." 2 KEYNOTE PAPER Performance Measures for the Academic Networked Environment Notes 1, 2 Dr. Charles R. McClure, Distinguished Professor Dr. Cynthia Lopata, Assistant Professor School of Information Studies, Syracuse University, USA ments) also comprise the academic networked envi- Abstract ronment. Both the networking of users and A A s more academic institutions spend larger sums resources within the institution as well the connect- money to network their campuses and pro- ing of these users to other persons and resources vide network connections to sources outside their outside of the institution are considered part of this campuses, questions concerning the selection and environment. configuration of appropriate network technologies, The notion of networked information services is and the appropriate types and levels of services to an evolving one. Such services can be offered by provide, are emerging. Increasingly, academic individuals, libraries, computer centers, publishers, administrators are asking questions about the bene- networks, government agencies, or a host of other fits and impacts of networking. organizations and groups with access to the Internet This paper reports on a study in progress and the evolving National Information designed to address these, and other questions relat- Infrastructure (NH) (Huth and Gould, 1994) and the ed to assessing network technologies and services at Global Information Infrastructure (Gil) (Gore, academic institutions. The purpose of this study is 1995). Networked information services comprise to develop performance measures and indicators of bulletin boards; email; list-servs; remote access to the impacts of networking on the academic institu- distant databases, software, and high speed comput- tion. Key components of the academic networked ing; and collaborative efforts among geographically environment will be defined and performance mea- dispersed individuals - to name but a few. A key sures of networked information technologies and aspect of 'networked information services' is that services at academic institutions will be developed there are numerous providers - local and remote; and operationalized. there are a range of electronic information services The paper provides an overview of the study's available to users; and access to and use of these progress to date, identifies and discusses key issues services continues to increase. and preliminary findings that affect successful eval- Despite the fact that many institutions of higher uation of networked services and the development education have built significant networks and are of performance measures, and describes an academ- connected to the Internet and the evolving Gil and ic networked environment performance measures NH, there is little knowledge of how such connec- manual that is currently under development. tivity has affected the academic institution. Thus, some key questions are: Introduction - How much networking activity and of what types are taking place on a particular academic The notion of the 'academic networked environ- campus? ment' encompasses a range of campus electron- - What types of users access the academic net ic networked activities and services. Minimally, the work and to what types of services and activities academic networked environment includes informa- do they connect? tion services, products, hardware and software, and - What are the costs for an academic network and resources which are received by campus users via various types of network activities and services? electronic networks. In this environment, information services are provided by regional and national - How has access to and use of networked infor- networks, although locally developed information mation resources and services affected teaching, services (ie. from the library, computing services, research, learning, service, and other indicators administration, individuals, or academic depart- of traditional academic performance? 63 3 McClure and Lopata: The Academic Networked environment Self-assessment for campus information technology To date, there has been little practical guidance services (Fleit, 1994) as well as Evaluation guide- offered to assess the impact of networking on these lines for institutional information resources traditional areas of academic institutional perfor- 1995). Those in the process of assessing mance. Moreover, performance measures related to (CAUSE, the academic networked environment may wish to network use by specific audiences within the institu- review the self-assessment technique and the guide- tion such as faculty, administrators, librarians, stu- lines developed by CAUSE. dents, and staff, are only now being developed. As described above, this is an exploratory study As nonprofit organizations implement new infor- in progress and as such it is based on two broad mation technologies, they are beginning to call for research questions: evaluation methods and measures to demonstrate that the resources invested in the new technologies - What is the academic networked environment? have had some positive impact on their organiza- - What performance measures can be developed tions, the services they provide, and the users they and tested to assess this academic networked serve. Traditional economic models that evaluate environment? the impacts of information technology in terms of In answering the first question, the study examines an organization's bottom line are neither appropriate information resources and services provided, orga- for nonprofit, service organizations, nor have they nizational structures within which they are provid- been used with much success in for-profit organiza- ed, the various classes of users involved, and users' tions (Brynjolfsson, 1993; Computer Science and activities on the network. To answer the second Telecommunications Board, 1994). question the study reviewed existing measures used There is evidence that a restructuring of the com- at individual institutions and is developing and test- puting and communications infrastructure as a result ing new measures of academic networking effec- of the availability and use of electronic information tiveness, efficiency, and extensiveness, as well as is occurring and that this will have a fundamental impact. A core set of possible performance mea- impact on educational institutions. Already, this sures as well as other evaluation techniques will be restructuring is affecting the communication cus- developed in a manual to assist those engaged in toms and expectations of researchers in a variety of assessing academic networked environments. fields. In a larger sense, this restructuring is affect- Because of the exploratory nature of this investi- ing the entire information transfer cycle from the gation, an inductive approach, using a variety of creation, structuring, and representation of informa- qualitative methods, is being taken. Among the tion to its dissemination and use by the members of methods being used are: focus groups, case studies, academic communities (McClure, 1993). site visits, and interviews. Individuals involved in A number of writers have attempted to describe the design, implementation, support, and use of net- the evolving academic networked environment and worked resources and services provide on-going consider possible problems facing its development feedback and comments on the study via an elec- (Lynch, 1991; Drake, 1993). Recent reports issued tronic discussion list (see Appendix A). Individuals by the Corporation for Public Broadcasting interested in learning more about the project and (1994) and the American Council on Education receiving updated project information are welcome (1995) summarize current developments on uses and applications of information and networking to join the list. technologies on campuses in the USA. But, to date, Performance Measures and few formal efforts have been made to develop techniques to produce performance measures and assess Evaluation the impact of networked information services on performance measures represent a broad manage- such an environment. r rial/evaluation concept that encompasses mea- Traditional criteria used in assessing information surement of inputs (indicators of the resources services may serve as a beginning model for net- essential to provide a service), outputs (indicators of worked information services assessment. For exam- the services resulting from the use of those ple, traditional performance indicators typically resources), and impacts (the effect of these out- examine a service, activity, or product in terms of: comes on other variables or factors). They are an extensiveness, effectiveness, efficiency, and impact essential means to assess the academic networked (McClure, 1991). In addition, CAUSE, an associa- environment. Performance measures serve a number tion for managing and using information resources of useful purposes. They can: in higher education, published an excellent tool, 64 4 McClure and Lopata: The Academic Networked Environment required to successfully telnet to a remote data- of the network that are - Identify those aspects base successful versus those aspects that are less successful - Effectiveness: how well the networked information service met the objectives of the provider or in the net- - Provide trend-data to assess changes of the user, eg. success rate of identifying and work and network services over time accessing the information needed by the user allocate or reallocate - Assist decision makers to - Impact: how a service made a difference in resources and to plan for future network devel- some other activity or situation, eg. the degree to opment which faculty network users increased their - Monitor network activities and services to research productivity or teaching effectiveness inform managers of any changes in activities or by use of networked information services the quality of services Although evaluations of networked information ser- satisfied - Determine the degree to which users are vices need to consider extensiveness and efficiency with the network and network services criteria, much more attention needs to be given to justify expenditures - Assist network managers to effectiveness and impact measures. As will be dis- expenditures and be accountable for those, cussed later in this paper, however, developing mea- Simply stated, performance measures ask decision- sures of impacts from networked services remains a makers to answer the question: How well is the ser- very difficult task. vice or activity doing what it claims to be doing? Because networked information services are Performance measures also assist managers to multi-dimensional, the type of evaluation needed formally evaluate the network. Thus, evaluation is typically will be multi-dimensional. A single mea- the process of identifying and collecting data about sure provides only one 'snapshot' of a particular specific services or activities, establishing criteria to service; multiple 'snapshots' from different mea- assess their success, and determining the degree to sures are needed. Moreover, evaluators of net- which the service or activity accomplishes stated worked information services will need to know objectives. As such, evaluation is a decision-making what type of evaluation approach and data collec- tool intended primarily to assist decision-makers tion techniques will be appropriate for what types of allocate resources that best accomplish organiza- services (McClure, 1994). An important point, tional goals. Evaluation reflects value judgements however, is that researchers need tb develop evalua- on the part of the evaluator regarding the adequacy, tion strategies that are user-based, that is, they appropriateness, and success of a particular service examine networked information services from the or activity. point-of-view of the user. In a broader organizational context, measurement Providers of networked information services must and evaluation of networked information services not accept as a 'given' that their services, resources, are essential for resource allocation, planning, and and technical procedures are efficient and effective; improving services. Without measures that can eval- rather, they must test their assumptions about the uate particular services, decision-makers must rely quality of networked information services through on intuition and anecdotal information as a basis for an ongoing process of evaluation. Ongoing evalua- assessing the usefulness and value of a particular tion activities are essential to support the provider's service. Perhaps most importantly, measurement planning process. Planning and evaluation are and evaluation provide feedback for users to make two sides of the same coin. Each will be more suc- known how well those services meet their needs. cessful when the other is part of the overall services Approaches for evaluating networked informa- design and implementation approach. tion services can be based on the following criteria: Developing, operationalizing, and validating a range of performance measures that encourages an the service has - Extensiveness: how much of academic institution to assess what types of net- been provided, eg. number of users logging-in worked information services have what level of per week on a bulletin board, or the number of quality, have what impacts on the educational participants of a particular list-sew process, and have what costs is essential if adminis- in providing or - Efficiency: the use of resources trators of networked information in the academic accessing networked information services, eg. setting are to justify such services and better meet cost per session in providing access to remote user information needs. users of an online catalog, or average time 65 5 McClure and Lopata: The Academic Networked Environment BARRIERS TO THE DEVELOPMENT OF NETWORK Selected Key Issues TECHNOLOGIES AND SERVICES this writing, a number of site visits, focus At In spite of the increasing demand for networking groups, and small group interviews have been and growing recognition on the part of administra- conducted at professional conferences and in select- tors of the importance of networking, a number of ed academic institutions. Participants were academ- barriers to the growth and development of network- ic computing professionals, librarians, and others ing exist. Among these are problems associated from a variety of institutions. The primary intent in with network technologies and pedagogical limita- conducting data collection activities was to inform tions. the study team's understanding of the research ques- A major challenge to network managers is . . . tions, the state of modeling and evaluation of acad- getting our systems to be easy to use. They're still emic networking in practice, and to obtain assess- not good enough. They're not like dialing a tele- ments of draft performance measures under devel- the systems are not phone to get what you need . . . opment by the study team. The key issues which intuitive and easy to use. And there are a trillion dif- emerged from data collection activities to date ferent kinds of systems and almost a trillion differ- include the following. ent interfaces to access them . . . Another aspect of the network technology which DRIVERS OF THE DEVELOPMENT OF NETWORK stands in the way of development is its distributed TECHNOLOGIES AND SERVICES nature and the problems that creates. 'Anyone can Participants identified changes in users' expecta- get an Internet address and hang a server on the net- tions and experiences, changes in technology, and work. There are a whole lot of issues that the main- changes in the nature of educational processes, and frame administrator used to handle. It's now distrib- institutional support for those processes, as drivers uted all over the network. And if they don't manage of the development of networking on their campus- whole network is exposed.' it properly . your . . es. This distributed environment presents an image, Both faculty and students now seem to expect oftentimes, that no one is really in charge or in con- that network access should, and will, be available at trol of networking developments on campus - these all times, and from a variety of locations. Such developments just 'happen'. expectations have been instrumental in causing net- Yet another barrier to the development of network providers to accelerate their planning and working is the lack of appropriate pedagogical mod- implementation schedules. This has been particular- els to take advantage of the technology. 'People are ly true in the case of incoming freshmen. One acad- of . most fundamentally automating old things . . emic administrator commented: our professors haven't really internalized how to use `Seeing a whole new crop of freshmen come in, the technology to really change the way they con- computer literate in a way that we had never seen duct their classes.' The full advantage of network- all of a sudden this class came in and before ing may not be realized until new pedagogical mod- . . . said, "This is our god given right and why isn't els are developed. By this, the authors mean that it there a connection in every dorm room?" We had may be too early to measure impacts of networking a plan to have that in a year and a half, and we . on the academic institution since traditional models . . have just spent the last two months wildly coming of teaching and learning are still in use. New mod- up with a plan to make sure that we got it by els for teaching and learning that exploit the September, 1995, because student demand is networked environment are still being developed. there.' As one interviewee commented, 'We are all still floundering a bit as to how best to use and apply In addition, there is a recognition among university networking services.' administrators that a network is essential in order to A final issue which was identified during the data attract faculty. As one participant explained, quoting collection was that of measures of the impacts of a university provost, 'The world is now very differ- networking. Participants discussed financial mea- ent and every faculty person we recruit needs a sures and impact measures and agreed that, while dowry, needs to understand what kind of worksta- traditional measures of technology impacts are often tion they are going to have on their desk, what kind inappropriate, new measures are yet to be devel- of networking connection.' oped. 'All of the traditional models, all of the accounting models, just don't apply any more . . . 66 McClure and Lopata: The Academic Networked Environment these impacts remains very difficult. 'We are at a [we need] to develop new models, and these models very immature stage where we really, I think, are are going to be squishier.' only getting glimpses of what the future is going to There was also some suggestion that academic hold. And so, it's going to be very hard, I'd say, to administrators don't ask for economic justification measure things, because it's very foggy just where of requests for investment in new technologies, or in all this is going to go.' This difficulty in identifying other improvements to the institution. 'Colleges and and measuring 'impacts' from networking is the universities don't make these decisions based on result of an exceedingly complex distributed direct economic issues. They don't ask what is the networking environment, a rapidly changing net- they don't ask that with respect to cost benefit . . . working infrastructure, and the lack of conceptual anything they do. I mean there is no bottom line, tools to describe this environment. there isn't.' THE NEED FOR MEASURES AND ASSESSMENT Attitudes, Problems, and Perceptions Moving beyond financial measures, participants dis- ased on the various site visits and other data cussed a developing interest in measures of the B collection activities, a number of similar views impacts of networking on teaching and learning. Administrators are beginning to ask questions like, and attitudes toward evaluation and the development of perfonnance measures is evolving. `Has it enabled an instructor to increase the contents Understanding these attitudes and the 'evaluation or broaden the contents or get deeper in the content culture' at an academic institution is important as of the class?' and 'Has it reduced their administra- they will affect the degree to which successful tive work in administering the class?' ongoing evaluation and use of perfonnance mea- In response to such questions, network administrators are beginning to develop and apply new measures can be implemented. sures. For example, 'We provided multimedia sup- EVALUATION OF NETWORK ACTIVITIES AND port for classrooms where we have evidence that the SERVICES IS A 'GOOD THING' BUT RARELY. DONE faculty member is spending less time on the In general, participants agreed with this view. But mechanics and more on the content. So that the stu- they also agreed that none of them conducts evalua- dents are getting more content and learning it tions regularly. There was an underlying assumption faster.' However, most evidence of networking's that the network is a good thing and that the need impacts on teaching and learning is anecdotal. for it is essential and growing. Therefore evalua- For example, one faculty member commented, tions to determine what's wrong with the network or 'We've got an architecture class, and we're doing whether the network is necessary are not needed. shared design projects with students in Norway.' `It's not like I'm going to come out with an earth- Another faculty member stated 'I have a small class shattering study that's going to prove technology is and there's another fellow in Nebraska with a worthless and we're all going to go back to books. small class, and we are collaborating using the It's not going to happen.' Internet. There are all kinds of things like that that you can point to that you can say that those are INEQUALITIES OF COMPUTING RESOURCES things that could not have happened any other way. A network administrator described his university as So those are tangible outcomes but you can't a 'very, very heterogeneous environment. Some col- measure them.' lege units are relatively resource rich some are rela- Another problem identified in trying to create tively resource poor and it's got more to do with the measures of the impacts of networking on activities historical situation than with anything that's evolved like teaching and learning is the lack of good mea- because of the structural needs of the information sures of these activities, regardless of networking, technology. Addressing that imbalance is going to and the lack of existing data on teaching and learn- be one of the immediate items on the agenda over ing in a non-networked environment at some institu- the next few years, addressing it in some formal, tions. 'We never really measured these outcome and addressing where the line is systematic way . measures or evaluated the quality of instruction or . . between central and distributed support.' He recog- learning or anything anyway. So now we are asking nized that having some type of performance mea- how has this proved something that we never measures could assist them to deal with this issue, and sured anyway.' over time, determine the degree to which 'progress' Although there are examples of the impacts of in equalizing resources had been made. networking, as described above, measurement of 67 7 McClure and Lopata: The Academic Networked Environment Individuals charged with providing network NON-SYSTEMATIC COLLECTION OF access to a growing. and an increasingly NETWORKING DATA demanding. group of users may not have the There was evidence that some data on network per- time and resources to conduct evaluations. formance are being collected, by different units Their focus is on keeping the network running within the institution and by different people, but and meeting users' demands for speed and there is little evidence that the data are being col- power. As one administrator described it, 'Life lected and analyzed in a systematic way or that they is very simple for me as a network planner. I are being used in planning and decision-making need to keep figuring out how to get the best, related to networking development. Interviewees biggest, fastest, cheapest pipe in here possible agreed that having a central MIS that identified, on the presumption that the need for bandwidth collected, organized, analyzed, and reported select- is going to get bigger and bigger and bigger . . ed networking statistics would be an important step forward and was essential to be able to better plan The lack of support (in the form of additional for networking services. They also noted a range of resources. a GA to do data collection. etc.) to problems and issues that would have to be resolved conduct such evaluations and for some. limited if such an MIS were to be established at this partic- knowledge on how to conduct such evaluations is also a barrier. Evaluation is 'just another ular institution. thing to do' in addition to a range of other REACHING AGREEMENT OF NETWORKING responsibilities and no additional institutional TERMS AND EVALUATION PROCEDURES support seems to be provided for such evalua- Some interesting discussions occurred regarding the tion. One person indicated that before he would use of different procedures and different terms for feel `comfortable' doing such evaluation, addi- the evaluation that had been done at some of these tional training would be needed. sites. There was little confidence that central com- The lack of incentives to conduct such evalua- puting services could obtain agreement from other tions. While generally agreeing that evaluation stakeholders on campus as to the appropriate defini- was a good thing, they also noted that there tions to be used in a larger institutional effort on were few tangible and direct incentives for con- evaluation. One person commented, 'it would take ducting such evaluations. As one person com- an act of God to reach such agreement.' Yet, it was mented, there was not an institutional mindset recognized that until campus-wide agreement on supporting ongoing evaluations. how best to define key terms occurred, development A lack of faith in the utility and applicability of of performance measures would be impossible. evaluation results. There is an underlying sense BARRIERS TO ONGOING NETWORKING in many academic institutions that ongoing EVALUATION evaluation of services and activities does not Participants offered a number of reasons for the lack produce useful results or offer specific recom- of systematic data collection and low priority placed mendations for how to improve networked on evaluation activities. information services. Further, there is oftentimes no tradition or culture of ongoing evalua- the goals - Some interviewees were suspicious of tion in the campus setting. of evaluation. When asked what his response would be if his director asked for this type of data, - The oftentimes confusing distribution of tech- one person said, 'My first question would be, nologies and services, and responsibilities for "What are you going to do with it?"' The fear of managing those technologies and services, com- how evaluation results might be used prompted pound the problems associated with evaluation. some to not want to know how well or poorly a It is not always clear who is responsible for service was provided. what. A number of the participants commented that they were unsure who did what in terms of uni- - There are 'power pockets' throughout the networking or were unsure who should be con- versity and a grossly unequal distribution of tacted to solve a particular networking problem. resources. Thus some groups have a vested Similarly, when a university provides network interest in not sharing information about the access to information services and resources extent of their resources lest they be pressured such as access to remote produced by others to share those resources. databases - it is unclear what exactly is being 68 McClure and Lopata: The Academic Networked Environment ronment can be developed. evaluated, services and resources provided by Although the manual describes standardized pro- the university or by the remote site or by the cedures for data collection and computing perfor- network providers? mance measures, the resulting measures are unlikely in defining - There are considerable difficulties to be comparable across different institutions of key networking terms and services in such A higher education. The networking infrastructure and way that they can be operationalized for mea- the manner in which data are available in different surement. Thus, before evaluation can occur, the institutions vary considerably. Furthermore, differ- institution may first have to reach agreement on ent institutions may use different definitions how to operationalize key networking activities for key terms. While these concerns will not hinder for measurement and clarify policies related to the use of these measures in one particular institu- networked information services. tion, they will limit the degree to which measures change in network- - The recent rapid growth and can be compared to results at other institutions. ing makes evaluation and planning extremely To some degree, users of the manual may have to difficult. It is not always possible to predict the develop policies and define data collection activities next direction in the technology. I remember within a range of organizational and network con- when someone said, 'Who needs a laserwriter? straints. Indeed, some institutions may not currently What are you going to do with that?' And desk- have the capacity to collect the data needed for top publishing. Nobody could have predicted some of these perfonnance measures. In such this stuff. About the time staff get 'geared up' to instances, the academic institution will need to first evaluate a particular service, it is no longer pro- determine how best the data can be collected, devel- vided or it is out-of-date. op a system or approach for collecting and analyz- ing that data, and develop policies that formalize a admin- - Networking infrastructures. services, and management information system to insure that the istrative organization for networking change data continues to be collected in a regular and stan- rapidly. Evaluation is difficult in such a volatile dardized fashion. environment. For example, on the day that the The research project revealed a number of differ- study team conducted interviews at one site ent views and experiences regarding which types of visit, the Vice President for Computing performance measures might be most useful given announced a re-organization of the management an institution's particular situation. Thus, the responsibilities for computing services on cam- approach taken in the manual is to identify and pus. describe a core set of measures. Depending on the In summary, interviewees agreed they would evalu- nature of the network, the administrative concerns ate network performance if: someone 'ordered them regarding that network, and networking/institutional to do so'; they believed that the results would bring goals and objectives, some of the following mea- them additional resources; they could expect to sures may be more useful for some institutions than receive additional personnel to conduct evaluations; others. they had training in evaluation methods; and if they The scores that result from these performance had the time to conduct evaluations. As one inter- measures take on greater usefulness when consid- viewee said, 'in an ideal world yes, we would have ered in the broader context of: ongoing evaluation; but the reality is we can't and objectives - Institutional and networking goals and don't.' at that particular institution Possible Performance Measures - Other performance measures of institutional rr activities, services, and participants he procedures and measures being developed Various time periods and the amount of change 1 for the manual are based on a research effort on this particular measure over time that obtained information, assessments, and input The amount of resources and the allocation of from a number of site visits and individuals knowl- those resources for networking infrastructure edgeable about academic computing and networking. A key finding from this work is the limited and services knowledge and use of performance measures in the Factors related to a particular institution, its net- academic environment. The complexity of the acad- working configuration, or other variables unique emic networked environment imposes some limita- to that institution. tions on the degree to which measures of this envi- 69 McClure and Lopata: The Academic Networked Environment that are - Services: the applications and services In short, value judgements as to whether a score on made available over the network a performance measure is 'good' or 'bad' are dependent on a range of other factors and should Online Public Access Catalog Measures not be considered in isolation of those factors. Number of users using the online library Finally, it should be noted that the accuracy of the catalog actual measures as computed by institutional offi- Number of campus logins to the online cials will be directly related to the quality of the library catalog data they collect, the use of standardized procedures, and perhaps, the development of institutional Number of off -campus logins to the online policies that define these data collection activities. library catalog To some degree, these performance measures might Number of logins to the online library cata- be best seen as estimates of the extensiveness, effi- log per user ciency, effectiveness, or impact of a service or activity rather than a precise measure of that partic- Cost per user to access the online library ular service or activity. Even if these measures are catalog best seen as estimates, such estimates are a signifi- User satisfaction with the networked online cant improvement over the very limited set of per- library catalog formance measures that are currently available and Distance Learning being used. The performance measures are organized in the Number of faculty offering distance learn- manual by key areas of assessment. And within ing courses each area the following measures are currently Number of student enrolled in distance being developed. learning classes as a percentage of all and types of users and the classes offered - Users: the number frequency with which they use the campus net- Distance learning courses as a percentage of work all courses offered - Count of Network Users Cost per distance learning course by type of user Technology involved in distance learning Network Users - Count of Active Student satisfaction with distance learning by type of user Support: the types of assistance that network officials make available to the users of the net- of financial resources - Costs: the total and types work that are expended to operate the academic network - Help Desk Technology Expenditures - Annual Information Response Time expenditures per - Information Technology Accuracy of Response capita Courtesy of Staff and types of traffic - Network traffic: the amount Additional measures are also under consideration flowing over the academic network for inclusion in the manual. Space does not permit a Measure of Overall - Router Traffic as a detailed description of these proposed measures, Campus Network Activity how they have been operationalized, and procedures for data collection and analysis. The draft perfor- into the Campus Network - Modem Traffic mance measures manual contains such information. (into and out of the campus) - Internet Traffic In developing these measures the study team of uses made of the - Use: the amount and types found that oftentimes the academic institution network would first have to deal with and resolve a range of issues before the performance measures could be Use - Frequency of Network computed. For example, the measures 'count of Percentage of Very Active Network Users network users' (CNU), ie. the number of identified Percentage of Inactive Network Users email accounts with access to the campus network, and 'count of active network users' (CANU), ie. the I 0 70