ERIC ED473390: Program Assessment in an Undergraduate Information Systems Program: Prospects for Curricular and Programmatic Enhancement.

16 Pages·2000·0.25 MB·English

by  ERIC

DOCUMENT RESUME
ED 473 390
IR 058 620
Pick, James B.; Kim, Jeff
AUTHOR
Program Assessment in an Undergraduate Information Systems
TITLE
Program: Prospects for Curricular and Programmatic
Enhancement.
PUB DATE 2000-00-00
15p.; In: Proceedings of the International Academy for
NOTE
Information Management Annual Conference (15th, Brisbane,
Australia, December 6-10, 2000); see IR 058 611.
Reports
Speeches/Meeting Papers (150)
Research (143)
PUB TYPE
EDRS Price MF01/PC01 Plus Postage.
EDRS PRICE
Curriculum Development; *Educational Assessment; *Evaluation
DESCRIPTORS
Methods; Higher Education; *Information Science Education;
*Information Systems; Models; Program Effectiveness; *Program
Evaluation; Undergraduate Study
*University of Redlands CA
IDENTIFIERS
ABSTRACT
This paper offers an assessment model approach for program
assessment of Information Systems (IS) undergraduate programs. The paper
presents the case example of results of program assessment for the B.S. in
Information Systems at University of Redlands (UR). It demonstrates the
usefulness of the approach to understanding the overall success of an IS
degree program as well as details of program elements. The example points to
many ways that the outcomes of assessment can be utilized for IS program
improvement. Four tables include: knowledge attainment goals of BSIS major
program; curriculum of bachelor of science in information systems (BSIS);
correlations of important knowledge/competency areas for current BSIS; and
correlations of career impact on alumni for important BSIS curricular areas.
Also includes two figures. (Contains 18 references.)
(AEF)
Reproductions supplied by EDRS are the best that can be made
from the original document.
PROGRAM ASSESSMENT
IN AN UNDERGRADUATE
INFORMATION SYSTEMS PROGRAM:
PROSPECTS FOR CURRICULAR AND
PROGRAMMATIC ENHANCEMENT
James B. Pick
University of Redlands
U.S. DEPARTMENT OF EDUCATION
Office of Educational Research and Improvement
PERMISSION TO REPRODUCE AND
EDUCATIONAL RESOURCES INFORMATION
DISSEMINATE THIS MATERIAL HAS
CENTER (ERIC)
le This document has been reproduced
BEEN GRANTED BY
as
received from the person or organization
originating it.
T. Case
Minor changes have been made to
Jeff Kim
improve reproduction quality.
University of Redlands
Points of view or opinions stated in this
TO THE EDUCATIONAL RESOURCES
document do not necessarily represent
INFORMATION CENTER (ERIC)
official OERI position or policy.
ABSTRACT
It presents an
This paper examines the assessment of an undergraduate program in Information Systems.
The data from program
approach to program assessment that has proven beneficial to one institution.
assessment can be useful to stakeholders and forms a baseline for program improvement.
presents the case example of results of program
INTRODUCTION
assessment for the B.S. in Information Systems at
It demonstrates the
University of Redlands (UR).
A major trend in U.S. higher education of the last ten
usefulness of the approach to understanding the overall
assessment.
towards
educational
years
been
has
success of an IS degree program as well as details of
Universities, accrediting organizations, professional
program elements. The example points to many ways
academic associations, and government review panels
that the outcomes of assessment can be utilized for IS
assessment
demanded
often
requested
and
have
program improvement.
information from
departments,
programs,
schools,
One example is the
projects, and individual faculty.
The B.S. in IS at UR is a 14 year old program that had
AACSB accreditation standards for business schools,
that a variety of
undergone three major curricula upgrades, with a
for accreditation
which require
fourth one planned in the fall of 2000.
In the field of
The prior
assessment processes be in place.
curricula changes were effected in the years 1986,
information systems, not much has been reported in
on program
1992, and 1997. The curricula were based on national
yet
assessment,
literature
the
it
is
model curricula, including the DPMA 90 curriculum,
important because IS programs are growing in size and
was available
97 curriculum
(it
in
the
their success needs to be monitored and informed steps
and
IS
preliminary form two years prior to its final release in
taken to improve the curricula.
1997.) The Redlands B.S. in IS is a degree completion
junior
senior
only
and
This paper offers an assessment model approach for
includes
that
program
undergraduate years, with the first two years of work
program assessment of IS undergraduate programs.
It
00
O
Proceedings of the 15th Annual Conference of the International Academy for Information Management
193
).1
2
BEST COPY AVAILABLE
experience and jobs, a more condensed curricular
done at another university or college. The Redlands 97
introduction was felt to be appropriate. The IS major
curriculum's knowledge attainment goals are shown in
is distinguished by two required courses that are not
Table 1. The Redlands 97 IS curriculum consists of the
standard in IS '97: a course in geographic information
IS major consisting of nine IS courses, as well as one
systems (GIS) and a course in computer ethics. Both
humanities course, one upper division mathematics
of these courses relate to strengths of the university: it
course, and four business courses (See Table 2). The
nine IS courses encompass nearly all of the IS '97
is known for its strong GIS faculty and it has a liberal
exception of several
model curriculum, with the
arts foundation that ethics relates to.
However, since the students
introductory courses.
enter as working adults, many with IS industry
TABLE 1
KNOWLEDGE ATTAINMENT GOALS OF BSIS
MAJOR PROGRAM
Understand an
structured programming
apply
techniques
conversant
fundamental language
C
in
Be
capabilities
Understand the Graphical User Interface (GUI)
principles
Be conversant
Basic
fundamental
Visual
in
language capabilities
Understand the basic terms and components of the
relational database model
apply
database
able
Understand
and
be to
application development
Access
using
in
conjunction with the VisualBasic programming
language
Understand and apply the fundamental concepts
and terminology associated with networking and
telecommunications
Understand and be able to build internet and web-
based applications
of
appreciate
the structure
design
and
To
geographic
systems
information
as
a
decision_making tool
To demonstrate knowledge of decision support
executive systems,
systems, information
and
expert systems including
some hands-on
applications
To understand how the decision making process
can be enhanced by modeling
of
appreciate
practical benefits
the
To
management support systems
business
real
in
settings and in society
To obtain skills in spatial analysis
To become knowledgeable of the ArcView GIS
Be able to implement
software applications.
hands-on basic GIS functions using ArcView
To appreciate how decision_makers in the public
and private sectors can better assess and make
Proceedings of the 15th Annual Conference of the International Academy for Information Management
194
3
decisions on managerial problems through the use
of GIS
of
principles
and
concepts
the
Understand
structured and object oriented methodologies in
systems development
Be able to apply effective system modeling tools
and skills including CASE
Understand the perspective of complete life cycle
activities, their value and application
systems
the
information
from
Understand
perspective of pieces of an organizational system
of
technological
magnitude
the
Understand
advances and their implications to ethical issues
Understand the theoretical perspective of ethical
relate
and
business
they
to
philosophies
as
organizations
Identify and understand examples of real world
ethical dilemmas in information systems including
the many sides of such dilemmas
Be able to discuss and properly communicate the
basic concepts of information systems
Be able to design and build a business information
system
Be able to develop and implement software in a
business project solution
project
systems
complete
Communicate
a
including verbally presenting and writing up and
documenting the project
Proceedings of the 15th Annual Conference of the International Academy for Information Management
195
ti
TABLE 2
CURRICULUM OF BACHELOR OF SCIENCE
IN INFORMATION SYSTEMS (BSIS)
Information
with
Productivity
ISYS 312
Technology
of
Foundations
MGMT 310
Philosophical
Management
of
Fundamentals
Structured
ISYS 219
Programming with C
Managing
and Leading
MGMT 330
Organizations
Programming
ISYS 317
Introduction
to
Object-Oriented
Techniques:
Programming
Applications
MTHW 303 Mathematical
for Information Systems
Political and Business Economics
BUAD 337
Database Concepts
ISYS 318
Accounting Information Systems
BUAD 469
Decision
Management
ISYS 425
and
Systems
Geographic Information Systems:
ISYS 365
Principles of Marketing
BUAD 340
Software Development
Applied
ISYS 489
Project II
Computer Ethics
ISYS 415
Systems Analysis and Design
ISYS 404
Development
Applied Software
ISYS 488
Project I
After the 'Redlands IS 97 curriculum had been run for
it gathers data from current students, alumni, corporate
two years, the IS faculty initiated a plan of program
It utilizes the methods of
advisors, and IS faculty.
assessment that was carried out in early 2000.
written survey, directed focus groups, and discussion.
The
assessment model consists of the following steps: (1)
All of this information is aggregated together to form
part of the IS program assessment report.
written survey of the 300 current students of the extent
Other
that their knowledge or competency of the program
information required by the university is added to the
objectives and on their suggestions
report including for instance data on the program,
for program
current status of the curricular and support services,
enhancements, (2) written survey of 700 alumni of the
program on the extent that program objectives have
advising, and math and writing skills. Altogether, this
larger amount of data than
is ordinarily collected
helped them in their careers and on their suggestions
provides a more in-depth insight into the program
for program enhancements, (3) focus group with the IS
Degree Program Corporate Advisory Committee on
successes and failures, strengths and weaknesses, and
their recommendations for curricular improvement, (4)
feeds information into design of the future curriculum.
focus groups with samples of current students about
their suggestions for curricular improvements, and (5)
Results of Assessment Procedures
of
faculty
review suggestions
and
current
IS
Program.
curriculum.
The' BSIS program was assessed through
three open-ended questions in two surveys, one of
current students and a second of alumni.
The approach is multi-faceted, i.e.,
A third
it includes many
stakeholders and several methodologies. In particular,
program assessment was conducted through two focus
Proceedings of the 15th Annual Conference of the
International Academy for Information Management
196
5
In the hour and a half
academic support and service elements.
groups with BSIS students.
focus group sessions, the students were first presented
of
Curriculum.
students
current
survey
The
with the results of the BSIS curricular survey. Then
measured Likert scale responses on the extent of
there was a segment of discussion of the curricular
current knowledge of 26 program objectives.
For
Finally, there was a 45 minute open group
results.
N=53 responses, the average current knowledge was
discussion of suggestions for the BSIS program and
2.59 on a 1 to 5 Likert scale, with 5 being the highest
curriculum.
This reflects an intermediate state of current
value.
to be expected for the full
knowledge, which
Only the survey responses responses from current
is
spectrum of students over the two years of the
BSIS students that made specific recommendations are
program. The comparison of students in the first half
included, i.e., a comment such as "great program" is
of the program i.e., junior year to those in the second
not included because it does not inform the assessment
senior year shows an
current
half
average
the
i.e.,
especially
to
pertains
Question
process.
3
knowledge of juniors of 2.45 and of seniors of 2.73,
program. There are several general points coming out
for an average gain of 0.28, or eleven percent. When
of these comments regarding the program. On is that
see the
juniors are compared with students who were seniors
technology
student would
latest
to
like
in the last six months or had just completed the
This implies that in
emphasized in the program.
program, the average gain in current knowledge was
program revision, attention needs to be given to the
These results provide
0.57 or twenty three percent.
most current and important technical skills. Responses
also support review for very recent and current
confirmation of the learning value of the IS major. We
interpret that the gains are not even higher because the
textbooks underpinning the technical skills. There are
student population consists of working adults students,
several comments about workshop/tutorial preparation
most of whom come into the program as IS or
in math and writing skills as important. In general the
technology professionals, so their starting knowledge
administrative mechanics and support services of the
base is higher than for traditional younger resident
program are not commented on, which would imply
students.
that they are working well.
The gains in current knowledge from the program can
The summary of the two BSIS student focus groups
disaggregated by the
sixteen program
further
underscore several points regarding the program. One
be
Comparing juniors to seniors in the last
counseling and
objectives.
advertising
that
pre-entry
the
is
nine months and a few very recent graduates, the
the program
materials need to
position
correctly
In particular, the BSIS program offers a
following are the ranked gains by program objective.
balance.
balanced blend of managerial and technical content.
Current BSIS Students
However, some incoming students have mis-read the
Av. Gain in
program as strictly technical. Students are not clear on
BSIS Program Objective
the level of technical expertise that they will be
Current
will they be intermediate
expected to achieve i.e.,
Knowledge from
Jr. Year to Last 9
programmers or master programmers? The program
Rank
Mos.
needs to articulate its goals early on. One comment in
that the
survey numerical
responses
Access in Development
response
1.06
is
1
indicate a considerable range of incoming technical
Dec. Making with GIS
1.05
2
knowledge/competency in particular between 1
and
Communicate Project
1.03
3
Hence, part of the
3.5 on a
to 5 Likert scale.
0.98
Database Applications
4
1
misunderstanding is that students technical mastery
0.94
Relational Database
5
levels will remain different throughout the program
0.94
Methods of Syst Analysis
6
and what can be standardized is the minimum level of
0.92
CASE & Syst Tools
7
technical knowledge/competency.
0.79
Visual Basic
8
Structured Programming.
0.78
9
Overall, the student suggstions/recommendations are
0.63
Syst Life Cycle
10
centered on the technical content of the program; they
0.57
ArcView Software
11
are seeking expanded technical coverage. They appear
0.56
Internet & Web Applications
12
administration
the
program
and
with
satisfied
197
Proceedings of the 15th Annual Conference of the International Academy for Information Management
6
0.10
Mgt Support Systems
0.54
25
GUI Interface
13
Networking Concepts
0.04
26
0.53
Communic. Basic Concepts
14
0.47
C Language
15
0.46
DSS/EIS
16
The information in the disaggregated table is valuable
0.45
Decision Making
17
in determining where the most learning is taking place
0.43
Design/Build an IS
18
and where learning is not as effective, so future
0.36
Spatial Analysis
19
improvements can be focused in those areas. The
0.35
IS Project Solution
20
results confirm extensive learning in GIS decision
0.33
Ethical Dilemmas
21
making, a new area for most students, in
Organiz. Systems
0.31
22
communication of project,
Ethical Philosophies
0.13
23
0.13
Ethic Imp lic of Techn.
24
dilemmas in the real world.
and in the mainstay areas of data-bases and systems
development. Also, CASE tools and visual basic are
The knowledge/competency results can be grouped by
high. Those are areas in the present curriculum that
major curricular areas. For instance, the Systems
appear to be working well and are likely to be retained
curricular area comprises the knowledge/competency
in the new BSIS 2000 curriculum although
objectives of methods of systems analysis, CASE and
modernized.
systems tools, systems life cycle, designing/building
an information system, and IS project solution. When
On the other hand, areas that are not doing well as
the knowledge/competency of current BSIS students is
measured by the extent of knowledge gain are
grouped by major curricular areas (see Figure 1), the
networking concepts, management support systems,
highest levels of knowledge/competency are in
certain aspects of computer ethics, organizational
communications/organization/ethics, while the lowest
systems, and solutions with IS projects. These are
levels are in DDS/GIS and systems, with technical in
pointing to areas in the BSIS 2000 curriculum in
the middle. This emphasizes that the students have
which to redesign courses and educational strategies
and to seek major teaching and learning
improvements. For instance, the low score for
networking concepts may stem from the lack, until
recently, of a hands-on networking laboratories for
training on NT administration and configuration.
Small practice networks have now been located in the
Jones Center and all the major regional centers. Also,
3.50
the networking course was in need of revision, a task
which has recently been accomplished, but may need
even more revision and attention in the new
curriculum.
There are a number of gender differences in
knowledge/competency by program objectives. The
statistically significant gender differences for
competency/knowledge of current students are for:
structured programming and C language (p=0.05) and
there are lesser differences for networking and ethical
O4'
dilemmas (p=0.10). In all these cases, the knowledge
Important Cunicular Areas
competency of males exceeds that of females. The
programming and networking differences may relate
1034R- STUDENT
to occupational differences prior to entry, while the
AV. GAIN .R. TO SR.
°AV. GAIN R. TO END
difference in knowledge of ethical dilemmas may
ar students" are gapped in the
NOTE
related to differential prior exposure to ethical
range fran 0 to 5, but actualy are in the range from 1 to 5.
Proceedings of the 15th Annual Conference of the International Academy for Information Management
198
BEST COPY AVAILABLE
7
by motivational factors i.e., that students are more
considerable knowledge of the softer sides of the
motivated to learn about the technical and systems
curriculum. Figure 1 also demonstrates a gender gap
areas due for job and career related reasons.
overall and for the four major knowledge/competency
areas. This mostly relate to the prior levels of
E1
experience an
'COMPETENCY OF
entered the BSIS
:RICULAR AREAS,
have had more exp
IMNI
To a less
differences in Mt'
Since tlf gende
differefte bie.ggr
regards tl
reason
-0
3.00
The @in in k
curricular
eas2(Fi
is hiiest in
.
--
lowest in the
i
2.00
organizati4s, and
learning gain of 23
curriculum malts
student entry know
curriculum is pro
areas mtilly. Un th
other hand;
moderate to wea
25
bt.a.Reill ince BSIS Start
the least learning
communications/organization/ethics. This is the area
of strongest initial knowledge. This may be explained
FIGURE 2
OVERALL KNOWLEDGE/COMPETENCY VS.
TIME IN PROGRAM, CURRENT BSIS
STUDENTS
EST COPY AVAILABLE
International Academy for Information Management
Proceedings of the 15th Annual Conference of the
199
S
competency with elapsed time, but not with the other
Correlation analysis for the aggregated knowledge/
areas. This is seen in the scatter plot shown in Figure
competency areas and with elapsed program duration
What is clear in all these areas is that there is a
in months shows signficant correlations among the
2.
large range of incoming competencies; again this
overall knowledge competency and the four important
relates to the population of adult students. For overall
areas (see Tables 3 and 4). This implies that generally
knowledge (Fig. 1) and technical knowledge, there is a
a student is higher or lower in knowledge competency
significant trend of increase over the two years.
Yet,
in all the areas i.e., they vary up and down together
range of
substantial
among individuals. This points to a more generic set
graduation,
there
at
is
still
knowledge/competency. The trend line has moved up,
This may
of knowledge/competency across areas.
This is helpful in
but the high variation remains.
reflect the real world experience of adult students. For
explaining teaching challenges even late in the BSIS
instance, a student who had worked in a systems
program of a large range of competencies in a single
analysis position for ten years before entering the
We interpret this as continuation of
cluster group.
BSIS would have built up knowledge in all of the four
for
contrast,
incoming
major areas, while an entering student without any IS
ranges.
By
job experience would be weak in
the
the technical,
average
communications/organizations/ethics,
incoming knowledge/competency is higher but with an
It is somewhat less clear
systems, and DSS/GIS areas.
even larger range and doesn't increase much with the
weak
would
why he/she
be
in
same large range throughout the program.
communications/organizations/ethics as well, but that
is the case.
The data from the
survey were equally
alumni
The key question related
to program
revealing.
The correlations also show the correlation with elapsed
objectives is different; it asked how well the IS degree
time in the program. There is a statistically significant
had impacted the alumni's career advancement.
The
correlations of overall and technical knowledge/
following were the ranked program objectives (on a 1
to 5 Likert scale, with 5 being the highest impact):
The alumni results point to the importance of the
"softer" sides of the curriculum on career impact
communications, organizations,
areas
such
as
designing and building an IS project, and the "building
block" areas of systems development and database.
The top two objectives are the area of interpersonal
This result corresponds to other
communications.
studies in the IS educational literature that stress the IS
career advances of
Proceedings of the 15th Annual Conference of the International Academy for Information Management
200
9
career attainment of BSIS alumni, so that eleven
and teamwork (see
literature
skills
interpersonal
of the
top management
are
alumni
review in Pick and Schenk, 1993). In general, this
percent
in
(President, VPs) and forty percent are
in middle
literature points to the "softer" areas as ultimately
In middle to top management
management or higher.
having more impact on IS graduates careers in the long
jobs, the areas of communications, teamwork, and
term. It is important to ask why these behavioral skills
organizations are essential.
rise to such importance. One reason may be the rising
TABLE 3
CORRELATIONS OF IMPORTANT
KNOWLEDGE /COMPETENCY AREAS FOR
CURRENT BSIS
Technical DSS/GIS Systems C/O/E
Months
Overall
Months
1
0.309*
Overall
1
0.923**
Technical 0.349**
1
0.863** 0709**
0.273
DSS/GIS
1
0.923** 0.798**
Systems
0.732 **
0.251
1
0.852**
0.771**
0.682**
0.186 0.891*
C/O/E
1
significant at 0.05 level
*
** significant at 0.01 level
TAB
LE 4
CORRELATIONS OF CAREER IMPACT ON
ALUMNI FOR IMPORTANT BSIS
CURRICULAR AREAS
Technical DSS/GIS Systems
C/O/E
Overall
Year
Graduated
Year
1
Graduated
-0.187
Overall
1
0.818**
-0.047
Technical
1
0.436**
0.754**
-0.287*
DSS/GIS
1
0.519**
0.817** 0.579**
-0.235
Systems
1
0.623**
0.560**
0.746** 0.351**
-0.202
C/O/E
1
* significant at 0.05 level
** significant at 0.01 level
Communicate Project
4.08
Alumni
I
Impact of BSIS
Communic. Basic Concepts
4.04
BSIS Program Objective
2
Program
Syst Life Cycle
4.00
3
on
Objective
Alumni's Career
Organiz. Systems
3.98
4
Rank
Advancement
3.97
IS Project Solution
5
201
Proceedings of the 15th Annual Conference of the International Academy for Information Management
10