DTIC ADA423604: Managing the Development of Technology-Based Courses: Success Factors from Eight Government Training Courses

22 Pages·0.19 MB·English

by  Defense Technical Information Center

Acquisition Review Quarterly — Winter 2003
76
Report Documentation Page Form Approved
OMB No. 0704-0188
Public reporting burden for the collection of information is estimated to average 1 hour per response, including the time for reviewing instructions, searching existing data sources, gathering and
maintaining the data needed, and completing and reviewing the collection of information. Send comments regarding this burden estimate or any other aspect of this collection of information,
including suggestions for reducing this burden, to Washington Headquarters Services, Directorate for Information Operations and Reports, 1215 Jefferson Davis Highway, Suite 1204, Arlington
VA 22202-4302. Respondents should be aware that notwithstanding any other provision of law, no person shall be subject to a penalty for failing to comply with a collection of information if it

does not display a currently valid OMB control number.
1. REPORT DATE 2. REPORT TYPE 3. DATES COVERED
2003 N/A -
4. TITLE AND SUBTITLE 5a. CONTRACT NUMBER
Managing the Development of Technology-Based Courses: Success
5b. GRANT NUMBER
Factors From Eight Government Training Courses
5c. PROGRAM ELEMENT NUMBER
6. AUTHOR(S) 5d. PROJECT NUMBER
John /Bennett; Ellen /Bunker; Kurt /Rowley
5e. TASK NUMBER
5f. WORK UNIT NUMBER
7. PERFORMING ORGANIZATION NAME(S) AND ADDRESS(ES) 8. PERFORMING ORGANIZATION
Defense Acquisition University Alumni Association 2550 Huntington Ave, REPORT NUMBER
Suite 202 Alexandria, VA 22303
9. SPONSORING/MONITORING AGENCY NAME(S) AND ADDRESS(ES) 10. SPONSOR/MONITOR’S ACRONYM(S)
11. SPONSOR/MONITOR’S REPORT
NUMBER(S)
12. DISTRIBUTION/AVAILABILITY STATEMENT
Approved for public release, distribution unlimited
13. SUPPLEMENTARY NOTES
14. ABSTRACT
15. SUBJECT TERMS
16. SECURITY CLASSIFICATION OF: 17. LIMITATION OF 18. NUMBER 19a. NAME OF
ABSTRACT OF PAGES RESPONSIBLE PERSON

a. REPORT b. ABSTRACT c. THIS PAGE UU 21
unclassified unclassified unclassified
Standard Form 298 (Rev. 8-98)
Prescribed by ANSI Std Z39-18
Managing the DevelopRmEeSnEt AofR TCeHchnology-Based Courses
MANAGING THE DEVELOPMENT OF
TECHNOLOGY-BASED COURSES
SUCCESS FACTORS FROM EIGHT
GOVERNMENT TRAINING COURSES
John Bennett, Ellen Bunker, and Kurt Rowley
A study was conducted to determine whether success factors identified in
traditional higher education distance learning research literature were important
to technology-based course development efforts at Defense Acquisition
University (DAU). The study included a literature review, a list of candidate
success factors from the literature, data collected through interviews with eight
faculty course development managers, and data analysis to correlate findings
with the research literature. The study indicates that many of the success factors
found in the literature were also important to management of the DAU course
development projects. A number of additional success factors identified were
important for the DAU courses and may be important for other distance education
development environments. Recommendations for development managers of
distance education courses are proposed.
D
efense Acquisition University storylines and hybrid (on-line and
(DAU) is a corporate university classroom) components. These eight
charged with training the Depart- courses were selected because they
ment of Defense (DoD) acquisition are representative of the courses that
workforce. This study investigated suc- won DAU two consecutive U.S. Dis-
cess factors for managing the develop- tance Learning Association (USDLA)
ment of eight technology-based courses Awards for Excellence in Distance
at DAU and identified success factors that Learning Programming for 2000 and
may be relevant to ongoing and future 2001. In addition, they have been suc-
DAU course development efforts. The cessfully deployed, meet or exceed all
courses studied (see Table 1) range from educational and administrative re-
entry-level courses taken on-line with no quirements, and cover the range of in-
required instructor interaction to higher- structional designs and delivery modes
level courses using sophisticated threaded DAU uses in most of its courses.
77
Acquisition Review Quarterly — Winter 2003
Table 1. DAU Course Summaries
Course Students Course Development Training Course
per Year1 Hours Time3 Certification Description
(Online/ Requirements Met
Classroom)2
Fundamentals 10,500/yr 25/0 13 months Level I DAWIA* training • Integrated entry-level
of Systems (Oct 97-Nov 98) certification in Acquisi- course covering eight
Acquisition tion Management. functional career
Management Required course for fields.
(ACQ 101) multiple career fields (cid:127) Lessons and exams
accessed via the
internet.
Intermediate 5,000/yr 40/36 18 months Level II DAWIA* train- (cid:127) Intermediate level
Systems (Jul 99-Mar 01) ing certification in integrated course.
Acquisition Acquisition Manage- (cid:127) Scenario-based
Course ment. Required course hybrid design using
(ACQ 201A/B) for multiple career both internet-based
fields and classroom
(5 days) delivery
modes.
Introduction to 2,000/yr 30/0 18 mos Level I DAWIA* (cid:127) Entry level course
Acquisition (Jan 99- Oct 00) required training with lessons and
Workforce Test certification for Test and exams accessed via
and Evaluation Evaluation career field the internet.
(TST 101)
Basic Software 700/yr 19/0 10 months Does not provide (cid:127) Internet-based
Acquisition DAWIA* training certifi- distance learning
Management cation (course targets course for all levels
(SAM 101) students from all of learners.
DAWIA* career fields (cid:127) Not a certification
and levels) course.
Program 720/year 56/24 3 months for Level II DAWIA* training (cid:127) Unique hybrid design.
Manager’s (Virtual 65% solution, certification in Program (cid:127) Internet based DL
Tools Course Classroom) 7 months for Management career (8 lessons) followed
(PMT 250) 85% solution field by synchronous four-
day virtual classroom
using phone confer-
encing and LMS file
sharing tool (Forum).
1 Students per year figures are approximate.
2 Course hours based on course design estimates.
3 Development time obtained from Course Manager interviews. Generally, from the time the development contract was awarded
to the start of the first production offering.
* Defense Acquisition Workforce Improvement Act (DAWIA)
**Business, Cost Estimating, Funds Management (BCF) functional area
Note: 100 series courses are entry-level, 200 series courses are intermediate level, and 300 series course are advanced level.
(continued)
78
Managing the Development of Technology-Based Courses
Table 1. DAU Course Summaries (continued)
Course Students Course Development Training Course
per Year1 Hours Time3 certification Description
(online/ requirements met
classroom)2
Program 700-1000/yr 50/232 18 months total Level III DAWIA* training (cid:127) Hybrid design.
Management (first offer- (DL fielded in certification in Program (cid:127) Integrated advanced-
Office Course ings started 12 months) Management career level course.
(PMT 352) in June 02) field (cid:127) 10 modules of
internet-based
distance learning.
(cid:127) 12 scenario-based
exercises over six
weeks in the
classroom.
(cid:127) LMS used to access
classroom material
and exams.
Fundamentals 700/yr 60/0 About 20 months Level I DAWIA* training Internet based
of Earned certification in BCF** distance learning
Value career field modules for entry-
Management level instruction in
(BCF 102) Earned Value
Management
Acquisition 170/yr 20 to 30/37 About 8 months Level II DAWIA* training (cid:127) Hybrid design,
Business certification in BCF** intermediate level
Management career field course.
(BCF 211) (cid:127) Students must pass
three tests online
within 60-day window.
(cid:127) Review material (no
structured lessons)
provided online as
prerequisite to five-
day classroom
portion.
1 Students per year figures are approximate.
2 Course hours based on course design estimates.
3 Development time obtained from Course Manager interviews. Generally, from the time the development contract was awarded
to the start of the first production offering.
* Defense Acquisition Workforce Improvement Act (DAWIA)
**Business, Cost Estimating, Funds Management (BCF) functional area
Note: 100 series courses are entry-level, 200 series courses are intermediate level, and 300 series course are advanced level.
The process for managing the devel- content development and programming,
opment of these courses mirrors the testing, and deployment. Interviews with
weapons system acquisition process. the eight course development managers
Both start with requirements generation (all DAU faculty members with practitio-
and progress through concept, design, ner experience in systems acquisitions),
79
Acquisition Review Quarterly — Winter 2003
confirmed that many of the course de- analysis of data. The success factors
velopment success factors are directly identified from the literature review
related to commonly accepted systems were then compared with success fac-
acquisition and program management tors developed from the DAU interviews
techniques. to determine which factors are common
The distance education literature re- to both and which are unique to DAU.
ports many success factors related to A set of proposed recommendations for
managing the development of technol- distance learning program managers
ogy-based university-level courses. was developed from the results of the
Those most commonly cited include study. The study provided a strong
effective use of chang- grouping of success factors and recom-
ing technologies, suffi- mendations that should apply to DAU
“The researchers cient resources for as well as to management of the devel-
used success
course development opment of distance learning courses at
factors described
and operation, creating similar institutions.
in distance learn-
educationally sound
ing research to
and engaging course
help guide data
designs, effective staff- BACKGROUND
collection.”
ing, detailed planning,
identifying and accom- From the inception of formal DoD sys-
modating the needs of instructors and tems acquisition training in 1971 until the
students, and ensuring sufficient tech- late 1990s, students have traveled to a DAU
nical expertise. classroom location to attend courses. Stu-
Much of the existing information fo- dents from more than 50 miles away incur
cuses on technology-based traditional temporary duty costs (travel, lodging, meals)
university training. Compared to tradi- that are paid by DAU. Until recently, class
tional universities, corporate universi- durations ranged from three days to 20
ties face unique challenges. Because the weeks. In response to downsizing and cost
corporation typically pays employee concerns in the mid-1990s, DAU developed
salaries while they are students at the a strategy to take advantage of emerging
corporate university, the training must technologies and join the movement toward
be as efficient as possible. Also, the technology-based distance learning (com-
training investment is expected to trans- monly known as E-learning).
fer directly into job performance. These In the context of this study, a technol-
are strong incentives for creating effec- ogy-based course is one that requires stu-
tive learning environments that mini- dents and instructors to use the Internet and
mize employee time in the training en- computer-based technologies to access and/
vironment. or manage some or all aspects of the course.
The researchers used success factors With contractor support, a unique Learning
described in distance learning research Management System (LMS) called the Vir-
to help guide data collection. Interviews tual Campus was developed in 1998 in an-
were conducted with the DAU course ticipation of hosting requirements for the
development managers, followed by to-be-developed Internet-based distance
80
Managing the Development of Technology-Based Courses
learning courses. Since 1998, more than interviewing course managers at DAU.
18 technology-based courses (commonly The success factors identified in the lit-
known as on-line courses) have been erature review were then compared with
developed by DAU, and more are success factors identified during analysis
planned. The number of graduates for of the DAU interview transcripts.
each course ranges from several hundred Due to the immense
to more than 10,500 students per year. body of knowledge re-
Collectively, over 20,000 students gradu- lated to distance educa- “Both mandatory
ate annually from these courses. The shift tion, selection criteria for and optional
technology-based
to on-line courses has significantly re- the search were very
training courses
duced the time students spend in the class- narrow, focusing on re-
are offered to
room environment, along with the asso- ports of success factors
over 130,000
ciated costs. for the management of
Department of
Both mandatory and optional technol- distance education de-
Defense acquisi-
ogy-based training courses are offered to velopment projects. The tion workforce
over 130,000 Department of Defense ac- search favored empiri- personnel in 11
quisition workforce personnel in 11 ca- cal results from con- career paths.”
reer paths. Some courses are conversions trolled studies where
from classroom courses, some are new possible. Additional
courses designed specifically for the on- sources that reflected collection of data
line environment, and some utilize a com- from experienced distance educators were
bination of new and existing material. The also included.
first courses were designed for Internet- A summary of common problems re-
based distance learning, with no physi- lated to innovation with on-line distance
cal classroom required. Later course de- learning (Robinson, 2001) provided a use-
signs, known as “hybrids,” included both ful method for success factor categoriza-
an Internet-based portion and an in-resi- tion. Based on the experience of 426 dis-
dence classroom portion. Table 1 sum- tance educators, Robinson classified dis-
marizes the eight courses studied for this tance education course issues related to
research project. innovation, leading to the four general cat-
egories of resource availability, organi-
zational issues, human resource capac-
REVIEW OF DISTANCE ity, and technology capabilities. These
LEARNING LITERATURE four categories were used to organize suc-
cess factors found in the literature.
The review identified success factors
critical to managing the development of RESOURCE AVAILABILITY
technology-based courses in the tradi- Resource issues figure prominently in the
tional university environment. General literature. Among the resources commonly
categories of success factors related to identified as lacking during distance edu-
managing the development of distance cation course developments are time, fund-
education courses provided a basis for the ing, personnel, and a sufficient technology
development of a research protocol for support infrastructure. Several authors
81
Acquisition Review Quarterly — Winter 2003
(Alexander, MacKenzie, & Geissinger, faculty, and staff toward the use of tech-
1998; Brigham, 1992) cite the importance nology must be part of the course devel-
of instructional support services, as well opment (Brigham, 1992; Volery, 2001).
as providing sufficient working time and
realistic production deadlines, as success HUMAN RESOURCE CAPACITY
factors. Robinson (2001) points out that Human resource capacity is tightly in-
distance education projects are often tertwined with other resource and organi-
underfunded as well as being too small in zational issues. Several primary human re-
scope to be financially viable, suggesting source issues were found in the literature.
that giving projects proper scope is neces- For example, development teams must
sary for success. produce quality materials and support the
instructional requirements of a distance
ORGANIZATIONAL ISSUES education environment (Brigham, 1992).
Organizational success factors are often Likewise, it is helpful to apply systematic
tightly interwoven with an organization’s and analytical methods of course design
structure and culture, and development (Alexander, MacKenzie,
and may be difficult to & Geissinger, 1998); Wagner, 1995). The
implement in some design and development needs require
“Human resource
capacity is tightly situations, especially course developers to go beyond general
intertwined with when introducing inno- conceptual planning and think through the
other resource vation into the organiza- details involved in a distance course
and organiza- tion. However, when the (Robinson, 2001).
tional issues.” development team con- In an analysis of 104 Australian tech-
siders certain factors, dis- nology-based learning projects, Alex-
tance learning project ander, MacKenzie, and Geissinger (1998)
outcomes can be enhanced. For example, determined that the instructional staff must
distance education courses require unique address specific student needs, use the
internal coordination and administrative technology to enhance learning in ways
practices (Robinson, 2001). Research not previously possible, use a sound and
shows that descriptions of processes for well-integrated instructional strategy, in-
determining course content and the ap- clude learner support, and design assess-
proval of that content must identify clearly ments appropriate for technology-based
the people to be included in the process delivery. A proper balance must be present
(Brigham, 1992). In addition, the organi- between the capabilities of the instructional
zation must provide appropriate technical staff and the technical and instructional
support (Alexander, MacKenzie, & support staff (Volery, 2001), leading or-
Geissinger, 1998) and consistent organi- ganizations with less technical support
zation-wide strategies for the use of tech- to invest additional resources in staff de-
nology in teaching and learning (Bates, velopment.
2000). Wagner (1995) identified the need
for adequate organizational learner and in- TECHNOLOGY CAPABILITIES
structor support. Finally, consideration of A broad range of technological suc-
the overall attitudes of administrators, cess factors are identified in the literature
82
Managing the Development of Technology-Based Courses
reviewed. Availability of adequate researchers. A single researcher conducted
technical support is mentioned repeat- all remaining interviews. All interviews were
edly and is tightly interwoven with the recorded and transcribed for the analysis.
resource, organizational, and staffing is- The interview questions (see Appendix)
sues described above. Lopez and were designed to provide field-based inputs
Nagelhout (1995) note three success fac- from the eight DAU developmental course
tors for the use of technology in on-line managers sufficient to allow the comparison
education: reliability, quality, and richness. of their experiences with the success factors
Alexander, MacKenzie, and Geissinger identified in the literature. The interview ques-
(1998) note numerous technology success tions were organized into three groups: stake-
factors including software testing, software holder issues (organizational category issues),
development expertise (where relevant), team-level issues in the development pro-
copyright issue resolution, and student ac- cess (human resources issues), and course-
cess to hardware and software. Bates level issues (resource availability and tech-
(2000) noted that there is a tension between nology issues).
the need for student technology access and The interview method
equity of access to higher education. Bates was face-to-face with fol-
also made the interesting note that due to low-on contact for clarifi- “The research
the high and recurrent investment cost in cation. Data analysis in- design was
largely qualita-
technology, the use of new technologies cluded identification of 99
tive and used
may not provide overall cost savings. independent issues in the
guided interviews
The literature indicates that not all transcripts followed by or-
as the primary
course developments end successfully, ganization of those issues
means of data
and failures can often be traced back to into themes and then into
collection.”
poor understanding at some level of candidate success factors.
how to balance the factors discussed The course managers re-
above, or even ignorance of some of viewed the results and a post-hoc analysis
those factors. correlated the data with results of the litera-
ture review. As part of the post-hoc analysis,
the DAU course managers reviewed the ini-
DATA COLLECTION AND ANALYSIS
tial list of 10 most commonly occurring fac-
METHODS tors and their relative rankings. Based on their
feedback, the initial list of 10 was reduced to
The research design was largely qualita- eight by eliminating some redundancies.
tive and used guided interviews as the
primary means of data collection. The
RESULTS
interview protocol was designed to fa-
cilitate the exploration of the course
managers’ experiences and relate The eight success factors, derived from
them to the general issues identified the interview data, are described below:
in the literature review. The interview
protocol was validated with an initial 1. Effectively blending technologies — This
interview conducted jointly by two success factor includes researching and
83
Acquisition Review Quarterly — Winter 2003
analyzing available technologies and optimum balance between cost,
the most efficient mix of technolo- schedule, and quality; developing
gies, and considering methods of baselines and metrics for the course
blending on-line and classroom de- development; continually tracking
livery methods. Also included is un- and monitoring the course’s progress
derstanding the impact of future against those baselines; and making
changes driven by new technology. changes as necessary to ensure ad-
At DAU, entry-level courses typically equate progress and performance.
require minimal instructor resources, These techniques are also critical to
while higher-level courses may have the success of any systems acquisi-
more interactive designs (e.g., tion.
blended (hybrid) on-line/classroom
components), which require more re- 4. Meeting student needs with instruc-
sources. tional design strategies — This suc-
cess factor is particularly critical for
2. Technical configuration control — This distance learning or hybrid course-
includes document version control, ware. Ineffective design strategies
harmonization of design and develop- will not hold a student’s interest in
ment versions, and assurance of source an on-line environment. Data from
documentation for all materials. Key the DAU interviews show that effec-
challenges of configuration control for tive strategies include the use of
course developers were problem-based and scenario-based
managing, tracking, up- training mechanisms along with
“Ineffective de- dating and documenting storylines integrated across part or
sign strategies will
the assignment of learn- all of the course. Several courses em-
not hold a
ing objectives to materi- ploy a highly blended strategy that
student’s interest
als, creating test items uses the on-line part of the course to
in an on-line
based on the learning ob- prepare students to work effectively
environment.”
jectives and course con- in the classroom as part of a team.
tent, and effectively man- Other important design factors in-
aging developmental and production clude planning adequate student time
courseware releases. Real world policy for course completion, matching the
changes must also be rapidly incorpo- course level with the students’ ex-
rated into courses, even as material is pected level of preparedness, and
being developed, which adds to the providing efficient and usable re-
configuration control challenge. sources to the students (some of
these are not directly controlled by
3. Project planning and management the course developer, but can often
techniques — This involves defining be indirectly influenced).
overall course requirements and
learning objectives before develop- 5. Availability of Subject Matter Experts
ing the course design; making man- (SMEs) — Ideally, full-time SMEs are
agement tradeoffs to achieve the dedicated to the project. This was a
84