ERIC ED341312: Evaluation of Pre-Departure Orientation of Long-Term, Off-Shore Graduate Education Participants: Theory & Practice.

36 Pages·1991·0.54 MB·English

by  ERIC

DOCUMENT RESUME
HE 025 204
ED 341 312
AUTHOR
Aagard, Steven D.
Evaluation of Pre-Departure Orientation of Long-Term,
TITLE
Off-Shore Graduate Education Participants: Theory &
Practice.
PUB DATE
91
NOTE
36p.
Reports - Research/Technical (143)
PUB TYPE
EDRS PRICE
MF01/PCO2 Plus Postage.
DESCRIPTORS
*Agricultural Education; *Evaluation Methods; Foreign
Countries; *Foreign Students; *Graduate Study; Higher
EducatioL; Orientation; Program Effectiveness;
Program Evaluation; Study Abroad; Surveys; Vocational
Education
IDENTIFIERS
*Malawi
ABSTRACT
This study presents an analysis of a previous
evaluation conducted by the Malawi (East Africa) government of the
usefulness of long-term, off-shore (mostly in the United States)
graduate education and results of a recent survey comparing career
attitudes of participants and non-participants in a pre-departure
orientation program. Following a discussion of evaluation theory, a
critique of the evaluation commissiohed by the Malawi government
includes backgrounl information on the evaluation, ar analysis of the
data gathering techniques and instrument, and a discussion of the
possible uses of the information. Application of previously developed
evaluation principles to the survey (N=60) of a pre-departure
orientation program found that there was no difference in the
perceived relevance of graduate education to career and national
development for those individuals uho participated in a pre-departure
orientation as compared to those who did not. However, the
demographic analysis of the two groups found that the bulk of
individuals who did not participate in the orientation program were
doctoral students who had previously completed master's level work in
the United States. Included are three tables and 10 references.
(JB)
***********************************************************************
Reproductions supplied by EDRS are the best that can be made
from the original document.
***********************************************************************
parture On ntation
Shore Gra uate
ants
by
Steven D. Aagard
"PERMISSION TO
O(PARTMINT Of IIDUCATiON
REPRODUCE THIS
OThel at Educations' Research and Imorovemant
MATERIAL HAS
BEEN GRANTED
BY
Steven
EDUCATIONAL RESOURCES INFORMATION
Aagard
CENTER IERIC)
1Thts document has ben reproduced as
wowed trom it* person or oroanitation
originating it
C Minor changes have been made to improve
reproduction dualdy
I Points ot view or (motions stated in this docu.
TO THE
EDUCATIONAL
ment 00 not nacessardy
present official
RESOURCES
INFORMATION
OERI posthon 31 policy
CENTER (ERIC)."
2
Kg Mtn MMUS!
Table of Contents
Introduction
1
5
Theory
15
Critique
20
Implementation
24
Findings
26
Conclusions and Discussion
28
Limitations and Recommendations for Further Study
30
Refet ences
3
1
Introduction
The process of development and improvement requires constant evaluation and
change. This is true of a person or an educational program. The evaluation element is
critical to determine which parts of the program are worthy of continuation; which need to
be modified to better fit the circumstances; and which need to be discarded/replaced. The
result is a spiraling effect of application of a treatment and evaluation of its effectiveness
which brings the person or the agency closer to the ultimate goal.
In focusing specifically on the evaluation portion of this process, one needs to
develop an evaluation strategy that is situationally appropriate. A theoretical framework
must be interwoven into the structure of the evaluation (Figure 1). The evaluator must
analyze the evaluation work that has been completed previously to determine what is still
needed. Finally, the application of the background and chosen design to the situation. This
writer has chosen to evaluate the relevance of long term U.S. graduate education for
students from Malawi, East Africa.
During the past decade, the developing countries of the world have invested more
than two billion dollars in training of nationals. Training is aimed at upgrading skills and
building organizational capacity through in-country and long-term, off-shore training.
Richard Clough, Regional World Bank Agricultural Training Specialist for the African Desk,
estimates that World Badc, USAID, and other donors have invested more than $50 million
in manpower development since 1980.
Major donors have invested in Malawi, most of
which was in the area of long-term, off-shore training at the masters and doctorate levels.
Some 75 officers from the Ministry of Agriculture have been sponsored for graduate
education by USAID since 1980.
4
2
Figure 1.
plementati
3
Officials from Malawi and USAID are vitally interested in the impact of this training.
A central question pertains to the relevancy of the training to both the individual and the
agricultural development of Malawi. The USAID position is well articulated by the Mission
is essentia. to determine both
Director Carol Pens ley in 1989 when she said,
"It
quantitatively and qualitatively the impact of the Mission's long term commitment to masters
and doctorate training of Ministry of Agriculture officers in the U.S. One of the first steps
is to view the training in terms of relevancy and impact of the agricultural development of
Malawi."
An evaluation process was established to determine the effectiveness of long-term
graduate education with Malawi students. The writer's analysis of this evaluation will be
broken down into three main areas:
The theoretical basis for the evaluation;
Critique of the Malawi evaluation within that framework;
Construction and application of evaluation techniques to a pre-departure
orientation program.
The theoretical framework includes the selection of the evaluation design given the
constraints of the situation and guarding against threats to the internal and external validity
of the study. An evaluation model will be selected and justification for its use given. The
research techniques used for data collection will also be explored. In addition to the text
used in the evaluation class, the writer will incorporate insight into the evaluation process
from outside sources.
After establishing the theoretical basis for the evaluation, the evaluation conducted
for the Government of Malawi (GOM) by the Department of Adult and Youth Education
This will include information on the
at Washington State University will be analyzed.
4
Information for this section was obtained
research procedures, design and constraints.
The analysis will be
through written reports and interviews with program evaluators.
the analysis
structured around the theoretical basis which has been developed. Included in
will be a exitique of the instmment used and a reaction to the information obtained.
The final section will report the application of the evaluation model and research
des;gn selected to a portion of the Malawian evaluation. A pre-departure orientation
long-term graduate study in the United
program conducted for students scheduled to begin
educational
States will be the focus of the evaluation. Included in this section will be the
adiiiinistration
outcomes to be evaluated, instrument chosen to evaluate the outcomes,
procedures, results and a discussion of the limitations and conclusions drawn from this
evaluation procedure.
7
Th
Ory
..
ost Facto
Ex
ram Outcomes
M
Pro
2
Studies
Case
Is
Site Vi
ter a
External vs.
I
S
5
Theory
The usefulness of information gained from an evaluation can depend largely on the
care that is taken by the evaluator when planni.ig the evaluation. The evaluation design and
evaluation model must be carefully selected while keeping in mind the specifics of the
program. There are three broad categories of evaluation designs: non-experimental, quasi-
experimental, and true experimental. Within each of these broad categories, there are
several types of evaluation designs. The elements used to distinguish one typc from another
include random vs. non-random selection, centralized or decentralized selection for the
treatment, administration of a pre-test, presence of a control or comparison group and the
timing of the assessment/treatment process. Selection is the "locus of decision regarding
which treatment a given subject will receive." (Mohr, 1988 p.45) This selection process can
determine which of the evaluation designs is the most appropriate for the evaluation.
Random selection designates that the participants are being assigned to treatment and to
comparison or control groups at random. Each individual has an equal chance of being in
either group, whereas in non-random selection this is not the case. Centralized selection
states that assignment to the groups is being made by one individual or group of individuals.
The same criteria for assignment is being used on each individual. Testing the participants
prior to the treatment to establish a baseline for interpretation of any differences produced
by the treatment is another facet of determining which evaluation design to be used. Non-
experimental designs do not utilize pre-testing procedures.
In selecting a specific design, the evaluator is concerned with two main things:
Internal validity is the capability of an evaluative research
external and internal validity.
study to establish that the observed program outcomes were actually produced by the
9
6
validity is the degree to
program operations rather than extraneous influences. External
which the research findings may be generalized to a larger population (Miller, 1979). Given
the circumstances of the program to be evaluated, the researcher must select the design that
guards against as many threats to validity as possible. Threats to validity are alternative
elements that could have brought about the change in the program participants. These
and
threats include history, maturation, test-wiseness, attrition, selection, contamination
spuriousness. The evaluation designs can be placed on a continuum according to how well
This continuum would proceed from non-experimental to
they counter those threats.
experimental with experimental being the most effective at guarding against the threats.
quasi
non-
-
-
experimental
experimental
experimental
-+
the
Experimental designs however do not often occur in real life and therefore they are
often more easily constructed, but
most difficult to relate to real life situations. They are
therefore less
the circumstances surrounding the experiment can be rather contrived and
often
applicable to the world outside the experiment. Since educational programs are not
alternate
constructed to meet all the criteria of experimental designs, evaluators must choose
be
methods that fit the situation. In choosing those alternate methods, the evaluator must
inherent with that design and the limitations posed on the
aware of the threats to validity
interpretatik. Lio use of the results.
ex-post-facto or
One of the designs categorized in the non-experimental group is the
therefore there are
"after the fact" design. This design is quite low on the continuum and
that
several threats to validity that the evaluator must factor in. There are two elements
characterize the ex-post-facto design:
reconstruction of events
! ()