ERIC ED444036: National Assessment of Vocational Education (NAVE): Overview of Evaluation Plan.

29 Pages·2000·0.56 MB·English

by  ERIC

DOCUMENT RESUME
CE 080 626
ED 444 036
National Assessment of Vocational Education (NAVE): Overview
TITLE
of Evaluation Plan.
National Assessment of Vocational Education (ED),
INSTITUTION
Washington, DC.
2000-00-00
PUB DATE
28p.; For NAVE Design papers, see CE 080 512-516.
NOTE
For full text:
AVAILABLE FROM
http://www.ed.gov/offices/OUS/eval/NAVE/evaluation.html.
Descriptive (141)
Reports
PUB TYPE
MF01/PCO2 Plus Postage.
EDRS PRICE
Academic Achievement; *Compliance (Legal); Educational
DESCRIPTORS
Change; *Educational Legislation; Educational Policy;
Federal Legislation; Outcomes of Education; Postsecondary
Education; *Program Evaluation; *Research Methodology;
Secondary Education; Special Needs Students; *Vocational
Education
*Carl D Perkins Vocational Technical Educ Act 1998;
IDENTIFIERS
*National Assessment of Vocational Education
ABSTRACT
The Carl D. Perkins Vocational and Technical Education Act
as amended in 1998 (Perkins III) requires that a national assessment of
vocational education (NAVE) be conducted. Several key policy issues are
(1) the
likely to shape the course of vocational education and of the NAVE:
small federal investment in vocational education relative to overall federal
(2) the focus of education reforms on improving students'
education spending;
academic achievement; and (3) changes in accountability and funding
provisions in Perkins III. The NAVE will seek to discover which strategies
improve the performance of vocational students, what are the pathways by
which sub-baccalaureate students prepare for careers, and whether the policy
shift to flexibility and accountability is likely to improve program quality
and student outcomes for regular and special needs students. The NAVE study
agenda calls for diverse data collection and analysis methods and the study
of key issues in both secondary and postsecondary vocational education.
Because NAVE results must be reported to Congress by July 2002, the study
will reflect the very early efforts made in response to new provisions in
Perkins III.
(KC)
Reproductions supplied by EDRS are the best that can be made
from the original document.
National Assessment of
Vocational Education
(NAVE)
Overview of Evaluation Plan
U.S. DEPARTMENT OF EDUCATION
Office of Educational Research and Improvement
EDU ATIONAL RESOURCES INFORMATION
CENTER (ERIC)
his document has been reproduced as
received from the person or organization
originating it.
Minor changes have been made to
improve reproduction quality.
Points of view or opinions stated in this
document do not necessarily represent
official OERI position or policy.
AVAIYABLIE
BEST COPY
NATIONAL ASSESSMENT OF
VOCATIONAL EDUCATION (NAVE)
KANSA!
OVERVIEW OF EVALUATION PLAN
V11)1.04E11101
YOCII1011At
ER1411140
ABSTRACT
Act (Perkins
The recently amended (1998) Carl Perkins Vocational and Technology Education
evaluation and assessment of
III) directs the Secretary of Education to complete an "independent
advi-
vocational and technical education programs under this Act," and to appoint an independent
Education on how to conduct this national assessment of
sory panel to advise the Department of
vocational education (NAVE).
Vocational education is a field in transition, prompted by sweeping changes in state and local
education priorities. Several key policy issues are likely to shape the course of vocational educa-
tion, and therefore of the NAVE: (1) the small federal investment in vocational education relative
improving students'
to overall federal education spending, (2) the focus of education reforms on
academic achievement, and (3) changes in accountability and funding provisions in Perkins DI
Given the educational and political context in which Perkins III operates, as well as specific leg-
islative changes recently enacted, the NAVE will be guided by several main questions:
What strategies improve the performance of "vocational students" and how does, or can,
vocational education contribute to improving academic and occupational skills, access to
postsecondary education, and earnings?
What are the pathways by which sub-baccalaureate students prepare for careers, and
what is the contribution of workforce reform efforts to improving their training?
Is the policy shift from set-asides and legislative prescription to flexibility and account-
ability likely to improve program quality and student outcomes? How do special popula-
tions fare?
Addressing the primary vocational
policy issues requires a set of
Figura 4
distinct
interrelated
but
studies.
-::
The overall research agenda calls
RESEARCH STRATEGIES TO ADDRESS
MAIN POLICY ISSUES
for diverse data collection and
analysis methods, including: quali-
tative case studies, national sur-
Secondary Vocational Education
veys, examination of existing data-
A. Participation, Outcomes, and Impact
bases, and econometric estimation.
B. Quality
C. Effectiveness of Promising Strategies
While some broad themes are
relevant to both secondary and
Postsecondary Occupational Education
postsecondary vocational educa-
A. Participation, Pathways, Outcomes, and Impact
B. Alignment with WIA
tion, each also has its own key is-
sues. Finally, given the timeline
Program Management
for implementing Perkins III, and
A. Accountability
the July 2002 date for reporting
B. Funding
NAVE results to Congress, the
Other
studies conducted will reflect the
very early efforts made in re-
sponse to specific new provisions
in Perkins III.
AVAOLABLE
BEST COPY
3
NATIONAL ASSESSMENT OF
VOCATIONAL EDUCATION (NAVE)
OVERVIEW OF EVALUATION PLAN
INTRODUCTION
I.
Federal support for vocational education, and for understanding its consequences, has a long his-
tory. As was true with previous vocational legislation, the recently amended (1998) Carl Perkins
Vocational and Technology Education Act (Perkins III), directs the Secretary of Education to
complete an "independent evaluation and assessment of vocational and technical education pro-
grams under this Act," in this case, by 2002. As currently designed, this national assessment plan
will include multiple components. This paper discusses the main issues that NAVE will address,
and the research strategies that will be undertaken to address those issues.
Perkins 111 (section 114) directs the NAVE to address a wide range of topics, including: (1) im-
plementation of state and local programs; (2) impact of changes in federal funding formulas;
(3) teacher quality and teacher supply and demand; (4) student participation in vocational educa-
tion, (5) academic and employment outcomes; (6) employer involvement and satisfaction with
vocation education programs; (7) education technology and distance learning; and (8) the impact
of accountability requirements on program performance. Perkins DI also directs the Secretary to
appoint an Independent Advisory Panel to provide advice on conducting the NAVE and to submit
to Congress its own independent analysis of NAVE findings and recommendations (see Appendix
A for list of panel members).
Congress has mandated a final NAVE report in 2002 to provide it with information that can guide
reauthorization of the Perkins Act in 2003. Of primary interest is how some of the new provisions
in the law have been implemented. Unfortunately, many of the important changes in accountabil-
ity, integration with workforce development, and increased flexibility will have barely begun.
Most states have opted for "transitional plans" which in effect, defer implementation of Perkins
III provisions until October 2000 (Figure 1). Given the July 2002 date for reporting NAVE
results to Congress, data collected during the 2000/2001 school year will reflect the very early
efforts made in response to specific new provisions in Perkins III. Conclusions regarding the
longer-term prospects for Perkins III will require further research.
Figure 1
NAVE TIMELINE
January
January
July October
mbar
1=
ACI
2000 2000
2000
1088
001
31/311111131113121.213171111111.1171112111211931111101311.311.11213111211111,311,710701111011311H313b
Interim
Most States
State
Final
WIA NAVE
Perkins Ill
NAVE
WIA
Report
Report
Data
Passes Passes
Data
Perkins III
int=en.
Implementing
Plans to
Collection
Collection n
Ends
Begins Perkins
Begins
ED
AVAILABLE
BEST COPY
4
At the same time, however, Perkins III continues to emphasize several major themes reflected in
the 1990 Perkins amendments. These include the integration of academic and vocational educa-
tion, broadening the focus of vocational education to emphasize industries and careers, and
strengthening the links between secondary and postsecondary education through tech-prep and
other strategies. The current NAVE will be able to track the extent to which these longer run
themes are reflected in school practice, and where possible, what effects they have on outcomes
for students.
All of these objectives will be addressed in a series of studies soon to get underway. Two factors
are guiding the design of the NAVE agenda: (1) the current policy environment, and (2) the spe-
cific research questions that reflect policymaker and practitioner interests.
POLICY CONTEXT FOR THE NAVE
A.
Over the past two decades, three prior national assessments of vocational education have been
conducted.' Each study sought to focus upon what were thought to be the major issues facing vo-
cational education at the timefunding, special populations, and economic competitiveness of
the workforce. The current NAVE must also be sensitive to the dominant educational issues of its
time.
Vocational education is a field in transition, prompted by sweeping changes in state and local
education priorities. New goals, program offerings, and terminology increasingly characterize
vocational education. Federal legislation has encouraged several major changesfrom an historic
emphasis on entry-level job preparation in semi-skilled occupations to a broader focus on prepa-
ration for careers that offer high wages and requires higher level skills; from preparing students to
enter the workforce directly after high school to providing students with the choice of pursuing
employment or attending college, or as is increasingly the case, doing both simultaneously; and
from expecting vocational students to do less well in school than other students, to holding such
students to the same academic standards as others. Many of the overarching issues that NAVE
will address will consider whether this transition in the field of vocational education is "on track."
Several key policy concerns are likely to shape this assessment:
Federal Funding For Vocational Education. Federal budgets are widely regarded as
a basic indicator of policy priorities. Although overall funding on Department of
Education (ED) programs has increased by 177 percent from FY 1980FY 1999, vo-
cational education funding increased by only 47 percent during the same period (Fig-
ure 2).2 In 1980, funding for vocational education was about 6 percent of total ED
expenditures; it has now shrunk to about 3 percent.
11982, 1989, and 1994.
2Figure 2 shows the trends in federal funding for vocational education over the past 20 years, and compares funding for
vocational education with overall funding for Department of Education programs.
NAVE Overview-2
Figure 2
FUNDING TRENDS IN EDUCATION PROGRAMS
Amount In billions
$40
as
15
10 .:
Perkins
Vocational
Education
5
1494 1999
1992
1989 1995
1985
1943
1980
These trends strongly suggest that for nearly two decades vocational education has
been increasingly viewed as less worthy of investment than other areas. One reason
for this pattern may be that vocational education has historically prepared students
for work at a time when priorities have shifted toward academic preparation for col-
lege. To the extent that vocational education has changed to address these as well as
more traditional objectives, it will be important for NAVE to describe and assess this
development.
Figure 3
Although overall federal funding
for vocational education has not
SELECTED EDUCATION DEPARTMENT
SPENDING ON HIGH SCHOOLS
changed substantially, the Perkins
Act remains the largest single
source of ED funds directed to
Trio
high schools. Comparing ED' s
three major sources of funding
TN* I
for high schools (only a share of
_
the funds from each program is
Vocational
directed at high schools), Perkins
Education
-- --,_
Act funds spent to help high
WO
school students are more than
$600
$400
$200
SO
(in millions)
Title I and TRIO funds combined
1999
(Figure 3).3
3Figure 3 compares ED funding of high schools from three major sourcesthe Perkins Act, Title I of Improving
America's Schools Act (IASA), and the TRIO program.
NAVE Overview-3
6
BEST COPY AVAIILABLE
Given the major role that Perkins funding plays at the high school level relative to
other ED programs, it makes sense to think about how this investment contributes to
high school improvement generally, and the contribution that vocational education
makes, or could make, to such efforts.
Academic Reform. There is little question but that the principal focus of recent edu-
cation reforms has been on improving students' academic achievement and increas-
ing their opportunities to attend college. Federal vocational education policy now
places top priority upon ensuring that vocational education students are academically
well prepared for both careers and/or success in postsecondary education. But what
is, or ought to be, the responsibility of vocational educators for ensuring that students
leave high school with both a solid academic foundation as well as technical skills?
One major goal of NAVE should be to help Congress better understand what the
contribution of vocational education is to achieving these objectives and how this
contribution can be increased.
Changes in Perkins III. Although Congress did not alter the basic structure of the
Perkins Act in the 1998 reauthorization, it did make several important substantive
changes. Among these are: (1) increased emphasis on academics; (2) greater flexibil-
ity in the use of funds through elimination of major set-asides for gender equity and
other rules governing the use of funds; (3) a higher proportion of funds directed to lo-
cal programs and the establishment of a 10 percent reserve fund; and (4) creation of a
"higher stakes" accountability system. At the same time, the Workforce Investment
Act of 1998 (WIA) also encouraged greater integration of vocational education and
the workforce development system. Implementing these changes will be a major
challenge for school districts and postsecondary institutions.
B.
MAIN POLICY ISSUES
Given the educational and political context in which Perkins III operates, as well as specific leg-
islative changes recently enacted, the NAVE will be guided by several main questions:
1. What strategies improve the performance of "vocational students" and how does, or
can, vocational education contribute to improving academic and occupational skills, ac-
cess to postsecondary education, and earnings?
Standards-based reform is fundamentally altering policies and practices from kindergar-
ten through high school. NAVE will examine the role of standardsboth academic and
occupationalin vocational education, the effects on students' academic and technical
preparation of promising reform strategies, and the role of federal policy, as implemented
by states and communities, in facilitating implementation of effective programs of study.
2. What are the pathways by which sub-baccalaureate students prepare for careers, and
what is the contribution of workforce reform efforts to improving their training?
In passing Perkins IQ and the WIA, Congress was concerned that federally supported job
training programs operated independently of vocational education. Congress expects that
linking Perkins DI and WIA together will encourage better integration of vocational edu-
cation and workforce development policies. Issues to be examined include: (1) the role of
NAVE Overview-4
7
(2) the importance
postsecondary vocational in developing state workforce training plans;
institutions (e.g., programs of
of unified planning; (3) the early impact on postsecondary
align Perkins DI and
study, one-stop career centers, accountability) of efforts to
participation in postsecondary
workforce development, and (4) patterns of enrollment and
occupational education, and their relationship to outcomes and impact.
flexibility and account-
Is the policy shift from set-asides and legislative prescription to
3.
How do special popu-
ability likely to improve program quality and student outcomes?
lations fare?
For the past two decades, federal policy has focused on serving those most at-risk, com-
monly termed the "special populations." Perkins DI represents a major shift in direc-
tioneliminating set-asides for "single parents...," requirements that local funds be pri-
oritized to serve the highest concentrations of special populations, and requirements to
coordinate with the Individuals for Disabilities Education Act. In its place is an increased
emphasis on accountability, including the requirement that states track the progress of
special population groups. Has increased flexibility resulted in changes in educational
priorities or practices? Have "at-risk" populations been helped or hurt as a result? Are ac-
countability requirements improving the quality of vocational education for all students?
II. NAVE RESEARCH AGENDA
Addressing the primary vocational
policy issues requires a set of inter-
Figure 4
related but distinct studies. No one
ADDRESS
RESEARCH STRATEGIES TO
study could encompass the broad
MAIN POLICY ISSUES
array of interests and questions.
research
overall
the
Moreover,
agenda calls for diverse data col-
Secondary Vocational Education
lection and analysis methods, in-
A. Participation, Outcomes, and Impact
B. Quality
qualitative case
studies,
cluding:
C. Effectiveness of Promising Strategies
national surveys, examination of
existing databases, and sophisti-
Postsecondary Occupational Education
cated econometric estimation. Fi-
A. Participation, Pathways, Outcomes, and Impact
B. Alignment with WIA
nally, while some broad themes are
to both secondary and
relevant
Program Management
educa-
vocational
postsecondary
A. Accountability
B. Funding
tion, each also has its own key
issues.
Other
For these reasons, the NAVE plans
to undertake studies in seven main
areas (Figure 4). These are described below, grouped together by major topic: secondary voca-
tional education, postsecondary vocational education, and program management.
NAVE Overview-5
8
SECONDARY VOCATIONAL EDUCATION
is evolving. Many policymakers, educa-
The role of vocational education at the secondary level
about whether secondary vocational programs serve students'
tors, and parents remain concerned
that completion of a vocational program
best interests in the long run. Previous studies suggest
if students do not pursue postsecon-
has a positive effect on employment, at least in the short run,
jobs.5 However, the earlier research also indi-
dary education4 and if they obtain training-related
neither academic achievement nor postsecondary
cates that vocational education contributes to
by the last
enrollmentthe path desired by most students and their parents. Studies undertaken
traditional, with
NAVE confirmed that vocational instruction in the early 1990s was still largely
of vocational
little focus on academic skills.6 These concerns about the quality and outcomes
1980s.
education have contributed to the decline in participation of high school students since the
These same concerns have raised the stakes for gaining a clearer understanding of: (1) how par-
ticipation in and impacts of secondary vocational education have changed since the earlier stud-
ies, (2) the extent to which secondary vocational education now reflects high quality practices,
and (3) whether promising career education reform strategies are proving to be effective.
PARTICIPATION AND OUTCOMES
A.
Perhaps the most important issue for vocational education, and for the current NAVE, is who
participates at the secondary level and how well they fare in school and beyond. At least through
the early 1990s, vocational education had come to be stigmatized as a high school "track" for stu-
problems.' This result alone
dents with poor academic capabilities, special needs, or behavioral
would not be problematic, but vocational programs also appeared to contribute little "value
added" to student outcomes. In the current climate, with increasing federal emphasis on program
performance and on high academic achievement for all students, vocational education has much
to prove.
Over the last five years, however, career education has been given new prominence by several
initiatives targeted to students other than those traditionally served by vocational programs. In
addition, some states and districts have worked to strengthen vocational courses.8 Whether any of
these efforts have successfully broadened the appeal and improved the impact of occupational
programs needs careful examination.
teffan Mane, Trends in the Payoff to Academic and Occupational-Specific Skills: The Short and Medium Run Returns
to Academic and Vocational High School Courses for Non-College Bound Students (Working paper # 98-07) (Ithaca,
NY: Center for Advanced Human Resource Studies, Cornell University).
5National Assessment of Vocational Education, Final Report to Congress, Volume I, July 1994.
61bhi.
gAlan Hershey, Marsha Silverberg, Joshua Haimson, Paula Hudis, and Russell Jackson, Expanding Options for Stu-
dents: Report to Congress on the National Evaluation of School-to-Work Implementation (Princeton, NJ: Mathematica
Policy Research, October 1998).
NAVE Overview--6
9
Figure 5
SECONDARY VOCATIONAL PARTICIPATION
AND IMPACTS
Study Plans
Research Issues
NAEP Transcript Study
Enrollment Trends
Analyses using NELS, HS&B,
Special Populations Participation
NLSY, and NAEP
Course Taking
NELS and HS&B Academic
Academic Achievement
Achievement Comparison
Postsecondary Enrollment
Analyses of state data
Labor Market Benefits
Who participates in secondary vocational education?
ao
Updating the enrollment trends and characteristics of students served by high school-level voca-
tional programs is of interest, particularly in light of growing reports from state directors that en-
rollment patterns have been changing. Congress and most educators, remain committed to pro-
viding students from special populations access to vocational education. However, many believe
the quality of vocational programs is unlikely to improve without attracting a broader segment of
the student population; or that the participation of a more diverse set of students will signal that
quality improvements are being made. Several questions must be addressed:
Who enrolls in secondary vocational education and to what extent has this picture
changed over the 1990s? How do the characteristics of vocational students compare
with those of nonvocational students?
Does enrollment vary by high school setting and vocational program area?
Has the course taking (academic and technical) of vocational students changed?
PLANNED STUDIES
Descriptive statistics from national data. NAEP transcript data ( 1998) provide the
most recent evidence of secondary vocational participation nationwide. Using NAEP
and student transcripts from other data sources, trends in student characteristics,
course taking and achievement will be documented.
NAVE Overview-7
10