ERIC ED604133: Effectiveness of WorkKeys Curriculum for Improving Foundational Workplace Skills: Nevada State High School Case Study. Technical Brief

0.37 MB·English

by  ERIC

ACT Research & Policy | Technical Brief | January 2020 1

Effectiveness of WorkKeys Curriculum for
Improving Foundational Workplace Skills:
Nevada State High School Case Study

Jeffrey T. Steedle, Senior Research Scientist, ACT
Wendi Hawk, Chief Academic Officer, Nevada State High School

Introduction Additional analyses detected significant
relationships between WorkKeys scores
This study sought to estimate the potential and grade point average at NSHS. In all,
benefits of using ACT® WorkKeys® results suggest that engagement with
Curriculum to improve foundational WorkKeys Curriculum helps improve
workplace skills and subsequent WorkKeys scores and that WorkKeys is a
performance on the ACT® WorkKeys® useful predictor of academic success in a
Applied Math, Workplace Documents, and dual-enrollment, college-into-workforce
Graphic Literacy assessments. Students at program.
Nevada State High School (NSHS) took the
WorkKeys Assessments before and after Nevada State High School
using ACT WorkKeys Curriculum. Using
data from the supervisor dashboard, NSHS is a dual-enrollment system of
students were classified as engaged or schools in Nevada for 11th and 12th
unengaged in lessons, practice questions, graders that has been operating since 2004.
and quizzes included in WorkKeys The students at NSHS attend college
Curriculum. Overall, students who engaged classes with college professors for a real
meaningfully with WorkKeys Curriculum college experience. NSHS provides classes
were expected to have greater score gains to help monitor and support students in their
and were more likely to increase their early college experience. The curriculum
WorkKeys level scores. In particular, focuses on readiness for the transition from
students who engaged in Applied Math college to careers with real world
practice questions had significantly greater applications. This involves working with
scale score gains than unengaged students, students to help develop workplace skills
and students who exhibited overall and career direction for their post-high
engagement in the Applied Math and school college planning. NSHS has been
Workplace Documents modules were more using WorkKeys Assessments since the
likely to increase their corresponding level 2008–2009 school year as a way to
scores. Engagement in WorkKeys compare student performance with the
Curriculum was related to performance foundational workplace skill levels required
improvements for Graphic Literacy but not for jobs nationwide. Moreover, WorkKeys
to a statistically significant degree. results are meaningful for students after

ACT.org/research
© 2020 by ACT, Inc. All rights reserved. | R1793
ACT Research & Policy | Technical Brief | January 2020 2

high school because prospective employers engaged with the Applied Math and
may use WorkKeys for hiring purposes, and Workplace Documents modules of
most students earn a nationally recognized WorkKeys Curriculum increased their
certification—the ACT National Career scores approximately 1 point more, on
Readiness Certificate. average, than students who did not engage
with the curriculum.2 Unfortunately, students
Background in 2017–2018 were not assigned to the
practice questions, which might be expected
ACT WorkKeys is an assessment of
to contribute significantly to the
foundational workplace skills used by
effectiveness of WorkKeys Curriculum. In
employers and workforce developers to
2018–2019, NSHS students were assigned
select job candidates and trainees.
all components of WorkKeys Curriculum,
WorkKeys Curriculum is designed to help
which made it possible to evaluate the full
users improve the skills measured by the
implementation of the curriculum. This
WorkKeys Applied Math, Workplace
report presents results from analyses of the
Documents, and Graphic Literacy
2018–2019 data.
assessments. For each skillset, the
curriculum materials are divided by levels
Data
(3, 4, 5, 6, and 7) corresponding to levels of
knowledge and skills that are required for Data from several standardized
different jobs or training programs.1 assessments were analyzed for this study.
Curriculum users have the option of taking a WorkKeys scores are reported on a 65–90
placement test to estimate their initial skill scale, and those scale scores are
level. For each WorkKeys level, the transformed to level scores with a maximum
curriculum provides instruction, practice of 7. Level scores are commonly used to
items, and a summative quiz to evaluate support hiring decisions and admissions to
mastery. workforce training programs. Examinees
who achieve certain level scores on the
All first-year students at NSHS (11th or 12th Applied Math, Workplace Documents, and
grade) take the WorkKeys Applied Math, Graphic Literacy assessments are awarded
Workplace Documents, and Graphic an ACT® National Career Readiness
Literacy assessments. Students who do not Certificate® (NCRC®).3 Specifically,
achieve a level score of 5 or higher on a examinees earn a Bronze NCRC by scoring
given assessment are assigned to the level 3 or higher on all assessments, Silver
corresponding modules of WorkKeys by scoring level 4 or higher, Gold by scoring
Curriculum. Students subsequently take level 5 or higher, and Platinum by scoring
certain WorkKeys assessments again to level 6 or higher.
attempt earning a higher score. This pre-
test/post-test design provides a unique The data also included scores from the
opportunity to examine the effectiveness of ACT® assessment, which is primarily used
WorkKeys Curriculum for improving for college admissions. For the ACT, each
WorkKeys scores. examinee completes four subject-area tests
(English, reading, math, and science), and
A preliminary analysis of 2017–2018 data scores are reported on a 1–36 scale. The
from NSHS indicated that students who average of the subject-area scores is the
ACT Research & Policy | Technical Brief | January 2020 3

ACT Composite score. Students at NSHS of the sample was eligible for free or
also took ACT Engage®, a measure of reduced-price lunch. Analyses were
social and emotional learning competencies conducted on smaller groups of students
associated with academic success. The full who retested on Applied Math (186),
instrument includes 108 items divided Workplace Documents (159), or Graphic
between 10 subscales. ACT Engage school Literacy (63). The distributions of
and district score reports include two demographic variables were similar in each
indices: an academic success index analysis, so only overall sample
reflecting the probability of earning a 2.0 or demographics are reported here.
higher GPA, and a retention index reflecting
the probability of completing high school; Results
both were analyzed in this study.
Descriptive Statistics
NSHS provided ACT a de-identified data file Table 1 provides descriptive statistics for
including records for students in 2018– WorkKeys, the ACT, Engage, and GPA for
2019. This file included demographics, the different (but overlapping) samples
grade point average, math courses taken, analyzed for this study. Pre-test WorkKeys
WorkKeys scores, and ACT Engage scores. scores reflect the earliest test date during
Using a unique identification number, the the 2018–2019 school year, and post-test
NSHS data were merged with records scores reflect that latest test date. The
exported from WorkKeys Curriculum. This average maximum WorkKeys scores were
included percentage of instructional lessons slightly higher than the average post-test
viewed, total time spent on lessons, number scores because students did not always
of practice questions completed, average achieve their highest scores on the last test
practice question response time, date. Comparing ACT Composite score, the
percentage correct on practice questions, Engage indices, and GPA between samples
number of quizzes completed, average quiz indicates that the Graphic Literacy sample
question response time, and percentage had relatively low average academic
correct on quizzes (overall and for each achievement and social-emotional learning
level). competencies.
Sample Table 2 shows the correlations between
variables analyzed in this study. ACT
The overall sample included 248 NSHS
Composite score had moderate and
students, 70.2% of which were female. In
statistically significant correlations with all
terms of race/ethnicity, the sample was
three maximum WorkKeys scores. That is,
6.0% Asian, 11.7% Black, 40.7% Hispanic,
students who earned higher ACT
1.6% Pacific Islander, 29.0% White, and
Composite scores tended to earn higher
10.9% multi-racial/ethnic. Four percent of
WorkKeys scores. Of the WorkKeys
students were classified as English
Assessments, social-emotional learning
language learners during 2018–2019; an
competencies were most strongly related to
additional 17.3% had been classified as
Graphic Literacy scores. Nearly all
English language learners in prior years. As
WorkKeys, ACT, and Engage scores
an indicator of socioeconomic status, 47.6%
correlated significantly with GPA.
ACT Research & Policy | Technical Brief | January 2020 4

Table 1. Average Assessment Scores and Grade Point Averages
Workplace Graphic
Applied Math
Documents Literacy
(n = 186)
(n = 159) (n = 63)
WorkKeys Pre-Test 76.2 77.9 75.6
WorkKeys Post-Test 78.1 79.8 78.0
WorkKeys Maximum 78.6 80.2 78.2
ACT Composite 17.8 17.8 16.6
Engage Academic Success .45 .45 .39
Engage Retention .41 .41 .36
Incoming GPA 3.13 3.19 2.94
Outgoing GPA 3.20 3.28 3.02
WorkKeys Pre-Test 76.2 77.9 75.6
WorkKeys Post-Test 78.1 79.8 78.0
WorkKeys Maximum 78.6 80.2 78.2
ACT Composite 17.8 17.8 16.6

Table 2. Correlations with Outcome Variables
Workplace Graphic
Applied Math Cumulative
Documents Literacy
Maxim um GPA
Maximum Maximum
Applied Math Pre-Test .65*** .28** .09 .29***
Applied Math Post-Test .93*** .43*** .28* .29***
Applied Math Maximum .43*** .22 .32***

Workplace Documents Pre-Test .23* .46*** .15 .37***
Workplace Documents Post-Test .46*** .92*** .25 .27***
Workplace Documents Maximum .43*** .22 .32***

Graphic Literacy Pre-Test .30* .14 .19 .19
Graphic Literacy Post-Test .24 .23 .96*** .36**
Graphic Literacy Maximum .22 .22 .27*

ACT Composite .56*** .53*** .40** .46***
Engage Academic Success .16* .07 .37** .52***
Engage Retention .16* .05 .35** .46***
* p < .05, ** p < .01, *** p < .001

WorkKeys Curriculum Engagement practice and quiz questions of two seconds
or less. On one hand, this observation is
Usage data from WorkKeys Curriculum
disappointing because students missed
made it apparent that some students
chances to improve their foundational
engaged meaningfully with the curriculum
workplace skills. For research purposes,
and some did not. For example, numerous
however, this provided an opportunity to
students had average response times for
examine the effectiveness of WorkKeys
ACT Research & Policy | Technical Brief | January 2020 5

Curriculum by comparing students who did The goal was to identify students who were
and did not engage. likely not engaged. Setting the cut points
higher might have identified students who
Engagement was inferred using data from were more engaged, but exploratory
the WorkKeys Curriculum dashboard analyses indicated that results were not
including number of lessons, practice sensitive to higher cut points. In most
questions, and quizzes; time spent on analysis, variables were combined because
lessons, practice questions, and quizzes; individual variables were unreliable
and percentage correct scores on practice indicators of engagement. For example,
questions and quizzes. Cut points were some students completed every practice
established through a combination of question but got only 30% correct—an
empirical and rational means. This generally indication of random guessing.
involved examining the distribution of a
variable, identifying a value that would As described above, engagement was first
isolate the group of apparently disengaged inferred by classifying students as engaged
students, and comparing that value with in the lessons (i.e., reading instructional text
expectations considering the amount of and reviewing example questions)
curricular materials and the amount of time according to the amount of time spent on
it would take to read or respond the lessons (40+ minutes). The percentage
conscientiously. For example, there was a of lessons viewed was not considered
sizeable group of students who spent less because students could easily demonstrate
than 40 minutes reviewing the lessons. substantial “progress” through the lessons
Considering that there were between 21 in a very short time. As shown in Table 3,
and 30 lessons per module, and each 90% of students engaged in the Applied
lesson entailed several pages of text and Math lessons, 77% in the Workplace
example questions, students who spent less Documents lessons, and 76% in the
than 40 minutes on the lessons could not Graphic Literacy lessons.
have reviewed many of them thoroughly.
The cut point for question response time (15 Students were classified as engaging in the
seconds) was based on the distribution of practice questions if they spent an average
response times and prior research on of 15 or more seconds responding to each
responding behavior indicative of rapid practice question, completed at least 30% of
guessing. The percentage correct cut point the available practice questions, and got at
(55%) was chosen to be higher than the least 55% correct on the practice questions.
percentage expected from random guessing The percentage engaging on the Applied
(25%) and indicative of effort to answer Math practice questions (61%) was greater
questions correctly, yet well below full than the percentage on the Workplace
mastery, which was not necessarily Documents (36%) or Graphic Literacy
expected of students. Cut points for (35%) practice questions (Table 3). The
percentage of completed practice questions main drivers of lack of engagement were
(30%) and number of quizzes (three out of short response times and low percentages
five) were selected based on distributions correct.
and what would be considered low to
WorkKeys Curriculum is divided into units
moderate effort.
based on WorkKeys levels 3 through 7, and
ACT Research & Policy | Technical Brief | January 2020 6

each unit concludes with a quiz to assess quizzes was particularly low for Graphic
mastery of associated skills. Students were Literacy (14%; Table 3). As with the practice
classified as engaged on the quizzes if they questions, students were most likely to be
spent an average of 15 or more seconds flagged for disengagement due to short
responding to quiz questions, completed response times and low percentages
three or more quizzes (out of five), and got correct.
at least 55% correct on the quiz questions.
The percentage of student engaged in
Table 3. Average Assessment Scores and Grade Point Averages
Applied Workplace Graphic
Math Documents Literacy

Mean Median Mean Median Mean Median
Average Progress 84% 100% 82% 100% 81% 100%
Total Time (min.) 229 208 117 101 122 93
% Practice Questions Completed 85% 31% 78% 100% 100% 0%
Mean Practice Response Time (sec.) 71 41 205 27 79 18
Practice % Correct 72% 74% 66% 68% 67% 71%
Quizzes Completed 4.2 5.0 4.0 5.0 4.0 5.0
Mean Quiz Response Time (sec.) 79 35 86 36 1527 20
Quiz % Correct 70% 78% 67% 75% 67% 77%
Engaged in Lessons 90% 77% 76%

Engaged in Practice 61% 36% 35%

Engaged in Quizzes 48% 40% 14%

Engaged in All 38% 22% 10%

Pre-Test/Post-Test Differences distributions4). The average differences
corresponded to moderate effect sizes.
On average, students increased their
WorkKeys scores from the initial
Figures 1, 2, and 3 illustrate changes in the
administration (pre-test) to the final
distributions of WorkKeys level scores over
administration (post-test) by an average of
time for Applied Math, Workplace
1.9 to 2.4 points on the 65–90 WorkKeys
Documents, and Graphic Literacy,
scale (Table 4). Average pre-test to
respectively. In all cases, retesting resulted
maximum score differences ranged from 2.3
in reductions in the percentages of level 3
to 2.6 points. All differences were
and level 4 scores and increases in the
statistically significant according to paired-
percentages of level 5 and level 6 scores.
sample t-tests. The average differences
Correspondingly, the percentages of
were transformed to effect sizes, which are
students with a Bronze or Silver NCRC
reported in standard deviation units (using
decreased and the percentage with a Gold
standard deviations from population
NCRC increased (Figure 4; NCRC levels
are based on maximum scores).
ACT Research & Policy | Technical Brief | January 2020 7

Table 4. Comparison of Mean WorkKeys Scale Scores
Mean Mean Difference Effect Size
Pre- Post- Pre- Pre- Pre- Pre-
Assessment N Test Test Max. Post Max. Post Max.
Applied Math 186 76.2 78.1 78.6 1.9*** 2.4*** 0.35 0.44
Workplace
159 77.9 79.8 80.2 1.9*** 2.3*** 0.37 0.45
Documents
Graphic
63 75.6 78.0 78.2 2.4*** 2.6*** 0.47 0.50
Literacy
* p < .05, ** p < .01, *** p < .001

Figure 1. Applied Math Level Score Distributions
Pre-Test Post-Test Maximum
80%
63%
60%
46%
40%
40% 34%
27%27%
19%
16%
20%
8% 8%
3% 3% 3% 3%
0% 0% 0% 0%
0%
Below 3 3 4 5 6 7
Applied Math Level Score

Figure 2. Workplace Documents Level Score Distributions
Pre-Test Post-Test Maximum
100%
82%
80%
60% 50%
44%
40% 28%28%
17% 16%16%
20% 11%
5%
1% 1% 0% 0% 0% 0% 0% 0%
0%
Below 3 3 4 5 6 7
Workplace Documents Level Score
ACT Research & Policy | Technical Brief | January 2020 8

Figure 3. Graphic Literacy Level Score Distributions
Pre-Test Post-Test Maximum
57%
60%
51%51%
43%
40%
37%
40%
20%
8%
5% 5%
2% 0% 0% 2% 2% 0% 0% 0% 0%
0%
Below 3 3 4 5 6 7
Graphic Literacy Level Score

Figure 4. National Career Readiness Certificate Level Distributions
Pre-Test Maximum
80%
62%
60%
48%
40% 34% 34%
16%
20%
4% 0% 0% 0% 2%
0%
Non-Qual. Bronze Silver Gold Platinum
NCRC Level

WorkKeys Curriculum Effect of engagement with the curriculum), the
analysis attempted to disentangle learning
The observed increases in WorkKeys scale
within and outside WorkKeys Curriculum.
scores and level scores are possibly
attributable to use of WorkKeys Curriculum.
The first analysis employed linear
However, students took the post-test an
regression—specifically, analysis of
average of six months after the pre-test, and
covariance—to examine the relationship
learning may have occurred through other
between engagement with WorkKeys
means, such as classroom instruction. The
Curriculum and post-test scores while
following analyses help address this
controlling for pre-test scores. Results
limitation by controlling for engagement with
indicated whether students who engaged
WorkKeys Curriculum. Assuming that
with WorkKeys Curriculum earned higher
students experienced similar learning
post-test scores than unengaged students
outside of WorkKeys Curriculum (regardless
with similar pre-test scores. A total of five
ACT Research & Policy | Technical Brief | January 2020 9

models were fit to the data for each statistically significant only for the Graphic
WorkKeys Assessment (see Appendix Literacy model (see Model 5 in Appendix
Table A1). The models included various Table A1). Note that the Academic Success
combinations of engagement indicators index reflects a combination of academic
(engaged in lessons, engaged in practice, discipline, social activity, and a self-reported
engaged in quizzes, and engaged in all ACT Composite score. In general, harder
three) and the ACT Engage Academic working and more conscientious students
Success index. tend to have better educational outcomes.
Thus, Academic Success was expected to
Of the engagement variables, engagement predict WorkKeys scores.
in practice appeared to have the strongest
association with post-test scores (Appendix Next, logistic regression was used to
Table A1). For Applied Math, students who examine the relationship between
engaged in the practice questions earned engagement in WorkKeys Curriculum and
post-test scores that were 1.9 points higher the probability of increasing by one level
than students with similar pre-test scores score or more (based on maximum
who did not engage (Model 3). The left WorkKeys scores). In this analysis, a
panel of Figure 5 illustrates this relationship. significant effect indicated that students who
This statistically significant difference engaged with WorkKeys Curriculum had
corresponded to an effect size of 0.35 higher odds of increasing their level scores
standard deviations (a moderate effect). through retesting than unengaged students
The expected effect of engagement in with similar pre-test scores. The general
practice questions was approximately 0.5 patterns in logistic regression results
points for Workplace Documents, but this (Appendix Table A2) were similar to the
coefficient was not significantly different linear regression results. That is, students
from zero. The models including only an who engaged in the Applied Math practice
overall engagement indicator each questions had significantly greater odds of
suggested a positive effect associated with increasing their level scores. The right panel
engagement (Model 4), but only the of Figure 5 reveals that engaged students’
coefficient for Applied Math (1.1) was chances of increasing their level scores
statistically significant (see Model 3 in were approximately 25% higher than
Appendix Table A1). Smaller sample sizes unengaged students. The negative slope
for Workplace Documents and Graphic indicates that that students with higher pre-
Literacy made it more difficult to detect test scores were somewhat less likely to
statistically significant effects. increase their level scores.5
The last model included engagement in Figure 6 illustrates how overall engagement
practice questions and the ACT Engage in the Workplace Documents curriculum
Academic Success index. The Academic materials related to expected post-test
Success and Retention indices were highly scores and the probability of increasing
correlated (r = .91); therefore, the Retention one’s level score. The effect of overall
index was left out of analyses to avoid engagement (0.73) was not significantly
multicollinearity. Consistent with the different from zero in the linear regression,
correlations reported in Table 2, the but the overall engagement coefficient in
coefficient for Academic Success was logistic regression was nearly so (p = .07).
ACT Research & Policy | Technical Brief | January 2020 10

Throughout the range of Workplace Literacy curriculum were positive and similar
Documents pre-test scores, students who in magnitude to those for Workplace
engaged in WorkKeys Curriculum had Documents, but neither was statistically
greater chances of improving their level significant (Table A2).
scores (by up to 19%). The estimated
effects of overall engagement in the Graphic
Figure 5. Applied Math Linear and Logistic Regression Plots for Engagement in Practice
Unengaged in Practice Engaged in Practice Unengaged in Practice Engaged in Practice
82 1.0
est 80 se) .8
T a
- e
st 78 cr
ath Po 76 ore In .6
M Sc .4
ed 74 el
pli ev
p L .2
A 72 P(
70 .0
70 71 72 73 74 75 76 77 78 79 80 70 71 72 73 74 75 76 77 78 79 80
Applied Math Pre-Test Applied Math Pre-Test

Figure 6. Workplace Documents Linear and Logistic Regression Plots for Overall Engagement
Unengaged Engaged Unengaged Engaged
st 82 1.0
e
T )
ost- 80 ase .8
P e
ents 78 e Incr .6
m 76 r
o
u c
Doc 74 el S .4
e ev
plac 72 P(L .2
k
r
o 70
W .0
70 71 72 73 74 75 76 77 78 79 80
70 71 72 73 74 75 76 77 78 79 80
Workplace Documents Pre-Test
Workplace Documents Pre-Test