ERIC ED610399: Rethinking Completion Analytics to Better Support the Student Experience across Diverse Ecosystems. The AIR Professional File, Fall 2020. Article 150

2020·0.56 MB·English

by  ERIC

The AIR
Professional File
Fall 2020 Volume
Supporting quality data and
decisions for higher education.
Copyright © 2020, Association for Institutional Research
LETTER FROM THE EDITOR
Welcome to the Fall 2020 volume of the AIR approach to better understand brain drain and
Professional File! In this volume, you will find two mitigate its impact.
articles whose authors use large databases from
community colleges and statewide systems to Wherever and however you are working these days,
address issues with significant policy implications. I hope you are healthy and happy. Soothe the stress
of this ongoing health crisis by treating your brain to
It seems that debates have raged for decades over some new ideas. Share yours with your colleagues
how to use completion rates to evaluate college by sending a manuscript to the AIR Professional File.
performance. No matter what formula is adopted,
the resulting computations often fail to reflect our Sharron Ronco
diverse missions. What if we used a student-focused Coordinating Editor
methodology instead? In his article, Rethinking
Completion Analytics to Better Support the Student
Experience across Diverse Ecosystems, Jeffrey L. Cornett
proposes an approach to completion rates that
Editors
adjusts to the institution’s ecosystem and range of
students we serve. Check out his CAT-scan graphics! Sharron Ronco
Coordinating Editor
In their study Brain Drain in Maryland: Exploring Marquette University (retired)
Student Movement from High School to Postsecondary
Education and the Workforce, Amber Bloomfield, Stephan C. Cooley
Bess A. Rose, Alison M. Preston, and Angela K. Managing Editor
Henneberger examine the migration of students Association for Institutional Research
who leave the state to attend college elsewhere.
Which students are most at risk of not returning ISSN 2155-7535
to the state workforce? Their analysis comprises a
unique application of propensity score matching
Table of Contents
4 Rethinking Completion Analytics to Better Support the Student Experience
across Diverse Ecosystems
Article 150
Author: Jeffrey L. Cornett
28 Brain Drain in Maryland: Exploring Student Movement from High School to
Postsecondary Education and the Workforce
Article 151
Authors: Amber Bloomfield, Bess A. Rose, Alison M. Preston, and Angela K. Henneberger
49 About the AIR Professional File
49 More About AIR
Rethinking Completion Analytics
to Better Support the Student
Experience across Diverse Ecosystems
Jeffrey L. Cornett Abstract
Single measures of college performance fail to
reflect the mix and diversity of students served.
About the Author First-time students vary widely in terms of their
readiness to succeed in college. Higher education
Jeffrey L. Cornett, PhD, was executive director of
environments (ecosystems) also vary widely and
institutional research at the central region of Ivy
afford students differing admission and transfer
Tech Community College of Indiana from 2011
opportunities. These factors confound college
to 2017. He also served in the same capacity at
completion analytics when those who define college
Valencia College of Florida from 2007 to 2011. He
goals focus only on what is best for the college. A
is an expert in information engineering and the
fresh way to rethink completion analytics would
graphical display of data insights.
be to measure college completion success from
the student point of view and how they wish to
Acknowledgments experience college. Student-focused completion
rate scorekeeping would improve research into best
This research would not have been possible without
practices. Student advising would also improve if
the kind and thoughtful support of several people.
completion analytics were disaggregated along a
I thank Dr. Sanford Shugart, president of Valencia
continuum of readiness to succeed.
College, for teaching me to value the student
experience and the importance of finding ways to
measure it. I am grateful to Dr. Sue Ellspermann,
Keywords: student experience, college readiness,
president of Ivy Tech Community College of Indiana
CAT-scan graphs, college ecosystem, sister university,
for encouraging me to publish this article, and for
transfer rates, 1+3 transfers, 2+2 transfers, second-
allowing me to reveal all the Ivy Tech Central Indiana
chance transfers, completion rates, vertical lift, right
data shown in this paper.1 Finally, Stephen Hancock
shift, institutional research, student advising
earns my thanks for devoting several years of his
career to building our CAT-scan data mart. This was
1. The Central Indiana service area of Ivy Tech Community College includes
an incredibly time-consuming and frustrating task
Indianapolis (Marion County) and eight surrounding counties (Boone,
Hamilton, Hancock, Hendricks, Johnson, Morgan, Putnam, and Shelby). In
since I continually added more layers to the CAT-
2017 Ivy Tech was reorganized statewide to formally eliminate the regional
layer of management. The name “Ivy Tech” in this article refers to the Central
scan graphs.
Indiana region or service area of that college, unless otherwise noted.
The AIR Professional File, Fall 2020 https://doi.org/10.34315/apf1502020
Article 150 Copyright © 2020, Association for Institutional Research
Fall 2020 Volume 4
all differences in performance outcomes to
INTRODUCTION
be attributed to the college, and none to the
In my first meeting with our college’s new chief students. This oversimplifies comparisons
information officer I began to explain problems across colleges and the interpretation of results.
with completion performance scorekeeping. It is too easy to presume that higher-scoring
After patiently listening for a while, he interrupted colleges must simply have better leadership,
to challenge me to prescribe the exact college academics, or support programs, and that
scorekeeping formula that I proposed. At first, I lower-performing colleges must be doing
did not know how to respond. Just changing the something dramatically wrong or must be
completion formula does not solve the problem. missing major improvement opportunities.
After pausing to think for a moment, I suggested
3| Single measures of college performance can
that his question was the wrong one to ask. When
mislead students and their advisors. Completion
judging success, it is not the college performance
analytics that adjust to the mix of students
that matters—it is the student performance! We
served could also provide data that allow
should try to analyze and improve how the student
student advising to be better tailored to the
experiences college, and not merely how the college
individual student. Students are not all equally
experiences students.
likely to succeed in college. Individual students
vary in backgrounds, motivations, and abilities.
Single performance measures for the college
PROBLEM STATEMENT
reflect the outcome for an average student
at that college, and not a diverse range of
Completion analytics that focus on the college
individual students with their varying goals and
experience rather than the student experience
conditions of learning.
present us with three problems:
1| Students’ goals might not match the college’s
goals. Colleges and their scorekeepers naturally RESEARCH HYPOTHESIS
want to experience students who have high
Institutional research, college benchmarking,
rates of retention and completion. Students,
and student advising could all be improved if
however, when given the opportunity to
performance metrics were stratified along a
transfer, might have different goals that make
continuous scale that reflects the range of diversity
them choose to transfer before graduating from
in a student’s readiness to succeed.
their first school attended.
2| Student mix varies across colleges. When This paper provides supporting evidence for this
analyzing college performance, it is much hypothesis through consideration of five research
too convenient to incorrectly assume that questions (see below for definition of terms):
all colleges have the same mix of students
operating within comparable higher education 1| Are 1+3 transfers successful at their transfer
environments. This false assumption allows destination schools?
Fall 2020 Volume 5
2| How much does the student experience vary A third type of transfer are second-chance
between college-ready and college-prep students? transfers, defined as students who did not do well
academically at their first college and who earned
3| How does an ecosystem affect transfer behavior
very few credits before leaving. After leaving and
and student mix?
perhaps even several years later, they found
4| Can we improve our understanding of college
a second chance to continue their education
readiness beyond just a ready-or-not split?
elsewhere. Because second-chance transfers were
5| How do you show performance improvement
not academically successful at their first school,
when there are differences in student mix?
these transfers should not be added into transfer-
adjusted completion rates.
TERMINOLOGY
College-Ready vs. College-Prep Students
Ecosystem Not all high school graduates are academically
prepared for college-level coursework. Community
The local mix of higher education opportunities
colleges sometimes use a placement test for their
varies for students across colleges and universities
first-time-in-college students to identify those who
in different regions. A different mix of competing
would benefit from developmental coursework in
colleges, educational programs, and enrollment
basic reading, writing, and math before they take
capacities can result in one region having a much
more-advanced college-level courses.
different mix of first-year students who begin at
community college rather than those who begin at
the local state university. Second- and third-year
THEORY
educational pathways and capacities can also vary
by region, resulting in much different student Florida state colleges include some of the most
transfer behavior. respected community colleges in the country. Higher
graduation rates lead to frequent performance
awards, as well as the ongoing study by consultants
Three Types of Transfers: 2+2, 1+3, and
and benchmarking organizations to learn the secrets
Second-Chance Transfers
to Florida’s success. Valencia College in Florida is
For those seeking a 4-year degree, 2+2 transfers viewed as an exemplary role model, and that college
are students who begin college at a 2-year school won the inaugural Leah Meyer Austin Award in
and graduate there first before transferring to a 2009 and the inaugural Aspen Prize for Community
4-year school. College Excellence in 2011.
1+3 transfers are students who did well academically Austin Community College in Texas, with high
at their first 2-year school, but who did not transfer rates and low graduation rates, provides
graduate there. Instead they chose to transfer to an interesting contrast to Valencia College.
another school to continue their education toward a Austin’s scorekeeping problem was addressed
4-year degree. by the president of Valencia College in his article,
Fall 2020 Volume 6
“Rethinking the Completion Agenda” (Shugart, 2013). is an award-winning school with comparatively high
Shugart’s article introduces the term “ecosystem” rates of retention and graduation.
to define the higher education environment and
how students behave in different ways across Valencia College provides an ironic contrast in
diverse ecosystems. He offers a compelling essay strategies to Ivy Tech. The number one strategic
that explains how low graduation rates can be priority at Valencia College is on student learning,
driven by high transfer rates attributable to the local and only secondarily on college completion
environment (ecosystem) and are not due performance metrics. Valencia leadership believes
to a deficiency in college leadership. He suggests the that greater learning will lead to better student
need for more-collaborative research of performance. Conversely, with its low completion
student behavior across colleges, including the rates Ivy Tech has always been compelled to focus
development of measures of the performance of the its strategic priorities on improving retention and
entire ecosystem. completion. How do you get students to stay
until graduation, and how do you get students to
This paper builds on Dr. Shugart’s working theories graduate sooner so that their completion can be
and provides additional supporting data. included within the limited number of years used for
performance scorekeeping?
METHODOLOGY
Unique Sources of Data
During our study both colleges had good sources
Case Studies
of data and capable institutional research (IR)
Two community colleges are the case studies for this departments. This allowed us to analyze and
paper. These two schools operate in much different compare the student experience in ways that go well
higher education environments, but both are greatly beyond traditional college scorekeeping formulas.
affected by the admissions practices of their local
sister state universities. Ivy Tech had built a data mart that tracked all first-
time-in-college students over an 11-year period
The Central Indiana region of Ivy Tech Community using data from the National Student Clearinghouse
College is in an environment where it is easy (2005–15). This allowed us to study transfers in
for students to transfer out before graduation. terms of their long-term outcomes and to explore
Neighboring state universities do not require a whether the individual student’s choice to transfer
degree before they admit successful students as was beneficial or harmful to that student.
transfers. Ivy Tech’s retention and completion rates
are scored very low. Valencia College routinely differentiated graduation
rates between college-ready and college-prep
Valencia College in Central Florida is in an students. Ivy Tech adopted a similar approach for
environment where state policy strongly discourages internal research and reporting. This differentiation
state universities from admitting community college enabled us to make comparisons in and across both
transfers that did not first graduate. Valencia College schools on how student readiness to succeed affects
graduation rate performance.
Fall 2020 Volume 7
Each community college had a close relationship reported. It helps to graphically show all student
with its local sister state university. This included outcomes in layers that add up to 100%. If you
coordinated research on issues of common want more students to graduate, a 100% layered
interest. The admissions practices of local sister graph encourages you to consider how to get fewer
state universities affect the aptitude mix of students students to do anything else.
attending the local community college. In both
Central Indiana and Central Florida, applicants Ivy Tech pioneered the use of CAT-scan (Cohorts
who are not admitted to their state university as Across Time) graphs to show the diverse ways
freshmen often start at the local community college students experience college. A CAT-scan graph
with the goal of transferring to the local state simultaneously shows all student outcomes for all
university as soon as they are allowed to do so. Data cohort outcome years. Ivy Tech built these CAT-
on the aptitude or achievement levels of entering scans (Cornett & Hancock, 2015) by first creating
students at sister state universities are available a data mart that matched up all annual starting
online from websites that have been designed to aid cohorts of students against each student’s
students in college selection. cohort-year outcomes as recorded in the National
Student Clearinghouse.
Valencia’s sister university is the University of
Central Florida (UCF), a school with high admission Ivy Tech IR staff invested many months of effort to
standards and low transfer-out rates. Due to rapid design and create their original CAT-scan data mart.
growth, UCF has always had very limited enrollment Their effort increased every year as the variety of
capacity to accept transfers-in as sophomores. reported student outcomes expanded. Once the
data mart had been created, the big reward was
Ivy Tech’s sister university is Indiana University– the ease by which precomputed outcomes could
Purdue University Indianapolis (IUPUI), a regional instantly be reported and disaggregated across any
campus shared by Indiana University and Purdue student group or research treatment variable. The
University. As a regional campus, IUPUI’s admissions data mart was delivered in a spreadsheet that
standards are naturally lower than their two could be explored using simple pivot-table methods.
separate main campuses. IUPUI, with high rates In a single afternoon, IR staff can produce scores of
of transfers out (including transfers to one of their CAT-scan graphs. The hardest part is labeling the
main campuses), had the available enrollment graphs and delivering the results so that users
capacity to easily accept students to transfer in well can understand all the information made available
before those students had completed a degree. to them.
Figure 1 is a basic 7-layer CAT-scan graph. Three
CAT-Scan Methodology different types of transfers are shown: 2+2, 1+3, and
second-chance transfers. When studying a CAT-
The first question asked when considering a
scan, it is useful to consider how results vary across
graduation rate is, “Why isn’t it higher?” The simple
cohort outcome years. In this example, by the end
answer is that the students did something else.
of Year 3 few students have graduated (3.8% = 1.2%
Yet it is rare to see a graduation rate reported at
+ 2.6%; areas in light green and dark green), while
the same time that all other student outcomes are
more have had a successful 1+3 transfer (11.3%;
areas in yellow).
Fall 2020 Volume 8
Figure 1. This 7-layer CAT-scan combines 11 starting cohorts across 9 cohort outcome years.
How Students Experience College In a "1+3Ecosystem"
students are free to transfer at any time:
7-Layer CAT-scan: Measures 7 major categories of student outcomes over time. 2nd Chance transfers = those who failwith us, then transfer
Degree-Seeking only Ivy Tech -Central Region only 1+3 transfers= successfulstudents who transfer with no degree
College-Ready or not; Full or Part-Time; FTIC only; Combined 2004-2014 Fall starting cohorts; Outcomes to Fall 2015 2+2 transfers = successfulstudents who graduate, then transfer
100%
95%
Failed Academically
8950%% 29.0% 28.9% 27.1% 25.4% 24.0% 23.1% 22.1% 21.3% 20.5% Perhaps "life intervened"
80%
"2nd Chance" Transfers
75% 4.9% 5.7% 6.7% 7.5% 7.9% 8.2% Were failing with us, then transferred
70% 1.7% 3.0% 4.0%
65%
60% 17.5% Dropoutsand Stopouts
55% 28.5% 33.4% 35.8% 36.0% 35.2% 34.3% 34.0% 34.3% Successful students but "life intervened"
50% 4.4%
45%
40% "1+3" Transfers
35% 8.4% Successful students
but no Associate's Degree
30% 11.3%
13.6% 15.2% 16.5% 17.4% 17.9% 18.1% Retained
25%
47.3%
Continuing with us that outcome year
20%
15% 30.1% 20.4% 12.8% 7.6% 5.6% 4.6% 3.5% 3.0% "2+0" Graduates
Did not transfer after graduating
10% 8.8% 9.8% 10.9% 12.1% 12.5%
5% 5.5% "2+2" Transfers
0% 00..47%% 12..26%% 2.1% 2.7% 3.1% 3.3% 3.3% 3.4% Graduated, then transferred
Year 1 Year 2 Year 3 Year 4 Year 5 Year 6 Year 7 Year 8 Year 9
n=31,584 n=28,177 n=24,418 n=20,671 n=17,664 n=15,196 n=11,861 n=8,650 n=6,081 Academic "success" means the student
Cohort Outcome Years (as of Fall) since first starting at Ivy Tech earmed a final GPA of at least 2.00 or
earned 12 total credit hours with us.
Figure 1
IPEDS (Integrated Postsecondary Education academically show up as transfers in National
Data System) serves as the national standard for Student Clearinghouse data. By Year 8 many initial
benchmarking the performance of colleges. The college failures can be seen to have returned to
National Center for Education Statistics (2020) school elsewhere to give college another try.
recently began reporting community college
completion outcomes out to 6 and 8 years instead Answering one research question often leads to
of just 3 years, as was their previous standard. asking two new questions, and new questions
quickly arise when first considering a 7-layer scan.
As seen in Figure 1, Ivy Tech’s 8-year outcomes show The National Student Clearinghouse provides data
an obvious improvement in graduation rates over that allow researchers to know much more about
3-year outcomes. Even so, after 8 years the 1+3 student outcomes. Additional research variables can
transfers (17.9%) are still more common than all be linked from a college’s own data warehouse. Ivy
graduates (15.4% = 3.3% + 12.1%) from Ivy Tech. Tech eventually built a 23-layer CAT-scan to study a
much richer variety of student outcome questions
Second-chance transfers also increase over time. (see Figure 2).
In the first outcome year, few students who fail
Fall 2020 Volume 9
Figure 2. A 23-layer CAT-scan shows many more types of outcomes than a basic 7-layer CAT-scan.
23-Layer CAT-scan (Cohorts Across Time) Within this 23-Layer CAT-scan,
there are three types of transfers:
23-Layer CAT-scan: Measures a more complete combination of student experiences over time. 2nd Chance transfers = those who failwith us, then transfer
Degree-Seeking only Ivy Tech -Central Region only 1+3 transfers= successfulstudents who transfer with no degree
College-Ready or not; Full or Part-Time; FTIC only; Combined 2004-2014 Fall starting cohorts; Outcomes to Fall 2015 2+2 transfers = successfulstudents who graduate, then transfer
100%
23 Dropout while Failing 0 credits
95% 12% 11% 10% 10% 9% 9% 9% 9% 9%
22 Dropout while Failing 1-11 credits
90%
21 Stopout while Failing 0 credits
85% 11% 12% 12% 12% 12% 12% 11% 11% 10% 20 Stopout while Failing 1-11 credits
80% 15 Failed > 2nd Chance Transfer > Dropout
7705%% 233%%% 333%%% 3221%%%% 3222%%%% 3211%%%% 33111%%%%% 331%%% 342%%% 342%%% 1143 FFaaiilleedd >> 22nndd CChhaannccee TTrraannssffeerr >> CGorandtiunautieng
65% 19 Dropout while Successful <30 credits
11%
18 Stopout while Successful <30 credits
60% 19% 22% 23% 24% 24% 24% 23% 23% 17 Dropout while Successful 30+ credits
55% 6%
16 Stopout while Successful 30+ credits
50% 4% 1% 12 Success <30 credits > 1+3 Transfer > Dropout
45% 7% 6% 5% 4% 2% 2% 1% 11 Success 30+ credits > 1+3 Transfer > Dropout
40% 2% 4% 6% 7% 7% 7% 8% 10% 10 Success <30 credits > 1+3 Transfer > Continuing
1223350505%%%%% 45% 1626%%% 78302%%%%% 744122%%%%%% 63235213%%%%%%%% 54321133%%%%%%%% 4532201231%%%%%%%%%% 4542201121%%%%%%%%%% 353130122%%%%%%%%% 000000987654 RRSSSGuuueeracccttaacccdeeeiiunnsssaeessst dd3<3e 300 >>>0++ CCC cccooorrrnneeenttdddtiiinniiintttuusssu iii>>>nnn gg111g +++ <3w33330i 0+tTTT hrrrccaaa rrueennnsddsssfffiieeettssrrr >>> GCGorraanddtiuunaauttieeng
10% 14% 13% 81%% 81%% 91%% 10% 12% 12% 03 Graduate & Done
5% 5% 02 Graduate > 2+2 Transfer > Continuing
0% 2% 01%% 210%%% 02%% 12%% 12%% 12%% 22%% 22%% 01 Graduate > 2+2 Transfer > Graduate
Year 1 Year 2 Year 3 Year 4 Year 5 Year 6 Year 7 Year 8 Year 9
n=31,584 n=28,177 n=24,418 n=20,671 n=17,664 n=15,196 n=11,861 n=8,650 n=6,081 Academic "success" means the student
Cohort Outcome Years (as of Fall) since first starting at Ivy Tech earmed a final GPA of at least 2.00 or
earned 12 total credit hours with us.
Figure 2
In this 23-layer CAT-scan, 1+3 transfers are For example, Indiana considered the creation of a
disaggregated into three transfer-destination 30-credit transfer certificate that could be earned by
outcomes: students who have graduated, those students who completed the right mix of freshman-
continuing in school somewhere, and those who have level courses. That certificate would be interpreted
dropped out. Each of these three categories is further by scorekeepers as a type of pretransfer completion
split between those who earned at least 30 credits credential. The use of 30-credits-earned layers in
before leaving and those who transferred out before the 23-layer CAT-scan shows how the introduction
earning 30 credits. This combination of six outcomes of a 30-credit transfer certificate might affect college
within the 1+3 transfer type allows us to explore a completion rates, and how these rates would vary
wide range of what-if scorekeeping questions. depending on how many cohort years are counted
in the scorekeeping.
Fall 2020 Volume 10