Careering After College Establishing The Validity Of Abilities Learned In College For Later ...

170 Pages·2007·4.21 MB·English

DOCUMENT RESUME
ED 239 556
HE 016 980
AUTHOR
Mentkowski, Marcia; Doherty, Austin
TITLE
Careering after College: Establishing the Validity of
Abilities Learned in College for Later Careering and
Professional Performance. Final Report: Overview and
Summary.
INSTITUTION
Alverno Coll., Milwaukee, Wis.
SPONS AGENCY
National Inst. of Education (ED), Washington, DC.
PUB DATE
May 83
GRANT
NIE-G-77-0058
NOTE
171p.; For related documents, see HE 016 981-990.
PUB TYPE
Reports
Research/Technical (143)
EDRS PRICE
MF01/PC07 Plus Postage.
DESCRIPTORS
Academic Ability; Alumni; Cognitive Style; College
Curriculum; College Graduates; Educational Benefits;
*Education Work Relationship; Higher Education;
Learning Experience; Liberal Arts; *Outcomes of
Education; *Research Methodology; Student Attitudes;
*Student Development; *Student Evaluation;
*Validity
IDENTIFIERS
*Alverno College WI
ABSTRACT
A series of studies that investigated the validity of
abilities learned in college for later career and professional work
are summarized. After describing the rationale for validating
outcome-centered higher education curricula, the following types of
measures for program evaluation and outcome validation are discussed:
performance measures of general abilities, cognitive-developmental
measures based on developmental theory, learning style measures based
on experiential learning theory, and students' views on the outcomes
of college. Attention is also directed to the framework of research
at Alverno College, including: the types of student abilities,
outcome definitions, performance evaluation, student and alumnae
outcomes, and validity assumptions. The research methodology is
described with attention to the sample, the validation model,
longitudinal and cross-sectional approaches, the input of students
and faculty/professionals, and the research instruments. Additional
attention is focused on outcomes of the college experience and the
relationship of outcomes to the world of work. Major study findings
are summarized, and dissemination of the results is discussed.
Abstracts of the 10 research reports in this series are appended.
(SW)
***********************************************************************
*
Reproductions supplied by EDRS are the best that can be made
*
*
from the original document.
*
***********************************************************************
THE VALIDITY
CAREERING AFTER COLLEGE: ESTABLISHING
LATER
OF ABILITIES LEARNED IN COLLEGE FOR
CAREERING AND PROFESSIONAL PERFORMANCE
Austin Doherty
Marcia Mentkowski
ALVERNO COLLEGE
EDUCATION:
FINAL REPORT TO THE NATIONAL INSTITUTE OF
OVERVIEW AND SUMMARY
U.S. DEPARTMENT OF EDUCATION
NATIONAL INSTITUTE OF EDUCATION
EDUCATIONAL RESOURCES INFORMA T'ON
CENTER IERICI
his document has been reproduced as
received from the person Or
orgdrii/i11101,
oTigirlating It
Minor LhdfIVS have been Made to improve
qiiaNy
Puee, of view of Opinions stated in this
ill n it NIE
not eliesSdrily repr,sorit
mslhort or pofe-;
Funded by a grant from the National Institute of Education:
Careering After College: Establishing the Validity of Abilities
Leariied in College for Later Success
(NIEG-77-0058)
Principal Investigators:
Marcia Mentkowski
Austin Doherty
Alverno College
3401 South 39th Street
Milwaukee, Wisconsin 53215
This overview and summary is the overarching paper in the series that corr prise M. Mentkowski
the \
'y of Abilities Learned in Colley,,,
and A. D
Careering After College: Establich;,-,
I:
to the National Institute
for Later ,areering and Professional Performanc
I RL,
Education. Milwaukee, WI: Alverno Productions,
3, 1390 pages.
The research reports described in this overview and summary and included in the final report are:
One:
Friedman, M., Mentkowski, M., Earley, M., Loacker, G., & Diez, M. Validating
Assessment Techniques in an Outcooe-Centered Liberal Arts Curriculum:
Valuing and Communications Generic Instrument, 1980, 54 pages.
Two:
Friedman, M., Mentkowski, M., Deutsch, B., Shover, M.N., & Allen, Z. Validating
Assessment Techniques in an Outcome-Centered Liberal Arts Curriculum: Social
Interaction Generic Instrument, 1982, 78
pages.
Three:
Assessment Committee /Office of Research and Evaluation. Validating
Assessment
Techniques in an Outcome-Centered Liberal Arts Curriculum: Insights
From the
Evaluation and Revision Process, 1980, 5 pages.
Four:
Assessment Committee/Office of Research and Evaluation. Validating
Assessment
Techniques in an Outcome-Centered Liberal Arts Curriculum:
Integrated Competence
Seminar, 1982, 58 pages.
Five:
Assessment Committee/Office of Research and E.
Validating Assessment
Techniques in an Outcome-Centered Liberal
Arts Curriculum: Six Performance
Characteristics Rating, 1983, 74
pages.
Six:
Mentkowski, M., & Strait, M. A Longitudinal Study of Student Change
in Cognitive
Development and Generic Abilities in an Outcome-Centered Liberal Arts
Curriculum, 1g83, 357
pages.
Seven:
Much, N., & Mentkowski, M. Student Perspectives
on Liberal Learning at Alverno
College: Justifying Learning
as Relevant to Performance in Personal and
Professional Roles, 1982, 83
pages.
Eight:
Mentkowski, M., Much, N., & Giencke-Holl, L. Careering
After College: Perspectives
on Lifelong Learning and Career Development, 1983, 124
pages.
Nine:
Mentkowski, M., DeBack, V., Bishop, J., Allen, Z.,
& Blanton, B. Developing a
Professional Competence Model for Nursing Education,
1980, 74 pages.
Ten:
Mentkowski, M., O'Brien, K., McEachern, W., & Fowler,
D. Developing a Professional
Competence Model for Management Education,
1982, 317 pages.
Also available from Alverno Productions:
d.
Mentkowski, M., Mueser, M., & Strait, M. Using the Perry Scheme of Intellectual and Ethical
Development as a College Outcomes Measure: A Process and Criteria for Judging Student
Performance. Vols. I & II, 1983, 516 pages.
()Copyright 1983. Alverno College Productions, Milwaukee, Wisconsin. All rights reserved under U.S.,
International
and Universal Copyright Conventions. Reproduction in part or whole by any method is prohibited by law.
PREFACE
During our dissemination at conferences and professional meetings of some preliminary findings
based on results described in detail in this report, many colleagues expressed interest in the
broader issue of validation. Concerned about validating their own programs, they wished to
know how we conceptualized the validation of a liberal arts, outcome-centered curriculum.
We responded to this interest by describing faculty questions that stimulated the initia. :+udy
of college outcomes at Alverno and by summarizing research results based on ongoing iadulty
questions. We explained that these questions and results were placed within the context of a
validation model that has guided our efforts at Alverno College for the past seven years.
This report follows that same approach. First, this overview and summary describes the
rationale for validating outcome-centered higher education curricula and our validation model
drawn from faculty questions. We then describe the research objectives based on tnese questions
and our approaches to instrumentation and methodology. Then we synthesize major
conclusions from ten research studies that follow this overview and summary. We relate the
findings to our overall purposes and discuss implications for higher education. We also include
orts, identify our dissemination strategies and list the range of
abstracts of the research r
porations and schools whose questions and insights have con-
colleges and universities,
tributed to our efforts during these past seven years.
This report represents the collaborative work of the Alverno faculty, Office of Research and
Evaluation staff, Alverno students and alumnae, and Milwaukee organizations and professionals
Our acknowledgments to them and to our other colleagues follow the preface.
This work is dedicated to our students, whose belief in our ability to improve education gives
us the faith and courage to continue learning, and to research the penetrating questions that
challenge higher education today.
Marcia Mentkowski.
Austin Doherty
Milwaukee, Wisconsin
May 1983
ACKNOWLEDGEMENTS
For the past seven years, we have worked with cfalleagues in a variety oi roles to complete the research
outcomes reported here. We gratefully recognize all contributors to this volume, and we hope to
convey by these acknowledgements that educators and researchers in higher education can collaborate
to achieve research and evaluation outcomes that benefit all of us.
This project was undertaken with the support of the Alverno administration, faculty and students;
colleagues from across the country; the National Institute of Education; and research participants
from the Alverno student body and alumnae, and the Milwaukee business and professional community.
The acin-Aistration anti faculty of Alverno College worked in close concert with Office of Research
and Evaluation staff
create and carry out the research. Joel Read, Vivien DeBack, Mary Hueller,
Rosemary Hufker, Theophane Hytrek, Celestine Schell, Alice Theine, Christine Trimberger,
Alen Wutzdorff, and several department coordinators assitteci in contacting participants.
Ail faculty assisted in some way in this project. Those who contributed directly to the research
reports are: Zita Allen, Jean Birkey, Robert Birney, Barbara Blanton, Vivien DeBack, Bernardin
Deutsch, Mary Oiez, Margaret Earley, George Gurria, Patricia Jensen, Joan Koehler, Cathleen Krzyminski,
Dim.itri Lazo, Georgine Loacker, William McEachern, Agnes Meysenburg, Marlene Neises, James
Newton, Kathleen O'Brien, James Roth, Jean Schafer, M. Nicolette Shovar, Judith Stanley, Kyle
Stewart, Alice Theine, Christine Trimberger and Allen Wutzdorff. Those who contributed indirectly
to the research reports are: Barry Burd, Patricia Burns, Katherine Couture, Rita Eisterhold, Jane
Halonen, Ruth Hoerig, Rosemary Hufker, Anne Huston, Patricia Hutchings, Everett Kisinger,
Gertrude Kramer, Nancy Maynard, Clare Novak, Maria Terese Patterson, Penelope Reed and Ann
Schlaefer.
Several current and former members of the Office of Research and Evaluation contributed to the
research reported here at some time during the last seven years. While their responsibilities and their
contribution varied, all were committed to respect for the involvement of the participants, to high
standards for duty collection, accurate recording, and careful analysis and writing. They include
Nancy Much, Michael Strait, Deborah Fowler, James Bishop, Miriam Friedman, Mary Moeser,
Elizabeth Davies, Eunice Monroe, Laura Giencke-Holl, Nancy Miller, Mary Ellen DeHaven, Susan
McAllister, Lois Grau, ZaZa Pbpovic, Maureen Wah!, Kathleen Denny, Nevenka Davis, Jerilyn Bar,
Donna Siekert, Judith Meehan, Margaret Quinn and Delores McCrimmon. Student assistants include
Jacqueline Guiilory, Vicki Lengyel, Lisa 'Nevins, Bernadette Mayer, Sue Schultz ar J Jean VanSciver.
Laura Giencke-Noll produced the finaj report with the assistance of Margaret Quinn. Mark Hein
edited this overview and summary. Joan Hahn and staff of Secretarial Services and the staff
of Alverno Productions, Patricia Kosz in particular, enabled us over the years to communicate
our work to a range of audiences.
Many production instruments wui e used in this project ant; assessors showed ingenuity and insight
in judging performance samples from students, alumnae and professiolials. Assessors include
Zita Allen, James Bishop, Elizabeth Davies, Vivien DeBack, Deborah Fowler, Jeanne Jung, William
McEachern, Nancy Miller, Mary Moeser, Nancy Much, Kathleen O'Brien, ZaZa Popovic, Jean Schafer,
Robert Scorgie, M. Nicolette Shovar, and Michael Strait. John Gibbs and Clark Power, while at the
Center for Moral Education at Harvard University, coded the Moral Judgment Instrument. McBer and
Company of Boston coded instruments from the Cognitive Competence Assessment Battery under the
direction of Ann Litwin.
i1 i
Several of our colleagues served as formal consultants. They are F. David Bertram formerly of
th,! University
Marquette University, Mark Davison of the University of Minnesota,
'
.,y of
of Georgia, Milton Hakel of The Ohio State University, Lee Knefelf
Maryland, Marcus Lieberman of Harvard University, and Joel Moses
nrough personal
contact and conversafions., others provided special insights. They include John Gibbs of The Ohio
State University, GYen Gish of Case Western Reserve University, Douglas Heath of Haverford College,
George Klemp ofhcBer and Company, Lawrence Kohlberg of Harvard University, David Kolb
of Case Western Reserve University, Jane Loevinger of Washington University, David McClelland
of Harvard University, William G. Perry, Jr. of Harvard University, John Renner of Oklahoma State
University, James Rest. of the University of Minnesota, Paul Pottinger of the National Center for the
Study of Professions, Howard Russell of McBer and Com Pany, and David Winter of Wesleyan University.
Other directors from National Institute of Education projects investigating competence assessment
and validation met together with us in several intensive meetings and helped spur us on to critique
and develop our work. They include Ken Alvares, Sheila Huff, George Klemp, David Kolb, Sudhansu
Metra, Willa Pettygrove, Glenn Varney, Maureen Webster and Donald Wolfe.
Jean Miller, project officer from the National Institute of Education, provided vision, encouragement
and intellectual stimulation. She helped us to see the connections and relationships between ouri;work
and the contributions it might make to higher education.
i v
CONTENTS
Pae
PREFACE
ACKNOWLEDGEMENTS
ABSTRACT
1
WHY FOCUS or: OUTCOMES IN HIGHER EDUCATION?
5
WHY VALIDATE OUTCOMES IN HIGHER EDUCATION?
9
DEFINING, ASSESSING AND VALIDATING COLLEGE OUTCOMES
11
colle
Outcomes:
Frameworks and Measures
.
finking Education and Work:
Generic Abiliiv
12
Measures
CognitiveDevelopmental
Developmental Theory:
13
Measures
Experiential Learning Theory:
Learning Style
14
Measures
Competence Assessment:
Performance Interviews
14
and Inventories
Perspectives on Learning and Careering:
15
Interviews and Careering Questionnaires
Matching Framewcrks and Measur,es to Curricular Goals
and Assessment Principles
DEFINING AND ASSESSING OUTCOMES AT ALVERNO COLLEGE
17
Do We Define (.-jitcomes'2
17
What Are the Abilities or Competences?
18
Developmental Abilities
18
Holistic Abilities
19
Genetic Abilities
ri
19
How Can We Develop These Abilities?
20
How Will We Know a Student Has Achieved erhese
Abilities?
20
Criteria
21
Multiple Judgments
.
22
Alternate Performance Modes
22
Expert Judgment
22
SelfAssessment
What Are Student Outcomes of the Learning Process?
23.
What Are Alumnae Future Outcomes?
,23
4
24
Learning Process?
What are the Components of
v
OUTCOMES
ESTABLISHING THE VALIDITY OF
26
Component
Fstablishing Evaluation/Validation as a
of the Learning Process
28
Identifying Assumptions About. Validity
28
Validation is Developmental
Process
Validation is an Illuminative, Diagnostic
29
and
Validation Relates Theory to Practice
29
Research to Evaluation
Validation is Contetual
30
31
Defining Validity
32
Desi.,u1-13ased Validity
,-!or!-.Ince-Based Validity
36
leloatlHiu,:. Validation Questions
Validity
Faculty Questions for Establishing
37
RESEARCH OBJECTIVES AND REPORTS
RESEARCH METHODOLOGY
ct
Sample
Characteristics of the Validation ',1,
'
49
Correlational caher A
im,
.-1p,'
i
5U
T\
Model
An Aggregate
nrignlai-
52
The Validation ?.Ndel
53
Characteristics of Research Designs
53
Longitudinal and Cross-Sectional Approaches
55
Fetal Sampling
55
Cohort
and
55
Class Cohort
55
Time Series Design
'58
Achievement Cohort
58
D,gree Completion
Matching Comparison Croups for
59
Participation
11)4 r asing Rate a
59
Procedures
60
Res,
Create a.ConiTt for Validation
Ii
60
and
as of Students, Facu Ly
Respond to C
,essionals
P:
61
Faculty Involvement
62
Student Involvement
62
Involvement of Professionals
62
and
Collaborate with Colleagues in Research
Curriculum Development
63
Program
Respect the Values and Coals of the
6.3
Use Ekisting Program Evaluation Structures
64
Provide Feedback for Program Improvement
vi
rage
Respect the Values and Coals of Research
65
Participants
Contacting Participants
65
Communicating Rationale and Confidentiality
66
Feedback on Study Results
67
Rate of Participation Achieved
67
Choosing, Creating, Validating and
Scoring
68
Instruments
Characteristics of InstruMents
68
Types of Measures
69
Recognition and Production Measures
69
Developmental Measures
71
External Criterion Measures
71
Create and Validate Instruments
72
Score Instruments
73
Select Data Analysis Strategies
75
Instrument Descriptions
76
Human Potent
Cogubt\.
76
Jevelopment
,sure
Human P,
Learning Styles
77
Human P
i Measure
Generic Abilities
77
Measures
tudent Per
,rmance in the Learning Process
78
,
Me-qtres of Student/Aluml.a Perceptions of Learning and
80
.:veering
81
Measures of Professional Performance, Perceptions, and
Professional Development
85
WHAT ARE THE OUTCOMES OF A COLLEGE EXPERIENCE?
Student Change on College Defined Abilities
85
Performance in the Learning ProceSs
85
Are Measures Valid?
Is Expert Judgment Reliable?
36
Integrated Abilities
86
Faculty Rating of Student Development
87
Generic Abilirios: ,Communications, Valuing,
87
tion
Creating Strategies for Evaluating and
88
Revising Instruments
Student Change on Broad, Developmental Frameworks
98
Examples of Patterns in Student Change
91
Cognitive Development
-91
Learning Styles
93
Student Perceptions of Learning
94
Page
TO THE WORLD OF WORK?
97
HOW DO COLLEGE OUTCOMES RELATE
Careering
.97
Alumnae Perceptions of Work, Learning and
98
Alumnae TransFer Abilities
ALi
m1:,1:11e .md A1.1t,.t
u,,11).ie
\
oo
co( limed Ee.frniu
lumo.1;
nu
`lore r(Irm I
I'll l ng
AI
El' :
99
Alumnae Experience Competence
100
Abilities Used by Professionals
100
for
Competence Models of Effective Performance
Higher Education Programs
101
Abilities Develop Through Education and
Experience
101
Technical Skills are not Enough
102
Both Perceptions and Pecrormance are Important
Sources for Validating Outcomes
102
!elrning is Critical
103
WHAT HAVE WE LEARNED?
103
Outcomes are Complex, Holistic Human Abilities
104
Outcomes are Developmental
104
Instruction
Outcomes Develop as the Result of
105
Outcomes Develop' at Different Times
105
Developmental Patterns are Subtle and Complex
106'
Growth
There are Individual Differences in
Patterns
107
H-1 Professional
College Outcomes Relate to Careering
Performance
.
107
Abilities Identified by Liberal Arts Educators.
Prof(;sSionals
are Demonstrated by
108
Abilities Need to be Adapted
108
Learning Continues After College
108
Abilities Learned in College are Used by
Alumnae
109
Competence is a Concept and an Experience
109
Methodological
Enhancing Feasibility Through
Contributions
109
ProdUction Measures'
Using Expert Judgment in
110
Validating Non-Traditional Assessment
TeOniques
111
Testing Out New Measures of Colleg._ Outcomes
111
Defining Validity as a Concept for Higher
° -
Education
112
Acccmplish Its Own
Can a Liberal Arts College
Evaluation and Validation?
113
Developing Participant Involvement Strategies
113
Researchio.g Women's Abilities
11,4
Disseminating .Results to Ensure Implementation
viii