ERIC EJ840077: Perceived Growth versus Actual Growth in Executive Leadership Competencies: An Application of the Stair-Step Behaviorally Anchored Evaluation Approach

2007·0.14 MB·English

by  ERIC

PERCEIVED GROWTH VERSUS ACTUAL GROWTH IN EXECUTIVE LEADERSHIP
COMPETENCIES: AN APPLICATION OF THE STAIR-STEP BEHAVIORALLY
ANCHORED EVALUATION APPROACH

Michael J. McCormick, Assistant Professor
Kim E. Dooley, Associate Professor
James R. Lindner, Associate Professor
Richard L. Cummins, Executive Director
Texas A&M University

Abstract

The purpose of this study was to describe student learning in executive leadership core
competencies after being engaged in a two-semester leadership education sequence. The
researchers used evaluative research techniques to compare perceived and actual growth in
learning of executive leadership competencies. Data collection consisted of a post-then (or
Before-After) self-assessment instrument and an evaluation rubric that was designed to score the
final exam. The results were reported as average learning scores based upon (Before-After)
measures, narrative verifications, and comparisons of perceived versus actual growth. Students
varied in their perception of their competencies using only the behavioral anchors. Based on the
findings of this study, new competency-based behavioral benchmarks were developed, at the
near novice, proficiency, and near expert levels of competence. Recommendations to insure the
fidelity of students’ self-assessment and appropriateness of the behavioral benchmarks were
provided.

Introduction agricultural education departments (Fritz &
Brown, 1998). A vital part of creating a

successful leadership education process is
A recent survey conducted by the
the evaluation component. As Townsend
industry publication, Training, found that
(2002) observed: “Leadership educators may
among training professionals leadership
be in the perfect position to evaluate
development is now their number one
programs as well as create them. Seeking
concern (Hall, 2005). Leadership education,
accountability in organizational leadership
likewise, is a high priority for colleges and
[development] efforts is critical to sustaining
universities given students’ interest in
the efforts of leadership education
enhancing their leadership capabilities and
programs” (p. 3).
employers’ demand for graduates with these
For a variety of reasons, however,
skills (Carry, 2003; Coplin, 2003; Levine,
leadership program evaluation (like most
2005; Mangan, 2002). Over 900 leadership
training programs) receives far less
programs now operate in higher education
attention. One reason for this is practical.
settings, and, furthermore, they vary in
The training community is rewarded
complexity from simple workshops to
primarily for designing and implementing
doctoral programs (Mangan). In a recent
human resource development activities, not
keynote speech, Acker (2005) noted that the
measuring their outcomes. A second factor
number one thing agricultural and extension
is the difficulty involved in accurately
educators can do to make a difference in the
assessing changes in individuals’ attitudes,
world is to “invest in student leadership”
beliefs and behaviors. Such undertakings are
(p. 12).
usually labor-intensive and time-consuming.
The challenge of providing productive
Thus, ease of use becomes an important
leadership development activities for college
consideration (Rohs, 2002). Without reliable
students has been taken up by a growing
and valid program evaluation, however,
corps of leadership educators. Many are in
Journal of Agricultural Education 23 Volume 48, Number 2, 2007
McCormick, Dooley, Lindner, & Cummins Perceived Growth Versus Actual Growth…
leadership learning will continue to occur defined as “characteristics of core
mainly in “an unplanned and serendipitous competencies associated with the mastery of
way” (Boyd & Murphrey, 2002, p. 36). content” (Dooley & Lindner, p. 25). A self-
Therefore, a need exists for an easy-to- assessment instrument was created based
implement but reliable and valid evaluation upon the core competencies and behavioral
methodology for measuring leadership anchors. The instrument used a stair-step
education results. figure to visually represent progression from
For departments of agricultural novice (0) to expert (7). The behavioral
education (broadly defined), the assessment anchors were presented to the students as a
and evaluation of both student and program self-reflective guide for them to base their
success is particularly relevant. In their before-and after-training evaluations of their
study of traditional departments of competencies.
agricultural education, Fritz and Brown Open-ended narratives were analyzed to
(1998) found that nearly 70% were offering describe a change in learning and to develop
leadership and human resource development competency-based behavioral anchors to
classes. They further determined that these serve as authentication tools. Competency-
courses were generating a substantial based behavioral anchors were defined as
portion of departments’ credit hour “performance capabilities needed to
production. For instance, in 28% of the demonstrate knowledge, skill, and ability
departments, leadership courses accounted (competency) acquisition” (Dooley &
for more than half of all contact hours Lindner, 2002, p. 25). Competency-based
generated. In recognition, some departments behavioral anchors were created for
have changed their names to include competency levels 2, 4, and 6.
“leadership” as one of their content areas.
Leadership training and development has Executive Leadership Competency Model
become a very large part of what current In a study on the need for leadership
agricultural education departments are training for students in a land-grant college
doing. It follows, therefore, that creating of agriculture, Schumacher and Swan (1993)
good assessment methods for evaluating offered a number of recommendations. Two
leadership development among key recommendations from Schumacher and
undergraduate students is essential for the Swan were a need for the “development and
continued success of agricultural education adoption of required leadership development
departments. programs for their students” (p. 9) and a
need for research “designed to identify how
Conceptual Framework the Agricultural Education profession can
best facilitate the leadership skill
The conceptual framework for this study development of students...” (p. 9). These
combines the behaviorally anchored self- authors noted, further, that Departments of
assessment methodology introduced by Agricultural Education (broadly defined) are
Dooley and Lindner (2002) with a uniquely qualified to lead larger efforts to
competency model of executive leadership provide for student leadership development
developed by Cummins (2005). and training. Others have also expressed this
view. For example, Birkenholz and
Stair-Step Assessment Method Schumacher (1994) noted that:
Dooley and Lindner (2002) developed a
self-assessment protocol to document The demand for future agriculture leaders
growth (learning) in distance education core is great... Changes in agriculture will
competencies in a graduate course. In undoubtedly create an environment in
developing their assessment tool, Dooley need of strong leadership. Educators in
and Lindner used content analysis to identify agriculture must recognize that need and
six core competencies areas for distance implement strategies to develop leaders
education professionals. Then they who are able to effectively guide and
developed behavioral anchors for each core direct the industry in the future (p. 8).
competency. Behavioral anchors were However, before a successful leadership
Journal of Agricultural Education 24 Volume 48, Number 2, 2007
McCormick, Dooley, Lindner, & Cummins Perceived Growth Versus Actual Growth…
development process can be put in place, a identified by Pernick (2001) as one of the
reasonably valid leadership competency ways in which organizations
model is needed. Having such a model determine critical leadership competencies;
enables the leadership educator to design that is, use generic leadership
and deliver effective leadership development competencies found in theory and the
activities as well as specify evaluation leadership literature. They were: (1) Stay
criteria. Focused on Purpose, (2) Communicate
To address the need identified by Purpose and Vision, (3) Establish and
Schumacher and Swan (1993), Cummins Maintain Environment, (4) Develop Others,
(2005) conducted a content analysis of the (5) Lead Teams, and (6) Make Decisions in
literature on executive leadership (similar to a Complex Environment (Ambrose, 1970;
what Dooley and Lindner did in their study Barnard, 1968; Bossidy, Charan, & Burck,
of distance education core competencies) as 2002; Hanson, 1999; Janis, 1983;
well as accounts of historical events and Keegan, 1988; Kouzes & Posner, 1988;
biographies of historically significant Lansing, 1999; Pressfield, 1998;
leaders. He identified six salient competency Watkins, 2003). Table 1 provides
clusters associated with effective operational definitions of each executive
executive leadership. The method Cummins competency.
used was similar to the technique

Table 1
Core Executive Leadership Competency Operational Definitions
Executive Leadership
Core Competency Operational Definition of Construct
Stay Focused on Ability to sort through incomplete, inaccurate, and sometimes
Purpose conflicting data and determine what should be done

Communicate Purpose Skill to express to others what the intended outcome will look like if
& Vision we get it right

Establish & Maintain Competence to put into place the behaviors, standards or
Environment performance, training and feedback essential to attain the goals of
the organization

Develop Others Expertise to increase the knowledge and skills of others in order to
enhance their capacity to perform and to utilize their leadership
ability for the organization

Lead Teams Capability to direct the work of others in the pursuit of a common
goal acceptable to all

Make Decisions in a Proficiency to take into account both the known and the unknown
Complex Environment and make informed and thoughtful decisions

It is noteworthy that the six executive (2004) for extension directors and
competencies identified by Cummins (2005) administrators if one re-classifies the 80
match reasonably well with the executive specific competencies into the six
competencies identified by Moore and Rudd competency cluster proposed by Cummins.
Journal of Agricultural Education 25 Volume 48, Number 2, 2007
McCormick, Dooley, Lindner, & Cummins Perceived Growth Versus Actual Growth…
For example, the competency “Demonstrate competencies among undergraduate students
Respect for Others (p. 27)” identified by taking a leadership education sequence. Data
Moore and Rudd (2004) can be collection consisted of a Before-After self-
encompassed in the “Establish and Maintain assessment instrument. A scoring rubric was
the Environment” competency from the employed to grade students’ final exams to
Cummins (2005) executive competency assure grading consistency and control for
model. Another example is “Think instructor bias, since the instructor was one
Strategically (p. 27)”, which fits well within of the researchers. Also, while the first
the “Make Decisions in a Complex author may have been the instructor, the data
Environment” competency in the w a s analyzed and interpreted by the second
Cummins model. As previously noted, and third authors, which is a recommended
having a competency model enables the technique for controlling experimenter’s
leadership educator to design and deliver bias (Borg & Gall, 1989). In addition, the
effective leadership development activities instructor had consistent opportunities for
as well as specify evaluation learning observation and interaction with the learners
criteria. (prolonged engagement).
Based on the set of competencies
Purpose identified by Cummins (2005), a curriculum
for a two-semester course on executive
The purpose of this study was two-fold. leadership was developed. The curriculum
First was to measure perceived growth of was reviewed and approved by the dean and
executive leadership competencies among a associate dean of student affairs and the
select group of undergraduate students after university curriculum committee.
they had completed a two-semester Furthermore, the course content was
executive leadership course. Second was to evaluated by a panel of university leadership
compare perceived to actual learning based educators from across the United States. The
upon final exam scores. panel agreed that the identified
competencies represented capabilities
Research Questions central to the executive leadership function.
Dooley and Lindner’s (2002) stair-step
The following research questions guided assessment protocol using a Before-After
this study: questionnaire based on the six executive
leadership competencies was developed and
1. What were the class average scores administered to all students in the executive
for each of the executive leadership leadership course at the conclusion of the
core competencies based upon t w o -semester course.
the Before-After self-assessment The natural setting for this investigation
measures? was a two-semester executive leadership
2. How did the individual’s perceived course conducted at Texas A&M University.
growth (“After”-score) compare with The students were part of a cohort that was
the individual’s actual growth required to take this course. The cohort was
(expert-authenticated final exam composed of 44 college seniors participating
score)? in a structured leadership development
3. Was the stair-step behaviorally program, and all the students participated in
anchored self-assessment instrument the study. Thus, the sample selected was
used in this study a reasonably essentially a convenience sample. All were
accurate measure of student members of a student organization that has
learning? as its purpose the development of the
leadership capabilities of its members.
Methods Respondents were coded with a number to
ensure confidentiality. Students attended 28,
This study used evaluative research one-hour weekly meetings, half in a fall
techniques to compare perceived and actual semester and half in a spring semester.
growth in learning of executive leadership Classes included lectures with handouts,
Journal of Agricultural Education 26 Volume 48, Number 2, 2007
McCormick, Dooley, Lindner, & Cummins Perceived Growth Versus Actual Growth…
case studies, class discussions, and analysis of competence for distance education
of film clips. This study was approved by professionals (Dooley & Lindner, 2002).
the Institutional Review Board for Human Reliability for this instrument was estimated
Subjects. by calculating a Guttman split-half
As mentioned previously, the coefficient on pretest or “Before” (r = .79)
and posttest or “After” (r = .70). The
researchers used a before-after self-
researchers used the stair-step approach
assessment instrument along
(rather than a continuum or summated scale)
with researcher/instructor authenticated
to visually represent progression from
assessment (that is, the final exam) to
novice (0) to expert (7). Since the students
document learning. The final exam was a came into the course sequence as
reflective paper based upon the movie novices regarding executive leadership
“Remember the Titans.” Students were competencies, the researchers determined
that a 4 on the step-stair model
asked to identify, describe, and discuss how
indicated proficiency in a core competency
the main character employed at least five of
area. The numbered stair-steps were
the six executive leadership functions. An
intended to serve as a reflection tool for the
expert-authenticated assessment rubric was students to measure their learning rather
used by the instructor and research team to than test for statistical significance.
substantiate the final exam and control for Averages were calculated to show trends in
bias. the data. Students were provided
The self-assessment instrument for this the operational definitions from which to
study was modified from an instrument that base their Before-After assessment
has been shown to be a consistent measure (Figure 1).

1. Stay Focused on Purpose Verification:

Ability to sort through incomplete, Expert

inaccurate, and sometimes conflicting
data and determine what should be

done.
7

6

5

4

3

2

1

0

Novice

Figure 1. Example of “stay focused on purpose” core competency and self-assessment format.

The open-ended narrative (the box with average of the perceived executive
the heading “verification”) was used to leadership competencies was calculated by
verify the level of executive leadership totaling the self-assessment scores provided
competence perceived by the students for by each student (their “After” scores) and
each of the six competencies. To dividing by six, because there were six
authenticate individual learning scores, the competencies. A projected exam score was
Journal of Agricultural Education 27 Volume 48, Number 2, 2007
McCormick, Dooley, Lindner, & Cummins Perceived Growth Versus Actual Growth…
calculated by dividing the average perceived (2) narrative verifications used to illustrate
score by seven, which was the highest competence in executive leadership, and
possible score for the stair-step method and (3) the comparison of individual’s
multiplying by 100. The projected exam perceived (self-assessment) versus actual
score was compared to the students’ actual growth (final exam) of executive
final exam score to determine the accuracy leadership competencies. In Table 2, the
of their projected final scores (based on their average growth is indicated for each of
self-assessment) to actual scores (Final the six core executive leadership
Exam scores) using the following scale: competencies.
Accurate = within a ten point range; For Staying Focused on Purpose,
over/under estimated = 11 to 20 point range; students reflected that their “Before”
and extremely over/under estimated = competence range was 0-6. By the end of
greater than 20 point difference between the executive leadership course sequence,
projected final exam score and actual final students’ “After” competence range was 2-7.
exam score. With 4 as the proficiency level, 33 of the 44
students (or 75%) were below in their
Results “Before” scores. In their “After” scores,
only 3 were below proficiency. The
The results of this study are reported in growth ranged from 2.4 to 5.0 with an
three areas: (1) the average learning scores average growth of 2.6, or 2 stair-steps
of executive leadership core competencies on the self-assessment instrument
based upon the Before-After measures, (Table 2).

Table 2
Average Growth in Executive Leadership Competency Clusters (N = 44)
Class Average
Core-Competency Before After Growth
1. Stay Focused on Purpose 2.4 5.0 2.6

2. Communicate Purpose & Vision 2.5 5.2 2.7

3. Establish & Maintain Environment 2.7 5.2 2.5

4. Develop Others 3.1 5.4 2.3

5. Lead Teams 3.0 5.3 2.3

6. Make Decisions in a Complex Environment 3.0 5.3 2.3

Journal of Agricultural Education 28 Volume 48, Number 2, 2007
McCormick, Dooley, Lindner, & Cummins Perceived Growth Versus Actual Growth…
For the Staying Focused on Purpose Twenty-five students reported a below
competency, one respondent, who proficiency score in their “then” assessment
rated himself as a “6” at the end of (57%) and only four after the course
the course (“After”) for that capability sequence. The average competence changed
noted: from a 3.1 “Before” to a 5.4 at the end of the
course sequence (“After”) with a change of
Before this year, I knew what I wanted, 2.3. The “After” score for Develop Others
but sometimes I got distracted and was the highest competence area.
deviated from my path to my purpose. One student who only rated himself as a
Now, when I decide what I want, I am 4 at the end of the course sequence offered
much more able to stay on task until it’s these bullets in his verification narrative:
done (3).
• Can develop others in many
For Communicate Purpose and Vision, ways…technical skills, leadership
students rated their “Before” competence skills, relationship skills, personal
from 0-5 and “After” competence from 3-7. improvement, etc…
The average growth was 2.5 to 5.2 with an • The quality of an organization is
incremental change of 2.7. The majority of resultant upon the quality of the
the learners (32) were below the proficiency individuals (9).
level in their “Before” competency scores,
which was 73% of the students. However, For the core competency of Lead Teams,
only three learners were still below students rated themselves between 0-6
proficiency in their “After” scores. One “Before” and 2-7 “After.” Twenty-six
respondent who perceived himself to be students (59%) rated themselves below the
below proficiency stated this: “I usually proficiency level when reflecting back.
have a hard time communicating purpose Only four students reported lacking
and vision to what I’m doing. I find myself proficiency after the course sequence. The
just trying to do it myself. I think I have average competence was 3.0 before
found more reasons to stop doing this” (24). participating in the course sequence and 5.3
For the third core competency, students at the end, with an increase of 2.3.
assessed their ability to Establish and A student who perceived his competence
Maintain Environment, ranging from 0-5 in this area to be proficient when he started
“Before” and 4-7 “After.” There were 32 and now felt he was at a level 6 indicated: “I
learners (73%) who were below the see this as my most resourceful function this
proficiency level in their “Before” scores, year. Most of the time I felt as a team leader
but no one reported being below proficiency getting others to jump on board and follow”
in the “After” assessment. The average (27).
“Before” score for this construct was a 2.7, The final core competency was Make
with students assessing their competence at Decisions in a Complex Environment.
5.2 by the end of the course sequence, a Students expressed beginning competence
change of 2.5. (“Before”) ranges of 0-6 and ending
A student who perceived growth from a competence (“After”) ranges of 2-7.
3 to a 4 on the Establish and Maintain the Twenty-eight students (64%) lacked
Environment competency had this to say: proficiency as indicated by their “Before”
“This function is hard to ensure during the scores. Four students, however, still lacked
experience as a senior, since we are not proficiency in their “After” self-assessment.
totally active with the training atmosphere. The average level was a 3.0 “Before” and
My awareness of its necessity has increased, 5.3 at the end of the course sequence
but maybe not my personal skills at doing it” (“After”).
(22). One student who reported a level 6
Develop Others is an important competence in this area stated, “Many times
dimension of leadership education. Student in difficult situations leaders are the ones
self-assessments for the “Before” scores everyone looks to in order to make complex
ranged from 0-6 with “After” scores at 1-7. decisions. It’s important to make your
Journal of Agricultural Education 29 Volume 48, Number 2, 2007
McCormick, Dooley, Lindner, & Cummins Perceived Growth Versus Actual Growth…
decisions quickly but with sound judgment. score (expert) of a 7 and multiplying by 100.
Then support your decision” (38). This number was then compared to their
Although the average class change in actual expert authenticated final exam score
perceived competence is interesting to (Table 3). Using the decision model
measure, the real challenge for described in the methods section above,
educational researchers is measuring actual estimates for student perceived
growth or learning as a result of an competence versus actual competence was
educational program. In order to address determined. Based on student self-
this concern, the researchers recoded each assessments, 22 of the 44 students
student’s average competence into a accurately estimated their competence. The
projected final exam score by dividing their remaining 22 students underestimated their
self-assessment score by the highest possible learning.

Table 3
Comparison of Perceived and Actual Learning of Executive Leadership Competencies
Average Self- Projected Final Actual Final
Respondent Assessment Scorea Examb Exam Scorec (%) Discrepancy
1 6.17 88 79 Accurate
2 6.67 95 91 Accurate
3 4.67 67 85 Extremely Under Estimated
4 4.67 67 74 Accurate
5 4.83 69 82 Under Estimated
6 5.50 79 91 Under Estimated
7 5.70 81 93 Under Estimated
8 5.50 79 81 Accurate
9 4.33 62 99 Extremely Under Estimated
10 5.50 79 72 Accurate
11 3.50 50 82 Extremely Under Estimated
12 3.83 55 84 Extremely Under Estimated
13 4.83 69 90 Extremely Under Estimated
14 4.83 69 75 Accurate
15 4.33 62 91 Extremely Under Estimated
16 4.67 67 78 Under Estimated
17 5.50 79 83 Accurate
18 5.33 76 85 Accurate
19 5.67 81 89 Accurate
20 5.17 74 87 Under Estimated
21 4.67 67 78 Under Estimated
22 4.17 59 92 Extremely Under Estimated
23 5.00 71 78 Accurate
Journal of Agricultural Education 30 Volume 48, Number 2, 2007
McCormick, Dooley, Lindner, & Cummins Perceived Growth Versus Actual Growth…
Average Self- Projected Final Actual Final
Respondent Assessment Scorea Examb Exam Scorec (%) Discrepancy
24 4.83 69 74 Accurate
25 5.83 83 76 Accurate
26 6.00 86 81 Accurate
27 5.67 81 95 Under Estimated
28 6.00 86 81 Accurate
29 4.50 64 79 Under Estimated
30 4.83 69 84 Under Estimated
31 5.17 74 94 Under Estimated
32 5.17 74 65 Accurate
33 5.83 83 77 Accurate
34 6.17 88 91 Accurate
35 4.83 69 73 Accurate
36 6.17 88 75 Over Estimated
37 5.83 83 81 Accurate
38 5.33 76 90 Under Estimated
39 6.50 93 84 Accurate
40 5.50 79 80 Accurate
41 5.50 79 83 Accurate
42 5.17 74 90 Under Estimated
43 4.50 64 91 Extremely Under Estimated
44 4.00 57 80 Extremely Under Estimated
aSelf-assessment score at the end of the course; bProjected Final Exam = average self-assessment score/7
x 100; cGraded final exam score on a 100 point scale

Conclusions and Implications positive change trend. Students tended to
improve at least two steps in each of the
Three conclusions emerged from of this core competency functions. This finding is
study. The first was the positive growth consistent with the findings in the Dooley
trends in students’ perceived average growth and Lindner (2002) study. Confidence that
in each of the executive leadership the self-reported changes were genuine was
competency areas. The second was the supported by the data displayed in Table 3.
implication of the discrepancy between Comparisons of students’ final exam scores
perceived learning and actual learning with their projected final exam scores
displayed by some of the students. And third showed that half of them were accurate in
was the need to create behaviorally anchored their personal assessments of what they
benchmarks at levels 2, 4, and 6 in order for knew and could do. Furthermore, with only
learners to better gauge their actual one exception, the remaining students under-
competence. estimated the amount of learning they had
As Table 2 revealed, for each of the six experienced. This suggests that students
executive competency areas there was a were conservative in their self-estimates of
Journal of Agricultural Education 31 Volume 48, Number 2, 2007
McCormick, Dooley, Lindner, & Cummins Perceived Growth Versus Actual Growth…
their improvement on each of the executive presented in Table 4. Bloom’s (1956)
competencies. By the end of the course they taxonomy was used to guide the
had actually learned more than they thought development of each behavioral anchor. For
they had. The two-semester class, therefore, instance, a step 2 level of competency
appears to have contributed positively to reflected that a student had achieved the
students’ mastery of the six executive knowledge and comprehension level in
leadership competencies. Bloom’s taxonomy for a given executive
Accurate self-assessment of personal and competency. A step 4 level required students
professional capabilities is essential for to apply leadership competencies in new
success in leadership (Cummins, 2005). settings and analyze the leadership situation.
One-half the students estimated their Finally, a step 6 level indicated a student
competence with reasonable accuracy and had developed an ability to synthesize and
about half of the students underestimated evaluate how different leadership
their growth in the leadership competencies. competencies could be used for making
Only one student overestimated his or her decisions and solving problems.
competence. Thus, the stair-step In summary, this executive leadership
behaviorally anchored evaluation approach course did improve students’ perceived
appeared to capture reasonably well growth in their leadership competencies.
students’ growth in executive leadership Growth trends were positive across all six
competencies, especially given that only one competencies. Furthermore, students’ self-
student overestimated his/her capabilities. assessments of what they had actually
However, since respondents were provided learned were essentially conservative; that
with only the operational definitions of each is, most either accurately estimated their
executive competency, an implication is that growth or underestimated it. This implies
students’ inability to accurately anchor their that the stair-step self-evaluation method
self-assessment to actual performance may provides at least a conservative measure of
have resulted from a lack of specificity in student learning, which is noteworthy.
the executive leadership definitions. However, future research is needed to test
Given that possibility, the authors have whether adding specificity to the
created competency-based behavioral competency-based behavior anchors
anchors for stair-steps 2 (near novice), 4 enhances the accuracy of students’
(proficiency), and 6 (near expert) for each of perceptions of their executive
the six executive competencies. They are competencies.

Journal of Agricultural Education 32 Volume 48, Number 2, 2007