ERIC ED610748: An Evaluation of the Teaching English Learners Early Mathematics (TEEM) Program; an Investing in Innovations (i3) Development Grant

2021·1.5 MB·English

by  ERIC

2021

An Evaluation of the Teaching
English Learners Early
Mathematics (TEEM) Program;
An Investing in Innovations (i3)
Development Grant
(U411C140073)
IMPLEMENTATION YEARS 1-4 FINAL REPORT
PREPARED BY: DR. LORIE A. SOUSA WITH DR. MADELEINE JETTER (CSUSB)
Table of Contents
Executive Summary 2
Introduction 6
Method 10
Results 18
Key Findings 52
Conclusions 54
References 56
Appendices 61
Logic Model
Fidelity of Implementation Table
Impact Contrast Table

1
Executive Summary
California State University, San Bernardino (CSUSB) and the Riverside County Office of
Education (RCOE) were awarded an Investing in Innovation (i3) Development grant in
2015 to investigate the impact of an early education mathematics program on English
Learner students at two rural school districts in Riverside, CA.
COREducational, an evaluation and research consultancy, conducted the external
evaluation of the findings over a four-year period from 2015-2016 to 2018-2019 school
years. A no cost extension was approved for the final and fifth year (2019-2020).
However, the COVID pandemic resulted in school closures that precluded data
collection. Thus, the decision was made to report on the four complete years of data
collected.
Project Activities and their relationship to project goals:
Teaching English Learners Early Mathematics (TEEM) had five major goals: to improve
the academic achievement of English Learners (ELs) (Goal 1); to improve teacher
knowledge and beliefs for teaching math to ELs (Goal 2); to improve the teaching of
math to ELs and all students (Goal 3); to develop principals as instructional leaders in
mathematics (Goal 4); and to build a learning partnership that develops, sustains and
shares a model of instructional change supporting ELs. (Goal 5). In support of these
goals, a combination of four mutually supportive interventions were implemented.
Intervention 1. Intensive Teacher Professional Development (PD) Activities
Summer Intensive Institutes were held every summer. The institutes addressed math
content, best practices for teaching English Learners, and interactive notebooking.
Over the course of the project, the mathematical focus of the Institutes progressed
through the major themes of elementary mathematics, including whole numbers and
algebraic thinking, multiplicative thinking, geometry, fractions, proportional reasoning
and expressions and equations. The Common Core State Standards for Mathematical
Practice were emphasized throughout the professional development. The use of
multiple representations, to support understanding for English Learners and all students,
was another focus of attention. Participants received curriculum units from the Contexts
for Learning Mathematics curriculum as a resource to support them in teaching through
problem solving. TEEM staff, including CSUSB faculty and Riverside County Office of
Education (RCOE) specialists, presented the PD. Lesson Study experts were also brought
in to cover special topics such as Teaching Through Problem Solving.
Intervention 2. Lesson Study
Each year, participants worked in grade-level groups of three to eight members each
to complete two three-day cycles of lesson study in each academic year of the
project. In a lesson study cycle, teams set shared instructional goals; studied relevant
content, pedagogy and formative assessment data; and planned, taught, observed
and reflected on a mathematics lesson. As a result of participation in the California
Action Network for Mathematics Excellence and Equity (CANMEE), lesson study
participants began crafting equity goals (in addition to mathematical goals) for the
2
classrooms in which lesson study is taking place. An example of an equity goal is
“Students see themselves as powerful mathematical thinkers through small-group and
whole-classroom discourse.”
Intervention 3. Interactive Math Notebooks for participating students
Each participating classroom teacher received a class set of composition books at
each TEEM Summer Institute. Notebooks are intended to contain a record of the
teacher’s teaching and the student’s learning: “input” pages on the right hand side
include note-taking, practice, or other elements directed by the teacher, while
“output” pages on the left provide opportunities for students to engage more actively
with the material through problem solving, summarizing or reflection on learning. Project
staff rated a sample of student notebooks each year using a rubric.
Intervention 4. Professional Learning for school leaders
Site principals participated in several activities designed to support them in
understanding the curriculum shifts described in the Common Core State Standards for
Mathematics.
TEEM staff provided 30 to 45-minute presentations at monthly district-based principals’
meetings (six at each district). These presentations addressed the practices and
strategies implemented in TEEM to aid principals in understanding what they would
expect to see in observations of TEEM classrooms. Topics included Lesson Study,
mathematics notebooking, and the importance of multiple representations in
mathematics learning, among others.
All principals participated in classroom walkthroughs with TEEM staff. In one of the
districts, each site principal hosted one walkthrough attended by all district principals
and TEEM staff. In the other district, the two principals walked classrooms at both sites
twice a year. The 5x8 Card developed by the Strategic Education Research Partnership
(SERP) was used to facilitate observation and discussion of mathematics classroom
practice. This led to more productive conversations with principals about mathematics
teaching and learning.
Independent Evaluation and preliminary findings:
The Independent Evaluation of the TEEM Project was conducted by TEEM’s
independent evaluator COREducational, headed by Dr. Lorie Sousa. Below is a
summary of results.
TEEM Teachers demonstrated growth in mathematics content knowledge, as measured
in the Learning Mathematics for Teaching (LMT) Assessment. They have showed
improving self-efficacy beliefs relative to a comparison group.
Participating teachers’ mathematics content knowledge improved relative to a
comparison group of non-participating teachers in each project year (although due to
sample size, the difference was not statistically significant in the last two project years).
3
The Mathematics Teaching Efficacy and Expectancy Beliefs Instrument (MTEEBI) was
used to measure teachers’ beliefs about developing mathematical practices in a
diverse student population. Teachers are asked to rate their level of agreement with
statements like “I am able to help students from impoverished backgrounds excel in
math.” Participating teachers’ beliefs improved relative to a comparison group to a
statistically significant degree, showing that participation in TEEM had a measurable,
positive impact on teachers’ beliefs -- both in their own teaching abilities and in their
students’ ability to learn mathematics.
Classroom observations were performed each project year using a project-specific
rubric. These results were analyzed in the evaluation report and demonstrated some
significant positive shifts in teacher classroom practices. For example, the percentage
of classrooms in which “English Learners produce language that communicates ideas
and reasoning, even when that language is imperfect” increased from 48% in Spring
2016 to 80% in Spring 2018.
Student outcomes were tracked using state-administered Smarter Balanced Assessment
(SBA) for grades 3 and higher. Because students do not take these tests prior to grade 3,
we used Assessing Math Concepts (AMC) interviews for grades PK-2. In grades 3 and
above, students taught by a TEEM teacher for at least one year outperformed students
in a comparison group in both the Mathematics and English Language Arts SBA tests.
This was true both for English Learners and the general student population. Comparison
data was not available for AMC interviews, so our evaluator compared participating
students’ results at the beginning and end of each project year and found significant
improvements in every section of the assessment.
Extent to which objectives, expected outcomes and annual targets were achieved:
Based upon the evaluation data, TEEM achieved many of its proposed outcomes for
teachers and students. Due to participant attrition, participation levels were below the
proposed target of 75 teachers participating for all four years. Some objectives for site
principals could not be achieved as proposed: for example, principal attendance at
the TEEM Summer Institute remained well below target in spite of an increase in Y4.
Sustainability planning and dissemination:
The TEEM Teacher Leadership Team consisted of eight teachers interested in
developing themselves as math leaders and disseminating best practices learned in
TEEM. Four bimonthly Leadership Team seminars were held in Y4. The Project director
lead each meeting using the Learning to Lead Mathematics Professional Development
curriculum by Carroll and Mumme, and participant feedback was very positive. Based
on teacher interest, Leadership Team teachers provided activities and material kits to
support Family Math nights at school sites. The leadership team also went on to plan
and present an online dissemination workshop attended by approximately 30 local PK-
TK-Kindergarten teachers at the close of the project.
4
Project dissemination goals were met. The Project Director gave two presentations at
regional conferences discussing the math notebooking practice in TEEM classrooms.
The first of these was done in collaboration with a project teacher at the Riverside
County MaTHinK conference; the other was to the California Math Council Southern
Section Conference. An article submission to NCTM on the same topic was not
accepted for publication. Alternate publication venues are being considered to
disseminate information regarding the notebooking practice and results of the study.
Challenges, best practices and lessons learned:
Attrition. Teacher attrition was a challenge. In the hope of compensating for attrition,
TEEM received permission to invite new teachers to participate in lesson study
beginning in Fall of 2018. As a result,10 new teachers were included in TEEM lesson study
in September 2018.
Substitute teacher availability. Occasionally, a school district partner was short on
substitute teacher coverage on a TEEM lesson study day. In these situations,
administrators did their best to enable teachers to keep their lesson study meeting (for
example, by splitting classes into other teachers’ classrooms), however on some
occasions district administrators cancelled a day of TEEM lesson study. This was difficult
for teachers and facilitators alike. TEEM attempted to mitigate this problem by
scheduling lesson study well in advance.
Lesson study. Work with the CANMEE Lesson Study Support Hub provided insight into
best practices for mathematics lesson study. The use of equity goals and an equity
commentator also encouraged a deeper look into student learning, causing teachers
to slow down and think specifically about targeted students including English learners or
those with special needs.
Impact beyond the partners and implementation sites:
Lesson study. Through the CANMEE Lesson Study Support Hub, TEEM staff support lesson
study across our region as a vehicle to promote equitable and rigorous mathematics
instruction. Further, educators involved in the project have developed their own skills
and capacity with respect to the lesson study practice.

5
Introduction
The Teaching English Learners Early Mathematics (TEEM)program was designed by,
California State University, San Bernardino (CSUSB) and the Riverside County Office of
Education (RCOE), in conjunction with two Local Education Agency (LEA) partners, and
the Head Start and State Preschool programs affiliated with one of the LEAs. The
purpose was to build on a history of close collaboration to develop, implement and
evaluate a replicable model for improving the academic achievement of English
Learner (EL) students in pre-K through grade 6.
TEEM program intentions were to increase the number and proportion of ELs completing
challenging courses in core academic subjects through a novel combination of
interventions designed to support communities of inquiry among students, teachers,
and school leaders. Students engaged in structured inquiry through the use of
interactive notebooks in mathematics; teachers sought to improve mathematics
instruction through the formation of communities of inquiry supported by Summer
Institutes and an adaptation of Japanese Lesson Study; and principals were developed
as instructional leaders in mathematics through a variety of supports. The project
intended to directly impact at least 4,000 students, 75 teachers and 7 principals.
The primary goal was to increase the number and proportion of ELs successfully
completing challenging courses in core academic subjects by implementing structured
inquiry-based approaches to learning, teaching and school leadership. These
approaches were intended to develop collective efficacy in schools and provide ELs
access to challenging math content from pre-K through grade 6.
Importance of math in early grades.
To maximize long-term impact, math-focused interventions were developed for ELs in
early grades. The risk of dropout is roughly twice as high for ELs as it is for non-ELs
(Callahan, 2013), and in many cases the dropout problem is a math problem. In the Los
Angeles Unified School District, students who passed Algebra I by grade 9 were twice as
likely to graduate as those who had not (Silver, 2008). In order to improve academic
outcomes through high school, ELs must succeed in challenging math courses, and
they must begin early. Academic performance in early grades is a strong indicator of
future dropout risk, and high-quality preschool decreases the risk (Rumberger & Lim,
2008).
In California, achievement gaps existing in grade 2 largely persist through grade 7, a
“pivot year” in which student achievement varies widely and determines future
opportunities in math (Terry and Rosin, 2011). In California, ELs are a large percentage
of early-grade students, and many struggle in math. In 2013, ELs made up 35.7% of CA
grade 2 enrollments, and 47% of those students tested below proficient in math.
The status of ELs statewide is mirrored in the partner, small districts serving the
unincorporated communities in Riverside County. The percentage of students receiving
free or reduced-price lunch is 76.1% and 65% at the two districts, while the state
average is 58%. Per the 2010 Census, the largest ethnic group in both communities is
Hispanic/Latino at 51.4% and 54.5%. In one district involved in the study, 45.4% of people
aged 5 or more years speak a language other than English at home. Results from the
6
2013 California Standards Test (CST) showed the percentage of students classified as EL
tapers through the elementary grades.
While the proportion of ELs decreases, the math achievement gap for ELs grows with
the complexity of the mathematics from grade to grade. In the seventh grade, 52% of
English-Fluent or English Only seventh graders were proficient or advanced in
mathematics while only 13% of ELs attained that level of achievement. A student
classified as EL in the grade 7 pivot year faces restricted access to advanced
mathematics, to success in high school and college, and to STEM careers.
Mathematics interventions supporting ELs and their teachers are especially timely in
light of the Common Core State Standards for Mathematics (CCSS-M). The CCSS-M call
for students to demonstrate proficiency through eight Standards for Mathematical
Practice (SMP). In order to give ELs access to the CCSS-M, Moschkovich (2013)
recommends that teachers balance concepts with procedures; maintain high
cognitive demand; develop productive student beliefs about math; and engage
students in the SMP. At the same time, teachers in the 22 states using the new Smarter
Balanced Assessment Consortium (SBAC) tests to assess student proficiency with respect
to the CCSS-M are challenged to prepare ELs for the higher language demands of
these tests. TEEM supported teachers in this shift by developing a replicable model of
intensive professional learning.
Improving outcomes in core academic subjects through attention to metacognitive
and non-cognitive skills and mindsets.
When students believe that intellectual ability is not fixed but grows through effort, they
show greater persistence (Dweck, 2006). The National Mathematics Advisory Panel
(2008) reported that interventions addressing affective and motivational factors can
improve the achievement of traditionally underserved groups. TEEM will help ELs
develop the metacognitive and non-cognitive skills key to success using interactive
inquiry notebooks for math. The notebooks were not only a tool for collecting and
recording classroom activities, but they sought to make student metacognition explicit
and provide a structured space for developing reading, writing, and study skills.
Unlike many accepted practices for supporting ELs which focus on general academic
language development, the program sought to achieve improved outcomes for ELs
while keeping a strong disciplinary focus on mathematics, combining four structured
inquiry-focused interventions targeting students, teachers, and school leaders. TEEM’s
approach to improving the achievement of ELs is based on the novel premise that
educators need deep mathematical understanding to provide ELs access to the SMP in
a CCSS-aligned curriculum, that “vocabulary is necessary, [but] not sufficient” to
provide that access (Moschkovich, 2013), and that providing that access will support
ELs in strengthening language skills through reasoning, talking, and writing about
complex disciplinary content.
To provide a comprehensive approach to increasing ELs’ access to rigorous
mathematics, the project combined four evidence-based interventions:
1. The adaptation of interactive inquiry notebooks – a best practice in science
teaching – to mathematics
7
2. Summer Professional Development (PD) to immerse teachers in math inquiry and
notebooking
3. The adaptation of the Japanese Lesson Study model of professional learning for
teachers
4. The purposeful, regular, and intensive participation of school principals in the
professional development.
Given the need to intervene early, another novel aspect of TEEM was the creation of a
PD pipeline for preschool educators who rarely receive opportunities for math-focused
PD. Through its focus on mathematics, its combination of research-based interventions,
and the purposeful inclusion of school leaders and early education providers, TEEM
intended to provide a blueprint for how to increase EL student academic performance
and pursuit of STEM in college and careers. TEEM interventions were built upon existing
research and promising practice, with adaptations intended to make it an exemplar for
new practice.
Interactive notebooks.
Notebooks have been used successfully in many science classrooms (Hargrove &
Nesbit, 2003). Used intentionally, student notebooks provide evidence of the teacher’s
teaching practices and the students’ learning (Shimojyo & Shimojyo, 2013) and are a
formative assessment tool of great value. They make student metacognition explicit
and provide a structured space for the development of reading, writing, and study
skills. It is a promising practice, and an inexpensive and sustainable one since paper
notebooks are easily obtained. It is a culturally responsive practice: in Mexican and
other Latin American classrooms, el cuaderno (notebook) is used in a more structured
and rigorous fashion than is common in the United States (Celedon-Pattichis & Ramirez,
2012). Amaral and colleagues (2002) found that the use of interactive notebooks in
elementary science classrooms in an impoverished section of inland California
improved EL achievement not only in science, but also in reading and writing. TEEM
hopes to move the field forward by adapting a promising practice in science
education to the mathematics classroom and measuring its impact. We tested the idea
that use of purposefully designed math inquiry notebooks would help develop
language, organizational, and metacognitive skills, and would accelerate EL
achievement not only in math but also in language learning.
The combination of interactive notebooks with the lesson study model for PD is a unique
and powerful way for teachers to develop effective formative assessment practices.
Lesson Study.
The transition to CCSS calls for teachers to make major shifts towards greater focus,
coherence, and rigor in math instruction (CCSS Initiative, 2014), but observers have
noted that fundamental teaching practices have changed little in the last century
(National Research Council, 2001). TEEM supported teachers in this shift using the
Japanese lesson study model.
Lesson study gained attention with the publication of The Teaching Gap (Stigler &
Hiebert, 1999) and is the subject of active research (Lewis et. al. 2009). Lesson study
8
incorporates many characteristics of effective PD and school change: it is intensive,
ongoing, and connected to practice; it is focused on student learning, it addresses the
teaching of specific curriculum content, and it builds strong working relationships
among teachers (Darling-Hammond et. al. 2009). In lesson study, teams of teachers
develop a practice of inquiring into their students’ thinking and investigating the
effectiveness of their instruction. In a cycle of lesson study (Fig. 1), teachers agree to
shared learning goals. They research relevant curriculum and pedagogy, and they
plan, observe, and refine a “research lesson” addressing the goals.
Based on its documented
effectiveness and on
experience gained in previous
projects, TEEM used lesson
study to create new
opportunities for schools to
build a knowledge base for
teaching core academic
subjects to ELs. Through close
observation, teachers
develop “the eyes to see
students” (Lewis, 2004), seeing
past language deficiencies
into the content of students’
mathematical thinking. TEEM
Figure 1 - The Lesson Study Cycle
teachers sought to build and
share knowledge of their
students’ language needs and learn to anticipate those needs. Student notebooks
were a key resource for lesson study, providing teachers with data to inform their study,
help discern student needs, set instructional goals and evaluate success of research
lessons.
Supporting Instructional Leadership in Mathematics.
Among all school-related factors, the impact of leadership on student learning is
second only to teaching, and this impact is greatest in high-need schools. At the same
time, there is little research on how principals continue their professional learning on the
job (Leithwood et. al, 2004). While Instructional Rounds (City et. al, 2009) are considered
best practice for principal learning, principals without a significant background in
mathematics receive little support in considering the mathematical content of the
lessons they observe in classrooms. To fill this gap, TEEM’s fourth novel intervention
sought to support principals as “lead learners” at schools (Liptak, 2005), developing their
capacity as instructional leaders in the transition to CCSS. Principals attended Summer
Institute sessions and monthly sessions built into district-wide principal meetings. Project
staff supported principals in Instructional Rounds.
The proposed interventions and the evaluation of their effectiveness will hopefully
benefit schools and districts nationwide: EL populations are growing rapidly across the
nation (by 53% from ’98 to ’08; NMAP, 2008) and, to our knowledge, research
investigating methods to accelerate EL growth via CCSS, lesson study, and interactive
9