Skip to Content

 

SIG2 / CalSTAT Activities and Outcomes Evaluation Summary Report
February 2005 – August 2008

Improving the Special Education System in California A State Improvement Grant (SIG) Program

Prepared By: Cheryl “Li” Walter, Ph.D. and Alan Wood

November 2008

For questions concerning this report, please contact:

Janet Canning, California SIG Director, California Department of Education, Special Education Division

Anne Davin, CalSTAT Director

CalSTAT (California Services for Technical Assistance and Training), at Napa County Office of Education, is a special project of the California Department of Education, Special Education Division. Visit CalSTAT at http://www.calstat.org.

CalSTAT is partially funded from federal funds awarded in Part B of Public Law 108-446, the Individuals with Disabilities Education Act (IDEA), as amended in 2004. Additional federal funds are provided from a federal competitively awarded State Program Improvement Grant to California (CFDA 84.323A) allowed in Part D of Public Law 108-447, the Individuals with Disabilities Education Act (IDEA), as amended in 2004. These dollars are considered local assistance funds. Both funds are to assist individuals serving children birth to 22 years of age and their families.

Table of Contents

California State Improvement Grant (SIG2)

In 1999, a five-year State Improvement Grant (SIG) was received from the federal Office of Special Education Programs (OSEP) to improve special education outcomes and services in the State of California. In 2005, a second SIG was granted, SIG2, continuing and expanding original SIG service delivery through June 2008. A new five-year grant, the State Personal Development Grant (SPDG/SIG3) was awarded in 2008 and will continue these SIG activities.

From February 2005 through June 2008, SIG2 implemented activities directed at personnel development, family education, and systems change. Those served by the grant activities included teachers and administrators from both general and special education, early intervention service providers, students and families, related agency personnel and service providers, university/college professors, and university students currently enrolled in teaching preparation programs.

A major focus of SIG2 was to communicate common messages to the field about selected topics. These common, or core messages, articulate critical research findings and essential components of effective practice. All these evidence-based core messages have been identified by experts in the field and have been approved by the California Department of Education, Special Education Division.

CalSTAT, through SIG2 funds, supported activities that reflect all of the following core messages:

SIG2 was fully aligned with improvement strategies identified in the California Performance Plan (CA SPP) and is cited explicitly as a part of improvement activities under several key indicators. The CA SPP describes the role of SIG2 in improving special education outcomes in California in the following passage:

SIG2 will be used to improve special education services in California in several areas such as the quality and number of teachers and other personnel who work with students with disabilities, coordination of services for students with disabilities, behavioral supports available for students with disabilities, academic outcomes, especially in the area of literacy, participation of parents and family members, and in the collection and dissemination of data. The grant has a significant site-based component that will include an entire network of educators who have been trained through the first SIG to assist schools in implementing research-proven behavioral approaches.
State of California Part B State Performance Plan (SPP) for 2005-10, page 7

Some of CalSTAT’s many activities include:

SIG2 Activities throughout California

Professional development delivered through the Leadership Sites, BEST, ERIA, and other Training/Technical Assistance (TA) activities included 565 events with 12,008 participants during SIG2

Training and Technical Assistance (TA)

Training and TA are provided around research-based core messages from the eight Core Message Areas through:

Leadership Site and Regional Institute TA are aspects of the Leadership Community activities, which are described in greater detail starting on page 14.

Over the course of SIG2 (not including BEST, ERIA, or Statewide Institute activities), there were 472 training and technical assistance (TA) events with 13,433 participants.

Event Topics and participant roles were tracked by CalSTAT during SIG2

TA Particpants by Core Message Area
472 events during SIG2 (Feburary 2005 to June 2008)
Core Message Participants Percent
GE/SE Collaboration 5,955 44%
Literacy 3,544 26%
Behavior 1,345 10%
Transition 1,297 10%
RtI 394 3%
Family Partnerships 388 3%
IDEA 280 2%
LRE 230 2%
TA Particpants by Role
472 events during SIG2 (Feburary 2005 to June 2008)
Role Participants Percent
Teacher: General Education 1,954 31%
Teacher: Special Education 1,661 27%
Other Certificated Professional 700 11%
Administrator: General Education 544 9%
Administrator: Special Education 339 6%
Paraprofessional 233 4%
Parent/Family 196 3%
Program Specialist 81 1%
Other 504 8%
Average Participant Responses to End-of-Event Surveys
Responses from 6,200 of 13,433 participants at 472 events during SIG2 (February 2005 to June 2008)
Knowledge Prior 2.9
Knowledge After 4.2
Usability 4.3
Overall 4.3

Follow-up surveys were emailed to recipients to measure the impact of TA between 3 and 6 months after events. Responses to follow-up surveys described implementation and sharing of strategies that worked well.

Participant Responses to Follow-Up Surveys
Responses from 1,238 participants at 472 events during SIG2 (February 2005 to June 2008)
  Implemented Strategies Strategies Worked Well Shared Strategies with Others
Responding with a 3 on a 5-point scale 30% 28% 25%
Responding with a 4 or 5 on a 5-point scale 53% 62% 62%
Total Responding with a 3, 4 or 5 on a 5-point scale 1,028 participants 1,114 participants 1,077 participants

Participants were given the opportunity, both in end-of-event and follow-up surveys, to comment on the TA they received. Responses include the following:

The training received has become a daily resource. Thanks to CalSTAT for providing the forum to have powerful learning and conversations around the latest research-based practices.

Perhaps what is most useful for us is being exposed to so many great practices and ideas then as a district team narrowing those down to some key practices that we implement.

The presentation was very effective because it was presented by teachers to teachers.

We have started to implement true professional learning communities; we are beginning with the math department. We have integrated clusters and are using conversation questions to look at our data more critically.

I enjoyed visiting the classrooms and seeing the strategies that have been put in place to address all students and students at risk. I enjoyed listening to how the district supports the schools and message that is given district-wide. I would enjoy visiting again!

Collaboration transformed our approach to our systems and structures. Teachers feel empowered as never before as they see immediate progress by students. We have shared with other schools and other districts multiple times! Thanks for the support.

CalSTAT has used California Standards Testing (CST) English-language Arts (ELA) proficiency as an indicator of academic outcomes at sites receiving TA. Not all TA events are directly focused on improving CST ELA scores, though most are and many others are intended to improve general academic outcomes, which include CST ELA scores.

Average Percent of CST ELA Students Proficient and Above at Sites Receiving Substantial TA
469 sites who received 3 or more days of TA in two or more years during SIG2
    2004
(Prior to SIG2)
2005 2006 2007 2008
All Students Sites Receiving TA 42.6 46.8 49.6 49.8 51.8
Statewide 37.4 41.9 44.8 45.5 48.5
Students with Disabilities Sites Receiving TA 16.6 19.9 23.4 23.8 27.9
Statewide 14.7 17.0 19.5 20.7 24.2
AYP Target (Approximate) 12 23 23 23 34

AYP targets increased to approximately 34% in 2008, and will increase again in 2009. Both leadership sites and statewide scores are currently exceeding these targets for all students, while neither are meeting these higher targets for students with disabilities.

Collaborative Sites Survey

Special Education/General Education Collaboration has been a major focus of TA through SIG2, with 51% of all trainings delivered in this core message area. Many of the other core message areas include a collaborative component as well, helping at-risk students through delivery of interventions which often include both general and special education teachers in the process.

Sites receiving more than three days of TA in collaboration were identified by CalSTAT and contacted with a survey regarding site-level collaboration between general education and special education. Of the 349 sites meeting this criteria, 63 surveys were returned.

a
Area Degree of Collaboration PRIOR to TA Degree of Collaboration AFTER TA Change
Assessment 4.5 7.1 +2.6
Intervention 4.6 7.4 +2.8
Core Curriculum 5.0 7.4 +2.4
Teaming 5.0 7.4 +2.4
Accountability 4.8 7.1 +2.3
Parent Role 5.1 6.4 +1.3
Administration 5.7 7.8 +2.1
Resources 5.0 7.2 +2.2
Professional Development 5.2 7.7 +2.5
Overall 5.1 7.4 +2.3

Sites responding to the collaborative sites survey were generous in taking the time to provide detailed comments describing their sites before and after receiving TA, their experiences along the way and their expectations for the future. These comments include the following.

Please describe your organization’s special education service delivery system prior to the training/TA:

Some pull out for specific core subjects, and some push into the classroom. In addition, both certificated teacher and aide ran groups include SE and Non-SE students. RSP [resource specialist program] teacher monitored all progress.

Prior to our district’s involvement with CalSTAT, we had a very traditional model for delivering services to students with IEPs [individualized education program]. SE teachers had caseloads and taught the students on those caseloads.

Students were referred to a student study team. There was not a systematic approach in providing intervention services in the GE setting prior to SST [student success team] referral. At the SST meeting the focus was usually on identifying the student for SE services. SE staff worked in isolation and developed their own curriculum and curricular modifications without a supportive structure that enables frequent communication or collaboration with GE staff. In addition, they did not always have access to a full range of core curricular materials.

The use of a formalized progress monitoring system was not being utilized. There was a disconnect/lack of structured communication regarding formalized progress between SE and GE in the area of literacy.

Please describe your organization’s special education service delivery system now:

SE and GE staff collaborates more effectively. School sites have GE interventions and most referrals for SE assessments happen after it is clear the interventions were not working. SDC [special day classes] students are more integrated in GE classes when appropriate. GE teachers and administrators are more engaged in the IEP process to determine needs, goals, and services.

Each district has a collaboration and intervention plan. New service delivery systems started in all districts. Training more focused on specific needs and interventions.

Teachers meet weekly to review data and discuss students. Students are served based on their needs as indicated by the shared data. Even students without an official label can access these services.

Now that over 36 school sites have attended the RtI trainings—supported by CalSTAT funding—for the past two years and the PEAK 3 Literacy series for one; [our county] now has more school districts looking to make an entire shift in all their schools to an RtI service delivery model. Systems change is happening in [the county].

While we still operate using a ‘blended services’ model, we have broadened our service delivery model so that it aligns more closely with the RtI model. As a result, we have far fewer ‘intensive’ intervention classes with the development and success of our approach. Additionally, we are identifying fewer students for SE under the SLD [specific learning disability] category. We have also been working on our strategic supports to prevent students from getting to that intensive level.

What challenges have you had to address in the process of working towards a more collaborative approach?

Main issue we have to deal with and we continue to deal with is time. We have increased instructional minutes and we are trying to implement a full collaboration model. When and how often do we meet to fully implement the principles of collaboration?

The greatest challenge has been the fundamental belief that not all students can meet the expectations and standards addressed in the core curriculum. Additionally, there is the belief that GE teachers may not have the skills to deliver curriculum to students with IEPs. “ I don’t know how to teach those students.” These two challenges are at the core of our ongoing professional development.

Lack of resources to provide more TA to support sustainability of systems change for making a shift to adopt collaborative models of services for special education students.

What has helped facilitate the process of working toward a more collaborative approach?

The statewide leadership institute has provided us with the exposure to best practices throughout the state and the opportunity to network with the practitioners who are implementing these best practices. Each year the institute has inspired a new group to think differently about serving students in the LEA.

Weekly collaboration time—school wide—strictly focused on curriculum and instruction. Training on how to use data to drive instruction. Building trust and relationships. Challenging the culture of the school. Celebrating successes. Setting yearly school-wide and department goals. Requiring objective, outcomes, and feedback for all lessons.

Now that the administrative team has focused knowledge of the goals and purposes of collaboration the support is much more effective and consistent in reaching the goals associated with collaboration

Once a site team gets excited and learns to facilitate the processes and has good data to measure progress, we have seen great movement.

What results have you seen since adopting these changes? In API scores? SE referrals? Behavioral referrals? Student/teacher attitude?

Students have more success in the GE setting and also we feel that the results of collaboration has also aided in the passing of the exit exam as well as completion of high school requirements.

One of the most dramatic results of our changing our service delivery is that we have decreased the number of students in SE by 1,100 over the last 5 years. Additionally, many regular education students are successfully served in collaborative classrooms that might otherwise be referred for SE. Several of our schools have seen a steady rise in test scores of their students with disabilities.

Since adopting the co-teaching model, our behavioral referrals have dropped drastically. The students are in with their peers and not isolated. We actually exited students who probably would not have been able to be exited in a non-collaborative model.

When we hear our teachers, parents, and students speak about collaboration they are passionate and enthusiastic and state they would never turn back to the way it used to be.

What future plans do you have for sustaining or expanding the move towards collaboration?

We hope to sustain the changes we have seen by creating the foundational systems that will support this work regardless of who is in charge or who is involved. Our teachers would never want to return to the old model of doing school. Our district is providing the resources we need to implement assessments and interventions with fidelity.

District is continuing to work with certificated staff and employees bargaining groups to expand collaboration. Additionally, the district is working on benchmark assessments, common formative and summative assessments, and “translating essential standards into student friendly language.”

SE is working more closely in the PLC’s [professional learning communities] to become part of the weekly rotation time. This is a time where the teachers identify students who need additional help in a particular skill set. The students divide into fluid, flexible groups among the grade level teachers to re-teach that particular skill to proficiency using differentiated instructional strategies. We hope to incorporate our SE students into this rotation.

Are there any areas where your site is in need of further training or technical assistance?

Absolutely-we need to be become more sophisticated in data management at multiple data entry points and translating that knowledge into practice with teachers to assist them in making shifts in their performance in the classroom.

We need to continue our work in looking at data and identifying what the data is really telling us. Also we need to further develop skills in peer reflection .

We look forward to gaining more training and information regarding research based intervention programs in the areas of math and reading.

Yes, we would support any future trainings that might become available to increase our skills base in helping teams communicate, address issues, and resolve obstacles that would interfere with the collaboration process.

SE/GE Collaboration at Leadership Sites

In addition to the 242 TA events in Special Education/General Education Collaboration, SIG2 also advanced this core message by making 16 of 28 leadership awards to schools with exemplary collaborative practices, strategies and programs. Detailed profiles of these sites are available online, offering a clear picture of Collaboration in action. Visit http://www.calstat.org/leadershipSites/index.html and click one of the links to any of the Collaboration site profiles including information about the site, the site’s story, and presentations and PowerPoints about their program.

The Leadership Site Award Program is detailed below.

The Leadership Community

SIG2 devotes a significant share of TA and resources to a “communities of practice” model throughout CalSTAT activities. This approach reflects effective, evidence-based practices in improving personnel knowledge and skills and involving collaborations between parents, administrators, and personnel representing all aspects of education, both within and between schools.

The Leadership Community expanded and proliferated these principles during SIG2. The community included three major components:

By building upon the support they received through the Statewide Leadership Institute and other resources throughout the year, these Leadership Sites and Regional Institutes delivered most SIG2 TA. Of the 472 TA events delivered during SIG2 (see page 6), 345 were based around a Leadership Site or Regional Institute host site (73%).

TA Delivery Within a Learning Community

In implementing SIG2, CalSTAT embraced the evidence-based principle of effective teacher training through learning communities. In successful programs, learning communities are effective in offering feedback to the community, answering questions, helping plan activities and set goals, discussing problems, and providing on-going support for these research-based activities. Peer-to-peer instruction in learning community settings utilizes personnel who have “been there, done that” and allows them to share their experience schoolwide.

Effective training programs also give participants the opportunity to reinforce what they have learned by testing, repeating and practicing. Over the course of a year, Leadership Sites and Regional Institute host sites continued to be a resource to sites receiving TA, sustaining lessons learned through repeated contact. Further, as members of a larger regional community of educators, sites who received Leadership Awards and hosted Regional Institutes often continued to serve as regional leaders, year to year.

While the formation of a thriving learning community was a key focus of the SIG2 Leadership Community, these principles were pervasive throughout all TA delivery and, often in modified forms, throughout all SIG2 activities. Behavior and literacy intervention activities, ERIA and BEST, were both sustained through statewide collaborative activities, including statewide meetings, conference calls, and online discussion forums. These activities further built on themes of collaboration and partnership in other contexts.

Statewide Leadership Institute

Each year during SIG2, CalSTAT hosted teams from leadership and regional institute host sites in a statewide institute. This three-day event is supported with wrap-around TA and online activities, presenting an opportunity for leadership sites to communicate with one another and further develop their programs.

Participants in the institute benefited from content training and discussions with experts in the core message areas. Site-hosted roundtables allowed sites to share learning and practice presentation skills. Participants were also given opportunities to network with other sites and meet with their own teams, to synthesise learning and plan for the future.

Statewide Leadership Institute Particpants by Role
3 events during SIG2 (Feburary 2005 to June 2008)
326 of 663 participants responding
Role Participants Percent
Teacher: Special Education 88 27%
Teacher: General Education 87 27%
Administrator: General Education 47 15%
Administrator: Special Education 46 14%
Parent/Family 27 8%
Other Certificated Professional 16 5%
Other 11 4%
Average Participant Responses to Surveys at Statewide Institutes
Responses from 326 of 663 participants at 3 events during SIG2 (February 2005 to June 2008)
Knowledge Prior 3.1
Knowledge After 4.3
Usability 4.6
Overall 4.4

Leadership Site Awards

As part of SIG2, high performing sites were identified through a rigorous, competitive process each year. These sites articulated their own site-level successes through this initial evaluation and demonstrate a continuity of their award-winning practices and high standards through annual review. Leadership sites continued to develop their programs, as well, by receiving ongoing TA activities and participating in the Leadership Community.

Additionally, leadership sites distributed TA to other sites based on their first-hand experience developing successful strategies, by offering site tours, and through other peer-to-peer contacts. These sites received awards for best practices in the core messages of literacy, behavior, collaboration, transition and family partnerships, and each established themselves as regional leaders in their own core message area.

During SIG2, CalSTAT issued Leadership Site Awards to 28 sites representing 22 LEAs in 14 counties, including:

Sites who received awards committed to:

During SIG2, 28 Leadership Sites were awarded, 23 to schools and 5 to school districts and county offices of education. Leadership site awards were made in five areas:

Integration with other SIG2/CalSTAT Activities

Leadership sites serve California by delivering much of SIG2 TA. Additionally, some of the sites participating in BEST and ERIA have also accessed these TA events, both to reinforce and expand upon the benefits of these programs and to support the systems change necessary to fully implement research-based practices with fidelity.

CST ELA Scores

CalSTAT has used California Standards Testing (CST) English-language Arts (ELA) proficiency as an indicator of academic outcomes at sites with Leadership Site awards.

Improving ELA scores have been an objective of the 25 leadership sites with academic goals. In addition to the 3 literacy sites, all 16 collaboration sites target reading education with collaborative activities. For the 5 behavior sites and 1 family participation site, improving academic outcomes is a perceived benefit of the program, though not the focus of the leadership site award. However, the 3 transition sites have specific goals which do not include academic outcomes and are not included here.

Average Percent of CST ELA Students Proficient and Above at Leadership Sites
25* sites who received a Leadership Site Award during SIG2
    2004
(Prior to SIG2)
2005 2006 2007 2008
All Students Sites Receiving TA 49.0 53.0 56.5 55.9 59.1
Statewide 37.4 41.9 44.8 45.5 48.5
Students with Disabilities Sites Receiving TA 15.0 17.8 24.2 21.6 25.1
Statewide 14.7 17.0 19.5 20.7 24.2
AYP Target (Approximate) 12 23 23 23 34
*3 sites who received Transition awards are not included here because this core message does not include an academic goal.

Leadership Site Profiles and Leadership Cross Site Analysis

Profiles for each leadership site at www.calstat.org offer program overviews, PowerPoint presentations, contact and other information to visitors who want to learn more about innovations and successes in the core message areas. These profiles are available online at http://www.calstat.org/leadershipSites/index.html and have been used as a training tool in a number of contexts throughout SIG2.

Additionally, during SIG2, interviews were conducted with staff from the Leadership Sites to learn more about their systems change processes. Questions focused on what led the site to develop its award-winning program, programmatic details, stages of implementation, and outcomes. Out of these interviews, stories have been developed describing each site’s unique experience and approach. These stories are included in the site profile for each leadership site.

These site profiles and an cross-site accompanying analysis describe “inclusive collaboration,” a term for the common characteristics that emerged from these site’s efforts to improve. The product of this cross-site analysis, They Are All Our Kids, summarizes these common elements and is available on the CalSTAT website at http://www.calstat.org/leadershipcommunity.html.

Stages Leading to “Inclusive Collaboration” at Leadership Sites

While there were a number of variations from one leadership site to another, a distinct progression of implementation was identified across all seventeen SIG2 leadership sites who were interviewed. For example, many sites pursued research-based literacy interventions as part of a data-driven effort to improve academic outcomes, and leadership sites universally experienced resistance by teaching staff to several changes. Leadership sites adopted inclusive collaboration by moving through four phases:

Successes and Results: Most sites reported experiencing positive changes in test scores, grades, student behavior, and school culture. Many sites also reported a reduction in the number of students receiving special education services, both due to fewer student entrances and an increased rate of exits. Additionally, almost all sites reported that inclusive collaboration created a sense of accountability for all students, transcending traditional general and special education divisions. The emergent sense that “they are all our kids” was attributed to the joint effort in teaming, blending classrooms, and creating interventions to assist all students in succeeding.

Regional Institutes

During SIG2, CalSTAT supported a series of Regional Institutes to create an ongoing learning community, sustain and expand systems change efforts, encourage and support meaningful family involvement, expand the capacity of attending school-site teams, and identify and share successful practices.

These institutes were locally sponsored by regional host sites who participated in a competitive award process to received CalSTAT support. Unlike the competitive Leadership Award process, which continued support of sites who maintained high academic standards, Regional Institutes were newly distributed each year. These sites received TA and coaching from CalSTAT on how to effectively deliver professional development activities regionally and locally and were participants at the Statewide Leadership Institute.

During SIG2, CalSTAT funded 21 Regional Institute Host Sites who hosted 49 events. These Regional Host Sites included 10 SELPAs, 10 LEAs, and 1 school representing 15 counties. Many regional Host Sites provided TA in multiple core message areas.

During SIG2, there were 49 Regional Institutes. The professional development provided in these multi-day events were expanded upon and sustained by follow-up TA delivered to institute participants at 73 additional events. The 49 actual Regional Institutes are described below while institute-related TA is summarized with other TA.

Regional Institute Participants by Core Message Area
49 events during SIG2 (Feburary 2005 to June 2008)
5,850 participants
Core Message Participants Percent
GE/SE Collaboration 4,031 69%
Literacy 909 16%
Transition 414 7%
RtI 345 6%
Behavior 120 2%
Family Partnerships 31 < 1%
Regional Institute Particpants by Role
49 events during SIG2 (Feburary 2005 to June 2008)
3,657 of 5,580 participants responding
Role Participants Percent
Teacher: General Education 1,173 32%
Teacher: Special Education 819 22%
Administrator: General Education 432 12%
Other Certificated Professional 502 14%
Administrator: Special Education 298 8%
Program Specialist 93 3%
Paraprofessional 60 2%
Parent/Family 48 1%
Other 232 6%
Average Participant Responses to End-of-Event Surveys
Responses from 6,200 of 13,433 participants at 472 events during SIG2 (February 2005 to June 2008)
Knowledge Prior 2.9
Knowledge After 4.2
Usability 4.4
Overall 4.4

Effective Reading Intervention Academy (ERIA)

The Effective Reading Intervention Academy (ERIA) supports schools in identifying struggling students and trains teachers in providing effective interventions to assist in improving specific student reading skills.

CalSTAT began working with local education agencies in 2004 to bring ERIA to school sites. Each cohort of approximately 10 schools receives training and ongoing support. Sites in West Orange County and Antelope Valley piloted ERIA starting in the 2004-05 school year (Cohort 1), followed by San Joaquin Valley in 2005-06 (Cohort 2) and San Diego County in 2006-07 (Cohort 3). Cohort 1 has conducted a series of district-wide scale-ups, increasing the size of the cohort to 27 sites. Cohort 2 and 3 will expand in the 2008-09 school year.

In the 2007-08 school year, there were 47 active ERIA sites in 18 LEAs and 7 counties, including:

During the 2007-08 school year, implementation of ERIA advanced considerably and student outcomes measurements showed gains.

Of note: Wildfires forced evacuations of the schools and communities around many Cohort 3 ERIA sites last October, interrupting interventions and impacting outcomes.

Key Elements of ERIA

ERIA delivers differentiated instruction to students depending on need. Students who are falling behind or are at risk of falling behind in their reading levels are identified for intensified instruction in key areas to reach proficiency in English-language arts (ELA). As a program, ERIA seeks to embed key principles of evidence-based literacy education in schools throughout California.

The ERIA program includes four key elements:

Assessing Students

Assessment of student reading levels is key to making data-informed decisions about student placement and interventions. Assessment is a multifaceted activity which involves examining existing data sources, such as scores from the California Standards Testing (CST), generating new benchmark data at regular intervals with a variety of measures, and interpreting these data against specific criteria. Three major skill areas are identified for assessment at most ERIA sites:

Three major skill areas are identified for assessment at most ERIA sites:

Assessment in all three areas is encouraged, but sites are only required to turn in scores from the SDQ and oral reading fluency as part of annual program evaluation. Most of these assessments involve one-on-one contact between test administrators and students.

Frequent and comprehensive assessment requires large time commitments; this issue has been mitigated by sites in a number of ways: Many sites use CST data and curriculum-embedded assessments as a primary assessment tool, referring students at risk in these measures for additional testing. One school which applies comprehensive assessments to its entire student population hires a team of substitute teachers to conduct testing, easing the burden on teachers and instruction schedules. Other sites have made use of parent volunteers in a similar role.

Student assessment is only the first step in implementation of ERIA, but many sites report that it has its own inherently beneficial effects in securing faculty interest and staff buy-in. Some sites have instituted weekly faculty meetings to review student progress, and many teachers have begun sharing assessment scores with students weekly. Organizational culture at ERIA sites appears to be shifting to make use of these newly-available sources of data, and sites are reporting that new attention to measurable outcomes has become a source of motivation at all levels.

Specific Reading Intervention Programs

Intervention delivers additional resources to students with additional needs, typically drawing from one or more research-based literacy education programs. Interventions are organized into three tiers, with a core tier serving all students and two additional tiers delivering additional help to students with additional needs. Tiers modulate the intensity of instruction and move towards more individualized intervention, providing struggling students with the resources and support necessary for them to succeed.

Student needs are matched to one of these intervention tiers through assessment data. Decisions about student placement are typically made on a student-by-student basis and are guided by specific criteria. These placement criteria are developed at the site-level, relating the needs of individual students to the reading level of the overall student population and the instruction provided in the core tier.

Instruction is differentiated within each tier as well, allowing students to benefit from literacy programs targeting specific reading skills. Some literacy programs being delivered as interventions at ERIA sites include Read Naturally, Rewards, Language!, Read 180 and The Six Minute Solution. Many literacy interventions in use address oral reading fluency and grade-levelling, while others such as Soar to Success target reading comprehension skills.

Response to Intervention (RtI)

Frequent, comprehensive assessment and data-informed decision-making are ongoing activities at ERIA sites. This data informs regular collaborative review of intervention placements, and adjustments are made in the intervention placement of students as student needs change.

As a program evaluation requirement, ERIA sites turn in student scores on the SDQ and oral reading fluency assessments representing fall and spring outcomes, but sites make many assessments in addition to these. Many sites “benchmark” their students with three annual comprehensive assessments, adding a mid-year testing phase to the schedule required by CalSTAT. Frequent formal and informal assessments are made between these testing periods, often through intervention-embedded assessments.

Like assessment, review of student placements happens through formal and informal processes throughout the year. Collaboration between teachers, administrators, and often parents and students is a key aspect of effectively monitoring and responding to student progress. Data, including recent assessment scores and classroom observations, are recorded and available for review in a variety of formats which vary by site, ranging from basic spreadsheets to sophisticated charting programs. Most sites support weekly or monthly staff meetings to review student outcomes. Changes in student placement can be made at any time during the year.

Systems Change

The ERIA program cultivates structures and expertise at the site level which support and sustain these best practices. Formation of a site-level ERIA program starts with professional development activities, delivered to administrators, teachers, and other educators. ERIA sites regularly access ongoing professional development for help with aspects of literacy education from data-informed decision-making to more active teaching styles that keep students engaged. Participants in these activities return to the school with these skills and proceed to build a site-level ERIA program.

The key component of ERIA at every site is the ERIA Site Team. Site teams are comprised of administrators and teachers who guide implementation, train and advise teachers, and find ways to leverage local resources and initiative to improve outcomes. Most site teams are led by a site team coach (a member of the team with additional training, expertise, and responsibilities) who works in association with cohort coaches (literacy specialists coaching many schools in many LEAs), district staff, and reading specialists including Anita Archer, Jan Hasbrouck, and Kevin Feldman.

Site teams have a high level of autonomy and are encouraged to adapt the core ERIA program into a unique site-level ERIA program. Site teams work closely with cohort coaches and district officials to adapt key elements in a way that works at the local level. Changes site teams have influenced include integrating time for assessments into the master schedule, reserving the necessary time in instruction to accommodate interventions, and establishing mechanisms for collaborative review. Experimentation and innovation in site team planning is both welcome and encouraged in ERIA.

Site teams have also found ways to meet challenges to full, schoolwide implementation of ERIA’s key elements. For many sites, implementation has been delayed by a lack of resources to conduct comprehensive assessments, too few teachers trained in interventions, and not enough time for collaborative review of student progress. Hiring substitute teachers or recruiting parent volunteers to conduct assessments are examples of how site-level innovation can create effective solutions.

Evaluation and Planning Tools

In planning the implementation of ERIA, site teams develop a Site Action Plan (SAP). This SAP is an electronic form which asks the site team to detail their site-level ERIA program by responding to 24 open-ended questions. The SAP is then revisited by the site team at least twice per year to facilitate reevaluation of strategies and progress, and allowing updates to be made as necessary. The SAP is being accessed as an evaluation instrument as well, creating a coordinated picture of how ERIA’s core elements have been adapted and implemented across all ERIA sites.

Another instrument, the Team Implementation Checklist (TIC), is a companion piece to the SAP collecting implementation data according to 26 specific items and helps the site team evaluate its progress in implementing the four key elements of ERIA.

ERIA in Practice

The 2007-08 school year was the first to see use of the Team Implementation Checklist (TIC). This tool allows ERIA Site Teams to summarize site implementation of ERIA according to 26 explicit criteria. The TIC is complementary to the Site Action Plan (SAP); these forms reinforce one another and are completed at the same time, at the beginning and end of the school year, reflecting changes in both ERIA planning and practice.

The TIC is distributed and completed by sites in an electronic form called the TIC Charting Program. This Excel-based form automatically generates longitudinal, graphic charts as site teams add data, so they can see and use this information in their planning processes without waiting for evaluator feedback.

The average of all 26 items at each site has been explored as an important measure of ERIA implementation. Responses to each item are given as “achieved,” “in progress” or “not started,” and interpreted as 100%, 50% and 0% implementation, respectively.

As of Spring 2008, a majority of sites were fully implementing ERIA.

  Fall 2007 Spring 2008
  Number Percent Number Percent
Full Implementation 14 sites 30% 28 sites 60%
Partial Implementation 29 sites 62% 18 sites 38%
Minimal Implementation 4 sites 8% 1 sites 2%

The table below explores implementation of each of the 26 checklist items as averaged across all ERIA sites.

Patterns of Implementation at the Site Level

Among all active ERIA sites, 20 are fully implementing ERIA schoolwide (43%). The 27 sites which have not yet reached full implementation schoolwide have typically followed one of two strategies:

Average Implementation of ERIA Elements

In ERIA’s key element areas of assessment, intervention, and response to intervention, these gains resulted in the full implementation of 12 of 16 TIC items as an average across all ERIA sites.

Implementation of systems change is advancing more slowly than other key elements of ERIA. Administrator and structural supports appear to be the most advanced aspect of this area (D1 and D8), with principals and site teams providing active leadership. However, most TIC items in this area are not yet being fully implemented.

Average Implementation of ERIA Elements
47 ERIA sites, 2007-08 school year
ERIA Key Element Specific Criteria Fall 2007 Spring 2008
Assessing Students A1. CST ELA Proficiency of each student is examined 96% 98%
A2. Decoding skills of students less than Proficient are assessed 86% 92%
A3. Fluency skills of students less than Proficient are assessed 95% 96%
A4. Comprehension skills of students less than Proficient are assessed 76% 79%
A5. Specific criteria exist for reading intervention placementA6. 81% 91%
A6. Specific-skill reading intervention needs have been determined forthe school as a whole, based upon student assessments 75% 82%
Specific Intervention Programs B1. Based upon the needs of the school, research-based specific-skill reading interventions have been purchased and are in place 76% 87%
B2. Staff have been trained in use of reading intervention programs 68% 78%
B3. Intervention placement criteria are used to match and exit students 68% 81%
B4. Students needing intervention(s) are receiving them regularly 74% 87%
B5. Reading intervention programs are being used with fidelity 69% 78%
B6. Periodic tests and/or measures from the intervention programs are being used to monitor student progress 68% 81%
Response to Intervention C1. Initial assessment tests (decoding, fluency, etc.) are repeated regularly to inform the review of intervention placements 74% 94%
C2. Progress monitoring data are recorded and charted for ease of use 62% 82%
C3. Regular, collaborative review of individual student progress and intervention placements is occurring 63% 74%
C4. Multiple levels of interventions are provided ranging in intensity 69% 84%
Systems Change D1. An ERIA Site Team is guiding implementation 65% 77%
D2. The school principal is active in leading ERIA implementation 72% 82%
D3. Site Team members communicate regularly (formally/informally) 66% 77%
D4. Coaching is provided to support implementation with fidelity 48% 56%
D5. Regular fidelity observations are done by administrators/coaches 32% 47%
D6. Ongoing professional development activities are taking place 60% 74%
D7. Time is provided for collaboration on a regular basis 70% 78%
D8. Schedule reflects required time to accommodate interventions 76% 88%
D9. The ERIA/RTI process is being implemented schoolwide 60% 70%
D0. The School Site ERIA program is part of a district-wide scale-up 59% 68%

ERIA Continues to Grow

CalSTAT has supported specific schools in cohorts across California in developing a literacy intervention program through teaching best practices, providing cohort coaches, and funding associated with the ERIA program. ERIA is prepared to expand further, as well. The 2008-09 school year will see additional scale-up activities grow the size of Cohorts 2 and 3 by roughly double; Cohort 1 will likely expand again as well.

District-wide Scale-up

Districts have taken a leading role in promoting implementation of ERIA. District administrators have motivated sites to participate in ERIA, worked with cohort coaches to provide content trainings to sites and supported site administrators in implementing ERIA at the school level.

Once the four key elements of ERIA are fully implemented in a portion of the schools within a district, the districts participating in ERIA have initiated scale-up activities, spreading these practices beyond the original ERIA sites to other schools in the district.

23 site teams (49%) noted achieving the checklist item “The School Site ERIA program is part of a district-wide scale-up” on the TIC, and another 32% said that it was in progress.

As of the 2007-08 school year, the Ocean View School District has expanded the ERIA program to every school in the district. Additional Cohort 1 school districts, Westminster Elementary and Huntington Beach City Elementary, are also expanding the ERIA program within their districts.

ERIA Sites Received CalSTAT Leadership Site Awards

Another CalSTAT program, the Leadership Site Award competition, identifies sites implementing effective, evidence-based best practices. These sites are given additional resources to develop their programs, give demonstrations to other schools, and serve as regional model sites. Three of the first ERIA sites applied for and received Leadership Site Awards in the core message area of literacy, all in the Ocean View School District: Marine View Middle, Mesa View Middle, and Vista View Middle.

Data Collection for Summary Reporting

Data generated by sites around two assessments—the SDQ and oral reading fluency—is also forwarded to CalSTAT for evaluation purposes, once in the fall and once in the spring. All ERIA outcomes described here are from the 2007-08 school year.

These assessment scores allow sites to make data-based intervention decisions, delivering specific, targeted help to students based on their individual needs. To allow CalSTAT to aggregate data from nearly fifty schools, sites have been asked to make assessments based on Summary Reporting Criteria, which standardize assessments between sites. 84% of student data received for the 2007-08 school year met these criteria, up from 57% in 2006-07.

Summary Reporting Criteria

Note: CalSTAT does not received student-level data from sites including any student identifiers. Student anonymity and privacy is preserved above the site level for all sites and students participating in ERIA.

Student Data Reported by Cohort
  Number Percent
Cohort 1 3,455 43%
Cohort 2 1,075 13%
Cohort 3 1,941 24%
Cohort 1 Expansion 1,620 20%
Total Student Data Received 8,091 100%
Student Data Reported by Content of Data
  Number Percent
Students Struggling with Decoding, Fluency, or Both 4,498 55%
Students Not Struggling in Decoding or Fluency 2,322 29%
Data Not Meeting Summary Criteria 1,271 16%
Total Student Data Received 8,091 100%

Assessments represented in this report include only those meeting summary reporting criteria.

Many sites conducted a large number of student assessments that were incompatible with summary reporting criteria. It is important to note that such measurements may still be valuable in improving student outcomes. While sites are asked to make and report measures to CalSTAT which can be aggregated with those of other sites, ERIA site teams are encouraged to continue making decisions about how best to assess students at their sites.

Additionally, many sites assess only a portion of their students with the SDQ or oral reading fluency measures. Because of time pressures on faculty implementing these tests, existing data is often used to identify students who may be at risk before additional testing is conducted. Consequently, the number of student data reports received by CalSTAT may not be fully indicative of the number of students being monitored for possible intervention across all ERIA sites.

San Diego Quick

The San Diego Quick assessment requires students to identify words on a grade-levelled vocabulary list, with correct identification of 8 or more of the 10 words representing grade-level skill. Student data is only included here if it meets summary reporting criteria and the student is testing below grade level in fall assessments (struggling).

Students who are only one grade-level behind (yellow in the chart) are students at risk, but not necessarily high risk. ERIA sites have shown progress helping these students grow, as well as helping students at greater risk (more than 1 grade level behind) move closer to grade-level literacy.

Struggling Students on the San Diego Quick by Cohort
  Cohort 1 Cohort 2 Cohort 3
Struggling Students in Cohort 1,698 174 951
Schools in Cohort 23 10 8
Spring 2007: Decoding at or Above Grade Level (total) 45% 43% 27%
Spring 2007: Decoding Below Grade Level (total) 55% 57% 73%
Struggling Students by Relative Grade-level Level
  Cohort 1 Cohort 2 Cohort 3
  Fall Spring Fall Spring Fall Spring
Ahead 1 Grade Level or More 0% 17% 0% 22% 0% 12%
Decoding at Grade Level 0% 28% 0% 21% 0% 15%
Behind 1 Grade Level 51% 27% 44% 22% 39% 18%
Behind More than 1 Grade Level 49% 28% 56% 35% 61% 55%
San Diego Quick Assessments of Struggling Students by School
Cohort School N Fall Spring
Behind > 1 GL Behind 1 GL Behind > 1 GL Behind 1 GL Decoding at GL Ahead 1 GL or More
Cohort 1 Cohort 1 Overall 1,698 49% 51% 28% 27% 28% 17%
Dwyer (Ethel) Middle 234 53% 47% 33% 27% 30% 10%
Finley Elementary 49 59% 41% 33% 29% 20% 18%
Issac L. Sowers Middle 32 44% 56% 38% 25% 28% 9%
Johnson Middle 355 58% 42% 31% 28% 27% 14%
Marine View Middle 20 15% 85% 5% 25% 45% 25%
Meairs Elementary 56 52% 48% 32% 14% 36% 18%
Mesa View Middle 88 51% 49% 38% 41% 16% 6%
Peterson (John R.) Elementary 41 42% 59% 22% 10% 37% 32%
Quartz Hill Elementary 190 28% 72% 10% 28% 33% 29%
Star View Elementary 28 50% 50% 4% 36% 50% 11%
Vista View Middle 439 48% 52% 31% 26% 21% 22%
Webber Elementary 61 56% 44% 30% 28% 36% 7%
Sites with Fewer than 20 Struggling Students (11 sites) 105 50% 50% 20% 30% 40% 10%
Cohort 2 Cohort 2 Overall 174 56% 44% 35% 22% 21% 22%
Alvina Elementary Charter 31 42% 58% 45% 42% 10% 3%
Oakhurst Elementary 23 39% 61% 22% 22% 44% 13%
Pioneer Elementary 22 32% 68% 14% 14% 14% 59%
Sites with Fewer than 20 Struggling Students (6 sites) 98 69% 31% 40% 17% 21% 21%
Cohort 3 Cohort 3 Overall 951 60% 40% 54% 18% 16% 12%
Campo Elementary 26 27% 73% 8% 19% 35% 39%
Dukes (James) Elementary 37 38% 62% 3% 38% 57% 3%
Mt. Woodson Elementary 23 57% 44% 30% 26% 35% 9%
Riverview Elementary / Winter Gardens Elementary 36 33% 67% 8% 31% 33% 28%
Spring Valley Middle 807 63% 37% 62% 16% 11% 11%
Sites with Fewer than 20 Struggling Students (3 sites) 22 50% 50% 9% 36% 50% 5%

The student outcomes of individual ERIA sites are identified above and below.

Sites with fewer than 20 struggling students are not identified individually in this report to protect student confidentiality.

Cohort 3 outcomes were suppressed this year compared to last year. This may be due, in part, to wildfires in October that disrupted ERIA implementation. Additionally, a large portion (85%) of data generated in Cohort 3 came from a single site, Spring Valley Middle School, which has implemented schoolwide decoding assessment but does not yet describe the ERIA’s four key elements being either fully implemented or implemented schoolwide.

Oral Reading Fluency

Oral reading fluency assessment tracks incremental changes in the number of words read aloud in a minute from a passage of text specifically calibrated by grade level. Student data is only included if it met the summary reporting criteria and the student showed oral reading fluency scores below the 50th percentile in fall (struggling).

In order to monitor whether ERIA has helped enable accelerated learning at sites, change in WCPM values was determined for all struggling students (spring WCPM - fall WCPM). This was further calculated as a percentage of typical change in WCPM, based upon Hasbrouck and Tindal’s 2004 oral reading fluency study.

Because struggling students are reading more slowly than the grade-level median, greater-than-typical growth is needed to reach the fluency level of other students. Growth above that typical for other students is shown below in blue and green.

Struggling Students in Oral Reading Fluency by Cohort
  Cohort 1 Cohort 2 Cohort 3
Struggling Students in Cohort 2,151 574 328
Schools in Cohort 26 8 9
Growth Above Typical Growth (total) 56% 55% 42%
Growth Equal to or Less than Typical Growth (total) 44% 45% 58%
Growth Above 150% 32% 27% 16%
Growth > 100% to 150% 24% 28% 26%
Growth > 50% to 100% 25% 29% 32%
Growth Below 50% or Decline 19% 16% 26%
Oral Reading Fluency Assessments of Struggling Students, by School
Cohort School N Growth Below 50% or Decline Growth > 50% to 100% Growth > 100% to 150% Growth Above 150%
Cohort 1 Cohort 1 Overall 2,151 19% 26% 24% 32%
Anderson Elementary 24 21% 58% 13% 8%
College View Elementary 32 16% 44% 16% 25%
DeMille Elementary 155 25% 34% 23% 19%
Dwyer (Ethel) Middle 261 20% 23% 16% 42%
Finley Elementary 86 17% 27% 33% 23%
Jessie Hayden Elementary 62 23% 37% 27% 13%
Johnson Middle 103 27% 29% 17% 26%
Webber Elementary 83 17% 22% 35% 27%
Lake View Elementary 21 29% 29% 29% 14%
Meairs Elementary 89 26% 40% 21% 12%
Mesa View Middle 90 31% 24% 22% 22%
Peterson (John R.) Elementary 58 21% 36% 24% 19%
Quartz Hill Elementary 354 3% 16% 32% 49%
Spring View Middle 94 39% 21% 13% 27%
Star View Elementary 27 30% 37% 26% 7%
Top of the World Elementary 49 12% 33% 27% 29%
Village View Elementary 21 29% 43% 14% 14%
Vista View Middle 446 18% 20% 22% 40%
Sites with Fewer than 20 Struggling Students (8 sites) 96 19% 34% 32% 15%
Cohort 2 Cohort 2 Overall 574 16% 29% 28% 27%
Alvina Elementary 66 30% 27% 23% 20%
Centerville Elementary 21 19% 33% 29% 19%
Oakhurst Elementary 58 12% 48% 24% 16%
Pioneer Elementary 75 11% 27% 29% 33%
Sierra View Elementary 20 15% 40% 25% 20%
Webster Elementary 51 18% 39% 29% 14%
Wilson Middle 270 12% 24% 30% 33%
Sites with Fewer than 20 Struggling Students (1 site) 13 46% 15% 23% 15%
Cohort 3 Cohort 3 Overall 328 26% 32% 26% 16%
Campo Elementary 48 21% 44% 29% 6%
Dukes (James) Elementary 74 27% 26% 34% 14%
Marshall (Thurgood) Elementary 64 34% 33% 22% 11%
Mt. Woodson Elementary 37 32% 22% 24% 22%
Riverview Elementary 60 30% 43% 13% 13%
Sites with Fewer than 20 Struggling Students (4 sites) 45 9% 22% 31% 38%

California Standards Test (CST) Proficiency

Monitoring CST English Language Arts (ELA) proficiency scores has been a focus of ERIA evaluation, just as it is a focus for all California schools. The following chart averages the percent of students proficient and above schoolwide, with all sites weighted equally.

For each cohort, CST ELA proficiency prior to involvement with ERIA is noted with a dotted line. Cohort 1 is divided between the original ERIA sites (Cohort 1) and those who joined in the 2007-08 school year (Cohort 1 Expansion).

Average CST ELA Proficiency of All Students at ERIA Sites by Cohort
  2004 2005 2006 2007 2008
AYP Target (approximate) 12% 23% 23% 23% 34%
Statewide Proficiency 37% 42% 45% 46% 48%
Cohort 1 (9 sites starting in 2004-05) 48% 54% 57% 57% 59%
Cohort 2 (11 sites starting in 2005-06) 41% 45% 49% 48% 53%
Cohort 3 (9 sites starting in 2006-07) 43% 44% 50% 53% 53%
Cohort 1 Expansion (18 sites starting in 2007-08) 48% 54% 58% 58% 58%
Average CST ELA Proficiency of Students with Disabilities at ERIA Sites by Cohort
  2004 2005 2006 2007 2008
AYP Target (approximate) 12% 23% 23% 23% 34%
Statewide Proficiency 15% 17% 20% 21% 24%
Cohort 1 (9 sites starting in 2004-05) 13% 17% 27% 21% 26%
Cohort 2 (11 sites starting in 2005-06) 7% 12% 14% 17% 20%
Cohort 3 (9 sites starting in 2006-07) 14% 14% 24% 26% 32%
Cohort 1 Expansion (18 sites starting in 2007-08) 20% 26% 32% 28% 32%

The chart below shows CST ELA trends for individual sites, as well as site-specific implementation data from the TIC.

Implementation of ERIA and CST ELA Proficiency at All ERIA Sites
Cohort School Start Year Implementation Schoolwide Implementation CST ELA: All Students CST ELA: Students with Disabilities
Fall Spring Fall Spring Start Year 2008 Start Year 2008
Cohort 1 Cohort 1 Average

2004-05

76% 85% 78% 84% 48% 59% 13% 26%
DeMille Elementary 2004-05 75% 88% Full Full 38% 44% 25% 26%
Dwyer Middle 2004-05 94% 84% Full Partial 58% 69% 10% 28%
L.P. Webber Elementary 2004-05 75% 87% Full Full 33% 45% 10% 23%
Marine View Middle 2004-05 52% 69% Partial Partial 59% 67% 20% 16%
Mesa View Middle 2004-05 75% 90% Full Full 54% 71% 8% 28%
Peterson Elementary 2004-05 69% 75% Partial Full 64% 69% 24% 58%
Quartz Hill Elementary 2004-05 79% 92% Partial Full 43% 48% 6% 25%
Spring View Middle 2004-05 67% 83% Partial Partial 47% 67% 7% 18%
Vista View Middle 2004-05 94% 96% Full Full 39% 56% 3% 11%
Cohort 2 Cohort 2 Average 2005-06 67% 74% 46% 55% 45% 53% 12% 20%
Alvina Elementary Charter 2005-06 58% 75% Full Full 33% 34% . 21%
Centerville Elementary 2005-06 63% 87% Partial Full 35% 60% . 0%
Exeter High 2005-06 69% 50% No No 50% 64% . 32%
Herbert Hoover High 2005-06 38% 38% No No 44% 51% 5% 8%
McLane High 2005-06 62% 65% No No 22% 30% 6% 8%
Oakhurst Elementary 2005-06 83% 85% Full Full 49% 51% . 20%
Pioneer Elementary 2005-06 67% 87% Partial Partial 60% 60% 36% 54%
Ranchos Middle 2005-06 56% 65% Partial Partial 56% 62% 15% 10%
Sierra View 2005-06 67% 83% No Partial . 61% . 15%
Webster Elementary 2005-06 92% 92% Full Full 59% 64% 4% 28%
Wilson Middle 2005-06 83% 87% Partial Partial 38% 48% 19% 3%
Cohort 3 Cohort 3 Average 2006-07 78% 85% 67% 75% 50% 53% 24% 32%
Campo Elementary 2006-07 92% 94% Full Full 36% 53% 22% 18%
Clover Flat Elementary 2006-07 90% 100% Full Full 38% 39% 29% 0%
James Dukes Elementary 2006-07 75% 75% Partial Partial 74% 72% 51% 46%
Marshall (Thurgood) Elementary 2006-07 87% 90% Full Full 62% 69% 23% 32%
Mt Woodson Elementary 2006-07 50% 85% Partial Full 60% 61% 38% 44%
Pine Valley Elementary 2006-07 . 85% . Full 59% 59% 15% 0%
Potrero Elementary 2006-07 90% 94% Partial Partial 28% 29% . 0%
Riverview Elementary/ Winter Gardens 2006-07 69% 69% Partial Partial 47% 50% 27% 42%
Spring Valley Middle 2006-07 63% 75% No Partial 46% 46% 11% 7%
Cohort 1 Expansion Cohort 1 Expansion Average 2007-08 63% 79% 56% 71% 58% 58% 28% 32%
Anderson Elementary 2007-08 . 56% . Partial 49% 53% 14% 22%
Circle View Elementary 2007-08 71% 81% Full Full 81% 83% 61% 49%
College View Elementary 2007-08 85% 96% Full Full 49% 50% 33% 25%
Finley Elementary 2007-08 58% 73% Partial Partial 32% 29% . 3%
Golden View 2007-08 75% 85% Full Full 57% 59% 22% 22%
Harbour View 2007-08 48% 81% No Partial 72% 75% 37% 32%
Hayden Elementary 2007-08 58% 69% Partial Partial 55% 56% 43% 56%
Hope View 2007-08 56% 62% Partial Partial 76% 78% 47% 57%
Johnson Middle 2007-08 40% 69% No No 40% 46% 16% 18%
Lake View Elementary 2007-08 65% 100% Partial Full 49% 55% 21% 46%
Meairs 2007-08 83% 96% Partial Partial 47% 46% 14% 10%
Oak View 2007-08 81% 100% Full Full 37% 37% 22% 15%
Sowers Middle 2007-08 54% 52% No Partial 73% 70% 31% 32%
Star View Elementary 2007-08 73% 88% Full Full 63% 66% 17% 30%
Sun View Elementary 2007-08 56% 54% No Partial 59% 44% 38% 24%
Top of the World 2007-08 40% 67% Partial Full 77% 79% . 57%
Village View 2007-08 50% 75% Partial Partial 71% 70% 45% 50%
Westmont Elementary 2007-08 88% 90% Full Full 54% 57% 20% 23%
Start Year: The first year ERIA began to be implemented at the site.
Implementation: Overall implementation from all 26 TIC items.
Schoolwide Imp: response to TIC item D9, “The ERIA/RtI process is being implemented schoolwide.”
CST ELA: Percent proficient and above for “All Students” and the “Students with Disabilities” subgroup.

Building Effective Schools Together (BEST)

BEST is a program based on positive behavioral supports (PBS) that helps schools develop and implement positive school rules, rule teaching, and positive reinforcement systems schoolwide. To date, over 450 sites and almost 5,000 people have been trained in PBS through the BEST program, including ongoing trainings that have reached over 100 sites and 800 people through the second California State Improvement Grant (SIG2, February 2005 to June 2008), administered by CalSTAT on behalf of the California Department of Education, Special Education Division.

Nearing the end of SIG2, a total of 105 BEST sites remain active. Of these sites, 54 were newly trained in the 2006-07 school year. The other 51 sites were initially trained earlier in SIG2 and have continued to implement PBS and receive booster trainings.

Each BEST school site is served by one of seven BEST regional cadres of trainers which are comprised of local program specialists, administrators, parents and others. Personnel from Institute on Violence and Destructive Behavior (IVDB) at the University of Oregon provided trainings and support to the BEST cadre of trainers and sites in California. These cadres of trainers then provide the initial and follow-up training for school sites as well as assisting each site in implementing BEST. The cadres of trainers are distributed statewide and serve a diverse range of California schools: both rural and urban, coastal and central, in Northern and Southern California.

California's SIG2 BEST/PBS Cadres and Sites

Implementation of BEST

BEST functions through school-level “site teams”—a collection of administrators and faculty, often including parents and district counselors. These site teams are formally trained in BEST and in turn guide and monitor implementation of the program at their school.

Site teams adapt standard PBS elements and develop new ones as necessary to make sure the program is functional and effective at the local level. This process is driven by site level dialogue and guided by attention to data. Because of the innovations of BEST site teams, there is a great deal of diversity in how BEST is implemented across California.

Common elements of BEST throughout California include:

“All of the teachers have formulated their classroom rules around the concepts of Be Safe, Be Respectful, and Be Responsible. We have trained the students on cafeteria behavior, playground behavior, assembly behavior, etc. Rules have been explicitly taught and practiced. We have a schedule for reteaching the rules as well. We are off to a great start with BEST!”
–General Education Administrator

We implemented the new reward system, and it is making a big difference. The kids love getting them and are trying their best to receive one.
–Parent

“We also have a ‘Caught Being Good’ program, and the kids get tags to hang from their backpack. They can also add perfect attendance tags and academic achievement tags.”
–General Education Teacher

It has been helpful to have a common language that all staff members can use with the students!
–General Education Teacher

Starting in Spring 2007, evaluation of BEST has included use of the PBS Team Implementation Checklist (TIC) developed by Rob Horner and George Sugai to monitor implementation of the program. This instrument asks sites to self-assess degree of implementation according to 17 items relating to activities such as staff buy-in, defining expectations, and teaching behavior expectations.

Responses to each item are given as “achieved,” “in progress” or “not started,” and interpreted as 100%, 50% and 0% implementation, respectively, for the purposes of aggregated analysis.

If all 17 checklist items average to 80% or more, a site is considered to be fully implementing BEST. Most sites averaged at least 50% but less than 80%, and are considered to be partially implementing. Below 50%, sites are considered to be minimally implementing.

BEST Sites by Degree of Implementation N=105 sites
  Spring 2007 Fall 2007
Fully Implementing 15 sites, 14% 27 sites, 26%
Partially Implementing 38 sites, 36% 39 sites, 37%
Minimally Implementing 19 sites, 18% 14 sites, 13%
Sites not Reporting 33 sites, 32% 25 sites, 24%
Total Fully and Partially Implementing 53 sites, 50% 66 sites, 63%

High staff transiency was described as a major hurdle to implementation in some areas, dissolving newly-created site teams year after year. Having time to meet and plan for implementation was another significant hurdle reported.

Implementation by Cadre

BEST is organized around the seven regional trainer cadres presented on page one. Sites within these regional cadres showed varying levels of participation in the Fall 2007 TIC data collection, as well as varying degrees of implementation.

BEST Cadre Sites by Level of Implementation of Start-up Activities
Fall 2007 -- 105 BEST Sites*
  Fully Implementing Partially Implementing Minimally Implementing Site did not submit data
San Francisco (18 sites) 10 sites, 56% 6 sites, 33% 2 sites, 11% 0 sites, 0%
Los Angeles (31 sites) 10 sites, 32% 13 sites, 42% 7 sites, 23% 1 site, 3%
Butte (11 sites) 3 sites, 27% 5 sites, 46% 3 sites, 27% 0 sites, 0%
Redding (14 sites) 3 sites, 22% 9 sites, 64% 1 site, 7% 1 site, 7%
Napa (8 sites) 0 sites, 0% 4 sites, 50% 0 sites, 0% 4 sites, 50%
Sonoma (14 sites) 1 site, 7% 1 site, 7% 0 sites, 0% 12 sites, 86%
Placer (9 sites) 0 sites, 0% 1 site, 11% 1 site, 11% 7 sites, 78%
Total (105 sites) 27 sites, 26% 39 sites, 37% 14 sites, 13% 25 sites, 24%
*Since Spring 2007, the number of BEST sites has declined slightly. Two Butte sites merged into a single school and one San Francisco site was moved out of the district. The Santa Barbara cadre did not continue to participate in BEST (3 sites). Also, two schools in Placer are served by a single site team, and are counted here as a single site.

Implementation of Individual BEST Elements

The chart on page seven shows the average degrees of implementation for each of the 17 TIC checklist items by all sites who completed a TIC in Fall 2007. Average responses from Spring 2007 are included as well for comparison purposes (as the gray shadow columns). Of the 17 items, 14 showed increases in implementation.

Items in the TIC are grouped into six categories which relate to different aspects of the BEST program. These and the individual items are explored here and on the next page as averages across all BEST sites. BEST empowers site teams to make implementation decisions at the site level, resulting in a diversity of approaches, so it is important to recognize that an individual school may deviate from these averages.

Exploring average implementation of these items individually, BEST sites overall have demonstrated their commitment and active use of schoolwide positive behavior supports. These elements are seen mostly in the TIC categories of establishing commitment, establishing and maintaining the site team, and establishing school-wide expectations.

Strong responses to these five items in the TIC demonstrate key aspects of the BEST program overall. Site teams, with support from administrators and staff with relevant expertise, appear to be in place in most BEST sites. BEST sites are also widely implementing use of a concise list of behavioral expectations which they reinforce with a formal reward system. Both of these latter two elements are described on pages two and three of this report.

Meanwhile, the use and integration of data in day-to-day implementation of BEST has seen more moderate progress across BEST sites overall, despite strong implementation and successes in many individual sites. These checklist items are seen primarily in the categories of self-assessment and establishing an information system.

Finally, many sites have also been slow to develop a plan to identify and establish systems for teacher support, functional assessment, and development and implementation of a support plan (item 17). It appears that many BEST sites would benefit from continued support in cultivating behavioral expertise and fostering the ongoing development of their programs through time.

Average Implementation of BEST Elements
Checklist Item Spring
(N=71 sites reporting)
Fall
(N=80 sites reporting)

Establish Commitment

1. Administrator’s support & active involvement.

93% 91%

2. Faculty/Staff support (One of top 3 goals, 80% of faculty document support, 3 year timeline).

63% 70%

Establish & Maintain Team

3. Team established (representative).

83% 89%

4. Team has regular meeting schedule, effective operating procedures.

54% 70%

5. Audit is completed for efficient integration of team with other teams/initiatives addressing behavior support.

42% 46%

Self-Assessment

6. Team/faculty completes EBS self-assessment survey.

80% 74%

7. Team summarizes existing school discipline data.

55% 64%

8. Strengths, areas of immediate focus & action plan are identified.

59% 63%

Establish School-wide Expectations

9. 3-5 school-wide behavior expectations are defined.

87% 92%

10. School-wide teaching matrix developed.

65% 64%

11. Teaching plans for school-wide expectations are developed.

50% 61%

12. School-wide behavioral expectations taught directly & formally.

60% 69%

13. System in place to acknowledge/reward school-wide expectations.

72% 79%

14. Clearly defined & consistent consequences and procedures for undesirable behaviors are developed.

55% 64%

Establish Information System

15. Discipline data are gathered, summarized, & reported.

51% 64%

Build Capacity for Function-based Support

16. Personnel with behavioral expertise are identified & involved.

65% 77%

17. Plan developed to identify and establish systems for teacher support, functional assessment & support plan development & implementation.

40% 51%

Ongoing Support for Positive Behavior

BEST’s successes are attributable to the partnerships and teamwork of the agencies and personnel working together to build safer, more productive schools. A theme of partnerships is pervasive in all levels of the program, with coordinated efforts between CalSTAT, the Institute on Violence and Destructive Behavior (IVDB), and schools and districts throughout California. Nowhere have these collaborations been more successful than between the cadres of trainers, school faculty and administrators, parents, and even students.

The positive outcomes of BEST are expected to continue for many schools through the ongoing work of many of these partners and with two additional sources of support.

CalSTAT administers many other education resources, and these have been open to BEST sites who have been interested in pursuing them. Interactions between BEST sites and CalSTAT’s Leadership Site Award program have been particularly productive.

In Los Angeles Unified School District, the school board recently approved a district-wide PBS program by passing the Discipline Foundation Policy in March 2007.

BEST sites in the district modeled many of the policies that will be advanced under this policy and will continue to receive PBS program support from LAUSD. Additionally, members of the Los Angeles cadre of trainers have become key members in the policy’s Implementation Task Force.

“Every student, pre-school through adult, has the right to be educated in a safe, respectful and welcoming environment. Every educator has the right to teach in an atmosphere free from disruption and obstacles that impede learning. This will be achieved through the adoption and implementation of a consistent school-wide positive behavior support and discipline plan for every school in LAUSD.”
Discipline Foundation Policy
Policy Bulletin, March 27, 2007

Stronger Communities, Safer Schools

The most pervasive outcome reported by the teachers, administrators, and parents involved was a cultural shift at schools implementing BEST. By emphasizing the teaching and support of positive behaviors among students, discipline became a proactive enterprise at many BEST sites, with faculty working with students to create a safer and more congenial learning environment.

BEST faculty members related many situations in which students took initiative in promoting this better environment, several of which have been described earlier in this report.

In another site, the Three Be’s were rebranded as “Cougar Pride,” after the school mascot. The site rewards good students by recruiting them to read morning PA announcements. These students adopted, without prompting, the practice of reciting the Cougar Pride tenets every morning at the conclusion of the morning announcements.

Another community-building effort enlisted faculty and staff in a “valet” service each morning, helping students out of cars and with their backpacks. By greeting students and parents each morning, the school created a safer and more friendly environment, starting off every school day on a high note.

One site has created a “village” environment by setting up “special friend” relationships between students and faulty members. These staff-student pairings are created so that faculty members (who intentionally do not have a professional relationship with the student) take an interest in the student, checking up on them and getting to know them. This creates a “trickle” effect where students see teachers caring outside of their roles as teachers, building a community environment.

Many sites have also found ways of enlisting student support in modeling and rewarding positive behaviors. For example, one school recruits students from third and fourth grade to serve as “play leaders” for first and second graders. These leaders apply for the position not unlike applying for a job, receive training and serve roles similar to faculty members in modeling and rewarding positive behaviors. Play leaders become part of the playground and classroom communities of these younger students, learning their names and working with them over time.

“We have started a BEST Club, where students help with how to follow the rules. They also run a table once a week that give the kids a chance to give drops [rewards] to students and faculty.”
–General Education Teacher

Measurable, Behavior-related Outcomes

Changes in the use and application of office discipline referrals (ODRs) varied considerably, with about half of the sites decreasing their use of ODRs, while the other BEST sites saw increases, including 39% whose use of ODRs doubled or more.

These measures are not based on negative behaviors themselves, but on the response of school officials to negative behavior. Increases may be due to changing school policies and practices, such as more consistent enforcement of school rules or enhanced awareness of negative behavior and its impact on a school’s academic mission. Consequently, the increase or decrease of ODR (or suspension) rates at a given site may or may not be evidence of increased incidence of negative behaviors. Clearly, data collected with these measures suggest a wide range of outcomes.

Change in ODR Rate at Partially or Fully Implementing Sites
N=31 sites, comparing 2006-07 School Year to Baseline (2003-04 to 2005-06)
  Number of Sites Percent
Decrease by More than Half 10 32%
Decrease by up to Half 6 19%
No Change or Increase Less than Double 3 10%
Increase by Double or More 12 39%

One of the measures showing the largest decreases was of the rate of students receiving more than a single day of suspension over the school year. For 44% of sites who reported a baseline for this statistic, this rate decreased by more than half. This indicates a sharp decline in the number of students suspended for major infractions or with continued, serious behavior problems.

The PBS program taught by BEST utilizes an RTI (response to intervention) approach. In RTI, students are identified for one of three levels of support according to need (support for all students, for at-risk students, or for high-risk students). Students receiving suspensions would be identified for additional support in this RTI-based behavior model, which may be contributing to the decreases seen in students receiving more than one day of suspension at these BEST sites.

Change in Rate of Students with More than One Day of Suspension at Partially or Fully Implementing Sites
N=27 sites, comparing 2006-07 School Year to Baseline (2003-04 to 2005-06)
  Number of Sites Percent
Decrease by More than Half 12 44%
Decrease by up to Half 4 15%
No Change or Increase Less than Double 7 26%
Increase by Double or More 4 15%

The rate of suspensions and ODRs at fully and partially implementing BEST sites for whom baseline data was available are presented on the following page. These sites represent considerable diversity both in student populations and in how they chose to pursue a schoolwide PBS program. These 38 sites suggest the following:

It is important to note that these rates can change for a number of reasons. Changes in school policy, which many BEST site teams chose to implement, could have had a large impact. Examples of this include greater reliance on non-suspension responses to misbehavior or expanding use of ODRs to respond to a wider spectrum of negative behaviors. Increased awareness of behavior issues and attention to accurate record keeping may also be responsible for increases or decreases in these statistics.

Many sites described positive outcomes as a response to increased attention to ODR and suspension data. Indicative stories include the following:

One cadre of trainers has used teleconferencing software to address data collection and interpretation, having all site teams complete surveys and compile data at the same time. This has resulted in 100% data collection for CalSTAT’s purposes, as well as helping sites share information and reinforce each other’s efforts.

Establishing a system for effectively recording, tracking, and accessing data at the site level was noted as a challenge for many sites. Many sites have used a service called SWIS but expressed concern about its cost. Development of less expensive (or free), usable and accessible data systems could be helpful for BEST sites and other schools which are already trying to track and access data for decision-making.

Helping us think about our data collection process was very helpful. We are well on our way and have incorporated it into our student information system.
–General Education Teacher

Yearly Rate of Suspensions and ODRs at BEST Sites
Rank School Type Degree of Implementation Suspensions
per 100 Students
ODRs
per 100 Students
Base*  2007  Change  Base*  2007  Change 
          1 Elementary Partial        10.8          0.0 -10.8        30.0        10.0 -20.0
          2 Elementary Full          7.6          0.2 -7.4        10.4          0.2 -10.2
          3 Elementary Partial          1.6          0.9 -0.8     110.0        11.9 -98.1
          4 Elementary Partial          0.5          1.0 +0.6          2.6
          5 Elementary Partial          5.6          1.1 -4.6        34.4        27.6 -6.7
          6 Elementary Full          8.3          1.9 -6.4        25.0     110.1 +85.1
          7 Continuation Full          4.8          1.9 -2.8        18.2        11.2 -6.9
          8 Elementary Partial          4.6          2.1 -2.4          6.5        84.4 +77.9
          9 Elementary Full          0.2          2.6 +2.4          4.9
        10 Elementary Partial        14.0          3.0 -11.0        67.4        43.6 -23.8
        11 Middle School Full        19.3          4.0 -15.4          5.2 +5.2
        12 High School Partial          3.9          4.6 +0.6        10.9          4.6 -6.4
        13 Elementary Partial          2.7          5.0 +2.3          8.6
        14 High School Partial          4.5          5.4 +0.9        22.0        25.3 +3.3
        15 Elementary Full          3.1          5.5 +2.4          0.5          9.2 +8.7
        16 Middle School Partial          0.5          6.3 +5.7          2.3     140.0 +137.7
        17 Elementary Partial          1.2          6.9 +5.7          0.5
        18 K-8 Partial          4.9          7.6 +2.8        33.3        33.9 +0.6
        19 Elementary Full        11.8          8.0 -3.7        57.4        31.7 -25.7
        20 Middle School Full          8.1          8.7 +0.7        15.8          7.1 -8.7
        21 Middle School Partial        28.8          8.8 -20.0     111.3        94.2 -17.1
        22 Elementary Partial          6.4          9.8 +3.4          8.1        44.9 +36.8
        23 Elementary Partial        10.8        13.4 +2.5        51.0          9.7 -41.3
        24 Middle School Full        18.9        13.8 -5.1          1.0        52.0 +51.0
        25 Middle School Full        10.4        13.8 +3.4        10.6          4.4 -6.2
California's Statewide Rate        11.7        13.9 +2.2 (data unavailable)
        26 Elementary Full          9.7        15.5 +5.8        14.7     113.4 +98.7
        27 Middle School Partial        21.9        18.7 -3.3     127.9
        28 Elementary Partial        10.9        19.0 +8.1     541.8
        29 Middle School Full        19.5        24.2 +4.6     108.1        24.2 -83.9
        30 High School Full        21.0        26.3 +5.4     127.0     176.3 +49.3
        31 Middle School Full        26.1        30.9 +4.8          8.9        90.7 +81.8
        32 Middle School Partial        37.5        34.5 -3.0
        33 Middle School Partial        21.3        34.5 +13.3        91.5        11.0 -80.5
        34 Community Full        46.2        36.4 -9.8        13.6 +13.6
        35 K-8 Partial        34.4        41.4 +6.9        37.4        97.8 +60.4
        36 Middle School Partial          2.0        55.2 +53.2          2.1     124.4 +122.3
        37 Community Partial     161.0     170.7 +9.7
        38 Community Partial        63.9     180.6 +116.6
*Base year is a single school year between 2003-04 and 2005-06, depending on when the site was trained. The base year selected for California was 2004-05, matching the midpoint for BEST site trainings.

Stakeholder Input and the SPP

Improving Special Education Services (ISES) is a collaborative stakeholder group convened by the California Department of Education (CDE), Special Education Division. The California State Performance Plan (CA SPP), Annual Performance Report (APR), and SIG2/SPDG are guided by this meeting in addressing issues such as personnel development, improving statewide assessment outcomes, and progress monitoring through a unified planning process. ISES is the unification of what were previously the Partnership Committee on Special Education (PCSE) and the Key Performance Indicator (KPI) group. The committee consists of a broadly diverse and representative group of individuals involved in, or concerned with, the education of children with disabilities.

ISES has met twice annually since the group was created in January 2007. ISES will continue to meet, following this schedule, to guide SIG3/SPDG.

Each ISES meeting includes stakeholders supported by CDE workgroup facilitators and CalSTAT staff. Participants include both new and returning ISES members.

ISES Meeting Particpants by Role
4 meetings from January 2007 to June 2008, each with between 68 and 109 participants
Role Percent
Administrator: Special Education 19%
Faciliator 18%
Other Agency Staff 16%
Parent Leader 13%
Other CDE Staff 12%
IHE 8%
Program Specialist 7%
Teacher: Special Education 4%
Administrator: General Education 3%

The CA SPP is developed by the California Department of Education Special Education Division, with input from broad stakeholder groups including ISES. The CA SPP describes how the state will meet the requirements of the Individuals with Disabilities Education Act (IDEA), as amended in 2004.

To provide input to the SPP, ISES committee members divide themselves into workgroups, each of which addresses 1-4 indicators described in the CA SPP. Experts are recruited by CalSTAT to facilitate discussion and planning, and attending stakeholders participate in the workgroup of their choosing. Workgroups generate recommendations around the workgroup content area, CA SPP indicators, and related SIG/SPDG improvement activities which are then reported to all ISES participants at the end of the event and posted online. This process has informed the writing of the most recent version of the CA SPP, updated in February 2008.

Average ISES Participant Responses to End-of-Event Surveys
4 meetings from January 2007 to June 2008, 13 participant responses
Overall, how was the meeting? 4.1
Was this meeting of value in advancing the goals/vision of the SPP? 4.2
How would you rat ethe primary workgroup in which you participated? 4.3
Were you able to contribute in a way that brought your best thinking to the table? 4.3

In end of event evaluations, stakeholders who participated in ISES meetings rated their experience highly.

The survey also asked participants, “What were the most positive aspects of this meeting, and why?” Responses include the following:

The variety of participants and their perspectives.

The small groups work and report out of their groups—it’s wonderful to see so many minds actually engaged in problem solving.

All stakeholders made contributions! We walked out with a product!

The diversity of participants was so helpful in getting all of the perspectives and stakeholders involved in the process.

Revisiting earlier stakeholder recommendations and revising as needed to reflect current trends and accomplishments.

Shared understanding—the networking and collaboration, better understanding of the SPP/APR process.

Teacher Recruitment and Retention

The Institutions of Higher Education (IHE) Leadership Project, funded by CDE, is focused on helping to address the continuing teacher shortage in California by delivering information resources to potential teachers through publishing activities and the convenient interface of www.teachcalifornia.org.

This website was established during SIG1 to directly deliver information about teaching requirements, opportunities, and benefits to the public. During SIG2, the website has seen more than 1.4 million visits, from over 600,000 visitors, viewing over 4.5 million pages, and continues to be a resource attracting an average of over 1,000 visits per day.

TEACHCalifornia.org Activity Summary during SIG2
  2005 2006 2007 2008* Total
Website Visits 425,000 384,000 453,000 176,000 1,438,000
Unique Visitors 187,000 157,000 174,000 96,000 614,000
Visitors who made one visit 152,000 126,000 140,000 78,000 496,000
Visitors who made two visits 18,000 15,000 17,000 9,000 59,000
Visitors who made 3+ visits 17,000 15,000 17,000 9,000 58,000
*SIG concluded in June 2008. These totals reflect only the first 6 months of the year.

A pop-up survey collected over 33,000 responses from website visitors during SIG2. Responding visitors represented a broad diversity of age and ethnicity. Additionally, responses indicate the website is being accessed by its target audience.

Are you thinking of becoming a teacher someday?
N=33,764, 2005 through 2007
Yes 77%
No 2%
N/A 21%
Are you taking steps to become a teacher right now?
N=26,127 who were thinking of becoming a teacher, 2005 through 2007
Yes 77%
No 2%
N/A 21%

Many of the visitors who responded “N/A” to the survey may be university professors and counselors who are accessing resources to aid in teacher recruitment. In addition to delivering information directly to potential teachers, the TEACH California website offers online ordering for recruitment materials, including videos and brochures; many orders placed are for multiple copies of these resources.

TEACH California provides a number of materials to individuals, schools, and organizations helping to bring new teachers into California classrooms. These pamphlets, videos, and other materials can be ordered or downloaded for free through teachcalifornia.org. Items can be ordered or downloaded separately, or a “toolkit” of nine related items can be ordered as a single unit.

IHE Materials Shipped and Viewed Online During SIG2
  2005 2006 2007 2008* Grand Total
Shipped Online Shipped Online Shipped Online Shipped Online
Pathways to Teaching 7,181 388 6,539 568 2,349 2,458 ** 477 19,960
Toolkit Items Becoming a SE Teacher in California 13,508 130 9,395 897 4,431 4,549 3,236 394 36,540
One Child at a Time (CD ROM) 3,422 ** 742 ** 570 -- 504 -- 5,238
One Child at a Time (VHS) 306 ** 712 ** 385 ** 537 ** 1,940
Video Guide 3,393 201 699 543 547 575 492 261 6,711
Paraeducator 130 ** 4,772 1,001 1,711 1,212 906 479 15,265
Teach Math 130 4 4,345 550 11,175 1,004 3,154 308 20,670
Teach Science 130 2 4,843 786 11,946 2,654 3,156 1,026 24,543
Special Educators Pamplet Career Choice 5,306 109 136 122 *** 159 ***   5,832
Financial Aid 4,762 283 136 104 *** 134 ***   5,419
Overview 4,730 243 136 214 *** 121 ***   5,444
*SIG concluded in 2008. These totals reflect only the first 6 months of the year.
**This document was not available in this format during this time.
***These pamphlets are out of print. THeir content has been integrated into Becoming a SE Teacher in California.

Generation and shipping of these resources is administered in concert with CalSTAT’s other publishing activities, detailed below.

Publications

CalSTAT administers a number of publishing activities as a resource for stakeholders at all levels of general and special education. These activities involve both print media and electronic resources, and include the following:

The Special EDge Newsletter

The Special EDge is published three times a year and delivered to approximately 50,000 print and email subscribers, including over 8,000 parents. It is designed to inform and support parents, educators, and other service providers on special education topics, focusing on research-based practices, legislation, technical support, and current resources. Current and past issues are available online in English and Spanish.

Some of the most-recent issues published during SIG2 include:

Transition to Adult Living: An Information & Resource Guide

This comprehensive handbook is written for students, parents, and teachers. It offers practical guidance and resources in support of transition efforts for students with disabilities as they move from their junior high and high school years into the world of adulthood and/or independent living.

Frequently updated, approximately 60,000 copies of the Transition Guide were printed and shipped during SIG2.

Podcasts

Podcasts are the newest addition to the CalSTAT website. Users can hear recorded conversations with experts from the field on a variety of topics featured in The Special EDge and CalSTAT activities. Eleven podcasts are currently available, with titles including What is Disproportionate Representation? and the multi-part series, Communities of Practice.

The RiSE Library

CalSTAT also manages the RiSE (Resources in Special Education) Library, which makes available documents that offer research-proven strategies and information recommended by educational stakeholders. Residents of California can check out these resources free of charge by ordering through phone, email, U.S. mail, fax, or by visiting the library in person. The RiSE Library is located at Parents Helping Parents of Santa Clara, a federally funded Parent Training and Information Center.

During SIG2, over 2,000 items have been checked out through the RiSE Library.

Family Outreach and Partnerships

The parents, guardians, and family members of children with disabilities play a significant role in their children’s lives. Building collaborative, responsive environments and relationships between family members and schools enables a greater number of family members to become active participants — and when family members are engaged, it can enhance the system’s capacity to address the needs of a diverse student population.

The California Department of Education has prioritized parents as key stakeholders and partners in the professional development of education leaders and staff. As a project of the Special Education Division, CalSTAT has funded several parent outreach activities and identified families as a key team member in the learning communities they support.

SIG2 activities have included elements of family outreach and partnership as well. Family-school partnership is one of the eight core messages which have guided SIG2 activities, and SIG2 activities have included parents wherever possible. Some of these other SIG2 activities which have contributed to the goal of family outreach and partnerships include:

Family Outreach and Partnerships as a Primary Focus

During SIG2, CalSTAT addressed this core message by providing resources directly to parents and parent organizations, as well. These activities include the following:

Both the Family Participation Fund (FPF) and the Community Advisory Committee (CAC) Capacity and Recruitment Project engaged family members, targeting those from diverse backgrounds, and supporting their participation within the education system while also building their capacity as advocates for their children and other students with disabilities. These projects are discussed in the following pages.

Family Participation Fund (FPF)

The Family Participation Fund (FPF) was developed to meet the need for fiscal resources to help encourage and support meaningful family involvement in local, regional, or statewide educational decision-making activities, events, and groups. The intent of this activity is to build partnerships that reflect the diversity of our population and that are aligned with state reform mandates and initiatives. During SIG2, nearly 5,000 requests for financial support were funded.

The FPF is administered through the California Association of Family Empowerment Centers (CAFEC), which works as a center of information, technical assistance, and systems change advocacy for the statewide network of local FECs. FECs work directly with families of children with disabilities to provide family education and empowerment, regularly referring families to the FPF when they show interest in participating as a member of a decision-making body.

Ethnic minorities and low-income families are often underrepresented in the membership of decision-making bodies. Surveys completed by stipend recipients demonstrate that a majority of FPF resources have served these communities.

Family Participation Fund Recipients by Ethnicity
Feburary 2005 - July 2008 ~ 4,834 recipients*
Ethnicity Recipients Percent
African American 2,259 49%
Latino / Hispanic 1,198 26%
Caucasian / White 904 20%
Asian or Pacific Islander 194 4%
Other 52 1%
*227 recipients did not specify their ethnicity
Family Participation Fund Recipients by Ethnicity
Feburary 2005 - July 2008 ~ 4,834 recipients*
Income Recipients Percent
Under $10,000 2,562 57%
$10,000 to 19,999 736 16%
$20,000 to 29,999 430 9%
$30,000 to 39,999 221 5%
$40,000 to 49,999 208 5%
$50,000 or more 371 8%
*306 recipients did not specify their income
Average FPF Participant Evaluation Responses
February 2005 - June 2008 ~ 4,834 recipients
I learned valuable, useful information at the meeting 4.7
The meeting will make a positive difference in the lives of children with disabilities 4.5
I felt that I was an effective participant at the meeting 4.5
I felt that other people at the meeting valued my participation 4.4
The financial support of attending the meeting met my needs 4.4

Another goal of the FPF is to increase awareness and build collaborative partnerships between special education (SE) and general education (GE) by encouraging family members of children with disabilities to join decision-making bodies focused on GE issues.

Reimbursements by Focus of Meeting
February 2005 - June 2008 ~ 4,834 reimbursements
Special Education 59%
General Education 41%

Comments from FPF recipients include:

At this meeting we as parents were welcomed, and our input into policies affecting our children’s education was valued and utilized. Our concerns were responded to, and we saw results.

There is a lot of information to learn and share!

This meeting presented an opportunity for parents to speak directly to the School Board members and for them to hear and understand a parent’s perspective on the current state of affairs.

This stipend is a great help!

To determine their impact on the system, 38 active FPF recipients (family members who received three or more stipends between January 2005 and February 2006) participated in an in-depth telephone interview process.

Each family member indicated a main area of focus in their participation. These areas of focus fall into five general categories:

Within this small sample, it appears that, as family members continue to be engaged in the education system, they progressively move their focus from the school level to the district level to the state level. For family members with 4 or fewer years of participation, typical focus was on the school level, such as school board meetings. For those with 4 to 10 years, typical focus was on the district or regional level, such as CAC meetings. For those with 10 or more years, typical focus was on the state level, such as stakeholder meetings. Family members typically described their contributions as valuable at all levels of focus.

Participant Responses to Follow-Up Surveys
Responses from 1,238 participants at 472 events during SIG2 (February 2005 to June 2008)
  Influenced Discussions Influenced Decisions Taking Action
Responding with a 4 on a 5-point scale 26% 32% 71% report Taking Action
Responding with a 5 on a 5-point scale 58% 47%
Total Responding with a 4 or 5 on a 5-point scale 84% 79%

Interviewed family members reported that within all levels of the education system, they were supporting efforts to keep the system accountable, providing ideas and feedback based on their experiences, and making change happen.

Holding the System Accountable through Monitoring What’s Happening

Contributing Ideas and Giving Feedback in Planning Processes

Actually Got Changes Made

CAC Capacity Building and Recruitment

It is the mission of all CACs to act as collaborative partners with Special Education Local Plan Areas (SELPAs) to ensure that students with disabilities receive free and appropriate public education and have equal access to all services. CACs serve as a liaison between the SELPA, COE, local districts, families, and community members. Therefore, it is important that the membership of a CAC be representative of its community—economically, ethnically, and by disability.

As a pilot project, in 2005 five Community Advisory Committees (CACs) were partnered with Family Empowerment Centers (FECs) in their areas to build the capacity of the CACs as advocates for students with disabilities, to develop support networks for parents, and to increase the membership of underrepresented groups. Collaborative and individual surveys have allowed CAC members to describe their recent experience, forming the basis of this evaluation.

Participating CACs included:

The CACs involved in this project were selected because they had identified the need to recruit a greater number of members who were representative of the community they served. These five CACs serve diverse communities with over 230,000 students, over 20,000 of whom are in special education. Each CAC has a distinct student population:

Between 2005 and 2008, this partnership has allowed the members of the CACs to benefit from the resources and experience of FEC personnel. This is in alignment with the general mission of local FECs to provide family education, empowerment, and parent-professional collaborative activities for families of children with disabilities.

Much of this support comes in the form of general assistance with procedure, budgeting, and communication with school officials. Material support has included providing advertising, transportation, and translation services.

Membership Recruitment

Official CAC members must be appointed by the local SELPA director, but many more parents participate in CACs unofficially. Each CAC recently reported that their membership (both official and unofficial) is anchored by a handful of participants (between 5 and 16) who attend regularly and actively. A larger body of participants attended trainings, events, and occasional meetings. As a central goal, these CACs continue to encourage parents to become more-frequent participants.

Since this program started in 2005, efforts to increase membership have been successful, increasing the overall number of regularly-attending members by 52% (from 27 participants in 2005 to 41 in 2007 across the four participating sites).

CAC Membership by Income
27 regularly attending CAC members in 2005, 41 regularly attenidn CAC members in 2008
  2005 2008
Income Below $30,000 13 members 19 members
Income Above $30,000 14 members 22 members
Increase 46%
CAC Membership by Ethnicity
27 regularly attending CAC members in 2005, 41 regularly attenidn CAC members in 2008
  2005 2008
Ethnic Minority 12 members 18 members
Caucacsian 15 members 23 members
Increase 50%
CAC Membership by Family wiht Disability
27 regularly attending CAC members in 2005, 41 regularly attenidn CAC members in 2008
  2005 2008
Disability in Family 20 members 36 members
No Disability in Family 7 members 5 members
Increase 80%

Activities and Successes

Recruitment continues to be a primary activity of the CACs. Successful, ongoing efforts include:

All four CACs distributed literature to parents. These activities included sending fliers and brochures home with students, distributing newsletters and developing websites. The goal of these communications varied from posting CAC meeting times to providing information about disabilities and special education.

Another function of the CACs has been to help parents take a larger role in the decisions effecting their children. This was achieved through a variety of strategies including:

CACs have provided parents within their community the opportunity to attend trainings on topics including assistive technology, alternative dispute resolutions, leadership and advocacy. Many of these trainings were given by outside agencies hosted by or in partnership with the CAC. All four CACs hosted at least one training or workshop during SIG2.

Additionally, CACs have a structural role as a point of contact between parents and school officials in the special education community. Ideally, the community will use the CAC as a forum to ask questions and provide input while school officials use the CAC as a collaborative partner and venue to disseminate information to the community. While this relationship is strained in some CAC locations, these contacts are still reported as successful.

Beyond activism and advocacy, CACs successfully function as an anchor point for their community. CACs help create social networks and provide emotional support for the parents and families of students with disabilities. One FEC facilitator described this in a survey response:

Our goal is to empower parents and the community with the importance of knowing, understanding, and utilizing their rights and benefits through their consistent and dedicated involvement with the CAC as a CAC member.

Critical Elements

When asked to identify what facilitated these successes, the respondents indicated the following three elements are critical for success:

Each of these three elements were found to have contributed to the successes reported at the sites. However, these elements are only partially in place at the CACs.

Barriers to Success

All four FEC representatives were very clear that they see the CACs being blocked from achieving their stated goals of representing community viewpoints and delivering input to education policy.

According to FEC representatives, parents who want to be involved in decision-making are often discouraged by the perception of an adversarial relationship with school officials. These parents suggest that some school officials want the CACs to function as a rubber stamp and have accused them of using their power to achieve this outcome—by refusing to appoint new CAC members, by restricting access to parents through student take-home packets and fliers, or by cutting CAC participants out of the process.

Frustration with this environment is named by many FEC representatives as the leading reason parents cite when they stop attending meetings and participating in CAC activities. Many parents, especially those with small incomes, are described as refusing to invest any more of their own valuable time in the CACs when they doubt they will ever be able to actually influence education policy or outcomes.

FEC representatives made a priority of improving relationships between CAC members and school officials over the last two years. While describing successes both incremental and broad, many CACs continue to operate in an uncertain environment. Streamlining the appointment process for new CAC members and reinforcing the critical elements of CAC success are seen by many FEC representatives as essential to building a functional network of CACs.

One FEC facilitator described how these barriers have created opportunities for growth:

One person, in particular, was threatening to file a complaint and her director caved in and elected her; she has become a leader in her community. She has worked hard and positively. Once she was elected, she wrote back to her director saying ‘thank you so much, I know we can get more done together than separately.’ She said she would look forward, not backwards, and I thought that was excellent.

Data Tools

SIG evaluators have used the evaluation of SIG activities as an opportunity to help build a culture of data use across California’s educational community. SIG evaluators committed to timely development of evaluation materials that will be relevant and accessible to site-level decision-makers. During SIG2, data tools were developed which allowed sites to track and use data at the local level, with instantaneously generated visual representations of data.

The two tools with the widest application are detailed in the following pages, including:

Other Evaluation Tools

Additionally, evaluation instruments were created for the BEST and ERIA programs using electronic formats allowing instantaneous, longitudinal site-level charting. The centerpiece of these efforts has been the TIC Charting Program, which includes separate versions for BEST, a schoolwide positive behavior supports program, and ERIA, a literacy intervention program.

The TIC, or Team Implementation Checklist, is a an instrument which site teams review and complete twice per school year. As the implementation of program elements advances at the site, additional checklist items can be recorded as completed. In both the BEST and ERIA program evaluations, the TIC is one of several instruments which together create a full picture of the degree of implementation.

The TIC Charting Program operates in Microsoft Excel. It includes data entry sections within the workbook and a number of automatically updating charts which create visual representation of evaluation data. As site teams complete the TIC using the TIC Charting program, they are instantly rewarded with a graphic demonstrating their progress. The TIC Charting program is configured to collect up to four years of data, allowing sites to monitor their progress through time.

Other electronic instruments have been integrated in the ERIA assessment. These include the use of word processing files to build a comprehensive, longitudinal Site Action plan (SAP) and a modified version of the CST Charting program which allows sites to access CST ELA scores more directly in their planning.

CST Charting Program

The CST Charting Program was designed to enable schools and districts to create a single page graphical chart showing the percentage of all students compared with students with disabilities as a subgroup performing at each of the five levels of proficiency on the California Standards Test (CST) for English-language arts (ELA). The chart shows the achievement gap, and that most students have a long way to go in achieving grade-level proficiency. Using the same tables of numbers that parents, teachers, and administrators have had access to for years, the CST Charting Program presents this information with colors and proportions and makes it simple for most people to grasp where students are at a glance.

Over 750 copies of a CD-ROM (or an emailed version) containing the CST Charting Program have been distributed to personnel and parents. Also included are reporting templates as well as a series of brief instructional movies for how to use the program.

The CST Charting Program is brilliant and invaluable. Simple to use, the program produces an elegant graphic that shows clearly whether we are closing the gap between groups/subgroups. Thanks to ERIA, we are doing just that!
–School District Administrator

The CST Charting program is available to download online at http://www.calstat.org/CSTProficiencyCharting.html

TED: Training Evaluation Database

The Training Evaluation Database (TED) was originally developed during the first State Improvement Grant to track the professional development training events funded by CalSTAT and to evaluate the success of these events. As part of the second State Improvement Grant this project was expanded and TED was further developed to be shared with SELPAs, COEs, and LEAs that were also providing professional development within their local area. The objective was to assist selected sites in effectively tracking the trainings provided while also offering easy-to-use evaluation functions that determine the success of events and inform future training decisions.

The TED Program

TED was developed to hold both basic and advanced event tracking information so that each site can customize their level of use. The following is a brief list of some of these functions:

Sites that are interested in using TED receive both initial and ongoing user support to ensure a smooth transition.

During SIG2 several sites also elected to participate in an advanced training with the SIG Evaluator that looked at the training data they had collected in conjunction with student test scores. The intent of this advanced training was to determine how the site might tailor future professional development training to support the areas in which they excelled and to intensify training in areas of deficit.

TED can provide essential training provision information and assist sites in answering a number of questions that might help them more effectively manage their professional development and training program. For example:

Users of TED who piloted the software during SIG2 described some of the benefits of using TED. Comments include the following:

“One of the most important features of TED (for us) is the ability to run reports for our Human Resources Department to show what trainings a particular employee has had. This is also useful in due process hearings to show what trainings a particular employee has had to demonstrate their qualifications for a particular position. Also, when participants call in asking for proof of their attendance (for CEU purposes), we are able to locate the events they attended easily.”

“I have worked with TED for two years. The program has been a great tool. As with any new program, there have been growing pains but [the SIG2 evaluation team] has always stepped right in and given immediate assistance. I am amazed at how many of our requests and needs have been implemented into the program since I first started using this program.”

“We love TED. We use it for all of our staff development workshops. It really makes tracking things over multiple years easy.”

“We provide copies of event summary reports to our special education directors to review. We also provide them with copies of the evaluation summaries and have found this a really helpful way to provide them with feedback on events.”

A demo version of TED which does not require Filemaker Pro to operate was developed and used as a marketing tool. Sites received this tool as a CD-ROM or by downloading it from the CalSTAT website, using it to determine if they wanted to purchase Filemaker Pro and pursue installation of the full version.

This demo version is available to download at http://www.calstat.org/ted.html.

TED Implementation During SIG2

During SIG2 the installation and ongoing support of TED was made available to SELPAs and LEAs for free. These sites became part of a pilot project, during which they used TED at their site and provided regular feedback on the usefulness of the available functions and made suggestions on additional functions that might be added to meet their tracking and evaluating needs. The objective was to include 20 diverse sites in the pilot project. During the course of SIG2, 19 sites elected to have TED installed and their staff trained in its use. Of these 19 sites, 15 have been consistent users and have provided feedback and suggestions that have contributed to the further development and refinement of the TED program. The 15 actively participating sites include 6 SELPAs, 6 districts, 1 COE, and 2 partnering organizations from northern, central, and southern California.

When asked to describe their experience, 12 sites responded and rated the TED program highly on a 5-point scale. Respondents were asked whether TED was user friendly (4.7 average rating), saved their staff time in tracking training events (4.5 average rating), and whether the reports provided useful information (4.5 average rating). Overall, respondents indicated that TED was effective at tracking all components of the professional development their site provided and made preparing for events easier. Of the functions provided in TED, each site indicating using a different assortment and thereby customizing their use to meet their need.

The TED program has also assisted CalSTAT in tracking SIG2-funded events. Currently the TED program holds information on 570 training events, 8,257 unique participants, and 12,136 training evaluations. A number of the evaluation data-points presented throughout this report were generated by the automated reporting function in TED. The capacity of this database to hold this level of information has allowed CalSTAT to track the large number of training events it has supported over the course of SIG2 and to easily summarize the information by multiple criteria.

Looking to the Future

With the close of SIG2, TED is a user-friendly training and evaluation tracking database that will continue to be made available to sites in the state of California to support their efforts in making data-informed decisions regarding the professional development they offer in their local area.

In the future, the TED program will continue to be a component of the State’s effort to support data-informed decision making. As part of the State Personnel Development Grant (SPDG) beginning in 2008, sites interested in using TED will continue to receive installation and ongoing user support. In addition, the SPDG Evaluation Team will be focused on developing a TED package that will sustain this effort beyond the SPDG. This package will likely include automated installers, installation instructions for possible set-up styles, a user manual, and a movie demonstration.

 


IDEAS that Work!

Project READ is a California Department of Education, Special Education Division project funded through a federal competitively-awarded State Personnel Development Grant to California (#H323A120019) provided from the U.S. Department of Education Part D of the Individuals with Disabilities Education act (IDEA), Office of Special Education Programs (OSEP). Opinions expressed herein are those of the authors and do not necessarily represent the position of the U.S. Department of Education.