Skip to Content

 

SIG / CalSTAT
Activities Evaluation Summary Report
July 1999 - January 2005

June 2005

Improving the Special Education System in California A State Improvement Grant (SIG) Program

Prepared By:
Cheryl "Li" Walter, Ph.D. California Institute on Human Services at Sonoma State University
and Kelly Bucy, MPA CalSTAT

This project is funded through a federal grant from the U.S. Department of Education, California's State Program Improvement Grant (CFDA 84.323A), as described in the Individuals with Disabilities Education Act (IDEA).

This report is available for review and printing online (as a PDF) at: www.calstat.org in the Evaluation and Results section.
SIG/CalSTAT Activities Evaluation Summary Report July 1999 - January 2005

For questions concerning this report, please contact:
Kelly Bucy -- kelly.bucy@calstat.org or (707) 849-2268 or
Cheryl "Li" Walter -- li@sonic.net

Introduction

In 1999, a five-year State Improvement Grant (SIG) was received from the federal Office of Special Education Programs (OSEP) to improve special education outcomes and services in the State of California by implementing activities directed at personnel development, family education, and systems change. Those served by the grant activities include general and special education teachers and administrators, early intervention service providers, consumers/families, related agency personnel and service providers, university/college professors, and university students currently enrolled in teaching preparation programs.

The partners involved in California's State Improvement Grant (SIG) effort undertook a broad range of activities between July 1999 and January 2005, the majority of which are presented in this report. Evaluations were conducted on the following activities:

The evaluation of these activities served several purposes:

Framework for this Report

First comes a summary of the general systems changes that have taken place in California, followed by maps showing the distribution of services provided. Then, for each program activity, there is a narrative and accompanying graphs summarizing the evaluation findings.

Systems Change in California

In California's State Improvement Grant (SIG) effort, systems change was the focus from day one. The motto "Do and Develop" was adopted to instill the spirit of willingness to take action, to learn from experience, and then to take new actions based on that experience. These efforts were guided by research-based core messages, while interventions in areas such as literacy and behavior served as mechanisms for change. Recognizing that things need time to grow, seeds were spread and then close attention was paid to where things moved and changed, and then that growth was supported. As was stated by a key SIG leader, "Systems change isn't systematic. It's chaotic and self-organizing."

This is perhaps a reminder of why asking people to change how things are done is not always a comfortable process. There have been many successes and challenges along the way, and much learning was gained from both. In this final summary report of California's SIG1 activities, the systems changes that have taken place overall -- and in some of the major activity areas -- have been examined.

The data for this systems change summary comes from observations of the entire process by evaluation staff, interviews with key informants, and discussion with the broad representation of the SIG Evaluation Task Force. Data-informed does not just mean numbers but also the impressions and perceptions of the participants in the unfolding of the story. This is not meant as a comprehensive study but simply a view-from-here-so-far to acknowledge some of what has happened and the influences along the way. And it gives a picture of where things appear to be heading.

In the chart at the right, you will see the major themes that emerged from the interviews and discussions. The findings seemed to fall into three themes, although as the diagram illustrates, these aspects are inextricably linked. The three aspects that appear to have coalesced into systems change in California are:

In the activity evaluation summary pages that follow, many of the major SIG activities contain brief narratives discussing systems changes that have emerged specifically in relation to that activity.

California's State Improvement Grant (SIG) Aspects of Systems Change
Relationships Sustainability Structures
  • SE/GE Collaboration
  • Site-to-Site TA/Coaching
  • Involvement of Parents
  • Learning Communities
  • Meetings of Stakeholders
  • Resources to Site-Level Teams Committed to Change
  • Multi-Layered Support for Ongoing Change
  • Leveraging Resources/Seed Money
  • Training of Trainers/Coaches
  • Scaling-Up
  • Direct Access for Schools and Teachers to TA Resources
  • Leadership Institutes
  • Partnership Committee on Special Education
  • Data Sharing
  • Research-Based Core Messages
  • Regional Coordinating Councils
SYSTEMS CHANGE

SIG/CalSTAT-Funded, RCC-Sponsored, BEST, Technical Assistance, and Regional Institute Trainings
July 1999 - January 2005

Region Number of Trainings Number of Participants
1. North Coast 102 4,675
2. Northeastern 114 4,344
3. Capital 163 7,416
4. Bay 267 9,814
5. South Bay 101 4,748
6. Delta Sierra 89 3,806
7. Central Valley 78 6,161
8. Costa Del Sur 118 5,704
9. Southern 261 10,268
10. Rims 255 6,412
11. Los Angeles 154 14,401
Throughout California's 11 Regions 1,702 77,749

Regional Coordinating Council (RCC)- Sponsored Trainings

The Regional Coordinating Council (RCC)-sponsored training activity was developed to meet the need for an augmented, coherent, and comprehensive system of training for educators, service providers, and families/consumers that focused on the state plan goals and objectives. The intention of using the RCCs as a delivery system for these trainings was to build a system that was both responsive and that could effectively serve local and regional needs.

From 2000 to 2005, the RCCs sponsored a total of 923 trainings, with over 52,400 participants attending. These trainings were primarily evaluated through paper-and-pencil forms conducted at the end of each event. In addition, training follow-up evaluations were conducted online, and the number of trainings provided and the number of participants attending was tracked in each core message area.

SIG-Funded RCC Trainings by Main Topic Presented
July 2000 - Jan 2005 -- 923 Trainings
Main Topic Presented Number of Trainings Percent
SE & GE Collaboration 110 12%
IDEA '97 98 11%
Transitions 45 5%
Parent Partnerships 47 5%
Least Restrictive Environment 33 4%
Positive Behavioral Supports 352 37%
Literacy 238 26%

The majority of the training events were concentrated in the core message topic areas of positive behavioral supports and literacy.

SIG-Funded RCC Participants by Main Topic Presented
July 2000 - Jan 2005 -- 52,442 Participants in 923 Trainings
Main Topic Presented Number of Participants Percent
SE & GE Collaboration 5,908 11%
IDEA '97 5,032 10%
Transitions 1,718 3%
Parent Partnerships 2,655 5%
Least Restrictive Environment 1,400 3%
Positive Behavioral Supports 22,670 43%
Literacy 13,059 25%

Participants were concentrated in the same core message topic areas -- positive behavioral supports and literacy.

SIG-Funded RCC Participants by Main Topic Presented
July 2000 - Jan 2005 -- 52,442 Participants in 923 Trainings
Participant Role Number of Participants Percent
Teacher: SE 11,194 36%
Teacher: GE 3,285 10%
Admin: SE 1,085 3%
Admin: GE 825 3%
Program Specialist 2,078 7%
Parent 1,369 4%
Paraprofessional 1,338 4%
Speech/ Language Pathologist 1,827 6%
Psychologist/ Counselor 3,368 11%
Other Certified Professional 426 1%
Other 4,864 15%
*20,783 participants did not specify their role

Special education teachers were the predominant group attending the trainings sponsored by the RCCs.

Evaluation Findings

SIG-Funded RCC Training Evaluations
July 2000 - Jan 2005 -- 923 Trainings
Survey Item Average Response
(5-point scale)
Before Training Knowledge 3.2
After Training Knowledge 4.2
(31% increase)
Overall Rating of Training 4.5
Will Share Learning 4.4
Will Use Learning 4.5
Learning Will Assist Me 4.4

There was a 31-percent increase in the average level of knowledge reported by participants after the training (from 3.2 to 4.2 on a 5-point scale). There was little variation in responses from the different role categories, except for special education administrators and program specialists who reported a slightly higher before and after training knowledge level.

Participants rated their experience high in every category. The overall training rating of 4.5 (on a 5-point scale) has remained consistent over the past four years.

SIG-Funded RCC Training Follow-up Evaluations
July 2000 - Jan 2005 -- 4,486 Responses from 923 Trainings
86% Implemented Learning AT LEAST ONCE
(responded 3, 4 or 5 on a 5-point scale) 		85% Shared Learning AT LEAST ONCE
(responded 3, 4 or 5 on a 5-point scale)
57% Implemented Learning REPEATEDLY
(responded 4 or 5 on a 5-point scale) 		60% Shared Learning REPEATEDLY
(responded 4 or 5 on a 5-point scale)

Follow-up emails were sent to approximately 11,000 training participants several months after they attended a training. They submitted 4,486 responses (41 percent).

Of those responding, 86 percent indicated that they had implemented strategies learned at the trainings, with 57 percent having implemented the strategies repeatedly.

Participant Comments from Follow-up Evaluations

Behavior

The behavior plan format is clear and very organized. I find myself being much more specific in describing the behavior and interventions. I feel the plans I have been completing would definitely have a higher rating then last year.

I work as an adjunct faculty member at a university that prepares future teachers. I shared some of the materials in a behavior management class. The majority feedback of the students indicated Diana's materials were the most useful and valuable tools they had received in any college-level class. They felt they were given practical ideas and solutions.

Literacy

I have totally changed the format of my reading group. I now use close reading, silent reading (with questions so they are reading for meaning), and choral reading INSTEAD of round-robin reading, which is what I did last year. Yesterday my principal observed my language arts program and commented that it was very effective. I told him that I had used ideas from your conference!!

We are looking at implementing an early intervention program based around models we learned about at the in-service. It is part of an overall plan to increase test scores and reduce special education enrollment.

Research base and excellent references assisted in talking with parents and other professionals about taking new directions in intervention. The handout has been an invaluable resource.

Using various testing instruments has been very useful for meeting individual needs. I was very excited to see the number of instruments on display at the workshop.

IDEA

The training was very valuable because it gave me hands-on practical time to use the knowledge throughout the training. The training helped me develop and write clear, concise goals according to the standards for different subject areas. This is my 6th year teaching and I only wish that someone had done this a long time ago.

Special Education/General Education Collaboration

The training was very informative and helpful in the development of a preschool program for students in West County. We are going to structure our program on best practices and current research. Having the opportunity to participate in the training provided the basis for developing our preschool program. I know the nine superintendents that also participated were very pleased with the information. Jeanie Harmon and Missy Danneberg did an outstanding job of gathering and presenting information. In the fall, we will move to the next phase of actual program development with our recently formed task force. Thank you.

I can't even begin to tell you how valuable this workshop was. I'm still telling other SLP's about it. Within a few weeks of the workshop I approached my administrators about training one of our bi-lingual assistants to help me assess bi-lingual students. It has worked out wonderfully! One of the most valuable tools is the parent interview done BEFORE any assessment. This happens after the SST team meets. I interview the parents and then return to the SST to discuss assessment if necessary.

The school psychologist and I paired together after the workshop and discussed how we could present the information to the staff in a fun and enlightening way. We presented the 9 Adaptations Grid during a staff meeting modeling as many accommodation/adaptations as we could. As a school we now use it to document intervention used for student study teams.

Systems Change Findings

At the beginning of the SIG, the RCCs were realigned from 12 special education regions to general education's 11 superintendent's regions, which changed the structure, service area, and relationships of many regions. The long-term intent of this was to enable closer working relationships with general education's professional development committees in the regions. Also, funding was shifted from a regional grant to a contract system requiring that the RCCs offer their trainings in the research-based SIG core message areas, link trainings to follow-up technical assistance, and participate in evaluation activities.

In this case, systems change was imposed on an existing system, and, while it seems to have worked in some regions to the extent that new relationships and ways of working together and utilizing resources have developed, many of the regions have struggled against a structure that did not seem to suit their needs. As it became clear that the federal SIG guidelines were requiring the linking of activities and very specific outcomes, this generalized regional approach to training individual teachers or staff from schools or districts throughout a region became unsustainble as a programmatic option for SIG2. Some regions have been able to shift their approaches and become Regional Institute Host Sites, which work from a different service delivery model.

Centrally Coordinated Technical Assistance (TA)

The Centrally Coordinated Technical Assistance (TA) activity was developed to meet the need for a centralized system of easy-to-access technical assistance for educators, service providers, and families/consumers that focused on the state plan goals and objectives and that prioritized strengthening partnerships between consumers, parents, and professionals at all levels. The intent of this activity was to provide TA in the form of trainings, facilitation, and coaching to schools, parent organizations, education agencies, and other organizations throughout the state.

From 2000 to 2005, a total of 659 events, with over 17,800 participants attending, were supported through TA. These events were primarily evaluated through paper-and-pencil forms conducted at the end of each event. In addition, follow-up evaluations to the event were conducted online, and the number of events provided and the number of participants attending were tracked in each core message area.

TA Events by Main Topic Presented
July 2000 - Jan 2005 -- 659 Events
Main Topic Presented Number of Events Percent
SE & GE Collaboration 325 49%
IDEA '97 105 16%
Transitions 31 5%
Parent Partnerships 60 9%
Least Restrictive Environment 7 1%
Positive Behavioral Supports 74 11%
Literacy 57 9%

The majority of the events were concentrated in the core message topic areas of special and general education collaboration and IDEA '97.

TA Participants by Main Topic Presented
July 2000 - Jan 2005 -- 17,806 Participants in 659 Events
Main Topic Presented Number of Participants Percent
SE & GE Collaboration 5,973 33%
IDEA '97 4,226 24%
Transitions 655 4%
Parent Partnerships 1,643 9%
Least Restrictive Environment 304 2%
Positive Behavioral Supports 3,040 17%
Literacy 1,965 11%

The participants were concentrated in events that addressed core message topic areas of special and general education collaboration, IDEA '97, and positive behavioral supports.

TA Participants by Role
July 2000 - Jan 2005 -- 17,806* Participants in 659 Events
Participant Role Number of Participants Percent
Teacher: SE 3,515 31%
Teacher: GE 2,252 21%
Admin: SE 460 4%
Admin: GE 722 7%
Program Specialist 752 7%
Parent 995 9%
Paraprofessional 346 3%
Speech/ Language Pathologist 395 4%
Psychologist/ Counselor 636 6%
Other 883 8%
*6,850 participants did not specify their role

Special education teachers were the predominant group attending the events provided through TA. General education teachers also had a strong presence, with the remaining roles fairly evenly distributed.

Evaluation Findings

TA Event Evaluations
July 2000 - Jan 2005 -- 659 Events
Survey Item Average Response
(5-point scale)
Before Training Knowledge 3.2
After Training Knowledge 4.2
(31% increase)
Overall Rating of Training 4.5
Will Share Learning 4.4
Will Use Learning 4.4
Learning Will Assist Me 4.4

There was a 31-percent increase in the average level of knowledge reported by participants after the event (from 3.2 to 4.2 on a 5-point scale).

Participants rated their experience high in every category. The overall rating of 4.5 (on a 5-point scale) has remained fairly consistent for this activity over the past four years.

SIG-Funded RCC Training Follow-up Evaluations
July 2000 - Jan 2005 -- 4,486 Responses from 923 Trainings
82% Implemented Learning AT LEAST ONCE
(responded 3, 4 or 5 on a 5-point scale) 		82% Shared Learning AT LEAST ONCE
(responded 3, 4 or 5 on a 5-point scale)
54% Implemented Learning REPEATEDLY
(responded 4 or 5 on a 5-point scale) 		60% Shared Learning REPEATEDLY
(responded 4 or 5 on a 5-point scale)

Follow-up emails were sent to approximately 5,500 participants several months after they attended an event. From that group, 2,285 responses (42 percent) were received.

Of those responding, 82 percent indicated that they had implemented strategies learned at the event, with 54 percent having implemented the strategies repeatedly.

Participant Comments from Follow-up Evaluations

IDEA

The information was very helpful in reviewing my son's IEP Goals and preparing for his annual IEP to write new goals for the next year. We can now focus on the essential standards at each grade level when we write goals for his IEP. Most importantly we can better understand how the teachers and RSP providers write their goals.

The methods that were learned from you on facilitated IEP's have been very helpful with some of our more difficult IEP's. They ran smoothly with less conflict and both parents and school district agreed upon the outcomes.

Literacy

The specific strategies regarding reading were excellent and I've implemented many. The concept of tracking struggling readers and collaborating with other teachers in an ongoing basis has been more challenging. We have not implemented the Wall chart, which I had hoped we'd do. I haven't given up and where I've started is teaming with one teacher (4th grade) regularly to share with her reading strategies, assessment and support for a particularly difficult group of students. I also meet regularly with all the primary grade teachers, which is a very collaborative group, and we share insights, observations and strategies to strengthen their reading. I think this group would be very receptive to the wall chart.

Special Education/General Education Collaboration

The connection and interaction of your academic innovation and the BEST practices was particularly valuable. The first hand observation of implementation of these strategies was especially helpful. Thank you for opening your classrooms to our team.

The Hesperia ExCEL model is a common sense, research verified program that pools the efforts of all staff, including all support staff in regular and special education to specifically meet the needs of individual students through collaboration, communication, teacher empowerment. We are looking at all of our students as OUR students, not mine. The staff is thrilled with the learning outcomes so far by instituting the ExCEL model and is true believers of the power collaboration and direct instruction of explicit skills and guided reading. We are specifically addressing individual student needs and remediating fluency and comprehension issues on a daily basis. The ExCEL training was an essential piece in aligning the goals of regular and special education and maximizing our resources and potential. We're thrilled. Our entire school will implement ExCEL in the 2004/2005 school year.

We took what we observed, shaped and molded it to fit our students, and ran with it. The results have been phenomenal! I taught the "low" group and they grew in leaps and bounds. It was like we did a hundred years ago when we did across grade level ability grouping.

Our district was impressed with the program and three schools have committed to full implementation in the Fall 04. We hope to have six additional schools go through the training next year. We are grateful to Hesperia for the coaching and sharing of expertise.

Systems Change Findings

TA was initially intended to serve as follow-up to larger trainings, but over time it became a method of delivering core messages to the field. Prior to this approach, the system had operated predominantly through RCCs, SELPAs, and content experts serving on a broad scale. System change came through the focus on giving the local level more direct access to TA. Now, school sites can define their own needs and access TA resources in the form of mini-grants to pursue change and learning, guided by researchbased core messages. Even a teacher at a school can access TA and be the change agent without having to go through their administrator. Change has become a process often initiated with a conversation along the lines of "Who can I talk to about…"

These empowered change agents have a chance to "prove" something works and then scale-up to schoolwide, district, or larger levels. In the process, more practicing teachers and administrators are becoming "experts" through their experiences and are then able to help other sites as "peer coaches."

The greatest impact of TA was seen in the "small glories stories" of very local systems. Where TA was tried and didn't work, a lack of functioning relationships between teachers and administrators and/or general education and special education was reported. In fact, special education/general education collaboration was the core message area in which the majority of TA was requested. Follow-up surveys of over 100 sites that received TA in this area evidenced a tremendous amount of self-reported change in the relationships and educational practices at their sites (turn the page for details).

Collaborative Sites Survey

Over the past five years, more than 100 schools, districts, county offices of education, and SELPAs have, through CalSTAT, received California State Improvement Grant-funded technical assistance (TA) to foster greater collaboration between special education and general education at their sites. The TA providers are primarily staff from "model sites" with experience operating collaboratively. A survey was conducted to find out what, if anything, had changed at the sites receiving the TA. Forty-two sites responded, with these results producing the following conclusions.

Average Degree of Collaboration Prior to and After Receiving TA
Collaboration Prior to Receiving TA
(10-point scale) 		Collaboration After Receiving TA
(10-point scale)
4.19 6.66

Before receiving TA, the average degree of collaboration at the sites was 4.19 on a scale from 1 (totally separate) to 10 (totally collaborative). This indicates that, on average, the sites were operating more separately than collaboratively.

When asked what their service delivery system looked like prior to receiving TA, the sites responded that they focused on using pull-out services and special day classes (SDC) for special education students; and, since there was little collaboration between special education and general education teachers, the special education students were seen as separate from general education and not a joint responsibility.

After receiving TA, the average degree of collaboration at the sites was 6.66 on a scale from 1 (totally separate) to 10 (totally collaborative). This indicates that, on average, the sites were operating more collaboratively than separately.

When asked what their service delivery system looked like after receiving TA, the sites responded that there was much greater inclusion of resource specialist program (RSP) and SDC students and teachers in the GE classroom; there were more pre-referral intervention options available; teachers were collaborating more and working as a team; and scaffolded/differentiated instruction was used for all students and was based on extensive assessment.

The greatest gains were made in the areas of teaming, intervention, and focus on the core curriculum. Assessment, resources, and accountability also showed increases. While collaboration with parents was fairly high to begin with, little gain was made during the period of the grant, suggesting this as a potential area of focus for future TA.

Average Degree of Collaboration Prior to and After Receiving TA
by Area of Collaboration
Area of Collaboration Prior to TA After TA Difference
Assessment 3.79 6.48 2.69
Intervention 3.83 6.98 3.15
Core Curriculum 3.95 6.77 2.82
Teaming 3.40 6.70 3.30
Accountability 4.17 6.74 2.57
The Role of Parents 5.05 5.98 0.93
Administration 5.02 7.12 2.10
Resources 3.93 6.60 2.67
Professional Development 4.55 6.88 2.33

Of all the collaborative sites receiving TA, 68 percent showed growth in their Academic Performance Index (API) scores between 1999 and 2003. Of those sites that did not show growth, all but one had received less than three days of TA and/or had just begun receiving TA in 2002- 2003 and thus had not had time to implement changes and see results. The chart below shows that there was API growth at 96 percent of the sites that accessed three-plus days of TA and had received some TA before 2002-2003, with an average growth of 81 points toward the goal of 800.

IVDB-Provided Building Effective Schools Together (BEST) Trainings

The state contracted with the Institute on Violence and Destructive Behavior (IVDB) after identifying a need for improved positive behavioral management training for educators, service providers, and families/consumers. The intent of this activity was to deliver a school-wide discipline program, titled BEST, based on positive behavioral supports and developed by IVDB. This program trains school teams to develop and implement positive school rules, rule teaching, and positive reinforcement systems.

From 2000 to 2003, a total of 57 trainings, with nearly 2,600 participants attending, were provided by IVDB. These trainings were primarily evaluated through paper-and-pencil forms conducted at the end of each event. In addition, training follow-up evaluations were conducted online.

IVDB-Provided BEST Training Participants by Role
2000-2003 -- 2,581* Participants in 57 Trainings
Participant Role Number of Participants Percent
Teacher: SE 254 14%
Teacher: GE 662 38%
Admin: SE 61 3%
Admin: GE 208 12%
Program Specialist 69 4%
Parent 50 3%
Paraprofessional 17 1%
Program Specialist 69 4%
Psychologist/ Counselor 87 5%
Other 348 20%
*825 participants did not specify their role

General education teachers were the predominant group attending these trainings. There was also a strong presence of special education teachers and general education administrators. The intent of BEST is to involve teams from each participating school that include representatives from each of several roles, which explains in part why there was a stronger presence of general education teachers and administrators compared to TA or RCC activities.

Evaluation Findings

IVDB-Provided BEST Training Evaluation
2000-2003 -- 57 Trainings
Survey Item Average Response
(5-point scale)
Before Training Knowledge 3.3
After Training Knowledge 4.3
(30% increase)
Overall Rating of Training 4.4
Will Share Learning 4.5
Will Use Learning 4.6
Learning Will Assist Me 4.5

There was a 30-percent increase in the average level of knowledge reported by participants after the training (from 3.3 to 4.3 on a 5-point scale).

Participants rated their experience high in every category, with an overall rating of 4.4 (on a 5-point scale).

IVDB-Provided BEST Training Follow-up Evaluations
2000-2003 -- 288 Responses from 57 Trainings
91% Implemented Learning AT LEAST ONCE
(responded 3, 4 or 5 on a 5-point scale) 		89% Shared Learning AT LEAST ONCE
(responded 3, 4 or 5 on a 5-point scale)
67% Implemented Learning REPEATEDLY
(responded 4 or 5 on a 5-point scale) 		74% Shared Learning REPEATEDLY
(responded 4 or 5 on a 5-point scale)

Follow-up emails were sent to approximately 700 training participants several months after they attended a training. From that group, 288 responses (41 percent) were received.

Of those responding, 91 percent indicated that they had implemented strategies learned at the trainings, with 67 percent having implemented the strategies repeatedly.

These percentages are much higher than the average implementation rates for the past four years of both RCC trainings and TA. The high implementation rates may be due in part to the design of the training. Attending sites were required to send a team to participate in a two day training and a follow-up training day several months later. This sequence provided ongoing support to the teams implementing BEST.

Participant Comments from Follow-up Evaluations

Behavior

It has been great to have the entire district on the "same page" as we tackle the increasing behavioral concerns. The majority of the district has implemented the same basic rules, which is nice to see the consistency across the district. Rules are being "taught" and "re-taught" so that we can hold the kids accountable. Many schools are actually having days in which the kids rotate from activity to activity and learn about the rules...this has also helped the staff be on the same page! Great in-service.

We have seen a great reduction in the amount of small discipline problems. The students like knowing that everyone in the school is following the same rules and that everyone is enforcing the rules.

We have almost complete buy in from the staff all the way to the bus drivers! We started to implement the program in common areas so if there was resistance, it wouldn't seem to single anyone out. It has passively carried over into the classrooms. So far so good!

The training has help our staff solidify our school-wide rules; it has taught us the importance of developing ways to support the base of the triangle (the 80 to 85% of our students); It has given us a common language for discussing students and discipline in general. I have had two teams go through the 3 day training and have a third team ready to go in October. It is great training.

Training of a California Cadre of Trainers to Provide BEST Trainings

After identifying a need for improved positive behavioral management training for educators, service providers, and families/consumers, the state contracted with IVDB to provide the positive behavioral management training, Building Effective Schools Together (BEST). In 2000-2001, these trainings received the highest overall average participant ratings in comparison with all other training activities. To increase the capacity of the state, the sustainability of this program, and the number of coaches in California that were qualified to provide BEST trainings, California was awarded $836,000 by the federal government to enhance its existing State Improvement Grant (SIG). The state's application was ranked highest in a competitive bidding process against twenty other states. The intent of this activity was to build local capacity by preparing a cadre of trainers in California to deliver BEST trainings--a school-wide discipline program developed by IVDB, based on positive behavioral supports. BEST trains school teams to develop and implement positive school rules, rule teaching, and positive reinforcement systems.

In 2003, 62 professionals and parents were chosen to become a part of the California cadre of trainers and received coaching in BEST. These trainers were brought together both in 2003 and 2004 for training in how to provide BEST training. The intent of this activity was to train trainers who then could go into schools and districts and provide training in BEST. Because of this, program specialists (who typically provide professional development in their job duties) and administrators (both special and general education) were the predominant groups attending the event. There were a fairly high percentage of parents and other agency staff, while there was a relatively low representation of teachers.

BEST Training of Trainers Participants by Role
2003-2004 -- 62 Cadre Coaches Trained in BEST
Role Number of Cadre Coaches Percent
Teacher: SE 1 2%
Teacher: GE 2 3%
Admin: SE 6 10%
Admin: GE 12 19%
Program Specialist 18 28%
Parent 6 10%
Paraprofessional 5 8%
SLP 1 2%
Other Agency Staff 6 10%
Other Certified Professional 3 5%
Other 2 3%

Evaluation Findings

BEST Training of Trainers Evaluations
2003-2004 - 2 Trainings
Survey Item Average Response
(5-point scale)
Before Training Knowledge 3.4
After Training Knowledge 4.4
(29% increase)
Overall Rating of Training 4.5
Will Share Learning 4.8
Will Use Learning 4.8
Learning Will Assist Me 4.7

There was a 29-percent increase in the average level of knowledge reported by participants after the training (from 3.4 to 4.4 on a 5-point scale). There was variation in the responses from the different categories of roles. For example, teachers and parents started at a lower level of knowledge before the training. On the other hand, program specialists responded with higher ratings for each question, which may be due, in part, to the coaching skills they had already acquired for their position.

Participants rated their experience high in every category. The overall rating of 4.5 (on a 5-point scale) in this particular activity is what we have typically seen from other training activities.

BEST Training of Trainers
2003-2004 -- Response to Question 10:
I am fully prepared to provide BEST I trainings to others
Response Number of Responses
5 20
4 27
3 12
2 5
1 3
BEST Training of Trainers
2003-2004 -- Response to Question 10:
I am fully prepared to provide BEST I trainings to others
Prepared
(3, 4 or 5 on a 5-point scale) 		88%
Very Prepared
(4 or 5 on a 5-point scale) 		70%
Average Response 3.8

When asked whether they were prepared to provide BEST trainings, a majority of the participants responded with a three or higher (on a 5-point scale), indicating that they are more prepared than not. But, because the average response was 3.8, this still leaves room for improvement.

Of participants responding, 88 percent felt prepared to provide BEST trainings, with 70 percent feeling very prepared. Follow-up coaching was provided to all cadre trainers to help each become fully prepared.

Cadre-Provided Building Effective Schools Together (BEST) Trainings

The intent of this activity was for a cadre of California trainers to deliver a school-wide discipline program titled BEST, based on positive behavioral supports and developed by IVDB. This program trains school teams to develop and implement positive school rules, rule teaching, and positive reinforcement systems. This activity is part of an effort to build local training capacity and sustainability.

From 2003 to 2005, the cadre of California trainers provided a total of 37 trainings, with over 1,600 participants from 190 sites attending. These trainings were primarily evaluated through paper-and-pencil forms conducted at the end of each event. In addition, training follow-up evaluations were conducted online.

Cadre-Provided BEST I & II Training Participants by Role
July 2003 - Jan 2005 -- 1,646* Participants from 190 Sites
Participant Role Number of Participants Percent
Teacher: SE 211 15%
Teacher: GE 545 38%
Admin: SE 39 3%
Admin: GE 222 16%
Program Specialist 27 2%
Parent 75 5%
Paraprofessional 84 6%
Psychologist/ Counselor 70 5%
Other Certified Professional 20 1%
Other 134 9%
*219 participants did not specify their role

Evaluation Findings

Cadre-Provided BEST I & II Training Evaluations
July 2003 - Jan 2005 -- 37 Trainings Including 190 Sites
Survey Item Average Response
(5-point scale)
Before Training Knowledge 3.4
After Training Knowledge 4.3
(27% increase)
Overall Rating of Training 4.3
Will Share Learning 4.4
Will Use Learning 4.5
Learning Will Assist Me 4.4

There was a 27 percent increase in the average level of knowledge reported by participants after the event (from 3.4 to 4.3 on a 5-point scale).

Participants rated their experience high in every category. The overall rating of 4.3 (on a 5-point scale) was nearly as high as the ratings regularly received by the original trainers that have been delivering this training for many years.

Cadre-Provided BEST I & II Training Follow-up Evaluations
July 2003 - Jan 2005 -- 298 Responses
91% Implemented Learning AT LEAST ONCE
(responded 3, 4 or 5 on a 5-point scale) 		92% Shared Learning AT LEAST ONCE
(responded 3, 4 or 5 on a 5-point scale)
64% Implemented Learning REPEATEDLY
(responded 4 or 5 on a 5-point scale) 		76% Shared Learning REPEATEDLY
(responded 4 or 5 on a 5-point scale)

Follow-up emails were sent to approximately 700 participants several months after they attended an event. From that group, 299 responses (43 percent) were received.

Of those responding, 91 percent indicated that they had implemented strategies learned at the event, with 64 percent having implemented the strategies repeatedly.

This indicates a very high degree of implementation among respondants.

Although it is still early in the process, many of the school sites trained by Cadre members have begun to implement various aspects of the BEST behavior program, as evidenced by the results of the Effective Behavior Support Survey conducted with 73 sites after the first year of implementation. The chart to the right, shows that expected behaviors have been defined and are being taught in the classroom. Numerous other aspects have begun to be implemented in the classroom, in non-classroom settings, and schoolwide. The Effective Behavior Support Survey will be conducted annually to serve as a measure of the degree of implementation of BEST in the school sites trained by Cadre members.

Participant Comments from Follow-up Evaluations

Behavior

The training helped us focus as a school site on school-wide issues. We were able to use models from other schools and apply them to Pine Crest. We modified, adjusted, invented, etc. to fit our unique situation. We improved playground and hallway behavior and generally impacted our campus for the better! The piece about modeling, discussing, and repeatedly teaching the rules were especially impacting. We also will be using assemblies at the beginning of the school year and after Winter Break to "indoctrinate" our students and reinforce the importance of our efforts to work together to create and maintain a safe and respectful environment where people make responsible choices and the focus is on learning (Safe, Respectful, Responsible). We emphasize a positive rewards system with "Pine Crest BEST" Tickets and weekly drawing, rewards (mechanical pencils & pizza tickets, etc.), and recognition on announcements and at monthly assemblies. We will be developing school wide and classroom rules as we begin this, the second year, of our work in Building an Effective School Together! More training on an ongoing and year-to-year basis would be very helpful. Our site BEST Team (teachers, counselor, principal, support staff, and parent) will have some new and returning members this year and in-services are very helpful in maintaining our focus and work together. Thanks for asking!

The wealth of ideas and strategies has been great. It provides a focus for our school community and addresses the issue of school-wide behavior. It supports teamwork and positive behavior recognition. We are thinking, always, about doing and being our BEST! It takes lots of time and energy and patience to implement. We can see why we have been advised to give this process 2 to 3 years! Maintaining staff focus, understanding, participation, and support is a constant challenge and we have been mostly pleasantly surprised by the response. It allows me, as principal the opportunity to get into classes regularly to give positive recognition for acceptable and model behavior. The importance of teaching and re-teaching and modeling rules has been very powerful in getting things more calm and focused on the playground and other common areas.

I had 10 wonderful little girls in my class, and 10 great boys -- all squirrels!! But the BEST Practices worked so well with them (on one in particular) that I am really pleased with the results and will continue to practice, study and implement what I've learned!

The approaches to at risk kids and to students who have anger control issues were excellent and have helped me to more effectively address difficult issues and share those with other staff members.

CA BEST Behavior Cadre-Trained Sites
Degree of Implementation after Year 1
n=73 sites
  Behavioral Support Item Item Average

Current Status

item in place : 1.6+
partially in place : 1.0-1.5
not in place: 0-0.9

Schoolwide Administrator Participation 1.5 Partially In Place/ Still Needs Work
Rules Defined 1.4 Partially In Place/ Still Needs Work
Emergency Situations 1.4 Partially In Place/ Still Needs Work
Families Informed 1.2 Partially In Place/ Still Needs Work
Behaviors Taught 1.1 Partially In Place/ Still Needs Work
Rewards 1.0 Partially In Place/ Still Needs Work
Problems Defined 1.0 Partially In Place/ Still Needs Work
Behavior Support Team 0.9 Needs Attention/ Is Not In Place
Consequences Defined 0.9 Needs Attention/ Is Not In Place
Instruction Continues 0.9 Needs Attention/ Is Not In Place
All Staff Involved 0.7 Needs Attention/ Is Not In Place
Office/Class Distinct 0.6 Needs Attention/ Is Not In Place
Regular Feedback 0.2 Needs Attention/ Is Not In Place
Support Team Budget 0.1 Needs Attention/ Is Not In Place
Booster Activities 0.1 Needs Attention/ Is Not In Place
Non-Classroom Supervisors 1.2 Partially In Place/ Still Needs Work
Expected Behaviors 1.2 Partially In Place/ Still Needs Work
Scheduling 1.1 Partially In Place/ Still Needs Work
All Staff Involved 0.8 Needs Attention/ Is Not In Place
Features Modified 0.7 Needs Attention/ Is Not In Place
Behaviors Taught 0.7 Needs Attention/ Is Not In Place
Rewards 0.6 Needs Attention/ Is Not In Place
Supervision Skills 0.2 Needs Attention/ Is Not In Place
Quarterly Data 0.2 Needs Attention/ Is Not In Place
Classroom Expected Behavior Defined 1.8 In Place/ Accomplished
Taught Directly 1.7 In Place/ Accomplished
Problem Behavior Defined 1.5 Partially In Place/ Still Needs Work
Acknowledged Regularly 1.2 Partially In Place/ Still Needs Work
Transitions 1.2 Partially In Place/ Still Needs Work
Consistent Consequences 1.2 Partially In Place/ Still Needs Work
Materials Match Ability 1.2 Partially In Place/ Still Needs Work
Instruction Continues 1.0 Partially In Place/ Still Needs Work
Procedures Consistent 1.0 Partially In Place/ Still Needs Work
High Academic Success 0.8 Needs Attention/ Is Not In Place
Teacher Opportunities 0.8 Needs Attention/ Is Not In Place
Student Teacher Assistance 0.9 Needs Attention/ Is Not In Place
Family/Community Involved 0.8 Needs Attention/ Is Not In Place
Identify Students 0.6 Needs Attention/ Is Not In Place
Functional Assessment 0.4 Needs Attention/ Is Not In Place
Team Responds Promptly 0.3 Needs Attention/ Is Not In Place
Family Training 0.3 Needs Attention/ Is Not In Place
Monitor/Feedback 0.2 Needs Attention/ Is Not In Place
Local Resources Used 0.1 Needs Attention/ Is Not In Place

Statewide Collaborative Leadership Institutes

The Collaborative Leadership Institute was developed to meet the need for a comprehensive statewide network of individuals with disabilities, parents, and professionals working together to effect systems change in California's public education system. The objective of this activity was to support the development of collaborative systems involving general and special educators and families in implementing effective, research-based educational programs and strategies. Through the institute, school site teams involved in collaborative change came together with other educators and family leaders to share learning, develop skills, and plan for action. Institute participants formed a learning community that remained connected over time through the use of a variety of communication and information-sharing strategies.

From 2001 to 2004, 6 three-day institute meetings were held with nearly 900 participants attending. This event was evaluated through paper-and-pencil forms at the end of the event.

Special education teachers were the predominant group attending the event, along with a strong presence of general education teachers, both special and general education administrators, and parents. This, of all the activities, had the best balance of special and general education representatives.

Statewide Leadership Institutes Participants by Role
2001-2004 -- 893 Participants in 6 Events
Participant Role Number of Participants Percent
Teacher: SE 222 26%
Teacher: GE 144 16%
Admin: SE 138 15%
Admin: GE 137 15%
Program Specialist 82 9%
Parent 116 13%
Other 54 6%

Evaluation Findings

Statewide Leadership Institute Evaluations
2001-2004 -- 555 Responses
Item Response
(5-point scale)
Overall rating of the January Institute meeting 4.3
Overall rating of the Learning Strands 4.4
This Institute was effective in preparing me to support system change/improvement 4.2
What I learned will assist me in addressing the needs of those w/disabilities 4.2

Participants rated their experience high in every category. The overall rating of 4.4 (on a 5-point scale) is similar to the overall ratings of other training activities.

A follow-up evaluation was conducted by adding questions to the end-of-event evaluation form used at the Leadership Institute held each following year. If they had attended the previous Leadership Institute, participants indicate whether they had implemented what they learned the previous year.

Statewide Leadership Institute Followup Evaluations
2002-2004 -- 522 Responses from 4 Events
97% Implemented Learning AT LEAST ONCE
(responded 3, 4 or 5 on a 5-point scale)
69% Implemented Learning REPEATEDLY
(responded 4 or 5 on a 5-point scale)

Of the 522 participants who responded, 97 percent indicated that they had implemented strategies learned at the previous event, with 69 percent having implemented the strategies repeatedly. The percentage of participants repeatedly implementing strategies exceeded those from any other training activity. This may be due to the intensity of the three-day institute, the participation by site teams, the online support activities before and after the event, and that evaluations were only completed by those who returned for another year of institute training.

Systems Change Findings

The Statewide Leadership Institutes focus on teaching systems change skills along with providing content expertise and giving site teams the opportunity to practice skills and discuss strategies for approaching their unique situations. These are not just events with speakers standing in front of groups, but dynamic days of sharing experiences and strategies among sites and learning to become change agents, with ongoing online contact. The mixed learning communities created in the Institutes involve teams from model sites and struggling focused monitoring sites learning together. Many participants and sites have become leaders in peer coaching and scaling-up efforts in the state. The tremendous response to the Statewide Institutes led to the development and adoption of a Regional Leadership Institutes model to provide opportunities for more sites to be involved. Parent participation in the Statewide Leadership Institutes has been strong but often not as part of teams. In SIG2, parents will be among the required team members to ensure that they are fully integrated into the process.

Regional Leadership Institutes

The Regional Leadership Institutes were developed to meet the need for a responsive service delivery system to bring together regional networks of individuals with disabilities, parents, and professionals working together to effect system change in California's public education system. The objective of this activity was to support the development of collaborative systems involving general and special educators and families in implementing effective, research-based, educational programs and strategies. These events were coordinated through a regional system to best and most efficiently serve local and regional needs and to build local capacity.

From 2002 to 2005, regional hosts sponsored a total of 25 events, with nearly 3,300 participants attending. These events were primarily evaluated through paper-and-pencil forms conducted at the end of each event. In addition, follow-up evaluations to the event were conducted online, and the number of events provided and the number of participants attending were tracked in each core message area.

Regional Institute Events by Main Topic Presented
July 2002 - Jan 2005 -- 25 Events
Main Topic Presented Number of Events Percent
SE & GE Collaboration 17 68%
IDEA '97 1 4%
Transitions 2 8%
Literacy 5 20%

The majority of the training events were concentrated in the core message topic areas of special and general education collaboration and literacy.

Regional Institute Participants by Main Topic Presented
July 2002 - Jan 2005 -- 3,269 Participants in 25 Events
Main Topic Presented Number of Participants Percent
SE & GE Collaboration 2,367 72%
IDEA '97 99 3%
Transitions 194 6%
Literacy 609 19%

Participants were concentrated in the same core message topic areas -- special and general education collaboration and literacy.

Regional Institute Participants by Role
July 2002 - Jan 2005 -- 3,269* Participants in 25 Training
Participant Role Number of Participants Percent
Teacher: SE 528 29%
Teacher: GE 625 34%
Admin: SE 103 6%
Admin: GE 182 10%
Program Specialist 110 6%
Parent 37 2%
Paraprofessional 18 1%
SLP 28 2%
Psychologist/ Counselor 60 3%
Other Certified Professional 22 1%
Other 110 6%

Special and general education teachers were the predominant groups attending the events. There was also a strong presence from special and general education administrators.

Regional Institute Evaluations
July 2002 - Jan 2005 -- 25 Trainings
Survey Item Average Response
(5-point scale)
Before Training Knowledge 3.3
After Training Knowledge 4.2
(27% increase)
Overall Rating of Training 4.4
Will Share Learning 4.3
Will Use Learning 4.3
Learning Will Assist Me 4.3

There was a 27 percent increase in the average level of knowledge reported by participants after the training (from 3.3 to 4.2 on a 5-point scale). There was some variation in responses from the special and general education administrators, who consistently rated the trainings above the average on each question.

Participants rated their experience high in every category. The overall rating of 4.4 (on a 5-point scale) is similar to the overall ratings of other training activities for the past four years.

Regional Institute Follow-up Evaluations
July 2002 - Jan 2005 -- 346 Responses
87% Implemented Learning AT LEAST ONCE
(responded 3, 4 or 5 on a 5-point scale) 		86% Shared Learning AT LEAST ONCE
(responded 3, 4 or 5 on a 5-point scale)
57% Implemented Learning REPEATEDLY
(responded 4 or 5 on a 5-point scale) 		63% Shared Learning REPEATEDLY
(responded 4 or 5 on a 5-point scale)

Follow-up emails were sent to approximately 1,100 participants several months after they attended a training. From that group, 346 responses (31 percent) were received.

Of those responding, 87 percent indicated that they had implemented strategies learned at the trainings, with 57 percent having implemented the strategies repeatedly.

Participant Comments from Follow-up Evaluations

Literacy

I began to implement a Words Per Minute reading fluency strategy in which students keep track of their fluency and answer comprehension questions. Students are motivated with their charted progress. Their reading comprehension has improved as a result of their increased fluency.

We are now examining data every three weeks. We use this data to set 30 day grade level goals which, are implemented for each grade level team to improve weaknesses in our program.

Our specialist has created a never-streaming program that is seeing great results.

Special Education/General Education Collaboration

The information that was presented during the in-service made a significant impression on me. This school year, instead of offering any RSP pullout classes, I have collaborated with all of my general ed teachers. The general ed teachers seemed to be really impressed with the services the RSP students are receiving and I see that my students are starting off with a successful start to this school year.

The institute is very professionally run. It was outstanding. Specific assessments & interventions were described, as well as general scheduling and qualifying criteria, etc. And the teams so clearly projected an aura of team spirit, forward motion, and job satisfaction! Wow! YES! YES! Our team is implementing a blended program. We are at the very beginning, and have some administrative support.

The team meeting guidelines have completely revamped and refocused our department meeting. The planning time and discussions regarding data and performance are streamlined and succinct. Everyone appreciates the focused direction.

Systems Change Findings

The Regional Leadership Institutes focus on teaching system change skills along with providing content expertise and giving site teams the opportunity to practice skills and discuss strategies for approaching their unique situations. These are not just events with speakers standing in front of groups, but dynamic days of sharing experiences and strategies among sites, and learning to become change agents, with ongoing online contact. The mixed learning communities created in the Institutes involve teams from model sites and struggling focused monitoring sites learning together.

Many of the Regional Leadership Institute Host Sites have grown into that role out of their experiences as site teams participating in the Statewide Leadership Institutes and/or through their experiences in providing personnel development as Regional Coordinating Councils. The Regional Leadership Institutes structure for disseminating system change support is designed to "scale-up" the successful Statewide Leadership Institutes model by empowering regional host sites to bring together site teams in their areas to create ongoing learning communities. After being initiated in the final stages of SIG1, this model will be scaled up in SIG2.

Family Participation Fund

The Family Participation Fund was developed to meet the need for fiscal resources to help build the capacity for meaningful family/consumer involvement in local, regional, or statewide decision-making activities, events, and groups. The intent of this activity was to build partnerships that reflected the diversity of our population and are aligned with reform mandates and initiatives.

From 2000 to 2005, nearly 3,500 requests for financial support were funded, allowing family members of children with disabilities to participate in decision-making activities. This activity was evaluated by asking each recipient of financial support to complete an evaluation that requested them to list their demographic information, to rate the meeting they participated in, and to rank the adequacy of the financial support they were given.

Family Participation Fund Participants by Ethnicity
July 2000 - Jan 2005 -- 3,459* Recipients
Participant Ethnicity Number of Participants Percent
African American 2,325 71%
Asian/ Pacific Islander 56 2%
Hispanic/ Latino 468 14%
Native American 157 5%
Caucasian/ White 256 8%
*197 recipients did not specify their ethnicity

The vast majority of recipients were African American (71 percent).

The Family Participation Fund targeted outreach to ethnic minorities and was successful at increasing the percent of recipients identifying themselves as ethnic minorities to 92 percent.

Family Participation Fund Participants by Income
July 2000 - Jan 2005 -- 3,459* Recipients
Participant Ethnicity Number of Participants Percent
Under $10,000 1,800 58%
$10,000 to $19,999 471 15%
$20,000 to $29,999 324 11%
$30,000 to $39,999 141 5%
$40,000 to $49,999 132 4%
Above $50,000 212 7%
*379 recipients did not specify their income level

The Family Participation Fund also focused on reaching lower income families. Out of the 3,080 recipients who responded to the evaluation, 2,595 (84 percent) were family members earning less than $30,000 annually, with over half earning under $10,000 per year.

Evaluation Findings

Family Participation Fund Participants by Ethnicity
July 2000 - Jan 2005 -- 3,459 Recipients
Item Average Response
(5-point scale)
I learned valuable, useful information at the meeting 4.7
The meeting will make a positive difference in the lives of children with disabilities 4.6
I felt I was an effective participant at the meeting 4.5
I felt that other people at the meeting valued my participation 4.5
The financial support of attending the meeting met my needs 4.3

The recipients responded with high ratings regarding their experience at the meetings they participated in--with a 4.3 (on a 5-point scale) or higher in every category.

Responses to these questions by the different ethnic groups were consistent over the course of the five years of the SIG grant.

When looking at the responses to the evaluation by income level, the recipients earning over $50,000 annually consistently rated their experience lower than the average of all recipients.

Comments from Family Members

At this meeting we as parents were welcomed and our input into policies affecting our children's education was valued and utilized. Our concerns were responded to and we saw results.

I got to meet with really knowledgeable special education representatives, exchange ideas and information, and discuss our concerns trying (hoping) to make a difference.

This meeting presented an opportunity for parents to speak directly to the School Board members and for them to hear and understand a parent's perspective on the current state of affairs.

Being involved has given me more insight into how the funds allocated by the school district are spent and on what items.

By attending this meeting I learned how to participate as a CAC member and make our CAC more effective. I plan to encourage other CAC members and leaders to become aware of the way other CACs function and deal with challenges.

I think all parents should know what is going on in their child's school. I would like to contribute to my child's education and by attending meetings I can learn what is going on.

It's important that professionals hear a parent's perspective on how law and a school's actions impact them and their children.

It is important to be part of the decision-making process for special education, and to monitor the special education administration.

It is critical that parents/advocates be proactive on behalf of vulnerable foster/special educations students.

I will tell other families about opportunities to participate in meetings like this so they will have the opportunity to learn more about what's really going on with their children.

Any parent interested in learning how to assist their child in learning should attend and participate in meetings like this.

This meeting was a chance to voice my opinion and/or needs for my and others' children. It was also a chance to share information with other parents and to meet other parents with children like my own.

Attending this meeting helped me to stay informed of changes from the district and to get all available services for my child.

I plan to continue attending meetings like this because I know I am making a change by participating.

African American Family Outreach Project

The African American Family Outreach Project was developed to meet the need for a system with a variety of supports to increase the responsiveness of the state's service delivery model for individuals, families, and community members. The intent of this project was to use outreach activities in order to identify African American families who would then be eligible for and interested in receiving ongoing support to be effective advocates for their children with disabilities.

From 2000 to 2005, 100 African American families were identified and received ongoing support. An initial needs assessment was conducted prior to providing support services, and then two follow-up assessments were conducted at approximately two-month intervals while delivering the services. These assessments were used to help focus the support provided, as well as to measure progress.

African American Family Outreach
Summary of Evaluations
July 2000 - Jan 2005 -- N=100 Families
Item Needs Assessment 1st Follow-up 2nd Follow-up Overall Change in Average
I Understand Laws & Services 1.6 2.6 4.0 +2.4
I Participate in IEP Meetings for My Child 2.0 2.8 4.2 +2.2
I am an Effective Advocate for My Child 1.6 2.6 4.2 +2.6
I Have Connected Learning at Home with Learning at School 2.0 3.0 4.5 +2.5
I am Involved in School Activities/ Committees 1.7 2.8 4.3 +2.6

The chart shows that continuing support to families appears to bring ongoing change. This conclusion is reinforced by the approximate 1-point average gain at the end of the first two-month period and another approximate 1.5-point average gain at the end of the second two-month period in almost all categories. This pattern of change has been consistent for each of the five years that this activity has been implemented.

Family Collaborative Leadership Project

The Family Collaborative Leadership Project was developed to meet the need for a system with a variety of supports to increase the responsiveness of the state's service delivery model for individuals, families, and community members. The intent of this activity was to reach out to a diverse population representative of the state's population.

Thirteen projects around the state received mini-grants to work with the families of children with disabilities. The grants placed a particular emphasis on reaching out to involve low income and ethnically diverse families. One of the primary activities the grant recipients provided was training to empower parents to be confident and active participants in meetings, both in the IEP meetings of their children, as well as in meetings of educational decision-making bodies at the local, regional, and statewide levels.

Nearly 750 participants were reported to have engaged in the activities of the projects. The majority of participants were from low income and/or ethnic minority families, and many participants were non-English speakers and/or limited in their literacy.

Demographics
2000-2002 -- N=555 Out of a Total of 740 Participants
Low Income Low Literacy Non-English Speaking Ethnic Minority
56% 24% 28% 58%
FCLP Participants by Ethnicity
2000-2002 -- 740* Participants Attendance Meeting
Participant Ethnicity Number of Participants Percent
Caucasian/ White 231 41%
Hispanic/ Latino 147 27%
African American 129 23%
Hmong/ Vietnamese 36 7%
Native American 10 2%
*187 participants did not specify their ethnicity

Evaluation Findings

Participants were asked to complete pre- and post-training surveys to determine where there were any reported changes in knowledge, confidence, or perceived influence. The findings indicated increases in every area, with an average increase of 1 point on a 10- point scale. The greatest increase was seen in the area of understanding laws and policies. There were also substantial increases in participants reporting that they had influenced decisions that will impact the lives of children with disabilities, confidence in their ability to participate effectively in committees and groups, and feeling that they could lead a decision-making committee or group.

FCLP Training Survey
2000-2002 n=330 completed pre/post surveys
Item Before Training Average Responses After Training Average Responses
I understand laws and policies that relate to children w/ disabilities 5.2 6.7
I use my voice to advocate for children w/ disabilities 6.8 7.5
I have influenced decisions that will impact the lives of children w/ disabilities 5.6 6.6
I am confident that I can participate effectively in committees and groups 6.8 7.8
I can lead a decisionmaking committee or group 5.4 6.5
In the past 6 months I have provided support to other parents of children with disabilities 5.9 6.5

An attempt was made to track the number of training participants that went on to attend IEP and educational committee meetings, but issues related to the projects tracking participants after the initial training and discrepancies in the definition of "meetings" versus "training" call the accuracy and usefulness of the data into question. Below is a table of what was reported.

Training Attendance
2000-2002
# of Participants Attending Trainings # of Trainings Attended
582 1,379
Meeting Attendance
2000-2002
# of Participants Attending Trainings # of Trainings Attended
267 1,231

Pre-Intern Programs

The Pre-intern Program was developed to alleviate the acute shortage in qualified special education teachers and to increase support for both students not yet finished with their credentials and beginning teachers in order to promote retention.

The SIG has provided ongoing funding for four pre-intern projects geared toward improving the recruitment, retention, and subject matter passage rates of pre-interns, particularly special education pre-interns. However, the project also worked to educate all pre-interns who may eventually interface with students with disabilities in the classroom. The ultimate goal is to increase the number of fully credentialed teachers in the state of California. Each project provides an array of services to project participants.

The subject-matter exam passing rate was 58.4 percent for pre-interns in the statewide program between 1998-2000, which is comparable to the statewide passing rate for all test takers (California Commission on Teacher Credentialing Pre-Intern Report October, 2001). The ambitious goal of the SIG-funded pre-intern projects, which provide subject-matter examination test preparation assistance, was to achieve a passing rate of 80 percent for participants who completed the test preparation modules.

Through three years, the SIG-funded pre-intern projects show a combined passing rate of 79.9 percent, which essentially met the goal. In 2003-2004, a new project was added that concluded with very low pass rates, pulling the collective pass rate down and the overall rate for the four years of the project down. Nevertheless, an important beginning was made in that region and efforts will be made to assist them in learning from the projects that had very high pass rates.

Year # SIG Pre-interns Tested # SIG Pre-interns Passed % SIG Pre-interns Passed
2000-2001 90 68 75.6%
2001-2002 81 73 90.1%
2002-2003 73 54 74.0%
2003-2004 82 46 56.1%
Overall 326 241 73.9%
Year # Took Test Prep Tested # Took Test Prep Passed % Took Test Prep Passed
2002-2003 51 43 84.3%
2003-2004 34 19 55.9%
Overall 85 62 72.9%

Retention of the 326 project participants--in the pre-intern program, in the classroom, or through enrollment in credential programs--was tracked in a variety of categories. Unresolved challenges with following program participants as they move from one institution to another makes these figures incomplete. A participant maybe be counted in more than one category.

Retention # of Pre-interns
Pre-interns Progressing to Full Certification 152
Continuing in Pre-intern Program 50
Placed in an Intern Program 115
Obtained Teaching Position 210
Still in Special Education 205

Systems Change Findings

All those preparing to become teachers that participate in the pre-intern programs in California receive exposure to information about teaching students with disabilities and working with IEPs, so they will be better prepared to serve students with disabilities regardless of whether they eventually become special education or general education teachers. The pre-intern program also provides an opportunity to actively recruit those preparing to become teachers for possible careers in special education.

Several pre-intern program sites have used this program as the foundation of efforts to improve their overall teacher preparation and retention approaches, and several have reported great success in this area.

Another activity enaged in by some of the pre-intern programs has been the development and dissemination of teacher preparation materials through written materials, videos, and online documents, providing valuable resources for other areas in the state.

Resources in Special Education (RiSE) Library

The Resources in Special Education (RiSE) Library was created to provide resources that inform and support parents, educators, and other service providers on special education topics. The resources that have been made available focus on research-based practices, legislation, technical support, and current resources. The RiSE Library was housed at Parents Helping Parents (PHP) of Santa Clara, a Parent Training and Information Center (PTI), in an effort to enhance parent and professional partnerships. Patrons can procure materials one of three ways: postal mail, e-mail, and personal checkout.

In 2002-2003, a major publicity and promotion effort for RiSE was mounted, resulting in a doubling of circulation. The increase was sustained in 2003-2004, although not at quite the same levels. Through the first seven months of 2004-2005, the circulation level remains high, and projections are that the final figures will likely meet or exceed the prior year's levels.

A total of 5,424 books/resources were accessed from the RiSE Library from July 1999 through January 2005.

RiSE Library Resource Circulation
July 1999 - Jan 2005
5,424 Resources
  Jul Aug Sept Oct Nov Dec Jan Feb Mar Apr May Jun Total
99-00 42 0 0 0 1 1 13 65 62 27 52 148 411
00-01 39 53 51 73 39 23 66 39 103 65 42 32 625
01-02 48 45 59 110 85 42 86 61 63 59 82 74 814
02-03 47 43 131 223 171 177 231 159 194 96 74 69 1,615
03-04 64 103 145 147 85 60 126 93 131 49 51 107 1,161
04-05 67 67 145 156 150 102 111 *partial year *798
Grand Total
5,424

Pilot Evaluation

An evaluation was developed and piloted starting in March 2003 to determine what types of resources were being accessed, who was accessing them (e.g., administrators, teachers, or parents), and to what degree resources from the library assisted the patrons in helping students benefit from their education. Implementation of the evaluation (207 respondents) was limited; therefore, the number of evaluations collected represents only nine percent of the resources accessed from March 2003 through December 2004. The following data are from a very small pilot sample that just begins to give us an idea of how the RiSE Library is being utilized.

RiSE Library Resources Accessed by Main Topic
2003-2004 -- Sample of 207* Resources
Main Topic Number of Resources Accessed Percent
SE & GE Collaboration 15 8%
IDEA '97 11 6%
Transitions 30 14%
Parent Partnerships 15 8%
Least Restrictive Environment 9 5%
Positive Behavioral Supports 52 25%
Literacy 15 8%
Parent and Professional Partnerships 15 8%
Assessment 3 2%
*7 resources did not have a specified core message
RiSE Library Patrons by Role
2003-2004 -- Sample of 207* Patrons
Patron Role Number of Patrons Percent
Teacher: SE 75 39%
Teacher: GE 4 2%
Admin: SE 17 9%
Program Specialist 13 7%
Parent 56 29%
Paraprofessional 1 1%
Program Specialist 13 7%
Psychologist/ Counselor 1 1%
Speech/ Language Pathologist 4 2%
Other 4 2%
*17 recipients did not specify their role

The charts above show what kinds of resources were being accessed and who was checking them out. Sixty-three percent of the resources accessed in this small pilot focused on either positive behavioral supports, learning disabilities, or transitions and the vast majority of resources were accessed by either special education teachers (39 percent) or parents (29 percent).

Evaluation Findings

RiSE Library Evaluations
2003-2004 -- 207 Responses
Item Average Response
(5-point scale)
The overall rating I would give this resource is 4.2
I have implemented the strategies presented in the resource 3.6
This resource assisted me in helping students benefit from their education 3.7

The patrons in this small pilot were asked several questions about the resource they accessed. The chart to the left shows the average responses to each of the questions and indicates that the resources were almost always highly rated. This is evident by the high overall rating of the resources (4.2 on a 5-point scale).

Institutes of Higher Education (IHE) Leadership Project

The purpose of the IHE Leadership Team is to implement and monitor the California Strategic Action Plan for the Recruitment, Preparation, and Retention of Special Education Teachers. A statewide task force of representatives from over 40 organizations and programs developed this strategic plan in 2003 to address the special education teacher shortage in California, and these organizations continue to work together in implementing the plan.

The implementation principles that guide this work are:

Much of the work of the IHE Leadership Team has involved the development and dissemination of materials and information through printed brochures and through the www.teachcalifornia.org website. A detailed utilization analysis of the TEACH California website and TEACH California brochures and media was conducted. A list of the products developed is presented here, followed by highlights from the evaluation.

Products Developed

Recruitment

Delivering on the Promise and Pathways to Teaching brochures and graphic

One Child at a Time video and CD-ROM target high school students, as well as individuals from traditionally under-represented groups

Service Learning brochure targets high school students

California Special Education Internship Monograph (State Improvement Grant (SIG) funds contributed to the printing, and supported this effort by Dr. Belinda Karge, CSU, Fullerton)

Matrix of California Special Education Internship Programs

Preparation

Model for Recruiting and Preparing Special Education Teachers Linking Community Colleges and Four Year Institutions

Beginning Teacher Support and Assessment/California Formative Assessment and Support System for Teachers adaptations for Mild/Moderate/Severe Special Education Teachers

California Shortage of Highly Qualified Personnel for University Faculty Positions in Special Education website document

Special Education Teacher Preparation Report based on an electronic survey and data analysis regarding SE teacher preparation in California conducted by the CSU, Chancellor's Office

Retention

Website Document that Highlights "Retention Data from Pipeline Study"

Website Literature Review that Addresses Special Education Teacher Retention Issues

Special Education Teacher Retention Report based on an electronic survey and data analysis regarding SE teacher retention in California conducted by the CSU, Chancellor's Office

Preparation/Professional Development of School Administrators (information on NERRC Strategies) for administrators, teachers, state department, and certification representatives.

TEACH California Website Evaluation Findings

The website became active on March 31, 2004. Visitor summary data were collected through December 31, 2004 (a period of nine months).

Materials Downloaded/Shipped from the TEACH California Website

From March 31, 2004 to December 31, 2004, over 10,000 items were disseminated.

Materials Downloaded Shipped Total
Become a Special Education Teacher in California 3,859 1,358 5,217
Finding Your Pathway to Teaching (English) 88 1,427 1,515
Special Educators, Delivering on the Promise-Financial 250 1,085 1,335
Special Educators, Delivering on the Promise-Recruitment 324 875 1,199
Special Educators, Delivering on the Promise-Overview 327 621 948
One Child at a Time Video Guide 164 23 187
Special Educators Delivering on the Promise-Poster 95 70 165

Nearly 20,000 additional downloads were made from the website of various materials providing information on financial aid, assistance, or other incentives for teachers. Based on this analysis, the IHE Leadership Team is considering developing a financial incentives summary brochure for distribution on the website.

TEACH California Website User Survey Results

To find out more about who is accessing the website, a user survey was recently developed and added to the website. Between January 7, 2005 and January 23, 2005, the survey was piloted, with 1,150 individuals responding to the survey. During that period, there were approximately 10,625 unique visitors to the website, so the survey response rate was approximately 11 percent. The following pie charts provide a description of the individuals accessing the website and completing the survey.

Who are you?
Response Percent
Career changer with a college degree 27%
Career changer without a college degree 3%
College student 14%
Community college student 6%
Credentialed teacher 13%
High school student 4%
Intern teacher 3%
Out of country teacher 4%
Out of state teacher 12%
Other 10%
Missing 4%
Are you thinking of becoming a teacher one day?
Response Percent
Yes 75%
No 2%
Not applicable 23%
Are you taking steps to become a teacher right now?
Response Percent
Yes 57%
No 17%
Not applicable 26%
What type of teacher are you thinking of becoming?
Response Percent
Elementary Teacher 34%
High School Teacher 24%
Special Education 13%
Math Teacher 8%
Middle School Teacher 7%
Science Teacher 5%
Missing 9%
Age
Response Percent
Over 45 15%
36-45 20%
25-35 36%
Under 25 26%
Missing 3%
Ethnicity
Response Percent
White 52%
Hispanic 15%
Asian 11%
African American 9%
Asian 11%
Native Ameriacn 1%
Other 7%
Missing 5%
Gender
Response Percent
Female 71%
Male 26%
Missing 3%

IHE Leadership Team Members and Consultants

Partnership Committee on Special Education (PCSE) Annual Meeting

The Partnership Committee on Special Education (PCSE) was developed to meet the need for a comprehensive, statewide network of individuals with disabilities, parents, and professionals that both reflects the diversity of the state's population and is aligned with reform mandates and initiatives.

The PCSE is the collaborative stakeholder committee responsible for developing the State Improvement Plan that provides the foundation for the State Improvement Grant. The committee is a broadly diverse and representative group of individuals involved in, or concerned with, the education of children with disabilities. This group continues to meet annually to provide guidance to the implementation of the grant.

Over the course of the grant there were approximately 188 different organizational partners that participated in the PCSE, with an average of approximately 100 stakeholders attending the meeting annually. As shown in the chart below, there was a balance of representation between parents, CDE staff, school/district/SELPA administrators, professors and staff from institutions of higher learning, other state and professional agencies, program specialists, and teachers.

PCSE Partners by Role
2001-2004
188 Partners
Partner Role Number of Partners Percent
CDE 34 18%
Admin: SE 26 14%
Admin: GE 7 4%
IHE 27 14%
Parent 40 21%
Program Specialist 11 6%
Teacher: SE 4 2%
Other Agency Staff 20 11%
Other 19 10%

Evaluation Findings

PCSE Meeting Evaluations
2001-2004 -- 178 Responses
Item Average Response
(5-point scale)
Overall, how was the meeting 4.2
Do you feel we used your time wisely 4.1
Were you able to contribute in a way that brought your best thinking to the table 3.9
Was this meeting of value in advancing the goals/vision of the SIG 3.8
Will what you learned assist you in addressing the needs of individuals with disabilities 3.6

Participants in the PCSE meeting rated their experience fairly high in every category, with an overall rating of 4.2 (on a 5-point scale).

Systems Change Findings

The Partnership Committee on Special Education (PCSE) meets annually to develop and discuss recommendations on the implementation and evolution of the SIG. The meetings of this ongoing stakeholder group, which entail a high degree of interaction, have built relationships between partners that had not previously been brought together, model cross-stakeholder processes for special education in the state, and build ownership of the work of the SIG. These discussions take place in a shifting environment as No Child Left Behind (NCLB), the federal focus of the SIG, and learning from experience all contribute to influencing the direction of the SIG and its activities.

The SIG Evaluation Task Force is a small group of representative stakeholders who meet with the SIG evaluation staff twice annually for a full day to review activity and outcomes evaluation data and to make recommendations to the PCSE based on that data concerning the implementation and direction of the SIG. This feedback loop ensures that close examination of the data can be used to learn from experience. As a result of this process, the SIG objectives have been aligned with the state's Key Performance Indicators, and SIG2 activities have been designed to be focused and intensive enough to begin to measure specific outcomes at the site level.

Making data accessible and understandable to stakeholders has been a cornerstone of California's SIG since its inception. Over the course of the SIG, collective attention has been brought to data on specific outcomes, and even though that data was not considered "clean" at first, the continued attention served to focus collective efforts on what was intended to be impacted by the SIG, and eventually brought about clarification of measures and mechanisms so that at the end of SIG1 there is a reasonably good baseline. The need to get activity evaluation data into the hands of those putting on the trainings more rapidly led to the development of a training evaluation database, affectionately know as TED, which is being piloted in SIG2.

 

IDEAS that Work!

Project READ is a California Department of Education, Special Education Division project funded through a federal competitively-awarded State Personnel Development Grant to California (#H323A120019) provided from the U.S. Department of Education Part D of the Individuals with Disabilities Education act (IDEA), Office of Special Education Programs (OSEP). Opinions expressed herein are those of the authors and do not necessarily represent the position of the U.S. Department of Education.