Skip to Content

 

Improving the Special Education System in California

The California Department of Education and the Partnership Committee on Special Education
A State Program Improvement Grants Program, CFDA 84.323A

Abstract

Background: The original State Improvement Grant, as delineated in the 1997 Reauthorization of the Individuals with Disabilities Education Act (IDEA), has provided the California Department of Education with a unique opportunity to forge a whole new era of broad-scale, collaborative partnerships. The second State Improvement Grant reaps this enormous benefit from the first—along with significant information about what’s most needed now and what works.

Objectives: The Partnership’s rationale for selection is that these objectives directly address key identified needs and reflect the research-based “core messages” selected by the Partnership:

Activities: To achieve these objectives, the project will fund the activities that were selected for inclusion based on 1) their contribution to important and quantifiable systemic change, 2) their strong foundation in research and effective practice, and 3) their capacity to be “scaled up” to meet the needs of California’s large and diverse population.

Project participants: All required partners and many optional partners are included. As appropriate, given the objectives, there is an emphasis on including parents/family of students with disabilities.

Expected outcomes: Achievement of the objectives will be measured by precise indicators that detail the expected outcomes.

Table of Contents

  1. Part I: Application for Federal Assistance
    • Form 424
    • ED Code Title 34: Protocol Summary Sheet
  2. Part II: Budget Information
    • Form 524
    • Budget narrative
  3. Part III: Application Narrative
    1. Need for Project
      1. Process for identifying gaps or weaknesses in services, infrastructure, or opportunities
      2. Nature/magnitude of weaknesses in services, infrastructure, opportunities
      3. How identified weaknesses will be addressed
    2. Significance
      1. Significance of the activities to address the identified weaknesses
      2. Likelihood that the project will result in system change or improvement
        1. Involvement of individuals with disabilities or parents of these individuals
        2. Historical success in creating systemic change/improvement
    3. Quality of the Project Design
      1. Clearly specified, measurable goals, objectives, and outcomes
      2. Relationship of the project design to the identified needs
      3. Activities constituting a coherent, sustained program of training in the field
      4. Project design reflecting up-to-date knowledge from research and effective practice
      5. Linkages with other appropriate agencies/organizations
      6. Project as part of a comprehensive effort to improve teaching/learning and support rigorous academic standards for students
    4. Quality of Project Personnel
      1. Encouragement of applications from underrepresented groups
      2. Qualifications of key project personnel
      3. Qualifications of project consultants or subcontractors
    5. Adequacy of Resources
      1. The adequacy of support, including facilities, equipment, supplies, other
      2. The relevance/commitment of each partner to project implementation and success
      3. Adequacy of the budget to support the project
      4. Reasonable costs in relation to project objectives, design, and potential significance
      5. Continued support of the project after Federal funding ends
    6. Quality of the Management Plan
      1. Management plan to achieve the project objectives on time and within budget
      2. Ensuring a diversity of perspectives are brought to bear in project operations
    7. Quality of the Project Evaluation
      1. Thorough, feasible, appropriate methods to evaluate goals, objectives, and outcomes
      2. Evaluation examines the effectiveness of project implementation strategies
      3. Objective performance measures producing quantitative/qualitative data
      4. Methods provide performance feedback and periodic assessment of progress
  4. Part IV: Assurances and Certifications
    • Form 424B: Assurances – Non construction Programs
    • Form 80-013: Certifications regarding lobbying; debarement, suspension and other responsibility matters; and drug-free workplace requirements disclosure of lobbying activities
    • Form 80-014: Certification regarding debarement, suspension, ineligibility and voluntary exclusion – lower tier covered transactions
  5. Appendices
    1. Appendix A: Staff Vitae
    2. Appendix B: Staff Time Commitment
    3. Appendix C: Evaluation Tools/Instruments
    4. Appendix D. Agreements
    5. Appendix E: Training and Technical Assistance State-Approved Core Message Areas

PRIORITY REQUIREMENTS LIST

Page # Requirements
Narrative, d.1 pp. 56 (a) Projects funded under this notice must make positive efforts to employ and advance in employment qualified individuals with disabilities in project activities. (See Section 606 of IDEA)
Narrative, b.2.a., pp. 20 (b) Applicants and grant recipients funded under this notice must involve individuals with disabilities or parents of individuals with disabilities in planning, implementing, and evaluating the projects. (See Section 661(f)(1)(A) of IDEA)
Narrative, all pages (c) Applicant must describe steps to ensure equitable access to, and participation in, its program for students, teachers, and other program beneficiaries with special needs. (See Section 427, GEPA)
Budget (d) Projects funded under these priorities must budget for a two-day Project’s Directors’ meeting in Washington, D.C. during each year of the project.
MOUs (e) The project shall, consistent with its partnership agreement established under the grant, award contracts or subgrants to LEAs, institutions of higher education, and parent training and information centers, as appropriate, to carry out its State improvement plan.
Budget (f) The project shall use not less than 75 percent of the funds it receives for any fiscal year to ensure that there are sufficient regular education, special education, and related services personnel who have the skills and knowledge necessary to meet the needs of children with disabilities and developmental goals of young children; or to work with other States on common certification criteria

Narrative

Introduction

The original State Improvement Grant (SIG), as delineated in the 1997 Reauthorization of the Individuals with Disabilities Education Act (IDEA), has provided the California Department of Education (CDE) with a unique opportunity to forge a whole new era of broadscale, collaborative partnerships. The Department has used this opportunity to change the entire system of decision making in the State, opening the planning and resource allocation process to the highly representative and inclusive group, the California Partnership Committee on Special Education (PCSE or Partnership) in compliance with IDEA Section 1452(b).

Though federal grants often have the goal of creating internal systems change, the amount of funding available frequently becomes subsumed or marginalized, especially in larger states, resulting in an ineffective effort of the proverbial tail attempting to wag the dog. The State Improvement Grant has proved to be the exception—a dramatic exception—in California. The original grant has changed nothing short of the entire vision of the special education support process, planning, implementation, and assessment.

As the State approached the planning for this second full State Improvement Grant (SIG2), there was no question about priorities, no discussion about the use of evidenced-based practice, and no controversy over process. These issues and many others have become part of the Partnership’s annual agenda, a process which has brought an unprecedented consensus of understanding and willingness to put individual agendas aside for the common good. The second State Improvement Grant reaps this enormous benefit from the first—along with significant information about what’s most needed now and what works. The Partnership is pleased to offer this proposal for the next stage of systems change in California.

(a) Need for Project

a.1. Process for identifying gaps or weaknesses in services, infrastructure, or opportunities.

This SIG proposal, SIG2, is based on the work of California’s Partnership Committee on Special Education (PCSE) that has been operating for almost 6 years to analyze the need for systemic change of the special education (SE) system, the priorities for resource allocation to meet these needs, and the results of the SIG and other efforts to make the needed changes. In the process of fulfilling their duties, the PCSE examines a myriad of data including the Government Performance Results Act (GPRA) measures reflected in the SIG Biennial Performance Report, information from the SIG Annual Report, data from the Continuous Improvement Monitoring Process (CIMP), the Self-Assessment, and the rank order of 618 data. The PCSE SIG Evaluation Task Force works closely with the SIG and other projects to determine the degree to which current efforts are effectively addressing the needs.

Annually the PSCE examines the data and current progress in meeting the identified needs and creates priorities. Those priorities then guide the staff of the California Department of Education (CDE), Special Education Division in their allocation of resources, including the potential resources requested in this current SIG application.

a.2. Nature/magnitude of weaknesses in services, infrastructure, opportunities.

The major areas of need in services, infrastructure, and opportunities are explored below.

Compliance issues: Compliance findings are evolving as changes in the direction and scope of the California Department of Education (CDE) monitoring processes have shifted to balance procedural guarantees with educational benefit.

Procedural Guarantees. From a procedural standpoint, the most frequent compliance issues are in the following areas:

These, and other high frequency items are checked in each review. The frequency of these findings has reduced over time, as more districts have participated in CDE monitoring processes.

Educational Benefit. For the last two years CDE has placed increased emphasis on monitoring for educational benefit. This has occurred in two ways. First, CDE reviews are customized based on the performance of districts on Key Performance Indicators (KPIs) related to the Department’s goals and based on concerns expressed by parents in the district. So, for a given district, the items reviewed are selected (via software) to assess procedural issues that are programmatically linked to the KPIs. The second method for monitoring educational benefit is an educational benefit file review process. Conducted jointly by CDE and district staff, the educational benefit review involves a detailed look at the IEPs planned for a child over 3-years. As a part of the review, the team evaluates the relationships between the present performance, the needs identified, goals and services established to support the needs, and the progress made in each need area. In addition, IEPs are compared across years to determine if the IEPs were reasonably planned to result in educational benefit (e.g., were the goals and services adjusted in response to the progress that was made by the student). These two processes have led to increased findings related to:

Largely, these results reflect several general problems in providing in-service and support to staff and planning services for students:

These monitoring findings may, in some measure account for the following statewide results:

Special education students meeting high standards for academic and non-academic skills: The results for California’s students with disabilities are delineated below.

Students with disabilities participation in STAR: STAR is the Standardized Testing and Reporting system in California that combines scores from a nationally normed test (currently the CAT 6) with scores from the criterion-referenced California Standards Test which is grade and subject specific. A student taking any section of either test is counted as participating in the STAR. In 02-03, 94.9% of special education students participated in STAR.

Academic performance in STAR for students with disabilities: The part of the STAR which is nationally-normed (initially SAT 9, now CAT 6) provides a percentage of students with disabilities scoring >50% on the combined language arts and math tests. In 02-03, 19% scored at this level, compared to 50% of all students nationally.

Students with disabilities exiting with a diploma: The chart below shows a clear increase in the percentage of students with disabilities exiting with a diploma, certificate or GED.

Percent of CA's SE Students Exiting with a High School Diploma, Certificate of Completion, or GED
Year High School Diploma Certificate of Completion or High School Proficiency through GED
99-00 37.6% 13.2%
00-01 51.6% 11.4%
01-02 58.8% 6.2%
02-03 58.7% 7.6%

Students with disabilities returning to GE: Between 1999-2000 and 02-03, California’s rate of students returning to GE dropped slightly from 6.4% to 5.6%. However, the field acknowledges that these figures are not highly reliable for comparative purposes due to the wide variance in protocols among the districts and the changing definitions over time.

Students with disabilities dropping out: The chart below shows that California has made steady progress in reducing the number/percentage of students with disabilities dropping out of school. The rate has declined over 50% between 1997-98 (7.3%) and 02-03 (3.6%).

Percent of California's Special Education Students that Dropped Out of School
Year Percent Dropped Out of School
97-98 7.3%
99-00 4.5%
00-01 4.1%
01-02 4.0%
02-03 3.6%

Performance of students with disabilities on measures of literacy: The performance of California’s students with disabilities on measures of literacy shows progress and promise at the elementary school level and very little or no progress at the middle/high school levels. The combined STAR tests scores in reading are shown below for 4th, 7th, and 10th grade students over the 5 year period of 1997-98 to 01-02; 02-03 was not included due to the change between SAT 9 and CAT 6 that year. The chart demonstrates the 14.8 point growth in averaged scaled scores for 4th graders, compared to a 6.9 point increase for 7th graders and a .3 point decrease for 10th graders.

STAR Reading Average Scaled Scores
CA's 4th, 7th and 10th Grade Special Education Students
  97-98 98-99 99-00 00-01 01-02
4th Grade 598.7 604.6 611.7 614.5 614.1
7th Grade 634.5 636.1 642.1 644.5 641.4
10th Grade 655.9 659.4 662.1 660.1 655.6

The problem becomes even more dramatic when data from the California Standards Test in English/Language Arts (CST E/LA) is examined. As previously discussed, the CST is a criterion-referenced test designed to carefully align with the State’s academic content standards. Grade level specific exams are given each year to all students whose scores are reported in 5 levels: Advanced, Proficient, Basic, Below Basic and Far Below Basic.

The CST levels of Proficient and above are considered as acceptable for NCLB Adequate Yearly Progress (AYP) for K-8 students. For 9-12, AYP was benchmarked at scores for math and English for 10th graders on the California High School Exit Exam (CAHSEE). Those scores (set considerably above the “passing” score for the CAHSEE) were found to be statistically equivalent to the Proficient score on the CST for grade 10. Thus the CST LA gives quite an accurate picture of progress on the AYP literacy measures, as well as the State standards, for students with disabilities. The chart of CST E/LA 2003 scores on the next page demonstrates several significant findings about literacy levels of California’s students with disabilities.

California Standards Test Scores ~ 2003
English Language Arts Proficiency ~ By Grade Level
  Total Proficient of Above Advanced Proficient Basic Below Basic Far Below Basic Total Less than Proficient
Grade 4, All Students 39% 15% 24% 35% 18% 8% 61%
Grade 4, Students with Disabilities 14% 5% 9% 24% 27% 34% 85%
Grade 7, All Students 36% 10% 26% 33% 18% 14% 65%
Grade 7, Students with Disabilities 6% 1% 5% 17% 24% 53% 94%
Grade 10, All Students 33% 11% 22% 30% 22% 14% 66%
Grade 10, Students with Disabilities 5% 1% 4% 14% 34% 47% 95%

Students with disabilities receiving educational services equal to their GE peers. Historically students with disabilities have not received equal treatment in the education system, leading the IDEA 97 to focus on two specific areas of equality—access to the GE curriculum and relative rates of punishment in terms of suspensions and expulsions.

Students with disabilities being integrated into GE: While the trend demonstrates an increase over the last 5 years in involvement of SE students in the GE environment, the data is not consistent enough to be credible. CDE has taken steps to create a more universal definition of percent of time students are integrated into GE, so data should be more reliable in the future.

Students with disabilities expelled or suspended compared to GE students: California’s capacity to collect valid data in this area has increased over the last 5 years, but the initial data is not clean enough to present a real trend line. In 02-03, 57,389 students with disabilities or 9.2% were suspended one or more times; the number and percentage of GE students is not available. However, the expulsion rates are comparable, with 906 or .15% of students with disabilities being expelled compared to .16% of GE students.

Students with disabilities reflected GE ethnicity/socioeconomic status: California continues to have an over-representation of some ethnic/racial groups and under-representation of others in various disability groups, though the over-all disparity index has dropped from 10.7 in 1999- 2000 to 10.3 for 02-03. The chart below shows the enrollment in SE compared to GE by ethnicity statewide. African-Americans continue to be most over-represented and Asian- Americans most under-represented, the exact opposite of the make-up of college undergraduates.

Racial/ Ethnic Disproportion in Special Education in California, 2002-2003
Enrollment Native American Asian or Pacific Islander Hispanic or Latino African American White Total
General Education 53,898
0.90%
700,671
11.30%
2,819,504
45.20%
515,805
8.30%
2,106,042
33.70%
6,244,403
100.00%
Allowable Range
(+/- 20%)
High 1.08
Low 0.72
High 13.56
Low 9.04
High 54.24
Low 36.16
High 9.96
Low 6.64
High 40.44
Low 26.96
 
Special Education 5,804
0.90%
39.180
5.80%
BELOW
293,779
43.50%
81,569
12.10%
ABOVE

255,000
37.80%

6,75,332
100.00%

Students with disabilities being served by highly qualified personnel: This performance measure includes data on both the quantity of personnel available and the degree of their professional training. Between 1998-99 and 02-03, the number of SE teachers increased nearly 28%, from 79,340 to 101,386. This marks substantial progress, as the number of students with disabilities in California during the same period increased by only 7%.

During the same period, California’s fully certified SE teachers decreased only slightly, from 80.1% being fully certified in 1998-99 to 76.4% in 02-03—a generally positive sign given the vast increase in total personnel during that timeframe. The remaining personnel are teaching under emergency permits or credential waivers, both of which require enrollment in SE coursework and progress toward SE certification.

California does not require Interstate Collaborative Agreements to meet personnel preparation needs, as in-State colleges/universities offer all of the following specialization areas:

a.3. How identified weaknesses will be addressed:

See section b immediately below.

(b) Significance

b.1. Significance of the activities to address the identified weaknesses. The chart below shows the summary of the identified weaknesses/gaps, the State’s response and the source of funding for that response. Those activities to be funded through the SIG2 are indicated and then explored in greater depth in the balance of this section. As noted above, the rationale for selecting strategies for inclusion in the SIG2 included 1) their contribution to important and quantifiable systemic change, 2) their strong foundation in research and effective practice, and 3) their capacity to be “scaled up” to meet the needs of California’s large and diverse population.

The other funding source cited is CalSTAT, the California Services for Technical Assistance and Training, a State-funded comprehensive service system providing a wide scope of activities to support improvement in services to children/youth with disabilities and their families. CalSTAT is operated through a sub-contract from CDE to the Sonoma State University California Institute on Human Services and is closely aligned to the SIG. Additional strategies addressing the needs are delineated, with funding sources, in section c.6, pp. 54-55.

Identified gap/weakness State response Funding source
Increase quantity of personnel through improved recruitment and retention California Task Force on Teacher Recruitment and Retention CalSTAT
Increase quality of personnel through pre-service training Pre-Intern Award Program CalSTAT
Objective 1: Increase quality of personnel through in-service professional development

Statewide Leadership Institute

Regional Leadership Institutes

Targeted core messages TA

Special Edge; Rise Library

SIG

SIG

SIG

CalSTAT

Objective 2: Improve educational coordination

State Leadership Institute

Regional Leadership Institutes

Site to Site TA

SIG

SIG

SIG

Objective 3. Improve academic performance of students with disabilities

Regionalized training

Regional Leadership Institutes

All core message TA

New model site ID/support

CalSTAT

SIG

SIG

SIG

Objective 4. Improve behavior supports or behavior outcomes California BEST TA SIG
Objective 5. Increase participation of parents and families

Family Participation Fund

African American Outreach

SIG

CalSTAT

Objective 6. Improve data collection and data dissemination

TED Dissemination

SIG Evaluation Design

SIG

SIG

Activities: The PCSE has chosen various strategies to address the objectives. Core message TA: Technical assistance (TA) is defined in the SIG as including a broad range of activities—any training, facilitation, modeling, problem-solving, trouble-shooting, or coaching that supports the efforts of IHEs, SELPAs, districts, schools, families and parent organizations, and other organizations to more effectively educate students with disabilities. TA may be delivered on-site, on-line, by phone, or via fax or hard-copy.

All SIG-supported TA must be linked to the CDE-approved “core messages”. Core messages convey fundamental research-based principles in the following priority topic areas; areas specifically targeted in SIG2 are asterisked:

The core messages are posted on the CIHS web site along with bibliographies of supporting literature and research; see www.calstat.org/cores.html.

In SIG2, core message TA is delivered in several ways: centrally coordinated TA, site-tosite TA, California’s BEST cadre TA, Regional Institute TA, and TED dissemination TA. The first is described directly below, with the others following.

• Centrally coordinated TA involves support and centralized coordination of TA services to educators (including institutes of higher education), education administrators, education-related service providers, and families/consumers. When requests are received, SIG staff respond with a type of triage process where one of the TA Coordinators first refers the caller to the general guidelines for TA posted on the website at http://www.calstat.org/ta.html and requests a call back, if the need/request conform to the guidelines. S/he then discusses the need and potential alternative resources to meet the need. If SIG-funded TA is appropriate at that point, s/he awards a number of TA days, usually 1-2 but up to 6 maximum.

TA requests are prioritized according to the following criteria: those that…

TA is budgeted at $400 per day. This “TA day” budgeting mechanism is used as the basis for subcontracts and invoicing. However, it has become clear that this framework must be implemented flexibly to accommodate the very diverse ways in which TA is provided. The budgeted TA days serve as target deliverables for the project, but TA requests may not align directly with TA days. Certain TA requests, such as customized field-requested training, may appropriately encompass several TA days. On the other hand, one TA day may stretch across weeks or even months when the TA is provided in increments via e-mail or fax.

The TA day funds are paid to the district/SELPA/Parent Training Institute, etc.—not to the TA provider. This has resulted in significant leveraging of funds, as many TA providers in California charge more than $400/day. The requesting site may hire their own consultant or may have their needs matched by SIG staff with a consultant from the over 500 in the current SIG/CalSTAT consultant pool.

  1. Adopting a whole systems approach (working as part of a systemic effort to improve student outcomes, including diversity of representation, offering opportunities to work in teams representative of local systems, and focusing on system change);
  2. Building connections over time (connecting the learning community year round through online conferencing/conference calls, and focusing on full team support);
  3. Emphasizing shared leadership and capacity building (viewing participants as experts, providing parents with an equal voice, structuring to foster connections, using model sites as presenters, using participants to lead round tables); and
  4. Delivering high priority, school focused and process-focused core message content in a format encouraging integration and application (developing content strands, focusing keynote presentations on models/skills for systems change, allowing teams time to integrate learning).

Regional Institute sub-contractors receive $5,000/day for up to 2 days to cover direct Institute and wrap-around follow-up costs and $1,500/day to support parent trainers.

b.2. Likelihood that the project will result in system change or improvement. The project design has a high likelihood of resulting in systemic improvement because…

As change theorists have known for years, the likelihood of systemic improvement is dependent on a complex of factors, not just the viability of the change design. In addition to the solidity of the project design in terms of system improvement, California is primed for change, as evident in the following factors:

b.2.(a) Involvement of individuals with disabilities or parents of these individuals. Individuals with disabilities and parents of persons with disabilities have been involved in the selection and planning of the State Improvement Grant activities and will be involved in the implementation and evaluation of the activities. This involvement is detailed below.

Planning: The PCSE is charged with the SIG planning, and this eclectic group is comprised of all required and many optional partners under law. Within this group, there are 5 persons who are self-described as individuals with disabilities and 49 persons who are selfdescribed as parents/family members of individuals with disabilities. Because the total membership is 100, persons with disabilities and parents/family members constitute approximately 54% of the whole—a very good representation for the planning body.

Implementation: Persons with disabilities and parents/family members will be highly involved in the project implementation. The following discussion focuses on Objective 5, “to increase participation of parents/family members of children with disabilities in the systems change process.” By including this as a separate objective rather than subsuming parent/family participation in other objectives, the SIG2 highlights the absolute necessity of improving parent/family involvement as a means of realizing the other objectives—increasing the quality of personnel who serve students with disabilities, increasing access to general education curriculum for students with disabilities, increasing the academic performance of students with disabilities, and improving behavior supports and outcomes for students with disabilities.

The indicators chosen to measure the success of this objective show the depth of the State’s commitment to include parents/family members in every aspect of the project.

Indicator 5.a. Increase the number and proportion of persons self-identifying as “parents/family members” in SIG-sponsored TA and Regional Institutes. To show progress in this indicator, the project will require that organizations/institutions conducting TA, receiving TA, participating in the Regional Institutes, and organizing the Regional Institutes place a premium on the attendance and involvement of parents/family members. Some methods the project may use to ensure progress on this indicator include…

Indicator 5.b. Increase the number of parents/family members taking an active leadership role on school teams at Institute activities. To show progress in this indicator, the project will require that organizations/institutions conducting TA and Regional Institutes increase the number of parents/family members participating on site Leadership Teams each year.

Indicator 5.c. Increase the participation of low-income families in local/regional SE advisory bodies. The typical committee having policy influence in SE issues at the local or regional level is the Community Advisory Committee. This indicator will be included in the contract for the provider of the Family Participation Fund.

Indicator 5.d. Increase the participation of parents/family members in decision-making committees that are not specifically SE-oriented. The Family Participation Fund speaks directly to the issue of increasing the participation of low-income parents/families of children with disabilities in decision-making level committees, and 15% of that fund will be set aside to support parents/family member participation in GE committees. In California at the site level, the required committee that has a significant effect on policy and resource allocation is the School Site Council. Other periodic policy/resource site committees include the Western Association of Schools and Colleges (WASC) family committee for high school accreditation and the parent/community committee for schools receiving funding under the Immediate Intervention for Under-performing Schools Program. All districts with significant percentages of English Learners must support an English Learner Advisory Committee which heavily influences policy and resource allocation; most also have a parent advisory group that works directly with the Superintendent. Many districts and SELPAs also support a parent advisory committee.

At the State level, there are a myriad of committees in the Department of Education, as well as other relevant Departments such as Developmental Disabilities. The project maintains a list of key committees and is on listserv notifications for openings which are then publicized to active parent/family members of children with disabilities in the K-12 community.

For involvement of committees at this level, the SIG2 will expand the efforts of the Family Participation Fund to increase the number of stipends and increase the publication of their availability to low income parents of children with disabilities.

Indicator 5.e. Increase the impact parents/family members have through their participation in educational reform committees. This indicator goes beyond merely increasing the numbers of parents/family members to target an increase in the impact they report that they have on local, regional, and statewide educational advisory and policy-making committees. An increased emphasis on training for effectiveness for parents participating in these committees can be realized through TA by CACs, Family Empowerment Centers, and other parent organizations around family partnerships core messages.

Evaluation: Parents/family members will be involved with the evaluation of the project on multiple levels. First, as discussed above, they will be directly responding to structured inquiry about their effectiveness in local, regional, and statewide educational advisory and policy-making committees. Second, they will be surveyed about the effectiveness of the SIGfunded TA in meeting their needs, and these findings will be included in the quarterly formative evaluation structure to generate continuous improvement in project outcomes. Third, a random sample of those who are involved in the delivery of TA will be interviewed to determine the quality of that experience, e.g. the support provided, the equality of their membership on the team, the importance of their contribution, etc. Fourth, as a result of their involvement in districtand SELPA-level program effectiveness analysis, parents/family members will increase their understanding of data-based decision making and using project-data to help make key decisions at the local level. Finally, as members of the PCSE, parents/family members will be evaluating the project and recommending adjustments to generate optimal outcomes for all objectives.

b.2.(b) Historical success in creating systemic change/improvement. The first SIG and SIG Supplement targeted a variety of outcomes, with significant success for most. The section below reflects the activities conducted under these first two grants. The content foci for all activities were the common principles or “core messages” developed by the PCSE and approved by the California Department of Education. These core messages articulate critical research findings and essential components of effective application in the following high-priority topic areas: connecting special education to the California Reading Initiative; positive behavioral supports for a safe and healthy school, collaboration between general and special education; family partnerships; transition from school to adult life; improving the IEP process—IDEA 97; and LRE. Within each content area are 3-5 core messages which appear in Appendix C “T/TA Core Message Areas;” the complete bibliography of supporting research for each is available at www.calstat.org/cores.html. The TA provider or organization receiving SIG dollars has been required to focus on one or more of these messages in all SIG funded TA.

Objective focus area 1: Improving Personnel Quality and Quantity. A major focus of the first SIG was instituting measures to address systemic change in the improvement of SE personnel quantity and quality. This was accomplished through several strategies, as discussed below.

Dissemination of Core Message TA. Over the 5-years of the SIG, research-based core message knowledge/skills have been disseminated through TA delivered through 2 mechanisms:

1.1. Regional Coordinating Councils (RCCs). The Regional Coordinating Council was designated by the PCSE as the vehicle in which SIG TA dollars could be used to meet local and regional needs while efficiently disseminating consistent high-quality personnel development and family education in the core message areas. In year one of the SIG, the existing Regional Coordinating Councils (RCCs) were realigned to match 11 education regions utilized by a variety of general education initiatives. In practice, RCCs provided TA based on core message areas through contractual agreements which included implementation of other selected SIG-funded activities. In the final year of the grant, RCCs delivered a minimum of 10 training days in core message areas based on local and/or regional need. RCCs were further supported by centralized TA funds for those sites who wished to augment training with follow-up TA. Actual distribution of TA content is shown for the last year in the chart below demonstrating a concentration in behavior—which led to the focus on positive behavior interventions in the SIG Supplement—as well as literacy, SE/GE collaboration, and IEP improvement through IDEA 97.

Regional Coordinating Council-Sponsored Trainings
2000-2003 # and % of Trainings by Main Topic Presented
656 Trainings - 37,881 Participants
Main Topic Number of Trainings Percent
SE & GE 78 12%
IDEA '97 69 11%
Transitions 35 5%
Parent/ Professional Partnerships 27 4%
Least Restrictive Environment 18 3%
Behavior 243 37%
Literacy 186 28%

In follow-up e-mail surveys at least 3 months after the TA event, 87% of participants responded that they had implemented their newly learned skills/knowledge, and 58% indicated that they had implemented their learning repeatedly. In addition, 85% reported having shared their learning with other professionals or parents, with 60% having shared their learning repeatedly. Data collection and analysis validity was carefully monitored as the intent of the project was to increase participant skills/knowledge, have them implement their new learning, and then have them share it with others to further disseminate the skills/knowledge and contribute to more widespread system change. For the 656 trainings provided in 2000-2003, participants averaged a 20% increase in knowledge/skills. Also, the trainings were rated highly, overall 4.5, and in terms of participants reporting that they will share and use their learning, and that they anticipate it will assist them in addressing the needs of students with disabilities.

1.2. Centrally Coordinated Technical Assistance. The second primary delivery mode for TA in the initial SIG was through a centralized, coordinated TA service to educators (including institutes of higher education), educational administrators, education-related service providers, and families/consumers. TA includes any training, facilitation, and coaching that supports the efforts of schools and families to more effectively educate students within the core message structure. Technical assistance may be delivered on-site, on-line, and utilizing other modes of communication such as phone, fax, and paper-copy distribution.

The chart below shows the considerable field-based interest SE and GE collaboration. It also reflects how the SIG project has used TA to build on the outcomes of the Leadership Institutes by offering Site to Site TA and providing follow-up/ongoing TA to individual sites to implement system changes.

Centrally Coordinated Technical Assistance
2000-20003 # and % of Events by Main Topic Presented
423 Events - 12,397 Participants
Main Topic Number of Events Percent
SE & GE 192 44%
IDEA '97 87 21%
Transitions 16 4%
Parent/ Professional Partnerships 39 9%
Least Restrictive Environment 3 1%
Behavior 49 12%
Literacy 37 9%

With 423 events and 8,782 responses to questions about learning, quality, and intent, the results were quite similar to those for RCC TA events—high rates of implementation and sharing of the training: 84% implementing the learning at least once, 57% repeatedly; 85% sharing the learning at least once, 64% repeatedly. Additionally, given the fact that this end-of-event evaluation instrument has been normed on over 30,000 participants, deviations from the normal high ratings become flags of success or concern.

Objective focus area 2: Improving Service Coordination

2.1. Collaborative Leadership Statewide Institute. This institute is a unique opportunity for professional growth that allows participants to contribute to a community of learners who are working to improve education in California. The goal of the Statewide Institute is to support the development of collaborative systems involving general and special educators and families in implementing effective, research-based, educational programs and strategies.

Through the Institute, school site teams involved in collaborative change efforts come together with other educators and family leaders to share learning, develop skills, and plan for action. Institute participants form a learning community that remains connected over time through the use of a variety of communication and information-sharing strategies.

The Institute has a face-to-face and an Internet online component. The online component involves up to four weeks of virtual conferences that take place before, during, and after the faceto- face meetings. Additionally, an ongoing, interactive website features participant-driven conferencing that focuses on priority topics and virtual site visits to successful schools.

The Institute has evolved over the past five years in both content and structure. Initially, three statewide meetings were held focusing on behavior, reading, and collaboration respectively. Thereafter, two meetings with up to 12 weeks of wrap-around online learning were held per year with learning strands in each previously listed area in addition to other forms of convening and educating adult learners such as: the use of open space technology, round table discussions sponsored by sites with promising practices or information, the establishment of online tools with year-round access, and the inclusion of high-level system change speakers such as nationally recognized author and speaker Dr. Margaret (Meg) Wheatley. The learning community has grown to approximately 700 participants and stays connected via the Internet. On average, in years four and five, 300 participants have gathered each year for one statewide meeting and four weeks of wrap around learning. Approximately half of the participants are new to the community each year, and the role designation is typically as follows: SE and GE teachers, administrators, program specialists, and parents of children with disabilities.

The chart below shows responses of participants in the Statewide Institute either 6 months or 12 months after the event. A very impressive 97% have implemented the strategies and 69% implemented them repeatedly. This somewhat masks the even more intense result that is generated because the Institute is attended by a site team, magnifying the effect.

Statewide Collaborative Leadership Insititutes
2002-2004 Implementation Follow-up
4 Events - 552 Responses
Implemented Learning at Least Once
(3, 4 or 5 on a 5-point scale)
97%
Implemented Learning Repeatedly
(4 or 5 on a 5-point scale)
69%

2.2. Collaborative Leadership Regional Institutes. In year five of the SIG, the project expanded its efforts to grow the Statewide Institute learning community by supporting regionally-based institutes at the local level making available both financial and administrative support to those entities demonstrating capacity and interest. Regional institute host sites contract to deliver training based on specific desired outcomes, specifically to create an ongoing learning community, sustain and expand system change, encourage and support meaningful family involvement, identify and share successful practices.

Sites focus on the core messages and deliver them through the guiding principals of the statewide meeting: adopt a whole systems approach, build connections over time, emphasize shared leadership, and deliver high priority content in a format that encourages integration and application. Sites follow a specific format and convene once per year with online wrap around learning to support their ongoing role as host regional institute sites. Last year 15 events involving 1,587 participants were held: 2 focused on IEP improvement/ IDEA 97, 3 on literacy, and 10 on SE/GE collaboration. Participation ratios are shown in the chart below:

Regional Leadership Institutes
2002-2003 Participants by Role
16 Events - 1,246 Responses
Role Percent
Teacher: SE 17%
Teacher: GE 21%
Administrator: SE 5%
Administrator: GE 11%
Program Specialist 7%
Parent 3%
Psychologist 6%
Resource Specialist 1%
Speech & Language Pathologist 1%
Not Specified 8%
Other 20%

The Regional Institute participants recorded an 18% increase in knowledge/skills and, at the 3 month follow-up, 86% reported that they have implemented and shared what they learned, 48% implemented their learning repeatedly, and 63% shared their learning several times.

2.3. Site-to-Site TA. A third method of increasing knowledge/skills in collaboration has been through site-to-site TA, including peer mentoring and model site visits. Currently there are 11 Collaborative Challenge Model Sites identified and awarded for demonstrating success in the collaboration core message area from 1999 to 2001. The Model Sites share information related to their programs based on research data and evidence of promising practices. Site to site TA has included online virtual tours, on-site visits, model site conference presentations and phone TA.

The project conducted a survey specifically of the 100 sites that received TA focused on SE/GE collaboration to determine if their service delivery system had changed post-TA. With 42 sites responding, findings showed that, prior to TA, the sites were operating more separately than collaboratively (4.19 on a 10 point scale). After receiving TA, the sites were operating more collaboratively than separately (6.66 on a 10 point scale). The chart below shows the relationship of TA days to change, with the trend line showing greater TA generating greater collaboration, with perhaps diminishing returns after 14 days.

Collaborative Site Survey
Average Difference Between Site Response Regarding Service Delivery Prior to TA Consultation and Now
Days of TA Received Average Difference in Degree of Collaboration
Recieved 1 Day of TA
10 sites
1.5

Recieved 2 Days of TA
5 sites

2.1
Recieved 3 Days of TA
8 sites
3.1
Received 4-6 Days of TA
4 sites
2.0
Received 7-9 Days of TA
8 sites
3.0
Received 10-13 Days of TA
3 sites
3.2

Received 14-17 Days of TA
4 sites

2.9
Overall Average Difference +2.47

These findings were echoed in written survey responses with 80% of sites reporting change most often from a pull-out service delivery model to one with much greater inclusion of RSP and SDC services in GE classrooms through differentiated instruction based on assessment. Participants reported, “Much more integration working collectively as a team and monitoring of progress;” “more measurable goals via the SST process;” ”more team-taught courses with SE and GE teachers in core subject areas;” “a three-tiered remedial reading program taught by one reading specialist, four resource teachers and two GE teachers;” and similar outcomes.

Objective focus area 3. Improving Student Academic Outcomes. A major SIG focus has been improving student academic outcomes, as measured through the model collaborative sites’ Academic Performance Indicator (API) level, California’s method of ranking schools primarily by standards-aligned, criterion-based student test scores. The scores of the 23 sites that had received >3 days of TA and had received some of that TA prior to 2002-2003 to allow time for change, showed a 78 point average gain in their API score between the 1999 and 2003. The model sites also reported other student outcomes including decreased special education referrals and improved attitudes among both teachers and students.

Objective focus area 4: Improving Behavioral Supports and Outcomes. The SIG focus on improving student outcomes was addressed, in part, through a behavioral management program developed at the Institute on Violence and Destructive Behavior (IVDB) at the University of Oregon. This program, Building Effective Schools Together (BEST), is a school-wide curriculum based on principles developed by the National Center on Positive Behavioral Interventions and Supports (an OSEP-funded research center) at University of Oregon. By SIG year 4, 1006 school team members from 166 schools teams were trained in BEST. With 58 trainings and 2,313 responses, participants reported a 20% knowledge/skills gain, 91% rate of implementation and 67% for repeated use, and 89% rate of sharing, with 78% repeatedly sharing. The SIG Supplement funded training/support of a cadre of 45 trainers who have delivered 20 BEST I and 12 BEST II trainings to school site teams, with the results shown in the graph below.

Building Effective Schools Together (BEST) Trainings
California Cadre Team Trainings
2003 Average Evaluation Responses
8 Trainings - 495 Responses
Item Response
(1 to 5)
Before Training Knowledge 3.4
After Training Knowledge

4.4 (a 20% increase)

Overall Rating of Training 4.4
Will Share Learning 4.4
Will Use Learning 4.5
Learning Will Assist Me 4.4

Objective focus area 5: Improving parent/family participation and knowledge. The SIG funded two specific efforts to address this objective: the Family Participation Fund and the African-American Outreach Program. Both are described below.

5.1: Family Participation Fund. This is a system-focused goal aimed at increasing the presence of parent and family representatives of individuals with disabilities on key initiative committees at decision-making levels. Stipends were made available and expenses reimbursed for parents from all cultural, linguistic, socioeconomic, geographic and disability groups to support their participation on policy and decision-making bodies. Outreach successfully targeted families from communities that are traditionally underrepresented in decision-making, such as low-income families and families of color. Especially in the last year, the Fund has supported predominately families making $29,000 per year of less, and the majority of recipients have been ethnic minorities, particularly African American and Latino families.

Family Participation Fund
2001-2003 # and % of Recipients by Ethnicity
995 Recipients
Ethnicity Number Percent
African American 480 48%
Asian/ Pacific Islander 34 3%
Caucasian/ White 233 23%
Hispanic/ Latino 145 15%
Native American 26 3%
Other 26 3%
Did Not Specify 51 5%

5.2: African American Family Outreach. This activity has been carried out through a sub-contract with Parents of Watts (POW), a community-based, multi-service, parent-led organization. Through their efforts, a limited number of very low-income families living in Watts received the intensive services they needed to become more effective partners in their children’s education. In addition, Sweet Alice Harris, Executive Director of POW, delivered motivational, strategy-focused presentations to families, educators, and administrators in various areas of the state to assist in the development of partnerships between families, especially lowincome families of color, and educators. This project has consistently shown high levels of change, although with a limited number of families being served.

African American Family Outreach
Summary of Family Needs Assessments and Follow-ups
2002-2003 N=2- Families
Evaluation Item Response
(1 "Not At All" to 5 "Very Much)
I Understand Laws and Services 1.8 2.9 4.1
I Participate in IEP Meetings for My Child 3.3 4.3 5.0
I am an Effective Advocate for My Child 2.2 3.1 4.4
I Have Connected Learning at Home with Learning at School 3.0 3.8 5.0
I am Involved in School Activities/ Committees 2.1 3.3 4.4

Objective focus area 6: Improving pre-service recruitment, retention, and qualification. Two components of the first SIG focused on improving pre-service recruitment, retention, and qualification: the pre-intern program, and the IHE task force. Both are discussed below.

6.1. The Pre-Intern Program. The SIG has provided ongoing funding for four preintern projects geared toward improving the recruitment, retention, and subject matter passage rates of pre-interns. The focus has been on special education pre-interns, but the project also worked to educate all pre-interns who will interface with students with disabilities in the classroom. The ultimate goal has been to increase the number of fully credentialed teachers in California. Each project provides an array of services to program participants.

The average subject-matter exam pass rate was 58.4% for all test takers, according to the California Commission on Teacher Credentialing Pre-Intern Report October, 2001. The ambitious goal of the SIG-funded programs was to achieve a pass rate of 80% for participants who completed the test preparation modules. The 2000-2001 pass rate of 75.6% exceeded the statewide average but fell just short of the program goal. However in 2001-2002, the pass rate of 90.1% far surpassed the program goal. The 2002-2003 figures are for all SIG pre-intern participants (including those who did not complete the test preparation modules.) Among those who completed test prep in 2002-2003, 84.3% passed their subject- matter exams. The combined pass rate for the three years of the program is 79.9%, which meets the program goal.

6.2. The IHE Task Force. In years one and two SIG dollars were allocated to Institutions of Higher Education (IHEs) to reduce the number of students that were “wait listed” for teacher credentialing classes with a special education degree emphasis. At the end of this period, the PCSE reviewed this activity, considering both its original purpose—to increase the number of fully credentialed teachers in the state—and the number of individuals that were actually being impacted by this activity. The PCSE determined the original purpose of this activity would be better served by appointing a subcommittee, the “IHE Task Force,” to make recommendations for a more comprehensive approach to such a large systemic problem. The committee members (eight total) have included IHE professors, the CSU Chancellor’s Office, the California Association of Professors of Special Education Board President and a representative of the California Commission on Teacher Credentialing.

The IHE Task Force met in year three and created the “California Strategic Plan for Teacher Recruitment, Preparation and Retention of Special Education Teachers” and presented the plan for review and further recommendation by the PCSE. This newly revised plan then moved into implementation in years four and five. The plan includes 14 distinct activities spread across three activity areas: teacher recruitment, preparation, and retention. The IHE Task Force presently meets six times per year to coordinate and review the progress of the strategic plan.

In year four, one activity the task force sponsored, the IHE Connections Meeting, was a 2-day leadership meeting to build public awareness and focus on issues related to SE teacher shortages. Findings included that the professors gained significantly from sharing resources and information. In fact, 12 of the 33 respondents (36%) specifically mentioned “making connections and networking” a great benefit of the meeting—a significant harbinger of system change.

Objective focus area 7: Improving Data Collection and Use. The last area of emphasis in the first SIG, improving data collection and use, was addressed through three elements: evaluation database, evaluation task force, and the Training Evaluation Database (TED).

7.1. Evaluation Database. Many charts displayed above were generated from a database developed to enable end-of-event and effect-over-time data to be entered and summarized, as well as automating follow-up email surveying of training participants. This data provides periodic feedback to SIG managers, Regional Coordinating Councils, and the SIG Evaluation Task Force in making recommendations based upon the data to the PCSE. The database has significantly increased the value of the SIG’s continuous improvement efforts.

7.2. SIG Evaluation Task Force. The SIG Evaluation Task Force meets twice annually to review data and make recommendations to the PCSE, reviewing data from the previous year’s activities in the fall and data from the previous year outcome objectives in the spring. This is a small stakeholder committee with significant parent participation.

7.3. Training Evaluation Database (TED). TED was developed as part of sustainability planning around the SIG and in response to the need to produce reports with the data collected by SELPAs. Many SELPAs collect but have no easy way to summarize, and therefore use, the data gathered for their planning processes. TED has standard reports that can be generated at the push of a button for specified data ranges and levels of aggregation.

(c) Quality of the project design

c.1. Clearly specified, measurable goals, objectives, and outcomes.

Goal: The central purpose of the State Improvement Grant is to improve outcomes for students K-12 with disabilities. To address this complex and multi-faceted purpose, the California Partnership Committee for Special Education (PCSE or Partnership) formally adopted as the goals for the State Improvement Grant the research-based “required elements of an effective educational system now and in the future” listed in IDEA Part D, Subpart I §1451(a)(6). As discussed above, the group then prioritized the needs, chose research-based and proven activities to address these needs, and assigned to the SIG funding those crucial elements required for systemic change.

Outcomes: This process resulted in 6 outcomes, detailed below, aligned with the SIG vision and targeting improved quality of in-service personnel, improved educational coordination, improved student academic outcomes, improved behavioral support and outcomes, improved parent/family participation, and improved data collection/use. From the prioritized needs and the outcomes directly related to those needs, the activity focal points for systemic improvement were developed as measurable objectives. All relate directly to the core messages.

These objectives were written in alignment with the Government Performance Results Act (GPRA) standards and call for 1) the dissemination of research-based content through scientifically proven methods; 2) scaling-up of proven effective strategies; 3) enhancement of dissemination systems; and 4) improvement of performance of children with disabilities. Each objective has clearly quantifiable indicators of important outcomes which include the assessment mechanism and benchmark of success. The PCSE has found that this very clear, inclusive method of presenting measurable objectives clarifies the process for all constituents—not just those who speak evaluation jargon.

Objectives and outcomes: The objectives/outcomes are detailed below by outcome areas.

Outcome 1: Improved Quality of Personnel Working with Students with Disabilities: Objective 1. To increase the quality of personnel through the provision of core message, research-based, in-service professional development through technical assistance (TA) and Regional Institutes, as demonstrated by:

Outcome 2: Improved Education Service Coordination for Students with Disabilities: Objective 2. To improve collaboration between special education (SE) and general education (GE) by:

Outcome 3: Improved Academic Outcomes for Students with Disabilities: Objective 3. To increase the academic performance of students with disabilities as demonstrated by:

Outcome 4: Improved Behavioral Supports and Outcomes for Students with Disabilities: Objective 4. To improve behavior supports and outcomes for students with disabilities, as demonstrated by:

Outcome 5: Improved Participation of Parents/Family Members of Students with Disabilities: Objective 5. To increase the participation of parents/family members of children with disabilities in the systems change process by:

Outcome 6: Improved Data Collection/Use of Data in Addressing the Needs of Students with Disabilities: Objective 6. To improve data collection and the use of data by local educational agencies in identifying and addressing the need for T/TA to support the continuous improvement of outcomes for students with disabilities by:

c.2. Relationship of the project design to the identified needs.

In section a, 6 primary needs were identified; those needs—and the response in the SIG2 project—are delineated below.

Need 1: Increase the number of fully qualified SE personnel serving students with disabilities. Response: These efforts will be funded outside of the SIG2, as the activities effectively addressing the need do not constitute systems change at this time.

Need 2: Increase the knowledge and skills of SE, GE, and other personnel serving students with disabilities. Response: The PCSE believes that in-service training is essential to increasing the quality of personnel and better informing parents/family members. Over 85% of the SIG2 budget is devoted to this effort which includes the following activities: the Statewide Leadership Institute, the Regional Institutes, the comprehensive core message TA, the Model and Learning Site Communities in the core message areas, Site-to-Site TA, the Family Participation Fund, and the data collection/analysis tool TA. The SIG2 in-service TA adheres to the researchbased strategies for maximizing effect in adult learning.

Need 3: Improve education service coordination, particularly between SE and GE. Response: Increased collaboration was a strategy implemented in the initial SIG through the model site program. This successful approach will be scaled up in SIG2 to provide opportunities for increased Site-to-Site TA in this area. Additional model and learning sites, as well as Regional Institutes focused on collaboration, will assist in the scale-up efforts.

Need 4: Improve outcomes for students with disabilities. Response: The PCSE determined to target two specific areas for improved outcomes—literacy and behavior. The literacy focus was additionally targeted, given the overwhelming need indicators, to the middle and high school levels. This is logical given the outpouring of federal and State resources to generate preschool reading readiness and a strong foundation of reading skills in elementary school. Students who have fallen through the cracks in this system, particularly students with disabilities, show a clear decline in language arts achievement in middle and high school. Thus the SIG2 targets TA specifically to this group.

The second area of improved student outcomes targeted is behavior. Based on the very effective BEST program introduced in the original SIG and scaled up in the SIG Supplement, the SIG2 funds will be devoted to the support and institutionalization of BEST in the existing sites and the spread of BEST training through the California cadre of BEST trainers. The cadre was originally trained and partially supported through the SIG Supplement. Those trainers—all of whom are charged with professional development in their jobs at districts, County Offices of Education, or SELPAs—will now also be increasing the numbers of sites introduced and supported to implement BEST as a sustainability effort.

Need 5: Increase the participation of parents/family members of children with disabilities in the systems change process. Response: The initial SIG highlighted the effectiveness of 2 approaches to parent participation activities that held the potential for scaling up: the emphasis in the Statewide Leadership Institute and Regional Institutes on including parents as team members and the Parent Participation Fund to encourage low-income and under-represented parents to participate in the systems change process.

In SIG2, the Statewide Leadership Institute will carry an even greater emphasis on the inclusion of parents both on participating site teams and as co-presenters. The Regional Institutes will also place a premium on both levels of parent/family member inclusion. The Parent Participation Fund will be targeted specifically to low-income families and those from racial/ethnic groups under-represented in college, specifically African-American, Hispanic, Native American, and some Southeast Asian groups. The PFP will also emphasize participation in GE policy-setting/recommending groups, as well as SE committees.

Need 6: Improve data collection and data use to address system needs and to ensure continuous improvement. Response: In addition to the on-going SIG effort to increase evaluation capacity in the field, the SIG2 specifically provides TA to SELPAs and County Offices of Education that wish to use a database to monitor their 1) TA offerings, including the core message areas being covered; 2) the district and school site origins of their attendees; and 3) the roles of the participants at those sites. The database, TED, offers agencies an efficient electronic data management system that can provide meaningful reports upon which professional development resource allocations may be based.

c.3. Activities constituting a coherent, sustained program of training in the field.

As previously discussed, pre-service activities to meet identified needs for improved quantities of SE personnel are part of the State’s comprehensive program but are funded through CalSTAT using State funds. The SIG2 focuses on in-service TA to up-grade the knowledge/skills of existing personnel and parents/family members of children with disabilities.

The intentional coherence in this system is provided by the scientifically proven core messages that form the foundation for all TA provided by SIG funding. The research base for the core messages targeted for the SIG2 is presented below.

c.4. Project design reflecting up-to-date knowledge from research and effective practice.

As noted above, the SIG2 effort will focus on specific, research-based core messages related to the objectives. The relevant research and effective practice related to each is detailed in the discussion below.

Objective 1. To increase the quality of personnel through in-service professional development. In-service professional development is the traditional vehicle for increasing personnel skills and knowledge. Indeed, the knowledge and skills teachers need to effectively teach children with special needs comes from preparation programs grounded in professional standards and on-going professional development (Boyer, 2003). Training makes teachers better able to do their jobs. Both the scope and quality of professional development experiences appear to be associated with SE teacher quality. Teachers whose recent in-service experiences covered a broad range of specified topics have higher aggregate teacher quality scores than those whose recent professional development covered fewer topics. Likewise, teachers who perceived their recent district-supported professional development as relevant and of high quality had higher teacher quality scores (Carlson, et al., 2002).

The need for in-service training is especially important amongst teachers of special education, as attrition rates for SE teachers exceed those for GE teachers by about one-third, and more continuing teachers transfer out of special education than in (Boe, et al., 1997). In a 1998 study, one third of entering SE teachers lacked standard certification for their assignments, as did 10% of all special education teachers (Boe, et al., 1998).

Decades of research have clarified the structure for highly effective professional development. Hocutt notes that “Several instructional methods appear to result in improvements in academic outcomes for students with mild disabilities. The more promising programs involve on-going teacher training, teacher planning time, and administrative support. When additional resources are provided, outcomes for non-disabled students may also be improved” (Hocutt, 1996). The principles of effective teacher training, all of which are addressed in SIG efforts, include: programs linked to school-wide efforts; teachers participating as helpers to each other and as planners, with administrators, of follow-up activities; emphasis on self-instruction, with differentiated training opportunities; teachers in active roles, choosing goals and activities for themselves; emphasis on demonstration, supervised trials, and feedback; training that is concrete and ongoing over time; and ongoing assistance and support available on request (Sparks, D. and Loucks-Horsely, 1989).

The SIG TA programs devote a significant share of training time to peer-to-peer instruction in learning community settings. Using highly qualified personnel who have “been there, done that” in successful programs is effective in offering feedback to the community, answering questions, helping plan future activities and set goals, discussing problems and plans, and supporting these research-based activities after participants return to their programs (Smollar, 1998). Effective training programs also give participants the opportunity to reinforce what they have learned by testing, repeating, and practicing, an element built in to all the SIG TA activities (Kennedy, 2003).

Clearly, to be effective, training programs must provide follow-up support to participants as they implement newly learned skills. When the intent of training is implementation of new knowledge and skills, specific plans for providing follow-up support to the participants must be woven into the training, not tacked on as an afterthought. Effective training assumes that the support provided to assure implementation is the second, but equally important, component of the training activity. (NTAC, 2000) This follow-up and support is inherent in all SIG TA activities, and a variety of support approaches utilized in the SIG have been demonstrated as effective including on-site visits, mentoring or coaching, video review, live video interactions, product review and feedback, and observations; these can be carried out face-to-face or on-line (NTAC, 2000.)

The SIG will address the need for ongoing teacher training through a variety of programs with demonstrated success, including the Statewide Leadership Institute, Regional Leadership Institutes, and the California BEST Trainers program. The project will implement continuous support for trainees through site-to-site TA as well as centrally coordinated TA. In addition, resources and information for teachers will be provided on an ongoing basis through the Special EDge newsletter and the RiSE Library, services funded through the State. The bottom line is “that special education can be no better – and no worse – than the quality of instruction provided by teachers” (Heward, 2003), and structured TA can affect that quality.

Objective 2. To improve collaboration between special education and general education. Improving the collaboration between SE and GE has a myriad of potential positive outcomes ranging from increased effectiveness in use of resources to increased access to GE curriculum for students with disabilities. The latter can have particularly wide-ranging results, including “a shared vision; national, state, and local policies; training and external support; organizational structure; and community advocacy” (Erwin & Soodak, 1995; Lieber et al., 2000, quoted in Soodak, 2002).

The landmark IDEA ’97 legislation stipulates that students with disabilities are entitled to access to the general curriculum. It also entitles students to participation and progress within that curriculum—language which provides great opportunities (Hitchcock, 2002). For teachers to successfully include students with disabilities in their classrooms, they must be provided with adequate staff development and technical assistance, based on the needs of the school personnel (e.g., information on disabilities, instructional methods, awareness and acceptance activities for students, and team building skills) (ERIC Clearinghouse on Disabilities and Gifted Education, 1993).

Special education directors in a nationwide survey chose information via the Internet as the most highly preferred form of TA (Thompson & Thurlow, 2003). But training needs to be reinforced through a shared vision that is most easily provided by working directly with professionals with like goals and successful, hands-on experience. The ideal combination, therefore, is a mix of face-to-face TA and Internet support. The State Leadership Institute is designed to offer both a face-to-face component and a web-based component to maximize effect while minimizing cost both to funders and to participants. These Institutes are enhanced and the messages reinforced by Regional Leadership Institutes.

Objective 3. To improve the academic performance of students with disabilities. Traditionally, special education has focused on improving the academic performance of students with disabilities by targeting efforts to those students and the personnel who provide special instruction/support. However, results from the field increasingly demonstrate that involving general education is key to student academic success, thus supporting the concept of SE/GE collaboration.

“Perhaps the biggest benefit for children with disabilities has been the focus of general education…. As schools make their plans for school improvement, they focus efforts on children with disabilities for the purpose of increasing the number of students scoring at the proficiency level. Because students with disabilities comprise a disproportionate share of students who are not at the proficiency level, they are the direct beneficiaries of district wide efforts to improve student performance” (a state SE director, quoted in Thompson & Thurlow, 2003).

The SIG will particularly focus on reading intervention, as that has long been recognized as fundamental for the academic success of students with disabilities. Through Reading First and other State/local initiatives, resources have been targeted to improve reading outcomes for all students in pre-K through grade 5, and the State is beginning to see the fruit of this effort.

Efforts for improving reading at the middle and high school levels have been less intensive and have had minimal impact to date. In California, the State has established reading intervention programs for middle schools and has recommended them for high schools. Some of these programs offer the type of intensive and individualized instruction Hocutt references: “Research suggests that the most effective interventions for students with disabilities…provide intensive and reasonably individualized instruction combined with very close cooperation between general and special education teachers; and careful, frequent monitoring of student progress” (Hocutt, 1996). The SIG will contract with model sites that are demonstrating significant success with students with disabilities—and general education students—using these programs to provide Site-to-Site TA. Additional TA will be provided to address specific site issues in implementing these types of programs.

Objective 4. To improve behavior supports and outcomes for students with disabilities. The PCSE supports the Positive Behavioral Support (PBS) approach for improving behavioral outcomes for students with disabilities. “Unlike traditional behavioral management, which views the individual as the sole problem and seeks to ‘fix’ him or her by quickly eliminating the challenging behavior, PBS views such things as settings and lack of skill as parts of the ‘problem’ and works to change those. As such, PBS is characterized as a long-term approach to reducing the inappropriate behavior, teaching a more appropriate behavior, and providing the contextual supports necessary for successful outcomes” (Warger, 1999).

PBS provides a broad range of systemic and individualized strategies for achieving important social and learning outcomes while preventing problem behaviors in all students (Sugai et al., 2000). PBS has been demonstrated to be an effective means of early intervention, especially when combined with parental involvement measures. If PBS is implemented early in a child’s school career, outcomes and experiences are greatly enhanced during the subsequent years of schooling (Fox, et al, 2002).

The California BEST Teachers program delivers a school-wide discipline program based on PBS methods, where school teams are trained to develop and implement positive school rules, rule teaching and positive reinforcement systems. BEST focuses on making civility and prosocial behavior a core value of the school’s culture which increases instructional time available for learning for all students, decreases time spent out of the classroom/school by behavioral offenders, and teaches lifelong skills in human relations. There is solid evidence that a pro-social behavioral culture creates safer and more effective schools. Decades of research (see, for example, D. Gottfredson, et. Al, 1997) demonstrate that a positive, inclusive, and accepting climate is essential for an effective school.

The California findings have been presented in section b above, pp. 32. The original Oregon study asked treatment and comparison schools to report the frequency of office discipline referrals pre/post BEST. All treatment schools reported reductions in referrals in the intervention year compared to the baseline and showed greater improvement than comparison schools. Middle school baseline year referrals ranged from 550-3167 and in the intervention year from 260-2608, making the average percent change a 36% decrease (range = -18 to -53%) compared to 82% increase in the comparison schools (range = -39 - +203%). Of treatment elementary schools with two years data, the average percent change was a 51% decrease (range = -18 to - 68.5%), with comparison schools showing a 7.5% decrease (range = -6 to 9%) (Sprague et al, 1999).

Objective 5: To increase the participation of parents/family members of children with disabilities in the systems change process. Decades of research have proven that “when parents become involved, children do better in school, and they go to better schools.” (Henderson & Berla, 1994). The proven value of parental involvement has corroborated the position of Joyce L. Epstein and Susan Dauber that parental/family involvement in a child’s education is the most significant factor in academic success — more significant than parental education level or socioeconomic status (Dauber & Epstein,1993). This is especially true for students with disabilities, where a national longitudinal study found that “parental involvement in a student's education was powerfully related to student performance. Controlling for other factors, students whose parents were more involved in their education missed fewer days of school and were much less likely to fail courses than were students whose parents were less educationally involved” (SRI International, 1997).

However, “Teachers must be trained to promote effective parent/family involvement in children’s education… It cannot be assumed that teachers will naturally know how to [do this]” (Carter, 2002). And likewise, parents/family members need information and support to work with the educational establishment. “It cannot be assumed that parents instinctively know how to involve themselves in their child’s education. In fact, many parents feel inadequate in teaching roles…and in dealing with the schools” (Simmons, Stevenson & Strand, 1993). This trend begins in the elementary grades and increases as students enter high school, even though “an adolescent’s success is influenced by his or her family even through the last year in school” (Trusty, 1999).

Thus one of the core messages to be delivered through TA is that educators have ways to work effectively with families and that families must have the skills/understanding to work effectively with educators. Effective working relationships require systematic two-way communication — school to home and home to school (Epstein, 2001). This requires that the project provide educators and families with strategies and techniques for connecting learning activities at home and in the community with learning at school. The SIG will provide this through targeted training and technical assistance.

Another core message in relation to parent/family member involvement is that the system must prepare families to participate actively in school decision-making and to exercise their leadership and advocacy skills. Among under-represented ethnicities and low-income families, including parents in assessment, placement, policy-making, and advocacy is recommended to restore the balance of power in parent-professional discourse (Harry, 1992). This is exactly what the SIG Parent Participation Fund does.

Objective 6: To improve data collection and the use of data by local educational agencies in identifying and addressing the need for T/TA to support the continuous improvement of outcomes for students with disabilities. “Developing accurate reporting procedures on the participation of students with disabilities in large-scale assessment programs has proven to be difficult, due to a lack of data, differing definitions of eligible testing populations, and the misalignment of data collection efforts and data management responsibilities” (Erickson, R., Thurlow, M. & Ysseldyke,1996).

Since those words were written, states have greatly improved their data collection for students with disabilities (Elliott, et al, 2000), but there is much work to do. However, California has a very limited data collection system at the local, regional, and State levels. To address the need, SIG will disseminate a SIG Evaluation Design that will provide extensive information on progress toward the program goals and implications for students with disabilities.

c.5. Linkages with other appropriate agencies/organizations.

California currently has formal MOUs on the delivery of services for students with disabilities with 5 agencies: The California Commission on Teacher Credentialing, the California Department of Developmental Services, the California Department of Health Services, the California Department of Social Services, and the California Department of Rehabilitation. The major linkages with appropriate agencies and organizations are within the Partnership. The types of agencies and organizations represented are roughly grouped below, with the numbers of individuals in attendance each year noted; the letters of commitment for these organizations is in Appendix A.

PSCE Role - Required Partners 1999 2000 2001 2002 2003
Governor's Office 1 1 1 1 1
Departments and Parent Organizations 11 23 28 35 21
Department of Developmental Services 1 3 4 1 1
CA State Advisory Commission on SE 1 3 3 2 2
Interagency Coordinating Council 1 2 2 2 2
Institutions on Higher Education 6 9 10 12 7
General Education 2 10 12 8 6
Special Education 52 36 45 45 27
Early Intervention Programs 2 6 6 5 4
Community-based/ Non Profit Agencies   19 22 19 12
PCSE Role - Optional Partners          
Department of Rehab 1 2 3 2 2
Department of Health Services 1 1 1 1 1
Mental Health   3 3 3 3
Other Agencies   24 36 46 21

c.6. Project as part of a comprehensive effort to improve teaching/learning and support rigorous academic standards for students.

The PCSE has provided a sounding board for resource allocation to support the comprehensive effort to improve teaching/learning and mastery of rigorous academic standards by students with disabilities. The funding behind this comprehensive effort is a patchwork. In section b.1, the chart shows efforts funded through CalSTAT and SIG. The following activities reflect another part of the comprehensive plan:

(d) Quality of project personnel

d.1. Encouragement of applications from underrepresented groups.

It is the policy of the California Department of Education, in alignment with federal Equal Opportunity Employment regulations, to encourage employment applications from underrepresented groups. In the case of the SIG, the only employment involving federal funds will actually be through Sonoma State University (SSU).

SSU is subject to all laws governing affirmative action and equal opportunity including Executive Order 11246 and Title IX of the Education Amendments Act. SSU has an outstanding history of nondiscriminatory employment practices for members of traditionally underrepresented groups such as racial/ethnic minority groups, women, and persons with disabilities, as documented by the annual reports filed with the State; the personnel policies explicitly guarantee equal treatment and encourage applications for employment from members of underrepresented groups. Reports over the last decade show full compliance with the affirmative action/equal opportunity plans filed with the State. All staff positions in this proposal will be filled under stringent enforcement of these policies. If vacancies occur during the project, employment of new personnel will be fully subject to these policies.

For the SIG, Sonoma State University will send electronic job notices to all Historically Black Colleges and federally listed minority institutions, in addition to other colleges/universities. Notices will also be sent to associations of education professionals from underrepresented groups. All notices will state “Persons belonging to groups underrepresented in higher education are encouraged to apply.”

Persons with disabilities already employed by the California Institute on Human Services are provided with special support by the CIHS Assistant Director, who holds an M.A. in SE, the purpose being to ensure that they have all possible resources/assistance to compete for promotional opportunities.

d.2. Qualifications of key project personnel.

SIG Administrators: The qualification for project administrators is to be a currently serving Director or Division Director for the California Department of Education—General Education, CDE—Special Education, and the California Department of Developmental Disabilities.

Deputy Superintendent, Curriculum and Instruction Branch, Sue Stickel, M.A. Sue Stickel is the current Deputy Superintendent of the Curriculum and Instruction Branch at the California Department of Education. She came to the Department from the field, having served as a classroom and mentor teacher, a director of instructional support, and the assistant superintendent of curriculum and professional learning for the Elk Grove Unified School District. She additionally serves as an instructor in educational administration and teacher education for the California State University system. Ms. Stickel has been involved in providing professional development across the State in standards-based curriculum, instruction, and assessment, particularly in mathematics.

Administrator (California Department of Education, Special Education). Alice Parker, Ed.D. Dr. Parker served as the Director of Student Services for the San Mateo-Foster City School District from 1991 to January 1998 when she accepted the position of Director of the Special Education Division of the California Department of Education. In her previous position, she served as District Compliance Officer, OCR Officer, and 504 Coordinator, as well as all programs for students with special needs (e.g., gifted and talented, at-risk counseling, psychological services, Healthy Start, Limited English Proficient, etc.) In her current position, she presents the public position of the State in relation to educating individuals with disabilities, as well as working behind the scenes to influence public policy. She has worked extensively to involve the entire community of persons concerned with services to individuals with disabilities in planning for systemic change to improve outcomes for children and families.

Administrator (California Department of Developmental Disabilities). Richard Ingraham, M.S., is the current Manager of the Children and Family Services Branch of the California Department of Developmental Services. In this position, he has direct responsibility for California’s Early Start program, Part C of IDEA, as well as other programs and initiatives. Previously Mr. Ingraham served as the Department’s Chief of Health and Wellness and Community Program Specialist. His background includes service as clinical director, program director, and director of specialist services at various therapeutic facilities and service as a school psychologist. He is also an adjunct professor at the University of the Pacific and holds a Marriage and Family Therapist professional license.

SIG Designated Director: Coordinating the SIG2 and working directly with the operational subcontractor is Janet Canning, M.S., a Consultant in CDE’s Special Education Division. Ms. Canning has served in this role for both the original SIG and the SIG Supplement. She is additionally the Division’s lead staff on NCLB, the liaison to the California Commission on Teacher Credentialing, and the Coordinator of the Comprehensive System of Personnel Development. She established the Division’s website and trained the current webmaster. Prior to coming to California, she was a psychometrist, a speech pathologist, and the Assistant State Director for the Oklahoma Department of Education. Person Loading Chart showing time in days per objective by key State/contractor personnel:

  Administrators CDE Director CIHS Co-Manag CIHS Evaluator
Objective 1 10 20 110 40
Objective 2 5 15 30 50
Objective 3 8 25 35 35
Objective 4 5 20 20 15
Objective 5 5 20 25 55
Objective 6 2 15 10 40

d.3. Qualifications of project consultants or subcontractors

d.3.(a) Qualifications of organizational subcontractors. The primary sub-contractor is Sonoma State University (SSU) which assists CDE in SIG administration through the California Institute of Human Services (CIHS). Tony Apolloni, Ph.D., the current Vice-President for Research and Sponsored Programs at SSU, is the Director of CIHS. Dr. Apolloni’s Ph.D. is in special education with an emphasis on using community supports for student success. He originated and has directed a number of system change projects which have trained over 20,000 state staff, teachers, administrators, parents, agency representatives, and service providers in effective educational reform. CIHS has successfully administered over 100 federal, state and private grants and contracts with combined total budgets of $50+ million. The Partnership’s first strategic planning meeting prior to the conception of the original SIG was designed by the Department of Education in conjunction with CIHS personnel.

CIHS has considerable experience as an organization in operating national/regional grant training and technical assistance projects. The Institute was created at SSU in 1979 to act as a catalyst for applying research knowledge to the solution of "real world" problems. The CIHS mission is to:

During the 25 years of its existence, CIHS has successfully completed federal, state, and private contracts and grants totaling approximately $34 million, successfully implementing two or more major contracts ($100,000+) from each of the following agencies or organizations: U.S. Department of Health and Human Services—Administration on Developmental Disabilities, and Head Start Bureau; U.S. Department of Education—Office of Special Education and Rehabilitative Services, Office of Bilingual Education and Minority Language Affairs, and Office of Vocational and Adult Education; Hilton Foundation; California Wellness Foundation; California Department of Education—Special Education Division, Career-Vocational Education Division, and Youth and Adult Alternative Education Services Division; California Council on Developmental Disabilities; California Department of Corrections; California Department of Rehabilitation; California Consumer Services Agency; California Department of Health Services; and California Department of Justice.

The above projects have resulted in the completion of assessment systems, curricula, program analyses, service evaluations, technical assistance plans, and professional development activities related to the service needs of children and adults with barriers such as physical and cognitive disabilities, limited-English proficiency, incarceration, and poverty. A full statement of qualifications for the Institute appears in the Appendix.

The CIHS Co-Managers for the SIG are Linda Blong and Anne Davin, Ph.D. Linda Blong, Ph.D. in progress, is the current SIG Project Co-Manager, Co-Manager of CalSTAT, and Director of the California Health Incentive Improvement Project. She previously served as a project coordinator for the Indiana Parent Information Network. Ms. Blong also currently serves as an appointed member of the Youth and Family Practice Network of the Natinal Collaboration on Workforce and Disability/Youth and a Board member of United Cerebral Palsy of the North Bay. She was the recent keynote speaker for the Parent Success conference.

Anne Davin, Ph.D., is the current SIG Project Co-Manager, Co-Manager of CalSTAT, and Manager of the IDEA T/TA project. Previously, she served as the Division Director of the Taos Pueblo Health and Community Services, Program Director for University of California Santa Barbara professional psychology training programs, and Program Director for the Human Relations Center. She also served as legislative liaison to New Mexico State Senator Carlos R. Cisneros. A well-published author, her expertise has been particularly helpful in addressing the need for behavioral improvement.

The CIHS Internal Evaluator for the SIG is Cheryl “Li” Walter, Ph.D. Dr. Walter is the current full-time evaluator for the SIG. Previously she evaluated a number of different grants for CIHS and SSU, served as the Project Director for the Center on Social Services Research—Health Research Group and the Bay Area Social Services Consortium at the University of California Berkeley, and as the Project Consultant to the Medi-Cal Managed Care Project for the Alameda County Health Care Services Agency.

d.3.(b) Qualifications of consultants. As mentioned above, a variety of consultants will be contracted with to support the activities, the greatest number being involved with the provision of training and technical assistance (T/TA) in the targeted core message areas. In the first SIG, CIHS developed a significant consultant pool based on the existing pools for the Hilton/Early Head Start Training Project (nationwide) and the Head Start Quality Improvement Center for Disability Services (Region IX). This pool has been subsequently divided into primary and secondary consultants, the primary consultants being recognized experts in their field. Brief bios on just a few of the primary consultants are offered below.

Kevin Feldman, Ed.D., is currently the Director of Reading and Early Intervention for the Sonoma County Office of Education. He also serves as an Adjunct Professor in Special Education at Sonoma State University and has served as a private consultant for local as well as national clients, such as the Association for Supervision and Curriculum Development and the Council on Exceptional Children.

Jeff Sprague, Ph.D., is currently the Director of the University of Oregon’s Substance Abuse Prevention Program, the Co-Director of the University’s Institute on Violence and Destructive Behavior, and a Senior Research Associate for the University’s Specialized Training Program. He is widely published and is an Editorial Board Member for both the American Journal of Mental Retardation and the Journal of Applied Behavior Analysis.

Sheri A. Wilkins, Ph.D/ABD and Consultant for the Desert/Mountain SELPA evaluation study, is currently the Program Manager for the Desert/Mountain SELPA. Prior to this position, she served as a Program Specialist for the SELPA and an elementary and middle school teacher of children with mild to severe disabilities.

d.3.(c) Qualifications of PCSE members. California’s Partnership Committee on Special Education serves as the advisory body for the SIG. The chart below lists the membership in terms of required and optional partners by agency affiliation or individual status, and shows the years of their attendance at the annual strategic planning meetings. Many members sit on active PCSE committees working on issues such as post-secondary transition, alternative assessments, and evaluation. See the chart of organizations actively involved 1999-2003 in section c.5, p. 53-54.

(e) Adequacy of resources

The PCSE is requesting OSEP funding that will be matched generously in in-kind contributions from the project partners. These combined funds form an adequate pool to support project activities.

e.1. The adequacy of support, including facilities, equipment, supplies, other.

The California Department of Education will provide as an in-kind contribution the facilities, equipment, supplies, and other resources required for the in-kind staff working on the project. These staff include…

Each staff person will be supported with an office, office furniture, telephone, computer, printer or access to printers, and access to copying machines, scanners, fax machines, etc. Personnel will receive normal office supplies to support their activities, as well as access to the full range of Department services, e.g. printing and publications, contracting, legal services, etc.

The California Department of Developmental Disabilities is providing an in-kind .03 FTE grant administrator, Dr. Rick Ingram, and will support of this contributed staff person with space, equipment, and supplies.

The primary contractor, Sonoma State University California Institute on Human Services, will provide similar support to their in-kind and grant-funded staff, including…

CIHS maintains a central office on the campus at SSU in Rohnert Park, California, approximately 1.5 hours northwest of San Francisco, and an off-campus office several blocks away. The Institute has a separate office at the California State University at Channel Islands to handle Southern California business.

The off-campus office, where the SIG staff are housed, is a self-contained facility under a 10-year lease agreement. The building has individual offices, meeting rooms, a production room, and staff facilities. Even though the office is off-campus, the project still has access to the lowcost University printing service, the large library facility and holdings, multiple meeting rooms, and other University accommodations. Both space and other resources are ample to support this project. A total of 1200 sq. ft. of office space and pro-rata allocated general use space houses the SIG program currently and will accommodate the projected personnel for the new program.

Grant funding is requested for normal project-related office supplies, activity-specific project supplies. No new computers or peripherals are required. All office furniture and other equipment, including telephones, additional printers, scanners, fax machines, copying machines, etc., will be provided as an in-kind contribution by Sonoma State University.

e.2. The relevance/commitment of each partner to project implementation and success.

Each partner has demonstrated commitment to the Partnership and the success of the SIG. The statements of these commitments are briefly summarized in this section and found in entirety in Appendix D.

The MOU with the required contractual partners indicates that all partners will…

  1. Serve on the California Partnership Committee for Special Education (PCSE) and appoint representative(s) to appropriate Committee ad hoc task forces, which may be established;
  2. Work with the PCSE members to provide oversight for the SIG, reviewing progress annually and making data-based recommendations for priorities and strategic approaches for the following year;
  3. Inform staff, colleagues, and families of the work of the PCSE and SIG strategies when appropriate; and
  4. Participate in joint training activities, including staff and families, when appropriate.

In addition to these functions, the Special Education Local Plan Area (SELPA) Organization and Special Education Administrators of County Offices (SEACO), the local education agency (LEA) partners, additionally agree to serve as a sounding board for LEAs throughout the State, gathering input on the relative effectiveness of SIG strategies in improving outcomes for individuals with disabilities, discussing issues relative to this goal, and providing considered recommendations from the field to the PCSE. The LEA partners also agree to work with the California Department of Education and appropriate other partners specifically to implement the activities related to the SIG’s 6 objectives.

The primary contractor, The California Institute on Human Services at Sonoma State University, additionally agrees to work with the California Department of Education (CDE) and appropriate other partners specifically to serve as the administrative and operational agency for the SIG, if funded. A sub-grant to CIHS from the State Department of Education will provide SIG funding for these activities.

Letters of commitment from each member of the Partnership feature the following agreements: The partner agrees to…

Beyond the significant contribution of in-kind staff, an additional major commitment from the Department of Education is that the State has charged no indirect fees for the project. Because the authorized rate is 24% of direct costs, this is a considerable contribution and one that speaks to the acceptance of the SIG as a key ingredient in the effort to improve outcomes for students with disabilities and an integral, essential mechanism for systems change.

These commitments are clearly relevant to the success of the SIG and to the goal of improving results for individuals with disabilities. They also represent a massive commitment of funds to help achieve this goal.

e.3. Adequacy of the budget to support the project.

Both the Department of Education and Sonoma State University have had extensive experience in planning and executing large scale training, technical assistance, and reform programs, including the original SIG and the SIG supplement. This experience is key in developing a budget that will adequately support the project activities.

Essential to fiscal adequacy are the literally millions of dollars of in-kind resources and services provided by the Department and the partners. Thousands of hours of pro-bono services are provided by the PCSE, augmenting the in-kind contributions enumerated above from the contractual partners and sub-contractor. Without these contributions, the budget would certainly be inadequate to support the scope of activities proposed. The partners’ major contributions, therefore, highlight two important Partnership attitudes:

The Partnership’s experience with the original SIG has proven these assumptions true.

e.4. Reasonable costs in relation to project objectives, design, and potential significance.

Reasonableness of cost for educational outcomes is simply not as easy to determine as price tags for military coffee pots or toilet seats. What is “reasonable” to spend to improve outcomes for individuals with disabilities is probably not an issue upon which Americans would find easy consensus. However, there are indicators of cost reasonableness in California’s approach that demonstrate the appropriateness of the federal investment in relation to return:

e.5. Continued support of the project after Federal funding ends.

These are difficult financial times for California, and the status of funding over the next few years will be decided by voters just days after submission of this application. Therefore, at this point, it’s difficult for the State Department of Education to commit to financial support of project elements in the future. However, the intent to do so is clear and bolstered by precedent. For example, four projects currently funded by the SIG will be transferred to state funding through the CalSTAT project: the Pre-Intern programs; the Institutions of Higher Education Task Force on the SE Workforce; the African-American Family Outreach project; the Strategic Planning on Teacher Recruitment, Retention, and Preparation; and the Regional Coordinating Council Core Message training program.

Additionally, the partners have committed to significant post-project support by taking on the institutionalization of many of the activities, such as the middle and high school reading interventions. The SIG will provide site-to-site TA and other support, but the schools will provide the personnel for training, purchase the materials, and support the programs with instructors, aides, etc.

(f) Quality of the management plan

f.1. Management plan to achieve the project objectives on time and within budget.

The SIG2 project management structure is based on the experience of project management for the initial SIG and the Supplement. The structure incorporates the intent of the PCSE to reflect and model interagency collaboration in the project management and operations, as well as in the activities. This intent to model interagency collaboration has been well captured in the management structure in the following ways:

Creating a functional management plan within these parameters has required some adjustment, but the structure outlined below has been functioning well for the past several years.

Project administration: The project will be administered by Director-level personnel from the Department of Education (both SE and GE divisions, representing Part B) and the Department of Developmental Services (representing Part C). As noted below, the role of the administrators is to attend the annual strategic planning meeting of the Partnership, meet with the CDE Designated Project Director and key contractor staff on a quarterly basis, and examine annual outcomes to ascertain the extent to which the agency mission is being met by the project and to determine ways in which the common missions can be fulfilled through re-direction of resources. In short, the administration provides annual oversight for the project as a whole but at least quarterly oversight for their internal agency responsibilities. Administrators will also review data provided by the project prior to the PCSE’s SIG Evaluation Task Force meetings in March and October; this review will be followed by a conference call between the administrators to discuss any concerns and/or endorse any changes in direction recommended.

The rationale for this approach is to provide overall, visible project leadership from the division/department level for the Parts B and C lead agencies, while allowing their committed time to be focused on “enlightened self interest,” that is the operation of the project—and the benefits accrued—in relation to their own department/division. The Partnership felt this level of management was needed to communicate the importance of the project as systemic change and to have the “bully pulpit” for both organizations available for use, if needed.

Project Directorship: The Project will be directed by Janet Canning, a California Department of Education, SE Division consultant. As described again below, she will devote .5 FTE to the direction of the project, including the following duties: providing daily oversight of CIHS as the primary sub-contractor, attending the annual strategic planning meeting of the Partnership, attending the two annual meetings of the PCSE SIG Evaluation Task Force, and examining annual outcomes to ascertain ways from the agency perspective to improve services through re-direction of resources and re-focusing of Grant funds. She is specifically responsible to ensure that activities are conducted in a timely and comprehensive manner to generate quality services that will generate the projected outcomes.

Project Management: CIHS employs two .5 FTE SIG Co-Managers. These same staff comanage CalSTAT, the CDE SE Division’s State-funded TA program, and this fact ensures very close coordination of services and maximum utilization of resources. The specific job duties of the co-managers are to…

The Co-Managers report directly to the SSU Vice-President for Sponsored Programs and Research, who is also the Director of CIHS, Dr. Tony Apolloni. Dr. Apolloni meets bi-monthly with the Co-Managers to ensure that the project is on target in terms of budget and deliverables, that the interim data shows reasonable progress toward meeting the objectives, and that the staff has the resources and support necessary to continue to produce quality goods and services.

Project Timeline: This timeline is designed to provide a summary of action steps required in each activity area in year one, not a daily management plan, and completion quarters relate to a projected start up of February 2005. Years 2 and 3 will have a similar pattern without the startup activities.

Activity/ Action Step Q1 Q2 Q3 Q4
Centrally coordinated TA        
Update and expand existing 500+ member field-based consultant pool x x    
Link new annual database for TA delivery to consultant database x      
Process consultant applications on request of TA requestors and others x x x x
Publicize widely the availability of SIG TA and how to access x x x x
Process TA requests according to SIG “triage” guidelines; allocate days x x x x
Maintain TED evaluation database on central TA and monitor progress x x x x
Analyze evaluation data and develop recommendations for improvement x x x x
Site-to-site TA        
Establish protocols for site-to-site TA in model site contracts x      
Train model sites in research-based TA delivery methodology x x    
Oversee implementation/conduct of site-to-site TA x x x x
Monitor provision of TA in model core message area x x x x
Analyze evaluation data and develop recommendations for improvement x x x x
California BEST TA        
Create contract with IVDB to provide evaluation and cadre support x      
Establish cadre teams as liaisons to all existing BEST-trained sites x      
Provide on-going support for cadre teams through IVDB contract x x x x
Monitor provision of BEST TA and evaluation x x x x
Monitor implementation of new BEST sites x x x x
Analyze evaluation data and develop recommendations for improvement x x x x
Regional Leadership Institutes TA        
Revise RLI subcontractor guidelines and conduct outreach x      
Identify sites and complete contracts x      
Monitor RLI planned agendas and focus on core message areas x x x x
Maintain web site with RLI information for sites to share x x x x
Analyze evaluation data and develop recommendations for improvement x x x x
TED Dissemination TA        
Finalize testing of all TED’s local features x      
Do demonstrations at conferences x x x x
Conduct other outreach for pilot sites, choose sites x      
Train pilot sites in basic use and report function   x    
Provide on-going TA support   x x x
Train pilots in joint use of summary reports and student outcomes data       x
Analyze evaluation data and develop recommendations for improvement x x x x
Model site programs        
Develop model site application, processes based on core messages x      
Print and post on-line application and directions for submission. x      
Conduct outreach soliciting applications through at least 5 methods x      
Develop scoring rubric, train readers, and conduct application reading x      
Conduct site visits of semi-finalists; select final sites; secure agreements   x    
Work with each site to develop web-based, interactive materials   x x x
Develop TA plans for each site focused on core message areas, etc.   x x x
Ensure full participation of all sites in statewide institute community   x x x
Monitor model site work in development/delivery of regional institutes   x x x
Analyze evaluation data and develop recommendations for improvement x x x x
Statewide Leadership Institute        
Revise facilitation team information packets; contract with facilitators x      
Revise Institute framework, outline agendas, establish roundtables x      
Coordinate outreach, registration, scholarships, and confirmation   x    
Develop and edit meeting materials and packets; post online   x    
Conduct wrap-around conferences, on-the-fly conferences, listserv   x x x
Conduct the 3-day conference, collecting all evaluation data     x  
Analyze evaluation data and develop recommendations for improvement       x
Family Participation Fund        
Contract with FPF sub-contractor detailing outreach, reimbursement, etc. x      
Monitor marketing through geographic and ethnic/racial use patterns x x x x
Conduct assessments, including sample interviews of effect x x x x
Monitor allocation of stipends to SE and GE committee participation x x x x
Analyze evaluation data and develop recommendations for improvement x x x x
Administer the program; support PCSE annual meeting and committees        
Review and revise as needed all protocols and procedures for SIG2 x      
Attend monthly CDE-CIHS meetings to monitor progress, problem-solve x x x x
Update PCSE database, finalize annual invite list, send out date cards   x    
Develop meeting agenda; mail out meeting packets; register participants   x x  
Plan and monitor all meeting logistics; prepare all materials   x x  
Provide full staff support for annual PCSE strategic planning meeting     x  
Transcribe meeting recommendations; update website       x
Analyze evaluation data and develop recommendations for improvement       x
Conduct SIG2 evaluation—See steps in Part g. Evaluation        

f.2. Ensuring a diversity of perspectives are brought to bear in project operations.

A diversity of perspectives permeates the project at many levels from the site planning teams which include parents, administrators, special/general ed teachers, and others to the Administrative Team with CDE general/special education directors and the Department of Developmental Services Director. The project is overseen by the broad-based Partnership, PCSE; see membership roster in section c.5, p. 53-54.

(g) Quality of the project evaluation

g.1. Thorough, feasible, appropriate methods to evaluate goals, objectives, and outcomes.

Evaluation structure: The evaluation will provide thorough, appropriate formative and summative assessments of service quantity and quality and the impact of these services on the intended audiences as described below.

Formative evaluation: The formative evaluation sessions will be held with the PCSE SIG Evaluation Task Force in March and October of each year. The timing of these sessions was designed to provide findings and recommendations for the Partnership's annual strategic planning session. CIHS handles the scheduling and logistics; attendees include the Task Force members, CDE Designated Project Director, Co-Managers, Evaluator, and other key staff. CIHS, working with the Department of Education, will supply relevant data available to date.

Formative evaluations will target the following continuous improvement issues: What is the number, nature, and quality of project activities actually implemented to date? What aspects need improvement and/or change? What approaches are working very well and might warrant expansion? What problems are anticipated in implementing the next phase, and how will these be overcome? Participants will discuss the available data on process objectives and outcomes, and develop recommendations for improvement that can be implemented immediately. The process thus creates a feed-back loop which repeats and generates continuous improvement.

Summative evaluation: The summative evaluation will establish the degree to which the objectives and outcomes have been met, including the quality of process and products, and the continued significance of the Project's work after the award period. The summative evaluation will be conducted annually and will contribute to the Partnership's annual strategic planning session. The annual summative evaluations will be forwarded to OSEP.

Evaluation management: The existing network of evaluation personnel, resources, and PCSE Evaluation Task Force, which have conducted/guided the evaluation for the initial SIG and supplement, will continue in SIG2.

SIG Evaluation Task Force: To ensure the ongoing involvement of stakeholders in monitoring progress toward the objectives, a SIG Evaluation Task Force meets twice a year. Comprised of a broad range of stakeholders, this 15-member group gives feedback on instruments and evaluation methods as they are developed, reviews data and progress reports, and makes recommendations based on the findings, which are presented to the PCSE annually.

Partnership Committee on Special Education (PCSE). The SIG Evaluation Task Force recommendations are considered, adjusted as needed, and accepted by the PCSE prior to recommendation to CDE Special Education Division. This process takes place annually.

SIG Evaluator. The SIG evaluation effort is being directed by CIHS employee, Cheryl "Li" Walter, Ph.D. Her role is to gather information and draft evaluation plans; develop evaluation instruments and methods, as needed; facilitate meetings of the SIG Evaluation Task Force and its subcommittees; draft progress reports and develop other avenues for presenting the findings; and design program evaluations for the activities of the grant subcontractors.

Additional key personnel: The SIG Evaluator works closely with Janet Canning, State Designated SIG Director from the CDE Special Education Division, and with the SIG Co- Managers. Kelly Riedel acts as the internal evaluation staff, managing the gathering and entry of data and generating the SIG activity reports; her work is supervised by the SIG Evaluator. This structure was established in order to address the SIG goal of building the capacity of the system to generate and use data to inform decision-making processes.

Evaluation design and methodology: Outlined below are the specific outcomes, objectives, and indicators developed for SIG2, along with the evaluation methodologies planned to assess progress toward meeting those outcomes/objectives.

Outcome 1: Improved Quality of Personnel Working with Students with Disabilities

Objective 1. To increase the quality of personnel through the provision of core message, research-based, in-service professional development through technical assistance (TA) and Regional Institutes, as demonstrated by:

Objective 1 Evaluation Methodology: Assessments will be conducted using end-of-event and follow-up evaluation questionnaires with all TA and Regional Institute participants.

1.1. End-of-event evaluation questionnaires. Increases in knowledge and overall ratings of TA events will be measured using an end-of-event questionnaire to be filled out by all participants; see a sample in Appendix C.1. Using a 5-point Likert scale from "Low" to "High," participants are asked to rate their level of useable skills/knowledge of the TA topic prior to the TA and after and assign an overall rating to the TA, among other items. Completed end-of-event evaluations are collected and submitted by TA providers along with the event invoice and a cover sheet specifying the core message areas addressed in the TA. Over the past four years, 72% of TA participants have completed evaluations.

Information on the core message addressed in each TA session and the data from the completed evaluations are entered into the SIG's searchable database. The database is capable of generating automated reports by event, trainer, core message area, and time period. Nearly 40,000 TA participants have completed end-of-event evaluation forms, thus establishing expected patterns of skill/knowledge gains and overall event ratings. Throughout the year, program staff can monitor the quality ratings for individual events and trainers, the TA being offered in specific core message areas, and the TA provided in given timeframes by comparing the targeted data to these norms.

Summary reports are generated annually showing the average of TA and Regional Institute participant ratings of increased knowledge/skills, quality of training, and intent for future action. The rating difference between prior skills/knowledge divided by the 5 points of the Likert scale generates the percent change, e.g., a 0.8 point gain in skills/knowledge represents a 16% increase. Aggregated levels of skills/knowledge gains are included in annual reports. Knowledge gains and overall ratings are generated for each core message area for use by SIG staff and the PCSE Evaluation Task Force in monitoring the quality of the TA and making adjustments through daily management and the formative evaluation process to ensure continuous improvement.

Follow-up Evaluations. Follow up evaluations are used to measure the degree of implementation and use of the knowledge/skills targeted in the TA and Regional Institutes and the degree to which the participants shared the core message knowledge/skills with other appropriate individuals or groups, e.g., parents, GE and/or SE educators, support personnel, health/social service agencies, etc. The follow up evaluation is conducted electronically using a follow-up evaluation questionnaire; see a sample in Appendix C.2.

The follow up evaluation questionnaire is sent to all participants ~3 months after the event. Participants receive an email that states the event title and date and requests their cooperation in completing the brief form. A link in the email takes participants to a website where they are asked to rate their degree of implementation and use of the knowledge/skills taught and the degree of sharing of their knowledge/skills from the TA on a 5-point Likert scale from "Not at All" to "Many Times."

Participant email addresses are obtained from event sign-in sheets and entered into the SIG database. Participants receive up to three reminder emails to complete the follow-up evaluation. Their responses are reported only in the aggregate, never with a link to the participant. In the past year, valid, functional email addresses were reported by ~31% of participants. Of those, the follow-up response rate was 35%. This amounted to approximately 11% of the T/TA participants allowing for a 95% confidence level of +or - 2.24%. Similar tests of confidence level and confidence interval will be conducted to ensure that future response rates are sufficient to generate valid findings.

When data is entered on the website, it is automatically added to the SIG database which has the capability of generating automated reports by event, trainer, core message area, and summaries for periods of time. Throughout the year, program staff use these reports to monitor California State Improvement Grant, p. 81 the degree of learning in the specific core message areas and the degree to which the learning in specific core message areas is being implemented and further disseminated through sharing. The data-to-date is also reviewed by the PCSE SIG Task Force in the formative evaluation.

At the end of each year, summary reports are generated showing the average TA and Regional Institute participant ratings of degree of implementation/use and sharing of their knowledge/skills. Responses of 4 or 5 on the 5-point Likert scale are recorded as the participant having repeatedly implemented or shared the knowledge/skills. The percent of participants at follow-up reporting having repeatedly implemented or shared their knowledge/skills is calculated and included in annual reports.

Outcome 2: Improved Educational Service Coordination for Students with Disabilities

Objective 2. To improve collaboration between special education (SE) and general education (GE) by:

Objective 2 Evaluation Methodology: Assessments will be conducted using the Collaborative Sites Survey with all sites receiving collaborative Site-to-Site TA or participating the Core Message Institutes. In addition, an in-depth research project focusing on the relationship between collaborative service delivery models and student outcomes will be conducted.

2.1. Collaborative Sites Survey. Increases in degree of collaboration in the areas of assessment, intervention, presenting the core curriculum, and teaming will be measured using a Collaborative Sites Survey. This survey is completed by the contact person at each of the California State Improvement Grant, p. 82 Collaboration Learning Sites and at all school/district sites receiving 3 or more days of Site-to- Site TA on Collaboration.

Site contacts are asked to rate the degree of collaboration at their site prior to receiving TA and after the TA on a 10-point Likert scale from "Totally Separate" to "Fully Collaborative" for each area of interaction in which collaboration might be taking place. In addition there are six open-ended questions inquiring about specific changes in the service delivery system, challenges faced in collaboration, factors that have helped facilitate the change process, results seen since adopting the changes, and future plans for sustaining or expanding the move toward collaboration. See Attachment C.3 for a sample of the survey.

The areas of collaboration examined in this survey were identified from two sources:

  1. The common strategies identified in the Schwab Learning-supported study of 16 collaborative challenge sites in California from 1999 to 2001. This study was the foundation for what has become the Collaborative Institute Learning Community model. In turn, that model has been expanded to other core message areas targeted in this application.
  2. A Collaborative Practices Client Technical Assistance Assessment Tool developed by collaborative consultant Dr. Paul Porter for use by sites in identifying areas of strength and weakness around collaboration.

    A survey pilot in 2003 examined sites in California that had received collaborative TA over the past four years. Of the 100 sites contacted that had received TA, nearly half completed the pilot survey. The results are reported in section b.2.(b), pp. 23-37.

The Collaborative Sites Survey will be conducted in the Fall of 2006 and will include all Collaboration Learning Sites and other district/schools that have received 3 or more days of TA California State Improvement Grant, p. 83 on collaboration and have been involved in these SIG activities for at least one year. Thus the time involvement of each site will vary, and this data will also be collected and considered in the outcomes evaluation.

Site contacts will receive emails announcing that the Collaborative Sites Survey is being sent and reminding sites of their commitment to complete the Survey and its purpose. Sites not returning completed surveys by the initial request date will receive reminder emails and a new deadline. Sites not responding by the deadline will receive phone calls and an offer to complete the survey through a phone interview. This methodology has been successful in the past in generating a relatively high rate of return.

Analysis of the quantitative data will focus on comparing the average degree of collaboration "Prior to TA" and "Now" for each of the areas of collaboration as reported by the sites to ascertain what degree of change occurred, if any. In addition, trend analysis will be used to look at the relationship between change in the degree of collaboration and the amount of TA received. Analysis of the qualitative data will focus on identifying the types of changes that were made in the service delivery systems of the sites, unique experiences, and common themes among the challenges faced, facilitating factors, results, and future plans reported.

The evaluator will develop a brief report that describes the quantitative and qualitative findings in both narrative format and through graphs easily understandable by lay persons. This report will be shared with SIG managers and TA coordinators to foster discussion of how the findings could be used in assessing needs and planning future activities. In addition, the findings will be shared with site teams at the Statewide Institute and with all survey participants, along with suggestions on how they might use the findings as a tool to initiate discussion of their site's process of moving toward collaboration. The report will also be used by the PCSE at their annual California State Improvement Grant, p. 84 strategic planning meeting. Finally, the report will be posted on the SIG website to afford easy access and sharing with the broader community.

2.2. Desert/Mountain SELPA Research Project. Collaboration between SE and GE is related in the research and literature to positive outcomes in other SIG objectives, specifically objectives 2--service coordination, 3--student achievement, and 6--use of data. To examine the relationship and bridge these objectives, the SIG plans an in-depth research project looking at the effects of a collaborative service model on student achievement.

The project will be conducted by the SIG Evaluator, in partnership with CalSTAT and the Desert/Mountain (D/M) SELPA. The D/M SELPA has been involved in the Statewide Collaborative Leadership Institute for several years and has sponsored 3 Regional Collaborative Institutes in the past 3 years. Several schools within the SELPA have received additional TA from the initial SIG and ongoing professional development and support from the D/M SELPA.

The Desert/Mountain SELPA is a consortium of 15 school districts, 5 charter schools, and the San Bernardino County Superintendent of Schools. Currently there are 2 collaborative model sites (Cottonwood Elementary and Big Bear Middle School) and 1 collaborative model district (Hesperia Unified School District) within the D/M SELPA region. The research project will allow the D/M SELPA to conduct an investigation of the changes in service delivery for students with disabilities and how the service delivery model relates to student achievement. The research project will be a 3-year undertaking, with the first year focusing on the effects of a collaborative service delivery model at the elementary level and subsequent years looking at the effects of a collaborative service delivery model at the middle school level.

The first year of the research project will be a 2-tiered endeavor, with one tier focusing on school-wide student achievement and the other on the academic achievement of students with California State Improvement Grant, p. 85 disabilities. The research will focus on the 51 elementary schools within the SELPA region. Schools will be divided into 2 groups: those schools operating a collaborative service delivery model and those operating a traditional, pull-out service delivery model.

Student achievement data from the 2001 California Standards Test (CST) in English/Language Arts will be compared with CST results from 2004 to determine whether a collaborative service delivery model is related to student achievement in the general curriculum. These data will be analyzed using chi square analysis, multiple regression, and analysis of variance (ANOVA).

The research will also target related areas. Data will be analyzed to determine whether a collaborative service delivery model is related to number of referrals to and/or placements in SE. In addition, the project will examine the amount of collaboration TA received by the site in relation to student achievement. Data will also be collected on the basic reading program used, as well as the intervention program(s) in effect during the study period, to analyze the relationship between reading curriculum and student achievement. On an individual student level, CST data will be analyzed to determine whether increased time in the GE classroom is related to improved student achievement for students with mild disabilities.

During the second year, the research will extend to middle and junior high schools in the D/M SELPA region, using the same type of measures: the relationship between collaborative service delivery model and student achievement, as measured by the CST E/LA; the relationship between amount of TA and student achievement; the relationship of the type of model and the number of referrals to and placements in SE; and the relationship between the reading curriculum and E/LA achievement. As in year 1, all data will be examined in relation to both school-wide E/LA achievement and E/LA achievement for students with disabilities.

During the third year, the research project will target schools in the D/M SELPA, K-8th grade and wrap up the analyses from the previous years by analyzing the data in an attempt to answer the question, "What kind of collaboration and what level of TA/support leads to improved student achievement at the elementary/middle school level?" Staff will also develop profiles of schools that have experienced systemic change and research which specific components of TA/support are most successful. The hope is that this model can then be replicated in other SELPAs and districts across the State.

This project will specifically analyze the effects of a collaborative service delivery model but secondary questions will deal with the effects of TA and ongoing support from the SIG and the SELPA, as well as literacy instruction and professional development at the elementary and middle school levels. The findings will then be widely disseminated through the SIG channels.

Outcome 3: Improved Academic Outcomes for Students with Disabilities

Objective 3. To increase the academic performance of students with disabilities as demonstrated by:

Objective 3 Evaluation Methodology: Assessments will be conducted using California's CST and API data reports for the Core Message Model and Learning sites and all school/district sites meeting the requirements stated in the outcome measures.

3.1. California Standards Test (CST) data reports. Increases in reading proficiency will be measured based on data reports of student performance on the CST E/LA. California's students are tested annually in the spring, with test scores by school and district available the following fall. As previously discussed, district, school, and student sub-group scores for reading proficiency are reported by percent in five categories: Advanced, Proficient, Basic, Below Basic, and Far Below Basic.

CST data is available from the CDE website CST Research Files. Where site level numbers of students with disabilities fall under the threshold for reporting on the website, data will be obtained either directly from the school or purchased from a specialized consulting firm. The evaluator will construct and regularly up-date a relational database, by site, for the CST school-wide and sub-group data, as well as the number of reading TA days received.

The percent of students scoring Proficient or Advanced and the percent of students scoring Far Below Basic will be proportionally averaged for all grades at each school/district in the database for the baseline year 2004 scores. Each year when CST scores become available, summary graphs will be generated comparing the baseline percentages to the current year percentages for each site and for the average of all sites. Progress toward the target percentage point increases and decreases will be highlighted. Site level data will be shared with each site and the SIG managers, while the project wide data will be shared with the Evaluation Task Force, at Institute meetings, through the CalSTAT website, at the PCSE annual strategic planning session, and in annual reports. A final report will provide results comparing the baseline year to the final year to produce the grant project period outcomes.

2.2 Academic Performance Indicator (API) Data Reports. Increases in overall academic literacy will be measured through API scores. The API score is a single number California State Improvement Grant, p. 89 between 200 and 1000 that synthesizes weighted CST and CAT 6 reading and math scores, along with High School Exit Exam results for secondary schools; this number is then translated to a decile rating of 1-10. Schools are considered to be performing adequately at 800+ for schoolwide and sub-groups; however, less than 5% of middle/high school sites currently score in that range. Schools also receive a Similar Schools Ranking on a scale of 1-10 in comparison with 100 other schools in California with similar parent, student, and teacher demographic characteristics.

API data will be obtained from the California Department of Education's website API Research Files. An up-dated database of the schools/districts participating as Core Message Model and Learning sites and sites receiving Site-To Site TA will include the amount of TA received, the API scores for all students and students with disabilities as a subgroup, and API scores for Similar Schools rankings. Each year when API scores become available, summary graphs will be generated comparing the baseline API and Similar Schools rankings to the current year APIs for each site, school-wide and for the disabilities sub-group. Site level data will be shared with each site and the SIG Managers, while the project wide data will be shared with the Evaluation Task Force, at Institute meetings, through the CalSTAT website, with the PCSE in the strategic planning meeting, and in annual reports. A final grant report will provide results comparing the baseline year to the final year to produce the grant project period outcomes.

Outcome 4: Improved Behavioral Supports and Outcomes for Students with Disabilities

Objective 4. To improve behavior supports and outcomes for students with disabilities, as demonstrated by:

Objective 4 Evaluation Methodology: Assessments will be conducted using a variety of BEST assessment tools by site teams and BEST cadre trainers with the assistance of the IVBD.

4.1. Combined Self-Assessment Survey and the System-wide Evaluation Tool (SET). Increases in the number of schools that have fully implemented BEST will be measured using the BEST Combined Self-Assessment Survey and the System-wide Evaluation Tool (SET), which assess intervention fidelity; see the sample in Appendix C.4 and C.5. Completed twice in the start-up year and annually thereafter, SET consists of 2 distinct tools to be completed by all school staff at each site: the Assessing Behavioral Support in the Schools Survey (EBS--Sugai et al, 1998) and the Oregon School Safety Survey (OSS--Sprague et al, 1999). This data is sent to the University of Oregon Institute on Violence and Destructive Behavior (IVDB) for analysis. A BEST Cadre member completes the SET annually, making observations and conducting interviews at the school sites to determine the level of program implementation and then forwarding the data to the IVDB for analysis. An implementation level of 80% or above on all measures is considered to indicate high intervention fidelity; IVDB expects sites to take at least two years to achieve this level.

IVDB analyzes and summarizes the data in graphs with text which are provided to BEST Cadre trainers to present to the school-level BEST Teams for discussion and use in refining their BEST implementation. In addition, IVDB summarizes information for all the sites and provides annual reports on baseline data and progress to SIG, specifically identifying the number of school sites that have fully implemented BEST. This information will be share widely with the PCSE and the field through the SIG annual report and on the website.

4.2. School Profile Reports. Decreases in office discipline referrals and discipline suspensions will be measured through School Profile Reports (SPR) submitted annually by each BEST site team; see the sample in Appendix C.6. In SPR, the schools provide demographic attendance, grades, and achievement score data, along with discipline office referral and suspension data for the previous year. The SPR is given to the site's BEST Cadre trainers who forward it to IVDB for analysis. The analysis, data sharing for continuous improvement, and dissemination of findings then proceeds as described above for SET.

BEST is a school-wide system change approach and so the focus of the objectives has been on improvements in behavioral supports and outcomes for all students. Nevertheless, there is a keen interest in ensuring that students with disabilities are directly benefiting from this change, beyond the improved school atmosphere and opportunity of the school's students and teachers to focus on learning. Data on discipline suspensions specific to student with disabilities at the school site level has recently begun to be collected through the California Special Education Management Information System (CASEMIS). By the end of the year, site level data on suspensions specific to students with disabilities will be available. Because of the small group size, this data will need to be totaled for all schools that have fully implemented BEST, with comparisons then drawn between baseline and end of the grant outcomes.

Outcome 5: Improved Participation of Parents/Family Members of Students with Disabilities

Objective 5. To increase the participation of parents/family members of children with disabilities in the systems change process by:

Objective 5 Evaluation Methodology: Assessments will be conducted using event registration forms and sign-in sheets, CAC progress reports, Family Participation Fund invoices, End-of-Event Evaluations, and follow-up phone interviews of parents/family members.

5.1. Event Registration Forms and Sign-In Sheets. The project will measure increases in the participation of parents/family members at TA and Regional Institute events and as site team members at Statewide Leadership Institute events using event sign-in sheets and registration forms which ask participants to specify their role; see a sample sign-in sheet in Appendix C.7. This data is entered into a SIG database, along with event invoices and evaluations. The database is capable of generating automated reports on the number/percent of parent/family member participation by event and by core message area, and by time period. Throughout the year, staff can access the database to monitor parent participation in individual events, Regional Institutes, and the TA being offered in specific core message areas. At the end of each year, summary reports will show pie charts of participating in TA and Regional Institutes by role and the percent of Statewide Institute teams with parent members. This data will be shared with the PCSE, posted on the website, and included in the SIG annual report.

5.2. Community Advisory Committee (CAC) Project Progress Reports. Increases in the active participation of low income parents/family members in Community Advisory Councils (CACs) in high poverty areas will be measured through semi-annual progress reports submitted by the California Association of Family Empowerment Centers (CAFEC). CAFEC administers the Family Participation Fund, linking potential low-income participants in 5 highpoverty CACs with training at the Family Empowerment Centers and the resources of the Family Participation Fund. Semi-annual progress reports on the CAC Project will provide details on the composition of the CACs in terms of the number of members, the degree to which they reflect the demographics of their community, support services provided to both the CAC and recruited parent representatives, and additional needs for future support. These reports will be shared and disseminated as discussed above.

5.3. Family Participation Fund (FPF) Invoices/End-of-Event Evaluations. Increases in the participation of low-income parents/family members in non-special education decision-making committees and in parents/family members reporting that their participation has made a difference will be measured using the FPF Invoice/End-of-Event forms. Parents/family members submit these forms following participation in a committee meeting in order to receive reimbursement; see a sample in Appendix C.8. The form inquires about income level and ethnicity to enable monitoring for receipt by the intended population of low-income, underrepresented minority parents/family members. The new forms will add questions about the nature of the committee (SE or GE) and ask participants to rate their effect through their presence and contributions on a 5-point Likert scale from "Not At All" to "Many Times." Responses of 4 or 5 will be counted as their having made a difference. The percent of participants having made a difference will be calculated and cited in annual reports.

Annually, FPF end-of-event evaluations will be forwarded from the FPF to SIG for data entry and summarizing into tables and graphs that will be shared with FPF staff and SIG managers to monitor utilization and effect of the funds. These data will be summarized and included in the SIG2 annual reports, at the annual strategic planning, and on the website.

5.4. Follow-up Phone Interviews. To generate some in-depth data on degree of difference parent/family members are making through their participation and how they might be better supported in enhancing their effectiveness, follow-up phone interviews will be conducted annually with selected FPF support recipients. Structured discussions focusing on the defined topic areas outlined above will be conducted by an experienced qualitative phone interviewer with a purposeful sample of up to 50 FPF recipients--including at least 15 each of FPF recipients who attended SE meetings and those who attended GE meetings--until the point of saturation/redundancy is reached. Each interview will last for approximately 30 minutes.

Phone interviewees will be recruited through a separate Consent Form attached to the end-of event evaluation. The form will discuss the questions to be asked, the time commitment involved, the nature of the sampling process, the $50 stipend for participation, and use of an outside consultant as interviewer. Those willing to be interviewed will submit the Consent Form along with their invoice and end-of-event evaluation. Identifying information will not in any way be connected with their interviewee responses in the recording, use, or reporting of the data. Prior to this phone interview process being conducted, the study protocol will be reviewed and approved by the Institutional Review Board at Sonoma State University.

The qualitative data obtained through the phone interviews will be analyzed for themes and unique experiences, specific kinds of effects parents report having had through their participation, and needs for additional training/support. These findings will be reported along with illustrative quotes from interviewees and recommendations. This report will be shared with the Association administering the FPF as well as CalSTAT managers to foster data-informed planning and more broadly to the PCSE, in the annual SIG2 report, and on the website.

Outcome 6: Improved Data Collection/Use of Data in Addressing the Needs of Students with Disabilities

Objective 6. To improve data collection and the use of data by local educational agencies in identifying and addressing the need for T/TA to support the continuous improvement of outcomes for students with disabilities by:

Objective 6: Evaluation Methodology. Increases in the number of SELPAs and COEs using TED to monitor data on their TA will be measured by the number of sites that complete the TED training, have TED installed at their site, enter TA data into TED, and submit their data to SIG electronically via TED. Sites that use TED data, along with school/district student outcomes, to directly inform decisions on future TA offerings to the continuous improvement of the outcomes of students with disabilities in their service areas will be identified and monitored.

The idea for TED emerged from a sustainability planning process carried out collaboratively by the SIG Evaluator, SIG Managers, CDE Designated Director/Contract Monitor, and PCSE SIG Evaluation Task Force in the final year of the initial SIG. The rationale was as follows:

TED fills the need of SELPAs, COEs, and districts to determine how much TA is being provided in each core message area, who is attending by role and school site, what evidence exists of change in the core message areas targeted, at what level of TA concentration does change begin/optimize, and who must be involved in the TA for maximum effect? Data-based answers to these questions will allow effective targeting of TA to areas of need (e.g., core message areas, districts/schools, personnel/parent roles); better choice of which TA to scale-up; clearer communication with GE about what works; and stronger applications for enhanced professional development funding.

It was concluded that the SIG electronic evaluation system and database developed as a management tool could be modified for broader application, making it a cost-effective instrument for tracking, and eventually evaluating, the use of SE-funded TA in the State as well as at the local level. Broadening efforts in this area is consistent with the State's intent of the SIG evaluation, which is to build capacity--at every level of the system--to gather, access, and use data to inform decision-making processes.

As currently developed, TED will allow SELPAs, districts, and COEs to track possible training effects and target training resources to areas of need, and most importantly to make reports based on that data as easy as possible to access in a useable form for decision-making processes at the local level; see sample screens from TED as well as samples of the automated reports TED is capable of generating in Appendix C.9.

TED will be made available for adoption on a voluntary basis by SELPAs and COEs. Currently TED is scheduled to be beta tested by two SELPAs in the spring of 2004 and will be ready for a broader pilot beginning in the summer. Under the SIG2, TED will be piloted in an additional 7-10 sites each year for the 3 years of the grant. Pilot sites will initially be recruited from among the SELPAs and COEs that have been actively involved in the SIG TA effort, due to their familiarity with the components of the SIG evaluation process and forms. Initial pilot sites will receive the TED database program and training in its use, with stipends to cover time/travel costs for 2 personnel from each site. During the year TA will be provided as needed to the pilot sites by SIG and the TED programmer. At the end of the year, a second meeting with the pilot sites will focus on how to generate and use summary reports from TED in conjunction with student outcomes data in planning and decision-making processes.

Feedback on TED will be elicited from the pilot sites at each stage of the process through brief reports and discussion at training meetings and will be used to refine the database program and add needed reporting capabilities. To broaden the piloting of TED, it will be demonstrated at Statewide Core Message Leadership Institute meetings and other statewide committee meetings such as those of the Comprehensive System of Personnel Development Advisory Committee (CSPDAC) and potentially the annual SELPA Director's meeting. This will help recruit additional pilot sites in years 2-3, as well as foster the use of data for decision-making around the use of TA resources by example.

g.2. Evaluation examines the effectiveness of project implementation strategies.

Examination of implementation strategies and performance is key to an assessment of the SIG Supplemental project. As shown in G.1 above, the objective activities will be assessed for effectiveness through collection of qualitative and quantitative data. Issues around implementation of the activities will clearly be documented by the post-TA evaluation surveys, the 3-month follow up surveys, test data analysis, and interviews. These will be addressed at the formative evaluation meetings.

g.3. Objective performance measures producing quantitative/qualitative data.

As shown in the narrative in G.1, the objective performance measures clearly produce both quantitative and qualitative data about the activities related directly to the project outcomes.

g.4. Methods provide performance feedback and periodic assessment of progress.

The Partnership has designed the SIG2 formative evaluation to provide performance feedback at the activity level, to summarize data-to-date in terms of progress toward achieving outcomes, and to permit implementation adjustment based on findings twice within the project year with the PCSE Evaluation Task Force. The annual summative evaluation will then review both project performance in completing the activities and to assess progress toward achieving outcomes during the last year.

Conclusion

As California educators, we continue to search for ways to improve the system, to involve the community, and to better serve the children. But in our efforts toward systemic improvement, it is essential that we focus on how those changes affect the individual student, the unique child with disabilities. John F. Kennedy said that we must "think of education as the means of developing our greatest abilities, because in each of us there is a private hope and dream which, fulfilled, can be translated into benefit for everyone and greater strength for our nation." It is our most sincere desire that, in creating a better system, the SIG2 will enable each child to realize their private hopes and dreams.

 


IDEAS that Work!

Project READ is a California Department of Education, Special Education Division project funded through a federal competitively-awarded State Personnel Development Grant to California (#H323A120019) provided from the U.S. Department of Education Part D of the Individuals with Disabilities Education act (IDEA), Office of Special Education Programs (OSEP). Opinions expressed herein are those of the authors and do not necessarily represent the position of the U.S. Department of Education.