Antecedents

This is the fourth in a four part series describing the Holgarth Middle School experience using the NCA CASI Performance Accreditation Framework and school improvement process. Together, these four articles chronicle the flow of events that take place during the typical school improvement cycle.

The first article (Journal of School Improvement, Spring 2001) viewed the Holgarth Middle School staff members as they began their school improvement cycle, traced the activities of the school's steering committee through the first four stages of the SIP process, and concluded with a visit by the external peer review chair. The second article (Journal of School Improvement, Fall 2001) followed the school staff as it completed and analyzed the Capacity Assessment Instrument, defined the essence of the reading comprehension and writing goals selected, developed a set of assessments designed to measure the degree of student growth throughout the cycle, and agreed upon a set of interventions designed to cause student growth to occur. A preliminary school improvement plan was developed, and the arrangements were made for the visit of the peer review team. The third article (Journal of School Improvement, Spring 2002) began with the arrival of the peer review team and described the series of events that took place during the two-day visit. The article concluded with the delivery of an exit report to the school community. This final article in the series begins with the arrival of the written report from the peer review team at the school and describes the events during the 30 months of the implementation phase of the SIP cycle. Major attention is directed toward the documentation of student growth during this time period and the work of the external team during the documentation visit.

The Written Report Arrives

As promised by Dr. Boris Heine, our external team chair, the written report of the first team visit arrived at Holgarth within two weeks of the exit report to the school community. The report was about 20 pages in length, and our principal, Sarah Artez, had copies made for each of the staff members as well as for non-staff members of the several committees. Ms. Artez had scheduled an early-release day for students so that the staff might consider the report and any suggested modifications to our School Improvement Plan (SIP). On the scheduled early-release day, the entire staff as well as non-staff members of the various committees, gathered in the Holgarth library. Fred Levendowski, our steering committee chair, distributed copies of the written report and offered brief remarks about the SIP process to date. He led us through the general recommendations and then asked Roberta Stepanov, the writing goal chair, and Don Martin, the reading comprehension goal chair, to gather with their committee members for an in-depth discussion of the recommendations offered for each of the goal committees. Teachers not assigned to ether of these committees were asked to join one of the groups and participate in a discussion of recommendations of the external peer review team. Specifically, we were asked to consider the rubric scores provided for assessments, interventions, and staff development as well as specific areas identified for further consideration by the school staff.

Modifications to the Plan

Some general recommendations were made regarding the SIP process. We decided to ask one of our steering committee members, Norm Sherman, to serve as the data specialist. His responsibility will be to receive all assessment data for both of the goal committees and develop a reporting system for the school community as well as for the final documentation of our SIP at the end of the cycle. He was asked to find ways of reporting these data in both narrative and graphic form. Additionally, Fred Levendowski asked Pamela Ruben, an English teacher on the steering committee, to serve as the publicity specialist. Her responsibility will be to develop news releases and stories regarding the school improvement efforts at Holgarth for the various news media serving the school community. Also, Sarah Artez committed to supporting an orientation to the SIP for each staff member joining the faculty.

The peer review team also made specific recommendations for each of our two goals. In this narrative, we will deal with those modifications regarding the writing goal (a copy of the First Peer Review Team Visit Report may be found at http://www.usd.edu/nca). Specifically, an additional measure to assess the quantity of writing was developed. The phrasing on our planned survey instruments was also standardized and Likert-type scales were added in order to make a statistical analysis more valid.

Implementation of the Plan

After incorporating recommendations from the peer review team, our staff felt that we were ready to begin implementation in the fall. Fred Levendowski reminded the goal committees to be sure that their baseline data for the multiple assessments were in place in the spring so that the interventions could be initiated at the beginning of the next school year. For the writing goal, this meant that we had the latest state writing assessment data gathered prior to beginning the interventions as well as the SAT-9 scores, a baseline writing sample for each student, and the survey perception data from parents and students and the follow-up study of promoted students. Norm Sherman, our data specialist, was charged with coordinating the collection of baseline data and displaying it in ways that would make analysis of student growth possible over the life of the SIP cycle. Pamela Ruben was asked to begin preparing publicity releases about the SIP for parent and community publications.

In the "Back to School" events of the next school year, the school improvement plan was highlighted to the parents, the students, and in school publications. Much of the staff development time on the days before students arrived was dedicated to ensuring that we were all ready to begin our part in bringing the SIP plan into operation. Norm Sherman had prepared graphic displays of the baseline data for each of our goals, and these displays were posted in the library, the teacher workroom, and in the outer school office. Pamela Ruben had established contacts with the local television stations, and a camera crew was on hand to do a short feature story on the school improvement effort that was about to "kick off."

Changes During Implementation

Throughout the school improvement planning process, our peer review chair as well as our steering committee chair talked about flexibility. We needed to be able to shift interventions and sharpen our assessments and staff development as circumstances dictated. Above all, we had to keep alert to changes in student achievement over the course of this SIP cycle so that we might be assured that we were providing optimum support for the improvement efforts underway.

Early on we discovered that our non-language arts teachers had some difficulty applying the 6+1 writing rubrics for the traits of sentence fluency and conventions. After staff discussions, we decided that the remaining traits were more appropriate for all teachers, and we left fluency and conventions in the hands of our language arts teachers.

Much to our disappointment, Sarah Artez, our principal, decided to accept a position as assistant superintendent for curriculum and instruction in another school district about half way through the SIP cycle. Dr. Ronald LeBeau was selected as our new principal. Dr. LeBeau had served as an assistant principal in another NCA CASI school and was familiar with the school improvement process. He had, in fact, chaired several team visits at other schools. We were all relieved that he had a basic knowledge of the SIP process, and he very quickly assumed a supportive role in our work.

Preparing the Documentation Reports

As we entered the fifth year of our school improvement cycle, the staff was becoming more and more excited about our school improvement efforts, especially in the area of writing. We were all anxious to see how our summative assessments measured up in relation to our initial goals. The formative assessments indicated what appeared to be good growth in the writing. We had experienced some problems with the implementation of the interventions for the reading goal and while we felt we were making gains with the goal, the assessments did not indicate any clear changes in reading performance among our students. The problems had been worked out with the reading improvement plan and the staff felt that it might be one or two more years before we would have sufficient documentation to make final decisions for the goal.

In the fall of the year Fred Levendowski asked the goal committee chairs to join the steering committee for a before-school meeting. Fred wanted to review the documentation phase with all of the leadership and establish a time-line for the NCA activities during the year. At the meeting, Fred presented the following outline for activities that needed to be accomplished before the spring of the year.

As a means of reviewing what we needed to accomplish, Fred reviewed Phases VII and VIII in the NCA CASI software suite, Tools for Team Chairs and School Leaders. The software provided us with a step-by-step review of the tasks in the two phases, a template for the preparation of the documentation reports, a rubric to evaluate the documentation reports, and an outline for the documentation visit. Fred also reminded us that we needed to complete the Capacity Assessment Instrument (CAI) as a staff before the documentation visit. The CAI results would be compared to our initial results from the beginning of the cycle to help us determine if we were growing as a system in our ability to support school improvement. Fred reviewed the results of the initial CAI that we had completed in the NCA CASI software Developing the School Profile at the beginning of our cycle.

As we began discussing our time-line, the dates of our assessments in March and April became a topic. Roberta Stepanov, from the writing committee, wondered if we needed to move the State Writing Assessment and our local writing sample administration dates up to January or February in order to have results from this year in time for completion of the documentation reports. Norm Sherman, our "resident statistician" on the steering committee, reminded us that we needed to use last spring's results at this time. Norm explained that the assessments must be given at a comparable time of year in order to be valid for statistical analysis. Even though our results at the end of year five would be important, the results from last spring would include two full years of implementation, a sufficiently long enough period of time to provide valid information about the results of our efforts. Since we would be using post-assessment results from last spring, it was decided that we could start the process of summative documentation in the next few months.

The group decided that the gathering of the data and preliminary analysis of the results by the goal committees should be finished before the holiday break in December. The results would be presented to the staff in January, and in February several staff and team meetings would be dedicated to the discussion of the results. The final documentation reports should be completed by mid-March, and we would plan on hosting the documentation visit in early April.

Roberta Stepanov called the writing goal committee together in early November to the review the tasks of the committee. After reviewing the template for the preparation of documentation reports, assignments were made to accomplish the tasks.

Two members of the writing goal committee had attended a workshop last year to learn about the NCA CASI Data Analysis Software and would be working with Norm Sherman to enter and analyze the data. Other committee members were assigned to writing the sections of the report that dealt with the implementation of the plan and the conclusions and curricular changes resulting from the school improvement efforts that would emerge from the data and discussions.

The writing goal committee began its work immediately, with the intent of completing the preliminary analysis of the assessment results by mid-December. Since we had planned carefully for our assessments in the school improvement plan, gathering the data was relatively easy. With the guidance of Norm, the committee members were able to determine that the data about writing contest entries and publication submissions, while providing useful information, were not valid for statistical analysis. Neither of these two assessments had any upper (mathematical) limits. The other assessments were valid for analysis, providing the committee members with an acceptable number of assessments that were both common metric and context bound in nature. By early December, the writing goal committee was able to view the results of the data analysis. The results indicated that the students had demonstrated verifiable (in many cases substantial) growth on all of the assessments for writing. The results had been disaggregated by gender, ethnic status, socio-economic classification, and by some disability categories. In discussions among the writing committee members (using effect size) it was noted that the context bound assessments indicated the greatest amounts of growth, the State Writing Assessment (common metric) indicated substantial growth, and the Stanford Achievement Test (common metric) verifiable growth worth mentioning. Some subgroup results merited further attention. While nearly all of our disaggregated groups demonstrated verifiable growth in their writing skills, some groups didn't appear to have benefited as much from the interventions as did other groups. The committee members noted the following:

The combined effect size of all the analyzed assessments was a .34 (Adapted Standard Units), indicating substantial verifiable growth for the goal. The committee was thrilled with the results and looked forward to sharing them with the rest of the staff.

Roberta shared the results for the goal with the steering committee shortly before the holiday break in December. A staff meeting was scheduled in mid-January at which both goal committees presented the preliminary results to the entire staff. We found that we had an abundance of data that supported positive growth on the writing goal but the data indicated mixed results for the reading goal. Both committees presented relevant disaggregated data for various assessments and offered preliminary conclusions about the progress on each goal from the data. Following the faculty meeting, each of the interdisciplinary teacher teams scheduled several meeting times to discuss the results and provide the goal committees with feedback. The teams were asked to focus on conclusions that could be drawn from the data, to identify the interventions/strategies that they felt were most powerful or effective, to reflect on curricular changes that should be considered based on the results, and to identify those interventions that should be institutionalized (become part of our way of doing things) and those that should be discarded. Each team was asked to provide the goal committees with its recommendations by mid-February.

The goal committees completed their final reports using the information from the staff and submitted the reports to the steering committee in March. Both committees used the Documentation Report Rubric from the NCA CASI Tools for Team Chairs and School Leaders to evaluate the final reports. The writing goal committee concluded that the data supported institutionalizing the 6 + 1 Writing Traits, seeking some assistance from the upper elementary teachers in preparing students for the use of the traits in writing, and continuing to promote writing through contests and student publications. (A copy of the final documentation report for the writing goal may be viewed at www.usd.edu/nca.)

The Peer Review Documentation Visit

Mr. LeBeau, our principal, contacted Dr. Heine in the fall of the year to establish April 3rd as the date for the documentation team's visit. In March, Fred Levendowski arranged for a conference call between the steering committee and Dr. Heine to review the details of the visit. Dr. Heine indicated that he would be bringing two other individuals with him for the one-day visit. The two individuals would each be assigned to review one goal.

Unfortunately a number of the original visiting team members had moved and one of the members of the current team would be new. Dr. Heine assured us that this individual would be familiar with the previous reports and our improvement plan before she joined the team. We also arranged to have the documentation reports mailed to Dr. Heine and the other team members and established times for the steering committees and goal committees to meet with the teams. As a final note, Dr. Heine reminded us that we needed to complete the Capacity Assessment Instrument prior to the visit. Since we had forgotten about the CAI, we made immediate arrangements for the interdisciplinary teams to complete the instrument and submit their results to the steering committee. A staff meeting was set aside to discuss the final ratings for each of the items on the instrument prior to creating the CAI summary in the NCA CASI software.

The team arrived at 8:00 a.m. on day of the visit and began with a coffee reception with the entire faculty. Dr. Heine told the faculty that the primary role of the team was to verify the implementation of the plan, review the documented results, and make some recommendations for the next school improvement cycle. He thanked the faculty members for their dedication and involvement in Holgarth's school improvement efforts and indicated that the team would be meeting with the steering committee and goal committees and, unlike the previous visit, would not be meeting with all of the staff. Following the reception the team met with the two goal committees. Each committee reviewed its documentation report with the team members. Each of these reports took about 45 minutes. Following the reports, the team members spent about 45 minutes visiting with some students, the counselors, Mr. LeBeau and his assistants, and some of the staff. They then joined the staff for lunch and met with Mr. Levendowski and the rest of the steering committee. During this meeting we discussed our conclusions about the current school improvement cycle, compared the results of the latest CAI with our original ratings on the instrument, and discussed some future directions. Dr. Heine indicated that the team would meet for about an hour and a half to complete its report and would depart before the end of the day. He also congratulated us on our hard work and success and indicated that the visiting team would be recommending to the state committee that Holgarth Middle School be accredited by NCA CASI as a result of our school improvement efforts. Following a review of Holgarth's portfolio, the NCA CASI Board of Trustees will make the final accreditation decision. Dr. Heine indicated that we would still be required to complete our annual report and meet the membership criteria on an annual basis. The final report completed with the NCA CASI Tools for Team Chairs and School Leaders would be available to us in about two weeks.

Celebrating the Success and Renewing the Process

We received the Peer Review Documentation Team report on the 18th of April. Mr. LeBeau and Fred Levendowski shared the results of the report with us at a staff meeting near the end of the month. Fred had met with our publicity chair, Pam Ruben, prior to the meeting; and they planned several releases for the media concerning the results of our efforts and the recommendations of the visiting team. Most noteworthy of the findings and recommendations included: