About the Author: Robert Armstrong is professor emeritus of Arizona State University and a research consultant for the NCA Commission on Schools, Tempe, Arizona.

Editor's note: This article is the third in a series of reports on the progress of the NCA COS school improvement endorsement plan, beginning with Wick (1997) and continuing with Armstrong (1998). The schools included in the data presented here achieved outcomes endorsement.

Introduction

In the present report is summarized the achievement score change results on 292 schools that were granted the outcomes endorsement in the 1997-98 or 1998-99 school years, and were reviewed at the annual meeting in 1998 or 1999. The purpose of this study was to chart the progress of the NCA COS outcomes-endorsed schools in their improvement efforts, and to continue the reporting sequence begun by Wick (1997) and continued by Armstrong (1998). As in the earlier reports, the emphasis in the present report is on achievement change scores (expressed in adapted standard score form for easy comparison), and on patterns of goal selection.

There are a number of points of comparison of the present report to the data in the 1998 report. However, there are a couple of exceptions that resulted from differences in the nature of the data or differences in perceived areas of interest. In the present set of schools, assessment types were more mixed within goals than in the previous two years. Thus, it was considered impractical to attempt to classify goals as having been tested by common metric or context-bound assessments¹, as was done in the 1998 report. Also, the question of whether a school received a goal score for each goal, or only part of the goals, received little or no attention in reader response. When an overall standard score is assigned to a school, it is calculated in a manner to yield assurance of being a reasonable representation of the school's overall change of achievement, even if not all goals could be scored (see below), so interest in the question of one or more goals missing a convertible score is debatable.

Review of the Improvement and Documentation Process

A school begins an improvement cycle by conducting a survey of the students/school/com-munity to create what is referred to as a student profile. The purpose is to gather information concerning student learning needs and school and community resources so that a focused and purposeful improvement plan can be developed. Immediately following the study, the student learning needs reflected in it will be used to determine target areas and goals.

As soon as goals are identified, the task of selecting or creating assessment means begins, simultaneous with the task of beginning the development of the improvement plan. When these

¹Also, in practice the distinction between common metric and context-bound assessments has been less clear than earlier hoped. Some forms of assessment were difficult to classify.

tasks are complete, and immediately prior to the implementation of the improvement plan, "pre-testing" is conducted. This "pretesting" consists of testing of all students (or all students at selected grade level[s]) who were taught with traditional methods in traditional programs. Student achievement on the curricular content of the goals is assessed and used as baseline data.

A given period of time later, generally at least two academic years, the equivalent students (all students, or the same grade level[s]) are posttested at the same time of the year as the pretests were done, and using the same assessment instruments. In this manner, comparable score data are obtained, so that student achievement resulting from the methods of the improvement plan can be compared to the achievement of students who were taught by the traditional methods. These difference scores, expressed as adapted standard scores, help the staff analyze the effectiveness of the improvement plan interventions as compared to the traditional methodology.

At the end of an improvement cycle the school summarizes the improvement effort, including the achievement results, in a final documentation report. If the process, and/or the achievement results are considered satisfactory by the visiting team, the state committee grants the school an endorsement. Depending on the number of completing schools in each state, all or a sample of the final documentation reports are forwarded to the North Central Association Annual Meeting for review by the commission. The present report was assembled from the achievement score differences in the final documentation reports that were sent to the annual meeting in 1998 and 1999. The assessment score differences were converted to adapted standard units at the annual meeting if they had not already been converted by trained state office personnel. Each goal was assigned a score if at least one of its assessment instruments produced a numerical score. Each school received an overall achievement score if at least half of its goals had assigned scores.

Data and Commentary

In Table 1 the average standard unit change of achievement is shown for each school level.

Table 1: Data By School Level

Also shown is the number of schools at each level, the number of schools that did not receive a score, the number of schools that recorded an apparent loss of achievement (negative standard score difference), and the highest and lowest school score in each school category.

The first obvious question is, "What do these scores (second data column) mean?" They are equivalent to standard deviations. The high school level, with an average gain of .25SU (standard units) per school, is showing an improvement in average overall achievement on the goals, equivalent to a quarter of a standard deviation in the middle of a normal distribution. This would be the equivalent of approximately 10 percentile rank points.

For the purpose of value estimation and school interpretation of the scores, an arbitrary but reasonable set of division points has been adopted to allow the use of descriptions for school scores. These guidelines are provided to schools to help them determine whether the differences resulting from their interventions are large enough to be meaningful. The division points are, as follows:

Due to normal test and aggregation error, the range from -.09 to .09 should be viewed as showing neither gain nor loss. (The distribution of school scores over the above descriptive range is shown in Table 2.)

As noted above, if fewer than half of a school's goals were assigned a score, the school would receive no score (third data column, Table 1). Approximately 9% of the 1998 and 1999 schools fell into that category. A negative score (fourth data column) indicates that the students who were taught under the improvement plan achieved less than those earlier students who learned by traditional methodology. On the face of it, this is an indication that for at least some of the goals of the school, the interventions developed for the improvement plan were not as effective as were the traditional methods. These results must be carefully interpreted, however, since the phenomenon of entire class levels varying from each other in academic ability is known to all teachers, particularly at the elementary level. In the current report 14 (less than 5%) of the schools achieved a negative score, and of those only five scored lower than -.09. Thus, only five of the 292 schools had a confirmable loss of achievement.

Comparing the data in the second column (mean change of achievement score) to the "totals" column of Table 2 in Armstrong (1998) shows that the change of achievement for the schools of the past two years compares very favorably with those reviewed at the annual meeting in 1997. In fact, for every school level but elementary, and for the total, the mean achievement of the schools of the present report exceeds that of the schools of two years ago. The elementary school average dropped by .03SU, from .31SU to .28SU. However, the unit school average rose from .15SU to.20SU, the JrHi/MidSch rose from .14SU to.22SU, High School rose from .19SU to .25SU, and the total for all schools rose from .23SU to .26SU.

Table 2: Distribution of School Scores

Another way to view school scoring results is exhibited in Table 2. Score differences are shown distributed across the aforementioned arbitrary division points of .10, .20 and .30 for the 266 schools that received overall scores. In the bottom two rows the total distribution for 1998 and 1999 combined is compared to the distribution for 1997. It is easy to see by the percent-of-total figures that there has been an upward shift in improvement of student achievement in 1998 and 1999 over 1997.

A look at goal selection, and an alternative view of achievement by goals, is offered in Tables 3a, 3b and 3c.² Table 3a exhibits the high school goal selection pattern. It can be seen that there is a heavy concentration in what might be called the "basic skills."

Table 3a: High School Goal Selection and Achievement by Goal

Reading, writing and mathematics computation account for 54% of the total. If the academic skills of critical thinking and problem-solving are added to the basic skills, the proportion rises to 84%. It should also be noted that whereas 89% of the cognitive goals were assigned standard scores, only 79% of the affective goals were so assigned, an indication of the greater difficulty of affective goal assessment, resulting either in verbal data or no data.

As to affective goals, virtually half are in the category of responsibility, while respect, responsibility and self-esteem together constitute 78% of all affective goals. The remaining 35 goals are widely scattered over a variety of generally unrelated outcomes such as community involvement (8), social interaction (3), and decision-making (4).

Table 3b: Junior High/Middle School Goal Selection and Achievement by Goal

Table 3b shows the same information for junior high/middle schools. Although the number of JH/MS schools is substantially less than the number of high schools or elementary schools, the results should still be meaningful. The goal distribution patterns are similar to those of the high schools for both cognitive and affective goals. As in the high schools, more affective goals had no numerical score, although for both cognitive and affective goals the junior high/middle schools had more schools without a score than did the high schools.

Table 3c: Elementary School Goal Selection and Achievement by Goal

The distribution pattern of goal selection for elementary schools, as shown in Table 3c, is also similar to the other school levels. Fewer affective goals had scores (68%) than cognitive goals (91 %). Of the cognitive goals, the basic academic skills comprise most of the goals (91 %), with reading the leader. In the affective goals, 78% are respect, responsibility or self-esteem, with responsibility the leader. It should be noted that the overall achievement change score for each school level in Tables 3a and 3b differs slightly from those for high school and junior high/middle school as shown in Table 1. The reason is that arithmetic mean is a weighted average. When scores were averaged for each school level in Tables 3a-c, all goals had equal weight. However, a school score may represent two goals, or three goals, or four goals, or more, depending on how many goals the school addressed, and depending also on how many of the goals in any given school were assigned a score. Conceivably, a school score might result from as few as one goal (in, say, a two-goal situation) on up to three or four goals, or more.

Discussion

An encouraging result of this two-year analysis is that average school change-of-achievement scores not only held up from the 1997 report but increased from .23SU to .26SU. Of all school levels, only elementary schools lost ground, but even at that the elementary level continued to lead the other school levels in overall gain in goal achievement.

If anything, goal selection seems to have narrowed in the past two years, in both cognitive and affective areas. In the cognitive area, two causes may be in effect. For one, at least some states are encouraging or requiring schools to improve in the basic skills, particularly "reading." As a result, many schools are selecting these basic skills as their cognitive goals. Also, when standardized test results are published, they are most often the scores of basic skills testing. Thus, these outcomes are most in the public eye. It also seems likely that the relative popularity of critical-thinking/problem-solving goals, particularly at the secondary level, reflects that educators at some level value independent thinking.

For affective goals, the one truly dominating goal was "responsibility." This seems to indicate a strong desire at all levels for students who function more cooperatively, who are better behaved, who consider the consequences of their actions. Since "responsibility" is the hallmark of the "mature" adult, the popularity of this goal is quite natural.

In light of the popularity of computers among the younger set, and the rapidly increasing number of uses of which a computer is capable, the low popularity of technology oriented goals (4%) seems difficult to understand. Further, less than two-thirds (22) of the 34 technology goals were assigned a numerical score. Low popularity of the goal and difficulty in measuring computer competence may have strong implications for school management of this important educational tool.

At all school levels there was a shift away from affective goals to cognitive goals. At the elementary level the average for affective goals fell below 1, meaning that some elementary schools had no affective goal, at all. There may be two possible causes for this. On the one hand, school people may be viewing cognitive achievement as more important than affective achievement, particularly since it is in the cognitive area that the most public pressure exists. Another possibility, of course, is that cognitive goals are generally easier to measure. The greater difficulty of measuring affective goals is attested by the greater proportion of affective goals that did not have an assigned score.

Concluding Comments

It seems apparent that the school improvement process that results in schools obtaining the NCA COS outcomes endorsement holds the potential for schools to improve student achievement. There are several characteristics of the model that may contribute to the relative success that has been experienced by the majority of the schools.

1. Within whatever state constraints exist, the school selects its own goals.
2. Faculty committees assume virtually complete authority and responsibility for the development of the improvement plan.
3. The faculty steering committee, or a committee designated by it, monitors the progress of the plan and the quality and degree of faculty involvement in it.
4. The improvement plan is normally in process for two or three academic years, providing enough time for learning effects to come to fruition.
5. The visiting team of peers offers support and advisory expertise throughout the improvement cycle.
6. The student achievement results on all goals are generally assessed by two or more instruments/formats, thus providing corroboration of results.

In spite of the high degree of faculty ownership of the NCA COS improvement process, there is no question that the full support of the building administrators and district office is critical to the success of the improvement process. For new and unfamiliar interventions, teachers need inservice to learn the skills. New interventions sometimes call for monetary expenditures. Not the least of the needs is the moral strength of the active encouragement of those in positions of management authority.

A fundamental aspect of the entire process is that curricular decisions are based on data. The assessment results of each improvement cycle are utilized in the process of deciding to continue or discontinue the selected interventions, to continue to emphasize certain goals or to move on to others. Such decisions based on authentic assessment evidence will ultimately result in better choices than decisions based on whim or hunch. Teaching is both a science and an art, and the best traits of both should be brought into play.

References

Armstrong, R. L. (1998). The outcomes endorsement: An update. NCA Quarterly, 72(3), 402-406.

Wick, J. W. (1997). Does the outcomes endorsement really work? NCA Quarterly, 71(3), 406-410.