|
|
Building Smart Education Systems |
Voices in Urban Education
Archives
Using Data for Decisions
VUE Number 18, Winter 2008
| VUE Home | Archives | Order Print Copy |
Beyond Test Scores: Leading Indicators for Education
By Jacob Mishook, Ellen Foley, Joanne Thompson, and Michael Kubiak
Jacob Mishook is a research associate, Ellen Foley is associate director of district redesign and leadership and assistant clinical professor in the Master's in Urban Education Policy Program, and Joanne Thompson and Michael Kubiak are research associates at the Annenberg Institute for School Reform.
> Author's biographies
A study of four leading-edge districts suggests what it might take to create a system that provides useful information about early signals of progress toward academic achievement.
Improving student outcomes and closing achievement gaps, both within a school and across a district, takes time — more time than is often allowed in typical big-city political environments. Education leaders and community members need a way of examining their schools and their school systems that allows them to understand when (and whether) progress is being made, before the results show up in indicators like student test scores.
Leading indicators — indicators that provide early signals of progress toward academic achievement — enable education leaders, especially at the central office level in a school district, to make more strategic and less reactive decisions about services and supports to improve student learning. The concept of leading indicators incorporates a way of viewing and using data to inform systemwide decisions about education. It builds on existing efforts by school districts to use “data-informed decision making.”
This article examines how four districts that are at the forefront of the field in using data to inform decisions are developing and using leading indicators for education. By describing how these four districts — Hamilton County (Chattanooga, Tennessee), Montgomery County (Maryland), Naperville (Illinois), and Philadelphia — have developed and used leading indicators within the context of a strong district “data culture,” the Annenberg Institute for School Reform hopes both to catalog specific indicators that have been useful to these districts in increasing student achievement and to expand the notion of a leading indicator beyond easily identified testing data to more difficult-to-measure but crucially important measures such as student engagement and central office practice. As one district partner put it:
What Makes an Indicator a Leading Indicator?
The most widely accepted and used indicators in education are scores on standardized tests that are given at the end of each school year. These and the other lagging indicators typically collected usually arrive too late to help individual children or schools that are struggling. These measures do not tell us whether the types of practices, people, strategies, materials, or technologies that school districts are investing in are likely to lead to improved student achievement.
Leading indicators, on the other hand, are:
Common Indicators
Early Reading Proficiency
Early reading proficiency was the most common leading indicator examined by our study districts. It was often the first thing district leaders and partners mentioned when asked if they could identify any high-leverage indicators.
Algebra Mastered in Eighth Grade
All of our study districts had developed some kind of mathematics initiative to help students master algebra sooner in their academic careers. They monitor enrollment and performance in mathematics classes, striving to help students understand algebra by the end of eighth grade.
Over-Age Students
Two of our study districts work to identify students who are “over age” in each grade level. In high school that typically might mean a student who has only accumulated enough credits to qualify as a sophomore but is actually old enough to be a junior or a senior. In elementary school, over-age students are those who are a year or more older than their peers in the same grade.
Grade-to-Grade Transitions
One district, in particular, focused on data around student transitions, especially from fifth to sixth grade, eighth to ninth, and ninth to tenth and has established “transition goals” to ensure that middle school students are academically prepared for a rigorous high school curriculum. The district has also used these data to develop a new policy: based on data showing the difficulties that students encountered in the ninth-grade transition, the district created ninth-grade academies in some high schools, as well as “mid-high schools” to both ease the transition and provide targeted support to keep students on a successful high school trajectory.
College Admission Test Scores
Two districts in our study have examined scores on college-entrance examinations (e.g., the SAT and the ACT) and their associated preparatory tests (e.g., the PSAT) and curricula (e.g., ACT PLAN and EXPLORE). They identify students who score high but are not enrolled in advanced courses or who are in danger of dropping out. One district is piloting the ACT's eighth- and tenth-grade college- and career-planning tests and is also utilizing a Web site that correlates state assessment scores to predict ACT scores and expected salary figures for future employment. Another district enrolls students, particularly students of color, in Advanced Placement courses if they score high on standardized tests.
Attendance and Suspension Rates
Districts have made headway collecting and sharing school- and district-level attendance rates with greater frequency. For example, in one district, attendance data reports had previously been delivered to schools each month and again at the end of each semester. Now, attendance data are shared on a ten-day cycle, allowing for principals to identify students and grades that have chronic attendance problems and to make necessary changes.
Multiple districts in the study have also improved the ways in which they attempt to correlate attendance data with suspension and “major incident” discipline rates. In this case, the key is to look not just at the overall percentages, but also at whether it tends to be the same students that are chronically suspended — and to build a subsequent understanding of how many instructional hours these students are missing and the academic cost of those absences.
Harder-to-Measure Indicators
All of the indicators described above are relatively easy to measure and data related to them have, for the most part, been collected — if not analyzed — by most districts for years. But there were also some indicators that our districts examined that were more difficult to quantify and are not collected widely by most school districts. These indicators include student mobility, special education enrollment, student engagement, and teacher and principal quality, including teacher turnover.
Student Mobility
Particularly in urban areas, high rates of student mobility make it more challenging to sustain each student's academic growth. Not surprisingly, districts in this study have found that, controlling for other factors, schools with higher mobility rates have lower student-achievement levels. Given that mobility will continue to be a fact of life for urban districts, the solution may in some respects lie in better data collection. Using a universal student identifier and relying more on technology to collect data are two strategies our study districts are using to improve the accuracy of their data about student mobility.
Special Education Enrollment
Special education students, under No Child Left Behind, receive a great deal of attention due to the need to make adequate yearly progress with all subgroups. All four districts in this study tracked data on special education students, though sometimes these data were not integrated. For example, interviewees in one district mentioned that information in special education students' Individualized Education Programs existed only on paper, not in the district's data warehouse, and that integrating these data was a priority.
Student Engagement
One district used “focus walks” to examine the level of student engagement in classrooms and benchmark “how we want students engaged in learning.” Districts also reported that they did frequent student surveys on topics ranging from technology use to students' social-emotional needs. However, some admitted that student engagement is not easily quantifiable. Thus, there's a belief that student engagement, as defined in myriad ways (e.g., school climate, time on task), is important. But the means of measuring student engagement are limited. The challenge is to develop a richer indicator that is more easily measured and can be understood and acted upon by administrators and teachers.
Because of the often-elusive nature of the concept of teacher quality, districts have approached this issue from a variety of vantage points. Getting easily quantifiable and usable data on teacher quality is complex and difficult.
Because of the often-elusive nature of the concept of teacher quality, districts have approached this issue from a variety of vantage points. At least one district is looking at teacher turnover. Another district has begun looking at measuring teaching practice through a coaching model that requires intensive examination of pedagogy. However, collecting data through this model has proved to be labor intensive, and it is difficult to use the information to train teachers to be more effective. Furthermore, like collecting data on student engagement, getting easily quantifiable and usable data on teacher and principal quality is complex and difficult in all four districts.
Another district has approached the measurement of teacher quality from several angles. The district has implemented a teacher evaluation system and reexamined surveys on teacher satisfaction to determine whether teacher satisfaction had any impact on student achievement. Another survey of teachers and administrators showed that supervisor ratings were meaningful to teachers. The district has also implemented an interview tool that scores teacher applicants and plans to determine whether this tool is, as a central office administrator put it, “actually sorting out who are the best teachers.” The district also tracks teacher professional development and teachers who are released and, with the collaboration of the union, has developed an exit survey.
The Data Wish List
Like most districts, the four districts in our study collect a lot of data. Still, there are areas where the data are thin. We asked our respondents to highlight data that weren't available to them but that they would like to have. While the items on their wish lists are not all leading indicators, the statistics described below could all be a part of a robust system of leading and lagging indicators.
Post-secondary Outcomes
All of our districts were able to minutely dissect student outcomes through the twelfth grade. But as soon as their students graduated, they had limited ways to track them. The ultimate proof of the education that districts provide is neither the students' scores on standardized tests nor their grades, but their success after the end of high school in college or the world of work. It was extremely difficult for our districts to know what happened to their graduates.
Social-Emotional Data on Students
Several of our districts expressed an interest in examining data related to the emotional well-being of their students but had found limited ways to get to these data. Participants from one district almost universally commented on the importance of these data.
The superintendent said,
Teacher Preparation and Training
Similarly to their interest in issues of teacher quality, the districts were particularly interested in gathering additional data about teacher preparation and training. For example, one district's vendor said:
The four districts in our study are among the most advanced in the country in using data to inform their decision making, and other school systems can learn a great deal from their successes and challenges. For us, at least, their experiences offer the following lessons.
Few school districts have the time, resources, or expertise to collect data on harder-to-measure concepts that reflect the kinds of rich learning environments we want our children to have.
This goal is consistent with the Annenberg Institute's vision of “smart education systems.” Smart education systems bring together schools, community organizations, and civic agencies and institutions to create a web of supports to develop a broad range of outcomes for children and youths. Using data differently is one of the key aspects of smart systems.
As we move forward with our work on data-informed decision making and leading indicators, we will focus on helping districts and communities think broadly about student engagement and figure out how to measure it. We will also collaborate with central offices to gather key information about their own services and more data about policy implementation. We will make efforts to link data, resources, and expertise both within and across specific communities as a kind of data network to advance our understanding of how educators can use and benefit from richer, more powerful, and more timely information.
Warning: include(../permissions_sub.html) [function.include]: failed to open stream: No such file or directory in /Library/WebServer/Documents/annenberg/VUE/winter08/Mishook.php on line 443
Warning: include() [function.include]: Failed opening '../permissions_sub.html' for inclusion (include_path='.:') in /Library/WebServer/Documents/annenberg/VUE/winter08/Mishook.php on line 443
Archives
Using Data for Decisions
VUE Number 18, Winter 2008
| VUE Home | Archives | Order Print Copy |
Beyond Test Scores: Leading Indicators for Education
By Jacob Mishook, Ellen Foley, Joanne Thompson, and Michael Kubiak
Jacob Mishook is a research associate, Ellen Foley is associate director of district redesign and leadership and assistant clinical professor in the Master's in Urban Education Policy Program, and Joanne Thompson and Michael Kubiak are research associates at the Annenberg Institute for School Reform.
> Author's biographies
A study of four leading-edge districts suggests what it might take to create a system that provides useful information about early signals of progress toward academic achievement.
Improving student outcomes and closing achievement gaps, both within a school and across a district, takes time — more time than is often allowed in typical big-city political environments. Education leaders and community members need a way of examining their schools and their school systems that allows them to understand when (and whether) progress is being made, before the results show up in indicators like student test scores.
Leading indicators — indicators that provide early signals of progress toward academic achievement — enable education leaders, especially at the central office level in a school district, to make more strategic and less reactive decisions about services and supports to improve student learning. The concept of leading indicators incorporates a way of viewing and using data to inform systemwide decisions about education. It builds on existing efforts by school districts to use “data-informed decision making.”
This article examines how four districts that are at the forefront of the field in using data to inform decisions are developing and using leading indicators for education. By describing how these four districts — Hamilton County (Chattanooga, Tennessee), Montgomery County (Maryland), Naperville (Illinois), and Philadelphia — have developed and used leading indicators within the context of a strong district “data culture,” the Annenberg Institute for School Reform hopes both to catalog specific indicators that have been useful to these districts in increasing student achievement and to expand the notion of a leading indicator beyond easily identified testing data to more difficult-to-measure but crucially important measures such as student engagement and central office practice. As one district partner put it:
At its best, data should be more than a number. It should tell stories. Measure capacity. Create, in a sense, a living picture in order to see the school and the system in a different way. Present “the everyday” in a precise
and meaningful way.
What Makes an Indicator a Leading Indicator?
The most widely accepted and used indicators in education are scores on standardized tests that are given at the end of each school year. These and the other lagging indicators typically collected usually arrive too late to help individual children or schools that are struggling. These measures do not tell us whether the types of practices, people, strategies, materials, or technologies that school districts are investing in are likely to lead to improved student achievement.
Leading indicators, on the other hand, are:
- timely and actionable: they are reported with enough time to change a course of action in order to improve lagging outcomes;
- benchmarked: users understand what constitutes improvement on leading indicators, whether through longitudinal
comparison of the same data or through research-based criteria;
- powerful and predictive: they can offer targets for improvement and show progress — or a lack of progress — toward a desired outcome before that outcome can be expected to occur.
Common Indicators
Early Reading Proficiency
Early reading proficiency was the most common leading indicator examined by our study districts. It was often the first thing district leaders and partners mentioned when asked if they could identify any high-leverage indicators.
Algebra Mastered in Eighth Grade
All of our study districts had developed some kind of mathematics initiative to help students master algebra sooner in their academic careers. They monitor enrollment and performance in mathematics classes, striving to help students understand algebra by the end of eighth grade.
Over-Age Students
Two of our study districts work to identify students who are “over age” in each grade level. In high school that typically might mean a student who has only accumulated enough credits to qualify as a sophomore but is actually old enough to be a junior or a senior. In elementary school, over-age students are those who are a year or more older than their peers in the same grade.
Grade-to-Grade Transitions
One district, in particular, focused on data around student transitions, especially from fifth to sixth grade, eighth to ninth, and ninth to tenth and has established “transition goals” to ensure that middle school students are academically prepared for a rigorous high school curriculum. The district has also used these data to develop a new policy: based on data showing the difficulties that students encountered in the ninth-grade transition, the district created ninth-grade academies in some high schools, as well as “mid-high schools” to both ease the transition and provide targeted support to keep students on a successful high school trajectory.
College Admission Test Scores
Two districts in our study have examined scores on college-entrance examinations (e.g., the SAT and the ACT) and their associated preparatory tests (e.g., the PSAT) and curricula (e.g., ACT PLAN and EXPLORE). They identify students who score high but are not enrolled in advanced courses or who are in danger of dropping out. One district is piloting the ACT's eighth- and tenth-grade college- and career-planning tests and is also utilizing a Web site that correlates state assessment scores to predict ACT scores and expected salary figures for future employment. Another district enrolls students, particularly students of color, in Advanced Placement courses if they score high on standardized tests.
Attendance and Suspension Rates
Districts have made headway collecting and sharing school- and district-level attendance rates with greater frequency. For example, in one district, attendance data reports had previously been delivered to schools each month and again at the end of each semester. Now, attendance data are shared on a ten-day cycle, allowing for principals to identify students and grades that have chronic attendance problems and to make necessary changes.
Multiple districts in the study have also improved the ways in which they attempt to correlate attendance data with suspension and “major incident” discipline rates. In this case, the key is to look not just at the overall percentages, but also at whether it tends to be the same students that are chronically suspended — and to build a subsequent understanding of how many instructional hours these students are missing and the academic cost of those absences.
Harder-to-Measure Indicators
All of the indicators described above are relatively easy to measure and data related to them have, for the most part, been collected — if not analyzed — by most districts for years. But there were also some indicators that our districts examined that were more difficult to quantify and are not collected widely by most school districts. These indicators include student mobility, special education enrollment, student engagement, and teacher and principal quality, including teacher turnover.
Student MobilityParticularly in urban areas, high rates of student mobility make it more challenging to sustain each student's academic growth. Not surprisingly, districts in this study have found that, controlling for other factors, schools with higher mobility rates have lower student-achievement levels. Given that mobility will continue to be a fact of life for urban districts, the solution may in some respects lie in better data collection. Using a universal student identifier and relying more on technology to collect data are two strategies our study districts are using to improve the accuracy of their data about student mobility.
Special Education Enrollment
Special education students, under No Child Left Behind, receive a great deal of attention due to the need to make adequate yearly progress with all subgroups. All four districts in this study tracked data on special education students, though sometimes these data were not integrated. For example, interviewees in one district mentioned that information in special education students' Individualized Education Programs existed only on paper, not in the district's data warehouse, and that integrating these data was a priority.
Student Engagement
One district used “focus walks” to examine the level of student engagement in classrooms and benchmark “how we want students engaged in learning.” Districts also reported that they did frequent student surveys on topics ranging from technology use to students' social-emotional needs. However, some admitted that student engagement is not easily quantifiable. Thus, there's a belief that student engagement, as defined in myriad ways (e.g., school climate, time on task), is important. But the means of measuring student engagement are limited. The challenge is to develop a richer indicator that is more easily measured and can be understood and acted upon by administrators and teachers.
Because of the often-elusive nature of the concept of teacher quality, districts have approached this issue from a variety of vantage points. Getting easily quantifiable and usable data on teacher quality is complex and difficult.
Because of the often-elusive nature of the concept of teacher quality, districts have approached this issue from a variety of vantage points. At least one district is looking at teacher turnover. Another district has begun looking at measuring teaching practice through a coaching model that requires intensive examination of pedagogy. However, collecting data through this model has proved to be labor intensive, and it is difficult to use the information to train teachers to be more effective. Furthermore, like collecting data on student engagement, getting easily quantifiable and usable data on teacher and principal quality is complex and difficult in all four districts.
Another district has approached the measurement of teacher quality from several angles. The district has implemented a teacher evaluation system and reexamined surveys on teacher satisfaction to determine whether teacher satisfaction had any impact on student achievement. Another survey of teachers and administrators showed that supervisor ratings were meaningful to teachers. The district has also implemented an interview tool that scores teacher applicants and plans to determine whether this tool is, as a central office administrator put it, “actually sorting out who are the best teachers.” The district also tracks teacher professional development and teachers who are released and, with the collaboration of the union, has developed an exit survey.
The Data Wish List
Like most districts, the four districts in our study collect a lot of data. Still, there are areas where the data are thin. We asked our respondents to highlight data that weren't available to them but that they would like to have. While the items on their wish lists are not all leading indicators, the statistics described below could all be a part of a robust system of leading and lagging indicators.
Post-secondary OutcomesAll of our districts were able to minutely dissect student outcomes through the twelfth grade. But as soon as their students graduated, they had limited ways to track them. The ultimate proof of the education that districts provide is neither the students' scores on standardized tests nor their grades, but their success after the end of high school in college or the world of work. It was extremely difficult for our districts to know what happened to their graduates.
Social-Emotional Data on Students
Several of our districts expressed an interest in examining data related to the emotional well-being of their students but had found limited ways to get to these data. Participants from one district almost universally commented on the importance of these data.
The superintendent said,
We have been struggling with issues around diversity — how to tackle it. Interesting question across the district:
kids in this school are tolerant of kids different from them on the survey. The number was still high, but there was a drop from last time. What can we do at the district level to modify what we are trying to do with social-emotional learning?
A district administrator summed it up best, saying:
How to assess social-emotional data is an area where we tend to go by gut rather than data. We need training on what tools are out there, what really is going to inform how we help kids in that area. Lots of research shows that social-emotional concerns can affect achievement. We do have some; we introduced the Manners Matrix and are trying to tweak [it] with social-emotional learning goals, and a school perceptions survey [was] completed recently. [We] got some; we need to collect [data] in a systematic way that will inform our decisions around the social-emotional piece.
The desired social-emotional data are related to the student-engagement data described in the section Student Engagement. Efforts to understand
student engagement are nascent attempts to get at the broader construct
of students' social and emotional development.
Teacher Preparation and Training
Similarly to their interest in issues of teacher quality, the districts were particularly interested in gathering additional data about teacher preparation and training. For example, one district's vendor said:
I also wonder how good universities are doing with teacher preparation for training teachers on how to use data. I doubt [the local university] has very much of this. [A data specialist] is invited once in a while to speak to students, but other than that, I don't know. How do we help our teacher preparation programs and the universities
prepare our teachers better to enter a data-driven system?
A district administrator from the same district said,
Another area we didn't talk about is K-16 — connecting with colleges around matriculation, training teachers.
Going to the schools, sharing information, talking with teachers, and realizing in every building there is something you can learn.
Several district administrators were interested in gathering additional information
from universities about student teachers and teachers coming from their programs:
I would like to gather data from student teachers. Talk to supervisors. What universities and colleges are they coming from? Are there areas where they are lacking? Areas where they excel?
[I would like] more data on the teacher. For example, what college they attended. Is it possible to use school codes like ACT does? Once we get that electronically, we can do more with the teacher piece.
Conclusions
[I would like] more data on the teacher. For example, what college they attended. Is it possible to use school codes like ACT does? Once we get that electronically, we can do more with the teacher piece.
The four districts in our study are among the most advanced in the country in using data to inform their decision making, and other school systems can learn a great deal from their successes and challenges. For us, at least, their experiences offer the following lessons.
- Though they might not be referred to as such, leading indicators for education exist and are being used to differentiate instruction and improve outcomes for students. In some cases these “leading indicators” are simply a prioritization of a few intermediate
outcomes; in others, they are a synthesis of multiple indicators that describe typical student trajectories toward success or failure. Either way, they go far beyond simply examining test scores.
- Many of the leading indicators already in use, such as third-grade reading proficiency and student age compared with credit accumulation, are data sets that school districts have long collected and that are relatively
easy to measure. But there are other indicators that are harder to measure and are essential to understanding
student success. Examples of such indicators include student engagement and teacher quality.
- School district central offices play a critical role in developing leading indicators as one part of a broader data-informed decision-making system.
Central office leaders do this by advocating for equity, especially in terms of outcomes by race and ethnicity; by providing time, infrastructure,
and supports that align all the work of the district; and, perhaps most important, by establishing a data culture, where information is sought out, discussed, and acted upon.
- For all the emphasis on understanding school, student, and teacher performance, there was no comparable focus on measuring the efficacy of central office supports. Central office staff relied primarily on anecdotal evidence to assess whether they were adding value to the work of school-based educators. Central offices need better and more standardized feedback tools for understanding their own effectiveness.
Few school districts have the time, resources, or expertise to collect data on harder-to-measure concepts that reflect the kinds of rich learning environments we want our children to have.
This goal is consistent with the Annenberg Institute's vision of “smart education systems.” Smart education systems bring together schools, community organizations, and civic agencies and institutions to create a web of supports to develop a broad range of outcomes for children and youths. Using data differently is one of the key aspects of smart systems.
As we move forward with our work on data-informed decision making and leading indicators, we will focus on helping districts and communities think broadly about student engagement and figure out how to measure it. We will also collaborate with central offices to gather key information about their own services and more data about policy implementation. We will make efforts to link data, resources, and expertise both within and across specific communities as a kind of data network to advance our understanding of how educators can use and benefit from richer, more powerful, and more timely information.
Warning: include(../permissions_sub.html) [function.include]: failed to open stream: No such file or directory in /Library/WebServer/Documents/annenberg/VUE/winter08/Mishook.php on line 443
Warning: include() [function.include]: Failed opening '../permissions_sub.html' for inclusion (include_path='.:') in /Library/WebServer/Documents/annenberg/VUE/winter08/Mishook.php on line 443
|
Annenberg Institute for School Reform Brown University, Box 1985 Providence, RI 02912 |
Telephone: 401.863.7790 Fax: 401.863.1290 |
 AISR_info@brown.edu
|
© all material AISR