Methods
Research Design | Methodology | Data Sources | Analytic Tools | Protocols
RESEARCH DESIGN
This six-year study, which began in September 2002, uses a multi-site case study design. In multi-site case studies, researchers choose several similar cases -- in our research, the organizing group functions as a "case"-- and compare and contrast findings across sites. By examining eight community organizing groups across the country, we enhance our ability to draw conclusions about the impacts of community organizing on school reform. In carrying out our research, we have engaged in a collaborative research process with our sites, sharing preliminary findings at each state of our analysis, so that their intimate knowledge of their work, and of the school, district, and community contexts, can inform our interpretation and understanding of the data.
This research study consists of two major phases. In the first, we aimed to delineate a conceptual framework linking organizing processes to school and community change. In the second phase, we use the conceptual framework as a guide in assessing the impact of organizing processes on school and district capacity. Each phase of the research required different methodologies.
METHODOLOGY (top)
Phase 1: Creating a Conceptual Framework (2002 - 2004)
We used a participatory theory-of-change methodology to define the organizing processes and school reform goals and strategies in each site and to illuminate how these strategies were shaped by the context (community, school, district, municipal, and state). This phase of the research relied primarily on qualitative data.
- Researchers interviewed staff and leaders in each group about the organization's theory and working assumptions about how to stimulate change and, thus, how specific activities link to their goals.
- Using each group's theory of change as a starting point, we conducted observations and interviews with community stakeholders to document how the group's strategies were implemented.
- Based on these data, we developed an overarching conceptual framework, or logic model, to show how community organizing groups aim to stimulate changes in: 1) community capacity, including the ways in which being involved in organizing impacts individual parent, youth, and community members in the group, and; 2) school and district capacity to educate students successfully.
Phase 2: Assessing Impacts of Community Organizing (2004 - ongoing)
In this phase, which is currently underway, we are employing a mixed methods strategy to assess the impacts of community organizing. We combine a thematic analysis of our qualitative data with a series of descriptive and inferential statistical analyses on a range of quantitative data. Triangulating qualitative fieldwork and quantitative data collection from multiple data sources enables us to identify points of convergence, as well as areas of divergence that require further investigation.
Our analysis of community capacity draws on interviews, observations of group activities, and surveys of youth and adult members of each organization. Our primary focus in this study, however, has been to understand how organizing campaigns influence school and district capacity. To guide our analysis of school capacity outcomes, we generated a list of the school improvement campaigns of each group, and defined a corresponding list of indicators to serve as key benchmarks in analyzing changes in the capacity of schools targeted by each group's campaigns. We measured changes on each indicator using survey data from teachers; school demographic, resource, and outcome data from publicly available state and district data sets; and interviews with critical stakeholders, including educators, ally organizations, organizers, parents, community members, and youth.
Data Sources (top)
The qualitative and quantitative data sources informing our analysis are described below in greater detail.
Qualitative Data Sources
Interviews
Researchers conducted 336 open-ended, semi-structured interviews with key stakeholders across the eight sites. Interviews were conducted between January 2003 and September 2006. Across the sites, we have collected 160 interviews with organizing staff, 77 interviews with parent and youth leaders, 56 interviews with educators, 28 interviews with allies, and 15 interviews with staff of national organizing networks, to which some of the sites belong.
Interviews in the first year of the study were conducted with organizing staff and leaders, and focused on organizational characteristics -- including mission, theory-of-change, leadership development, strategy, and capacity. Early interviews also focused on understanding the impetus for and strategies underlying groups' campaigns for school improvement. To understand the evolution of campaign strategies, we interviewed organizing staff multiple times over the course of the study. Interviewees in subsequent years also included allies, teachers, district administrators, superintendents, and other key stakeholders. These interviews explored the extent to which groups were perceived to be effective and powerful, and the ways in which their organizing efforts may have impacted school and district capacity.
Observations
Researchers observed meetings, training sessions, negotiation sessions and public actions, again between January 2003 through September 2006. Research team members also observed leader development and organizer development trainings led by the national organizing networks, to which some of the sites belong. We have compiled more than 75 field notes written by research team members documenting their observations.
Document Review
We reviewed documentation and archival materials produced by the groups, including newsletters, organizational charts, and training materials, across five years of the study.
Context Review
In addition to conducting extensive background research on the local and state context for each group (e.g., defining the critical policy reforms, state-level issues, governance structure for each school system, political landscape), we followed the local media coverage of education issues in all of our sites (January 2003 - May 2007). We have compiled a database of more than 1700 articles. These articles, combined with the interview data, provide a picture of the shifting context for reform in each site.
Quantitative Data Sources
Adult member surveys
As a complement to the interviews, observations, and archival research conducted in each site, we conducted a survey of 241 adult members across the eight organizing groups. The paper survey was distributed during a one-month period in October 2003. Survey questions probed member participation in organizing activities, as well as member perceptions of how participation in the group has influenced their engagement with schools and their involvement in their community.
Youth member surveys
We collected 124 surveys from the three youth organizing groups in our study -- Sistas and Brothas United in New York, South Central Youth Empowered thru Action in Los Angeles, and Youth United for Change in Philadelphia, in June through August 2006. Surveys asked young people about their involvement in organizing, and the impact of their involvement on their worldview, sense of agency, political engagement, academic motivation, and knowledge of the school and school system.
Teacher surveys
We administered online teacher surveys in three sites -- Austin, Miami-Dade, and Oakland -- where organizing groups have used a school-based strategy and have mounted highly visible campaigns for several years. Surveys were conducted through the Survey Monkey website. We surveyed teachers in Miami-Dade County Public Schools in Spring 2005 and teachers in the Oakland Unified School District and Austin Independent School District in Fall 2005. The survey probed teacher perceptions of district and community support and involvement in their school, as well as of their school's climate, professional culture, and instructional core. Survey questions were drawn from a variety of established measures, but primarily from scales developed by the Consortium on Chicago School Research.
In each of the three sites, we conducted surveys of teachers in a sample of schools associated with the group, as well as with a group of demographically similar comparison schools. Schools participating in the survey received a $100 gift certificate in appreciation. In addition, teachers completing the survey received an individual $10 gift certificate.
A total of 509 teacher surveys were collected from the three sites.
Public datasets
Baseline statistical data on a variety of community variables were collected for each site. Using the 1990 and 2000 U.S. Bureau of the Census database, we mapped the census tracts that are within or intersect a one-half- to one-and-a-half-mile radius of each school that groups were working in. For each school zone, we then collected data on a variety of neighborhood indicators, such as percentage of school-age children in poverty, median household income, educational attainment, and percentage of homeowners. These data were used to analyze the contexts in which the organizing was taking place.
We have also downloaded a range of publicly available teacher and student data from all eight districts, covering school years from 2000 - 2006. In districts where data prior to 2000 was available, we examined those data as well. Data vary from district to district but include measures of teacher and student race/ethnicity, years of teaching experience, dropout rates, graduation rates, student performance on standardized tests, and a range of other variables.
Analytic Tools (top)
Indicators framework
We have developed an indicators framework to guide our analysis of the impact of each group's organizing. Our conceptual framework posits that school reform organizing attempts to increase the capacity of schools to improve student learning. Therefore, we began by identifying the critical capacities schools need to have to deliver a high-quality learning experience. This analysis drew on a review of school reform literature as well as conversations with education research experts.
Using this analysis of school capacity we examined our field data to identify key campaigns (both current and past) that were sustained over a long enough period of time to warrant an assessment of impact. We then identified appropriate measures of change in the specific areas of school capacity that the group's organizing aimed to improve. We shared the indicators framework and measures with each site through one-on-one phone conversations. These conversations were particularly important to insuring that we correctly identified all of the relevant campaigns and corresponding indicators. The indicators framework we have developed is thus the product of an iterative process of analyzing and interpreting the data we've collected and verifying these observations with the organizers who are the most familiar with the progress and impact of local campaigns.
Our indicators framework encompasses four broad areas of school capacity: a) district and community influences; b) school climate; c) professional culture; and d) instructional core. In each area, we have specified indicators that draw on both qualitative and quantitative data. The use of multiple forms of data enables us to produce a rich picture of how community organizing efforts are influencing domains of school capacity.
Coding schema and qualitative software
We developed an extensive coding schema for all of our qualitative data, including interviews, observations, and media articles. Codes include: organizational characteristics and capacity; education organizing activities, impacts, and challenges; school and district dynamics; and state and national network support. To aid our data analysis, we are using a qualitative software program called Nvivo 7, which allows us to organize our interview, observation, and context data and, thus, to conduct sophisticated "runs" of all forms of coded data. Through these runs, we are able to integrate and analyze data within and across sites, roles, and time periods within one unified database.
Protocols (top)
Our interview and survey protocols are available on a password-protected website. To access this website please contact: Seema Shah.