METHODS
Research Design | Methodology | Data Sources | Analytic Tools | Protocols

 
Research Design

This six-year study, which began in September 2002, uses a rigorous mixed methods design, using both qualitative and quantitative data. By examining eight community organizing groups across the country, we enhance our ability to draw conclusions about the impacts of community organizing on school reform. In carrying out our research, we have engaged in a collaborative research process with our sites, sharing preliminary findings at each state of our analysis, so that their intimate knowledge of their work and of the school, district, and community contexts can inform our interpretation and understanding of the data.

This research study consisted of two major phases that required different methodologies. In the first, we aimed to develop a conceptual framework linking organizing processes to school and community change. In the second, we used the conceptual framework as a guide in assessing the impact of organizing processes on school and district capacity

The study was guided by two central questions:

• Does the effort to equalize power dynamics — the core of the organizing approach — change the nature of accountability and quality of engagement between schools and communities?
  
  
• Does the new responsiveness and/or collaboration generated by community-based efforts to equalize power dynamics lead to new priorities and capacities within schools and communities that can facilitate successful learning for all children?

Researchers interviewed staff and leaders in each group about the organization's theory and working assumptions about how to stimulate change and, thus, how specific activities link to their goals.

Using each group's theory of change as a starting point, we conducted observations and interviews with community stakeholders to document how the group's strategies were implemented.

Based on these data, we developed an overarching conceptual framework, or logic model, to show how community organizing groups aim to stimulate changes in:

• community capacity, including the ways in which being involved in organizing impacts individual parent, youth, and community members in the group;
  
  
• school and district capacity to educate students successfully.

Phase 2: Assessing Impacts of Community Organizing (2004-2008)

In this phase we employed a mixed-methods strategy to assess the impacts of community organizing. We combined a thematic analysis of our qualitative data with a series of descriptive and inferential statistical analyses on a range of quantitative data. Compiling qualitative fieldwork and quantitative data from multiple sources enabled us to identify points of convergence, as well as areas of divergence that require further investigation.

Our analysis of community capacity drew on interviews, observations of group activities, and surveys of youth and adult members of each organization. Our primary focus in this study, however, has been to understand how organizing campaigns influence school and district capacity. To guide our analysis of school capacity outcomes, we generated a list of the school-improvement campaigns of each group and defined a corresponding list of indicators to serve as key benchmarks in analyzing changes in the capacity of schools targeted by each group's campaigns. We measured changes on each indicator using survey data from teachers; school demographic, resource, and outcome data from publicly available state and district data sets; and interviews with critical stakeholders, including educators, ally organizations, organizers, parents, community members, and youth.
 

• INTERVIEWS. Researchers conducted 321 open-ended, semi-structured interviews with key stakeholders across the eight sites. Interviews were conducted between January 2003 and September 2006. Across the sites, we have collected interviews with organizing staff, parent and youth leaders, educators, interviews with allies, and staff of national organizing networks to which some of the sites belong.

  Interviews in the first year of the study were conducted with organizing staff and leaders and focused on organizational characteristics, including mission, theory of change, leadership development, strategy, and capacity. Early interviews also focused on understanding the impetus for and strategies underlying groups' campaigns for school improvement. To understand the evolution of campaign strategies, we interviewed organizing staff multiple times over the course of the study. Interviewees in subsequent years also included allies, teachers, district administrators, superintendents, and other key stakeholders. These interviews explored the extent to which groups were perceived to be effective and powerful and the ways in which their organizing efforts may have impacted school and district capacity.

• OBSERVATIONS. Researchers observed meetings, training sessions, negotiation sessions, and public actions, again from January 2003 through September 2006. Research team members also observed leader development and organizer development trainings led by the national organizing networks to which some of the sites belong. We compiled more than 75 field notes written by research team members documenting their observations.
  
  
• DOCUMENT REVIEW. We reviewed documentation and archival materials produced by the groups, including newsletters, organizational charts, and training materials, across five years of the study.
  
  
• CONTEXT REVIEW. In addition to conducting extensive background research on the local and state context for each group (e.g., defining the critical policy reforms, state-level issues, governance structure for each school system, political landscape), we followed the local media coverage of education issues in all of our sites (January 2003 to May 2007). We have compiled a database of more than 1700 articles. These articles, combined with the interview data, provide a picture of the shifting context for reform in each site.

• ADULT MEMBER SURVEYS. As a complement to the interviews, observations, and archival research conducted in each site, we conducted a survey of 241 adult members across the eight organizing groups. The paper survey was distributed during a one-month period in October 2003. Survey questions probed member participation in organizing activities, as well as member perceptions of how participation in the group has influenced their engagement with schools and their involvement in their community.
  
  
• TEACHER SURVEYS. We administered online teacher surveys in three sites — Austin, Miami-Dade County, and Oakland — where organizing groups have used a school-based strategy and have mounted highly visible campaigns for several years. Surveys were conducted through the Survey Monkey Web site. We surveyed teachers in MiamiĞDade County Public Schools in spring 2005 and teachers in the Oakland Unified School District and Austin Independent School District in fall 2005. The survey probed teacher perceptions of district and community support and involvement in their school, as well as of their school's climate, professional culture, and instructional core. Survey questions were drawn from a variety of established measures, but primarily from scales developed by the Consortium on Chicago School Research.
  
  In each of the three sites, we conducted surveys of teachers in a sample of schools associated with the group, as well as with a group of demographically similar comparison schools. Schools participating in the survey received a $100 gift certificate in appreciation. In addition, teachers completing the survey received an individual $10 gift certificate.
  
  A total of 509 teacher surveys were collected from the three sites.
  
  
• PUBLIC DATASETS. Baseline statistical data on a variety of community variables were collected for each site. Using the 1990 and 2000 U.S. Bureau of the Census database, we mapped the census tracts that are within or intersect a one-half- to one-and-a-half-mile radius of each school that groups were working in. For each school zone, we then collected data on a variety of neighborhood indicators, such as percentage of school-age children in poverty, median household income, educational attainment, and percentage of homeowners. These data were used to analyze the contexts in which the organizing was taking place.
  
  We have also downloaded a range of publicly available teacher and student data from all eight districts, covering school years from 2000 to 2006. In districts where data prior to 2000 was available, we examined those data as well. Data vary from district to district but include measures of teacher and student race/ethnicity, years of teaching experience, dropout rates, graduation rates, student performance on standardized tests, and a range of other variables.

We have developed an indicators framework to guide our analysis of the impact of each group's organizing. Our conceptual framework posits that school reform organizing attempts to increase the capacity of schools to improve student learning. Therefore, we began by identifying the critical capacities schools need to have to deliver a high-quality learning experience. This analysis drew on a review of school reform literature as well as conversations with education research experts.

Using this analysis of school capacity, we examined our field data to identify key campaigns (both current and past) that were sustained over a long enough period of time to warrant an assessment of impact. We then identified appropriate measures of change in the specific areas of school capacity that the group's organizing aimed to improve. We shared the indicators framework and measures with each site through one-on-one phone conversations. These conversations were particularly important to ensuring that we correctly identified all of the relevant campaigns and corresponding indicators. The indicators framework we have developed is thus the product of an iterative process of analyzing and interpreting the data we've collected and verifying these observations with the organizers who are most familiar with the progress and impact of local campaigns.

Our indicators framework encompasses four broad areas of school capacity: district and community influences; school climate; professional culture; and instructional core. In each area, we have specified indicators that draw on both qualitative and quantitative data. The use of multiple forms of data enabled us to produce a rich picture of how community organizing efforts are influencing domains of school capacity.

Coding Schema and Qualitative Software

We developed an extensive coding schema for all of our qualitative data, including interviews, observations, and media articles. Codes include: organizational characteristics and capacity; education organizing activities, impacts, and challenges; school and district dynamics; and state and national network support. To aid our data analysis, we are using a qualitative software program called NVivo, which allows us to organize our interview, observation, and context data and, thus, to conduct sophisticated "runs" of all forms of coded data. Through these runs, we are able to integrate and analyze data within and across sites, roles, and time periods within one unified database.