|
INTRODUCTION TO SURVEYS:
Essential Steps for Conducting Surveys
1. CLARIFY YOUR PURPOSE OR FOCUS
The first step in any school or district evaluation process is to clarify what you want to learn from your evaluation or research process. This will help you design your process and choose the most appropriate instrument:
- If your purpose is to get a profile of student achievement, surveys may not help you.
- If you are just beginning a school-improvement process and want to know how the school community – both inside and outside the school – views the current program, surveys alone may be all you need.
- If you have already begun changes and wish to track them, surveys can be useful in combination with other instruments such as observation tools and outcome measures.
Surveys are well matched to some purposes, such as the following:
- to conduct a needs assessment
Across the country, schools are using surveys to uncover the concerns of parents and other stakeholders before those issues become insurmountable problems. In 1996, the Alaska state legislature required schools to find out which issues parents felt were most important. Anchorage schools responded by asking parents to rank teachers on sixteen statements about their knowledge of subject matter, contact with the parents, and understanding and treatment of children. The Rhode Island legislature mandated annual surveys in all communities in the state beginning in 1997. Rochester, New York, initiated parent surveys in 1998.
- to measure school climate
The connection between school climate and learning is becoming increasingly clear. Elements that influence learning include the degree to which students perceive opportunities to participate in planning and decision making, feel classmates are supportive, feel the school is safe, and trust and respect teachers. Also influential are academic motivation, social skills, feelings of self-worth, and concern for others.
- Sample Tool: The School-Climate Survey
The principal and teachers at Concord (CA) High School wanted to reduce the number of students leaving school, and were aware of the importance of school climate. They developed a "homegrown" survey to gather baseline data on student perceptions of the school, and planned to administer the survey every year to judge the effectiveness of their efforts to improve the school climate.
- to measure progress toward change
If you have adopted goals for change and have defined the behaviors to be found in schools and classrooms that meet the goals, then surveys can be helpful for tracking your progress. After establishing a baseline from the first survey, you can administer the same survey items on a regular basis over time. In this way, you can track how different stakeholder groups view or experience the changes.
Top
2. DESIGN YOUR SURVEY STRATEGY
If, after clarifying your purpose, you found that survey data will suit your evaluation needs, it is important to develop a strategy to collect and manage the kind of evidence that you need. In this way, you can avoid ending up with unsuitable data. There are several questions to ask at this stage.
- What evidence do you already have on hand?
- What evidence do you need to collect?
- Which survey instruments will offer the evidence or information you need?
- Does your purpose suggest a time frame?
- From whom will you need to collect data?
- Are there needs specific to your school or district that should influence the design?
We elaborate on the last three questions in this section. We will take up the bigger question of choosing a survey instrument in section 3, Choose a Survey Instrument.
Does your purpose suggest a time frame?
Your school or district may want to look close up at the effectiveness of different aspects of the school program at a particular point in time. Or you may want to step back and look at programs over time, for example, to measure the difference before and after instituting a program change. Each perspective has implications for your survey design.
The first can usually be accomplished by looking once at the views of a cross-section of stakeholders, perhaps along with student outcome data from that same time. The second suggests a longitudinal approach that requires administering the same survey, observation, or test instruments at intervals over time.
> Tip 5: Cross-Sectional and Longitudinal Approaches
From whom will you need to collect data?
If your purpose is to poll the broader community about your school program, then a key question is whether the data you collect will reflect the demographics of your population. You may want to consider administering your survey to a stratified random sample that reflects the community’s demographics rather than distributing it in a mass mailing.
In addition to the demographics of the sample, sample size is important to consider as well. Some of the survey manuals referred to in the Resources section offer suggestions about how to determine the sample size required for your purposes. Once the sample has been determined, in order to maximize the response rate, some schools take advantage of school events to distribute the surveys and to ask participants to complete the surveys on the spot.
Are there needs specific to your school or district that should influence the design?
Sometimes schools need to adapt their survey strategy to meet a specific need. For example, some of the people with whom we consulted found that comparative data are very important to their communities. They chose to administer the same survey across the district or even across the state or the region. The data could be compared across schools and communities because the responses were based on the same criteria. Other schools found that the best way to build a strong commitment to change was to involve stakeholders in developing questions around their concerns. A middle position might be to combine questions from a statewide or national pool with some locally focused items.
Top
3. CHOOSE A SURVEY INSTRUMENT
An essential step in your survey process is to select the appropriate survey instrument. While the connection between the instrument and the purpose is fairly apparent, it is easy to overlook the importance of considering respondent preferences. Some people respond better to an informal interview; others prefer a more anonymous written questionnaire; some find it easier to answer questions with rating scales; others find such scales confining. The method will make a difference in the quality and number of responses.
Three common survey instruments
The written questionnaire is the most familiar survey instrument.
Questionnaires usually pose closed-ended questions that offer options from which to choose. The respondent can complete the questionnaire quickly and independently.
Sometimes questionnaires ask open-ended questions that require more of respondents' time and thought. Less frequently, questionnaires are a combination of both.
> Tip 6: Close-Ended and Open-Ended Questions
Written questionnaires can be mailed to very large numbers of respondents, and the results can be easily quantified and compared among subgroups. Because the questionnaire has to be self-explanatory, its purpose and all of its questions need to be very clear. If the questionnaire is mailed, it also has to be easy to complete and return. Formal questionnaires or polls are good ways to get a wide range of information but a disadvantage of the mailed questionnaire is that the response rate can be very low.
The oral interview is the most common alternative or supplement to the written questionnaire.
In this format, an interviewer poses a set of questions to each individual and records the responses. The oral interview offers the potential of immediately expanding on or clarifying responses by asking more open-ended questions. Interviews can be a useful way of following up on the written questionnaire with a small sample of respondents. Because of the personal contact, the response rate is higher than with mailed questionnaires. However, interviews are more costly; expenses include training the interviewers and analyzing open-ended questions. Comparability of responses is more difficult to achieve in an interview than in a questionnaire.
Focus groups, like interviews, are conducted orally. But, unlike interviews, they bring a number of respondents together in small groups.
A facilitator poses open-ended questions to the group. The tone of these meetings is usually informal and somewhat conversational, which can encourage otherwise reticent respondents to participate. These, like open-ended interviews, take more time than questionnaires. Facilitators need to be trained and groups need to be organized. Because the data are more difficult to record and analyze, the group conversation is sometimes transcribed for later analysis. While focus groups have the advantages of reaching more than one person at a time and of drawing out otherwise reluctant participants, the respondents' views can be influenced by hearing the opinions of other, more articulate members.
Top
4. ANALYZE THE DATA
Three lessons stand out from the schools we have observed and researchers we have consulted.
Survey results are useless unless schools and districts devote time to discussing and using them.
It is important to get help from experts to analyze data.
Unless your survey is very simple, analyzing survey data is not easy. Depending on the purpose of your survey design, the type of instrument you used, and the sample size, you may need to use sophisticated statistical methods. If your questions are open-ended, you may need to establish reliability among your scorers. Check Getting Help or Resources for assistance suggestions.
Data should be analyzed in the context of an ongoing process of inquiry.
Researcher Becky Crowe of Stanford University's Center for Research on the Context of Teaching cautions: "People doing surveys have to make sure their work is embedded in a strong cycle of inquiry. . . . People doing the survey need to know why the questions are being asked and have a plan for using the data, a plan for reflecting on it with more than one person. There needs to be a plan to take action."
To be effective, survey use must be explicitly linked to a larger school-improvement process and a sound system of accountability. One such process – a school self-study cycle of inquiry – is described in the School-Improvement Guide pages of the School-Improvement Planning Tools Web site. A suggested accountability framework is outlined below.
Top
|
|
