It is essential to check to be sure surveys – homegrown and prepackaged – are both reliable and valid.

Reliability is the consistency of the measurement, either across like respondents or across administrations of the survey. In a written questionnaire this means that the same item measures the same thing across similar respondents; if you ask a number of questions about the same factor, all should receive similar responses. The same applies in an interview. In addition, an interview requires reliability across interviewers – each asking the question in the same way. One way to increase reliability is to ask a number of different questions about the same phenomenon. (Note: A measurement that combines the responses to a number of related survey items into one score in order to measure a bigger idea or phenomenon is frequently called a survey scale.) Piloting or pretesting the survey can also help you check its reliability. If you use prepackaged surveys, be sure they describe the reliability of the instruments.

Validity is the degree to which the instrument measures what it is intended to measure. One essential validity question is, "What is the match between information provided by the instrument and what we want to know – the inferences we want to draw from the results?" (Herman and Winters 1992, p. 37 – see Resources). This means that a lot of thought has to go into making clear questions and aiming them accurately at the kind of information you want.

Piloting or pretesting the survey can help increase both the validity and reliability. Using survey scales – combining a number of related items into one score to measure a large phenomenon can do so as well.