Stages of Survey Development

Over the course of the past week or so, I have had several contacts from faculty related to surveys they are in process of developing and for which they wish guidance on IRB application. As a result of looking over some of the proposed surveys I thought I would offer a few comments about the nature of developing them. Czaja and Blair, authors of a fine text on survey development (1), suggest a 5-step process for developing and completing a survey and I thought I would use their approach in offering these comments to you.

Stage 1: Survey design and preliminary planning. This stage specifically looks at the research problem and the research questions that the survey is designed to address. What is your goal in doing this survey? Is it to test a hypothesis, to test a causal model or to estimate the proportion of people who hold certain beliefs or attitudes? In asking this question, there are important issues embedded in it. For example, who is our population of interest? Is it all students at PCC, for example, or a particular trimester, or a specific subgroup from one of these? What is the sampling frame, since it is unlikely we will be able to capture every student in our target population? How will we contact them? In our initial planning, we also need to consider the kinds of questions we will ask, how much time we will need, and how we will analyze the results.

Stage 2: Pretesting. This is where we begin testing our initial design decisions. We need to think about how to reach our sampling frame, what kinds of records we will use for collecting information, how to word questions and so on. We have to draft an initial version of our questionnaire, and it is okay to borrow them from other past research, ensuring that we contact the originator for his or her permission. But keep in mind that past use in a different population is not a guarantee of success in using it in ours. Ru our draft past others for their comments, and incorporate them. Then we need to pretest the questionnaire in a small population of individuals. For example, in the PCCR we might use our research fellows to pretest a questionnaire. Once done, we gather information from that group, either via written comments or via personal interviews and/or focus group meetings, and from their comments we revise our questionnaire accordingly. We need to feel that we are asking what we think we are asking; pretesting helps to accomplish that goal.

Stage 3: Final survey design and planning. This is revision of our initial draft based on the input we have collected. It may also lead us to revise who our sampling frame is or how we analyze the data (we may find that a question we felt was close-ended is actually better asked as an open-ended question, shifting us from a quantitative analysis to a qualitative one.

Stage 4: Data collection. In this stage, you must monitor the results of the sampling and data collection activities. You should also begin coding your answers and preparing your data files ( for example, if you have a “yes- no” question, are you coding the yes response as “1” and the no as “2” or something else? Are you using Excel or SPSS or something else? What do you do with non-answers or double marks? Are you using double data entry, having a second person also code the data for reliability purposes?

Stage 5: Data coding, date-file construction, analysis and final report. This is now where you ensure all your data was properly entered, that you have coded it properly for analysis, that it makes sense when you do if you are looking at anything more than a simple descriptive analysis (how many answered each question).

All of these steps ensure that in the end you collect data that answers questions you are interested in clearly and without miscomprehension of your questions. These are necessary steps; please consider using them as you develop your own surveys.

References
1. Czaja R, Blair J. Designing surveys: a guide to decisions and procedures. Thousand Oaks, CA: Pine Forge Press, 1996:11-30