Survey Development: Continuous Judgments and Direct Estimation Methods

Many of us develop surveys for use in classroom or clinical settings. When we do, we have to consider the variables that we are interested in, and often we find that these are continuous rather than categorical. One method of collecting continuous data is via a direct estimation method, where subjects are required to indicate their response by a mark on a line or by checking a box. One example of such a method would be a visual analogue scale for pain: a line of 10cm anchored with descriptors at one end of “no pain” and at the other with “worst pain imaginable.” But if you think about it, you could ask this question using different means of collecting the data. For example, you could offer the subject a series of choices, such as boxes with these descriptors: no pain, minor pain, moderate pain, severe pain, worst pain. This adjectival scale gives 5 choices from which to select. This is also called a unipolar scale, in that the descriptors range from no or little of the attribute at one end to the maximal amount at the other. This is in contrast to the Likert scale, which is bipolar, which means that it is often used to tap agreement (i.e., strongly disagree, disagree, no opinion, agree, strongly agree). But the choice of a proper scale for a question is important, both for continuous variables or categorical/dichotomous ones.

These are some of the issues you should consider when constructing a survey which uses continuous scaling:

1. How many steps should there be? This is an important consideration; if the number of levels is less than the participant’s ability to discriminate, the results will be a loss of information. Research on this question is unclear, but suggests that 5-7 steps seems to provide best information with little loss of reliability.

2. Is there a maximum number of categories? There seems to be some evidence that people cannot discriminate beyond 7 levels. But keep in mind that it is also known that people have “end aversion,” meaning that they typically avoid the end points of any scale.

3. Should there be an even or odd number of categories? For a unipolar scale, this would not matter, but for a bipolar scale the use of an odd number allows you to build in a “no opinion” option.

4. Should all the points on a scale be labeled, or only the ends? While research indicates that there is relatively little difference between scale with adjectives under each box and end-anchored scales, that same research indicates that respondents are more satisfied when many or all the points on a scale are labeled. Also, if you do not label all boxes, research has shown that people will often select a box that has been labeled, not one that has not.

5. Should the neutral point always be in the middle? Where positive or negative responses to an item are equally possible, it makes sense to have the neutral point in the middle. There are more technical reasons to unbalance a scale but I will not describe them here.

6. Do the adjectives always convey the same meaning? What does it mean to agree or to strongly agree? How does “often” differ from “not too often?” The use of these quantifiers requires care and consideration.

7. Do numbers placed under the boxes influence response? Yes, to be direct. When a set of VAS scales were used in one study, with the only difference being that one scale ran from 0-10 and the other from -5 to +5, a much higher percentage of respondents did not use the lower half of the scale in the latter group (87%), while 34% of the first group did use it.

8. Do questions influence the response to other questions? Yes. People wish to seem consistent and will often refer back to questions when answering new ones. Also, participants often try to interpret what the question is asking them, so they can respond appropriately.

The take home message from all this is that it is very hard to develop good questions in surveys and a great deal of thought and testing needs to go into testing the questions you write before you use them in survey research. This is but a small part of information on this topic, but my reference for this is from the following book: Streiner DL, Norman GR. Health measurement scales: a practical guide to their development and use, 3rd edition. New York City, NY; Oxford Press, 2003.