It’s surprising but true. One of the most common kinds of survey questions, used widely in course evaluation surveys, employee attitude surveys and other surveys of opinion or attitude in education and training, is actually a source of survey error and should be avoided.
It’s very common in surveys to ask whether a respondent agrees or disagrees with a statement (sometimes called a Likert Scale question). Here’s an example:
My overall health is excellent. Do you:
- Strongly Agree
- Agree
- Neither Agree or Disagree
- Disagree
- Strongly Disagree
Is this a good question or a poor one? The answer: it is usually a poor question. It’s much better to replace it with something direct. Here’s an example:
How would you rate your health overall?
- Excellent
- Very good
- Good
- Fair
- Bad
- Very bad
One reason why agree/disagree questions are weak is due to something called “acquiescence bias”.
People are much more likely to agree with a statement than disagree with it. Whether due to people wanting to be polite or just because they don’t want to put in the effort, people tend to agree with survey statements much more than they disagree. This is a particularly strong tendency for some of those with less formal education, who are more likely to agree, as they may think the survey author knows more than they do. This factor has been shown repeatedly in research. On average, studies have shown that with two questions asking the same thing in different ways, around 14% more people agree with an assertion compared to those who express the same point of view via a direct question.
Another reason that agree/disagree questions should be replaced is that they require the respondent to evaluate the statement and then answer. The respondent has to think “what do I feel about the statement?” and then “how should I answer?”, and this complexity reduces how accurate the results are. Furthermore, the distinction between “Strongly Agree” and “Agree” is imprecise and different people will select one or the other with different criteria.
To quote learning researcher Will Thalheimer in his excellent book on smile sheets (course evaluation surveys) “Performance-focused smile sheets”:
“Likert-like scales are ubiquitous on smile sheets – but they are very problematic and should be avoided in most circumstances”
For more information on why these questions can be improved on, read Will’s’ book, see this survey research article by eminent researchers Krosnick and Presser or read my article on good survey practice on the Questionmark blog.
It takes a bit more time to write direct questions than agree/disagree ones, but you will get more accurate results.
If you have an existing survey with past results and you want to ask it again to compare trends from today with those of the past, it can make sense to stick with the previous question format for comparison purposes. But if you are writing a new survey (in SAP Assessment Management or other systems), it's usually better to ask the question directly, not make a statement and ask the respondent whether they agree with it.
Agree or disagree or have your own thoughts – feel free to comment below!
