5.2 Questionnaire Design

People resist a census, but give them a profile page and they’ll spend all day telling you who they are. [Max Barry, Lexicon, 2013]

A questionnaire is a series of questions designed to obtain information on a topic from respondents. Of course, design principles vary depending on the subject and method of data collection, but it is considered good practice to test various questionnaires on random pilot populations before rolling it out on the study population.

5.2.1 Basic Concepts

In general, a questionnaire should

  • be as brief as possible, and free of unnecessary questions;

  • be accompanied by clear and concise instructions;

  • keep the respondent’s interests in mind;

  • emphasize confidentiality;

  • keep a serious and courteous tone;

  • be error-free and attractively presented;

  • be clearly and precisely worded;

  • be designed so that it can be answered accurately, and

  • neatly arranged.

The quality of the collected data depends to a large extent on the quality of the questionnaire – this is a practical aspect of the discipline on which much more time should be spent than on data analysis; reputable survey firms employ specialized teams for questionnaire design.

There is an added challenge for Government of Canada (GoC) federal departments that are collecting and reporting information about the public and representatives of businesses or other entities, including federal public servants: see Public opinion research in the Government of Canada for details. Some of the information presented in this section will overlap with the POR guidelines, but at other times, our (generic) advice will differ. When working with the GoC, the POR guidelines must obviously take precedence.54

5.2.2 Question Types

The basic unit of the questionnaire is, of course, the question, which comes in two forms:

  • closed questions, with a fixed number of predetermined, mutually exclusive, and collectively exhaustive answer choices (and which should always include an “Other (please specify)” category to counteract loss of expressiveness), and

  • open questions, which are used primarily to identify common response choices for use in closed-ended questions in a subsequent questionnaire; any closed-ended question should have been an open-ended question at some point.

In everyday conversation, closed-ended questions are not appropriate:

Asking open-ended questions is a friendly way to approach others in discussions. Knowing the difference between open and closed questions will be invaluable in your career and social life. [How to ask open-ended questions, WikiHow]

In a survey, it is rather open-ended questions that are not appropriate: closed-ended questions require less effort on the part of respondents, and they are generally easier to quantify, allowing more questions to be asked in a restricted amount of time and for a given budget.

For example, compare the two following questions.

Open question: What is the most important issue facing Ontario in 2022?

Closed-ended question: Which of these is the most important challenge for Ontario in 2022?

  • economy and unemployment

  • impact of COVID-19

  • reconciliation with indigenous communities

  • taxes

  • budget deficit

  • the environment

  • organized crime

  • gang violence

  • racism

  • other (please specify)

However, closed-ended questions can also lead to:

  • a loss of an opportunity to test the waters in order to obtain further clarification;

  • introducing response bias by presenting alternatives that respondents would never have thought of, and

  • a potential loss of interest if the choice of answers does not match a respondents’ expectations.

Adding open-ended questions to the questionnaire can mitigate these risks. The use of text analysis and natural language processing methods can also help to extract the main meaning or sentiments of an answer to an open-ended question (see Modules ?? and ?? for details and for limitations of such approaches).

5.2.3 Design Considerations

It is well known that the formulation of questions can influence the responses of a questionnaire; it is good idea to keep the following wording considerations in mind when developing questionnaires:

  • avoid abbreviations and jargon: “Does your organization use TTWQ practices?”

  • avoid using complex terms when simpler terms will do: “How many times have you been defenestrated?” vs. “How many times have you been thrown out a window?”

  • ensure that all respondents can answer the questions, by asking relevant and appropriate-level questions;

  • clarify the framework: “What is your annual income?” vs. “What was your total household income from all sources, before taxes and deductions, in 2021?

  • make the question as accurate as possible: “How much fuel did your moving company use last year?” (answers received: 2,500 liters, 800 gallons, $13500, more than the previous year, etc.) vs. “How much did your moving company spend on fuel last year?”

  • avoid “double-barreled” questions: “Do you plan to leave your car at home and take LRT to work?” vs. “Do you plan to leave your car at home? If so, do you plan to take LRT to work?”, and

  • avoid leading questions: the always excellent Yes, Prime Minister gives a clear-cut example (that is not nearly as facetious as it appears, in the final analysis):

    Yes, Prime Minister | [S04xE02](http://www.youtube.com/embed/G0ZZJXw4MTA?rel=0) | Leading Questions | *The Ministerial Broadcast*

    Figure 5.3: Yes, Prime Minister | S04xE02 | Leading Questions | The Ministerial Broadcast

    On the one hand, Sir Humphrey demonstrates that asking leading questions in a particular order can lead a respondent to support the reintroduction of national service:

    • Are you concerned about the number of unemployed youth?

    • Are you concerned about the increase in teenage crime?

    • Do you think there is a lack of discipline in our comprehensive schools?

    • Do you think young people would appreciate some leadership in their in their lives?

    • Do you think they would respond to a challenge?

    • Would you support the re-introduction of national service in the UK?

    The first five questions are designed and presented in such a way as to elicit support – the obvious answer to each is “yes”.

    After this pattern of agreement, Sir Humphrey launches the crucial question, framed in such a way that it proposes national service as a supposed solution to all the above problems.

    In the second part of the exchange, Sir Humphrey demonstrates that another set of leading questions can lead the respondent to oppose the reintroduction of national service:

    • Does the danger presented by war worry you?

    • Does the arms race worry you?

    • Do you think it is dangerous to arm young people and teach them to kill?

    • Is it a bad idea to force people to take up arms against their will?

    • Would you oppose the reintroduction of national service?

    Sir Humphrey’s first four questions are deliberately designed to produce agreement.

    In keeping with the survey design, the fifth question does the same: a person who answers “yes” to each of these questions is necessarily opposed to the reintroduction of national service. [Based on an idea by Nagesh Belludi].

5.2.4 Question Order

The order in which the questions are presented is as important as their wording. Questionnaires should be designed to be seamless and follow a logical process (from the perspective of the respondents):

  1. begin with an introduction that provides the title, topic and purpose of the survey;

  2. ask for cooperation from respondents and explain the importance of the survey and how the results will be used;

  3. indicate the degree of confidentiality and provide a deadline and contact address;

  4. follow up with a series of easy and interesting questions to build respondent confidence;

  5. group similar questions under the same heading;

  6. only introduce sensitive topics when a relationship of trust is likely to have been established with the respondents;

  7. leave some space and/or time for additional comments, and

  8. thank respondents for their participation.

Questionnaire design is discussed in the following references:

  • Hidiroglou, M., Drew, J. and Gray, G. [1993], “A Framework for Measuring and Reducing Nonresponse in Surveys,” Survey Methodology, v.19, n.1, pp.81-94 [45]

  • Gower, A. [1994], “Questionnaire Design for Business Surveys,” Survey Methodology, v.20, n.2, pp.125-136 [46]

  • Survey Methods and Practices, Statistics Canada, catalogue number 12-587-X [42]

It is worth remembering that without a “sound sampling plan”, collected data may be of such poor quality that it is impossible to use it to draw any meaningful conclusions. It is also essential to capture demographic information that allows classification of units into stratas (StS) or clusters (ClS); we will revisit those concepts in subsequent sections.

Example: Consider the following video:

Transcription of the video

In May, your household will receive a letter to complete the 2021 Census questionnaire. On your letter, you will find a secure access code that allows you to complete the questionnaire online. Once online, you can complete the questionnaire in three easy steps. Simply log on using your secure access code, complete the questionnaire and select “Submit.” If you need help or require a paper version, please call the Census Help Line. For more information or to complete the 2021 Census questionnaire, visit census.gc.ca. It’s safe, quick and easy.

Message from the Chief Statistician of Canada

Thank you for taking a few minutes to participate in the 2021 Census. The information you provide is converted into statistics used by communities, businesses and governments to plan services and make informed decisions about employment, education, health care, market development and more. Your answers are collected under the authority of the Statistics Act and kept strictly confidential. By law, every household must complete a 2021 Census of Population questionnaire. Statistics Canada makes use of existing sources of information such as immigration, income tax and benefits data to ensure the least amount of burden is placed on households. The information that you provide may be used by Statistics Canada for other statistical and research purposes or may be combined with other survey or administrative data sources. Make sure you count yourself into Canada’s statistical portrait, and complete your census questionnaire today.
 
Thank you,
 
Anil Arora
Chief Statistician of Canada

References

[42]
Survey Methods and Practices, Catalogue no.12-587-X. Statistics Canada.
[45]
D. Hidiroglou M. and G. Gray, “A framework for measuring and reducing non-response in surveys,” Survey Methodology, vol. 19, no. 1, pp. 81–94, 1993.
[46]
A. Gower, “Questionnaire design for business surveys,” vol. 20, no. 2, pp. 125–136, 1994.