Best Practice for Preparing a Proper Survey

June 19, 2019

In the modern digital world, it takes less than 3 minutes to find out what parts of your product or service your audience visits the most frequently, or which portion of your user base contacts you the most. 

Accessing this type of digital data with close to 100% accuracy is easy, but nuanced insight requires thorough research.

Companies should apply a strategic approach to find out why certain visitors don't communicate with your brand, or why most users only engage with limited features within a product.

Making survey results reflect the absolute or objective truth is impossible. All you can do is to take researcher-and-respondent-induced trends and resulting biases into account that lead to deviations in the results.

During the preparation and dissemination phases, try to keep these deviations at the lowest possible level. 

This article will lay out the most common types of survey variations, specific examples in the application, and tips to help you avoid common pitfalls. 

Preventing Response Bias in Surveys

Response bias has many different causes. Types of survey content and excluded target segments can both cause significant distortions in the results.

The image below features the example of types of credit cards, with these options for answers:

  • Visa
  • Mastercard
  • American Express
  • Discover
  • Diners Club

A survey question about types of credit cards with responses, different answer types, and label options

To the right, this example shows tag features that specify answer type:

  • Radio Button
  • Checkbox
  • Dropdown

Lastly, sort functionality options include the below:

  • Default 
  • Ascending
  • Descending
  • Randomized

Surveys often include sensitive subject matter that some people might find difficult to disclose. For instance, tax evaders might be reluctant to participate in a survey conducted by the Ministry of Finance.

If the results of a study exclude a critical relevant group, the results can’t offer an unbiased takeaway. 

The survey medium may also lead to non-responsiveness. Surveys should be accessible to people with disabilities and different levels of education. Since blind or illiterate people can’t respond to a written survey, consider offering an audio version. 

Technology also plays a critical role in survey data collection. E-mail surveys that display incorrectly on mobile phones will fail to account for the views of mobile users.

Minimize response bias by aligning your survey medium with your target audience and if necessary, test it in advance.

A common method of collecting data about sensitive subjects is to blend in questions about the topic with a series of entirely irrelevant ones.

Be Aware of Knowledge Limitations as a Survey Researcher

One of the most critical problems that can be detected is the researcher's lack of knowledge regarding the audience and the survey subject. Taking a survey is fruitless if the data you collect doesn’t overlap with the information your business requires for market research. 

Multiple-choice surveys present another set of problems, especially if surveys don't include common options the target audience are most likely to pick.

If your audience lives in Turkey, make sure to include choices that are available in their country. 

Asking "Which online shopping site do you use most often?" with American-centric multiple-choice options such as Amazon, eBay, and Walmart won’t account for regionally-specific preferences. 

The best way to avoid this situation is by conducting a preliminary survey on a smaller group, using open-ended questions. Include the most popular results they provide in your larger survey to ensure accuracy. 

Account for Survey Respondent Error

Two of the most common respondent-related deviations are incorrect or misleading answers. The first is that the respondent has incorrect answers due to not knowing the answer to the question. Other times, the respondent does not want to give an honest answer.

Not everyone who responds to your survey will have an answer to every question it includes. For instance, a customer support employee might not know how many people work in their company’s marketing department.

Only car owners will indicate how many liters of gas they consume per month, while people who don’t own cars might just mark the smallest value option. Preempt misleading results by offering opportunities for respondents to skip questions, or to answer “not applicable” or ”I don't know.”

Respondents might also avoid picking certain options out of a sense of goodwill. People tend to share opinions that show themselves in the best light. 

Account for this natural human tendency by adding screening questions to your survey.

For the examples above, start off by asking “Which department do you work in?”  “Do you have a car?” to eliminate people who are most likely to give the wrong answers to related questions.

On the other hand, the tendency of the respondent to give the correct answer is usually caused by social desirability.

Respondents may choose to respond in ways that society (or researcher) will welcome, rather than responding to specific questions honestly. 

This type of deviation frequently occurs in intimate matters such as:

  • Alcohol consumption
  • Sexual Orientation
  • Ethical or legal issues

Respondents might refrain from stating a negative opinion about disabled people to an interviewer in a wheelchair.

Accusatory questions such as “Have you ever stolen?” that pose a risk of legal consequences might not always receive honest answers.

Minimize dishonest tendencies by creating an environment that makes respondents feel safe. Formulate questions impartially using language that’s simple to understand. 

Platform Type-Related Survey Deviations

Online surveys, in particular, are the most likely versions where responses won’t always reflect the truth since no interviewer is present to help clarify or assess behavioral responses and reactions. 

Take preemptive measures by considering the best way to go about these factors:

  • How the questions are asked (tone, language usage)
  • Answer formats such as scales, multiple choice, or free text
  • Options provided
  • Visual structure of the survey
  • Question sequence

An example of a series of questions about a website's user-friendliness with responses on a scale from "strongly agree" to "strongly disagree"

 

Formulating questions and the answer choices should be as neutral as possible. Keep these consistent throughout the survey, too.

If you use a scale, apply the same set of labels with similar wording and emotional neutrality.

 “Good - No idea - Bad” has a different impact on respondents than “Love-Like-Dislike,” even though they are evaluations on a 3-point scale.

For lengthy questionnaires, display details such as the total number of and questions to maximize completion rate. 

Positioning difficult or complicated questions at the end of the study may cause fatigue-induced false responses.

If you put five questions on the first two pages of a 5-page survey, participants might get discouraged or bored if they encounter 15 questions on the third page, which increases the risk of opting out without completing it. 

Final Steps: Testing and Audience Selection

Being knowledgeable about the subject of a survey can offer your company valuable insight.

Subject matter experts who partake in your questionnaire may raise the possibility that your questions and response options do not coincide with the facts.

Even if you accounted for every possible deviation while preparing your survey, there’s still an essential step left in the process: testing the survey with a small group.

The best-case scenario is to solicit feedback from people who have experience in survey and market research