← Back to the principles list

Survey Bias in ecommerce

Next principle →

Free answers often lead to wrong guesses about what shoppers will buy.

Ticking the green tea soda on a survey costs nothing, and it can sound like the healthy choice. Asked to pay in advance, many of the same people go for the peach soda they really want.

Survey bias
Made-up example: customers said one flavor and paid for the other.

Survey bias, the kind also called response bias, is when the way you ask a question, or people's wish to look good, bends their answers away from what they really think or do.

Keep surveys for finding problems and learning the words shoppers use. Before a big spend, check the answer against something that costs people money, such as a paid pre-order. Asking what someone last bought usually tells you more than asking what they might buy.

Survey bias example: choosing the flavor for a $40,000 soda batch

A soda brand can afford one $40,000 batch, in Green Tea or Peach & Honey. The team asks customers which they would buy. Green Tea gets 82% and Peach & Honey gets 18%, so the team brews Green Tea and $40,000 goes to unsold cans. Now run the same made-up story with paid pre-orders deciding. Of the customers who pay in advance, 12% pick Green Tea and 88% pick Peach & Honey. The team brews Peach & Honey and wastes $0. Saying you would buy something is free, and paying for it shows what you really want.

How survey bias works with other principles: 4 store examples

Survey bias bends what people tell you. These four pairs separate it from two look-alikes and show two other biases that can push answers further off. Every store below is made up to show the idea.

Survey bias vs the observer-expectancy effect

Often confused with: Survey bias usually comes from the question and the setting, while the observer-expectancy effect comes from the person running the session, whose hopes leak out and steer people.

Illustrative store example: A sofa store's form asks, “How comfortable does our new sofa look?”, and that wording is survey bias. In person, the owner smiles whenever testers reach her favorite feature, which is the observer-expectancy effect, so have someone neutral read from a script.

Observer-expectancy effect: how your hopes can steer a test →

Survey bias vs survivorship bias

Often confused with: Survey bias bends the answers people give, while survivorship bias decides who is left to answer at all.

Illustrative store example: A garden tool store emails buyers, “Wasn't our delivery fast?”, and the wording pushes them toward yes. Fix the question and the shoppers who quit over shipping costs still never see it, so ask people as they leave too.

Survivorship bias: why the shoppers who left never reach your data →

Survey bias and the spotlight effect

Works with: The spotlight effect makes people feel more watched than they are, and feeling watched can push survey answers toward whatever looks good.

Illustrative store example: A men's grooming store asks how often customers trim their nose hair, on a form with their name at the top. Some picture a staff member judging the answer and pick the tidy one, so drop the name field and say the form is anonymous.

Spotlight effect: why people feel more watched than they are →

Survey bias and self-serving bias

Works with: Self-serving bias makes people credit themselves for wins and blame outside things for failures, so a survey about what went wrong can lean toward the store's faults.

Illustrative store example: A sewing machine store asks why a project went badly, and most answers blame the machine while few mention skipping the manual. Add a question about what they tried first, and read the support chats before you change the product.

Self-serving bias: why customers credit themselves and blame the product →

Put it into practice

How do I write survey questions that don't push people toward an answer?

Ask about things that already happened. “When did you last buy a sugar-free drink?” tells you more than “Would you buy one?” Cut friendly words like “love” and “easy” from the question, since they hint at the answer you want. Give a “none of these” choice, shuffle the order of the options and skip the name field when the topic is personal.

Can a paid pre-order mislead me too?

Yes. Pre-orders mostly reach fans who already follow you, so they can overstate how shoppers you haven't met will behave. A big early discount also draws bargain hunters who won't pay full price later. Treat a pre-order as a better clue than a survey, and keep the first batch small when you can. Give a clear delivery date and an easy refund so nobody feels tricked.

Is survey bias the same as sampling bias?

They are related, and people often use “survey bias” for both. Sampling bias means you asked the wrong people, such as only your most loyal email subscribers. Response bias means you asked the right people and the question or the setting bent their answers. Check both: who got the survey, and how the question would sound to someone who doesn't like you yet.

Why do people say one thing and buy another?

A survey answer costs nothing, so people often answer as the person they hope to be: healthier, greener, more careful with money. Some also pick what sounds good to whoever is asking. At the checkout, price, taste and habit get a vote too. Most of them aren't lying. People are often poor at predicting what they will do later, so past purchases tell you more.

How do I check a survey result before I spend on it?

Turn the winning idea into a small real offer. A limited pre-order works, and so does a “Join the waitlist” button on something labeled “Coming soon.” Some people call that a fake door test. Compare how many people act with how many said they would. Waitlist clicks are weaker evidence than money, so treat them as an early sign. If the numbers disagree, believe the people who acted.