Great Moments in Selection Bias

I conducted an informal survey today, asking 22 residents of my hometown, Charlotte, North Carolina, what they liked best about our city.


Jana Barrett

February 14, 2014

Back to Resources

I conducted an informal survey today, asking 22 residents of my hometown, Charlotte, North Carolina, what they liked best about our city. Here are the results:


Do those results surprise you? (Keep in mind, Charlotte averages about 5 inches of snow per year.) Maybe a picture from this morning will help explain things:


That’s right. The Queen City got our once in a decade blizzard today and I conducted the survey while we were walking through our snowy neighborhood. The error I committed in my survey is an example of a problem called Selection Bias.

Let me save you a click:


In a simple sense, selection bias occurs when data is collected from a biased group. Here are some other examples:

Survey Idea: Ask 100 fans to name their favorite baseball team.Introduction of Selection Bias: Select participants at Fenway Park in Boston.

Survey Idea: Ask 50 people if they intend to drink so heavily tonight that they may need to call a taxi.Introduction of Selection Bias: Conduct survey on December 31st

Survey Idea: Ask 100 people on Election Eve 2012 whether they intend to vote for Obama or Romney.Introduction of Selection Bias: Only survey people who have landline telephones. (This one really happened, by the way. That poll suggested that Romney was going to win by 14 points.)

In much the same way, it’s easy to unintentionally use today’s survey technologies in a highly biased way. For example, here’s a survey that landed in my inbox recently:


Ask yourself, would you labor through this pinch-zoom obstacle course on your phone? Neither would I. Neither would most people.

Somewhere, someone in corporate America spent real time thoughtfully crafting this survey. But the survey tool they used betrayed them at a basic level by automatically biasing them away from mobile users.

At this point, someone might ask: So what? Why should I care if mainly desktop users answer our survey? (It’s a good question.) Well, here’s one reason of many: When you skip mobile, you disproportionately skip the younger generation.

mobile ages

That’s part of the idea behind GetFeedback. You make the survey, we’ll make it look good – no matter which platform the recipient takes it on.

By the way, the snow was melting by afternoon. Here’s what sunset looked like:


Subscribe for the lastest CX content

© 2021 GetFeedback, Inc.|888-684-8821|Privacy notice|California privacy notice|Terms of use|Cookie policy

*Net Promoter, Net Promoter System, Net Promoter Score, NPS and the NPS-related emoticons are registered trademarks of Bain & Company, Inc., Fred Reichheld and Satmetrix Systems, Inc.