“There are three kinds of lies: lies, damned lies, and statistics.”
This is part of a series we’re thinking of running on decision making biases. There are tons of well documented biases that hinder our ability to actually make rational, well reasoned decisions and we thought it might be fun to talk about a few of our favorites. One of my personal favorites is selection bias.
Selection bias basically describes what happens when you cherry pick the data, sometimes on purpose to make a point but most of the time we’re unaware that our expectations heavily influence the way we use information to draw inferences. Not surprisingly this can completely invalidate conclusions drawn from the data and send us all in the wrong direction when seeking guidance.
Don’t believe everything you read
This is the type of bias that many pop scientists fall prey to. You know, the guys who write the thousand-plus-word think pieces for the New York something or other? Do not confuse them with the “hard” scientists who write detailed reports in peer-reviewed journals. The pop scientists’ job is to distill the incredible detail in the “hard” scientists’ reports and make it palatable for the public. And that’s totally fine… Except when they screw up. Take this example from The Scientific American,
In 1993 the columnist George Will was mathematically correct when he wrote in the Washington Post that “the ten states with the lowest per-pupil spending included four – North Dakota, South Dakota, Tennessee, Utah – among the ten states with the top SAT scores. Only one of the ten states with the highest per-pupil expenditures – Wisconsin – was among the ten states with the highest SAT scores. New Jersey has the highest per-pupil expenditures, an astonishing $10,561…New Jersey’s rank regarding SAT scores? Thirty-ninth.”
And herein lies the problem with pop science sensationalism: it neglects to mention hugely important details like this,
One important fact: in New Jersey all college-bound students take the SAT, whereas in North Dakota, South Dakota, Tennessee, and Utah, only the kids applying to out-of-state schools take the SAT. And you can bet these students applying out of state are the cream of the crop. This is selection bias, and it pops up everywhere.
Scientists vs writers
So that’s selection bias for ya. The real problem, however, is how a vital detail, like the one about New Jersey above, gets lost on it’s way to the headlines. It’s like a game of telephone: It starts with the “pure” scientist who conducts the study and publishes the report. They are precise; rigorous.
Then, the writer/pop scientist gets a hold of the report, and that person’s job is to create a story. They’re not peer reviewed, but they are fact-checked. It’s not like they’re fudging numbers, they just want to be interesting. So, he/she reports “the raw numbers” leaving out the all-important difference in student testing between the states that makes the numbers less compelling. Then the piece gets picked up by a major news outlet, many page views and social shares ensue, many readers are fleeced, and that’s that. End of story.
Caveat emptor: Be Skeptical
The point is that this science writing thing is getting more and more popular. And I don’t mean the peer-reviewed stuff, I mean the distilled, statistics-for-dummies, pop science stuff that doesn’t account for selection bias or any number of 56 experimenter’s biases. Finance is regularly hit by the tsunami of “new” research with conclusions and recommendations meant for you to use in managing your client’s money. While some good research is being done, particularly in behavioral economics, much of what we read is crap. Well-packaged but it’s just lipstick on a pig.
New research is critical and can be very illuminating. It can also be very misleading. Take your statistics with a grain of salt, always. Look at the underlying data for an article. Ask yourself what might be missing or left out, who is telling the story and why. Think for your self, draw your own conclusions apart from the author’s. Your clients are depending on you.