Five questions that catch the tiny base, the cherry-picked window and the missing spend figure.
CTFM Team
Case studies are the main way marketing knowledge spreads, and they are almost all written by the party that benefits from them looking good.
That does not make them useless. It means you have to read them with a specific set of questions. Here are the questions, and the tricks they catch.
This applies to our case studies too
Everything below is a standard you should hold this site to as well. If a page here reports a result without a baseline, a time window or a spend figure, you are entitled to be sceptical of it.
"Increased sign-ups by 300 percent" means nothing without the starting number. Three to twelve is a 300 percent increase. So is 3,000 to 12,000. Only one of those is interesting.
What was the time window, and who chose it?
If the window is oddly specific, such as a seven-week period, ask what happened in week eight. Windows are frequently selected after the fact to contain the good part.
What did it cost?
A result with no spend figure is not a result. "We tripled revenue" while quadrupling ad spend is a story about shrinking margins, told as a success.
What else changed at the same time?
Most case studies describe one change. Most companies make several. If they also launched a product, hired a salesperson, or ran a seasonal promotion, the attributed cause may not be the cause.
Who is telling me, and what do they sell?
An agency's case study about their own work, or a vendor's case study about their own tool, is a sales document. It can still be true. It has simply been filtered.
The most common one. Percentages from small numbers produce enormous figures with no substance.
Reported
What happened
"400% increase in leads"
2 leads became 10
"Doubled conversion rate"
0.4% became 0.8%, within normal variance
"10x growth in traffic"
300 visits became 3,000, on a new site
None of these are lies. All of them are chosen because the percentage sounds better than the number.
Always convert back to absolute numbers
If a case study gives you only percentages, that is a signal in itself. Ask for the raw counts. If they are not available, assume they are unimpressive, because a genuinely large absolute number would have been the headline.
The question a case study almost never answers: what would have happened anyway?
A company that grew 40 percent after a campaign, in a market growing 35 percent, has a much less interesting story than the headline suggests. Seasonality does this too. Retail results from November are not comparable to results from February.
"We started posting daily and revenue doubled." Possibly the posting caused it. Possibly they also hired two salespeople, or a competitor collapsed, or their category had a good year.
Watch for the word "after" doing the work that the word "because" has not earned.
If a case study reports engagement, impressions and reach but not revenue, retention or cost, ask why. The metrics chosen tell you which numbers were good.
A reliable tell
The more impressive-sounding the metric and the further it is from money, the more likely it was selected because the money metric was disappointing.
Useful as a checklist for reading, and for writing your own.
Present
Starting numbers. End numbers. Absolute values, not just percentages. Time period with a reason for its length. Total spend. What else changed. What did not work. Who is telling you and what they sell.
Absent
Percentages with no base. Vague timeframes. No cost figure. A single clean cause. Only flattering metrics. No mention of anything that went wrong.
That last item matters more than people expect. A case study that mentions what failed is dramatically more credible, because selective reporting is the default and honesty about failure is costly to fake.
The same discipline applies to any marketing statistic, and it is worth practising because the field is full of numbers with no origin.
Try following a familiar statistic to its source. Search the exact phrase in quotes and see where the citations lead. Frequently you find blog posts citing blog posts citing nothing, a chain that terminates in a dead link or a company's own marketing material.
Two well-known examples of claims that do not survive this test: the story that the Chevrolet Nova failed in Latin America because "no va" means "it doesn't go", which Snopes has debunked and which the car's actual sales contradict, and the claim that human attention spans are shorter than a goldfish's, which a BBC investigation traced to an analytics note about roughly 25 people.
Famous is not the same as verified
Both of those claims appear in marketing textbooks and conference talks. Neither is true. Treat the fame of a claim as a reason for more scepticism, not less, because widely repeated things get repeated rather than checked.
