How to read health research without being fooled

The other guides on this site summarise research and ask you to accept our reading of it. This one is different: it's about the tools to check that reading yourself, and to evaluate the next health headline you encounter without needing anyone's summary.

Almost all recurring failures in health reporting reduce to a handful of patterns. Once you can spot them, most of the noise becomes legible.

Relative risk versus absolute risk

This is the most common single distortion in health reporting, and it's worth mastering first.

Relative risk describes the change as a proportion of the baseline. Absolute risk describes the change in real incidence. They can differ enormously in impressiveness.

Suppose a condition affects 2 in 10,000 people annually, and an exposure doubles the risk to 4 in 10,000. The relative risk increase is 100% — a headline reading "doubles your risk." The absolute risk increase is 2 in 10,000, or 0.02 percentage points. Both are true descriptions of the same finding, and they leave very different impressions.

The rule: a relative risk is uninterpretable without the baseline rate. A doubling of a common risk is serious; a doubling of a rare one may be trivial. When a report gives you percentages without absolute numbers, the most important information is missing — and its absence is often what makes the story publishable.

Reverse causation

The exposure and the outcome are associated, but the outcome is causing the exposure rather than the other way round.

This appears repeatedly in the guides on this site because it explains several major findings. Long sleep looks harmful partly because illness causes long sleep. Low BMI looks harmful partly because disease causes weight loss. Social isolation looks harmful partly because poor health causes isolation.

The tell to look for: could the outcome plausibly have caused the exposure, including years before diagnosis? Chronic diseases have long silent phases, and during them they change behaviour, appetite, weight, sleep, and activity. Any study of behaviour and mortality has to reckon with this.

The standard defences are excluding deaths in the first several years of follow-up, adjusting for baseline health, and using genetic instruments. When you see those in a paper's methods, it's a sign the authors took the problem seriously.

Confounding and the healthy-user effect

A third variable causes both the exposure and the outcome, producing an association with no direct causal link.

The version that dominates lifestyle research is the healthy-user effect. People who adopt one health-promoting behaviour tend to adopt others, and they also tend to differ in income, education, healthcare access, and health literacy. Someone who takes a daily supplement is, on average, also more likely to exercise, attend screening, not smoke, and have money. When that supplement shows an association with better outcomes, disentangling the pill from the person taking it is genuinely difficult.

Statistical adjustment helps but is inherently incomplete: you can only adjust for what you measured, and you measured it imperfectly. Residual confounding is the norm, not the exception.

This is the main reason the alcohol J-curve collapsed. Moderate drinkers in Western cohorts were systematically better-off and healthier than abstainers in ways that adjustment failed to capture — plus the abstainer group was contaminated with people who'd quit because they were ill.

What study designs can and can't show

Design	Can show	Main vulnerability
Case report	That something happened once	Nothing generalisable at all
Cross-sectional	Association at one moment	No time order — can't tell what came first
Case-control	Association, efficiently for rare outcomes	Recall bias; control selection
Prospective cohort	Association with correct time order	Confounding; reverse causation
Mendelian randomisation	Causal inference using genetic variants	Assumes variants only act via the exposure
Randomised controlled trial	Causation, if well conducted	Short duration; selected populations; often infeasible
Meta-analysis	Pooled estimate across studies	Only as good as its inputs; publication bias

Two things people get wrong about this hierarchy. It isn't strict — a large, well-designed cohort study can be more informative than a small, brief, poorly conducted trial. And RCTs are frequently impossible for the questions that matter most here: you cannot randomise people to smoke for thirty years, so essentially all long-horizon lifestyle evidence is observational and must be read with the limitations above.

A meta-analysis deserves particular scepticism about its inputs. Pooling twenty weak studies produces a precise-looking estimate of whatever bias they share. Precision is not accuracy.

Surrogate endpoints

A study measures something assumed to stand in for the outcome you care about — cholesterol instead of heart attacks, bone density instead of fractures, a biomarker instead of illness.

Surrogates are used because they're faster and cheaper. Waiting for heart attacks takes years; measuring cholesterol takes a week. The risk is that interventions can move the surrogate without moving the outcome, and the history of medicine contains a number of cases where a drug improved a marker while failing to improve, or actively worsening, the clinical result.

When you read that something "improves markers of inflammation" or "increases a longevity biomarker," the question to ask is whether anyone has demonstrated that the marker moving in that direction, by that means, leads to people living better or longer. Frequently no one has. This is especially worth remembering in the longevity and supplement space, where surrogate-based claims are the norm.

Why findings reverse

Health advice appears to flip-flop, and this erodes trust. Several distinct reasons, worth separating:

• Small early studies are noisy. An initial striking result is often partly chance. Replication in larger samples produces a smaller, truer effect — regression to the mean, not a reversal.
• Publication bias. Striking positive findings get published and covered; null results often don't. The literature is skewed toward exciting results before it corrects.
• Better methods correct earlier errors. Mendelian randomisation and improved adjustment overturned the alcohol J-curve. That's science working, not failing.
• Reporting amplifies preliminary work. Much apparent flip-flopping is a single small study covered as though settled, then contradicted by the next one.

The useful response isn't cynicism. It's weighting: findings replicated across many independent populations with multiple methods — smoking harms, activity helps, blood pressure matters — have been stable for decades and are unlikely to reverse. Single-study findings about specific foods, supplements, and biomarkers reverse often. Calibrate confidence to which category you're in.

A practical checklist

Applied to any health claim, in rough order of usefulness:

1. Absolute or relative? If only relative numbers are given, find the baseline rate before forming a view.
2. What design? Observational or randomised? If observational, is a causal claim being made anyway?
3. How large is the study, and over how long? Small and brief means preliminary.
4. Could causation run backwards? Ask specifically whether the outcome could have caused the exposure.
5. Who else differs from whom? Name the plausible confounders yourself before checking whether they were adjusted for.
6. Real endpoint or surrogate? Did anyone actually get healthier, or did a number move?
7. Has it replicated? One study is a hypothesis. Several independent ones are a finding.
8. Who benefits from the result? Funding and commercial interest don't invalidate a study, but they justify closer reading.

Applied to this site, that checklist should make you appropriately sceptical of us too. Our factor weights come from observational cohorts with all the limitations described above, our model is additive when reality has interactions, and several of our inputs are self-reported. We've tried to say so in each guide. If you find a place where we've claimed more confidence than the evidence supports, the contact page exists for that.