Statistical Fallacies and Biases in Clinical Research

When Numbers Deceive and Minds Comply: Recognizing and Resisting the Illusion of Causality

Introduction

Clinical research is not immune to cognitive illusions. While statistics may offer a veneer of objectivity, they are often interpreted through the fog of human bias. This lesson confronts the subtle, but potent, errors in reasoning that pervade clinical science and practice, with particular focus on causal illusions: false beliefs in causal relationships arising from flawed study design, data interpretation, or human judgment.

We now bring the series full circle: from introducing p-values and confidence intervals to confronting the human tendencies that persistently mislead us, even when the statistics seem sound. These are not just “errors in math”, they are errors in meaning.

Study Designs Answer Different Questions

Before we explore specific fallacies, it’s important to zoom out and consider the landscape in which they occur. Not all clinical studies are built the same. And crucially, not all are equally vulnerable to bias, or equally capable of reflecting the complexity of real-world care.

Clinical study designs differ in the questions they can answer, the assumptions they require, and the biases to which they are vulnerable. They do not occupy a single continuum from causal rigor to real-world relevance:

• Randomized controlled trials use random assignment to make treatment groups exchangeable in expectation. Explanatory trials may emphasize tightly specified conditions, while pragmatic trials may be conducted in routine care. Randomization does not prevent attrition, nonadherence, measurement problems, selective reporting, or limited transportability.

• Observational studies investigate exposures without random assignment. They can represent varied populations and settings or highly selected ones. Causal interpretation requires explicit assumptions about confounding, selection, measurement, timing, and the analysis used to address them.

• Case reports and case series provide detailed observations about unusual or informative events. Without a concurrent comparison, they usually cannot distinguish treatment effects from natural history, regression to the mean, cointerventions, expectations, or measurement change. Their detail is valuable, but detail alone does not identify a cause.

Here’s the core lesson:

Changing design changes the inferential task. A pragmatic randomized trial can have substantial contextual relevance; an observational dataset can be highly restricted; and a detailed case can reveal a signal without estimating a treatment effect.

None of these designs is “wrong.” Each serves a purpose. But each amplifies different risks, and those risks must be understood if we are to interpret findings with epistemic care.

Good appraisal therefore asks two separate questions:

• What claim can this design identify under its assumptions?

• To which population, intervention, outcome, and setting might that claim transport?

Confusing those questions creates fertile ground for statistical, causal, and narrative errors.

Where an Error Enters the Inquiry

One useful dimension is where a problem enters. It may enter through study design and conduct, through interpretation of an otherwise sound result, or through unstructured learning from experience. This dimension is different from the kind of problem involved. Selection bias is a study bias; confounding threatens a causal contrast; multiplicity is a statistical problem; and confirmation bias is cognitive. Several can occur in the same claim.

Design and Conduct

Eligibility, treatment assignment, follow-up, measurement, analysis, and reporting determine which comparisons are available and whether they remain credible. Randomization addresses baseline treatment assignment; it does not repair every later source of bias.

Interpretation

A well-estimated result can still be exaggerated by treating statistical significance as practical importance, ignoring base rates, reversing causal direction, overlooking uncertainty, or transporting the estimate beyond the population and setting it supports.

Experience and Narrative

Firsthand experience can detect patterns and generate hypotheses, but it does not provide its own counterfactual comparison. Temporal sequence, vivid outcomes, repeated action, and a coherent story can produce strong causal belief even when alternative explanations remain compatible with what was observed.

These illusions dominate at the case report and anecdotal level, where context is rich but control is absent.

• “I did X and then Y happened, so X caused Y.”

• “When I use this technique, patients feel better — so it must work.”

These are examples of personal induction: the natural tendency to extract causal meaning from co-occurrence, especially when outcomes are emotionally or socially reinforced. Such observations can begin inquiry, but their causal interpretation requires structured comparison and attempts to discriminate among alternatives.

These stories are powerful because they feel true, and because they’re rarely subject to structured scrutiny.

The categories below classify what kind of inferential problem is present. They are not mutually exclusive and should not be treated as a one-label taxonomy.

Core Fallacies and Biases in Clinical Research

Below we distinguish statistical errors, study biases, causal fallacies, cognitive biases, and narrative errors.

Statistical Fallacies

Multiplicity and Selective Emphasis

What it is: Testing many hypotheses increases the probability of at least one false-positive result when error rates are not addressed. Selectively emphasizing favorable analyses adds a reporting problem.

When to look for it: Any study reporting many outcomes, subgroup analyses, or testing multiple variables without adjusting p-values or correcting for false discovery.

How to guard against it:

• Use correction procedures (e.g., Bonferroni, False Discovery Rate).

• Pre-register primary outcomes.

• Evaluate effect sizes, confidence intervals and consistency rather than isolated p-values.

Regression to the Mean

What it is: The statistical tendency for extreme values (high or low) to move closer to the average on subsequent measurement, independent of intervention.

When to look for it: Especially common in before-and-after studies, pilot trials, or case reports where patients are selected based on unusually severe symptoms.

How to guard against it:

• Include a comparison group or randomized control.

• Interpret single-subject or uncontrolled improvement cautiously.

• Consider baseline variability when evaluating treatment effects.

Statistical Significance as Practical Importance

What it is: Treating evidence against a null value as evidence that the estimated effect is large, precise, patient-relevant, or worth its costs and harms.

How to guard against it: Examine the effect estimate, compatibility or credible interval, meaningful-effect threshold, outcome validity, and consequences rather than the p-value alone.

Base-Rate Neglect

What it is: Ignoring prior probability or prevalence when interpreting predictive values or posterior probabilities.

How to guard against it: Translate sensitivity, specificity, and likelihood ratios through a relevant pre-evidence probability, as in Lesson 4.

Study Biases

Selection and Attrition Bias

Selection bias occurs when inclusion or analysis depends on variables that distort the target comparison. Attrition can create selection bias when loss to follow-up differs in ways related to treatment and outcome. Compare who enters, remains, and is analyzed across groups.

Measurement and Detection Bias

Measurement error becomes bias when variables are measured systematically differently across groups or relevant states. Awareness of treatment, unequal assessment, changing instruments, and differential outcome surveillance can all distort estimates.

Performance and Co-intervention Bias

Groups may receive different attention, concurrent treatment, or implementation beyond the intervention being compared. Blinding, protocol monitoring, and transparent reporting help evaluate these differences but do not guarantee their absence.

Causal Fallacies

Post Hoc Ergo Propter Hoc

What it is: Inferring that X caused Y because Y followed X. Natural history, regression to the mean, cointerventions, expectations, and measurement change may remain compatible with the sequence.

Confounding

What it is: An open noncausal path creates an association between exposure and outcome. A DAG can state the assumed structure; design and an appropriate adjustment set are needed to address it.

Reverse Causality

What it is: Attributing X as a cause of Y when Y causes X, or when timing is insufficient to distinguish the direction. Longitudinal measurement helps establish temporal order but does not resolve every causal ambiguity.

Collider and Overadjustment Bias

Conditioning on a collider can open a noncausal path. Conditioning on a mediator can remove part of the total effect. Variable inclusion should follow the causal question and graph rather than a rule to control every available covariate.

Cognitive Biases

Confirmation Bias

What it is: The tendency to favor, search for, or interpret information in ways that confirm one’s existing beliefs or hypotheses.

When to look for it: In both research interpretation (e.g., cherry-picking supportive findings) and clinical reasoning (e.g., sticking with an initial diagnosis despite conflicting signs).

How to guard against it:

• Actively seek disconfirming evidence.

• Use structured diagnostic checklists or Bayesian updating.

• Involve peers for second opinions or external critique.

Outcome Bias

What it is: The tendency to judge a decision by its eventual outcome rather than the quality of the decision-making process at the time.

When to look for it: Retrospective case reviews, audit meetings, or clinical debriefings, especially after bad outcomes.

How to guard against it:

• Focus evaluations on the logic and evidence used at the time, not in hindsight.

• Train teams to distinguish process quality from result.

• Practice simulation-based decision-making with probabilistic outcomes.

Narrative-Based Errors

Causal Coherence Bias

What it is: The cognitive tendency to impose causal connections between events in a story to create a coherent narrative, even when those connections don’t exist.

When to look for it: In clinical case discussions, journalistic summaries of research, or patient storytelling — any setting where sequences are interpreted causally without sufficient evidence.

How to guard against it:

• Use causal diagrams (e.g., DAGs) to differentiate correlation from causation.

• Use causal diagrams and structured causal considerations to expose the assumptions behind the claim.

• Ask: What other explanations are consistent with this sequence?

Conviction Narrative Fallacy

What it is: The belief that a compelling, emotionally resonant story must be true, especially under conditions of uncertainty.

When to look for it: Under diagnostic uncertainty, in treatment decisions when mechanisms are unknown, or when charismatic experts offer anecdotal support.

How to guard against it:

• Separate plausibility from empirical support.

• Cross-check with structured evidence reviews or mechanistic models.

• Ask: What evidence would falsify this story?

Why Do These Illusions Persist?

Psychological research shows that illusions of causality are not random. They are systematic, predictable, and hard to extinguish:

• High Outcome Density (many positive outcomes) leads people to overestimate treatment efficacy.

• Action Frequency can exaggerate perceived control: the more one intervenes, the more one sees improvement — regardless of actual effect.

• Prior Beliefs act as filters, biasing what evidence is noticed and how it’s weighted.

“Illusory beliefs can block learning from evidence. This may explain why ineffective treatments remain popular in practice.” Matute et al., 2015

Using Critical Realist Models to Expose Assumptions

A Critical Realist Graphical Causal Model (CR-GCM) can distinguish what was observed from the events and causal powers proposed to explain it. That distinction makes assumptions available for criticism, but a more detailed graph is not automatically less biased or more accurate:

• Empirical domain: experiences and measurements, such as a recorded pain score after intervention.

• Actual domain: events that occur whether or not they are observed, such as participation in loading or a change in motor behavior.

• Real domain: mechanisms and structures with causal powers, such as adaptive capacity, social constraints, and learning processes.

Explicitly modeling proposed mechanisms can reveal where a causal explanation depends on background knowledge rather than the study’s observations. It can also create a more persuasive but unsupported story. The protection comes from provenance, alternatives, empirical implications, and criticism—not from causal depth alone.

Here is one proposed Critical Realist Graphical Causal Model (CR-GCM) that distinguishes:

• Empirical measurements and experiences,

• Actual events and processes that may occur, and

• Real mechanisms and structures proposed to generate those events.

Study Example:

How Aerobic Exercise and Inspiratory Muscle Training affect the 6-Minute Walk Test, mediated and conditioned by deeper causal layers.

Measurement does not move something from one domain to another. A measurement is an empirical experience of an actual event, while real mechanisms and structures may generate that event. Better instruments can give us new empirical access to events or traces of mechanisms, but what exists is not determined by whether a particular study observes it.

Debiasing Strategies for Clinicians and Researchers

1. Teach Fallacies Explicitly: Integrate fallacy recognition into curricula.

2. Use Narrative Carefully: Teach how clinical stories can mislead as well as illuminate.

3. Use Bradford Hill’s considerations carefully: They are prompts for examining an association, not a checklist that establishes causation. Temporality is necessary for a cause to produce an effect; the other considerations can strengthen or weaken an argument without being individually necessary or sufficient:

   1. Temporality: Did the cause precede the effect?

   2. Strength & Consistency: Is the association robust and reproducible?

   3. Biological Plausibility & Coherence: Does it fit with known mechanisms? If not, are the mechanisms realistic with what you do know?

   4. Dose-Response: Does more exposure lead to greater effect?

4. Model Mechanisms: Ask not just if something works, but how and for whom.

5. Use Critical Realist Review: Distinguish observations, events, and proposed mechanisms while asking what evidence supports each representational claim.

6. Normalize Uncertainty: Cultivate tolerance for ambiguity and promote probabilistic reasoning.

Conclusion

Scientific claims can fail through statistical error, biased design or conduct, unsupported causal inference, cognitive bias, and persuasive narrative. Calculation matters, but so does knowing what a design can identify and what alternatives remain compatible with the evidence.

Recognizing these failure modes is essential to discovery that remains open to correction.

“Statistical thinking will one day be as necessary for efficient citizenship as the ability to read and write.” H.G. Wells

But it must be paired with epistemic humility and mechanistic insight.

Final Thoughts

Beyond Fallacy: When Belief Becomes Identity

“The lie is halfway around the world before the truth has its boots on.” — Attributed to Twain, Churchill, and a dozen others.

The point is clear: falsehood is fast, sticky, and socially contagious.

Statistical fallacies and cognitive biases are not the only threats to causal reasoning. Some errors persist not because people fail to understand the data, but because the data threatens a sense of self.

In The Liar’s Paradox, Jeremy Campbell explores how our age has made room for a new kind of falsehood. One not rooted in ignorance, but in the rejection of shared standards of truth. The line between fact and fiction blurs when truth becomes optional and falsehood becomes identity.

“What I encountered wasn’t disagreement, it was insulation.”

In teaching pseudoscience, I’ve seen beliefs treated not as hypotheses, but as expressions of personal truth. To question them is to violate a code, not to engage in inquiry.

We must therefore ask not only what makes reasoning flawed, but what makes it fragile. Vulnerable not to poor logic alone, but to social performance, cultural identity, and emotional allegiance.

This is where causal reasoning meets courage.

To teach it, model it, and live it, we must recognize that the most persistent fallacies are shields. Sometimes, to see clearly is to feel vulnerable.

Carry the Method Forward

The course began with observations and the claims we build from them. It then separated association from causation, made uncertainty explicit through Bayesian updating, distinguished transportability from patient-specific instantiation, and used DAGs to expose causal assumptions. This final lesson adds the discipline of looking for the ways every one of those steps can fail.

The practical habit is simple but demanding: describe what was observed, state what is being inferred, quantify what remains uncertain, make the causal assumptions visible, seek mechanisms and alternatives, and keep every conclusion open to revision. That is how observations become warranted claims without becoming certainty they cannot support.