Data · reproducibility
A whole subfield’s accuracy numbers are too good to be true
Many published clinical machine-learning models report near-perfect accuracy that comes from a data mistake, not real skill. One correct change can drop a model from 0.98 to 0.79.
· Steps Ventures
If you read machine-learning papers in medicine, you notice something odd. The accuracy numbers are amazing. Models that predict who will get sepsis or die in the ICU, scoring 0.95, 0.98, sometimes a flat 1.00. Real medicine is not that predictable. When a whole subfield reports numbers that good, the most likely explanation is not that everyone is a genius. It is that a common mistake is quietly inflating the scores.
The mistake has a boring name, data leakage, and it means the model got to peek at the answer during training in a way it never could in the real world. It is easy to do by accident and it is everywhere. We decided to stop talking about it and demonstrate it on data anyone can download.
How the peeking happens
Two flavors show up constantly. The first is splitting the data by row instead of by patient. One patient has many hours of records. If some of that patient’s hours land in training and some in testing, the model can half-memorize the person instead of learning the disease. On public ICU data, fixing just that dropped a model from about 0.93 to 0.80. The second flavor is balancing the classes before splitting. Medical outcomes are lopsided, so people manufacture synthetic patients to even it out. Do that before you split, and copies of the same synthetic patient end up on both sides of the wall. On the same public data, that one error pushed a model from about 0.79 all the way to 0.98. The fix is a single line, moving the balancing step to after the split. The inflation was the whole gap.
Why this matters beyond the leaderboard
These are not just numbers in a table. A leaky model looks like it works, gets published, gets cited, and sometimes gets built into a product that touches real patients. Then it fails quietly in the hospital, because the skill was never there. And the published record looks far more confident than reality, which sends the next lab chasing a signal that was always an artifact.
What we built
We are building an open screener. You give it the identifier for a paper, it pulls the public summary, and it scores the described setup against a fixed checklist of leakage patterns. It does not accuse anyone. It flags risk. On a first batch of clinical-ML papers it flagged a chunk as high-risk and correctly left the careful ones alone, including papers that simply do not describe their split, which is silence, not a crime. Where a paper also has public data, we go further and reproduce the authors’ own number first, then show what one correct change does to it.
The rules we hold ourselves to
This work only earns trust if it is fair. So the guardrails are strict. Methods, never motives. Reproduce the original number before we critique it. One mechanically demonstrable flaw per paper, not a pile of vague doubts. Right of reply before anything goes public. And we report risk, not verdicts. This is a public good, not an accusation. The point is a subfield with numbers you can believe, and an open tool that lets anyone check a paper before they trust it.
The larger idea is the same one that runs through the rest of our research. Re-test published claims on their own data. If the signal survives the honest test, wonderful. If it evaporates, that is worth knowing too, and knowing it early is cheaper for everyone.