Twitter/X

Yishan (@yishan) re-ran the paper authors' open-sourced evaluation framework…

2026-06-28 · 04:54 UTC ·@emollick ·2 min read

Brief

Yishan (@yishan) used the authors' publicly released evaluation framework (github.com/aiden-ygu/health-...) and posted his reproduction at github.com/ywong137/health-a..., re-running the paper's radiology-interpretation tests on both the original models and current frontier models. He found little baseline drift and measured an improvement from 69/100 (prior best) to 79/100 for GPT-5.5 Pro, but stresses this numeric gain still falls short of the paper's stricter clinical-readiness requirements—robustness under perturbation, proper abstention when data are insufficient, and clinically valid reasoning without hallucinations—tests he could not replicate qualitatively. He notes some testing data remained closed due to JAMA/NEJM copyrights, praises the authors for releasing what they could, and argues that open-sourcing evaluation code is essential so others can reproducibly run benchmarks on newer models over time.

Why it matters

Yishan (@yishan) re-ran the paper authors' open-sourced evaluation framework (github.com/aiden-ygu/health-...) and published his reproduction and results at github.com/ywong137/health-a..., successfully reproducing tests on the public datasets the paper included.

Key details

  • Baseline re-runs on the paper's older models showed little performance drift; the most capable current model tested (GPT-5.5 Pro) scored 79/100 versus the prior best 69/100 — a clear improvement that nonetheless remains below the paper's standard for clinical readiness.
  • The paper's 'fit for reliable medical use' criterion demands robustness to perturbation/bad data, correct refusal/abstention when information is insufficient, and clinically valid reasoning without hallucinations; Yishan was unable to reproduce those qualitative robustness evaluations in his automated reruns.
  • Some test data could not be open-sourced because of copyrights held by JAMA/NEJM, but partial open-sourcing of the evaluation code enabled rapid external reproduction; Yishan urges all AI-paper authors to release evaluation frameworks so others can re-run tests on newer models.
Source evidence

Nice example of the increasing benefits of open science and transparent methodologies when writing papers about AI.

Yishan (@yishan)

A big problem with research studies on AI models is that given how long the peer review process is, the results are always out-of-date by the time the paper is published.

This time, we have something better!

The typical reaction to research results like this roughly goes "You're just testing on old models. Today's models are way better and surely can do it now!"

But the best solution is for these papers to also open-source all of their testing framework so that upon publication, others can reproduce their results, as well as run it on the newest models of the day - and into the future. After all, "this is the worst they'll ever be" so what really matters is determining when they DO pass the threshold.

As it turns out, the authors of this paper DID open-source their evaluation framework!

Here:
github.com/aiden-ygu/health-…

So I figured... let's re-run the tests on the latest models!

Summary of our results are here:
github.com/ywong137/health-a…

One drawback is that, unfortunately, the authors didn't (or weren't legally able to) open-source ALL the testing data, since apparently some of it is copyrighted by JAMA/NEJM etc. That's a separate problem with the medical research publishing industry for another time.

However, we were able to reproduce the test on the public datasets they did include!

First, we re-ran the same tests (as closely as we could) on the old models the paper claimed to use, in order to establish a baseline and determine how much "drift" there would be. (Answer: not too much)

Then we ran those tests on the newest frontier models we could find.

The results are: the most capable models today (GPT-5.5 Pro) did outperform the best models from before (79/100 vs 69/100), but did not improve enough to be considered sufficient for reliable medical use.

In fact, the paper's criterion for "fit for reliable medical use" is more stringent, requiring the models to be robust under perturbation and bad data, knowing when to say there's not enough information, give clinically valid reasoning rather than hallucinations, etc. Those sound pretty reasonable to me.

I wasn't able to reproduce that kind of qualitative evaluation, but even on the basic pass/fail test using public datasets of interpreting radiology images, the newest models are better, but not yet quite good enough.

Nevertheless, I would like to praise the paper's authors for at least open-sourcing what they could, enabling me to (fairly quickly) attempt to reproduce their results. This is definitely a step in the right direction!

While my reproduction wasn't able to be comprehensive, it certainly gave me useful directional info and - perhaps more importantly - allowed me (a random dude on the internet) to directly reproduce the results in their paper and validate them.

I would like to encourage ALL authors of research papers on AI models to do similar open-sourcing of their experimental frameworks!

— https://nitter.net/yishan/status/2070742742133780960#m