Twitter/X

On 2026-08-07 @Afinetheorem argues that AI should perform technical review…

2026-08-07 · 15:16 UTC ·@Afinetheorem ·2 min read

Brief

AI-assisted peer review: @Afinetheorem (reacting to Paul Novosad) argues that AI should handle the technical validation of manuscripts—running authors' code, checking proofs, verifying citations and table consistency—before human referees see papers, so humans only assess whether the work is important and accurately presented. Novosad lays out a 5-step workflow: human reads the intro; an AI digests the paper and presents findings; the human asks seminar-style questions while the AI answers strictly from the paper with precise references; a second AI vets the QA transcript for hallucinations; the human then writes the review. They invoke the reality of 100+ page appendices that reviewers do not realistically read and give the example question about occupational ranks in censuses; AI parsing can save reviewers time (the thread cites ~30 minutes of digging) and a journal-run platform could operationalize this while noting it won’t stop cheating but lowers the cost of honest, efficient review.

Why it matters

On 2026-08-07 @Afinetheorem argues that AI should perform technical review tasks—running code, checking proofs, verifying citation accuracy and table consistency—before human reviewers see a paper, leaving humans to judge only whether the work is important and fairly described to speed up refereeing.

Key details

  • Paul Novosad (@paulnovosad) proposes a concrete 5-step AI-assisted review workflow: (1) human reads the intro, (2) AI digests the paper and presents results, (3) human asks seminar-style questions while AI answers strictly from the paper with exact references/appendices, (4) a second AI reviews the transcript for hallucinations/groundedness, (5) human writes the review.
  • Novosad and @Afinetheorem highlight a systemic problem: authors produce 100+ page appendices that humans rarely read; AI can parse those appendices so reviewers can get targeted answers (e.g., “How did they handle occupational ranks in census years where education wasn't reported?”) instead of spending ~30 minutes digging.
  • They propose a journal-hosted platform that limits AI responses to paper content (no unsolicited judgment), lowers the cost of honest reviewing, enables seminar-style interaction, but acknowledges it wouldn't eliminate reviewer cheating.
Source evidence

I like this but I actually think even simpler: AI handles technical reviewing, runs the code, and checks the proofs, citation accuracy, table consistency, etc. before humans see it. Humans then evaluate for "is this important and fairly described" only. Much quicker refereeing!

Paul Novosad (@paulnovosad)

A proposal for AI-assisted article review.

The goal: use AI to make human review more efficient, without sacrificing / biasing expert human judgment.

  1. Human reads the intro to get the big idea in the author's words.
  2. AI reads and digests the paper and presents the results.
  3. Human asks questions, seminar style, AI answers strictly from the content of the paper, with exact references / appendices where possible.
  4. A second AI reviews the transcript for accuracy / groundedness, ensuring that there are no hallucinations or dropped balls.
  5. Human forms judgment and writes the review.

The current system: authors generate 100+ page appendices and we pretend that humans are reading them carefully. This is a fiction. AI in production is overwhelming the capacity to review.

But we do want human expert judgment. You can put a fully AI-generated review in the packet, but you want the human expert's opinion, ideally without getting biased by the AI's ideas.

AI generation in a paper makes it better — if there are incentives for honest work, vs. overwhelming the reader with slop. A 100+ page appendix is good for transparency and detail, it's just not feasible to ask people to read it. But with AI, you don't need to — you can just dig into the part you care about.

Reading a paper as a human is super inefficient. You have some question, "How did they handle occupational ranks in census years where education wasn't reported?" You can spend 30 minutes digging in and getting an answer, but you're only going to do that a limited number of times. With AI-supported review, you can ask and get the authors' answers to many more challenging questions.

This whole process could exist on its own platform, even provided by the journals. It could be appropriately prompted so that it only answers questions about the paper, doesn't provide judgments or try to do the review. (If you just put a paper into ChatGPT, it can't resist giving you all its good and bad review ideas.)

It wouldn't stop reviewers from cheating, but it lowers the cost of being honest. I would much rather read / review a paper this way, the same as I'd rather go to a seminar on a paper than read it.

AI-assisted paper reading is probably the future of paper reading anyway. Does anyone actually read a paper linearly? When reading papers for research, I read the intro, and then I jump around to try to get answers to specific questions. What's the main table? How did they address X? Where did they get these data?

A platform that does this well will be a huge lift for both research and review.

— https://nitter.net/paulnovosad/status/2085739437233750354#m