Sakana AI 发表了 Beyond Imitation，这是一篇 TMLR 研究论文，主题是基于错误检测的 LLM 辅助同行评审。大多数 AI 审稿人的评分标准是它们复制人类评审意见的接近程度。这项工作提出了一个更难的问题：AI 审稿人能否发现被植入的错误？研究团队交付了两部分内容：一个 Contradiction Benchmark 和一个 Multi-Layered Review（MLR）系统。对于构建研究智能体的开发者来说，这个教训很实用。系统设计和模型选择都会影响错误检测。
TL;DR
规模：来自 5 个会议的 257 篇论文中植入了 1,164 处矛盾。MLR 最多读取 10 页正文。
运行于：现成的 API 模型（Claude Sonnet 4、Claude Haiku 3.5）。无需 GPU，无需微调。每篇评审约 $0.47。
性能：在测试的全部 4 个系统中错误检测率最高，且评分与人类一致。
最佳：用 4 次评审捕获了 73.43% 的核心主张错误，而最佳基线为 14.81%。
最差：在真实被撤稿的 arXiv 论文上，仅有 16.11% 的精确匹配。
底线：
最佳：先阅读再评判，能发现远更多严重错误。
最差：仍然会中隐藏的提示注入。
什么是 Multi-Layered Review？
Multi-Layered Review 是 Sakana AI 推出的一个智能体式 AI 评审系统，它在批评一篇研究论文之前会先理解它。它在现成的 Claude 模型上使用 3 个智能体：
Appendix Agent（Claude Haiku 3.5）：从附录中总结实验和实现细节。
Literature Review Agent（Claude Sonnet 4）：使用网络搜索将论文置于先前工作中。它是可选的。
Review Agent（Claude Sonnet 4）：运行一个受 Keshav 的三遍阅读法启发的 3 遍提示链。
第 1 遍写出高层大纲。第 2 遍详细阅读并标记弱点、假设和空白。第 3 遍将所有智能体输出合并为优点、缺点、问题、建议、评分和待办清单。PDF 被直接传入，因此图表和公式得以保留。
Contradiction Benchmark 如何运作？
该基准测试将错误植入真实论文中，并检查审稿人是否能发现它们。研究团队从 ACL、AISTATS、CVPR 和 ICML 2025，以及 NeurIPS 2024 收集了 257 篇 CC 许可的论文。
Gemini 2.5 Pro builds a knowledge graph of each paper’s claims, evidence and methods. Node distance from a “main claim” sets severity. Distance 0 hits a core claim; larger distances hit details. GPT-4.1 then rewrites 1 node per distance into a contradiction, yielding 1,164 data points.
An o3 judge scores each review 10 times. On clean papers it reached 99.9% accuracy. It showed 86.8% sensitivity on manually confirmed catches, so reported scores may be conservative.
How well does MLR detect errors?
MLR led every baseline on the benchmark. With 4 reviews, it caught 73.43% of distance-0 contradictions and 40.95% overall. The best baseline, AgentReview, caught 14.81% at distance 0. A single MLR review still caught 60.79%.
An ablation separates model from design. Swapping GPT-4.1 for Claude Sonnet 4 inside LLM-Review lifted distance-0 detection from 14.56% to 35.40%. MLR’s design added about 25 more points on a single review. Accuracy falls as node distance grows, which supports the severity scoring.
On real retracted papers from WithdrarXiv-Check (211 papers), gains shrink. MLR scored 26.07% on ‘similar’ matches and 16.11% on ‘exact’ matches. The strongest baselines scored 18.48% and 9.00%.
Does MLR agree with human reviewers?
On scores, mostly yes. On ICLR 2025 submissions, MLR’s predicted scores reached a Pearson correlation of 0.586 with human scores. The human-to-human reference was 0.742. On ICML 2025, the AI Reviewer edged it, 0.439 versus 0.429.
On focus, no. MLR stresses validity and experiments, while humans weigh clarity and novelty more. The authors frame this as a complementary perspective, not a replacement.
What does it cost to run?
MLR costs about $0.47 per review, excluding the optional literature agent. It uses 189,062 input tokens, about half of the AI Reviewer’s 403,654. A single-prompt variant cut cost by about two-thirds. Its detection dropped about 3.5 points on a subset of the benchmark.
How does MLR compare with other AI reviewers?
FeatureMLR (Sakana AI)LLM-ReviewAI ReviewerAgentReviewLLM used in this studyClaude Sonnet 4 + Haiku 3.5GPT-4.1o4-miniGPT-4oDesign3 agents, 3-pass chainSingle prompt, text truncated5-review ensemble, meta-review, reflectionReviewer, author, area chair rolesContradiction Benchmark, full40.95% (4 reviews)6.39%6.50%5.95%Core-claim errors (distance 0)73.43%14.56%11.17%14.81%WithdrarXiv-Check, similar / exact26.07% / 16.11%5.21% / 2.37%13.74% / 9.00%18.48% / 5.69%ICLR 2025 Pearson vs human0.586-0.0130.5380.195Input tokens per review189,0626,517403,654310,964Cost per review~$0.47~$0.01~$0.49~$0.81Open codeOn requestYesYesYes
All benchmark, correlation, token and cost figures come from the Beyond Imitation paper.
Key Takeaways
Sakana AI scores AI reviewers on catching errors, not copying humans.
MLR caught 73.43% of core-claim errors, about 5x the best baseline.
Model swap and 3-pass design each add large detection gains.
Real retracted-paper errors remain hard: 16.11% exact matches.
Hidden prompt injection still sways every AI reviewer tested.
Check out the Paper here. All credit goes to the researcher of this project. Also, feel free to follow us on Twitter and don’t forget to join our 150k+ML SubReddit and Subscribe to our Newsletter. Wait! are you on telegram? now you can join us on telegram as well.
The post Sakana AI’s LLM Peer Review System Catches 73% of Core-Claim Errors appeared first on MarkTechPost.