Where Do LLMs Go Wrong? Diagnosing Automated Peer Review via Aspect-Guided Multi-Level Perturbation
Image created by chatGPT-5Abstract
We introduce an aspect-guided perturbation framework to diagnose vulnerabilities of Large Language Models (LLMs) in peer review. By perturbing papers, reviews, and rebuttals along dimensions such as contribution, soundness, presentation, tone, and completeness, we reveal where LLM reviewers are most error-prone. Our analysis across major LLMs (GPT-4o, Gemini 2.0, LLaMA 3, etc) highlights recurring weaknesses, including misjudging methodological flaws, over-weighting strong rejections, mishandling incomplete rebuttals, and misinterpreting poor critiques as rigorous. These findings provide actionable insights for building balanced human–AI peer review partnerships.
Type
Publication
In Proceedings of the 34th ACM International Conference on Information and Knowledge Management
I co-examine the robustness of LLMs when used as automated peer reviewers.
- Co-Designed an aspect-guided multi-level perturbation framework.
- Co-Evaluated multiple LLMs on review tasks.
- Found systematic cognitive biases and stability gaps.
