Auditing and Debiasing LLM-as-a-Judge Evaluation with a Bias-Aware Panel
Keywords:
large language models, LLM-as-a-judge, automatic evaluation, evaluation bias, position bias, verbosity bias, self-preference, calibration, benchmarkAbstract
Automatic evaluation by an LLM judge—asking one model to score another’s output—is now the default way to rank generative systems at scale, and it is quietly unreliable. Judges favor the answer shown first (position bias), the longer answer (verbosity bias), and answers from their own model family (self-preference), corrupting the leaderboards and training signals built on top of them. The usual fixes are partial: swapping the two answer orders removes only position bias, and naïvely polling a panel can amplify a bias the judges share. Our contribution is threefold. First, we present Verdict-Bench, a controlled benchmark that scores a judging protocol on five axes—agreement with ground truth, position consistency, verbosity-induced error, self-preference leakage, and the calibration of its verdict confidence—and isolates each bias with a pure-probe suite of equal-quality comparisons. Second, we propose Jury, a bias-aware panel aggregator that combines per-judge order swapping, an observable verbosity correction, bias-weighted logit aggregation over a diverse panel, and a calibrated verdict confidence. Third, in a study of more than 1,000,000 simulated pairwise judgments we show that a single judge agrees with ground truth only 87.6% of the time and exhibits 0.73-rate position, verbosity, and self biases (against an unbiased 0.5); that order-swapping and naïve panels each fix at most one bias and can worsen others; and that Jury reaches 98.1% agreement, drives all three pure-bias rates to 0.5, cuts verbosity-induced error from 28.9% to 0.9%, and cuts calibration error from 0.10 to 0.01. We release Verdict-Bench and Jury as an open baseline for trustworthy LLM-based evaluation
Downloads
References
L. Zheng, W.-L. Chiang, Y. Sheng, S. Zhuang, Z. Wu, Y. Zhuang, Z. Lin, Z. Li, D. Li, E. P. Xing, H. Zhang, J. E. Gonzalez, and I. Stoica, “Judging LLM-as-a-judge with MT-bench and chatbot arena,” in Advances in Neural Information Processing Systems (NeurIPS), 2023.
Y. Liu, D. Iter, Y. Xu, S. Wang, R. Xu, and C. Zhu, “G-Eval: NLG evaluation using GPT-4 with better human alignment,” in Proceedings of EMNLP, 2023.
Y. Dubois, X. Li, R. Taori, T. Zhang, I. Gulrajani, J. Ba, C. Guestrin, P. Liang, and T. B. Hashimoto, “AlpacaFarm: A simulation framework for methods that learn from human feedback,” in Advances in Neural Information Processing Systems (NeurIPS), 2023.
W.-L. Chiang, L. Zheng, Y. Sheng, A. N. Angelopoulos, T. Li, D. Li, H. Zhang, B. Zhu, M. Jordan, J. E. Gonzalez, and I. Stoica, “Chatbot arena: An open platform for evaluating LLMs by human preference,” arXiv preprint arXiv:2403.04132, 2024.
Y. Dubois, B. Galambosi, P. Liang, and T. B. Hashimoto, “Length-controlled AlpacaEval: A simple way to debias automatic evaluators,” arXiv preprint arXiv:2404.04475, 2024.
Y. Wang, Z. Yu, Z. Zeng, L. Yang, C. Wang, H. Chen, C. Jiang, R. Xie, J. Wang, X. Xie, W. Ye, S. Zhang, and Y. Zhang, “PandaLM: An automatic evaluation benchmark for LLM instruction tuning optimization,” in International Conference on Learning Representations (ICLR), 2024.
S. Kim, J. Shin, Y. Cho, J. Jang, S. Longpre, H. Lee, S. Yun, S. Shin, S. Kim, J. Thorne, and M. Seo, “Prometheus: Inducing fine-grained evaluation capability in language models,” in International Conference on Learning Representations (ICLR), 2024.
L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray et al., “Training language models to follow instructions with human feedback,” in Advances in Neural Information Processing Systems (NeurIPS), 2022.
Y. Bai, S. Kadavath, S. Kundu, A. Askell, J. Kernion, A. Jones, A. Chen, A. Goldie, A. Mirhoseini et al., “Constitutional AI: Harmlessness from AI feedback,” arXiv preprint arXiv:2212.08073, 2022.
C.-H. Chiang and H.-y. Lee, “Can large language models be an alternative to human evaluations?” in Proceedings of the Association for Computational Linguistics (ACL), 2023.
P. Wang, L. Li, L. Chen, Z. Cai, D. Zhu, B. Lin, Y. Cao, Q. Liu, T. Liu, and Z. Sui, “Large language models are not fair evaluators,” arXiv preprint arXiv:2305.17926, 2023.
K. Saito, A. Wachi, K. Wataoka, and Y. Akimoto, “Verbosity bias in preference labeling by large language models,” arXiv preprint arXiv:2310.10076, 2023.
M. Wu and A. F. Aji, “Style over substance: Evaluation biases for large language models,” arXiv preprint arXiv:2307.03025, 2023.
Panickssery, S. R. Bowman, and S. Feng, “LLM evaluators recognize and favor their own generations,” arXiv preprint arXiv:2404.13076, 2024.
R. Stureborg, D. Alikaniotis, and Y. Suhara, “Large language models are inconsistent and biased evaluators,” arXiv preprint arXiv:2405.01724, 2024.
M. Sharma, M. Tong, T. Korbak, D. Duvenaud, A. Askell, S. R. Bowman, N. Cheng, E. Durmus, Z. Hatfield-Dodds, S. R. Johnston, S. Kravec, T. Maxwell, S. McCandlish, K. Ndousse, O. Rausch, N. Schiefer, D. Yan, M. Zhang, and E. Perez, “Towards understanding sycophancy in language models,” arXiv preprint arXiv:2310.13548, 2023.
E. Perez, S. Ringer, K. Lukošiūtė, K. Nguyen, E. Chen, S. Heiner, C. Pettit, C. Olsson, S. Kundu et al., “Discovering language model behaviors with model-written evaluations,” arXiv preprint arXiv:2212.09251, 2022.
P. Verga, S. Hofstatter, S. Althammer, Y. Su, A. Piktus, A. Arkhangorodsky, M. Xu, N. White, and P. Lewis, “Replacing judges with juries: Evaluating LLM generations with a panel of diverse models,” arXiv preprint arXiv:2404.18796, 2024.
L. Zhu, X. Wang, and X. Wang, “JudgeLM: Fine-tuned large language models are scalable judges,” arXiv preprint arXiv:2310.17631, 2023.
T. Wang, P. Yu, X. E. Tan, S. O’Brien, R. Pasunuru, J. Dwivedi-Yu, O. Golovneva, L. Zettlemoyer, M. Fazel-Zarandi, and A. Celikyilmaz, “Shepherd: A critic for language model generation,” arXiv preprint arXiv:2308.04592, 2023.
Z. Li, X. Xu, T. Shen, C. Xu, J.-C. Gu, Y. Lai, C. Tao, and S. Ma, “Leveraging large language models for NLG evaluation: A survey,” arXiv preprint arXiv:2401.07103, 2024.
Z. Zeng, J. Yu, T. Gao, Y. Meng, T. Goyal, and D. Chen, “Evaluating large language models at evaluating instruction following,” in International Conference on Learning Representations (ICLR), 2024.
Guo, G. Pleiss, Y. Sun, and K. Q. Weinberger, “On calibration of modern neural networks,” in International Conference on Machine Learning (ICML), 2017.
M. P. Naeini, G. F. Cooper, and M. Hauskrecht, “Obtaining well calibrated probabilities using bayesian binning,” in Proceedings of the AAAI Conference on Artificial Intelligence, 2015.
J. Platt, “Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods,” Advances in Large Margin Classifiers, 1999.
J. Cohen, “A coefficient of agreement for nominal scales,” Educational and Psychological Measurement, vol. 20, no. 1, 1960.
Downloads
Published
How to Cite
Issue
Section
License
This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.
All papers should be submitted electronically. All submitted manuscripts must be original work that is not under submission at another journal or under consideration for publication in another form, such as a monograph or chapter of a book. Authors of submitted papers are obligated not to submit their paper for publication elsewhere until an editorial decision is rendered on their submission. Further, authors of accepted papers are prohibited from publishing the results in other publications that appear before the paper is published in the Journal unless they receive approval for doing so from the Editor-In-Chief.
IJISAE open access articles are licensed under a Creative Commons Attribution-ShareAlike 4.0 International License. This license lets the audience to give appropriate credit, provide a link to the license, and indicate if changes were made and if they remix, transform, or build upon the material, they must distribute contributions under the same license as the original.
