Auditing and Debiasing LLM-as-a-Judge Evaluation with a Bias-Aware Panel - Veera Ravindra Divi (International Journal of Intelligent Systems and Applications in Engineering, 2025)
Automatic evaluation by an LLM judge—asking one model to score another’s output—is now the default way to rank generative systems at scale, and it is quietly unreliable. Judges favor the answer shown first (position bias), the longer answer (verbosity bias), and answers from their own model family (self-preference), corrupting the leaderboards and training signals built on top of them. The usual fixes are partial: swapping the two answer orders removes only position bias, and naïvely polling a panel can amplify a bias the judges share. Our contribution is threefold. First, we present Verdict-Bench, a controlled benchmark that scores a judging protocol on five axes—agreement with ground truth, position consistency, verbosity-induced error, self-preference leakage, and the calibration of its verdict confidence—and isolates each bias with a pure-probe suite of equal-quality comparisons. Second, we propose Jury, a bias-aware panel aggregator that combines per-judge order swapping, an observable verbosity correction, bias-weighted logit aggregation over a diverse panel, and a calibrated verdict confidence. Third, in a study of more than 1,000,000 simulated pairwise judgments we show that a single judge agrees with ground truth only 87.6% of the time and exhibits 0.73-rate position, verbosity, and self biases (against an unbiased 0.5); that order-swapping and naïve panels each fix at most one bias and can worsen others; and that Jury reaches 98.1% agreement, drives all three pure-bias rates to 0.5, cuts verbosity-induced error from 28.9% to 0.9%, and cuts calibration error from 0.10 to 0.01. We release Verdict-Bench and Jury as an open baseline for trustworthy LLM-based evaluation