AI Safety & Leaderboards

arXiv:2610.04812 [cs.CL, cs.AI]● Camera-ReadyOctober 2026 Plain English + Original arXiv Layout

Plain English Explanation

Why AI Leaderboards Can Be Tricked by Long & Polite Answers

Testing 3.7 million human votes to uncover why chatbots win votes even when their answers are wrong

What We Discovered In Simple Terms

Popular AI ranking leaderboards ask everyday users to compare two anonymous chatbot answers and pick the winner. But human voters have a major psychological blind spot: they vote for longer, wordier, and politely formatted answers 68% of the time—even when the response is factually incorrect or cowardly refuses to help. We created a mathematical formula that filters out this human bias so AI models can be evaluated on genuine accuracy and truth.

Why This Matters To Everyday People

You shouldn't trust a chatbot just because it tops a public leaderboard. Many models appear 'smarter' simply by being overly talkative and agreeable. Our research shows how to evaluate AI for actual problem-solving.

Formal Academic Title (for Researchers & Citations):

Beyond Elo: Deconstructing the Four Hidden Confounders of Crowdsourced LLM Leaderboards

The Alignment Refusal Tax, Structural Markdown Inflation, Evaluator Calibration Drift, and Cross-Lingual Failure

Ajmal Baba•Zannix Labs (Independent Research)

33,000

Blind Pairwise Battles

20

Evaluated LLMs

13,383

Human Evaluators

96

Languages Covered