AI Needs Ranked Mode

Every chess player has a rating. Every Counter-Strike account has a rank. Every Dota player knows their MMR within ten points. We have spent twenty years building rigorous systems for measuring messy human skill — at games where execution dominates, at games where strategy dominates, at games where the meta shifts patch by patch — and we did it because the alternative is a year of forum arguments about who's actually good.

Now we have a new skill, and we are doing it badly.

AI use — actually using a model under time pressure to produce a verifiable artifact — is a skill. Not in the soft "everything is a skill" sense, but the way StarCraft is a skill. Two people sit down with the same model, the same tools, the same task, and the same clock, and one of them finishes and the other doesn't. The skill is real. We just haven't built the ladder to measure it.

What we have built instead is three years of vibes-based ranking. Twitter threads about whose prompt is cleverest. Self-rated quizzes — AIQ Rank scores you on tool diversity and "context leverage," whatever that means. Token-consumption leaderboards — CostHawk literally ranks people by how much money they spent on Claude this month, which is the AI-skill equivalent of ranking chess players by how many ELO points they bought from boosting services. The benchmark world ranks the engines: Chatbot Arena has six million votes for which model people prefer. The operators are unranked.

Call it ranked AI. The thesis: import twenty years of competitive-gaming rigor into a discourse that's been flailing at messy-skill measurement for three.

What ladders actually do

Watch a Counter-Strike pro on stream and the rank is the spine of the narrative. Without rank, every clip is just a clip. With rank — "this is a Global Elite playing a Silver smurf, watch how he moves" — the same clip is a lesson, a story, a tier list, a meta change. The ladder is what converts isolated performances into a culture.

Three things every ranked system does. It surfaces skill — you can't be quietly great anymore, the number is there. It creates stakes — each match changes the number, which makes each match matter. And it generates narrative — the climb, the plateau, the patch, the upset, the season. Pros study replays not because the replays are interesting but because the replays produce the meta.

We have none of this for AI. We have screenshots.

The benchmark trap

The first wave of AI ranking did the obvious thing. Rank the models. SWE-bench tracks which model solves the most real GitHub issues. MMLU tracks which one knows the most trivia. LMArena pits models head-to-head and computes a Bradley-Terry score over millions of human votes. This work matters — frame rate matters in esports too — but it is not the same as ranking the player.

Two people using GPT-5 to fix the same bug produce different outcomes. Two people using Claude Haiku 4.5 to ship the same feature produce different outcomes. The model is the engine. The driver is what we don't measure. And when nobody measures the driver, "I'm good with AI" floats in the same epistemological soup as "I'm good at music" — defensible, unfalsifiable, and useless for picking who to hire.

Worse: when we DO try to measure the driver, we measure the wrong thing. The two largest existing "rank AI users" products both rank by raw token consumption. The harder you make your AI work, the higher you rank — which is the inverse of skill. It is as if Chess.com ranked you by how many pieces you moved per minute. Tokens are like APM in StarCraft: low APM doesn't mean you're bad, high APM doesn't mean you're good. What matters is whether the motion was wasted. Ranking by APM alone is silly. Ranking by tokens alone is the same shape of mistake.

Where LMArena stopped

Chatbot Arena is the closest thing to ranked AI that has ever existed, and it is incredibly good at what it does. Six million human votes have produced Bradley-Terry coefficients with bootstrap confidence intervals on every major model since 2023. It's the gold standard.

It is also the wrong layer.

LMArena answers the question "which model is better." Ranked AI needs to answer the question "which operator is better at directing this model." These are different problems with different mathematics and different audiences. The first is procurement. The second is sport.

The trick LMArena pulls — pairwise preference over crowd votes — does not translate directly. A 1v1 ranked AI match cannot be settled by "vote which output you preferred"; that's slow, noisy, and gameable. It has to be settled by hard graders. Did the test pass. Did the API contract validate. Did the patch close the security bug. The judge has to be objective, and the task has to be designed so that objectivity is possible. Code, data transforms, scrapes, regexes, schema fixes — these can be judged in milliseconds by pytest. Open-ended creative tasks cannot. That's fine. Chess doesn't launch with three-dimensional variants. Ranked AI starts with the tasks where the win condition is unambiguous.

What ranked AI actually looks like

The competition analog is not Formula 1. F1 is the constructor championship — the car matters more than the driver, and a ranked AI ladder that lets you bring your own model is structurally the same: it ranks the team that bought the most compute, not the operator. F1 is fun. It is not what we are building.

The right analog is Battlecode, the MIT contest that has run every January since the early 2000s. Every team gets identical compute, identical APIs, identical robot capabilities. The only thing that varies is the code you write. The skill being measured is how well you direct a constrained agent toward a goal. Ranked AI shifts this one level up the stack: instead of writing the controller, you're writing the prompts.

So: every match runs against the same model — Claude Haiku 4.5, fixed — with the same tool surface, the same context window, the same temperature. Each player gets a 40,000-token hard cap and a ten-minute clock. Tasks come from a hidden pool of two hundred small bug-fix scenarios with light per-match randomization, so memorized prompts don't transfer cleanly between matches. A hidden pytest suite runs at the buzzer. Tests passed determines the winner. Tokens used breaks the tie. Glicko-2 updates the rating.

The token cap is the most important constraint on the page. It is the chess clock of ranked AI. Without it, every other guardrail leaks — players just throw more compute at the problem and the ladder collapses into a wallet rank. With it, "use Opus ten times until something works" stops being a strategy. The cap forces decomposition, prompt precision, and the kind of restraint that separates someone who actually knows what they're doing from someone who is brute-forcing the model. The token ticker, displayed live next to the test board, is also the most watchable element of the entire product. It is what a casual viewer can feel without understanding the game.

The shape of what comes next

The ladder is the start, not the end. Every ranked community I admire — chess locals, Geoguessr Battle Royales, Minecraft Speedrun seasons, fighting-game brackets — has the same arc. The online ladder is the funnel. The in-person tournament is the brand-defining moment. The first six months of ranked AI is online matches: closed beta, then public. The next six months are seasonal physical tournaments in cities where the AI energy already is — New York Tech Week, SF, then locals everywhere else. Top sixteen from the online ladder qualify; the rest watch. After that: nationals, worlds, the whole arc.

The ranking is the skeleton. The community is the body.

For three years AI has been the most competitive software ever shipped without a single competitive ladder for the humans using it. Imagine the same statement about chess engines and chess players. We would not have tolerated it for a month.

RANKEDLLM.COM. The first ladder for AI operators.

Queue up.