sequenced.ai
Articles/Models & infrastructure/Blueprint//8 min read

How Arena turns head-to-head model comparisons into public leaderboards

Arena collects blind preferences from people comparing models. Here is how to interpret a leaderboard, commission feedback and distinguish both from API inference.

By Sequenced deskAI-assisted, source-led · how we work
Visit Arena website ↗
Blind pairsBattle formatPeople compare answers before seeing model names.
Paired votesRating inputA Bradley–Terry model combines preferences.
Public ranksLeaderboardScores follow the model listing policy.
Direct or routedAPI choiceThe preview coding router conceals the serving model.
Arena mark
Arenaarena.ai · independent research

Represent this company? Verify your work email to access its workspace, or send the desk a factual correction.

Two models rewrite the same unhelpful error message. One sounds patient and reassuring. The other tells the reader exactly where to find the missing setting. Someone has to decide which answer helped more. Arena turns choices like that into public evidence about what people prefer. A product team can learn a great deal from those comparisons if it also checks whether the setting exists.

In brief
  1. 01The comparison People submit a prompt, inspect competing answers and vote. Arena combines paired preferences across many encounters.
  2. 02The data boundary Prompts submitted to the public service can reach providers and public research datasets. Use material cleared for that audience.
  3. 03The buying decision Reading a leaderboard, commissioning feedback on a model and paying for API inference purchase three different things.

01 / ProductFrom a research experiment to a model comparison service

The project began with researchers from UC Berkeley. It developed from Chatbot Arena into LMArena and then Arena. The company behind it is Arena Intelligence Inc. Its subject has expanded beyond written conversations to work such as generating images and building websites. That history matters when following older links or reading an evaluation announcement under the LMArena name. There is one continuing platform here. Arena company history From LMArena to Arena

In a public battle, a visitor supplies the task and sees two responses before learning which systems produced them. Hiding the names helps someone concentrate on what is in front of them. A brand reputation need not decide which paragraph reads better. Aggregating the votes through a Bradley–Terry statistical model gives Arena a way to estimate relative preferences across a field where every possible pair need not face the same visitor. How Arena battles work Arena voting and data FAQ

The result answers a particular question. It says something about the tendency of participants to choose one response over another under the conditions of that arena. Imagine two answers that both give the wrong return deadline. Voters may consistently prefer the clearer one. That preference is real evidence about presentation. A retailer still has to check the deadline before either version goes into a customer email.

02 / AudienceWho can use a preference signal

A designer choosing a writing assistant can start here because the experience of reading the output is part of the product. A fluent answer that makes the reader hunt for the first step can be a poor fit. Looking through example comparisons can help the designer name qualities to evaluate at home, such as whether a response leads with the next action or gives an explanation before the reader has enough context.

A model developer has a different question. A change to a post-training recipe might improve instruction following on a fixed test while making replies unnecessarily formal. Feedback from people bringing their own prompts can expose cases the developer never wrote down. A lab then needs examples and enough detail about the comparison to investigate changes. A public ranking by itself cannot identify which training change caused a preference to move.

A regulated service or a team handling confidential material should plan the exercise around what it can disclose. Arena explains that providers can receive conversations and that some conversations may be released as research data after de-identification. An unpublished customer complaint may identify someone through its details even after removing a name. A fictional, publishable scenario can preserve the writing problem while keeping private records out of a public battle. Arena voting and data FAQ

03 / WorkflowA proposed comparison for a help centre rewrite

Start with a section of an already public help centre that explains an easily misunderstood policy. Write a prompt asking for a plain-language rewrite for a first-time customer. Include a real constraint from the published material, such as an exception that applies only after the first renewal. Save the original source and the exact input so colleagues can inspect what the models were given.

Before viewing an answer, ask the policy owner what a reader must be able to do afterward. Perhaps they must distinguish exchanging an item from cancelling a recurring order. Sketch a separate preference question for the editor, such as which version a new reader could navigate more easily on a phone. A model can win the editorial comparison and still fail the policy check. Recording those judgments separately makes a disagreement interpretable.

Use the public comparison only for the cleared example. Examine both responses for missing exceptions, invented deadlines and a clear first action. After choosing a preference, record what influenced it before looking at the model labels. To choose two named models, use Arena’s Side by Side mode. The matchup is visible, and these votes are excluded from the public leaderboard. Use direct endpoints when the trial also needs controlled settings or repeated samples. Arena voting and data FAQ How Arena battles work

Bring the responses to a colleague who has never seen the policy. Ask them to resolve a concrete situation using each version. A preference accompanied by the wrong application of a rule should send the team back to its editorial brief. Conversely, two factually correct answers that lead to different levels of reader effort reveal something a fact-check alone would miss. Treat this as a proposed product exercise, with no claim that one model won it.

04 / PricingWhat the public site, a paid evaluation and the API provide

Arena describes the public leaderboard as freely shared. Its commercial AI Evaluations service provides analysis based on community feedback, representative samples for inspection and delivery timelines under a service agreement. The announcement mentions flexible pricing for open-weight models and nonprofit institutions. It gives no universal evaluation price. Ask which tasks, modalities, sample details and delivery obligations a quote covers. Commercial AI Evaluations

RouteCommercial basisWhat it delivers
Public leaderboardFreely accessible rankingsA view of observed community preferences
Commissioned evaluationsScope and pricing by arrangementCommunity feedback, representative samples and a delivery agreement
Arena API direct modelsEligible paid or all-model promotional balanceInference requests using an explicitly selected model ID

Commercial routes, consulted 21 November 2026. Commercial AI Evaluations Arena API model selection

The Arena API is another product. Its documentation distinguishes direct model calls from a coding router in preview. Direct model calls require paid funds or promotional funds eligible for all models. With the preview router, the response identifies the router and does not disclose the underlying model that served a particular request. A team reproducing a comparison between named models should choose direct model IDs and retain the settings. A router trial answers a different deployment question. Arena API model selection

05 / DistinctionsWhere a public leaderboard fits in an evaluation stack

The Braintrust Blueprint addresses teams assembling their own examples and evaluating application changes. A private regression set can repeatedly ask the exact renewal question after an edit to a system prompt. Arena brings a broader population of tasks and voters. A useful division of work is to use public results for a first shortlist, then ask whether each shortlisted model survives the same product-specific gates.

The OpenRouter Blueprint is relevant when the next decision is how an application reaches different models. A model identity on a chart does not establish that an integration supports the required tools or behaves the same way under load. Keep a record of the endpoint actually tested. Otherwise a change to the delivery route can look like a change in what users prefer about the model.

Arena also separates evaluating a model from listing a public score. Its policy specifies routes to eligibility, including public weights, an accessible public API or a generally available public service. Anonymous pre-release tests and preliminary scores are subject to separate rules. A research team tracking releases should preserve the date and status of the score it cites, especially while a new version is collecting votes. Arena leaderboard policy

06 / QuestionsWhich aspects of a result travel to another audience

A popular answer format can attract votes on its own. Arena's FAQ describes style controls in the rating system. An editor should still look at what happened in the application: did a bulleted response make the next action clearer, or simply look more organised in a side-by-side comparison? Preserve a few examples of both outcomes so a future prompt change can be discussed in terms of readers' experience. Arena voting and data FAQ

The same caution applies to the mix of tasks. A broad text leaderboard can include activities far removed from reading a contract exception or following a troubleshooting procedure. A narrow application may depend on situations that are rare among public prompts. Ask the person proposing a switch to show one real requirement for which the public signal is relevant and one that needs separate evidence. Both questions belong in the decision record.

07 / DecisionThree reasons to open Arena

01

Shortlisting a writing model

Choose the leaderboard closest to the intended task, keep the observation date, then run the candidates on publishable examples with an editor and a policy owner.

Advance candidates that produce an answer readers can use correctly.
02

Investigating a model release

Define the behavior that changed, ask what representative feedback a commissioned evaluation can return and read the public listing policy before announcing a score.

Agree on the evidence and the delivery date with the evaluation provider.
03

Selecting an API route

Decide whether each response must be attributable to a named model. Check credit eligibility and compare the direct and preview-router documentation for the tool that will send requests.

Use a named endpoint when attribution determines whether the trial is valid.
What should we explore next?

A business worth understanding.

Suggest your business or one you find interesting. Tell us what you want to understand about its product, positioning, design or workflows.

Suggestions are free. Selection and publication stay with the desk.

Sources

Continue reading

All in this category