3,844 comparisons among 13 agents.
| Description | RepoChat Arena preference votes, 3,844 battles between 13 named language models, 2024-11-30 to 2025-02-11. A person pastes a GitHub repository link, asks a question about it, and judges two anonymous models' answers. The contestant is really a pair: each side runs a generating model AND a retriever that chose which repository files to put in context, drawn independently (1,932 battles used gemini-1.5-flash-002 as retriever_a, 1,912 used gpt-4o-mini). agent_a and agent_b name the generating model only, because that is what the vote is nominally about and what the leaderboard ranks, while retriever_a and retriever_b are kept as columns so the confound stays visible -- a Bradley-Terry fit that ignores them is fitting a mixture. `repo` is the GitHub URL asked about, 2,377 distinct projects. agent_a wins 1,264, agent_b 1,343, 1,237 draws; the source's own winner field pools tie with both_bad, and `outcome` recovers the split (717 tie, 520 both_bad) from the separate booleans. `date` is a day, not a time: every source value is midnight UTC. No homefield column -- slot assignment is balanced and agent_a wins 48.48% of the 2,607 decided battles, z = -1.55. |
|---|---|
| Reference | LMArena. (2025). repochat-arena-preference-4k [Data set]. Hugging Face. https://huggingface.co/datasets/lmarena-ai/repochat-arena-preference-4k |
| Licence | CC BY 4.0 |
| Source data | https://huggingface.co/datasets/lmarena-ai/repochat-arena-preference-4k |
| Comparisons | 3,844 |
|---|---|
| Agents | 13 |
agent_aagent_bdateoutcomereporetriever_aretriever_bscore_ascore_bwinner
Or load it directly in R (no account needed with irw 1.4.0 or later) or Python (needs a free Redivis account; the first call opens a browser to sign in):
# R
install.packages("irw")
library(irw)
df <- irw_fetch("repochat_arena_2024_2025", source = "comp")
# Python
pip install irw
import irw
df = irw.fetch("repochat_arena_2024_2025", source="comp")
Or in Stata, which reads the same no-account CSV as the button above:
* Stata
net install irw, from("https://raw.githubusercontent.com/itemresponsewarehouse/Stata-pkg/main/")
irw_fetch repochat_arena_2024_2025, clear source(comp)
@dataset{lmarena_2025,
title={repochat-arena-preference-4k},
author={LMArena},
year={2025},
publisher={Hugging Face},
url={https://huggingface.co/datasets/lmarena-ai/repochat-arena-preference-4k}
}
| IRW version | v712 |
|---|---|
| Redivis dataset | irw_competitions v7.1 |
| Redivis dataset DOI | 10.57761/3r1w-t660 |
| Manifest pin for this IRW version | v7.1 |
| Metadata source | irw_meta v68.0 |