3,355 comparisons among 6 agents.
| Description | MT-Bench expert human pairwise judgments, 3,355 judgments by 65 named judges over 80 questions and 6 chat models (alpaca-13b, claude-v1, gpt-3.5-turbo, gpt-4, llama-13b, vicuna-13b). Each conversation is judged twice, once after turn 1 and once after turn 2, so `turn` is 1 or 2 and a row is a judgment rather than a battle. This is the small, careful counterpart to the crowd arenas: `rater` is the judge label as published ("author_N" for a paper author, "expert_N" for a recruited graduate-student expert), already a pseudonym in the source and kept verbatim because that distinction is informative. agent_a wins 1,293, agent_b 1,282, 780 draws; `outcome` keeps the source wording. `question` is the MT-Bench question id, 81 to 160. No date and no homefield column: the deposit has no timestamp, and on sides, slot assignment is balanced while agent_a wins 50.21% of the 2,575 decided judgments (z = +0.22), so no advantage is recorded. The companion gpt4_pair split of the same deposit, 2,400 rows of GPT-4 judging the same pairs, is deliberately NOT built -- those are machine responses, not human ones. |
|---|---|
| Reference | Zheng, L., Chiang, W.-L., Sheng, Y., Zhuang, S., Wu, Z., Zhuang, Y., Lin, Z., Li, Z., Li, D., Xing, E. P., Zhang, H., Gonzalez, J. E., & Stoica, I. (2023). Judging LLM-as-a-judge with MT-Bench and Chatbot Arena. Advances in Neural Information Processing Systems 36, Datasets and Benchmarks Track. Data set: lmsys/mt_bench_human_judgments, Hugging Face. |
| Licence | CC BY 4.0 |
| Source data | https://huggingface.co/datasets/lmsys/mt_bench_human_judgments |
| Comparisons | 3,355 |
|---|---|
| Agents | 6 |
agent_aagent_boutcomequestionraterscore_ascore_bturnwinner
Or load it directly in R (no account needed with irw 1.4.0 or later) or Python (needs a free Redivis account; the first call opens a browser to sign in):
# R
install.packages("irw")
library(irw)
df <- irw_fetch("mtbench_human_2023", source = "comp")
# Python
pip install irw
import irw
df = irw.fetch("mtbench_human_2023", source="comp")
Or in Stata, which reads the same no-account CSV as the button above:
* Stata
net install irw, from("https://raw.githubusercontent.com/itemresponsewarehouse/Stata-pkg/main/")
irw_fetch mtbench_human_2023, clear source(comp)
@dataset{zheng2023mt_bench,
author = {Zheng, Lianmin and Chiang, Wei-Lin and Sheng, Yujia and Zhuang, Shuhuai and Wu, Zhuohan and Zhuang, Yonghao and Lin, Zi and Li, Zhuohan and Li, Dacheng and Xing, Eric P. and Zhang, Hao and Gonzalez, Joseph E. and Stoica, Ion},
title = {Judging {LLM}-as-a-judge with {MT}-Bench and Chatbot Arena},
year = {2023},
publisher = {Hugging Face},
howpublished = {\url{https://huggingface.co/datasets/lmsys/mt_bench_human_judgments}},
note = {Advances in Neural Information Processing Systems 36, Datasets and Benchmarks Track}
}
| IRW version | v712 |
|---|---|
| Redivis dataset | irw_competitions v7.1 |
| Redivis dataset DOI | 10.57761/3r1w-t660 |
| Manifest pin for this IRW version | v7.1 |
| Metadata source | irw_meta v68.0 |