Weekly
A focused newsletter on model evaluations and benchmarks, for people who want to know what a new model can actually do
What an issue holds
Every week: a new model's evaluation results with the benchmark used and its known limits, new or revised benchmarks and what they leave out, and results later disputed or independently checked. For someone who wants to know what a model can actually do, not a marketing chart.
What it leaves out
Not a benchmark score reported with no context on what it measures, not marketing charts with no methodology attached, not model release announcements with no evaluation data, not general AI industry news better followed as its own subject.
Where the news comes from
LMArena blog, arXiv: machine learning, Hugging Face blog, Quanta Magazine, arXiv: computation and language.
Nobody gets this one yet. It starts when you do.
Start this one