Benchmarks measure benchmarks
When a model tops a leaderboard, it has proven one thing beyond doubt: it is very good at the leaderboard. That is not nothing. But a test is a controlled, artificial slice of the world, and being excellent at the test is not the same as being useful to you. Optimize hard enough for a number and you get a system exquisitely tuned to the number, and quietly indifferent to everything the number left out.
A benchmark score is a promise about a test. It is not a promise about your Tuesday.
The gap between the demo and the deployment
A demo is a performance staged under perfect conditions. Deployment is a stranger, tired, on a bad connection, asking something the script never anticipated. The distance between those two is where the actual work of the field lives, and it is the part almost nobody competes on, because it does not film well and there is no leaderboard for it.
What is worth optimizing for instead
We chose a harder set of questions with no scoreboard attached. Does it actually help you, or just impress you. Does it tell you the truth when the truth is unwelcome. Does it remember who you are next week. None of those have a benchmark, which is exactly why they are worth building for. The industry is optimizing what is easy to measure. We would rather build what is hard to fake.