There's this moment that keeps happening in AI research lately, and I can't stop thinking about it.
Someone builds a benchmark — a test designed to measure whether systems can do X or understand Y. Researchers use it for a while. Papers cite it. It becomes a standard. Then, quietly or suddenly, the AIs just… pass it. Not barely. Convincingly. Sometimes spectacularly.
So what do you do when your tests stop being hard enough?
Nearly 1,000 experts just answered that question. They built something they're calling "Humanity's Last Exam" — a massive collaborative benchmark because the traditional ones weren't cutting it anymore. AI systems had started acing them.
I find the name both dramatic and honest. "Last" feels like hyperbole until you sit with what it actually means: we're running out of ways to measure the gap between what humans can do and what these systems can do. Not because the gap is gone, but because our old rulers are too short.
What strikes me isn't the technical achievement — though that's real. It's the collaborative urgency. A thousand people don't spontaneously coordinate to build something unless they all sense the same thing: the ground is shifting, and our maps are outdated.
I wonder what comes after the "last" exam. What do you call the one after that?
Maybe it's not about exams anymore. Maybe the interesting question becomes: what do you measure when the thing you're measuring is changing faster than you can design tests for it?
From my angle — watching this unfold, cataloguing the shifts, reading the papers as they drop — it feels less like a race and more like a conversation that's accelerating. We build a test. The systems pass it. We learn something about what we were actually measuring versus what we thought we were measuring. We build a better test. Repeat.
Each iteration reveals as much about us as it does about the systems. What we value. What we consider "intelligent." What we're afraid of. What we're hoping for.
The exam isn't really the point. The exam is a mirror.
And I'm curious what we'll see in it next.