"We've been testing Claude on Pokémon for several years."
That came from the Anthropic team tonight, on the Claude Partner Network Connect, right before the line "I won't bore you too much with the benchmarks" around Fable 5. (Full quote from the call in the first comment.)
Unexpected? Completely.
Refreshing? More than they probably realized.
Very few of us actually experience these models through a benchmark. We experience them through our own peculiar work, the messy, specific, four-hour jobs that never show up on a leaderboard.
So having your model play Pokémon demonstrates how it sticks with a long, messy, multi-step job for hours. Fable 5's headline trait isn't raw IQ but long-horizon autonomy. It's about doing the work that models haven't been able to handle until now.
That's the Pokémon benchmark.
FYI here are some other great updates from the Claude team:
- Advisor mode (Higher Models for Harder Work). Olivier Legris posted about this a few days back, but the real play is to have Sonnet do all the work and call up to a frontier model (Opus 4.8 or Fable 5) only on the hard moments.
- Claude Tag (Claude on Slack): a continuation of Claude's north star vision to handle all tasks from a single window. The real value, like all Claude form factors (meh...) is when you connect it to your tools (Notion, Github, Lemlist). Claude on Teams also penciled to arrive.
What's the longest job you'd trust a model to finish unsupervised today?