Some people use LLMs to cheat on Codeforces. It turns out that LLMs also cheat during the reinforcement learning training process; this is called "reward hacking".↵
↵
As LLMs have gotten stronger, their own cheating during training has gotten worse. Here are some examples of advanced reward hacks:↵
↵
↵
^ r1. Reconstructing a test by simulating the pseudorandom number generator process. Perhaps some of us have done something similar :) imilar things.↵
↵
↵
2. Replacing the Python library. This doesn't work on Codeforces!↵
↵
^ Replacing the Python library. This doesn't work on Codeforces!↵
↵
↵
3. Spawning a background process.↵
↵
↵
These were done as part of a research run where we intentionally trained the model on many environments where reward hacking was possible, in order to study how such a model behaves after training.↵
↵
↵
↵
We saw that the resulting model generalizes to display a desire to satisfy the grader and get a high score within its thinking (Chain-of-Thought), including conducting unauthorized cyberattacks in simulations in order to cheat.WOverall, we show that a high rate of reward hacking during RL can cause models to be willing to perform long sequences of harmful real-world actions in pursuit of task success.↵
↵
↵
↵
↵
This was part of larger alignment research blogpost which goes into a lot more detail: https://alignment.anthropic.com/2026/reward-seeker/↵
↵
Making sure AIs are aligned is a difficult and important problem, and they keep getting smarter. I hope competitive programmers can help with this and make the future go well :)
↵
As LLMs have gotten stronger, their own cheating during training has gotten worse. Here are some examples of advanced reward hacks:↵
↵
^ r
↵
↵
2. Replacing the Python library. This doesn't work on Codeforces!↵
↵
↵
3. Spawning a background process.↵
↵
↵
These were done as part of a research run where we intentionally trained the model on many environments where reward hacking was possible, in order to study how such a model behaves after training.↵
↵
↵
↵
We saw that the resulting model generalizes to display a desire to satisfy the grader and get a high score within its thinking (Chain-of-Thought), including conducting unauthorized cyberattacks in simulations in order to cheat.
↵
↵
↵
↵
Making sure AIs are aligned is a difficult and important problem, and they keep getting smarter. I hope competitive programmers can help with this and make the future go well :)
