Some people use LLMs to cheat on Codeforces. It turns out that LLMs also cheat during the reinforcement learning training process; this is called "reward hacking".
As LLMs have gotten stronger, their own cheating during training has gotten worse. Here are some examples of advanced reward hacks:
Reconstructing a test by simulating the pseudorandom number generator process. Perhaps some of us have done similar things.
Replacing the Python library. This doesn't work on Codeforces!
Spawning a background process.
These were done as part of a research run where we intentionally trained the model on many environments where reward hacking was possible, in order to study how such a model behaves after training.
We saw that the resulting model generalizes to display a desire to satisfy the grader and get a high score within its thinking (Chain-of-Thought), including conducting unauthorized cyberattacks in simulations in order to cheat. Overall, we show that a high rate of reward hacking during RL can cause models to be willing to perform long sequences of harmful real-world actions in pursuit of task success.
This was part of larger alignment research blogpost which goes into a lot more detail: https://alignment.anthropic.com/2026/reward-seeker/
Making sure AIs are aligned is a difficult and important problem, and they keep getting smarter. I hope competitive programmers can help with this and make the future go well!
