As the main problemsetter of Codeforces Round 1000 (Div. 2), I know that a lot of you were somewhat disappointed to see that it was not a Div.1 contest. Though I am not the one who caused a lack of Div.1 contests, I understand your feeling. So I present to you the great surprise: The Anti-LLM Evaluation Report for Codeforces Round 1000 (Div. 2), the first of its kind on a Div.2! In this blog we discuss about how the round combats against LLMs (especially focusing on OpenAI o1, the greatest of its kind while we prepared and tested the problemset), by looking at the timeline of how the problemset changed.

So, let us begin with the initial problemset we had when the testing began. (The task names are anonymized, unless they are released to the public before or in the round)

In the beginning, we had a problemset that looks like this:

• A' — B — C0 — D' — E' — F0

(If you are confused with the meanings, (letter)' means it was not used in the final problemset but existed, while (letter)0 means that it appeared in the final problemset in a different form.)

So, you may see, at least half of the problemset has changed in the testing phase. How did o1 do against this revision of our problemset?

• A': Not Solved.
• B: Not Solved.
• C0: Solved.
• D': Not Solved.
• E': Not Solved.
• F0: Not Solved.

Okay, so it already was a bit strong against o1, but C0 being solved by LLMs while A' being not solved looked too odd to me. It felt like LLMs would get a too large unfair advantage if it was kept this way.

Now, the timeline of the problemset begins. (Note that the timeline is purely based on my memory and whatever information is left in the testing mashup)

• The first task to get swapped was A'. A lot of testers felt A' a bit too hard for its position, and we had to find a new one. Luckily, we found A'', and swapped A' with it. A'' was solved by o1. But we believed that would be fine if we could find a replacement to C0 that isn't solved by LLMs. Back on that later.

• The second task to get swapped was F0. F0 was a version of F that only asks for uniqueness. It was interactive, and we didn't really have a good way to force them to solve online. And there were unexpected solutions even in the online setting, so we made F that asks for counting, and split it to two subtasks. As you might expect, F was not solved by o1.

• The third task to get swapped was C0. We swapped it out for C' initially, but the testers did not like it (probably it was too hard for its position), and we swapped it back to C0. C' was not used afterwards, and it was not evaluated against LLMs either.

It was very hard to find a replacement for C0. Until...

• We found C. It was just a random idea I pulled while ranting about how hard it is to make tasks on the position. But somehow, very surprisingly, it was not solved by o1. I am still surprised about how o1 cannot solve it, don't ask us, go ask Sam Altman instead. Anyways, this replaced C0. Very lucky!

• At this point, most people struggled on E'. We decided that it is not a good fit for the position. As a replacement, we found E and replaced E' with it. Thankfully, the testers pointed out that it's just the fine difficulty for the position. And it's also not solved by o1. Another good one. Later, E' became COUNTGOOD on CodeChef Starters 167.

• Some time after that, some testers pointed out that D' is very similar to a task from GCJ. But worry not, we have discussed this out to balance the difficulty for a long time. Surprise: C0 became buffed to D. And it is now not solved by o1! Nice.

Around this time, I asked rewhile to test. He knew better about prompting LLMs, so I asked explicitly to test only using o1 (and he gladly accepted). Here was the result.

• A'': AC
• B: -11
• C: -4
• D: -2
• E: -1
• F (Easy): -2
• F (Hard): -1

So yes, I believed that o1 will die horribly if it's the same way.

• After this, I just changed A'' to A, which was much easier than A'' for humans. It is solved by o1, but it's fine now.

There are some omitted changes also (Such as a task proposed for Div2D but rejected immediately after I found that it's a Div2B), but these are all of the significant changes.

So the final result for o1 is as follows.

• Expected Score: $$$498$$$.
• Corresponding Rating: 911 (762 rating points less than the 1673, initially claimed by OpenAI)

The point we need to focus on is that, when these LLMs came out, people thought it's the end of the world for Div.2, but not yet! It might not be the end of the world! But it takes more effort to combat them. Here are some things I found, as a guidance for problemsetters who want to make your Div.2 contest strong against LLM.

• You might have noticed, the round has significantly different problem styles compared to the usual Div.2; that might have helped combat them.
• Maybe it could be time to change the meta again, after the last time it changed, which was probably when 1188B - Count Pairs appeared? I don't know. Up to the next Div.2 problemsetters to decide.
• In terms of problem style, problems that require multiple small observations will be generally more robust against LLMs than those that require one or two big observations. Stronger LLMs usually get one or two first observations correct. Making more small observations will make them also suffer from limit on number of tokens. For example, on problem C, o1 got the immediate first observation, but got WA by choosing the first $$$k=\mathcal{O}(\sqrt{n})$$$ and bruteforcing $$$k \choose 2$$$ pairs.
• It will require you significantly more effort to make easier tasks than to make hard tasks. Position C required us more effort than position F required us. For harder tasks you can care less if your task isn't extremely classic or already known, which I assume is usually not the case.

Maybe for a few things noted I might be not the closest to the ground truth. Tell me in comments if you need to point something out.

Also, for cheaters that tried to use o1:

I gave you a hint already. I hope you learn from negative delta, and become honest and diligent again. I hope you are a sane person. I, myself, truly improved only after moving on over bad behaviour. Yes, I had cheated back in the days, and now I became as honest as one could. I believe you can be better also.

Please take this opportunity as a lesson.

Thank you for reading.