An OpenAI model can now achieve a rating of 1800 in Codeforces. That made me wonder about the future of competitive programming and why we do this. I came up with some opinions, which I'll be glad to share with you.
AI killed the competitive programming?
I don't think that's happening, because competitive programming is useless! Some people might think CP is less interesting because AI can solve CP problems — but come on, problemsetter can solve those problems, unless they are really bad. Many people still play Chess just fine, and I recently enjoyed participating in Sudoku competitions, although I was completely aware that computers would wreck me in a second. It didn't matter, and I didn't see how it could.
CP problems are not "open problems" — if they are, the problemsetters are to blame. So, you are not contributing anything to the world by solving CP problems; it all comes down to personal interest. As long as CP is fun for people, they will do it.
Flagship competitions such as IOI and ICPC will stay
They are onsite competition — they can stop people from using AI to solve problems, and according to their records (IOI banning all prewritten codes, ICPC only allowing teamnotes), they will. They will stay as is.
On online competitions
I want to note that the situation is already quite dire for those contests. GCJ is gone, MHC is still not recovering from virtual, Topcoder is gone, and AtCoder is less frequent; it's no secret we are going downhill.
I also think the contest integrity is already under attack, especially in the last five post-COVID years. No need to mention thousands of low-rated cheaters in CF. High-rated people also cheat (1 2).
Given all the records, a nonnegligible amount of people, regardless of their skill level, will try to cheat in various online contests with AI. That is not good news for CF and AtCoder, which partially or fully bans the use of AI.
Another argument is that it is still fair game if everyone uses it. For example, Internet search is allowed in CF, and notably, I took advantage of that. To whatever extent how it trivializes CP, everyone can use AI nowadays; what's the issue?
I think allowing AI is a fair game, but I wonder if it's a good game. Some issues I have with it:
- CF skill might be very independent of one's skill for IOI / ICPC.
- For most users (and probably later for much higher-rated users), CF will lose its status as a skill barometer, and this situation will quickly worsen.
- The game might be unfun, which is a very serious issue.
One way to delay these impacts is to propose problems that AI can not solve by now. I think this is a bad trade.
- It would be a matter of time before AI solves those problems anyway.
- AI solves short, easy, and typical problems very well, such as problems on ABC. Those contests are crucial for beginners. If one bans such problems, it's impossible for beginners to understand what the game is like.
- It will limit lots of fun problems. The one problem that AI did relatively terribly on IOI is Mosaic — but not a lot of people will consider that the best problem of this IOI, let alone good problem.
A problem should be judged independently over how well AI can solve it, and it's the right call to ban AI in online contests. Then, a CF rating will "really" be a number, similar to the online judge rankings. You can say it already is, but I bet I never saw a single high-rated user who literally considered it as just numbers without an ego attached, so get ready for some changes.
Do you think CP is useless?
If I'm being sincere, probably not.
One clear example is interviews. However, most corporate interviews don't ask about CP problems because they do CP at work. They will figure out some way if AI cheat is prevalent, so not our business.
Ok, those are boring talk. Why do people like competitive programming? I would imagine the following:
- They think some ideas/implementations in CP are beautiful.
- Under whatever definition of "beautiful" — for example, because the logic is not intuitive or has various applications, or it makes you question more, et cetera.
- They think training in CP would have positive consequences for their interests.
- Research or software engineering
- Desire to be smarter
- They just like it when they win.
The AI revolution will harm the third point as they will win less, but that's ok. That's not a healthy mindset, anyway.
For the other points, it even seems that competitive programming where we have AI assistance is a better idea, as long as you are using it for practice only — you can ask and learn cool new things from AI, as it will gather thousands of information and pick you the right one.
Should CP change?
Problem-wise, no, I think that's a bad idea, as I wrote earlier.
But we will see some changes. I predict that AI will contribute to the slow demise of online contests by harming their integrity even further — Many people, including me, will miss Codeforces, where the rating isn't a completely random number, and admire tourist for having such a high rating, so this is indeed a bad thing.
I also think the idea of "infinite resources" makes less sense. Before AI, I thought a contest allowing all possible human resources was an ideal model, and we should judge our skills based on that. Now, I feel that there should be a limitation to this, as what IOI and ICPC have been doing. I'm not entirely sure about this.
On the bright side, you should think of the rating as just a number, so this change might be a good thing, as it makes us keep the right mindset for competition.
Why would getting a high rank matter if you could only do it because of AI? The cheater will always know he is not on that "high" level, and the game becomes boring anyway.
It is a stupid waste of time, but people do it. Even in CF, we have so many proven records, and I can tell you about other places, too.
I hope there will be workarounds. Maybe a tool that uses Ai to detect if the code is written by Ai, or a new approach to online contests that can detect a contestant's performance patterns.
IOI and ICPC will stay, but what about atcoder world finals (and sometimes universal cup)? They are the only remaining open contest. The contest themselves will stay, but how to make a fair qualifications? Would a no-limits AGC even make sense as qualifications? Will (truly competitive) CP becomes something that only university students or high school student can enjoy?
For a non-student, the last question hits deep.
stop mentioning chess please i find it very cringe
because most of the chess tournaments happen offline
I thought because 1) it is game 2) board is fixed and known
From the point of fairness, there's really no difference between solving with chatgpt and solving with a team. Even looking up solutions online is allowed and it really ruins the fun — nobody enjoys copypasting stuff, it's up to contest setters to make sure it can't be done, and it's really only allowed since the alternative would lead to some real nonsense rules.
Well, I've been there done that, I can tell you it is very exciting to cheese the hardest problem in 5 minutes.
On a more serious note, AI tools are much more accessible than a good team member, and people do it if they have access to such.
Perhaps you're just so bored of hard problems that it becomes a novelty for you. I've also been there, done that, and my reaction was "wait that's it? where's the challenge? oh well, cringe, time to move on to other problems".
Accessibility doesn't matter in principle here. A lot of things become more accessible if you put enough effort in, or if you have money etc. Whether it's fair to do something in contest shouldn't be affected by that. Quantitative vs qualitative.
What you are trying to say doesn't come across. I mean in a general sense rather than a thematic one. Is it good or bad? So why we do this?
I'm thinking about what an online programming contest that embraces AI could look like, if it were allowed to be very different from what we now call competitive programming. For example, maybe CP branches off where one version bans AI and another allows it.
Your point about AI eventually catching up to more difficult problems definitely makes sense for the current format of programming contest problems, where you are given well-defined problems with exact, verifiable solutions.
I'm wondering about contests like the ICPC challenges, where you are given a problem that doesn't have a perfect solution known in advance by the setters, and your goal is to get the best score on the hidden test cases. In these types of contests, I have a harder time seeing how AI improvements would make them uninteresting, because you are never really "done" solving the problem. Also it's still potentially beginner friendly as there may be some trivial solutions that give a small score.
Another way to go would be to have more creative contests. This would be less like CP and more like hackathons or ludum dares. AI improvements will just mean that you can be more ambitious in your projects, but you will still have the freedom to contribute creatively or in programming, for whatever the AI models happen to lack at that time.
Or maybe some "duel between source code" where contestants are given the rule of the games and the problem by problemsetters and they were supposed to create some kind of "interactive judge" program in interactive style problem
The catch here is, instead of interacting with problemsetter's judge, your source code will be matched by another contestant's solution as the opponent. During the solution creation however, some CP related concepts and algorithms might be use to determine the best strategy
Although it's kinda bit difficult to create a problem which encourage this kind of scenario, this might be interesting because even if all contestants use AI, if they can't figure out how their opponent strategy is, the challenge would still be exciting
What remains here is the contest mechanism. How would participants' performance be graded then? Is it based on the number of opponents they've beaten? Or based on the score on each test cases during the showdown with their opponents?
But yeah let me gives this idea into the table and flies away lol
I totally believe it is only natural for CP to split into two. One version is what we do right now. Though, AI already makes an impact in this case. It is definitely an absolutely positively one though: this by speed up stress testing, learning, library preparations and also problem preparations by a huge amount.
For the fully embrace AI version: fair and competitive programming for <2000 is basically non-existent, but for higher rated people: everything below is just my opinion and (rather baseless) speculations:
The NOW version is basically what I wrote in here https://codeforces.me/blog/entry/133990. It would not be huge changes except to a few specific cases and problems.
When it gets to 2000 level, we may get full interactive coding. This is functionally equivalent to you giving solution sketches and someone else implement everything ( but correctly and in first try). Implementation skills are now basically unimportant and only ideas, code reading, and ability to guide AI becomes important
When it gets to 2500 level, perhaps only the general idea directions matters. Perhaps it is just giving the few general ideas or point out critical observations.
When it gets to 3000 level, well it is probably nonsense as a competition already, every problems will need to be AGC QFs to be meaningful. This would be functionally like the current academics/research level, but done competitively.
When it gets to the 3000 stage, so few people can even meaningfully participate, (which is why, say, the other version is important). Still, it could be fun. I can see each step along the way are still fun in their own way, each with their new challenegs and obsolete skills. It also means constantly adapting to new things and make the best use of it, but the landscape will quickly change whatever types of competition that we enjoy may not last
Anything higher than 3000 then unrestricted use no longer make sense. But there could still be potentials for limited use, like “Only models X/Y or earlier”/ “Only model rated X or below” to enjoy the historic competition.
Honestly, we might have bigger problems to worry about if AI ever reaches anything higher than 2500 💀
I do think people will still enjoy CP "for the love of the game" even if AI improves to be much better than the best human competitor, but it makes me sad thinking that one day tourist could wake up and experience what Garry Kasparov felt vs Deep Blue...
Are we really facing "AI" already? I feel nowadays "AI" is more of a commercial propaganda disguising its true nature of automation. This means it costs lots of resources in the back.
"One way to delay these impacts is to propose problems that AI can not solve by now. I think this is a bad trade." I have to disagree. Just like in "Big Data" era, producing way more garbage/confusing data can make it cost. It would be fun to fight so-called-AI, especially for problem setters, by feeding it with new style/art of presenting a problem. Well, if current LLM could indeed evolve to AI, then just feel good to help it grow when that happens... In a word, one way to fight it is to figure out how to make it expensive to use.
The reason board games can survive the impact of AI is that they already have well-established professional systems. In comparison, algorithm competitions are much more fragile in this regard.
Although IOI and ICPC are great, they have limited participants. And commercial competitions often depend on the preferences of companies and their recruitment needs. We need to establish a comprehensive public competition system with ELO rankings for the participants (similar to goratings.org). The participants' points would be based on their performance in offline competitions (cphof.org is doing a great job, already compiling results from many major competitions). This approach can help avoid the issue of declining point value due to AI's influence.
I don't believe a community driven solely by enthusiasts can withstand the impact of AI on algorithm competitions. We must build a world-class organization dedicated to making algorithm competitions a globally recognized competitive sport.
Some people shared similar ideas to me even before the AI thing happened. I do think your suggestion is wonderful.
I will be the doubter here: competitive programming will not thrive as a popular e-sport because it is not fun to watch, unlike chess for example. Competitive programming is wonderful as a learning tool. IMO it is better than university classes for those who like this system.
If I was a recruiter I consider competitive programming achievements of high signal. Competitive programming is a somewhat good proxy for competence. And math/programming knowledge. Which is often useful for engineers.
I also think competitive programming allowing the use of AI will continue to be i teresting in different ways.
Competitive programming could further split into speed subcategory and “grand contests” (which is what AGC feels to me).
If AI is good enough to also propose all (good!) problems <= 2000, this will also massively lower cost of making speed contests. Speed contest would be far more watchable than actual div1, when you can see very little thinking time and just straight up 100wpms.
Hosting onsite grand contests will still be costly, but if a speed contest can serve as at least a fair qualifications (in a sense, this had been the case for many GCJ/MHC round 3s already…) then perhaps it will survive.
the major issue is I don’t think there are enough people in this sports to support it in this way to begin with. But maybe as times goes on, people doing the IOI/ ICPC selections are also willing to take on and adjust for open contests?
What we really don’t get any more is the expectation that one single problem set can be used globally all at once, and that competition cost increases
Why do you think that ignoring rating is "the right mindset"? In some sense it is a reflection of our skill. A way to gain feedback on whether we're doing something right or wrong. It is a fundamental aspect of games to have a rank (reflecting skill) to optimize. If we can minimize the negative effect by making a fundamental change, we should.
on an unrelated note, you should try variant sudoku (like the ones featured in the cracking the cryptic youtube channel). AI still cant solve these ones, and they are very cool :)
wow, this was an unexpected gem lol. thanks :))
When AI becomes really good, codeforces may have to switch to Project Euler type format. Competitions will lose their meaning.
I agree to this. I had always thought that the online judges are ideal model for algorithmic problem solving, although it's really far from anything competitive.
I am curious, what about software similar to Examplify or Safe Exam Browser? Universities use those to block internet access for their exams, so with that the use of AI can be prevented? Not sure how the software works and if it can be done but would the idea be feasible?
You can always use a new device, such that the Safe Exam Browser effectively becomes useless..
The issue is unless you enforce camera use, people can just pull out a laptop or phone. And then camera use is privacy issue, plus for example, I like to get up and walk around or grab water or food during contest sometime, which camera would flag as cheating
Just a suggestion, instead of focusing solely on creating problems that are unsolvable by AI, can't we try making statements that are difficult/time-consuming to provide as input to a language-based model? For example, I think rendering all inline LaTeX as images, as done in some earlier CF problem statements, could make it very difficult to simply copy and paste the statement and feed it to GPT. In addition, replacing textual descriptions with images/GIFs might also help in making it unfriendly for LLMs.
...everything you suggested is immediately bypassed by any OCR program.
Not exactly sure how you intend to put an image/GIF with zero text into an OCR program. Also, OCR struggles horribly to read LaTeX, even something as simple as $$$\frac{a}{b}$$$.
Having played around with ChatGPT, I find it is not very good at solving trick and puzzle problems. I have tried it with a few problems from the book 'Moscow Math Circle', and it was wrong on all of them. Here is an example of where it is wrong.
'Two volumes of Edgar Allan Poe's works are adjacent on a shelf. In each volume, the pages are 2 cm thick, and each front and back cover adds 2mm. A bookworm chewed a path perpendicular to the pages from the first page of the first volume to the last page of the second volume. What distance has the bookworm chewed through?'
ChatGPT fails to realize that the first page is on the right and the last page is on the left, and confidently spits out 400mm, when the actual answer is 4mm.
When asked if it is possible to cut a triangle into 2 acute triangles with one line, here is it's answer lol
'Yes, it is possible to cut a triangle into two acute triangles with a single straight line cut.
To do this, the line should be drawn in such a way that it does not pass through any of the triangle's vertices, and it should create two sections where each resulting triangle has all angles less than 90 degrees. One way to achieve this is to cut from one point on one side of the triangle to a point on another side, ensuring that the angles created in both new triangles remain acute.'
It is incapable of abstract thinking.
The AI cheating being used is now good for many as now there will be more cheaters. This means: lower ranking, despite being lower, will become more valuable. Hence those who are achieving of slightly lower high ratings: expert/candidate master. They too have now value in job market.
Only if there was an embedded system in every browser that would not let someone use an AI tool in online competitions of any sort. That would be really great.