I don't feel like my attention-seeking senses are fulfilled with just comments, so I think it makes more sense to write a blog about it. Codeforces has evolved over the years and I think it might be a good move to talk about some relicts of the past that are still there, or even possibly do something about them. I'd like to note, that it's possible that there've been some blogs about this topic from the admins and I've missed them and lack information because of that. In this case, please pardon me.
After the first two Spectral::Cup rounds I've commented on things that in my opinion are somehow harmful.
The first comment was this one. It was mostly about the solution being judged twice on the same tests. I'm not much against it, as it seems to be well defined in theory: you have to pass both times to get AC (where the second time it might include extra tests), which is fine — not perfect, but ok, rules are rules. But from what appears here (or at least from what I understand), this guy's solution got rejudged after it got TLE on the second (system) testing. For me seeing that my solution passed the pretests means nothing definitive (mostly because of possible extra systests, that will be touched more in the next paragraph), but apparently that's not the case for everybody and maybe it's bad of me to not follow what I'm observing about recent contests.
The second comment was this one. This one is less about something being unfair in a direct way, but rather about hurtful conventions/assumptions. During this contest having the knowledge that systests are not going to contain any additional tests (except for possibly unexpected hacks) would get me a better place -- but seeing no announcement that in some problems the pretests are equal to systests (which appeared multiple times if I'm not mistaken) I've assumed that I can't assume so and resubmitted the code (I think that both version of the code are actually correct if you make some extra observations, but the discussion is about decision making). And that would be totally fine, but I kind of think that this assumption is actually correct nowadays and systests are actually supposed to be equal to pretests and that people who assume so are ahead of others (because of being less likely, or even fully unlikely to resubmit).
I kind of imagine the Codeforces team to try to be nice to users and therefore contain as much tests in the pretests as possible, but because it totally wasn't the case some years ago, it's hard for me to just assume so without seeing any clarifications. I also assume that the rejudge of the solution from the previous paragraph happened because of cf team wanting to be kind, but that couldn't happen if pretests wouldn't be equal to systests, right? So if pretests = systests is really such a "rule" right now -- why not come clear about it? Otherwise it favors people who make unspecified assumptions, or even worse, people who are close with cf team and possibly got a (in theory) not very meaningful bit of information like "yeah, we usually just set systests=pretests nowadays" or something like that.
As I'm already writing this blog, I think it might also make sense to remind about another Codeforces' bad choice, that I think is still there (I might be mistaken though and maybe it got fixed). This one is a pure choice and if wanted might stay here, but I still think that's the worst option out of three -- to come clear, by a pure coincidence, it's something that also have stripped me out of a better place (and even a victory) some years ago, but I don't think that that's that crucial, as it probably influenced the results multiple other times as well, including ones beneficial to me or the opposite. It's about whether a submission which fails on sample tests should get a penalty or not. Assigning a penalty would obviously make sense as I think on most other competitions this is the case and it just works. Not assigning a penalty is also a right (and I guess preferable) choice as Codeforces wants to be nice and from what people were saying, it's possible to submit a wrong code/to a wrong problem or format the output incorrectly or something -- this makes sense, as then the problemsetters might try to "catch" such solutions and prevent them from getting a penalty, and I totally support this option and I think it should be the active one. So it turns out, that there is also a third option, which I don't really understand, but it turns out to be the active one -- to not assign the penalty only if the solution fails on the first test, no matter how many tests are marked as samples. For me, this is clearly because it was just implemented this way temporarily and not fixed later. In this setting it obviously contradicts with the idea of problemsetters being able to detect the wrong formatting for example -- imagine two samples, one with answer YES and second with answer NO -- it depends on the pure randomness whether spelling each of those incorrectly results in a penalty (yes, I know that most times outputting something like "yEs" is also accepted, just try to stretch the example a bit if you need it). I can also obviously imagine cases where it wouldn't save someone from getting a penalty when submitting a wrong file -- I imagine a problem with the output on the first sample is "0" and the next samples have more complex outputs -- it could be rare, but I'd assume there is a non-zero number of such problems and I totally imagine the wrong code outputting "0" just because. So again, not assigning the penalty for failing any of the samples seem strictly better to me?
I also know that in theory requiring extra work from the CF team wouldn't be super nice, but in the past I learned that they are super responsive about some stuff -- for example I remember preparing a round and saying something about the polygon and it got fixed/improved within a day or so. I also assume that Codeforces is very much a site "from competitive programmers for competitive programmers", so I don't think that mentioning MikeMirzayanov and KAN and inviting them to the discussion is something bad. I'm also doing so because I really think that some things are simply a bit dusty and talking about them might make sense and improve the platform.
Thank you for your attention, good night.
UPD: I forgot to mention something. I understand that not clarifying whether the pretests are equal to systests is somehow "comfortable" for admins/problemsetters, but I think that the biggest usage of that fact (that I'm aware of) happened at this one round (it was a couple of years ago) when one of the problems had deterministic model solution, but it was possible to come up with non-deterministic one and I think that only I and one other guy somewhere down the standings did it this way. So for the systest phase, the problemsetters added like 700 extra tests to test against such solution and if I'm not mistaken I've won this round, so that definitely felt like a personal attack on me. So I'd say that the biggest usage of the comfortableness should've not really happened and rolling with whatever the state of the systests is is the way to go (except for saying that all unexpected hacks will be added for example).
Based on the Polygon rules given to authors, it has been common practice to set systests=pretests(+hacks) for the past few years. Perhaps there are some niche cases where there is reason to deviate but in most contests I believe this convention is followed (in fact, Polygon will give you a warning if systests!=pretests).
Another convention of note is to have at least one sample for each possible output format. This is relevant for problems in which there are two possible output formats (e.g. a construction or -1 if impossible) but do not have any samples for one case. Based off this convention, it would be fair to assume that only one output format is possible (e.g. there is always a construction).
So what does this all mean? Maybe it would be better if CF made it a known convention to have systests=pretests(+hacks) unless otherwise announced. As was discussed here there should also be a well defined rule on how many rejudges a solution gets. As for the second convention, imo we should just stop setting problems like that (e.g. say that there is always a construction) unless there is some benefit I am missing.
Enforcing only one output format sounds limiting — I’d say not assigning a penalty for failing any of the samples sounds strictly better as well.
To clarify, I'm not saying that every problem should only have one output format. I have no issues with problems where you output a construction or print -1 (in which case, according to the convention, both output formats should be included in the samples). My point applies to problems that say "there may or may not be a construction" but secretly there always is a construction. In such cases, people who make an assumption based off "oh there are no samples with -1" are benefitted which doesn't make much sense.
Totally agreed, failing to format the output as desired by the problem and similar issues shouldn't be a criteria for judging the correctness of a solution.
I might be wrong but isnt it the user who has to prove and decide whether a construction always exist or not?
Yes, that is the intention with the problem not specifying whether there is always a construction. The issue is that if someone knows the convention "at least one sample per output format" and sees that every sample has a construction, they can assume without proof that there is always a construction. This would give them an (in my opinion, unfair) advantage over people that spend time proving whether a construction always exists.
I would say it might be a gamble because if they found out a case if they cant find a construction then they will waste some time in implementation. But yeah it makes sense to include such samples or state otherwise.
You're going at this from a wrong angle. If there's always a solution but the problem hides it from you, that's already poor design.
Fair enough, seems like more reason to stop setting problems this way (perhaps this can be another unspecified convention...)
A good counter-convention to the output format issue would be to always make the not-constructable sample a "stupid" one like $$$n=0$$$ or $$$n=1$$$ degenerate case, but all "normal" cases are constructable. I think it would usually be easy to tweak the problem statement to have such an edge case.
Love you too
Yeah I don't think the extra rejudge on that one submission was fair. Especially since AFAIK TLEs get reran anyway to ensure it's not a one time spike in the judging system.
Agree. Especially about the pretests and system tests.
Um_nik said the same thing 2 years ago about this information not being public and the advantage it creates, and it's still not solved.
Why are your submission got skipped?
If you resubmit an AC in contest, the previous AC gets skipped.
huh?
Yup — passing the pretests doesn’t mean that you’ll get an AC — or at least it didn’t mean that some years ago, that’s why I happen to resubmit (I’d say quite often compared to other people). According to the above ss of Um_nik’s blog, I was just uninformed and it seems that setting a round (or reading um_nik’s blogs) is a way to improve your performance and get you MacBooks — which at least is better than learning it by knowing somebody.
Regarding TLE,
As far as I understand, a submission is judged as TLE if it gets the TLE verdict twice in a row during the system tests. You can best observe this in the system tests in Edu-style rounds, when some hacks cause many submissions to fail with TLE. In system tests in such contests, you'll observe a small set of submissions get requeued periodically, and they all get TLE eventually (so you'll see a burst of submissions getting TLE in the status page). I assume this is done to save submissions whose running time is close to the limits.
My opinion: Regardless of the existence of run-twice system in the system tests, the rejudge in this case shouldn't have happened. Submitting a slow solution that's close to the limit is itself a risky move, and you shouldn't complain if it becomes TLE eventually. This is true even for problems where pretests = systests.
If I understand correctly and both "systests = pretests" and "Don't worry, if you tle on systests, it will be rejudged." hold, then the only reason to resubmit is because of being afraid of hacks, right? I'd say then that the number of resubmits would be way less if this information would be public (assuming that they are true).
To show that the fact that this is not public really influences results and performances, here are some statistics about all current LGMs — for each of them I've checked in how many rated rounds (during the last 3 years, as I don't know for how long it's been the case) they had at least one submission judged as "skipped":
It varies from not resubmitting at all to resubmitting in more than 20% of rounds. Not sure if I'd call it a skill issue though.
The conclusions seem rather clear (at least for me), but I'm still hoping for some official comment on the matter, as it's hard to simply accept those, for example because of the situation mentioned in UPD.
While the subject of your concern may indeed affect chances winning macbooks, I think you're missing a more important problem, which also affects your chances for material prizes.
NanYan in your LGM list is very likely a cheater. User with similar nickname who scored high places in rounds was banned on AtCoder.
XVIII is a cheater who scored even higher places and even won one Div 1. He also scored 3rd in Spectral::Cup 2026 Round 1 (Codeforces Round 1094, Div. 1 + Div. 2), and despite numerous blogs calling him out, according to this admin's comment https://codeforces.me/blog/entry/153179?#comment-1362536, they contacted him to give instructions to receive a prize for 3rd place.
Good news is that XVIII is banned now. Bad news is that he was only banned after about a month after he reached LGM, and after he could've received the material prize, and there are other LGM cheaters who are not banned.
Wouldn't it make perfect sense to automate cheater banning? I think that most people being able to vote would result in a few disasters, but for example allowing only people with rating at least something to vote should do, right? If not voting about banning someone, but even about suggesting these people for a check, I think that if admins would be able to enter a profile of a person and see that for example "95 out of 100 grandmasters voted that this person is a cheater", it would significantly help with cheater detection, right?
In most online games (even in the calm ones, like on chess.com or something) you can report people, maybe cf should have something similar?
I implemented a community overwatch system here. Although, I worked on this idea before your comment.
Maybe if it gets popular, the CF admins will do something similar.
Let me bump this once, maybe was simply skipped.
i see what you did there
Yes, I guess there's no official document pointing out the fact that pre=sys, but I heard from my friends who were problemsetters that it seems to be a convention, which affects the performance of those who aren't aware of this rule.
Based on the Polygon rules, "Pretests must be equal to tests.".
Hi! Thank you for publishing your concerns, they are very valid.
It's true that the conventions have been updated several times, and various issues arose since the official version of the rules has been published many years ago. I'm working on an updated set of rules that includes these updates.
For now you can assume that systests are always equal to pretests plus hacks unless mentioned otherwise, and hacking is disabled for several easiest problems of the contest.
Also, although solutions are physically judged twice on pretests due to a system bug, it is not supposed to work like that. Therefore we try to rejudge any unexpected verdict arising during system testing on a test that was included in pretests. Hopefully soon the system will be fixed and we won't have this double-judging anymore.
As far as penalty on non-first example test cases is concerned, this is purely a feature against submitting a wrong file, and not against preventing incorrect formatting. I believe the probability of passing the first example with a completely unrelated code is very low, in particular because of multi-tests now. So I don't think it's worth changing.
P. S. please do not consider the Polygon rules document linked in the comments as something guaranteed to be followed in a particular round. Although these are default rules for authors, they can always be overridden by the coordinator.
The Codeforces help page is still that 15 years ago page. Too many stuff have changed in 15 years. This should be routinely updated or at least point to a wiki maintained by the community.