I'm the coordinator of WorldQuant Round 1 (Codeforces Round 1128, Div. 1 + Div. 2). After the contest ended, I am disappointed to see some people complaining about the weak samples in the comments of the announcement. I agree that the samples in some problems (A, C, D1) are weak. But this was an intentional decision during the problem preparation process, and I am here to defend this decision. This post will contain an overview of what I believe samples should be used for, what good samples (not necessarily strong samples) look like from both a participant and problemsetting point of view.

What are samples for?

The most important function of samples and notes are for making sure the participant understands the problem statement. If samples and notes do not clarify confusion a participant may have about the problem after reading the statement carefully, I would say that's a bad set of samples.

The second function of samples, especially for interactive and constructive problems, are for showing the output/interaction format. This is especially important in problems where the format may be more complicated than usual, for example 2261F1 - Plegma II: Ask Me Any Cell (Easy Version). Getting Wrong Answer on test 1 does not count as extra penalty on Codeforces, so if someone otherwise has the correct logic but the wrong output format, the samples would be able to catch their mistake and save them from penalty or debugging.

What are samples not for?

While samples are of course helpful for all participants to either motivate the correct solution, or verifying their logic on small test cases as a sanity check, I believe that samples should not be there to help participants "guess" the correct solution by itself.

For example, consider 1339A - Filling Diamonds, where (spoilers) the solution is to simply print the input. The examples are weak, with only $$$n=1$$$ and $$$n=2$$$ provided. Even with just these two, I think some people may already have a good guess of what the solution is, and start implementing. Imagine if the authors also included $$$n=3,4,5,6$$$ in the samples. Do you think a single person will try to solve the problem as intended? It would be a problem wasted.

For this reason, I also don't think samples should be strong on problems to artifically decrease the difficulty of the problem. Similarly, I don't think samples should be weak on problems to artificially increase the difficulty of the problem.

When should the sample be strong?

When I say a set of samples are "strong", I mean that it's less likely that someone who submits code that passes samples will fail on a hidden test case. This usually means including a large number of samples, including edge cases, difficult cases, etc.

I believe that samples should be strong when it's difficult to make meaningful progress by purely looking at the samples rather than trying the problem conventionally. For example, I believe in strong samples for hard problems (because most hard problems have some logic that is counterintuitive or hard to guess). I also believe in strong samples for counting problems, because it's easy to generate a strong, large case where the answer is large or even exceed modulo, so it serves as a great way to catch many implementation mistakes (e.g. forgetting to take modulo somewhere), while not being very hinting by itself.

I also believe in having samples catch implementation errors: if someone has the correct logic, but incorrect implementation, I think it's usually better to create samples that can catch these mistakes. For edge cases, in an ideal world, samples should not help participants find what exactly are the edge cases, but if someone finds an edge case and handles it incorrectly the samples should catch it. This is of course not always possible.

Regarding round 1128

Again, like I said at the beginning of this blog, we intentionally made samples weak on some problems. I believe they are consistent with what I outlined in this blog.

In problem A, we did not include the case where the first player starts with only two numbers (and the numbers aren't adjacent). This is the only edge case where the answer is not the range of numbers at the start. I believe that if you did not find this edge case, you did not truly solve the problem, and I find it difficult to imagine someone finding this edge case and implementing it wrong. So, I think omitting it from the samples is correct.

In problem C, there are only two cases where the second player wins. We included one of them in the sample. My guess is lots of people didn't find the second case, and got Wrong Answer on the second test case. But again, if we included both cases, in my opinion, the problem loses a lot of its problem solving value.

In the future, I hope people stop blaming problem authors for their own skill issue, to put it bluntly. I get it, everyone hates submitting their solution and getting Wrong Answer on test 2. But if you missed an edge case because it wasn't included in samples, maybe it's because you didn't truly understand the problem as much as problem setters hope.