Plasmatic's blog

Why does codeforces take so long to return non-AC verdicts?
By Plasmatic, history, 5 years ago, In English

It's come to a point where if I see that my solution is 'stuck' on a case, I automatically assume that it's incorrect somehow.

Does anyone know why this happens?

  • Vote: I like it
  • +435
  • Vote: I do not like it

»
5 years ago, # |
← Rev. 3   Vote: I like it +50 Vote: I do not like it

Running on test x .

A few moments passes .

Wrong answer on test x .

Its been common in last few days .

→ Reply
  • »
    »
    5 years ago, # ^ |
      Vote: I like it -30 Vote: I do not like it

    Running on test x.

    A few moments passes.

    Wrong answer on test x + y.

    → Reply
»
5 years ago, # |
  Vote: I like it +22 Vote: I do not like it

This might be the case for solutions near the Time Limit which are executed multiple times. Not sure what is the used strategy, but I guess is something among the lines, if execution time of the submission is more than time limit, but less than time limit + some threshold execute again (do it 1 or 2 times). Solution will be considered TLE if all executions were out of time.

→ Reply
  • »
    »
    5 years ago, # ^ |
      Vote: I like it +133 Vote: I do not like it

    It doesn't happen only with TLE submissions. I observed similar things with WA submissions which are not close to TL. Testcase number that is executed is updated every ~3s but then it got stuck on test e.g. 12 and after 10s it shows WA on test 14.

    → Reply
    • »
      »
      »
      5 years ago, # ^ |
        Vote: I like it +87 Vote: I do not like it

      Also, recently I've seen a few times that after "Running on test N" I briefly see "Running on test N+1", which then swiftly changes to "WA on test N".

      → Reply
  • »
    »
    5 years ago, # ^ |
    ← Rev. 2   Vote: I like it +39 Vote: I do not like it

    It is a possible reason, but I think maybe the main problem is that the judge divides the test cases into different (processes or threads), in case of failure of any test, killing all (processes or threads) for higher numbered tests takes time

    → Reply
    • »
      »
      »
      5 years ago, # ^ |
        Vote: I like it +18 Vote: I do not like it

      In that case CF could show submission's verdict before killing all processes or threads.

      → Reply
  • »
    »
    5 years ago, # ^ |
      Vote: I like it 0 Vote: I do not like it

    I also find that the result of submissions always appears faster in the problem page of contests.

    → Reply
»
5 years ago, # |
  Vote: I like it +26 Vote: I do not like it

On the contrary, I notice that if you submit the same code for the same problem, but under two different problem IDs (like div2C and div1A), the submission result is very fast, as if they cached my submission or something.

→ Reply
  • »
    »
    5 years ago, # ^ |
      Vote: I like it +39 Vote: I do not like it

    They definitely cache submissions, I suspect they're caching binaries. If you submit the same file with only comments that differ you'll always get the same time.

    → Reply
»
5 years ago, # |
  Vote: I like it +69 Vote: I do not like it

I always thought this had to do with the diagnostics running extra stuff to figure out what happened when the code was wrong, though I have no clue if this is actually the case.

→ Reply
»
5 years ago, # |
← Rev. 2   Vote: I like it +63 Vote: I do not like it

because after non-AC verdicts, if you use C++ then CF run some diagnostics on your code to catch TLE, RTE, and errors. I don't know if there are other diagnostic tools for other languages.

→ Reply
»
5 years ago, # |
  Vote: I like it +2 Vote: I do not like it

I always thought that this is a way to stop participants from submitting too frequently, like when you know you didn't AC, sometimes you submit once again with some modified code, now you have to wait for 1 min more.

→ Reply
»
5 years ago, # |
  Vote: I like it +13 Vote: I do not like it

The comments about diagnostics are correct, you can read more about it here https://codeforces.me/blog/entry/55902

→ Reply
  • »
    »
    5 years ago, # ^ |
    ← Rev. 2   Vote: I like it +26 Vote: I do not like it

    Diagnostics don't change submission's verdict. CF could show submission's verdict, only then start the diagnostics(also show something like "diagnostics are running...").

    Aren't the diagnostics run after submission's first failed test?

    → Reply
»
5 years ago, # |
  Vote: I like it +17 Vote: I do not like it

The comments about diagnostics are obviously correct, though sometimes I have observed that after submission, when seeing the status, the page may say 'Running on test x' but upon refreshing you see 'Running on test y' instead (y>x). So sometimes the text on the status page is updated with a delay of few seconds, if you do not manually refresh the page.

→ Reply
»
5 years ago, # |
  Vote: I like it 0 Vote: I do not like it

I just handed over the task, I go to the status tab and there I am no longer there.WTF?

→ Reply