I_love_tigersugar's blog

too many test cases on Codeforces Round
By I_love_tigersugar, 11 years ago, In English

Today in Codeforces Round 244 (Div. 2), I found that there were too many test cases in problems C,D,E.

Problem C and E have more than 100 tests each, and problem D has about 170 tests.

As a result, the system testing phase become very slow.

Do you think that it is necessary to create such many tests? In my opinion, having more than 100 test cases doesn't make the test stronger.

What about you? Thank you.

  • Vote: I like it
  • +38
  • Vote: I do not like it

»
11 years ago, # |
  Vote: I like it -29 Vote: I do not like it

Testing is slow, huh? So... you should buy a new computer, I think.

→ Reply
»
11 years ago, # |
  Vote: I like it +39 Vote: I do not like it

It would be interesting to calculate number of failed solutions each test (i.e. test being the first failed)

My guess is that there would be ~5 tests out of 170 that failed at least one solution

→ Reply
  • »
    »
    11 years ago, # ^ |
      Vote: I like it +60 Vote: I do not like it

    A better statistic would be "tests that would leave a failing solution pass if removed", but that is a considerably harder statistic to make. (For instance, a lot of people got time limit exceeded on test 120, which probably means that solutions that TLEd earlier would also TLE there if earlier cases were removed).

    But as I have too much time in my hands, here's the list (including current practice): 1-23,26,29,35,38-39,43,46,50,57-58,63,66,107,115,117,120,124,125,126,127,130,142,146,150

    → Reply
    • »
      »
      »
      11 years ago, # ^ |
        Vote: I like it +22 Vote: I do not like it

      The likelihood of a solution being failed on a particular test can't be known before most of the solutions are ran against it. But there could a better solution to this problem through dynamic order management of test cases. The cases that are most probable to break a solution can be placed at first of the test case stack for each problem and it's probability changes once a solution fails. Starting order sequence could be setter-assigned or most successful ones from hacks. As more solution fails on a particular case, the priority of that cases changes and it gets pushed upward in the test queue. Since the system test pertains that one has to pass all the test-cases of a single problem. So I think, it doesn't matter if the solution fails on the first case(most probable one at first position of the queue). There could also be some cases with pre-defined priority from setter and tester that are placed prior to the start of the contest, it changes as system tests progresses. It wouldn't bring much of a change in the system testing procedure, but it may yield faster execution as probable cases are placed at first of the queue.

      → Reply