Today in Codeforces Round 244 (Div. 2), I found that there were too many test cases in problems C,D,E.
Problem C and E have more than 100 tests each, and problem D has about 170 tests.
As a result, the system testing phase become very slow.
Do you think that it is necessary to create such many tests? In my opinion, having more than 100 test cases doesn't make the test stronger.
What about you? Thank you.
Testing is slow, huh? So... you should buy a new computer, I think.
Hodor !!
Lannister's !!
It would be interesting to calculate number of failed solutions each test (i.e. test being the first failed)
My guess is that there would be ~5 tests out of 170 that failed at least one solution
A better statistic would be "tests that would leave a failing solution pass if removed", but that is a considerably harder statistic to make. (For instance, a lot of people got time limit exceeded on test 120, which probably means that solutions that TLEd earlier would also TLE there if earlier cases were removed).
But as I have too much time in my hands, here's the list (including current practice): 1-23,26,29,35,38-39,43,46,50,57-58,63,66,107,115,117,120,124,125,126,127,130,142,146,150
The likelihood of a solution being failed on a particular test can't be known before most of the solutions are ran against it. But there could a better solution to this problem through dynamic order management of test cases. The cases that are most probable to break a solution can be placed at first of the test case stack for each problem and it's probability changes once a solution fails. Starting order sequence could be setter-assigned or most successful ones from hacks. As more solution fails on a particular case, the priority of that cases changes and it gets pushed upward in the test queue. Since the system test pertains that one has to pass all the test-cases of a single problem. So I think, it doesn't matter if the solution fails on the first case(most probable one at first position of the queue). There could also be some cases with pre-defined priority from setter and tester that are placed prior to the start of the contest, it changes as system tests progresses. It wouldn't bring much of a change in the system testing procedure, but it may yield faster execution as probable cases are placed at first of the queue.
nice idea