Hello Codeforces community.
Tl;dr: The script is specifically for *Unix and can be used to stress test `C++` solutions. It can be easily modified to work with python or java. To use it in windows you can download Ubuntu terminal which is officially available on Microsoft Store and then use it or you can modify the script to work in cmd by changing the .out files to .exe and a bit of syntax for cmd.
Note: If you don't want to read this lengthy blog, just go to this repo and follow the instructions of readme.
First let me tell you what stress testing is if you don't know, Stress testing is technique using which we can run our solution (which we think might fail/is failing) on a number of random test cases and match it's output with the output of a solution which is a brute force solution or accepted solution of someone else.
If you don't know why and when to use stress testing you can use this detailed article on Stress Testing by ADJA.
I want to share my script for stress testing your solution.
Requirements:
1) A solution which we want to test.
2) A brute force solution which gives correct solution.
3) A generator for generating test cases according to the problem.
About script:
Go through the code once, i have added comments and you will understand most of the part. To understand the script all we need is just basics of the language bash. So we have three .cpp files:
1) gen.cpp // our cpp solution for generating test cases
2) solution.cpp // our orignial solution
3) brute.cpp // our solution which uses brute force and guaranteed to give correct output
About test case generator:
For generating test cases you can either use "testlib.h" here (i personally don't use it because it takes 6-7 seconds on average to compile in my pc(idk there might be ways to reduce it) and also i need to remember the methods to generate things.) or write your own since most of the time you just need arrays, strings, trees or simple graphs or simple some integers.
I use the below code for generating test cases most of the times(file gen.cpp):
You can use above code and modify it according to your needs. Just go through the code once and everything is self explanatory(i have added some comments too). Usage of above gen.cpp code:
1) to generate an array calling:
gen_array(10, -5, 10);
it will return an array(vector more specifically) with length 10 and elements in the range [-5, 10].
2) to generate a tree calling:
gen_tree(10):
will return a tree with 10 nodes.
3) to generate a simple graph calling:
gen_simple_graph(10, 12);
will return a simple connected graph with 10 nodes and 12 edges.
You can add things as you need them or use testlib which is the best if you know how to use it.
Usage:
Download this repository manually or by using git clone on terminal.
- Copy your original solution which you expect might fail in the file
solution.cpp. - Copy your brute force solution which is expected to give correct output in the file
brute.cpp. - Change the
gen.cppfile so as to generate test cases according to the question.
Now open your terminal in the directory where file s.sh resides and run command:
$ bash s.sh
or
$ ./s.sh
If it shows permission denied then give it execute permission using:
$ sudo chmod +x s.sh.
In file s.sh you can change the value of variable max_tests as you wish on how many random test you want to run the solution.
Verdict: the verdict of running file s.sh on every test case is either Accepted if your solution's output matches the brute solution output or Wrong Answer and will show the input on which the solution failed, the output of your solution and expected output according to the brute force solution on terminal and the script will be terminated.
If you wish to terminate the script at any moment you wish use the command ctrl + c in terminal.
quick demo: Here the output of first 3 test cases matched with the brite and as soon as it didn't, it printed the test case, output, and expected output to the console and terminated. Now you can debug your original solution and check where it goes wrong.
(
)
You can watch this video by Errichto to understand more about stress testing.
Any suggestions or corrections are welcomed.
Apologies for my poor English.
UPD1: Updated the script to take brute and main file via flags, no. of testcases as optional, added help/usage flag and updated readme for better readability & usage. Tested on Mac & Ubuntu, bash 5.x. Please refer updated README here for usage.
