CounterGen: Auto Counter-Example Generator for Debugging
Tired of getting "Wrong Answer" on gym problems with no clue why? Or being handed a giant unreadable testcase that doesn’t help at all?
Introducing CounterGen — an LLM-assisted tool that automatically generates minimal, human-readable counter-examples for your buggy programs.
Uses LLM APIs (Gemini, Claude, or OpenAI)
Each generated code is automatically tested to ensure it functions correctly before being used.
Once verified, these codes are combined into a stress-testing pipeline that runs your buggy solution against many testcases.
Produces minimal failing testcases automatically.
Workflow at a glance:
Try It Out!
The full source code is available in Github.
Just follow the instructions in the README to get started!
Important Notes
Beta disclaimer: CounterGen is still in development. Sometimes the counter-examples may not be successfully generated (especially in harder problems). We’re actively improving it, and your feedback would be valuable.
Contest rules: Please DO NOT use CounterGen during official contests. As per this Codeforces blog post , using AI assistance during contests may violate the rules. CounterGen is meant for practice, learning, and debugging outside of contests.
Any feedback or suggestions are very welcome!
Looks intriguing! This reminds me of a paid platform that also stress-tested code and provided incorrect cases for debugging (obviously the code was never made public). I recall the creator mentioning that they manually built a generator for each problem.
Your workflow, where you're using AI for everything, is interesting, but I think there are some limitations to consider. For example, the AI could generate a solution that passes the sample tests but is ultimately incorrect. This would lead to a wrong base solution. While there's an option to paste in your own solution code, this still feels somewhat overlooked.
Another point is the barrier to entry due to the need for users to create their own API keys. I understand why this is necessary, but imagine a centralized system where AI is used once to generate a stress tester for each problem (e.g., generating samples), storing this in a database, and then reusing it. This would mean each problem only requires one API call, as opposed to the current system where each call is made locally.
One added benefit of a centralized system would be that you (as the creator) would have access to a vast collection of stress testers for each problem. This could also allow for a 'correctness verification' system, where you manually set or approve a test generator (perhaps pulling from the editorial or a copy-paste solution). This would also solve the issue introduced by the AI sometimes being inaccurate.
Long story short, I think it’s best to only involve AI where it's absolutely necessary (specifically for problem statement parsing -> test case generation). This would reduce the probability of mistakes occuring in the workflow and make the system lighter for users. They wouldn’t need an API key for problems that have already been stress-tested.
Anyway, maybe I’m overlooking some things as well, considering the feasibility of implementing all of this, but just some thoughts!
Thanks a lot for your thoughtful and detailed feedback — I really appreciate you taking the time to share your perspective!
You’re absolutely right that AC code generation is the weakest link if the user doesn’t provide their own correct solution. That’s kind of inevitable, and I definitely encourage users to paste in their own AC code whenever possible. What I’ve tried to mitigate this is asking the LLM to generate naive (brute-force) solutions, prioritizing correctness over efficiency. It’s also encouraged to select stronger models for this part (e.g., Gemini 2.5-pro over 2.5-flash).
Regarding the idea of a centralized system or shared database — that’s a great point, but honestly I don’t plan to host or maintain a server for this project. As a student (and with no profit motive here), it would be overwhelming to run and moderate such infrastructure. My current goal is just to provide a generally convenient tool that works locally for most problems across most sites.
What I can take away from your advice though are some practical improvements on the UX side:
Allowing users to preview the generated codes (so they can sanity-check what the AI produced).
Allowing users to provide their own validator/checker, just like the AC code, to further reduce the risk of incorrect generations.
And I agree with your last point — users should not blindly trust the output. I’d recommend always manually checking that the generated counter-example actually proves the AC code is correct and their own code is wrong.
Once again, thanks for the feedback!