Inspiration
When your solution fails a judge's hidden test, you get a verdict and nothing else. You never see the input that broke it. Refute finds that input and shrinks it as small as possible.
What it does
Give Refute a problem statement and a solution that passes the samples but is secretly wrong. It finds the smallest input where the solution fails. For a greedy coin-change solution, that input is 3 10 / 1 5 6: the code prints 5, but the right answer is 2. Refute then writes a one-page report with a suggested fix.
How we built it
A language model writes four test tools from the statement: two independent reference solutions, an input validator and an input generator. Every tool is sandboxed and checked by running it. Refute then stress-tests your code, and a failure counts only when both references agree. The model never decides your code is wrong; running the code does. All code was written with Claude Code on Oct 10–11, and the model at runtime is gpt-oss-120b on Groq.
Challenges we ran into
Making the model's tools trustworthy was the hard part, so each one is verified by execution. Shrinking needed several rounds: early versions stopped at 3 32 / 1 16 30, and the final one reaches 3 10 / 1 5 6. Free-tier rate limits meant caching every model output.
Accomplishments that we're proud of
Refute finds the smallest failing input for a wrong greedy solution, and its suggested fix is verified on 200 fresh random tests. The test suite has 394 passing tests. The report is one HTML file styled like marked working in a squared school notebook.
What we learned
Treat model output as a proposal and check it by running it. "No failure found" is evidence, not proof, and the report says so.
What's next for Refute
A larger benchmark of real buggy solutions that reports its misses, C++ support, and automatic single-edit repair. Refute is for practice and learning, not for live contests.
Prior art and AI disclosure
CounterGen (Codeforces blog entry 145626) and CodeHacker (arXiv 2602.20213) explore similar ideas. Refute adds cross-checked references, a calibrated validator, validator-constrained shrinking and verified fixes. The plan was first drafted Oct 9, and the design brief was written with Claude in chat.
Log in or sign up for Devpost to join the conversation.