About Fallacy Fencer

Inspiration

Most people never get to practice arguing until it actually matters: a job negotiation, a policy debate, a heated family dinner. By then there is no coach in the room. Logical fallacies are usually taught as a static, lecture-hall concept, a list you memorize for a test and never actually feel yourself committing. We wanted to flip that around and turn fallacy detection into something you experience in the moment, mid-sentence, while you are still trying to win the argument. That tension, being sure you are right but not sure if that last line was actually a Straw Man, felt like a fun and replayable idea for a hackathon build.

What it does

Fallacy Fencer turns any topic into a live duel against an AI opponent who argues the other side. Every turn, the app does two things:

1. Generates the opponent's in-character rebuttal, at one of three difficulty profiles: Novice, Sharp, or Ruthless.
2. Analyzes your argument for fallacies, quoting the exact phrase that triggered each one, and scores its strength from 1 to 10.

The header keeps a running fallacy tally across the whole debate

How we built it

• Built with Next.js 16 (App Router), TypeScript, and Tailwind v4, with the entire debate living in React state. No database, since a 48 hour hackathon demo does not need persistence.
• A provider agnostic LLM layer (src/lib/llm/) exposes a single chat() and structured() interface, with two interchangeable backends: Anthropic (Claude) for the submission build, and a local Ollama model for free, offline iteration during development. Neither API route knows or cares which one is answering.
• Structured output is forced through tool calls. Instead of parsing free text for fallacy names and scores, both the analysis and report endpoints force the model through a submit_analysis or submit_report tool call, so the response is a typed object every time rather than something we have to regex out of prose.
• Two calls happen per turn, fired in parallel with Promise.all from a single API route. The opponent's rebuttal and the fallacy and strength analysis run concurrently, so the wait per turn is the max of the two call times instead of the sum.
• Difficulty is written as a persona, not a parameter. Novice, Sharp, and Ruthless are not scalar knobs on temperature. Each is a distinct system prompt describing how polished, aggressive, and gap exploiting the opponent should be, which produced far more distinct feeling opponents than tuning sampling parameters ever did.

Challenges we faced

• Small local models do not always follow the rules. Running Ollama during development meant the 8B model would occasionally skip the forced tool call entirely and reply in plain text, or return object and array fields as JSON encoded strings instead of native JSON. We handled this with a one shot retry (a blunter instruction to respond only by calling the function) and a coercion pass that tries to parse any string field that looks like an object or array before giving up.
• Keeping the opponent honest. An LLM arguing the other side with real conviction has an incentive to reach for fabricated statistics or fake citations to sound more persuasive. The system prompt had to state plainly that it could reference general, plausible knowledge but never invent specific, verified sounding facts. Otherwise the debate stopped feeling like a fair duel.
• Scoring that does not feel random. Early on, strength scores swung a lot turn to turn for arguments of similar quality. We anchored the rubric directly in the prompt, stating that a 10 should be reserved for excellent arguments and most decent arguments should land in the 5 to 8 range, so the meter felt earned rather than arbitrary.
• Swapping LLM backends with zero code changes. Designing chat() and structured() as the only surface the rest of the app touches meant the Anthropic and Ollama swap had to be handled through provider detection at the edge, based on environment variables, instead of scattered conditional checks throughout the route handlers. That extra bit of upfront design saved us from a rewrite before the demo.

What we learned

Forcing structured output through tool calls is far more reliable than prompting a model to return JSON and hoping for the best, but it shifts the failure mode from malformed output to no tool call at all, especially on smaller local models. That meant designing for a graceful retry rather than assuming the happy path. We also learned that persona driven system prompts, like a Ruthless trial lawyer versus a Novice debater, produce much more legible, demo friendly differences than turning numeric model parameters ever would.

Built With

• api
• app-router
• css
• javascript
• llama-3.1
• next.js
• node.js
• npm
• ollama
• react
• rest
• tailwind-css
• typescript