ClaimCheck

Inspiration

Students paste claim-heavy essays and posts into ChatGPT and get one vague answer with weak or invented citations. For Reverie Hacks’ ML Prompt Engineering track, we wanted a real multi-step workflow that separates extraction, retrieval, and verification—and can prove it beats a single prompt.

What it does

ClaimCheck fact-checks pasted text by:

1. Extracting atomic claims
2. Retrieving evidence with Firecrawl
3. Verifying each claim with a Featherless LLM (supported / unsupported / unclear + confidence + citations)
4. Synthesizing a report
5. Optionally rewriting unsupported parts It also includes A/B evals against a one-shot “fact-check this and cite sources” baseline.

How we built it

• Python pipeline with Pydantic schemas between nodes
• Featherless (OpenAI-compatible) for extract / verify / rewrite
• Firecrawl for web evidence
• Streamlit demo UI
• Fixture set + baseline/score scripts for samples
• Docs + workflow flowchart for the track submission

Challenges we ran into

• Making retrieval degrade safely when sources fail (never crash; mark unclear)
• Getting structured JSON reliably from LLMs
• Showing a fair A/B comparison vs free-form single-prompt output
• Keeping the MVP shippable for a student hackathon timeline

Accomplishments that we're proud of

• End-to-end live pipeline with real citations (not mock-only)
• Clear claim cards judges can scan quickly
• Measurable lift in structured citation coverage vs the baseline
• Submission-ready artifacts: workflow PNG, samples, documentation, public GitHub

What we learned

Structured multi-node prompting + retrieval grounding is more reliable than one mega-prompt. Conservative verification (prefer unclear over false “supported”) matters for student trust and safety.

What's next for ClaimCheck

Classroom/API mode, source caching, stronger eval scoring with human labels, and a browser extension for drafts.

Built With

• fact-checking
• featherless
• firecrawl
• llm
• openai
• prompt-engineering
• pydantic
• python
• streamlit