ProofHire AI

Inspiration

Technical recruiters and hiring managers have no fast, reliable way to verify whether a candidate's GitHub code actually implements the architecture claimed on their resume. Meanwhile, pasting a resume and a repo name into standard AI chat tools often leads to hallucinated answers without real code evidence. I built ProofHire AI ("Proof, not promises") to verify resume claims directly against actual GitHub source files with strict 1-based line citations.

What it does

ProofHire AI is an evidence-driven GitHub repository and resume claim auditor built for technical recruiters, hiring managers, and engineering leads:

• Bounded, Line-Grounded Verification: Breaks a candidate's resume claim into distinct technical signals and scans the repository's top 8 prioritized files (README, dependency manifests, and up to 6 core source files) with safe line truncation.
• Anti-Hallucination Verdicts: Assigns every assertion a strict status (SUPPORTED, PARTIALLY_SUPPORTED, CONTRADICTED, or INSUFFICIENT_EVIDENCE) backed by exact file paths and 1-based line numbers. It never guesses on files outside the 8-file scan boundary.
• Interactive Technical Viva: Automatically converts unverified or partial claims into targeted interview questions at the bottom of the dashboard, allowing recruiters to paste a candidate's explanation and grade it against the actual codebase in real time.
• Explicit Failure Policy & Fallback: Supports zero-key local deterministic auditing as well as gemini-2.5-flash and gpt-4o-mini modes, with configurable failure policies (llm_cascade, llm_required, allow_heuristic) so rate limits or API outages transparently fall back to the local engine without crashing.

How I built it

• Backend: Python, FastAPI, Pydantic, and Uvicorn exposing structured REST endpoints (/audit, /grade-answer, /health, and interactive OpenAPI /docs).
• AI & Verification Engine: Hybrid local deterministic signal-matching engine paired with optional Google Gemini (gemini-2.5-flash) and OpenAI (gpt-4o-mini) reasoning, guarded by strict citation line-boundary validators.
• Frontend: Responsive dark-mode single-page interface built with HTML, Tailwind CSS, vanilla JavaScript, and Lucide Icons, keeping user API keys strictly in browser page memory.
• Quality & Testing: Backed by 112 automated unittest tests and a multi-scenario benchmark suite (benchmarks/).

Challenges I ran into

1. Preventing False Positives on Large Repositories: Scanning an entire 50,000-star repository like Python's requests library in real time would time out or exceed token budgets. Bounding the scanner to the top 8 prioritized files and explicitly marking uninspected claims as PARTIALLY_SUPPORTED rather than guessing solved both speed and accuracy.
2. Handling Cloud LLM Rate Limits Gracefully: Live LLM calls can hit quota limits or return out-of-bounds line citations during an audit. Building citation validators and a transparent heuristic fallback engine ensured the app stays reliable even when an external API fails.

What I learned

Following the structured plan-first workflow (scope.md -> prd.md -> spec.md -> checklist.md) before writing application code made a huge difference:

• Writing scope.md first forced me to define hard boundaries early—specifically the 8-file scan cap and keeping API keys in page memory—which prevented scope creep.
• Designing the data contracts and failure policies inside spec.md before coding helped me catch edge cases like malformed LLM citations and API timeouts early, making the final build much cleaner and easier to test.

Built With

• fastapi
• gemini
• github-api
• html5
• javascript
• openai
• pydantic
• python
• tailwindcss
• uvicorn