SupportTrace Evidence Gate

A keyless RAG support console that attaches source evidence, scores citation confidence, and routes risky or weak drafts to human review.

Comment

Inspiration

What it does

How we built it

Challenges we ran into

Accomplishments that we're proud of

What we learned

What's next for SupportTrace Evidence Gate

Inspiration

Support drafts can sound confident even when retrieval is weak. Operators need more than fluent prose: they need to see the exact evidence, understand why a draft is being released or held, and retain control over sensitive cases.

What it does

SupportTrace retrieves approved knowledge, composes a deterministic draft, attaches the exact source excerpts, and calculates a transparent citation-relevance heuristic. It then emits one explicit decision: ready_to_send, human_review_required, or blocked. Low-confidence and policy-sensitive drafts enter a human review path; prompt injection is blocked before release.

How we built it

The public judge path is a keyless TypeScript application with a Fastify API, React/Vite console, deterministic local embeddings, an in-process vector store, Vitest coverage, Docker files, and CI. The Evidence Gate is exposed through POST /api/support/draft and in one judge-visible dashboard.

Codex with GPT-5.6 was used during Build Week to audit the pre-event baseline, design and implement the evidence decision state machine and API contract, add the dashboard and regression tests, run privacy and build verification, and prepare reproducible documentation. The /feedback Session ID is 019f734e-0c95-78a1-85c7-227b9d9a4ba9.

Challenges

The hardest part was making uncertainty useful without misrepresenting it. The 0.40 score threshold is deliberately documented as a deterministic demo policy, not a calibrated probability. The routing logic also keeps evidence quality separate from policy risk: a well-supported refund request can still require human approval.

Accomplishments

  • Exact source excerpts on every supported draft.
  • Transparent evidence confidence and reasons.
  • Three inspectable outcomes: release, review, or block.
  • Automatic review routing for weak and policy-sensitive drafts.
  • Prompt-injection blocking.
  • Eight passing tests and a keyless local judge path.

What we learned

Human review works best as an explicit product state, not a vague fallback. Showing the evidence and the reason for a hold makes the same workflow easier to trust, test, and tune.

What's next

  • Calibrate the threshold with labeled support traffic and error costs.
  • Add SQL-backed evidence and review persistence.
  • Add role-based approval policy and signed decision receipts.
  • Evaluate citation precision and false-release rates on a public benchmark.

Created by

I selected the product and publication boundaries, required a public-safe keyless demo, and reviewed the final behavior. Codex with GPT-5.6 performed the Build Week repository audit, detailed design, implementation, testing, and documentation. GPT-5.6 was the development partner through Codex; the runtime judge demo itself is deterministic and does not claim an online GPT-5.6 call.

Built With

Share this project:

Updates