EvidenceTrail

Inspiration

GitHub issues can be valuable but incomplete. Maintainers often need to decide whether a reported behavior is real, reproducible, and worth acting on—but AI summaries alone are not enough to trust. We built EvidenceTrail to make that investigation process inspectable. Instead of asking maintainers to accept an AI verdict, it gives them the trail behind it.

What it does

EvidenceTrail is a read-only workspace for investigating GitHub issues. It turns an issue into a bounded evidence trail: structured issue extraction, a focused test change, terminal output, Git diff, structured JUnit results, and a conservative classification. The dashboard makes the evidence easy to review through a triage queue, Evidence Brief, Evidence Results, artifact views, and timelines. A behavior gap is confirmed only when deterministic validation passes. EvidenceTrail does not decide whether that gap is a bug, feature request, documentation issue, or intended behavior—that decision remains with the maintainer.

How we built it

We built EvidenceTrail with Python, FastAPI, SQLAlchemy, SQLite, React, TypeScript, and Vite. Live investigations use fresh repository clones and short-lived Docker containers. Codex helped locate relevant code paths, propose focused executable test changes, run targeted tests, and retain the resulting artifacts. GPT-5.6 was used for structured issue extraction and evidence classification. The dashboard now presents the work as a modern reviewer workspace: a read-only triage queue, clear status indicators, repository filtering, Evidence Briefs, and results views that keep the proof close to the conclusion.

Challenges we ran into

The main challenge was preventing false certainty. A failed command might reflect missing dependencies, a broken environment, a flaky test, or a syntax problem—not the behavior described in the issue. We addressed that with structured JUnit evidence, focused-test checks, confirmation runs, reproducibility manifests, and conservative outcomes such as NEEDS_INFO and WONT_REPRO.

Accomplishments that we're proud of

We are proud that EvidenceTrail produces reviewable evidence rather than a black-box AI answer. We also created a no-key local judge demo. Reviewers can inspect persisted investigations and artifacts without GitHub access, OpenAI credentials, Docker, or rerunning a live investigation.

What we learned

We learned that trustworthy AI developer tools need boundaries as much as intelligence. Codex and GPT-5.6 were most valuable when paired with deterministic validation, preserved artifacts, and a clear separation between automated evidence gathering and human judgment.

What's next for EvidenceTrail

Next, we want to expand test-runner support, improve setup detection for more repositories, and make cross-investigation comparisons easier. We also want to explore opt-in maintainer workflows, such as draft follow-ups and review queues, while keeping every external action explicitly human-approved.

Built With

• alembic
• codex
• docker
• fastapi
• github
• gitpython
• gpt-5.6
• junit-xml
• npm
• openai
• pydantic
• pytest
• python
• react
• sqlalchemy
• sqlite
• typescript
• uvicorn
• vite
• vitest