Codex Contributor

Codex Contributor

💡 Inspiration

Every AI coding tool takes a GitHub issue and executes it blindly. Devin, Sweep, Copilot Workspace — issue in, PR out, no questions asked. But careful engineers don't work that way. They investigate before they write. They read the codebase, verify the issue's claims, and only then implement. We asked: What if Codex worked like a careful engineer — and produced proof of its investigation before writing a single line?

⚙️ What It Does

Codex Contributor forces Codex to investigate before it writes. Every GitHub issue gets an evidence-cited Engineering Review before any code is generated. Three named agents coordinate the entire workflow:

• Investigation Agent: Reads the issue and repository to produce a seven-section Engineering Review. Every evidence bullet points directly to a specific file path and line range.
• Implementation Agent: Writes scoped code changes only after a confidence gate verifies the investigation (halting automation if confidence falls below 0.50 to request human review).
• Validation Agent: Runs tests with up to five self-repair iterations.

The pull request handoff opens a real PR with the Engineering Review embedded as its first section — ensuring maintainers see the reasoning, not just a diff.

Measured Results:

• 5 real issues investigated across openai-python, langchain, flask, and requests.
• 18 evidence citations generated, 100% verified at the exact cited path and line range.
• Top Finding: Identified a security-relevant, cross-provider bug in openai/openai-python#3516 (confirmed at 94% confidence) where redirect-following in _base_client.py:L835-L839 combines with Azure's custom api-key header injection in lib/azure.py:L382-L386.

🛠️ How We Built It

Codex served as both the build environment AND the runtime engine in a full recursive loop.

1. The Infrastructure: Codex built every stage, including deterministic repository intake (git, AST mapping, test runners), the GPT-5.6 Sol reasoning contract (with strict schema validation and mandatory file-level citations), the confidence gate, the pipeline, the GitHub workflow, and a 4-tab Streamlit dashboard.
2. The Execution: Codex itself performed the Investigation Agent role for the demo sweep, running GPT-5.6 Sol against five real repositories to produce schema-validated reviews.
3. The API Path: The OpenAI Responses API path is fully implemented and environment-gated for judges to run using their own keys.

🛑 Challenges We Ran Into

• Trustworthy Citations: Making citations reliable was our core battle. The schema validator rejects any evidence claim that doesn't resolve to a real file path—invented paths fail loudly.
• API Credit Constraints: Keeping the demo honest under zero-API-credit constraints forced a better architecture. Codex desktop became the execution layer directly, proving the workflow doesn't depend on a single integration path.
• GitHub 2FA Lockout: A lockout an hour before the deadline proved why our deterministic trust layer matters—the stored-token git workflow kept shipping code when the humans couldn't log in.

🎉 Accomplishments That We're Proud Of

• 18/18 evidence citations verified at their exact cited path and line range with zero hallucinated references.
• A real security finding in openai-python confirmed with forensic file:line evidence.
• A strict confidence gate that demonstrably refuses to act on weak or incomplete evidence.
• 18 passing tests, a premium Engineering Review document view, and a complete recursive story: Codex built the tool that makes Codex investigate.

🧠 What We Learned

• Evidence > Code: The most valuable output of an engineering agent isn't code—it's the moment it produces evidence a human can verify.
• Trust Through Constraints: Requiring every claim to cite a file and line transformed the system from a "plausible assistant" to a "tool an engineer can trust."
• Guardrails Win: Constraining the agent (one confidence gate, citation-or-refuse, five-iteration repair cap) made it more useful, not less.

🚀 What's Next for Codex Contributor

• Webhook Integration: A GitHub webhook mode to trigger investigations automatically on every new issue.
• PR-Check Annotations: Displaying Engineering Reviews directly as annotations inside PR code checks.
• Continuous Benchmarking: Expanding the benchmark across more repositories and publishing verification rates continuously.
• Maintainer-Side Queue: A dedicated queue where every incoming AI-generated PR must arrive with a verified Engineering Review—making evidence-before-code the global default for AI contributions.

Built With

• codex
• git
• github-api
• gpt-5.6
• openai
• python
• streamlit