SignalLoop — AI-Native Coding Assessments

SignalLoop is an AI-native coding assessment that evaluates how engineers solve, verify, and own real work with a constrained AI collaborator.

Comment

Inspiration

AI is now part of everyday software engineering. Developers use it to understand unfamiliar code, debug failures, explore alternatives, and generate tests. A coding assessment that bans AI feels increasingly unrealistic—but giving candidates an unrestricted coding agent can erase the very signal an assessment is meant to capture. We built SignalLoop to explore a third path: let candidates work with AI, but constrain the collaboration so employers can evaluate framing, verification, judgment, and ownership—not only the final code. SignalLoop is a collaboration between Sreenivas Makam and Ritesh Dhoot. It started as a proof of concept before Build Week and has evolved into a runnable hosted pilot and open-source reference implementation. It is not yet a production-grade hiring system.

What it does

SignalLoop is an AI-native candidate evaluator for software engineering hiring. Employers can create an assessment manually or provide a role and job description for guided role matching. SignalLoop maps those requirements to the closest supported assessment while clearly identifying tested, partially tested, and unsupported skills. Candidates receive a browser-based coding workspace with Monaco Editor, public tests, and a constrained AI collaborator. The assistant can explain candidate-visible code, clarify concepts, and help reason through candidate-identified problems—but it cannot enumerate every defect, generate the complete solution, reveal hidden tests, or write the final explanation. Throughout the assessment, SignalLoop captures code snapshots, test runs, candidate-created tests, AI interactions, decisions, and final explanations. The resulting Engineering Evidence Report combines technical correctness with FAVO: Frame: Did the candidate understand the problem and constraints? Ask: Did they use AI as a focused collaborator or delegate the work? Verify: Did they test assumptions and validate AI-assisted changes? Own: Can they explain and defend the final result? A bounded GPT-5.6 advisory can summarize evidence gaps and suggest interview focus areas. It is deliberately non-scoring and cannot change deterministic scores, integrity signals, or hiring recommendations.

How we built it

SignalLoop uses Next.js, React, TypeScript, and Monaco Editor for the candidate, employer, and administrator experiences. The backend is built with FastAPI, Python, PostgreSQL, SQLAlchemy, and Alembic. Candidate code runs through an execution-provider abstraction: Docker-based isolation locally, a direct execution backend for the hosted pilot, and an ECS/Fargate production-isolation path for operators who configure it. Clerk provides employer authentication, Supabase provides hosted PostgreSQL, and the web and API services run on Render. The constrained collaborator uses multiple safety layers: A deterministic pre-gate blocks obvious solution-generation requests. A policy classifier evaluates the intent of the conversation. A separate response component produces bounded assistance. An anti-decomposition policy prevents candidates from assembling a complete solution through many smaller requests. Candidate-facing AI never receives hidden tests, scoring internals, evaluator notes, or reference solutions. SignalLoop existed as a working proof of concept before Build Week. During the submission window, we used Codex and GPT-5.6 across feature implementation, security and release hardening, multi-agent review, hosted browser testing, documentation, and the complete demo-video lifecycle. Build Week work included the bounded GPT-5.6 advisory, product improvements, CI hardening, full-history security scanning, hosted validation, and preparation of the public release.

Challenges we ran into

The hardest challenge was making the AI collaborator useful without letting it become an autopilot. A policy that is too restrictive makes AI irrelevant; a permissive policy allows the candidate to delegate the assessment. Progressive disclosure and anti-decomposition became essential to maintaining that boundary. A second challenge was adaptation. Assessments should reflect the company, role, and job description, but generating a different scored test for every candidate would undermine comparability. Our current design adapts across roles while keeping the scored assessment consistent within a hiring cohort. A third challenge was separating deterministic evaluation from generative advice. We wanted GPT-5.6 to help employers interpret evidence without allowing it to modify scores or receive evaluator-only data. We solved this using an explicit evidence allowlist, schema validation, provider separation, and a fail-open, non-scoring advisory. Finally, turning a proof of concept into a credible public release required security scanning, history sanitization, CI coverage, hosted end-to-end testing, documentation, and careful communication about what is production-ready and what remains experimental.

Accomplishments that we're proud of

We built and validated a complete hosted workflow spanning employer setup, candidate invitation, constrained AI-assisted coding, public and hidden test execution, evidence capture, and employer reporting. We are particularly proud of: Treating AI collaboration as evidence instead of automatically treating AI use as misconduct. Enforcing the candidate/evaluator context boundary. Implementing FAVO—Frame, Ask, Verify, Own—as an evidence model around technical evaluation. Preserving deterministic scoring while adding a bounded GPT-5.6 employer advisory. Supporting guided role matching without pretending unsupported skills were evaluated. Completing automated API, worker, migration, frontend, and browser validation. Publishing SignalLoop as an Apache-2.0 open-source project after full-history secret scanning and release hardening. Using Codex for implementation, testing, security review, browser automation, multi-agent analysis, documentation, and video production while retaining human ownership of product and safety decisions.

What we learned

AI-native assessments should not focus only on detecting whether AI was used. The more useful question is how responsibility was divided between the engineer and the AI. We learned that the candidate’s process—how they frame a problem, ask for help, verify suggestions, and explain the result—can be as informative as the final code. We also learned that deterministic systems and generative AI work best when their responsibilities are explicit. Scores and recommendations remain deterministic; GPT-5.6 provides bounded interpretation over an allowlisted evidence payload. Codex significantly accelerated not only implementation but also the less visible work required to ship responsibly: adversarial review, security hardening, hosted browser testing, release validation, documentation, and media preparation.

What's next for SignalLoop

SignalLoop is currently a hosted pilot and open-source reference implementation, not a production-grade hiring platform. Our next directions include: Composing role-level assessments from an approved and calibrated question bank. Expanding beyond the current FastAPI assessment coverage. Strengthening adversarial testing of the constrained collaborator. Improving and validating FAVO evidence mappings. Studying fairness, accessibility, calibration, and resistance to superficial gaming. Completing production-grade isolated execution and operational hardening. Adding clearer employer review and evaluator-calibration workflows. We would especially welcome help from assessment designers, AI-safety researchers, hiring practitioners, and engineers interested in reliable evidence capture, constrained AI collaboration, and fairer AI-era technical evaluation.

Built With

Share this project:

Updates