TestSenseAI

Inspiration

Every pull request means someone has to stop and write tests — fast, without missing edge cases, in a style that actually matches how the rest of the codebase tests things. Generic AI test generators get the "fast" part right and fail at the rest: they produce plausible-looking tests that ignore the repo's own conventions, because they have no memory of what came before.

The hackathon's theme — agentic memory that's "not a toy, not an afterthought" — was the missing piece. We didn't want an AI that generates tests in a vacuum. We wanted an agent that actually remembers a repository: every past PR, every merged test, retrievable and useful the next time a developer opens a new PR. CockroachDB's distributed vector indexing, running as the same durable store as the rest of the app's data, was exactly the kind of "memory that never goes down" the challenge asked for.

What it does

TestSense AI connects to a GitHub repository and learns from it. It ingests the repo's history of merged PRs, embeds them with Amazon Titan V2, and stores them in CockroachDB behind a real C-SPANN vector index. When a new pull request opens, the agent:

Retrieves the diff and changed files via the GitHub API.
Runs a cosine-similarity vector search against the repo's own historical tests to find the closest few-shot examples.
Hands those examples plus the PR diff to Claude Sonnet 4.6 on Amazon Bedrock, which generates 2–5 targeted unit tests.
Surfaces the generated tests in the UI with a "why this test" rationale, so a developer can accept, edit, or reject each one.
Feeds that accept/reject signal back into analytics — acceptance rate, retrieval hit rate, and average similarity score are all computed live from the same CockroachDB tables the pipeline writes to.

The longer a repo is connected, the more PR history the memory has absorbed, and the sharper retrieval gets — the whole point of agentic memory doing real work instead of a one-shot demo.

How we built it

Frontend: React 19 + Tailwind, a five-step "Connect Repository" flow, a Project Workspace with PR/Memory/Analytics tabs.
Backend: AWS Lambda behind API Gateway — one Lambda for the auth/CRUD/analytics surface, and a separate set of Lambdas (webhook handler, PR retrieval, similarity search, test generation) orchestrated by AWS Step Functions for the actual pipeline.
Memory layer: CockroachDB Cloud (Serverless, AWS us-west-2). test_embeddings.embedding VECTOR(1024) holds Titan V2 embeddings, backed by a C-SPANN vector index — CREATE VECTOR INDEX ... ON test_embeddings (project_id, embedding vector_cosine_ops) — partitioned per project and opclass-matched to the cosine operator the queries actually use. We verified via EXPLAIN that CockroachDB's planner picks a real vector search scan, not a full table scan, before and after seeding 200 vectors.
AI: Amazon Bedrock — Titan V2 for embeddings, Claude Sonnet 4.6 for generation, both called through a shared retry/backoff client.
Auth & webhooks: a GitHub App (not a classic OAuth App) — sign-in uses the standard user-to-server OAuth flow, but PR webhooks are configured once at the App level, so any repo the App is installed on starts sending events automatically. No per-repo webhook bookkeeping, no scope guessing.
Infra as code: AWS SAM — API Gateway, six Lambdas, a Step Functions state machine, and an S3 + CloudFront static frontend, all in one template.

Challenges we ran into

Every one of these was a real bug we hit taking the project from "code that reads fine" to "actually running against live AWS" — none of them showed up until we stopped trusting the code and started exercising it for real.

Two backends that had never met. The auth/CRUD system and the AI pipeline Lambdas were built independently and never wired to the same HTTP surface — the original infra template had no API Gateway at all, so even a "successful" deploy would have produced Lambdas nobody could reach.
A vector column isn't a vector index. The schema had a VECTOR(1024) column and a similarity query, but no actual index — just a full scan ordered by cosine distance. We built a real C-SPANN index and confirmed via EXPLAIN that it's actually used, including catching that the default opclass accelerates L2 distance, not the cosine operator our queries use.
A CloudFormation circular dependency between the API Gateway and the Lambda that referenced its own domain in an environment variable — classic chicken-and-egg, fixed by deferring that value to a second deploy pass.
Wrong AWS region on the first deploy attempt — everything else (the database, Bedrock, the secrets) lived in us-west-2; the first deploy went to us-east-1 and silently couldn't find any of it.
A Step Functions state machine that had never once deployed. The template used the raw CloudFormation resource type instead of the SAM one, so DefinitionUri — the whole point of pointing at an ASL file instead of inlining it — was rejected outright. One layer deeper, the ASL itself was invalid too: a Catch block tried to inject a fallback value directly, which Step Functions doesn't allow; fixed by routing failures through a dedicated Pass state instead.
A Secrets Manager ARN we constructed by hand was simply wrong. AWS appends a random suffix to every secret's real ARN — the template's !Sub-built ARN didn't have it, so every webhook lookup 404'd against a secret that, from AWS's point of view, didn't exist. Swapping to the plain secret name (which the API accepts just as well) fixed it.
IAM scoped for reading, not writing. The Lambda that stores a user's GitHub token on sign-in only had GetSecretValue — every real OAuth callback failed silently until PutSecretValue/CreateSecret/DeleteSecret were added.
API Gateway's own CORS config went stale and stayed stale. Redeploying with a new allowed origin updated the resource definition (verified directly via the API) but never triggered a new deployment to the live stage — CloudFormation doesn't always regenerate that hash on a parameter-only change. The fix wasn't a deploy trick; it was removing API Gateway's auto-generated CORS handling entirely and letting the Lambda's own (already-correct) logic own it.
A session cookie that "worked" and didn't. SameSite=Lax is fine when frontend and backend share a domain — ours don't (CloudFront vs. API Gateway are genuinely cross-site). Sign-in would complete, the token would get stored, and the app would still show signed-out, because the cookie never rode along on the next fetch. SameSite=None; Secure fixed it.
Two subsystems, two different ideas of where a token lives. The Lambda that fetches a PR's diff guessed a secret path (projects/{id}/oauth-token) that nothing ever wrote to — the real token lived under github-oauth-token/{userId}, set at sign-in. Same bug shape as the API Gateway split: code that looked plausible but had never been checked against what the other half of the system actually does.
GitHub's Mermaid renderer is strict. A diagram that rendered fine elsewhere silently fell back to raw text on GitHub because of <br/> tags and styling directives it doesn't support — no error, just a wall of code where a diagram should be.
Choosing GitHub App over OAuth App under deadline pressure — the "correct" architecture (App-level webhooks, no per-repo permission guessing) meant migrating auth mid-hackathon. We scoped it down to just what mattered: keep the existing sign-in flow, add App installation only for webhook delivery — avoiding a full rewrite of token handling.

Accomplishments that we're proud of

Every claim about the AI pipeline is backed by a real run against real infrastructure — real Titan V2 embeddings, real Claude Sonnet 4.6 generations, real CockroachDB reads and writes — not mocked, not stubbed.
Took the project from "never successfully deployed once" to a working AWS stack: API Gateway, six Lambdas, Step Functions, and a CloudFront-served frontend, built from a template that had fundamental infrastructure gaps.

What we learned

CockroachDB's vector index opclass has to match the distance operator your queries actually use (vector_cosine_ops vs the default L2) — easy to get a "working" index that never gets picked by the query planner.
A GitHub App's installation-scoped webhook is a meaningfully better integration pattern than manually registering per-repo OAuth webhooks — fewer permission edge cases, no secret-matching bugs, and a UX every developer already recognizes.
Infrastructure claims need to be verified live, not trusted from reading the code — several "should work" pieces (the analytics endpoint, the Step Functions resource type, the API routing) turned out to be silently broken until actually exercised end to end.
Calling a Lambda's handler function directly in a local script proves the logic works — it proves nothing about IAM, Secrets Manager, CORS, or cookies, because none of those layers exist outside a real deployment. We learned this the hard way after an early "everything works end-to-end" turned out to only mean the business logic was sound; the actual deployed path needed nine more real bugs fixed before a PR opened in a browser could make it all the way through.

What's next for TestSense AI

Scheduled (not just on-connect) memory refresh, so retrieval quality keeps improving on a repo even between PRs.
Multi-repo and org-level installs of the GitHub App.
Feeding accept/reject feedback back into future generation prompts, closing the loop between what a team actually keeps and what the agent learns to produce next time.