RaceHunter — Autonomous Race-Condition Investigator

• 

Inspiration

Some bugs crash a system in full view. A race condition can let the system answer two people with confidence and break one of those promises later.

One unit remains in inventory. Two customers try to buy it at nearly the same time. Both requests read the available quantity before either update completes. Both customers receive a successful response.

Every operation looked valid on its own. The final result was still wrong.

By the time an engineer investigates, the requests are over. The application appears healthy. Unit tests pass. The logs may show two ordinary transactions without revealing the small timing window that allowed both to succeed.

The customer still experienced the failure.

As a software engineer, I came to RaceHunter with a specific discomfort. Rare concurrency failures can leave a team responsible for something it cannot reproduce, explain, or confidently repair. A support ticket says two people bought the last item. The code suggests that should be impossible. The test suite agrees with the code.

The engineer is left trying to prove an event that has already disappeared.

A proposed fix may look correct in code review and pass every sequential test, but the team still needs to know whether the original collision can survive it. Without a reliable reproduction, “we think this closes the race” may be the most honest answer available.

That uncertainty matters when software is responsible for someone’s order, payment, reservation, credit, or data.

Traditional testing tools answer only part of the problem. Unit tests check individual operations. Integration tests check expected workflows. Load tests measure throughput and latency. They do not necessarily ask whether a business promise remains true when valid operations collide at exactly the wrong moment.

I wanted a tool that could ask that question directly.

I wanted to give it a rule written in the language of the business:

• Successful orders must never exceed available inventory.
• A coupon may be redeemed only once.
• Two users must never hold the same reservation.
• One logical command must not create two payments.
• A resource must move only through allowed states.

The tool would create controlled collisions, observe what happened, and keep searching when the first experiment found nothing. When it found a failure, it would have to reproduce it and preserve the evidence. A developer should be able to replay the same experiment after changing the code and see whether the failure remained.

That idea became RaceHunter.

Using an AI agent for the search raised the standard. A model can suggest a clever experiment and adapt its strategy, but a plausible explanation cannot establish that a race condition occurred. Every finding needs evidence produced and evaluated by deterministic code.

RaceHunter gives Gemini the exploratory work of deciding where to look next. Application code controls synchronization, traffic limits, invariant evaluation, finding promotion, minimization, and replay.

The included demonstration uses one item and two buyers, but the problem reaches much further. The same failure pattern appears in payments, appointments, ticketing, coupons, distributed jobs, and stateful workflows. Two individually reasonable actions collide, and the system makes a promise it cannot keep.

RaceHunter exists to catch that collision before a customer has to report it. It also gives the engineer repairing the bug a replayable test that can demonstrate whether the same failure survives the fix.

What it does

RaceHunter begins with a business rule expressed in plain language.

In the demonstration, the developer asks it to test:

Successful orders must not exceed available inventory.

Gemini 3.5 Flash converts that objective into a schema-validated experiment plan. The plan defines the actors, operations, observations, invariant, execution strategy, and safety budget.

The developer reviews and approves the plan once. RaceHunter then runs the investigation asynchronously:

1. It resets the authorized reference target to a known state with one unit of inventory.
2. It launches controlled concurrent buyers using bounded schedules.
3. It captures each actor’s request, response, timing, correlation identifier, and observed state.
4. A deterministic evaluator decides whether the inventory rule passed, failed, or produced inconclusive evidence.
5. Gemini reviews sanitized observations and chooses the next bounded strategy.
6. RaceHunter repeats a suspected violation three out of three times before promoting it to a verified finding.
7. It minimizes the failure to the smallest useful reproduction.
8. It creates an immutable replay artifact with a stable fingerprint.
9. It replays that exact artifact against both the vulnerable and corrected implementations.

The final finding contains the complete chain of evidence:

• the business rule that was tested;
• the approved experiment plan;
• each strategy selected by the agent;
• the schedule seed and actor steps;
• an ordered causal timeline;
• the observed responses and inventory state;
• the deterministic invariant result;
• the three reproduction attempts;
• the minimized two-actor failure;
• the replay artifact and fingerprint; and
• the vulnerable-Fail versus fixed-Pass comparison.

Gemini can decide which allowed experiment to try next. It cannot mark its own work as correct. Only the deterministic evaluator can promote a finding.

That boundary lets RaceHunter use AI for adaptive investigation while keeping every factual claim inspectable and replayable.

How we built it

RaceHunter is a modular .NET 10 and React application deployed on Google Cloud.

The React and TypeScript interface follows the investigation from beginning to end. It includes the New Hunt form, generated-plan review, human approval, live campaign progress, verified finding, causal timeline, minimized reproduction, and Verify Fix experience.

An ASP.NET Core API validates every request, applies authorization and safety limits, persists commands, and exposes durable progress to the interface. Long-running experiments never execute inside an HTTP request.

A separate .NET worker performs the investigation. It consumes authenticated Google Cloud Pub/Sub messages, acquires a durable lease, resumes the campaign from persisted state, and runs the plan-execute-observe-adapt loop.

PostgreSQL is the system of record locally, with Cloud SQL for PostgreSQL in staging. It stores experiment plans, approvals, campaign state, run attempts, model decisions, trace events, findings, reproduction attempts, and replay artifacts.

Cloud Run hosts three deployable services:

• the public RaceHunter API;
• the private background worker; and
• the private reference target containing vulnerable and corrected inventory modes.

I integrated Gemini 3.5 Flash through the Google GenAI SDK and Vertex AI. All model exchanges use versioned schemas. Gemini can generate an experiment plan and select from an allowlisted set of strategies, but its output passes through independent validation before anything executes.

The deterministic concurrency engine uses bounded channels, barriers, seeded schedules, cancellation tokens, and separate global and per-target concurrency limits. It records trace events in a stable order without allowing an unbounded number of tasks or requests.

The reference target contains a real overselling defect. In vulnerable mode, two buyers can read the same inventory value and both complete successfully. In corrected mode, transactional protection allows only one buyer to succeed.

The replay system binds the scenario version, invariant, actor steps, schedule, target snapshot, and sanitized request data into an immutable artifact. RaceHunter calculates a stable fingerprint from that content and validates it again before replay.

Terraform provisions the Google Cloud environment, including Cloud Run, Cloud SQL, Pub/Sub, Secret Manager, IAM, budgets, logging, and tracing. Deployments use immutable image references and approval gates so an unexpected infrastructure change cannot silently continue.

The security model limits what an autonomous concurrency tester may touch. Targets must be explicitly authorized and allowlisted. RaceHunter applies SSRF defenses, DNS validation, redirect denial, request budgets, concurrency limits, secret references, and trace redaction. The agent cannot turn a business-rule experiment into arbitrary traffic against a public service.

Challenges we ran into

The first challenge was preventing a race-condition investigator from producing ambiguous state through its own recovery behavior.

Google Cloud Pub/Sub provides at-least-once delivery. A worker may receive the same message again after a timeout or restart. A process can stop after sending a request but before saving the result. A lease can expire while an earlier worker is still alive. A model call can succeed while the process crashes before recording that it consumed part of the budget.

Each boundary created a different duplication risk.

I had to make inbox processing, leases, checkpoints, model-call reservations, run attempts, finding promotion, and replay claims idempotent. Stale workers must be unable to overwrite progress created by the current lease owner. Recovery must continue the same logical campaign without creating a second run, finding, or model decision.

This work is mostly invisible in the demonstration, but it determines whether the evidence can be trusted after a restart.

Reproducibility created another challenge. A schedule seed does not control how an external server, database, or operating system will order every request.

Claiming perfect schedule determinism would have been misleading.

RaceHunter instead preserves the complete experiment identity and measures the observed reproduction rate. A suspected violation must succeed in three consecutive reproduction attempts before it becomes verified. Results can remain failed or inconclusive when the evidence does not meet that standard.

“We saw it once” and “we can reproduce it” are different claims.

Gemini also needed enough context and choice to adapt the investigation without receiving unrestricted control over execution.

I defined a bounded action vocabulary. The model can select actor counts, timing patterns, and schedule strategies that the application has already approved. It cannot invent an arbitrary endpoint, exceed the campaign budget, bypass target authorization, redefine the invariant, or declare a finding.

Preserving identity through minimization and replay was equally difficult. A minimized reproduction is useful only if it still tests the same rule against the same scenario and target assumptions. The replay artifact therefore carries its complete identity, and RaceHunter validates its fingerprint before and after replay.

Google Cloud staging introduced problems that local validation could not reproduce. Cloud Run cold starts affected demo timing. Private services required exact OIDC audiences. Secret Manager version ordering mattered during deployment. Cloud SQL configuration, Terraform state migration, Pub/Sub authentication, and private-service routing each introduced failure modes outside the application code.

I built a resumable release process that stops safely, inspects existing state, and requires fresh approval before continuing after a material change.

The final challenge was explaining the complete system in less than four minutes.

RaceHunter contains an asynchronous agent, concurrency engine, deterministic evaluator, recovery system, immutable replay format, cloud architecture, and security boundary. Every screen had to earn its place.

The final demonstration focuses on one promise, one collision, one verified finding, one minimized reproduction, and one replay against the fix.

Accomplishments that we're proud of

I am proud that RaceHunter continues beyond generating a plan.

From one rule and one human approval, it can run a background campaign, adapt its strategy, discover an oversell, reproduce it three out of three times, reduce it to two actors, create an immutable artifact, and replay that artifact against vulnerable and corrected implementations.

The complete comparison matters most. The same artifact fails against the vulnerable target and passes against the corrected target. The developer can inspect the causal evidence behind both outcomes.

I am also proud of the boundary between AI judgment and factual evidence. Gemini helps RaceHunter decide what experiment to try next, but every verified finding comes from deterministic invariant evaluation and trace-linked observations.

The product carries operational safeguards that are easy to omit from a hackathon demo:

• durable restart recovery;
• duplicate-message protection;
• stale-lease rejection;
• model and request budgets;
• global and per-target concurrency limits;
• explicit target authorization;
• SSRF and redirect protection;
• secret redaction;
• exact-audience service identity;
• least-privilege IAM;
• immutable deployments; and
• hundreds of automated checks across domain logic, concurrency, recovery, architecture, integration, security, and browser behavior.

I wanted RaceHunter’s claims to remain defensible after the demonstration ended. The persisted evidence, deterministic evaluation, and replay artifact make that possible.

What we learned

Trustworthy autonomy needs boundaries.

RaceHunter has scoped tools, explicit budgets, versioned schemas, durable state, observable decisions, and a safe way to stop. Gemini can make meaningful choices inside those limits without controlling the evidence or the safety boundary.

AI and deterministic software also have different strengths.

Gemini is good at translating a human objective into possible experiments. It can review observations and decide that a different actor count, timing pattern, or schedule strategy may expose the failure.

Deterministic code is better at synchronization, request accounting, invariant evaluation, security enforcement, finding promotion, and replay validation.

Giving each side a clear responsibility produced a system I could reason about and test.

I also learned that reproducibility means preserving identity. Repeating an HTTP request is not enough.

A useful reproduction needs the same scenario version, invariant, actor steps, schedule, target assumptions, and setup data. RaceHunter binds those elements into a fingerprinted artifact so a later replay can prove which failure it is testing.

An inconclusive result is also necessary. A trustworthy tool must be willing to say that it does not have enough evidence.

An external target may schedule requests differently. A timeout may obscure the final state. A violation may appear once and fail to reproduce. RaceHunter preserves those distinctions instead of turning every suspicious observation into a finding.

The project also reinforced why concurrency bugs matter.

They are usually described using transactions, locks, isolation levels, idempotency keys, and compare-and-swap operations. The consequence is easier to understand: the system told two people yes when it could honor only one answer.

Software correctness protects the promises a system makes to the people using it.

What's next for RaceHunter

The next step is to make RaceHunter useful beyond the included inventory demonstration.

OpenAPI import would allow developers to bring in an authorized API contract and map operations without defining each endpoint manually. Additional invariant families could cover duplicate payments, conflicting reservations, coupon reuse, exactly-once processing, invalid state transitions, and stale-write overwrites.

I also want to expand the schedule-strategy library. Different failures require different collision patterns, including simultaneous starts, staggered reads and writes, repeated idempotency attempts, delayed confirmations, and controlled retry storms.

Verified findings should become executable regression tests. A team could export the minimized reproduction, run it in CI, and prevent the same race from returning months after the original fix.

Pull-request integration could attach the evidence directly to a change:

• the business rule that failed;
• the minimized actor schedule;
• the vulnerable replay result;
• the corrected replay result; and
• the artifact fingerprint used for both.

Longer term, RaceHunter could run bounded campaigns continuously against authorized staging environments. Teams could maintain a library of business promises and let the agent search for new violations as their systems change.

The immediate goal is to let a developer take a concurrency fix into review with the original failure preserved as a replayable test, then keep that test running in CI.

Built With

• .net
• asp.net-core
• c#
• cloud-logging
• cloud-run
• cloud-sql
• cloud-trace
• docker
• entity-framework-core
• gemini-3.5-flash
• google-cloud
• google-genai-sdk
• opentelemetry
• playwright
• postgresql
• pub/sub
• react
• restapi
• secret-manager
• terraform
• testcontainers
• typescript
• vertex-ai
• vite
• xunit