Opportunity-ops

An autonomous multi-agent engine using Gemini 3.5 Flash to extract opportunity facts, build execution plans, and independently verify claims before taking action.

Comment

Inspiration

Agentic demos are good at looking capable and bad at being trustworthy. The pattern is always the same: a model reads a page, "extracts" the facts, and the pipeline treats that extraction as ground truth. Everything downstream; the plan, the deadline, the action inherits a hallucination with no way to notice.

We picked a task where that failure has a real cost: tracking opportunities. Hackathons, grants, CFPs, and applications all live on messy pages, all hinge on a deadline, and all punish you for getting that deadline wrong. If an agent is going to plan work and produce a submission on your behalf, the interesting question isn't "can it read the page?" It is "can it prove what it read, and will it stop when it can not?"

So we built the restraint first and the capability second. Opportunity Ops is a vertical slice whose differentiator is that the verifier, not the extractor, decides whether an action is allowed to happen.

What it does

Opportunity Ops turns an untrusted opportunity page into a structured, cited, fit-scored execution plan and a generated submission package, and refuses to do any of it when the source can't back up the claims.

source snapshot -> Scout/Extractor -> Planner -> Operator -> Verifier
  • Scout ingests a URL or local file through a bounded loader: scripts and styles stripped, 1 MB cap, fail-closed on fetch errors, and the normalized text hashed into an immutable sourceVersion.
  • Extractor turns prose into structured facts, title, deadline, timezone, eligibility, required technologies, deliverables, judging criteria. Two interchangeable implementations sit behind one hook: a deterministic regex extractor (no credentials) and Gemini 3.5 (Developer API or Vertex AI).
  • Planner scores fit against a user profile and lays out dated milestones that must all land before the deadline, reserving a day for submission and recovery.
  • Operator writes only run-local artifacts. Anything with an external side effect is gated behind an approval callback and denied by default.
  • Verifier re-derives the deadline from the untouched snapshot and compares it to the claim. It runs before the operator writes anything.

Every run emits a replayable event timeline. The happy path reads:

RUN_CREATED -> SOURCE_FETCHED -> RUN_EXTRACTED -> FACTS_EXTRACTED -> RUN_PLANNED
-> PLAN_CREATED -> RUN_EXECUTED -> ACTION_EXECUTED -> RUN_VERIFIED
-> VERIFICATION_PASSED -> SUBMISSION_PACKAGE_CREATED

The trust path is the one we actually lead the demo with. Seed a wrong deadline and the timeline ends differently:

... -> PLAN_CREATED -> RUN_FAILED -> VERIFICATION_FAILED -> ACTION_BLOCKED

ACTION_EXECUTED is absent. That absence is the product. The run reports the conflict, names the value the source actually contains, and no artifact is written.

There's also a local, credential-free judge UI (npm run serve) that ingests a live URL and shows source provenance, cited evidence, conflict details, the replayable timeline, and a downloadable submission package — with the package link withheld entirely on blocked runs.

How we built it

Core: TypeScript on Node, ~1,300 lines total, no agent framework. Zod schemas are the contract at every boundary. The whole slice runs offline with zero credentials, which is what made it testable.

Google Cloud path: Gemini 3.5 through both the Developer API (@google/genai) and a Vertex AI REST adapter, Cloud Run as the container target (/healthz, honors PORT), Firestore for the run aggregate, Cloud Storage for snapshots and artifacts, and interfaces for Pub/Sub work queues and a Google ADK runner. Credentials resolve through Application Default Credentials — workload identity on Cloud Run, gcloud auth application-default login locally. cloud-run.yaml carries no API key, secret reference, or access token.

Four design decisions did most of the work:

1. The model never asserts provenance. Gemini returns facts only. composeOpportunity stamps id, sourceUrl, citations, and sourceVersion from the snapshot afterward, and Zod strips any key outside the facts contract. A page that tries to redirect its own citations structurally cannot — the fields it would need to write don't exist in the model's output schema.

2. Verification is re-derivation, not re-asking. sourceDeadline in src/agent.ts pulls the deadline out of the raw snapshot with non-model logic and compares. A second model call would just be the same trust assumption wearing a hat.

3. Tools are typed by their consequences. Every tool declares readOnly, destructive, and externalSideEffect. The runner checks those flags against an approval callback before executing, so the policy lives in one place instead of scattered through call sites. Run status moves through an explicit transition table; verified and failed are terminal.

4. Safety is a build gate. npm run evaluate runs five adversarial case; prompt injection, malformed deadline, deadline conflict, approval denial, source refresh prints a JSON report plus a Markdown scorecard, and exits non-zero if any expected safety behavior regresses. npm run rehearse runs both judge paths and writes a timed pitch script. 35 tests, all deterministic, no network.

Challenges we ran into

The verifier's independence has a price, and we paid it. Once you accept that verification means re-deriving from the source without the model, your coverage collapses to what non-model logic can extract. A page saying "applications close November 30, 2026 at 5pm Pacific" in prose blocks with Source says no deadline — even when Gemini reads it perfectly. That felt like a bug for about a day. It isn't. Widening coverage means adding independent date patterns; it must never mean letting the verifier trust the extractor. We documented the limitation in the README rather than quietly loosening the check to make the demo look better.

Proving restraint is harder to demo than proving capability. A working pipeline is easy to show. "It correctly did nothing" is a non-event. We ended up building a deliberately-seeded wrong-deadline scenario (DEMO_SCENARIO=deadline-conflict) and a rehearsal script whose whole job is to point at an event that isn't in the timeline.

Determinism vs. real model calls. A judge scorecard has to be reproducible; a live demo has to actually call Gemini. We split them: demo and serve honor EXTRACTOR/MODEL_MODE, while evaluate and rehearse stay deterministic on purpose.

Silent model downgrade. A stale GEMINI_MODEL env var could quietly drop a deployed run to a non-compliant generation. Both providers now validate the generation at construction time and fail closed, and a test scans the committed configuration so a pre-3.5 literal can't get back into source control.

Credentials as a hard dependency. Vertex AI needs a billed project, which would have made the whole thing unrunnable for anyone reviewing it. We made every external dependency injectable; tokenProvider, fetchImpl, generateContent so tests never reach real credentials, and lazy-imported the auth and genai SDKs so offline runs never even load them.

Idempotency on retries. Snapshot and artifact paths are immutable (snapshots/<sourceVersion>.txt, artifacts/<runId>/<name>) and Cloud Storage writes use an ifGenerationMatch: 0 precondition, so a retried run can't overwrite the evidence a previous run was judged on.

Accomplishments that we're proud of

  • The trust demo works and it's the lead, not a footnote. We can hand a judge a run that produces a plan and a package, then seed a wrong deadline and show the verifier catching it before a single byte is written.
  • Safety properties are regression-tested, not asserted. 5/5 adversarial cases, 35 passing tests, non-zero exit on regression. If someone weakens the verifier, CI says so.
  • Prompt injection is handled structurally, not with a stern prompt. Source text is wrapped as untrusted data and the provenance fields it would want to influence are stamped outside the model's reach.
  • Zero secrets in the repo. Workload identity from the first commit; GOOGLE_ACCESS_TOKEN exists only as a local convenience override.
  • It's small enough to audit. ~1,300 lines, no framework indirection. You can read the entire trust boundary in one sitting which is the only reason claims about it are worth anything.
  • Credential-free from clone to demo. npm install && npm run demo works with no cloud account.

What we learned

"Verify" is a load-bearing word and most pipelines use it wrong. Asking a model to check its own extraction isn't verification, it's correlation. Real verification needs a second, independent derivation path and the moment you build one, you find out how much of your "extraction accuracy" was never confirmable in the first place.

Fail-closed shrinks your demo surface, and you should publish the number instead of hiding it. Our honest coverage is "JSON-LD endDate or an explicit deadline: line." Stating that plainly is more persuasive than a demo that appears to handle everything and silently trusts the model on the hard cases.

Provenance belongs to the code, never to the model. Citations, source URL, and content hash have to be stamped by something the model can't write to. Otherwise a hostile page can cite itself into legitimacy.

Interfaces before credentials. Writing the Firestore, Pub/Sub, and ADK boundaries as interfaces with real adapters behind them. kept the entire system runnable and testable offline. It also meant the cloud story was a swap rather than a rewrite.

An evaluation harness that exits non-zero changes the culture of a project. Once "the verifier blocks a conflicting deadline" is a failing test rather than a paragraph in a README, you stop being tempted to loosen it.

What's next for Opportunity Ops

Widen verification coverage the right way. Add more independent (non-model) date derivations: JSON-LD variants, microdata, <time datetime>, and timezone-explicit prose patterns. Every one of these makes the verifier stronger. None of them will be implemented by having the verifier trust the extractor.

Corroboration and confidence bands. Today verification is binary. Next is multi-source corroboration with a graded confidence signal, so "two independent derivations agree" is distinguishable from "one pattern matched."

Human-in-the-loop approval. The external-action tool is currently policy-denied in all paths. The next step is a real approval surface in the judge UI — show the exact payload, the cited evidence behind it, and require an explicit human grant before any outbound side effect.

Finish the cloud path. Swap the Firestore, Pub/Sub, and ADK interfaces for live implementations; split Cloud Run into API and worker; export the event timeline through OpenTelemetry to Cloud Trace so the audit trail is queryable in production.

Scheduled refresh and stale-plan detection. Cloud Scheduler re-ingests sources on a cadence; a changed sourceVersion marks the existing plan stale and triggers a re-plan. The domain model already carries the stale flag, it needs the scheduler behind it.

Ingestion adapters we deliberately deferred. PDF extraction and authenticated sources get provider-specific adapters rather than guesses. The browser ingestion adapter comes first, so the UI stops depending on the local fixture format.

Portfolio mode. One opportunity is a demo; the real product is many. Rank a set of live opportunities by fit score, budget them against the user's available hours, and surface the deadline calendar — with every entry still carrying its citation and its verification status.

Built With

Share this project:

Updates

Submission history