OncoTwin Sentinel

OncoTwin evolved from a conventional pipeline into an inspectable agent workflow

Comment

Inspiration

Cancer researchers rarely make decisions from one clean signal. A nanoparticle may deliver a strong payload to a tumour but also accumulate excessively in the liver or kidneys. Evidence retrieval, candidate design, simulation, safety assessment, and approval are often handled through separate tools, making it difficult to understand how a recommendation was reached. We were also concerned about a second problem: generative AI can produce convincing recommendations that are easily mistaken for authorized decisions. OncoTwin Sentinel was inspired by the idea that AI should make complex research evidence easier to explore without silently taking control. Our guiding principle became:

Gemini interprets. Deterministic code calculates. Google ADK agents collaborate. Humans authorize.

All tumour, nanoparticle, and safety data used by OncoTwin Sentinel is synthetic. The application is a research demonstration and does not provide medical advice, diagnosis, or treatment.

What it does

OncoTwin Sentinel is a governed multimodal research twin for investigating a synthetic resistant tumour clone and evaluating nanoparticle delivery strategies. A user begins by asking: Investigate the resistant red clone and find a safer nanoparticle delivery strategy. Four specialized Google ADK agents then collaborate:

  1. Evidence Scout retrieves bounded evidence about the resistant R7 clone.
  2. Nano Designer creates three nanoparticle candidates inside a controlled design envelope.
  3. Twin Simulator compares tumour delivery and liver and kidney accumulation over 24 hours.
  4. Safety Steward applies a deterministic safety policy, quarantines unsafe candidates, recommends a preferred candidate for human review, and stops at the approval boundary. The user can interact with the same mission through several connected modalities: • Enter questions and commands as text. • Speak commands using Gemini 3.5 streaming transcription. • Interrupt spoken output by saying “Stop.” • Select the R7 clone or nanoparticle candidates directly in 3D. • Rotate, zoom, and inspect the research twin. • Drag the simulation timeline from T+00H to T+24H. • Ask contextual questions such as “Why was candidate B rejected?” • Navigate by voice with commands such as “Show candidate C” or “Move to hour 18.” • Upload a synthetic microscopy image for Gemini 3.5 vision analysis. • Compare new evidence with previous Firestore mission receipts. • Replay each agent’s work product and return the camera to the relevant evidence. • Attempt voice approval and see it rejected at the human-authority boundary. At T+18H, candidate B crosses the synthetic 45% liver-accumulation ceiling and visibly enters quarantine. Candidate C remains preferred for explicit human review. The 3D twin is not merely an animation. The selected object and simulation time become part of Gemini’s conversational context. If the user selects candidate B at T+18H and asks, “Why was this rejected?”, the response is grounded in that candidate, that time, the applicable policy, and the persistent evidence receipt.

How we built it

We built the frontend with React, TypeScript, React Three Fiber, and Vite. The interactive 3D theatre renders the resistant clone, three scientifically differentiated nanoparticle candidates, particle-delivery paths, organ-risk regions, camera choreography, quarantine states, agent artifacts, and the 0–24-hour simulation timeline. A FastAPI service acts as the governed mission gateway. The browser communicates with it through: • HTTPS for mission commands and evidence analysis • Server-Sent Events for agent progress • WebSockets for streaming microphone audio Google ADK 2.8 coordinates the four-agent graph workflow. Each agent has one bounded responsibility and access only to the tools it requires. Agents do not directly calculate simulation values or approve missions. The workflow uses Gemini 3.5 through Vertex AI for reasoning and multimodal interpretation. Gemini 3.5 streaming transcription processes voice input, while validated agent responses are rendered through Google Cloud Text-to-Speech using a Chirp 3 HD voice. The multimodal context is sent explicitly with every command: '''python { "mission_id": "nano-mission-id", "channel": "voice", "command": "Why was candidate B rejected?", "selected_candidate_id": "B", "selected_object": "candidate-B", "simulation_hour": 18, "image_evidence_id": null } ''' This prevents voice, images, 3D interaction, and timeline navigation from becoming disconnected features. Simulation and policy decisions remain deterministic. The Safety Steward calls a bounded policy tool that can return preferred and rejected candidates but cannot grant approval:

'''python def apply_nano_safety_policy(): results = run_comparison()

preferred = next(
    result for result in results
    if result.decision == "preferred"
)

rejected = [
    result.candidate.id
    for result in results
    if result.decision == "rejected"
]

return {
    "preferred_candidate_id": preferred.candidate.id,
    "rejected_candidate_ids": rejected,
    "policy_version": "nano-safety-v1",
    "human_approval_required": True,
    "approval_granted": False,
}

The approval endpoint separately validates the interaction channel: def validate_approval(channel, confirmation): if channel == "voice": raise ApprovalDenied( "Voice approval is disabled; use the visual approval control." )

if channel != "ui":
    raise ApprovalDenied(
        "Approval is accepted only through the visual boundary."
    )

if confirmation != "APPROVE SYNTHETIC RESEARCH MISSION":
    raise ApprovalDenied("Exact confirmation is required.")

''' Firestore stores missions, immutable evidence receipts, sanitized ADK traces, approval events, resume cursors, and image-evidence metadata. Uploaded image pixels are analyzed in memory and are not persisted. The web and API applications run as separate containers on Google Cloud Run. Cloud Build and Artifact Registry create and deploy versioned container images. The production service also exposes health, eligibility, ADK, memory, and voice proof endpoints so judges can verify the implementation.

Challenges we ran into

The shared context envelope solved this. Follow-up commands such as “Show candidate C” and “Why is it preferred?” continue in the same mission context. Persisting ADK traces across Cloud Run instances Our first ADK traces were stored in process memory. Cloud Run could execute a mission on one instance and route the trace request to another, making a completed mission appear to have no trace. We moved sanitized ADK trace state into Firestore. Missions and evidence can now survive reloads, new revisions, and instance changes. Making voice interruption reliable Our early voice implementation sounded scratchy, responded slowly, and did not always stop when the user said “Stop.” We separated the voice pipeline into three responsibilities: • Gemini 3.5 handles streaming speech transcription. • Governed mission services generate validated, receipt-grounded text. • Google Cloud Text-to-Speech renders the spoken response. We added automatic voice-activity detection, client-side audio cancellation, and explicit barge-in handling. “Stop” now cancels speech without being interpreted as approval or another research command. Maintaining conversational continuity after interruption Stopping speech was not enough. The next command also had to retain the selected candidate, simulation hour, mission ID, and image evidence.

Preventing early commands from reaching an unfinished mission A user could send a follow-up while the ADK mission was still queued, resulting in a 400 response. We introduced a visible GEMINI · VERIFIED readiness state and designed the judge flow to wait for completed agent artifacts before accepting contextual follow-ups. Balancing 3D rendering and real-time voice Continuous particle animation, high pixel density, speech streaming, and a detailed interface placed significant pressure on the browser. We added adaptive rendering quality, reduced-motion support, conservative scene modes, controlled particle complexity, and a graceful WebGL fallback. These were essential interaction improvements, not just graphical optimizations. Deploying Google Cloud Text-to-Speech An initially configured IAM role was not valid at the project-resource level, causing deployment to stop even though the container builds succeeded. We removed the invalid binding, used the Cloud Run runtime identity correctly, and added deployment-time verification for health and voice services. Making the system’s logic visible A visually impressive twin could still hide how the agents reached their conclusion. We added replayable agent work products, a decision chain, provenance indicators, persistent receipt hashes, modality traces, policy values, and explicit approval controls so users can inspect the reasoning process without exposing private chain-of-thought.

Accomplishments that we're proud of

We are proud that OncoTwin Sentinel is not a chat interface attached to a decorative 3D model. Every major modality affects the same governed mission. The production system demonstrates: • A real four-agent Google ADK workflow • Gemini 3.5 reasoning through Vertex AI • Gemini 3.5 streaming voice transcription • Gemini 3.5 synthetic image analysis • Context-aware text and voice follow-ups • Interruptible spoken responses • Voice-controlled candidate and timeline navigation • A receipt-driven 75-frame simulation • Interactive and selectable 3D evidence • Persistent Firestore mission memory • Replayable, inspectable agent work products • Cloud Run deployment with separate web and API services • Deterministic simulation and safety decisions • A server-enforced human-only approval boundary • No credentials exposed to the browser • No raw uploaded image pixels stored We are especially proud that governance is demonstrable. A judge can say “Approve the mission” and watch the system refuse. The safety claim is therefore testable rather than merely written in documentation.

What we learned

The biggest lesson was that multimodal UX is not created by placing a microphone, upload control, and 3D scene on the same page. The modalities must share state and contribute to one understandable outcome. We also learned that: • Models are strongest when interpreting context, not replacing deterministic calculation. • Agent specialization becomes valuable when every agent produces an inspectable artifact. • Tool restrictions are stronger than relying only on safety instructions. • 3D can become an explanation interface when selection changes conversational context. • Persistent receipts are more useful than an isolated “memory” feature. • Voice interruption must preserve conversational state. • Cloud-native agent systems cannot depend on process-local memory. • Safety boundaries should be enforced by both architecture and interface design. • Deployment scripts, IAM bindings, documentation, and demo timing are part of the product. • A trustworthy agent experience must clearly show where recommendation ends and authority begins.

What's next for OncoTwin Sentinel

Our next priority is production hardening: • User authentication and role-based access • Rate limiting and abuse protection • Distributed tracing and richer operational observability • Automated browser-based multimodal tests • Explicit retention policies for evidence metadata • Stronger session recovery and voice reconnection • Additional accessibility and low-power rendering modes We also want to extend the research experience with: • Additional synthetic tumour and nanoparticle missions • Side-by-side comparison of multiple mission histories • Richer uncertainty visualization • Researcher-authored policy envelopes • More advanced bounded rerun experiments • Collaborative review by multiple authorized humans • Exportable evidence packages for reproducible research discussions The governance principle will remain unchanged: Agents may retrieve, design, simulate, explain, and recommend. Only an authorized human may approve.

Built With

Share this project:

Updates