BioReason: Multimodal Lab Agent

A multimodal agent that visually audits lab setups and reasons through research protocols to predict experimental failures and safety risks using Gemini 3.

Comment

Inspiration

The scientific community faces a significant and costly challenge known as the Reproducibility Crisis. It is estimated that $28 billion is spent annually on preclinical research that cannot be replicated.

We identified three primary vectors for this failure:

  1. Ambiguity in Textual Protocols: Natural language descriptions of experiments often lack precise temporal or quantitative constraints.
  2. Human Error in Execution: Discrepancies between the intended protocol and the physical setup (e.g., wrong pipette usage, uncalibrated heating blocks).
  3. Silent Failures: Reagent incompatibilities that do not result in immediate failure but compromise downstream data.

BioReason was born from the realization that we need an AI agent that "understands" science not just as text, but as a physical process. We built this to utilize Gemini 3's ability to reason across modalities to detect these errors pro-actively, effectively solving the "28 Billion Dollar Problem" one experiment at a time.

What it does

BioReason is an autonomous safety agent designed to address the reproducibility crisis in synthetic biology and chemical engineering. By leveraging the advanced reasoning capabilities and multimodal context window of Gemini 3, BioReason functions as an active "in-silico" supervisor.

It ingests experimental protocols (PDFs) and real-time laboratory visual data (images/video) to perform zero-shot safety auditing, logical consistency checks, and reagent compatibility analysis before a single physical experiment is conducted.

The core process is modeled as a verification function

:

Where:

  • represents the experimental protocol (parsed from PDF/Text).
  • represents the observed image data of the lab setup.
  • The output is a set containing the Safety Score (

) and sets of Logical Errors (

) and Safety Hazards (

).

This project demonstrates a novel application of Neuro-Symbolic AI principles applied to the physical sciences.

How we built it

BioReason operates on a novel architecture we call the Semantic-Visual Alignment Engine (SVAE). Unlike standard RAG (Retrieval-Augmented Generation) pipelines, SVAE does not just retrieve information; it actively aligns the logical state of a protocol with the physical state of the lab bench.

1. Utilizing Gemini 3's Reasoning Capabilities

We utilize Gemini 3's Chain-of-Thought (CoT) capabilities to decompose complex biological procedures into atomic logical units.

For example, a standard PCR protocol step:

"Add 50µL of Taq Polymerase to the master mix."

Gemini 3 decomposes this into a logical constraint set:

  • Agent: Researcher
  • Action: Pipette Transfer

  • Volume:

  • Target: Master Mix (Temperature Sensitive: True)

If the visual input

shows the Master Mix sitting at room temperature (

) rather than on ice (

), the system triggers a Thermal Degradation Warning. This reasoning requires the model to possess internal knowledge of enzymatic stability, which Gemini 3 retrieves and applies zero-shot.

2. System Implementation

The Multimodal Ingestion Layer (Frontend): Built with React and Tailwind, this serves as a high-fidelity scanning interface. Users upload Protocol Documents (PDF/DOCX) and Visual Context (High-resolution images of reagents, equipment, and bench setup).

The Reasoning Core (Backend): The backend (Python/FastAPI) orchestrates the interaction with the Gemini 3 API using a multi-turn agentic workflow:

  • Extraction Phase: The model parses the PDF to extract a structured JSON representation of the experiment's "Happy Path."
  • Visual Audit Phase: The model analyzes the uploaded image to identify equipment and reagents (OCR for labels, object detection for glassware).
  • Adversarial Simulation Phase: This is the critical innovation. We prompt Gemini 3 to act as an "Adversarial Reviewer," specifically looking for edge cases where

and

conflict.

# Pseudo-code for Adversarial Simulation Prompt
prompt = f"""
ACT AS: Senior Principal Investigator.
INPUT: Protocol Step 3 ("Heat to 95C") AND Image_B (Glass Beaker).
TASK: Simulate the physical interaction.
CONSTRAINT: Glass beakers can shatter at high thermal gradients.
REASONING: Does Image_B satisfy the safety requirements for Step 3?
OUTPUT: Boolean verification + Chain of Thought explanation.
"""

Challenges we ran into

1. Hallucination in Scientific Data

Challenge: Early iterations of the model would sometimes "invent" chemical reactions that do not exist. Solution: We implemented a "Citation-Based Verification" guardrail. We force Gemini 3 to cite why a reaction is dangerous (e.g., "Referencing standard exothermic properties of Acid + Water mixing"). If it cannot ground the reasoning in general scientific principles, the confidence score is lowered.

2. Latency in Visual Processing

Challenge: Analyzing high-res lab images took too long for a real-time assistant. Solution: We leveraged Gemini 3's optimized multimodal latency. By preprocessing images into localized crops (focusing only on labels and equipment), we reduced the token load while maintaining high accuracy, achieving a response time of <2 seconds.

Accomplishments that we're proud of

To validate BioReason, we tested the system against a common failure mode in CRISPR-Cas9 editing: Buffer Incompatibility.

The Scenario: A user uploads a protocol requiring a "High-Salt Buffer" for DNA extraction but uploads an image showing a bottle of "Low-Salt Wash Buffer." To the human eye, the bottles look nearly identical (clear liquid, similar labels).

BioReason's Analysis:

  1. OCR Extraction: Detected text "Low-Salt" on the bottle in the image.
  2. Protocol Cross-reference: Matched against step requiring "High-Salt."
  3. Chemical Reasoning: Gemini 3 inferred that using Low-Salt buffer at this stage will cause the DNA to elute prematurely, ruining the yield.

Result: The system flagged a Critical Error with 98% confidence. This successfully demonstrated the "Silent Killer" detection capability we set out to build.

What we learned

We learned that the key to applying LLMs in science isn't just "knowledge retrieval," but Adversarial Simulation.

By prompting the model to actively look for why an experiment might fail (rather than just summarizing how it should work), we unlocked a much deeper layer of utility. We also validated that Citation-Based Verification is essential for any AI tool operating in high-stakes environments like a wet lab; without grounding reasoning in scientific principles, the "trust" factor is impossible to achieve.

What's next for BioReason: Multimodal Lab Agent

BioReason is the first step toward a fully autonomous "self-driving" laboratory.

  • Immediate Impact: We aim to deploy this to university labs to reduce the 50%+ failure rate in junior researcher experiments.
  • Long-term Vision: Integration with robotic liquid handlers. Instead of a human reading the alert, BioReason could send a STOP command directly to a robotic arm if it visually detects a safety violation.

We believe that by building AI that understands the physical constraints of science, we can accelerate the rate of discovery and make biotechnology safer and more accessible.

Built With

Share this project:

Updates