We will be undergoing planned maintenance on Oct 7th 6:00AM UTC / Oct 7th 2:00AM ET

Latch

Gemini evolves verified observers on five FPGAs for instruments producing up to 6.5 Tb/s. Blind tests matched three spacecraft; Latch also preserved a Fermi event NASA’s systems missed.

Comment

Inspiration

Telescopes, radio dishes, solar spectrographs, and SETI spectrometers produce data faster than observatories can preserve it. They therefore retain reduced products with fewer channels, integrated intensity, and coarser time resolution. Once a circular buffer is overwritten, phase, polarization, per-antenna voltages, microsecond pulse structure, and wideband context cannot be reconstructed from those summaries.

Quantitative trading systems address a similar timing constraint by placing FPGAs directly on the data path. We applied that architecture to scientific instruments, where an FPGA must identify useful structure and freeze the richer record before the buffer wraps.

What it does

Latch is an agentic harness for a population of five physical FPGA cards that receive the same ordered scientific stream. Gemini designs independent hardware observers by writing Verilog, adjusting thresholds and capture policies, spawning specialists, cloning successful parents, splitting hypotheses, merging redundant observers, and retiring ineffective branches.

Each FPGA runs its own observer RTL and maintains an independent circular buffer. When an observer detects useful structure, it timestamps the event and freezes the richer data before the buffer is overwritten. This preserves information such as phase, polarization, microsecond pulse structure, and wideband context that reduced scientific products cannot reconstruct.

The observations, misses, and counterexamples return to the next campaign. Gemini uses that evidence to revise the observer population until the set provides complementary coverage and survives reserved holdout evaluation. External source identities remain outside the optimization process until the final population and its evidence are sealed.

How we built it

Quantitative trading systems inspired the architecture because they place FPGAs directly on the data path when waiting for later analysis would lose the opportunity. We applied the same approach to telescope, radio, solar, SETI, and photon event streams.

An agent orchestrator assigns logical observers to five physical slots while preserving an exploration seat, a falsifier seat, and a known-good observer. Every observer has its own RTL, lineage, configuration, and capture policy. The loaded image configures the observed tile, threshold, drift mask, freeze window, and retained data product.

Every revision is content-addressed and linked to its parents. Simulation, oracle agreement, synthesis, timing, and holdout checks gate promotion. A failed revision returns evidence to Gemini but cannot replace the known-good image. A successful population is sealed with hashes covering its RTL, loaded images, event tuples, freeze policies, and retained frames.

Gemma 4 independently audits the sealed evidence without deployment authority. Cloud Run executes the bounded agent harness, Firestore stores campaign state and immutable receipts, and OpenTelemetry records redacted spans for model turns, tool calls, simulations, oracle checks, timing checks, and promotion decisions. Early development loops used OpenAI because free credits allowed us to test the campaign mechanics. Gemini is the submitted agent and the default provider.

Challenges we ran into

The first challenge was giving Gemini meaningful control without placing a language model in the real-time path. Gemini can redesign observers and manage the population between deployments, while the FPGA alone performs the deadline-sensitive detection and freeze operation.

The second challenge was convergence under a fixed hardware budget. Five cards cannot host every proposed specialist, so the agent orchestrator must preserve useful diversity, maintain exploration and falsification capacity, compare evidence across branches, and retire observers that no longer justify a physical slot.

The third challenge was separating source identity from physical evidence. A source name could bias the agent toward a memorized rule, so identity-bearing context was isolated from the agent-readable campaign. Horizons matching occurred only after the observer population and its evidence were sealed.

The final challenge was creating promotion gates that constrain deployment without making scientific decisions for the agent. The runtime can reject invalid RTL, failed simulations, oracle mismatches, timing failures, and unsafe slot changes, but it cannot choose the replacement observer or invent a better scientific threshold.

Accomplishments that we're proud of

Five physical F2 cards processed identity-blind voltages through the same hardware interface and source-independent observer logic. Empty GBT X-band input produced no hardware events, providing a negative control before the populated runs.

The most important result was that the sealed FPGA population produced recognizable hardware descriptions before receiving source identities. Each description combined observer RTL with its configuration, loaded bitstream, event tuple, freeze policy, and retained frames.

One observer produced GBT event (394, 27, 1) on tile 9394 with a measured drift of $-0.419,\mathrm{Hz,s^{-1}}$. Its retained dual-polarization voltages showed that 96.6% of the aligned-track power occupied one polarization, although the compact FPGA summary had already mixed that structure away. Nothing in the RTL, configuration, journal, or bitstream contained the name Voyager 1. Horizons associated the sealed physical description with Voyager 1 afterward.

Two other observers produced sealed ATA events (394, 19, 6) and (396, 20, 6). Their distinct configurations described separate physical tracks through the same identity-blind stream interface. Horizons later associated those sealed descriptions with ESCAPADE-Blue and ESCAPADE-Gold.

The population therefore converged on RTL and configurations that described the physical behavior of Voyager 1 and both ESCAPADE spacecraft before their names entered the system.

The same architecture preserved a $9.6,\mu\mathrm{s}$ Crab pulse on physical F2 whose shape cannot be reconstructed from a $256,\mu\mathrm{s}$ intensity bin. On OVRO data, another physical FPGA observer retained a 768-bin window behind a 64-channel trigger, preserving the context needed to distinguish quiet solar emission from radio-frequency interference.

The most unexpected result came from Fermi GBM continuous time-tagged event data on the trusted local ring. The observer retained a roughly $70,\mathrm{ms}$ event near $7,\mathrm{keV}$ with about eleven thousand excess photons across neighboring NaI detectors n4 and n5. The onboard trigger did not record it, and the ground miss-catcher published two other seconds from that day but not this one. The retained event reaches $196\sigma$ against quiet background under a 16 ms Li-Ma analysis, while the higher-energy magnetar band and the other detectors remain quiet. Latch preserved an event that neither NASA event path filed!!

What we learned

A scientific agent does not need to control the real-time instrument directly. It can operate at a slower timescale by designing hardware observers, evaluating their frozen evidence, and revising the population between deployments.

We also learned that observer diversity is useful scientific state. A population can preserve competing hypotheses and counterexamples that a single optimized trigger would discard. Convergence therefore means reaching a defensible set of complementary observers rather than selecting one universal detector.

Reduced summaries do not merely postpone analysis. They permanently remove possible analyses. Improving what the instrument preserves can therefore be more valuable than applying a more sophisticated model after the information has already disappeared.

The sealed population also became a machine-readable physical description of the stream. Source identities could be recovered from that description after sealing without being supplied to the agent during optimization

What's next for Latch

The next step is to deploy the Fermi TTE observer from the trusted local ring onto physical FPGA hardware and evaluate it through the same simulation, oracle, timing, and holdout gates used by the voltage observers.

We also want to extend the population across additional continuous scientific streams, increase the number of available FPGA slots, and run longer campaigns that measure how quickly observer populations specialize and converge under changing noise conditions.

Future work will add stronger population-level convergence metrics, broader independent holdouts, automatic rollback studies, and integrations with observatories that already maintain short raw-voltage buffers. The long-term goal is a general instrument layer where scientific agents continuously improve what hardware recognizes and preserves while the underlying evidence still exists

Built With

  • cloud-run
  • cloud-trace
  • firestore
  • fpga
  • gemini
  • gemma
  • google-adk
  • verilog
Share this project:

Updates

Submission history