Sentinel Observatory

Sentinel + Gemini 3

Inspiration

Modern sky surveys observe the night sky continuously and produce enormous numbers of brightness anomalies every night. Most of them are harmless noise, sensor artifacts, or ordinary variable stars, but a few represent rare astrophysical events such as supernovae.
The difficult problem is not detecting light — it is deciding when evidence is convincing enough to react.

Human astronomers often monitor candidates over time before requesting expensive follow-up observations. I wanted to explore whether an AI system could take on that role: not just identifying patterns in a single image, but supervising an observation process and making decisions under uncertainty.

This led to Sentinel — an autonomous observing agent that watches the sky, gathers evidence across multiple observations, and decides when a transient event becomes scientifically credible.

What it does

Sentinel Observatory simulates a telescope monitoring a star field and autonomously manages the observation process.

The system:

observes a simulated sky at regular time intervals
detects candidate brightness anomalies
tracks their behavior across multiple observations
evaluates persistence, position stability, and brightness evolution
decides whether to continue observing or trigger an alert

Instead of reacting to a single image, Sentinel reasons over time.
It waits, gathers evidence, and only escalates when repeated observations support a real transient event.

In the demo, one hidden transient is injected into the simulated universe, and Sentinel independently monitors, verifies, and confirms it.

How we built it

The project combines a physics-inspired simulation with an agentic reasoning loop using Gemini.

Sky Simulation

A synthetic star field is generated with atmospheric noise, optical blur (PSF), and detector noise. A transient source is injected but not revealed to the agent.

Image Processing Pipeline

Each observation produces a reference image, current image, and difference image. A detection module extracts measurable candidate sources and their brightness.

Agentic Decision Loop (OODA Loop)

Sentinel follows a recurring Observe–Orient–Decide–Act cycle:

Observe: receive measurements from the telescope
Orient: analyze candidate history and environment
Decide: determine next action
Act: observe again or trigger alert

Gemini evaluates structured observations and maintains candidate history across iterations.

Telemetry & Visualization

The system tracks confidence levels and brightness curves, allowing the agent to base decisions on evolving evidence instead of single-frame classification.

Challenges we ran into

The biggest challenge was separating perception from reasoning.

Initially, the agent analyzed raw difference images and produced unstable results because astronomical data is extremely noisy. We discovered that real observatories do not reason directly from images — they reason from measurements extracted from images.

We had to redesign the architecture so:

detection produces candidate measurements
the agent reasons only on measured evidence
no detections means continued observation

We also encountered coordinate alignment issues, duplicate detections of the same source, and false positives caused by subtraction artifacts. Stabilizing the decision logic required careful handling of uncertainty rather than simply tightening thresholds.

Accomplishments that we're proud of

We successfully built a fully autonomous observation supervisor that:

monitors a telescope across simulated time
maintains memory of candidate history
delays action until evidence accumulates
triggers alerts only after confirmation

The system does not simply detect a bright pixel — it demonstrates cautious scientific behavior.
In testing, Sentinel observed multiple iterations, rejected uncertain signals, and confirmed the injected transient with 100% detection accuracy in the final run.

The most satisfying moment was seeing the agent wait instead of reacting immediately. That hesitation was the actual intelligence.

What we learned

We learned that intelligence in scientific systems is not about classification accuracy — it is about decision timing.

Astronomical data is inherently uncertain. A useful system must manage incomplete evidence, not just recognize patterns. By forcing the agent to reason over time, the behavior became more stable and realistic.

We also learned how important system boundaries are: perception systems detect signals, but reasoning systems decide actions. Mixing those roles led to instability; separating them produced reliable behavior.

What's next for Sentinel Observatory

Future improvements include:

multiple simultaneous transients
prioritization of follow-up targets
adaptive observation scheduling
integration with real astronomical alert streams
longer horizon monitoring over days instead of hours

The broader goal is an AI system capable of supervising scientific monitoring tasks — not replacing astronomers, but assisting them by filtering uncertainty and highlighting events worth attention.

Sentinel is a step toward autonomous systems that observe, reason, and decide when evidence becomes meaningful.