Soar-On: Persistent Eyes In the Sky

Prediction markets on what the balloon counts, on SpacetimeDB
Live map: counts, frame confidence and the lot price with its reason (stand-in aerial footage until our flight)
Beach alerts: filed in Jira
student showcase
Architecture

Inspiration

Some questions can only be answered from above, right now, for hours. Is a rip current forming at Ocean Beach, and is anyone in the water near it? How many spaces are open in the lot? Is the crowd on this corner growing?

The tools we have miss one of those three things. A satellite photo is days old and too large to see a parking space. A drone can't stay up for more than 20 to 40 minutes, it's loud, and drones are banned across the Golden Gate National Parks, which include Ocean Beach. Ocean Beach also has no lifeguard towers, and SF's coast sees close to 200 rescues a year.

A small helium balloon on a tether doesn't have those limits. The helium does the work, so a power bank only has to run the camera and computer, and the balloon can stay up for hours, silently, at a fixed spot.

We talked to several beachgoers and swimmers outside the Palace of F. They told us lifeguards are often far outnumbered by the people in the water, and that a constant watch from above would make them feel safer.

What it does

A Raspberry Pi with a downward camera hangs from a tethered balloon and saves a frame every 2 seconds. A laptop ground station turns each frame into counts, and three groups use them:

"Beachgoers and patrols." - A model we trained on aerial rip-current imagery flags rips, and a people detector counts swimmers near them. Claude writes the alert a patrol member actually reads ("Rip forming north of the stairs, two people in the water near it"), with an explicit uncertainty line. A person confirms it on the live map, and the confirmed alert becomes a Jira incident assigned to whoever is on shift.

"Drivers." - Each parking space is marked free or taken on a live map, and a surge-pricing agent sets the lot price from occupancy within hard limits: at most one change every 5 minutes, at most 25% per step, a floor and a ceiling, and no change when the frame is unreliable. Every decision is logged with its inputs, so "why did the price go up at 4:10?" has an answer.

"Markets." - People bet play credits on what the balloon will count ("40 or more cars in lot A at 5 PM?"). A market agent proposes markets from the day's history and explains each one. Markets settle on the median of the good frames in the minute around the close, and are voided and refunded when there aren't enough good frames.

How we built it:

Balloon payload: Raspberry Pi 4, Logitech webcam, Grove barometer and accelerometer, on a power bank. The Pi records stills, video and sensors to its SD card no matter what, and serves a dashboard.

Detection: YOLO11n fine-tuned on VisDrone2019 (real drone imagery, 6,471 training images) for cars and people, reaching mAP50 0.72 for cars and 0.31 for pedestrians on the held-out VisDrone set (people are only a few pixels tall from above, so we count them and never try to identify anyone). A separate oriented-box model is trained on RipAID (2,186 images, CC BY 4.0), validated on two beach cameras it never saw in training: mAP50 0.32 (precision 0.49, recall 0.27), so a person confirms every rip alert.

Ground station: aligns every frame to a reference map of the site (ORB features and RANSAC), so drawn parking spaces stay put as the balloon drifts, and scores each frame's confidence from alignment, tilt and count stability.

Convex: the live map. Frames, observations, prices, decisions and alerts sync to every phone instantly.

SpacetimeDB: the prediction market. Accounts, markets, positions and an LMSR market maker run as a TypeScript module on maincloud, with integer money so balances always reconcile. The ground station is the only oracle allowed to post counts and settle.

Atlassian Forge: "Balloon Patrol" turns confirmed alerts into Jira incidents: de-duplicates repeats, assigns the on-shift lifeguard, escalates when people are in the water, queues low-confidence frames for a human, and writes an end-of-shift Confluence report. Every agent action is audited and revertible, and alert text is treated as untrusted data, so instructions hidden in it are flagged and ignored.

Challenges we ran into

Getting a headless Raspberry Pi onto a network at a venue that bans personal hotspots and isolates both Wi-Fi and wired clients. The Pi ended up running its own Wi-Fi network.
People are only 7 to 11 pixels tall from 100 m, so we count people and never try to identify anyone.
Keeping an autonomous pricing and market agent honest: guardrails in code, confidence gates, and voiding rather than guessing.

Accomplishments that we're proud of

Markets that settle from a live camera count, with money that provably balances.
An agent whose every action has a reason a judge can read, and that can't be talked into acting differently.

What we learned

Check the hardware before writing drivers. Our barometer turned out to be a DPS368 and our accelerometer an MMA7660, not the parts we planned for.
Venue networks block devices from talking to each other, on Wi-Fi and on the wired jacks, so the Pi now runs its own Wi-Fi network.
A rip-current model is much weaker on beaches it has never seen (mAP50 0.32), which is why a person confirms every alert.

What's next

Talk to SF Fire's surf rescue team about a trial at Ocean Beach, add a crossing counter for foot-traffic markets, and use the balloon's sharp labels as ground truth to calibrate satellite counts of the same places.

Tracks

Anthropic, Claude for Public Benefit. Community: swimmers and lifeguards at SF beaches, where lifeguards are outnumbered by swimmers. Claude reads each frame, writes the rip-current alert with an explicit uncertainty line, and a lifeguard confirms before anything is filed. We measure impact on the Alerts page: alerts drafted, seconds from frame to confirmed alert, people in the water near a rip.

Hyphenate, Autonomous Systems for Finance. A surge-pricing agent acts as the lot's revenue manager: it sets the parking price from live occupancy inside hard limits (one change per 5 minutes, 25% max per step, floor and ceiling), logs every decision with its inputs and a written reason, and refuses to act or settle on unreliable frames.

Convex. The live map is built on Convex live queries: frames, counts, prices and alerts sync to every phone instantly, and confirming an alert triggers a Convex action that files the Jira incident.

Spacetime. The prediction markets are a SpacetimeDB module: LMSR market maker, integer play credits so balances always reconcile, scheduled closing, and the ground station as the only oracle allowed to settle.

SpaceXSI. Built with Cursor. Grok's voice API reads every beach alert aloud so lifeguards keep their eyes on the water. We also compared the balloon to free satellite imagery: Sentinel-2 was 9 days old at 10 m per pixel, while the balloon sees about 7 cm per pixel every 2 seconds.

Atlassian, track: Best Agent. Balloon Patrol is an agent that does the patrol's triage in Jira (files, de-duplicates, assigns, escalates, writes the shift report), with the security layer built in: audit log, revert, and injection defense.