Birdseye

Inspiration

Most social media apps feel like they’re pulling you away from spending time with friends outdoors. Birdseye was made to do the opposite. Using machine learning to develop real time bird recognition and encouraging users to share their best finds and discover new trails and birding groups, we built the first ever social networking app for birdwatchers. By using AI to teach birders about their local avian ecosystems and connecting them with other birders near them, Birdseye brings what may seem like an old hobby to the modern age.

What it does

Start your adventure with a click. You tap the bird walk! sign and put your phone in your pocket.

Live audio ID: your phone records short audio chunks while tracking your route. The server runs each chunk through BirdNET, a neural network trained on bird vocalizations, and species pop up on your screen seconds after they sing.
Photo ID: snap a photo of a bird, and a proprietary vision/classification model suggests the species for you to confirm.
Recap: when you end the walk, you get a Strava-style summary with your route on a map, a pin where each bird was heard, and a 5-second clip of each bird's actual call from your recording. An AI-written field note explains how to recognize each bird and how unusual it is for that place, season, and time of day.
Rarity and points: rarity comes from live eBird sightings near you. Common birds are worth 10 points, uncommon 20, and rare 50. Hearing and photographing a bird score separately. A bird heard outside its normal range or season is flagged as an anomaly, and it only pays out (with a 100-point bonus) once a photo confirms it.
Social: walks post to a feed where friends can chirp (like), comment, and reply in real time.
Community: weekly quests fill a nest of pixel-art eggs, and a community map shows where everyone is hearing birds.
Privacy: exact routes are visible only to mutual friends. Everyone else sees a neighborhood label.

How we built it

Frontend: a React + TypeScript + Vite PWA installed to the iPhone home screen, with Tailwind, Mapbox GL, and hand-drawn pixel art.
Backend: FastAPI in Docker behind Caddy on a Vultr VPS. BirdNET is loaded once at startup, and ffmpeg handles audio conversion and clip cutting.
Data: Supabase (Postgres + PostGIS) for auth, storage, row-level security, and realtime updates for chirps and comments.
AI: BirdNET for audio, an OpenAI + internal algorithm vision model for photos, and an LLM for the per-bird field notes. The field notes come from one batched JSON call per walk.

Recording pipeline. The phone restarts its recorder every $T = 6$ s, so every chunk is a self-contained audio file. Chunks are saved to the phone's local storage and uploaded in order with retries, so a dead zone on a trail doesn't lose a bird.

Filtering out false IDs. Every BirdNET label is mapped to the eBird taxonomy, and non-bird sounds like voices, engines, and dogs are dropped. For each detection with confidence $c$:

$$\text{keep} = \begin{cases} \text{normal} & \text{if expected here and } c \ge 0.6 \ \text{anomaly} & \text{if not expected and } c \ge 0.85 \ \text{drop} & \text{otherwise} \end{cases}$$

"Expected here" comes from BirdNET's list of species likely at that location and date.

Scoring. A walk's points are

$$P = \sum_{s \in \text{species}} \Big( v_s\,\mathbb{1}[\text{heard}_s] + v_s\,\mathbb{1}[\text{photo}_s] + 100\,\mathbb{1}[\text{confirmed anomaly}_s] \Big),\quad v_s \in {10, 20, 50}.$$

Confirmed anomalies are scored at the rare value, $v_s = 50$. Every point is a row in a points log, so the math can be re-run whenever a late photo confirmation arrives.

Challenges we ran into

Three engineers, 36 hours. We split into frontend, ML/audio, and game/social workstreams, sharing a written data model and API contract. Every change to the contract was logged, so we could build in parallel against stubs.
Bird Detection We struggled to find an existing good bird detection model that was free and available to use, so we built our own custom framework.

Accomplishments that we're proud of

The amazing bird detection via both audio and visual sources. With tested 99% accuracy.
Every recap includes the actual recording of each bird you heard, not a stock sample.
It's genuinely fun. We kept "testing" by going on walks, touching grass. -Teaching the community about local nature and the environment while immersing users in it.
Duck-ing awesome hand-drawn graphics.

What we learned

Audio ML in the real world is mostly about data plumbing: formats, chunk boundaries, timestamps, and thresholds matter as much as the model itself.
Browser APIs on iOS have sharp edges, and building the riskiest piece first (the iPhone recording test) saved us.
Making every step safe to re-run (scoring, clip generation, summaries) turns production bugs into a one-command recovery instead of lost data.
Whimsical Hackathons are the most fun ones!

What's next for Birdseye

Citizen science: exporting confirmed sightings to eBird so every walk contributes to real research.
Recording in the background with the screen off, and an offline mode for trails with no signal.
Make it an iOS app available in the Apple App Store!