Earthdex

Inspiration

We grew up wishing the Pokédex was real. Pokémon made catching and learning about creatures feel like an adventure, and the real world is full of creatures nobody ever stops to look at. We wanted to turn every walk outside into that same adventure, and make nature easy and fun for kids to learn about. EarthDex is our version of a real-life Pokédex as the app scans an animal, plant or fungus, learn about it, and add it to your collection.

What it does

Scan: Point your phone camera at an animal, plant, or fungus and tap the photo button on our handheld Pokédex.

Identify: The photo goes to Google’s Gemini AI, which returns the common and scientific name, a short Pokédex inspired description, diet, habitat, and a safety note when relevant.

Verify: We check the scientific name against GBIF, an open biodiversity database, and pull the species’ official IUCN conservation status. The app labels each result as verified or as an AI suggestion.

Listen: The Pokédex reads the description aloud.

Collect: Each catch is saved with your location and shown on an interactive 3D globe.

Level up: New species earn more XP than repeat sightings, and XP builds your level.

Compete: A leaderboard ranks explorers by distinct species discovered.

How we built it

Frontend: React and Vite, with react-globe.gl for the 3D globe. Browser camera access (getUserMedia) and geolocation handle the scanning and map pins. AI backend: A serverless function on Vercel calls Gemini, so our API key never reaches the phone. Pokédex interface: We drew our own handheld Pokédex artwork. Because the buttons are irregular shapes, the app reads each button image’s transparency so only the drawn part is tappable. The zoom and D-pad buttons are split into pieces that press separately. Scoring: XP is configurable (50 for a new species, 10 for a repeat), and the level follows a simple formula: $$\text{level} = \left\lfloor \frac{\text{XP}}{100} \right\rfloor + 1$$ Leaderboard: It has a podium, ranked rows, and All time, Weekly and Monthly filters. The shown leaderboard is just a Demo explaining each player’s distinct species and does the ranking, so scores can’t be edited in the browser.

Challenges we ran into

We found that AI will happily name a species even when the photo is blurry. We told Gemini to report low confidence, and the app asks for a clearer photo instead of forcing an answer. GBIF verification catches invented names. Species versus breed. We kept the identification at the level the photo supports, such as genus or family, and avoided guessing breeds or cultivars. Camera permissions are needed since the Browser cameras needs HTTPS and user permission, and they behave differently on phones. We added a fallback that opens the phone’s camera app. There were issues with irregular buttons as getting taps to land on the right part of our artwork took pixel-level hit testing. It was also a challenge to stay within scope of our project since Camera, AI, voice, location, a 3D globe and accounts could each have broken the project. We built the app in stages and tested each one before adding the next.

Accomplishments that we're proud of

That our scanner works! Many features within the Earthdex also work, so we are proud of everything we accomplished.

What we learned

How to run an AI model safely from a serverless backend without exposing keys. That AI answers need verification against trusted sources. That a small, working end-to-end flow beats a large unfinished one. How to design, data, and AI work fit together best when each teammate focuses on one part, then connects through Git.

What's next for Earthdex

We wish to integrate a real sign-in and saved accounts, so the leaderboard shows real players instead of labelled sample data Trophies and coins for avatar clothes (we will hand-draw the clothing options) A personal collection page and a “Learn More” Q&A with the Pokédex