SideQuest

Explore your world, one quest at a time.

Comment

Inspiration

I love scavenger hunts, and I wanted to see if I could use Gemini 3’s AI to create truly unique quests based on any location in the world — not just templated missions, but culturally-aware adventures with real narratives.

When I discovered Gemini 3’s full multi-modal capabilities (text generation, vision, audio analysis, and image synthesis), I realized I could build something special: a game that transforms any city into an interactive adventure playground.

The retro 16-bit aesthetic was inspired by the SNES/Genesis games I grew up with, where every quest felt meaningful and every location told a story.

What it does

SideQuest transforms any location worldwide into an AI-powered scavenger hunt with:

  • Personalized quests
  • Pixel art visualizations
  • Intelligent multi-modal verification

The Complete Experience

  1. Enter any location
    “Da Nang, Vietnam” or “Brooklyn, New York” — works anywhere with Google Maps geocoding

  2. AI generates personalized campaigns
    Gemini 3 Flash creates 2–5 culturally-aware quests with narratives, objectives, hidden verification criteria, and real GPS coordinates

  3. 16-bit pixel art for every quest
    Gemini 3 Pro Image generates unique visualizations in a consistent SNES/Genesis aesthetic

  4. Navigate with live GPS
    Real-time distance tracking, Google Maps integration, and journey recording

  5. Multi-modal quest completion

    • Photo quests: Capture landmarks, street art, or local dishes
    • Video quests: Record 5–30 second clips of performances or environments
    • Audio quests: Capture ambient sounds, street music, or spoken content

  6. AI verification
    Gemini 3 Pro analyzes your media with advanced reasoning, combining visual/audio analysis with GPS context

  7. Appeal system
    If verification fails, explain your context and the AI reconsiders

The game tracks your entire journey (distance, duration, waypoints) and generates a complete route map at the end.

How I built it

Tech Stack

Frontend

  • Next.js 16
  • React 19
  • TypeScript 5
  • Tailwind CSS 4
  • Framer Motion

AI & APIs

  • Google Gemini 3 API
  • Google Maps API (Geocoding, Places, Distance Matrix)

Storage

  • IndexedDB (~50MB image cache)
  • localStorage (session persistence)

Gemini 3 Model Strategy

SideQuest showcases all three Gemini 3 models, each selected for its strengths:

┌────────────────────┬──────────────────────────┬───────────────────────────────────────────────────────────┐
│ Model              │ Use Case                 │ Why This Model?                                           │
├────────────────────┼──────────────────────────┼───────────────────────────────────────────────────────────┤
│ Gemini 3 Flash     │ Quest generation         │ Fast structured JSON output with cultural context         │
│ Gemini 3 Pro       │ Multi-modal verification │ Superior reasoning for complex decisions with GPS context │
│ Gemini 3 Pro Image │ Pixel art generation     │ Visual synthesis with consistent aesthetic                │
└────────────────────┴──────────────────────────┴───────────────────────────────────────────────────────────┘

Architecture
Quest Generation Pipeline
Location
→ Google Geocoding
→ Gemini 3 Flash (JSON)
→ Gemini 3 Pro Image (parallel batch)
→ IndexedDB Cache

Verification Pipeline
User Media (Photo / Video / Audio)
→ Gemini 3 Pro Analysis
→ [Multi-Modal Understanding + GPS Context]
→ Verification Result

Key Technical Decisions

  • Cultural context injection based on geocoded location data
  • Hidden verification criteria not shown to users to encourage authentic exploration
  • GPS proximity boost — more forgiving verification within 30 meters of the target
  • IndexedDB caching — saves approximately $0.20 per session resume by eliminating redundant image generation

Challenges I ran into

1. Choosing the Right Model for Each Task

Initially, Gemini 3 Flash was used for everything. While fast, verification accuracy suffered on complex multi-modal decisions.

Switching verification to Gemini 3 Pro dramatically improved accuracy — it better understands subtle visual cues and contextual nuances in photos, videos, and audio.

2. The “Hallucinated Coordinates” Problem

Early quest generation returned GPS coordinates pointing to rivers, private property, or inaccessible areas.

Solution: Explicit prompt constraints combined with Google Distance Matrix validation.

3. Multi-Modal Verification Accuracy

A photo of “a coffee shop” might get rejected because the AI expected very specific features.

Solutions

  • GPS proximity boost (more forgiving within 30 meters)
  • Appeal system where users explain context
  • Hidden criteria to encourage authentic exploration

4. Mobile Battery Optimization

Continuous GPS tracking initially drained 30%+ battery per hour.

Optimized to <10% battery drain per 30 minutes using:

  • Smart waypoint recording
  • Tuned geolocation accuracy settings

5. Image Generation Reliability

Implemented:

  • Adaptive timeouts (20–30 seconds)
  • Retry logic with exponential backoff
  • Graceful degradation when image generation fails

Accomplishments I'm proud of

🎯 Full Gemini 3 Model Family Showcase

Uses all three Gemini 3 capabilities — Flash for speed, Pro for reasoning, and Pro Image for visuals — in a single cohesive application.

🧠 The Appeal System

When verification fails, Gemini 3 Pro reconsiders using user-provided context, creating a genuinely intelligent and forgiving experience.

🎨 Consistent 16-Bit Aesthetic

Through prompt engineering, every generated image maintains a consistent SNES/Genesis pixel art style across thousands of potential quests worldwide.

🌍 Global Cultural Awareness

Generates culturally appropriate quests anywhere:

  • “Marrakech, Morocco” → souk exploration
  • “Copenhagen, Denmark” → hygge culture

📱 Client-Side Architecture

Zero backend servers — everything runs client-side with direct Gemini API calls.

The entire app is deployed as static HTML + JavaScript on Vercel.

What I learned

Multi-Modal AI Opens New Game Design Space

Analyzing photos, videos, and audio in a single model enables quest mechanics that weren’t possible before.

“Record the sound of the market” becomes a valid, verifiable objective.

Model Selection Matters

Using Gemini 3 Flash for everything was fast but sacrificed accuracy.

Switching verification to Gemini 3 Pro dramatically improved complex decision-making.

GPS Is Messier Than Expected

Mobile GPS has 10–50 meter accuracy drift.

Building verification systems that handle this gracefully required careful engineering and prompt tuning.

AI Needs Human Escape Hatches

The appeal system transformed user frustration into engagement.

Building human override mechanisms is crucial for real-world AI applications.

Caching Saves Real Money

Without image caching:

  • A 5-quest campaign costs $0.20 per resume

With 100 users resuming once:

  • That’s $20 saved

IndexedDB caching eliminates this entirely.

What’s next for SideQuest

🎮 Multiplayer Campaigns

Race friends on the same scavenger hunt.
Compare completion times, photos, and routes.

📚 Themed Quest Packs

  • History Mode — educational quests around historical landmarks
  • Food Tour — culinary exploration challenges
  • Photo Safari — artistic photography with composition criteria
  • Night Mode — evening-optimized quests for safe locations

🌐 Community Quest Platform

Users create and share custom campaigns.

Gemini 3 enhances user-created quests with narratives and verification criteria.

🔮 AR Integration

Overlay quest hints using device camera and orientation for augmented reality exploration.

📴 Offline Mode

Pre-download quest campaigns for areas with poor connectivity.

Sync completions when back online.

🏆 XP & Progression System

Level up, unlock themed campaigns, and earn badges for exploration milestones.

Built With

  • next-js
  • react
  • typescript
  • tailwindcss
  • framer-motion
  • gemini-3-flash
  • gemini-3-pro
  • gemini-3-pro-image
  • google-maps-api
  • indexeddb
  • geolocation-api
  • vercel

Try it now

Try it now: https://sidequest-seven.vercel.app

Enter your city, choose a campaign type, and let Gemini 3 transform your surroundings into an adventure 🗺️✨

Built With

Share this project:

Updates