Explore Art in 3D Space with Museum.Bingo

Inspiration

The idea for Museum.Bingo came from a universal frustration: visiting a museum with friends or family often turns into a scattered, passive experience. Kids get bored within minutes, adults feel overwhelmed by the sheer volume of artworks, and groups inevitably fragment. Traditional audio guides are solitary and formal; paper scavenger hunts feel like homework.

We asked ourselves: What if exploring a museum felt like a game night? The museum.bingo domain (from the name.com Domain Roulette challenge) was the perfect spark. Bingo is universally understood, social, and inherently satisfying. By combining it with augmented reality and on‑device AI, we could turn every gallery into a playful, competitive adventure. The goal was not just to build a gimmick, but to create a tool that museums would actually want to license – one that increases dwell time, encourages repeat visits, and makes art appreciation genuinely fun.

What it does

Museum.Bingo is a cross‑platform mobile app (React Native) that transforms any museum visit into an interactive AR scavenger hunt.

Automatic museum detection – Using GPS geofencing and the Google Places API, the app knows when you step into a museum (or you can select one manually).
Custom bingo card – Each museum gets a 3×3 or 4×4 bingo card filled with playful, visual prompts (e.g., “Painting with a dog”, “Sculpture that looks uncomfortable”, “Self‑portrait with suspicious eyes”). Prompts are generated dynamically from the museum’s collection using an LLM (GPT‑4 via Nimble’s Answer API).
Real‑time artwork recognition – Point your phone’s camera at an artwork. The app runs a quantised MobileCLIP model on‑device, compares the frame’s embedding against a pre‑computed index (FAISS), and instantly validates the matching bingo tile.
AR polish features –
- Bingo chip drop: A playful chip falls onto the screen with a “cha‑ching” sound and haptic feedback.
- Heat vision (hint mode): If you’re stuck, toggle a compass overlay that points to the nearest unvalidated artwork using iBeacon / Wi‑Fi RTT indoor positioning.
- Virtual confetti: When you complete a row, confetti explodes and lands on the actual floor using ARKit/ARCore plane detection.
Multiplayer competition – Create or join a room. Everyone sees the same bingo card. Real‑time leaderboards (via WebSockets + Firebase) show who’s winning – perfect for families, school trips, or friend groups.
Voice & multilingual support – Speak commands (“validate tile 3”, “give me a hint”) and get spoken feedback. The UI and bingo prompts automatically translate into 5+ languages using on‑device ML Kit.
Museum analytics – Anonymised visitor behaviour (completion rates, favourite prompts, time per gallery) is aggregated via Tower pipelines and presented to museum partners, helping them improve exhibitions.

How we built it

Frontend (React Native)

react-native-vision-camera v4 with Skia plugin for real‑time frame processing and GPU‑accelerated overlays.
@shopify/react-native-skia for drawing the bingo chip and compass arrow directly on the camera feed.
react-native-fast-tflite to run a quantised MobileCLIP model (fine‑tuned on museum artwork).
FAISS index (downloaded from S3) for offline nearest‑neighbour search.
react-native-voice and expo-speech for voice commands and text‑to‑speech.
Google ML Kit Translate for on‑device translation of prompts.
Zustand for state management (bingo progress, multiplayer room).

Backend (Node.js + Firebase)

Express server with Firebase Admin SDK.
Firestore stores user profiles, museum metadata, game sessions, and multiplayer room definitions.
Firebase Realtime Database caches ephemeral leaderboard scores for sub‑second updates.
Socket.IO server broadcasts tile completions and leaderboard changes to all clients in a room.
Stripe handles premium subscriptions (family plans, unlimited museums).

Data pipeline (Tower + Nimble + S3)

Museum onboarding: Admin provides a URL or CSV.
Nimble Crawl API extracts artwork titles, images, descriptions from the museum’s website (handles JavaScript‑rendered sites).
Images are downloaded, resized to 224×224, and stored in S3.
Tower GPU pipeline runs CLIP (ViT‑B/32) to generate 512‑dim embeddings.
Embeddings are indexed with FAISS (IndexFlatIP) and serialised to S3.
The mobile app downloads the index (≈50 MB for 2000 artworks) and performs offline searches.

AR features

Bingo chip: Skia canvas inside the frame processor worklet; animation driven by Reanimated with withTiming / withSequence.
Heat vision: Indoor positioning via iBeacons (using react-native-ble-plx) or Wi‑Fi RTT; bearing calculated from compass and artwork coordinates.
Confetti: Native bridge to ARKit (iOS) and ARCore (Android); plane detection attaches particles to horizontal surfaces.

Multilingual & voice

i18next with 5 locale files (en, es, fr, de, zh).
On‑device translation of bingo prompts using Google ML Kit Translate (models downloaded once).
Voice command parser uses simple keyword matching (extensible to NLU).

Deployment

Backend on Firebase Cloud Functions + Cloud Run.
Tower pipelines scheduled or triggered via webhooks.
S3 (AWS) for image and index storage.
Mobile app distributed via TestFlight (iOS) and internal track (Android).

Challenges we ran into

On‑device CLIP latency – Running a full CLIP model on every frame was too slow. We solved it by quantising to int8 (MobileCLIP‑S2), lowering input resolution to 224×224, and running inference only every 5th frame. Result: <50ms inference on modern phones.
Indoor positioning accuracy – GPS fails inside museums. We integrated iBeacon scanning with trilateration, but RSSI noise caused jitter. A Kalman filter smoothed the signal, and we added a fallback to Wi‑Fi RTT where available.
FAISS index download size – A 2000‑artwork index is ~50 MB, which is large over cellular. We implemented incremental downloads (only embedding vectors, not images) and used gzip compression, reducing to ~18 MB.
AR confetti plane detection – ARKit/ARCore sometimes fails to detect floors in low‑light or pattern‑less galleries. We added a fallback to a screen‑space particle effect (non‑AR) that still feels celebratory.
Multiplayer state sync – Keeping leaderboards consistent across multiple clients with flaky network was hard. We used Firebase Realtime Database as the source of truth and WebSockets for push updates; clients also pull on reconnection to resolve conflicts.
Nimble crawling museum sites – Many museum websites use infinite scroll or client‑side routing. We configured Nimble’s VX8 driver with custom JavaScript injection to trigger lazy loading before extraction.
Translating bingo prompts – Machine translation often loses the playful tone. We implemented a hybrid approach: translate with ML Kit, then run a small LLM (via Nimble Answer) to rephrase into a natural, fun prompt in the target language.

Accomplishments that we're proud of

🏆 Fully functional AR bingo loop – From camera frame to CLIP inference to tile validation to AR confetti, the whole pipeline runs at 30+ FPS on a standard iPhone 12.

🌍 Offline‑first design – After the initial index download, the app works entirely offline. No internet? No problem – the bingo game continues.

🤝 Real‑time multiplayer – We witnessed two strangers at a coffee shop join the same room and compete to finish a bingo card. The joy and trash talk were priceless.

🧠 End‑to‑end AI pipeline – We built a complete MLOps flow: Nimble data extraction → Tower GPU embedding generation → FAISS indexing → mobile download → on‑device inference. All in 13 days.

🎉 Delightful AR polish – The bingo chip drop, heat vision compass, and floor‑landing confetti were the features that made test users smile every single time.

📦 Clean, documented code – The entire codebase is TypeScript (mobile + backend) with extensive comments, plus a 10‑page README with Mermaid diagrams.

🧪 Cross‑platform – The app runs identically on iOS and Android, including AR features (via native bridges).

What we learned

MobileCLIP is surprisingly capable – A quantised 12 MB model can recognise artworks with >85% accuracy when fine‑tuned on a few hundred examples per museum.
Indoor positioning is still hard – iBeacons are the only pragmatic solution for room‑level accuracy, but they require museum partners to install hardware. Wi‑Fi RTT is promising but not yet widely available.
Nimble is a game‑changer for hackathons – It turned weeks of web scraping into a few API calls. The ability to run JavaScript‑rendered pages and get structured data saved us days of work.
Tower simplifies AI pipelines – Instead of wrestling with Kubernetes or Airflow, we wrote Python functions, added a Towerfile, and deployed in minutes. The GPU‑accelerated CLIP step cost less than $2 for a 1000‑artwork museum.
AR polish > AR complexity – Simple overlays (bingo chip, compass arrow) had a bigger impact on user delight than complex 3D models. Less is often more.
Firebase is great for real‑time, but watch your reads – Multiplayer leaderboards can generate many RTDB reads. We reduced cost by batching updates and using socket.io for most broadcasts.

What's next for Museum.Bingo

Short‑term (post‑hackathon)

Museum self‑onboarding portal – Allow museum staff to upload a CSV or URL and automatically generate a bingo card and embeddings without engineering help.
Smart prompt difficulty – Use a contextual bandit to adjust prompt difficulty based on user skill (easy/medium/hard).
More AR effects – Seasonal confetti (snowflakes in winter, flower petals in spring) and customisable bingo chip skins.

Medium‑term (3‑6 months)

Museum gift shop integration – Stripe Connect to let users redeem bingo wins for real‑world discounts.
Exhibit‑specific challenges – For special exhibitions, generate temporary bingo cards with time‑limited prompts.
School & group plans – Teacher dashboard to create private rooms, track class progress, and export participation reports.

Long‑term vision

Cross‑museum leaderboards – Compare your bingo scores across multiple cities or countries.
AR navigation – Full indoor AR navigation (not just a compass arrow) using visual SLAM.
Open‑source museum dataset – Share anonymised embedding indices to help other developers build museum apps.
Partnership with name.com – Offer free .museum domains to museums that adopt Museum.Bingo, creating a network of “bingo‑ready” cultural venues.

How you can help