What's Up, New York!

What’s Up, New York! is a data driven platform that converts natural language requests into complete, optimized NYC itineraries with real subway directions, timing, and costs

Comment

Inspiration

Planning a day in New York City is overwhelming. The project is motivated by the gap between “I have a day in NYC” and actually turning that intent into a realistic, efficient plan that adapts in real time.
The repo frames the app as voice-first, real-time, and map-centric, so users can discover places conversationally while moving through the city.
As stated in the README, the goal is an AI-powered NYC adventure planner with real-time 3D visualization, voice control, and automated reservations.

What it does

  • Displays an interactive 3D NYC map with buildings, overlays, and cinematic effects.
  • Shows real-time subway service and train movement using MTA GTFS-Realtime feeds.
  • Enables semantic discovery of restaurants, events, and landmarks using vector search and embeddings.
  • Generates AI itineraries from natural-language requests with budget- and transit-aware routing.
  • Supports voice input and spoken summaries (speech-to-text and text-to-speech).
  • Executes automated reservations by parsing intent and completing booking flows via browser automation.

How we built it

  • Frontend: React + TypeScript + Vite, Mapbox GL JS for 3D maps, Three.js for custom 3D elements, Tailwind CSS for styling, and Framer Motion for animations.
  • Backend: Node.js + Express server handling transit data ingestion, itinerary planning, vector search queries, and reservation orchestration.
  • Data layer: MongoDB Atlas for structured place data and vector search, with scripts for embedding, seeding, and index validation.
  • AI & voice: Google Gemini for planning, parsing, and embeddings; ElevenLabs for spoken output; Web Speech API for voice input.
  • Automation: Fetch.ai (ASI:One) for intent parsing and Playwright for end-to-end reservation execution.

Challenges we ran into

  • GTFS-Realtime inconsistency: transit feeds vary in structure and reliability, requiring defensive parsing and heuristics.
  • Vector index setup: MongoDB Atlas vector indexes require special handling and explicit validation.
  • Transit visualization: normalizing route IDs and service colors for accurate map rendering.
  • Itinerary feasibility: constraining LLM output so plans are time-realistic, transit-aware, and mappable.
  • Reservation reliability: booking flows differ across platforms, requiring selector fallbacks and robust automation logic.

Accomplishments that we're proud of

  • A fully interactive 3D map with live subway trains and service status.
  • Cross-category semantic search across restaurants, events, and landmarks.
  • An end-to-end voice workflow: speak → plan → visualize → hear the summary.
  • Automated reservations triggered directly from user intent.

What we learned

  • Real-world transit data is messy; robust fallback logic is essential.
  • Vector search quality depends heavily on indexing and normalization, not just embeddings.
  • LLM planning requires hard constraints to be reliable in production contexts.
  • Browser automation for real services demands defensive engineering and resilience.

What's next for What's Up, New York!

  • Expand and continuously refresh the NYC data pipeline for places and events.
  • Integrate more precise routing and scheduling for tighter itinerary timing.
  • Harden auto-reservation flows and add clearer user confirmation and safety checks.
  • Optimize 3D rendering performance and ship a stable public demo.

Built With

Share this project:

Updates