Subway Mate

home_page
prompts
prompts

Inspiration

Navigating the NYC subway for the first time is a notoriously disorienting and stressful experience. Deep underground, GPS signals die, cell service drops, and static overhead signs become easy to miss in dense crowds. For tourists, taking the wrong stairwell means an exhausting backtrack through multi-level concrete corridors. For individuals with disabilities, parents with strollers, or travelers with heavy luggage, the challenge is critical—finding step-free paths, accessible entrances, and functioning elevators often feels impossible.

We set out to build Subway Mate to solve this underground routing problem. Instead of relying on a jumping blue GPS dot that doesn’t work subterranean, Subway Mate anchors navigation to the physical world around you—using recognizable visual cues, station storefronts, elevator ID tags, and signage to guide you step-by-step to your platform.

What it does

Subway Mate is an accessibility-first navigation assistant available through a responsive web app and natively via iMessage:

AI-Powered Camera Navigation: Point your phone camera at a nearby landmark or directional sign and type or speak your destination. The app identifies your anchor node and generates clear instructions: "Take a left past the storefront, head down the corridor, and the Queens-bound platform will be on your left."
Zero-Install iMessage Bot: Downloading a heavy app underground with weak cell coverage is frustrating. With our iMessage integration, commuters simply text a dedicated number. Text where you are ("I see Elevator 412") or send a quick photo of a nearby sign, and the bot instantly replies with step-by-step directions.
Conversational Voice Loop: Users can speak natural language questions into the app, and it responds with text-to-speech audio guidance, acting as a virtual guide.
Step-Free Accessibility Routing: Accessibility is baked into the core routing logic. A single toggle filters out all stairs in favor of barrier-free paths, accessible doors, and elevators.

How we built it

The Interfaces: We built a responsive React Web App utilizing Vite for real-time camera-capture, alongside a Node.js/Express-powered backend managing iMessage and SMS webhooks. We also utilized the Photon API alongside targeted data extraction via Apify to enrich location context and handle broader transit geocoding queries.
Custom Topological Spatial Graph: To solve the complete absence of underground GPS, we constructed a localized JSON graph database linking physical visual landmarks (storefronts, signs, elevator ID tags) directly to navigational nodes across station levels.
The OCR-to-LLM Pipeline (Tesseract + Grok): Live camera frames and uploaded photos are processed using Tesseract.js. Because real-world OCR is often messy due to motion blur and bad subway lighting (e.g., reading "1Jptown" instead of "Uptown"), we feed the raw, noisy OCR output directly into Grok. Grok acts as a fuzzy-matching engine, intelligently resolving spelling mistakes and mapping the messy OCR to the exact station node.
Zero-Shot Spatial Reasoning Fallback: If a user texts a landmark not hardcoded in our JSON database, the system dynamically prompts Grok to rely on its own pre-trained weights. Grok's internal knowledge of NYC transit allows it to infer the user's location based on unmapped landmarks and map them to the nearest known node.
Smart Routing Engine: Once the starting node is confirmed, our custom Dijkstra pathfinding algorithm calculates the shortest route. To implement the accessibility toggle mathematically, we dynamically alter the edge weight $w(u,v)$ between nodes $u$ and $v$ based on the user's requirements:

$$w(u, v) = \begin{cases} \infty & \text{if edge is 'stairs' and accessibility is TRUE} \ \text{base_distance}(u, v) + \text{congestion_penalty} & \text{otherwise} \end{cases}$$

This ensures the pathfinding algorithm mathematically bypasses all non-accessible routes. Grok then translates the resulting graph traversal into conversational, human-readable directions.

Challenges we ran into

Taming Messy OCR & Unmapped Landmarks: Grabbing clean text from a shaking smartphone camera in a dimly lit subway is incredibly difficult. Early tests failed because the OCR output garbage characters, breaking our strict database queries. Routing the OCR output through Grok allowed us to turn a fragile exact-match system into a resilient, AI-driven fuzzy matcher.
The "Black Box" of Indoor Transit Data: Our biggest technical hurdle was the complete lack of indoor station data. While the MTA provides APIs for real-time train arrivals, they offer zero data or API endpoints for station interiors. There is no official database detailing sign placements, vendor locations, or accessible elevator coordinates. We had to manually map out station layouts and build our structured JSON matrix from scratch.
The AR Pivot (From Live Video to Visual Anchor Cues): Our original vision was a continuous Augmented Reality (AR) experience rendering floating 3D directional arrows on screen in real time. Because our team was not physically located in NYC during the hackathon, we lacked the live video datasets of station corridors needed to train and test reliable spatial tracking. We pivoted to static visual cues and photo-anchored LLM processing. This pivot ultimately made Subway Mate far lighter, faster, and better tailored for low-bandwidth environments.

Accomplishments that we're proud of

Resilient AI Pipeline: Successfully chaining raw Tesseract OCR into Grok to create a system that can infer, guess, and correct real-world visual noise with high accuracy.
Omnichannel Accessibility: Delivering a fully functional navigation assistant through iMessage so users can navigate without downloading a new app underground.
Accessibility-First Design: Building a routing engine where step-free pathfinding is a core algorithmic parameter, not an afterthought.

What we learned

LLMs as Data Sanitizers: Vision and Language models like Grok are incredibly powerful as middleware to sanitize, correct, and format messy real-world sensor data before it hits a traditional routing algorithm.
Designing for GPS-Denied Environments: When traditional hardware location sensors fail, anchoring spatial logic to human-recognizable visual landmarks (signage, storefronts, elevator IDs) is significantly more reliable.
Agile Problem-Solving: We learned how to rapidly pivot away from technical roadblocks (like our failing live WebAR stream) toward a practical, low-bandwidth solution that still solved the core user problem.

What's next for Subway Mate

Expanding Station Coverage: Mapping visual nodes and accessibility routes for major multi-level hubs like Penn Station, Grand Central Terminal, and Fulton Center.
Live Elevator Outage Integration: Incorporating real-time MTA elevator status updates to dynamically recalculate step-free routes around out-of-service elevators.
Crowdsourced Visual Anchors: Allowing commuters to report new visual cues, pop-up shops, or temporary station obstructions to continuously update our JSON routing graph.