Inspiration
Every Aggie knows the scene: class lets out at 10:20, a hundred people walk to the same stop, and the first bus pulls up already full. TAMU's buses carry about a million riders a month (per the National Transit Database). Their public map shows where each bus is, but you can't tell whether you'll fit on it or whether waiting ten minutes would help. On a home game day, with 100,000 people heading for Kyle Field, it's even harder to tell.
We noticed the buses' automatic passenger counters (APC) already report how many people are on board, live. We wanted to turn that into something a student can act on: is my bus going to be packed, and should I leave earlier?
What it does
- Live map: every bus with a ring showing how full it is. Search routes by name, and replay recorded trips.
- Forecast: expected fullness 15, 30 and 60 minutes ahead per route and direction. It gives a likely range, the chance of being crowded, and plain-language reasons ("classes letting out", "rain", "already busy").
- Events: campus events and service alerts mapped to the routes they affect, with the Tennessee game at Kyle Field featured first.
- Plan my trip: search two stops by name and compare options by wait, ride, walk and expected fullness.
- History: real readings next to what the model predicted.
- Genie and an ops dashboard for asking questions of the data in plain English.
How we built it
Everything runs on Databricks serverless:
- Collection. A collector polls TripSpark's public map feed every 30 s, which is fewer requests than one open browser tab. Batch jobs pull TAMU GTFS, the campus events calendar, Open-Meteo and NWS weather, and NTD ridership.
- Lakehouse. The raw files land in a Unity Catalog Volume. Auto Loader feeds Lakeflow Declarative Pipelines through Bronze → Silver → Gold, with data-quality expectations: stuck passenger counters, GPS outside the service area, stale pings. The Silver layer derives stop visits and headways (to measure bunching), and Gold aggregates into 15-minute route-direction windows.
- Forecasting. For a bus with $o$ riders on board and capacity $c$, the load factor is $L = o / c$. For each route-direction and 15-minute window $w$, the target is the busiest moment in the window:
$$y_w = Q_{0.9}\big({L_t : t \in w}\big)$$
We train LightGBM quantile regressors for $\alpha \in {0.1, 0.5, 0.9}$ by minimising the pinball loss
$$\ell_\alpha(y, \hat y) = \max\big(\alpha\,(y - \hat y),\; (\alpha - 1)\,(y - \hat y)\big)$$
and a classifier for $P(\max L \ge 0.75)$. We trained one set per horizon (15, 30 and 60 min), tuned each with Optuna, and logged everything to MLflow. Models are registered in Unity Catalog and served from the @champion alias. Each new model must beat two baselines before it replaces the current one:
- persistence: the next window looks like the last one;
- seasonal: the average for that route, day type and time slot.
- Explanations. Exact TreeSHAP contributions are grouped into families (time, class change, event, weather, recent load, bunching), and the top three become the reasons shown in the app.
- App. A Databricks App with a FastAPI backend and React, Vite and MapLibre on the front. Every SQL query is a fixed statement; anything that needs input uses typed parameters.
Jobs refresh the pipelines and rescore the forecasts every 15 minutes, retrain nightly, and email us if a health check fails.
Challenges we faced
- We had one weekend of real data. Collection started on 7 November, so there was no history to train on. We wrote a simulator calibrated to NTD so that simulated boardings add up to the real monthly totals:
$$\sum_{\text{trips in month } m} \text{boardings} = \text{UPT}_m^{\text{NTD}}$$
Its time-of-day profile, class-change waves and event effects are our assumptions, so every modelled row carries a data_origin of observed or simulated. We never mix the two in a metric, and the UI labels anything simulated.
- Measuring honestly. On the simulated test set LightGBM clearly beats the seasonal baseline (MAE 0.047 vs 0.081 at 30 min). On real riders the p10–p90 range covers only about 30% of outcomes, against an 80% target. So we built conformal widening: on held-out observed windows, compute the nonconformity score $s_i = \max(\hat q_{0.1} - y_i,\; y_i - \hat q_{0.9},\; 0)$ and widen both ends of the range by the score at rank $\lceil 0.8\,(n+1) \rceil$. This, and judging promotion on real data, both switch on automatically once there are at least 200 labelled real windows across at least 2 weekdays.
- Network walls.
- Serverless compute couldn't reach TAMU's domains, so a laptop collector uploads finished files, and a 30-second snapshot keeps the live map current.
- Auto Loader assumes files never change once they land, so we rotate files every 5 minutes and upload only closed ones.
- Late in the weekend the campus DNS filter flagged our brand-new app domain as malware.
- Time zones. TripSpark reports local Central time with no offset; everything is stored as UTC and shown in Central time.
- "Data stale" at midnight. Our freshness indicator panicked when the buses stopped for the night. We made it read the schedule, so it now says "Buses are off for now · next service 9:00 AM" instead.
- Making it readable. Our first Events page listed
01-04 / 0, 03-05 / 0…. Rewriting every page in plain language, with named routes, stop-name search, and layouts that work on phones, took longer than the model.
What we learned
- Labelling where each number came from builds more trust than a polished accuracy figure.
- Simple baselines are hard to beat; on real data, persistence is still competitive.
- Lakeflow expectations and Unity Catalog lineage made it easy to show where every number came from.
- A forecast only helps if a student can read it in two seconds at a bus stop.
What's next
- Collect a full week of weekday data and the 14 November game day, then let the observed promotion gate and interval calibration take over.
- Host a public read-only copy, and share the project with TAMU Transportation Services.
Built With
- databricks-apps
- fastapi
- genie
- lightgbm
- maplibre-gl-js
- mlflow
- pandas
- pyarrow
- pyspark
- python
- react
- scikit-learn
- sql
- typescript
- vite
