ApexPilot

An AI-assisted driver modeling platform using Mamba architectures to learn, visualize, and analyze racing behavior. Discover actionable insights on throttle, braking, and steering across every corner.

Comment

Inspiration

I love looking at pro racing data and trying to answer a very simple, very hard question:

“In this exact corner, at this exact moment, what would a faster driver be doing?”

Race broadcasts and games like Forza/F1 show ideal lines and ghost cars, but real-world amateur and semi-pro drivers usually only get raw telemetry dumps and a timing sheet. ApexPilot is my attempt to turn that into an AI driving coach that can sit on top of real race data and say: “Here’s what the fast laps would have done in your place.”

What it does

ApexPilot takes CAN/telemetry from the Toyota GR86 Cup at Sonoma 2025 Race 1 and:

  • Reconstructs each lap as a continuous progress axis around the track.
  • Automatically detects corner regions from steering and lateral G, and colors the lap by corner on the plot.
  • Trains a sequence model on the best laps of the front-running cars to learn “expert” steering, throttle and brake behavior.

  • For any selected car + lap, it runs the AI model over that lap’s telemetry and visualizes:

    • Steering vs AI steering, plus ΔSteer (AI − driver)
    • Throttle vs AI throttle, plus ΔThrottle
    • Brake vs AI brake, plus ΔBrake
    • All plotted over progress_dist with shaded corner blocks and sector markers.

The result is an interactive web page where you can scrub through a lap and instantly see, corner by corner, whether the AI thinks the driver should:

  • Turn in harder or softer
  • Get on throttle earlier or later
  • Brake more or less

How we built it

Data & preprocessing

  • Started from multi-gigabyte race telemetry CSVs (speed, steering, throttle, brake, RPM, gear, GPS-like progress, etc.).
  • Split and cleaned them by vehicle_id, then identified true flying laps using time filters and metadata.
  • Built a distance-based progress metric (progress_dist) by integrating speed over time.
  • On a reference fast lap, auto-detected corners by:
    • Rolling smoothing of steering and lateral acceleration
    • Thresholding (|steer| > 5° or |accy| > 0.2 g)
    • Merging contiguous segments into corner intervals on progress_dist.
  • Used this to assign a corner_index to every sample point in every lap.
  • Smoothed steering, throttle and brake with a short rolling window to remove jitter but preserve real driver intent.

Model

  • Built a sequence-to-sequence policy model (first LSTM, then upgraded to a Mamba-style state space model) in PyTorch.
  • Inputs per time step:
    • progress_dist, corner_index
    • Speed, longitudinal & lateral accel, RPM, gear, etc.
  • Outputs per time step:
    • Smoothed steering, throttle, and front brake pressure.
  • Trained on the best laps of the top-finishing cars to approximate an “expert driver” mapping: state → ideal actions.

Visualization

  • Exported model predictions back into the full telemetry dataset as:
    • pred_steer / pred_ath / pred_brk
    • diff_steer / diff_ath / diff_brk (pred − actual)
  • Built a lightweight web UI that:
    • Lets you select vehicle and lap.
    • Renders six stacked subplots (Steer, ΔSteer, Throttle, ΔThrottle, Brake, ΔBrake) vs progress_dist.
    • Adds colored background bands per corner and dashed vertical lines for sector boundaries.
    • Uses responsive layout and a height slider so you can zoom into the details without overwhelming the screen.

Challenges we ran into

  • Data quality & alignment
    • Original logs were huge, noisy and sometimes misaligned in time.
    • Speed channels and control inputs had different sampling rates and missing chunks; laps could overlap if naïvely integrated.
  • Finding “true” laps
    • Warmup, pit in/out and safety car segments had to be filtered out without losing edge cases like partial spins and re-joins.
  • Corner detection without GPS map
    • Had to derive corners purely from steering and lateral G, then match them consistently across all laps.
  • Sequence modeling on small, messy data
    • Needed enough smoothing to let the model see patterns, but not so much that it erased real driver corrections.
    • Getting loss down while keeping predictions physically reasonable took a lot of iteration.
  • Making the visualization readable
    • With six traces + diffs + corner bands it’s easy to create “spaghetti plots”.
    • Took several design passes to keep it clean: color choices, line styles, selective legends, adjustable chart height.

Accomplishments that we're proud of

  • Went from raw race telemetry to a working AI driver coach UI end-to-end.
  • Built a robust pipeline that:
    • Detects corners automatically,
    • Aligns every sample onto a shared progress axis,
    • And enriches the dataset with expert-policy predictions and diffs for every lap.
  • Achieved a low prediction error on steering and meaningful predictions on throttle/brake, good enough that overlays look like a faster ghost driver tracing over the real lap.
  • Created an interactive visualization where you can visually debug a driver’s technique corner by corner, instead of staring at spreadsheets.
  • All of this runs on real race data (GR86 Cup Sonoma 2025 R1), not simulated laps.

What we learned

  • Cleaning and aligning telemetry is half the project.
    The fancy model means nothing if laps are mis-cut or channels are misaligned by a few hundred milliseconds.
  • Corner-aware features are critical.
    Adding progress_dist and corner_index helped the model understand where it is in the lap and dramatically improved convergence.
  • Smoothing is a superpower.
    Light rolling smoothing on labels turned noisy pedal traces into learnable patterns and stabilized training.
  • Visualization drives trust.
    Seeing true vs predicted traces overlaid, with per-corner color bands, immediately revealed both model strengths and data issues.
  • AI coaching needs context, not just numbers.
    RMSE is useful, but what drivers care about is: “In T7 I’m lifting too early and adding unnecessary steering corrections.” The per-corner diffs are a first step toward that language.

What's next for ApexPilot

  • More signals & richer coaching
    • Incorporate line information (GPS / racing line) and maybe yaw rate to relate actions to path, not just pedals.
    • Translate per-corner diffs into natural-language feedback like
      “Turn 2: later turn-in, hold a bit more brake, trust the front grip.”
  • Driver-specific models
    • Learn each driver’s style and compare them against the global expert, or against their own best laps.
  • Multi-track & live use
    • Generalize the pipeline to other tracks and car classes.
    • Explore running a lightweight version live for post-session debriefs at the circuit.
  • Training integration
    • Generate “AI ghost traces” that can be overlaid in sims or HUDs, letting drivers practice following the AI coach line and timing.
  • Open tools
    • Package the data pipeline + visualization as a reusable toolkit so other series/teams can plug in their telemetry and get an ApexPilot-style dashboard for their own drivers.

ApexPilot is still an early prototype, but it already shows how combining sequence models with clean telemetry and good visualization can turn raw race logs into actionable coaching for real drivers.

Share this project:

Updates