Ctrl+Alt+Drive

AI Racing Coach: Race replay platform using real Toyota GR Cup telemetry. Drivers, teams, and fans get AI-powered racing insights impossible for human coaches to deliver at scale and cost.

Comment

Inspiration

Driver Development Simulator with AI Racing Coach We are developing an innovative race replay platform that leverages real Toyota GR Cup telemetry data. Drivers, team members, amateur racers, and fans can follow a driver in a race and get racing performance recommendations from an AI Racing Coach. This will allow users to fully immerse themselves in the race as it really happened. The AI Racing Coach takes it to a new level where its not practical for human instructors to analyze so much data and race context. This produces new and unique insights that were not possible before.

What it does

The platform features true GPS traces of any car the user wishes to follow in a particular race. Users can use familiar video controls to playback the race, jump to different laps, or pause at any moment in time. All the while the actual telemetry data is displayed as it happened on that day at that time. The AI Racing Coach has extensive awareness of the selected car's telemetry data, its position relative to other cars, laps times, weather, and more. With highly granular data available to the AI Racing Coach, it is able to answer direct questions about driving performance and specific improvements that the driver can make to improve. The AI Racing Coach can also create full PDF reports for common recommendations such as "Compare my lap performance to my best lap", or "Compare my lap performance to the winner's best lap".

How we built it

  • Flask & Flask-CORS for web API - An API server loads up the full telemetry data as well as the other lap timing and weather data provided
  • Pandas & NumPy for data processing - Aggregations and comparisons can be made in memory based on the feature
  • Boto3 for AWS integration - Make calls to Amazon Bedrock to support the AI Racing Agent
  • Strands Agents for AI Race Coaching - This is the AI Racing Agent that can reason, plan, and invoke a series of custom tools to answer questions
  • D3.js for frontend visualization
  • jsPDF for PDF report generation ## Challenges we ran into Initially we tried loading telemetry data into browser memory for speed, but found this was flawed: You had to pre-process all the telemetry files to gather a sampled set of data The sampled data was not precise enough to support moment-by-moment telemetry playback ## Accomplishments that we're proud of API server architecture that provides full telemetry data to the UI when needed, including a smoothing capability for realtime playback of an entire race Overlay of GPS trace onto a track map in the correct orientation API Racing Coach that has extensive awareness: Where is the car cursor at any moment in time and how does that relate to the turns in the track What lap is the car on and how do the lap times compare for this car, and also how do they compare to other cars What is the racing position of the car compared to the other cars at this moment What are the telemetry metrics for this moment and for any turn on the track How does the weather affect driving performance and track times ## What we learned Its difficult to solve the "too much telemetry data" problem. If you try to feed too much data to a large language model when asking questions, you will simply overwhelm the context window. Its tricky to create Agent tools that provide only the needed data and context it needs to answer the question and no more. ## What's next for Ctrl+Alt+Drive Visualize more that one car at a time on the race playback map. Overlay info on top of the GPS trace such as when the car changed gears, was braking, was on the throttle, top speed, etc. Offer 'overtaking' advice and visuals for positions battles

Built With

Share this project:

Updates