VALORANT Decision Support AI

Inspiration

Competitive VALORANT rounds are intense and fast-paced, and even professional IGLs (In-Game Leaders) struggle to make optimal decisions under pressure. We wanted to build an AI that could support live decision-making and provide actionable tactical options in real time.

What it does

SkyFinal AI Coach is a real-time decision support system for VALORANT teams. It:

Monitors live game data through the GRID API and screen captures.
Detects in-game events like kills, deaths, and player positions.
Provides exactly two tactical options for the IGL to choose from during live rounds.
Allows hands-free interaction using speech-to-text and responds via high-quality voice output.

How we built it

Language & Framework: Python 3.12+ with LangChain and Ollama LLM (llama3.2:1b).
Vision Processing: OpenCV and mss to capture and interpret game screens in real time.
Speech Interaction: Whisper for STT and Kokoro ONNX for TTS to provide natural voice responses.
Data Integration: GRID GraphQL API for live player states, inventory, and tactical events.
Agents:
- Brain: Routes live queries to the Mid-Game Agent.
- Mid-Game Agent: Generates two actionable tactical options based on current round data.
- Data Agent: Combines API and visual data for accurate context.
- VLM: Detects events autonomously from the screen.

Challenges we ran into

Latency: Ensuring tactical advice was delivered in time for in-round decisions required careful optimization of the data and inference pipeline.
Data Fusion: Aligning GRID API data with VLM visual recognition to produce accurate recommendations was complex.
Limited Context: The LLM has a small context window, making it tricky to process multiple round states while keeping advice relevant.

Accomplishments that we're proud of

Built a fully functioning Mid-Game decision support system that can monitor live rounds and provide actionable options.
Successfully integrated real-time API data, computer vision, and speech I/O into a cohesive multi-agent system.
Achieved a hands-free interface that allows IGLs to receive tactical advice without breaking focus.

What we learned

Real-time tactical AI requires balancing speed, accuracy, and cognitive simplicity.
Multi-modal inputs (API + vision) must be reconciled carefully to produce reliable recommendations.
Modular agent design greatly simplifies testing, maintenance, and future improvements.

What's next for SkyFinal AI Coach

Reduce latency further by implementing asynchronous event handling.
Improve Mid-Game Agent reasoning with more sophisticated models and richer context.
Enhance vision models to detect more nuanced in-game events for better tactical recommendations.