Spectal/SpectAI

Realtime AI coaching system for gaming

Comment

Inspiration

  • The Problem: Improving at competitive games is frustrating when you don't know what mistakes you're making mid-match.
  • The Cost Barrier: Real coaching is too expensive for most casual players, turning the game into a stressful grind instead of a fun hobby.
  • The Goal: We wanted to make personalized coaching accessible to everyone, removing the need to spend hours reviewing old gameplay footage just to find one mistake.

What it does

  • Live Coaching Overlay: A transparent, in-game window that watches your match and gives you immediate, actionable advice based on what is happening on your screen.
  • Built-In Playbook: Gives you access to high-level tactics and strategies on demand, so you don't have to study the game outside of playing it.
  • Post-Match Dashboard: Automatically tracks your performance to build a skill radar, then generates custom 3D aim training drills based specifically on your weak points.

How we built it

  • Live Data Feed: We use OpenCV to read the screen and the local Valorant API to pull health, economy, and map data directly into Gemini.
  • Voice Integration: Players can ask the coach questions mid-game using a standard push-to-talk key, powered by speech-to-text and ElevenLabs audio.
  • Lineup Tool: An integrated overlay automatically pulls up character-specific map setups, so players never have to minimize the game to search for a YouTube guide.

Challenges we ran into

  • Connection Drops: The connection to the AI would drop during long matches, so we had to build a custom script to automatically reconnect without the player noticing.
  • Data Limits: We had to figure out how to compress dense visual and game data into a small enough package for the AI to process quickly without losing context.
  • Training Quality: Generating aim drills from scratch was resulting in flat, boring maps. We solved this by giving the AI modular map pieces it could snap together.

Accomplishments that we're proud of

  • Speed: We heavily optimized the code with multithreading so the AI speaks to the player with almost zero delay.
  • Accuracy: Combining screen-reading with actual game data means the AI gives genuinely helpful advice instead of making wild guesses about what is happening.

What we learned

  • User Experience: We learned how to build an interface that sits over a competitive game without being distracting or annoying to the player.
  • Intentional Design: We realized that keeping the tool focused strictly on helping the player improve was much better than adding unnecessary features.

What's next for Spectal/SpectAI

  • Speed Upgrades: We plan to switch to a smaller, faster AI model to cut response times down even further.
  • Wider Access: We want to keep building tools that let any player, regardless of their free time or budget, get better at the games they love.

Built With

Share this project:

Updates