AI Coach

Developer-first AI coaching and best practices. Real-time guidance that prevents failures and evolves through feedback, powered by Gambit-based self-learning on Gemini 2.0 Flash, Google ADK and Weave.

Comment

Full Story

Inspiration

Walk into any tech company and you'll find dozens of AI tools. But look closer—almost all of them are built for executives. Adoption dashboards. Compliance metrics. ROI calculators. Top-down rollout strategies.

Who's helping the developers actually get better at using AI?

I built AI Coach because I was frustrated. After months using Claude Code, I was still making the same mistakes: vague prompts that wasted time, context bloat that killed productivity, and preventable errors that cascaded into multi-layer debugging nightmares.

The tools existed to track my usage, monitor my compliance, and measure my "AI adoption." But nothing was teaching me to be better at AI-assisted coding. I wanted a system that would coach me in real-time, learn from my feedback, and help me develop genuine AI coding skills—not just measure my adoption for someone else's dashboard.

What it does

AI Coach is a developer-first coaching system that makes you better while you work—not through separate training sessions or workshops, but through real-time guidance integrated directly into Claude Code.

Here's how it works:

  1. Real-Time Coaching - Before you waste 30 minutes on a vague prompt, AI Coach catches it. "Which component specifically? What's the expected vs actual behavior?" It learns what makes prompts effective and helps you write better ones.

  2. Context Health Monitoring - Your context window is like a gas tank. Most developers don't notice it's running low until they're stranded. AI Coach monitors context health in real-time and proactively suggests /compact before productivity tanks.

  3. Continuous Learning - Every intervention can be rated 1-3. Rate something helpful? The system learns to do more of it. Rate something annoying? It backs off. The coaching adapts to your style through reinforcement learning.

  4. Session Quality Tracking - The Session Quality Index measures your improvement across prompt quality, test success rate, time to green, and context management. You can see yourself getting better over time.

  5. Real-Time Dashboard - Watch your context health, failures prevented, and coaching insights update every 5 seconds with live data from W&B Weave.

How we built it

Architecture: Claude Code Hooks API → Google ADK Agent → W&B Weave → Next.js Dashboard

Google Gemini 2.0 Flash + Agent Development Kit powers the intelligence. The ADK provides the framework for building self-improving agents, while Gemini 2.0 Flash handles the actual decision-making. Each prompt gets analyzed for vagueness, scope, context health, and error patterns. The agent decides whether to intervene based on learned weights stored in a decision tree.

W&B Weave provides complete observability. Every coaching decision, every intervention, every user rating—all logged in real-time using both Python and TypeScript SDKs. I can see exactly what the AI is learning, which interventions work, and how the decision tree weights evolve over time. The Weave traces show the complete decision flow from prompt analysis to intervention selection.

Claude Code Hooks API integrates the coaching system directly into the development workflow. UserPromptSubmit hooks catch prompts before Claude sees them, and PostToolUse hooks verify outcomes with automated testing.

Daytona provides sandboxed test execution for PostToolUse verification. When Claude makes changes, the hooks automatically run tests in isolated Daytona environments to verify improvements without risking the main development environment. This enables safe, automated outcome verification.

Next.js 14 Dashboard pulls real-time data from W&B Weave every 5 seconds, displaying context health, Session Quality Index, AI insights, and intervention logs. Built with shadcn/ui and Tailwind CSS for a clean, Duolingo-inspired "Forest on Porcelain" design system.

W&B Inference (planned deployment) will scale the trained model for production use. Multi-armed bandit algorithms optimize intervention selection in milliseconds.

The result? A dual learning loop: The system improves both the developer through coaching AND itself through reinforcement learning. Gemini powers the intelligence, Weave tracks the learning, Inference scales the deployment.

Challenges we ran into

Hook Integration Complexity: Getting Claude Code hooks to work reliably was tricky. UserPromptSubmit hooks need to be fast (< 2 seconds) or they block the user. I had to optimize the Gemini analysis pipeline and implement caching for feature extraction to keep latency low.

Balancing Intervention Frequency: Too many interventions feel naggy, too few and the system doesn't help. I built a multi-armed bandit algorithm to learn the right intervention frequency per user, adjusting based on ratings. The challenge was tuning the exploration vs exploitation tradeoff.

Real-Time Data Synchronization: The dashboard needs to show live data from W&B Weave, but API rate limits and latency made this challenging. I implemented smart polling with 5-second intervals and client-side caching to balance freshness with performance.

Decision Tree Learning: Getting the reinforcement learning right was hard. I had to experiment with different weight adjustment formulas and learning rates. Too aggressive and the system oscillates, too conservative and it doesn't learn. The current formula uses a dampened gradient that converges smoothly.

Testing the Agent: How do you test a system that learns from user feedback? I built comprehensive test scenarios with mocked rating sequences to verify the agent learns correctly. Achieving 100% test coverage for a learning system required creative thinking about edge cases.

Accomplishments that we're proud of

100% Test Coverage: The coaching agent has comprehensive test coverage across 5 major scenarios (vague prompts, context health, large scope, error patterns, learning validation). For a system that learns from user feedback, achieving complete test coverage required creative test design with mocked rating sequences.

Real Dual Learning Loop: This isn't just observability—it's genuine reinforcement learning. The decision tree adjusts weights based on user ratings, and I can prove it works. Run npm run weights before and after rating an intervention, and you'll see the weights change. The system learns.

Production-Ready Integration: The Claude Code hooks work reliably in real development workflows. UserPromptSubmit interventions complete in < 1 second, and PostToolUse verification runs in sandboxed environments. This isn't a prototype—it's production-grade integration.

Measurable Improvement: After 369 decisions across development sessions:

  • 146 interventions - Caught issues before they became problems
  • 1.43/3 average rating - Users find the coaching helpful
  • 76% context health - Proactive management prevents productivity collapse
  • 90 Session Quality Index - Measurable improvement in AI-assisted coding

Developer-First Design: Successfully positioned as on-the-job training for developers, not another exec dashboard. The About page and demo script clearly articulate this unique positioning.

Real-Time Observability: Every decision, intervention, and rating logged to W&B Weave in real-time. The dashboard updates every 5 seconds with live data. You can watch the system learn as you use it.

What we learned

Google ADK is Powerful: The Agent Development Kit made building a self-improving agent much easier than building from scratch. The framework handles tool integration, state management, and execution flow. I could focus on the decision logic and learning algorithms instead of infrastructure.

W&B Weave is Essential for AI Observability: Without Weave traces, debugging the agent would have been impossible. Being able to see every decision, feature extraction, and weight adjustment in real-time made iterative development fast. The Python and TypeScript SDKs worked seamlessly together.

User Feedback is Gold: The rating system (1-3) provides clear signal for learning. Even simple feedback drives meaningful improvement. The average rating of 1.43/3 tells me there's room to improve, and the weight adjustments show the system is learning from that signal.

Developer-First Positioning Matters: Framing this as on-the-job training for developers (not exec adoption metrics) resonated immediately. The positioning makes the value clear and differentiates from existing tools.

Hook Performance is Critical: UserPromptSubmit hooks block the user if they're slow. Keeping latency under 1 second required aggressive optimization of the Gemini analysis pipeline. Every millisecond matters in developer tools.

Testing Learning Systems is Hard: Traditional test approaches don't work for systems that learn. I had to design test scenarios that verify learning behavior over time, not just correctness at a point in time. Mocked rating sequences proved essential.

What's next for AI Coach

The MVP is complete and working. But the vision is bigger:

Immediate Next Steps:

  • Deploy trained models to W&B Inference for production-scale intervention selection
  • Add hallucination detection coaching (catch when Claude makes up APIs or facts)
  • Expand coaching patterns (tool usage, test quality, code review prompts)
  • Multi-language support beyond English

Medium-Term Vision:

  • Team collaboration features with shared learning across organizations
  • Curriculum-based training modes (beginner → intermediate → expert paths)
  • Custom coaching rules per team (enforce org-specific best practices)
  • Integration with GitHub Copilot and other AI coding tools

Long-Term Vision:

  • Build the Duolingo of AI-assisted coding—gamified skill development with lessons and achievements
  • Competitive challenges with leaderboards (fastest to green, highest SQI)
  • Cross-IDE support (VS Code, JetBrains, Cursor)
  • Enterprise features (SSO, team analytics, compliance reporting)

The Ultimate Goal: Make every developer better at working with AI, one coached prompt at a time. Transform AI coding from trial-and-error into systematic skill development with measurable progress.

Why This Matters

AI coding assistants are powerful, but most developers are self-taught through trial and error. We learn bad habits, waste time on preventable mistakes, and never get systematic feedback on what makes prompts effective.

AI Coach changes that. It's the first system designed specifically to make developers better at AI-assisted coding through on-the-job training, measurable progress tracking, and continuous learning.

This isn't about adoption metrics for management. It's about making you better at your craft, one coached prompt at a time.

Tech Stack Summary

  • Google Gemini 2.0 Flash - Intelligent decision-making and feature extraction
  • Google Agent Development Kit (ADK) - Self-improving agent framework
  • W&B Weave - Real-time observability and learning tracking
  • W&B Inference - Production deployment and optimization
  • Claude Code Hooks API - UserPromptSubmit, PostToolUse integration
  • Daytona - Sandboxed test execution for safe automated verification
  • Next.js 14 - Real-time dashboard with App Router
  • Node.js + TypeScript - Backend processing and agent logic

Links

Built with ❤️ by Josh & The Pack for developers who want to level up their AI promptcraft.

Stop debugging. Start coaching.

Built With

Share this project:

Updates

Submission history