S(ai)mon

Inspiration

Simon's audience loves productivity, great design, and building systems. They want coaching to help them follow through on their goals, but traditional coaching is expensive and scheduling is a hassle. Generic productivity apps don't understand their unique context, values, and preferences.

We were inspired by Simon's vision: "Create a clean, minimal mobile app where users can browse, create, and share AI coaches, add personal context/values, and start chatting immediately. No complex setup, just guidance at the right moment."

The challenge was clear: build a coaching app that's as beautiful as it is functional, as simple as it is powerful, and as accessible as it is intelligent.

What It Does

Simon AI Coach is a minimalist iOS app that brings personalized AI coaching to your pocket. Here's what makes it special:

🤖 Browse & Create AI Coaches

• Curated Library: 12+ pre-built coaches for focus, planning, creativity, decision-making, wellness, business, and more
• Custom Creation: Design your own coaches with AI-generated avatars, personalized specialties, and coaching styles
• Save Favorites: Build your personal coach library with one-tap saving
• Smart Filtering: Find the perfect coach by specialty, style, or saved status

💬 Multi-Modal Chat Experience

• Text Chat: Natural conversation with real-time streaming responses
• Voice Input: Record voice messages for hands-free coaching
• Document Upload: Share PDFs, images, and documents for context-aware guidance
• Voice-Over Responses: Coaches respond with natural, human-like speech via ElevenLabs TTS
• Streaming: See responses as they're generated, word by word

⚡ Moments — Quick Actions

• Quick Templates: 6 pre-built prompts for common coaching needs (clarify next step, weekly review, etc.)
• Direct Chat: Ask anything and get automatically routed to the most appropriate coach
• Upcoming Events: View calendar events, reminders, and notifications created by your coaches
• Multi-Modal Input: Text, voice, photos, and documents — all in one place

🛠️ Intelligent Tools (9 Total)

Your coaches don't just talk — they take action:

Client Tools (iOS Integration):

• 📅 Calendar Events: Schedule meetings, time blocks, and recurring events
• ⏰ Reminders: Create task reminders with due dates and priorities
• 🔔 Notifications: Schedule push notifications for check-ins and nudges
• 📤 Share Export: Export plans and reviews as PDF or Markdown

Server Tools (Backend):

• 🧠 Memory: Save and recall your preferences, goals, commitments, and values
• 📋 Plans: Create structured plans with objectives, milestones (max 8), and next actions (max 12)
• 🔄 Plan Updates: Mark actions complete, add milestones, track progress
• 📆 Check-ins: Schedule recurring coaching sessions (daily, weekly, monthly)
• 🔍 Web Search: Access current information when needed (future feature)

📚 Library & History

• Past Conversations: Browse all coaching sessions with search
• Plans & Actions: View and manage your structured plans
• Events: Track calendar events, reminders, and notifications
• Context Vault: Your preferences, goals, and commitments in one place

💎 Premium Features (RevenueCat)

• Free Tier: 3 messages per session with any coach
• Pro Subscription: Unlimited messages, publish custom coaches, advanced features
• Flexible Plans: Weekly ($9.99), Monthly ($29.99), Yearly ($79.99)
• Restore Purchases: Seamless cross-device subscription sync

How We Built It

Tech Stack

iOS Client (Native Swift):

• SwiftUI: Modern, declarative UI framework
• AVFoundation: Audio recording and playback
• Firebase SDK: Authentication, Firestore, Storage
• RevenueCat SDK: Subscription management and paywalls
• Server-Sent Events: Real-time streaming responses

Backend (Go):

• Gin: Fast HTTP web framework
• Cloud Firestore: NoSQL database for users, coaches, sessions, messages
• Firebase Admin SDK: Authentication and authorization
• Vertex AI: Google's Gemini 3 Flash model for LLM
• ElevenLabs API: Text-to-speech with natural voices
• Server-Sent Events: Streaming responses to client

Infrastructure:

• Firebase Authentication: Google Sign-In
• Cloud Firestore: Real-time database
• Firebase Storage: File uploads (documents, images)
• Google Cloud Run: Serverless backend deployment
• Vertex AI: Gemini 3 Flash for conversations

Architecture

We built a multi-agent orchestration pipeline that handles:

1. Router Agent: Classifies user intent and determines if structured outputs are needed
2. Context Builder: Fetches coach configuration (CoachSpec), user context, and active plans
3. Coach Agent: Generates streaming responses using Gemini 3 Flash with coach-specific prompts
4. Planner Agent: Extracts structured data (plans, actions, reviews) from coach responses
5. Safety Filter: Validates outputs against coach policies
6. Memory Agent: Updates user context vault in background

Streaming Flow:

User Message → Backend → Gemini 3 Flash → Stream Text → Client
                    ↓
              ElevenLabs TTS → Stream Audio → Client

Both text and audio stream simultaneously for the lowest latency experience.

Key Features Implementation

Voice-Over Streaming:

• Backend creates ElevenLabs WebSocket session
• Streams text to both client and ElevenLabs
• ElevenLabs returns MP3 audio chunks
• Client decodes and plays audio in real-time with AVAudioEngine
• Latency: ~200-500ms from text generation to audio playback

Coach Creation:

• User provides minimal input (name, promise, specialty, style)
• Backend generates complete CoachSpec with:
  • Identity (role, expertise, personality)
  • System prompt (specialty and style-specific)
  • Conversation style (tone, verbosity, formatting)
  • Capabilities (tools, frameworks, protocols)
  • Guardrails (refusals, privacy, safety)
• AI-generated avatars (currently placeholder SVGs, Gemini 3 Image Generation planned)

Intelligent Tools:

• Client tools require user confirmation (iOS permission sheets)
• Server tools execute automatically (no confirmation needed)
• Tool requests detected via conversation analysis
• Structured outputs (plans, actions) extracted via secondary Gemini call
• All tool executions saved to Firestore for history

Subscription System:

• RevenueCat SDK for cross-platform subscription management
• Message limit enforcement (3 per session for free users)
• Paywall appears when limit reached
• Restore purchases for cross-device sync
• Settings screen shows subscription status and benefits

Challenges We Faced

1. Real-Time Voice-Over Streaming

Challenge: Streaming text and audio simultaneously with low latency.

Solution:

• Implemented double-stream pipeline: Gemini → Client (text) + Gemini → ElevenLabs → Client (audio)
• Used ElevenLabs WebSocket API with optimistic generation
• Built custom AVAudioEngine-based player for MP3 chunks
• Achieved ~200-500ms latency

Learning: Real-time audio streaming requires careful buffer management and error handling.

2. Multi-Agent Orchestration

Challenge: Coordinating multiple AI agents (router, coach, planner) while maintaining streaming.

Solution:

• Two-phase approach: Stream conversation first, extract structure second
• Router agent determines if planner is needed
• Planner runs asynchronously after streaming completes
• Structured outputs (plans, actions) sent as separate SSE events

Learning: Streaming and structured outputs don't mix well — better to separate concerns.

3. Tool Detection & Execution

Challenge: Detecting when coaches want to use tools and extracting parameters.

Current Solution:

• Heuristic-based detection (keyword matching)
• Empty tool payloads (parameters not extracted)
• User confirmation required for all client tools

Future Solution (planned):

• Gemini function calling for structured tool requests
• Automatic parameter extraction
• Smarter tool routing

Learning: LLM function calling is essential for reliable tool use — prompt engineering alone isn't enough.

4. Message Persistence with Voice-Over

Challenge: Messages weren't being saved when voice-over was enabled.

Solution:

• Added message tracking to voice streaming handler
• Save assistant message when stream completes
• Mirror the same saving pattern as regular streaming

Learning: Every code path needs to handle persistence — easy to miss edge cases.

5. Audio Indicator Timing

Challenge: Audio playing indicator appeared at wrong times or didn't disappear.

Solution:

• Simplified to two-state logic: first chunk → show, final marker → hide
• Removed complex buffer counting
• Immediate state updates instead of delayed

Learning: Simpler state management is more reliable than complex tracking.

6. Subscription Integration

Challenge: Integrating RevenueCat, enforcing limits, showing paywalls at right time.

Solution:

• Per-session message counting (not global)
• Paywall as full-screen overlay (not sheet)
• Success/error feedback with auto-dismiss
• Settings screen with subscription status

Learning: Subscription UX is critical — users need clear value proposition and easy upgrade path.

What We Learned

Technical Learnings

1. Streaming is Hard: Real-time streaming requires careful error handling, buffer management, and state synchronization. But the UX improvement is worth it.

2. Multi-Modal is the Future: Users love having options — text, voice, documents. Supporting all input types makes the app accessible to everyone.

3. LLM Orchestration: Multi-agent systems are powerful but complex. Clear separation of concerns (routing, generation, extraction) is essential.

4. Mobile-First Design: Building for mobile forces you to prioritize. Every feature must justify its screen real estate.

5. Subscriptions are Complex: RevenueCat makes it easier, but there are still many edge cases (restore, cross-device, family sharing, etc.).

Product Learnings

1. Minimalism Wins: Users don't want complexity. They want to open the app, pick a coach, and start chatting. Every extra step is friction.

2. Voice is Powerful: Voice-over responses make coaching feel personal and human. It's the difference between reading advice and receiving it.

3. Tools Create Value: Coaches that take action (create plans, schedule events) are more valuable than those that just talk.

4. Context is King: Coaches that remember your preferences, goals, and commitments provide better guidance over time.

5. Free + Premium Works: Free tier lets users try the app risk-free. Premium tier provides clear value (unlimited messaging).

Design Learnings

1. Consistency Matters: Using the same patterns across the app (attachment UI, voice recording, etc.) makes it intuitive.

2. Feedback is Essential: Loading states, success messages, error alerts — users need to know what's happening.

3. Accessibility First: VoiceOver support, dynamic type, high contrast — these aren't optional.

4. Animation Adds Polish: Smooth transitions, subtle motion, haptic feedback — these details make the app feel premium.

5. Dark Mode is Expected: Users expect apps to respect their system theme preference.

What's Next

Short-Term (Next 4 Weeks)

1. Gemini Function Calling: Replace heuristic tool detection with structured function calls
2. Document Processing: Actually process uploaded documents (currently just stored)
3. Coach Sharing: Publish custom coaches to community marketplace
4. Avatar Generation: Integrate Gemini 3 Image Generation for real AI avatars
5. Analytics: Track usage, engagement, conversion metrics

Medium-Term (Next 3 Months)

1. Web App: Build web version for desktop users
2. Android App: Port to Android for broader reach
3. Advanced Tools: Email sending, SMS, calendar event updates
4. Coach Templates: Pre-built templates for common use cases
5. Team Features: Share coaches within organizations

Long-Term (Next 6-12 Months)

1. MCP Integration: Support Model Context Protocol for external tools
2. Agent-to-Agent: Coaches can consult other coaches
3. Custom Frameworks: Users can create and share coaching frameworks
4. Voice Cloning: Clone your own voice for personalized responses
5. Multi-Language: Support for 10+ languages

Built With

• Swift — iOS client
• SwiftUI — UI framework
• Go — Backend server
• Gin — HTTP framework
• Firebase — Authentication, database, storage
• Firestore — NoSQL database
• Vertex AI — Gemini 3 Flash LLM
• ElevenLabs — Text-to-speech
• RevenueCat — Subscription management
• Google Cloud Run — Serverless deployment
• AVFoundation — Audio recording/playback
• Server-Sent Events — Real-time streaming

---

Built With

• avfoundation
• elevenlabs
• firebase
• firestore
• gin
• go
• google-cloud-run
• revenuecat
• sse
• swift
• swiftui
• vertex-ai