InteractEngine

Build trust, unlock secrets: an AI-powered character interaction engine where relationships drive narrative progression

Comment

💡 Inspiration

Even though AI, chatbots, and human-computer interactions are at an all-time high, I noticed a critical gap: dynamic character interaction. While AI excels at answering questions, I had never seen it used to create characters with evolving emotions and knowledge that unlocks progressively through relationship building.

This sparked the question: Why not use AI to create virtual characters with genuine human qualities? Characters that remember how you treat them, adapt their behavior based on trust, and gradually reveal secrets as your relationship deepens.

📚 What I Learned

This project pushed me far beyond my comfort zone:

  • TypeScript Mastery: Learned TypeScript, including advanced features like Zod schemas, type inference, and interface design
  • Rapid Development: Built a complex multi-component system solo in a compressed hackathon timeframe
  • Systems Integration: Successfully integrated multiple technologies (LLMs, voice synthesis, state management, React UI) into a seamless experience
  • Character Psychology: Discovered how to make virtual characters feel alive through emotion modeling, memory systems, and conditional knowledge
  • Performance Optimization: Learned to balance AI quality with response speed through model selection and async processing
  • API Integration: Navigated complex API authentication, rate limiting, and error handling across multiple services

🛠️ How I Built It

Phase 1: Core Engine (Foundation First)

I started by building the character interaction engine using deterministic methods—no AI initially. This included:

  • Character state management (trust, anger, fear)
  • Conditional knowledge system with unlock conditions
  • Memory events and relationship tracking
  • State update functions based on input classification

This allowed me to validate the core logic before adding AI complexity.

Phase 2: LLM Integration (Dynamic Intelligence)

Once the engine was functional, I integrated AI:

  • Input Analysis: LLM classifies player tone and intent (polite, aggressive, reassuring, etc.)
  • Response Generation: Claude 3.5 Haiku generates contextual, character-appropriate dialogue
  • Emotion Detection: AI suggests appropriate emotions based on character state and conversation

Phase 3: Visual Interface (Bringing It to Life)

Built a clean React UI featuring:

  • Turn-based dialogue system (player → character → player)
  • Character sprites with emotion states
  • Real-time character state panel showing trust/anger/fear meters

Phase 4: Voice & Polish (The Final Touch)

Added ElevenLabs voice synthesis:

  • Emotion-aware voice modulation
  • Mouth animation synchronized with audio playback
  • Background

🚧 Challenges I Faced

Challenge 1: Processing Time

Problem: Initial response times were 5-8 seconds (unacceptable for interactive gameplay).

Solution:

  • Switched from local Ollama (llama3.1:8b) to cloud-based Claude 3.5 Haiku
  • Implemented async voice generation (non-blocking)
  • Reduced response time from 5-8s to 1-2 seconds

Challenge 2: API Complexity & Restrictions

Problem: Multiple APIs with different authentication methods, rate limits, and response formats.

Solution:

  • Built modular LLM provider system supporting multiple backends
  • Implemented graceful fallbacks when APIs fail
  • Added comprehensive error handling and retry logic
  • Created abstraction layer for easy provider swapping

Challenge 3: Balancing AI Freedom vs. Control

Problem: Needed AI to be creative but stay within character constraints and respect the state system.

Solution:

  • Designed prompt engineering with clear sections (personality, state, allowed knowledge, forbidden topics)
  • Used Zod schemas for response validation
  • Implemented deterministic state updates (AI generates dialogue, but doesn't control character state)

🎯 Key Achievements

  • Built a fully functional character interaction engine in 24 hours
  • Integrated 3 different AI services (input analysis, dialogue, voice)
  • Created a polished UI with real-time state visualization
  • Designed a reusable, modular system that can support any character
  • Achieved sub-second response times with high-quality AI

🤖 AI Assistance Disclosure

This project was developed with AI assistance (Claude by Anthropic) used in the following ways:

What AI Helped With:

  • Code Architecture Guidance: Suggesting TypeScript patterns, Zod schemas, and modular design structures
  • API Integration: Providing implementation examples for OpenRouter, ElevenLabs, and error handling
  • Debugging: Identifying issues with secret unlocking, prompt engineering, and async processing
  • Documentation: Helping write documentation and technical explanations
  • Problem-Solving: Brainstorming solutions for performance optimization and UI/UX challenges

The result is InteractEngine—a system where conversations have consequences, relationships evolve naturally, and secrets unlock through earned trust.

Built With

Share this project:

Updates