Avatar Podcast Studio

Talk ideas into life with an AI host that listens, speaks, uses your knowledge, and creates live visual explanations while the conversation continues.

Comment 1

Inspiration

Voice assistants are natural to talk to, but their ideas disappear as soon as they are spoken. Presentation and diagram tools preserve ideas visually, but they interrupt the conversation and require manual design work.

I wanted to create a workspace where speaking, understanding, and visual thinking happen together. The result is Avatar Podcast Studio: an AI host that can listen, respond, use trusted knowledge, and turn a spoken request into a visual while the conversation continues.

What it does

Avatar Podcast Studio combines three experiences in one responsive web application:

  • A natural, low-latency voice conversation with an animated AI host.
  • Selectable plain-text knowledge bases that provide trusted context.
  • A live canvas that turns spoken requests into diagrams, roadmaps, comparisons, charts, and concept maps.

Users can select an avatar and voice, attach reference material, and go live. The avatar listens, reacts, speaks, and lip-syncs with the generated audio.

When the user asks for a visual, the avatar sends an asynchronous tool request to the canvas. The conversation does not stop while the visual is being created. When it is ready, the host confirms completion and the result appears as interactive HTML, CSS, and SVG.

Users can also generate custom avatar pose sheets with consistent listening, blinking, and speaking states.

How I built it

The frontend was built with React, TypeScript, and Vite. The backend uses Node.js and Express and serves both the API and the production frontend.

OpenAI Realtime provides the live voice experience through WebRTC. The API key remains on the server, and the browser receives only the temporary connection information required to establish the session.

GPT-5.6 Terra powers the visual canvas. It receives the visual request and streams concise, self-contained HTML, CSS, and SVG. The client safely renders that response inside the canvas.

GPT Image generates and edits avatar images into consistent sprite sheets. Audio levels and conversation events control the listening, blinking, and mouth poses.

SQLite stores avatars, knowledge bases, usage information, and application data. A persistent Docker volume protects this information across deployments.

The server also includes authentication, per-session and per-IP quotas, global concurrency limits, and feature switches to protect paid OpenAI capabilities.

How I used Codex and GPT-5.6

Avatar Podcast Studio began as an earlier experimental studio prototype. During OpenAI Build Week, I used Codex to transform it into the current OpenAI-only product.

Codex helped me audit the original architecture, remove legacy provider paths, migrate the voice and image workflows to OpenAI, implement GPT-5.6 streaming canvas generation, and coordinate changes across the React frontend and Express backend.

It also helped reproduce and resolve difficult runtime problems involving WebRTC events, microphone turn detection, occupied ports, stale services, missing avatar assets, blank routes, avatar selection, lip sync, asynchronous tool execution, responsive mobile layouts, SQLite persistence, Docker packaging, and Easypanel deployment.

I made the core product decisions: keeping the conversation active while visuals render, making knowledge selectable, using an OpenAI-only architecture, removing the legacy image gallery and direct canvas prompt box, and prioritizing a coherent voice-first experience.

GPT-5.6 Terra is part of the shipped application, not only the development workflow. It powers the live HTML, CSS, and SVG canvas while OpenAI Realtime handles natural voice and GPT Image creates custom avatars.

Challenges

The most important challenge was making visual generation asynchronous. A diagram can take longer than a voice response, so the avatar could not remain blocked while waiting for the canvas. I separated the tool request from the conversation lifecycle so the host can acknowledge the request, keep talking, and report completion later.

Another challenge was creating convincing avatar presence without generating video. I used consistent sprite-sheet poses and connected them to realtime audio levels and conversation states.

Deployment also required careful handling of same-origin API routes, reverse-proxy settings, protected credentials, and persistent SQLite storage.

Accomplishments

I am proud that the project feels like a complete product rather than a disconnected collection of AI demos.

The voice, avatar, knowledge, and visual systems work together in a single experience. Users do not need to stop speaking, open another tool, or write a separate prompt to make an idea visible.

The application is responsive, deployable as one Docker service, protected against uncontrolled API usage, and usable by judges through a hosted demonstration.

What I learned

I learned that the best multimodal experiences depend as much on orchestration as on individual models. Voice latency, tool execution, visual consistency, avatar state, context management, and user feedback must all be designed as one timeline.

I also learned that asynchronous tools make an AI conversation feel significantly more natural. The assistant should remain present and responsive even when another part of the experience is still working.

What's next

Next, I would add reusable visual design templates, collaborative sessions, richer knowledge retrieval, exportable canvas artifacts, additional avatar animation styles, and tools for recording and publishing complete visual conversations.

Built With

Share this project:

Updates