Inspiration

Fast-moving Twitch chat makes it easy for streamers to miss repeated questions and audience ideas while they’re focused on playing and entertaining. We wanted to make chat easier to act on without taking control away from the streamer.

What it does

StreamPilot AI turns chat into a prioritized Smart Queue, grouping similar questions so streamers can spot what viewers are asking about. It can draft grounded replies and suggest audience polls or interactive challenges. Streamers review and choose whether to feature a question, send a reply, or approve and launch a challenge. An OBS browser overlay can show challenge voting and a featured question. After a stream, the app can summarize community activity.

How we built it

We built a React, TypeScript, and Vite dashboard backed by a FastAPI service, SQLAlchemy, and SQLite. Twitch OAuth connects a broadcaster account, while EventSub delivers chat and stream events. Deterministic filters and fuzzy matching identify and group questions; bounded Gemini requests handle selected tasks such as grouping remaining questions and drafting structured replies. The backend validates model output, scopes data to the broadcaster, and keeps Twitch credentials encrypted. WebSockets update the dashboard.

Challenges we ran into

Live chat is noisy, and similar questions can be phrased in very different ways. We combined deterministic filtering and fuzzy matching with optional, batched AI grouping, and added safeguards for malformed output, provider failures, duplicate events, late votes, and moderated messages. Another challenge was preserving streamer control: AI-generated content stays a draft, and consequential actions require an explicit streamer action. We also had to distinguish automated test results from actual Twitch and Gemini behavior.

Accomplishments that we're proud of

We brought chat intake, question triage, reply drafting, audience interaction, and stream recaps into one streamer workflow. The Smart Queue can explain its priority, remove moderated content from future context, and keep private viewer messages off the public overlay. Reply drafts can be edited or discarded, and only an explicit send action posts to chat. Moderation support is flag-only; it does not automatically punish viewers.

What we learned

We learned that a useful AI feature needs clear boundaries around what it can do. Structured output validation, untrusted-input handling, broadcaster-scoped data, and explicit approvals are as important as the model call itself. We also learned to report verification precisely: passing mocked tests or enabling an EventSub subscription does not prove a complete live integration.

What's next for StreamPilot AI

Next, we want to resolve the queue-count and grouping issues recorded during the live rehearsal, then repeat controlled checks for chat delivery, moderation events, reply sending, polls, and stream recaps. We also plan to improve evaluation with a broader set of real-world phrasing and strengthen deployment and multi-instance readiness.

Built With

Share this project:

Updates

Submission history