Inspiration

Twitch chat moves fast, and nothing on screen can tell you why it shows what it shows. I wanted viewers to have something that is theirs on stream, and a stream that can explain itself: every value traceable to the exact inputs and rule that produced it.

What it does

Viewers type !me in chat and each one gets their own graph, shown as a bubble on the stream. Each graph holds that viewer's Twitch data (profile, stream, channel, followers). The streamer's graph also receives local face-tracking data, and a rule recomputes live:

$$ \text{stream.focus} = (0.4 < \text{face.x} < 0.6) $$

When the face is centered it is true, off to the side false, and with no face undefined (never an invented false). explain() shows the exact input, for example face.x = 0.53. Same state, same answer.

The streamer can inspect any node, share it to the OBS overlay, and control OBS from the same app.

How we built it

  • Graph: this.me 4.2.1, a deterministic, reactive knowledge graph. Rules recompute only what depends on what changed.
  • Twitch: OAuth with token refresh, EventSub for chat, replies through Send Chat Message.
  • UI: the .GUI library for the cards, the inspector, the theme-linked overlay tint and the face-tracking hooks.
  • OBS: a transparent Browser source overlay, plus OBS WebSocket v5 controls on loopback.
  • Codex built the integration in one dedicated session: OAuth and EventSub, the per-viewer graph registry, the inspector with real explain(), face tracking into the graph, the overlay sync and the OBS controls.

Everything runs locally on the streamer's machine. No LLM at runtime, no AI keys.

Challenges

  • Keeping the data honest. Imported Twitch values are stored facts, so their explanations have no fake derivation, and missing tracking gives undefined instead of a made-up value.
  • Security on a local companion. Chat can join, leave, read and explain, but it can never run code, write rules or touch another viewer's graph. Write actions require the authenticated broadcaster, an exact Origin and CSRF checks.
  • Showing what the graph actually does. Storing data isn't enough to show it off. A live rule fed by a real camera made the recompute-and-explain loop visible.
  • Syncing the app and OBS without capturing the stream inside itself.

What we learned

A deterministic graph and an LLM do different jobs. A model can propose, but something has to compute and explain the state it describes. Making every value explainable also forced us to be strict about what is a fact, what is derived, and what is unknown.

Why an O(k) reactive graph matters

This demo is the first, small implementation. The kernel underneath is the point. In .me, every value is a node in a graph, and rules declare what depends on what. When an input changes, the kernel recomputes only the $k$ nodes that actually depend on it, not the whole graph: $$ \text{cost of a write} = O(k), \quad k \ll n $$ In a 242-node graph, changing one route recomputes 4 nodes; changing a shared policy recomputes all 242. The cost follows the real dependencies, not the size of the data. That changes what a stream can be:

  • Every viewer, every signal, one graph. Chat, camera, game state, sensors or polls can all become facts, and each rule updates only what it touches.
  • Explainable by construction. Every derived value carries its inputs and rule, so the overlay can always answer why.
  • Deterministic. The same state always gives the same answer, so a model can propose changes while the kernel computes and validates them.

What's next

  • Each user brings their own .me (with QR.me), so identity and keys live in their own graph rather than in the app.
  • More live inputs (game events, polls, sensors) feeding rules that viewers can read and explain.
  • LLM proposals validated by the kernel before they touch the stream.

Built With

  • face-tracking
  • knowledge-graph
  • oauth
  • obs-studio
  • obs-websocket
  • openai-codex
  • reactive-programming
  • twitch
  • twitch-eventsub
Share this project:

Updates

Submission history