-
-
The human selects the authoritative direction that Codex must follow—not a decorative approval button.
-
Codex pairs securely through a one-time code, with every requested capability reviewed and approved by a human.
-
A spacious HITL workspace expands context, evidence, safeguards, uncertainty, and consequences before a decision.
-
Codex continues from the approved human decision and turns it into an actionable, visible plan.
-
SceneBoard turns compressed AI shorthand into a clear, presentation-ready explanation people can understand at a glance.
-
Every board update becomes an immutable visual revision that reviewers can navigate and compare.
-
Codex renders a polished 2D scene whose final color direction was explicitly chosen by a human.
-
SceneBoard safely hosts interactive Three.js and WebGL experiences in an isolated artifact runtime.
Inspiration
Codex is excellent at building software, but long development sessions can be difficult for a person to follow. The meaning of the work becomes scattered across terminal output, source files, plans, logs, and internal references such as D1, D2, or ADR-014. A reviewer may have to open several Markdown files just to understand one recommendation, while an important human decision can disappear inside a scrolling transcript.
We built SceneBoard to give Codex a second surface: a visual, persistent, presentation-grade workspace where it can explain complex work clearly and people can make informed decisions.
Our guiding idea is simple:
SceneBoard is the whiteboard Codex can operate freely.
Codex builds. SceneBoard explains. Humans decide.
What it does
SceneBoard connects to Codex through a plugin and Model Context Protocol server. After a user approves a one-time pairing request and exact capabilities, Codex can create and operate boards directly from the CLI or IDE.
Instead of displaying only compressed AI shorthand, SceneBoard expands the context a person needs for the current decision. A reference such as D1 is shown together with the governing rule, why it exists, what it allows or forbids, and its consequence for the work at hand. Each scene is designed to stand on its own without requiring the viewer to reopen an earlier chat or supporting document.
Codex can use the medium that best explains the work:
- structured documents, tables, diagrams, and architecture flows;
- charts, Scalable Vector Graphics, and animated 2D Canvas scenes;
- interactive HTML prototypes;
- isolated WebGL and Three.js experiences; and
- spacious Human-in-the-Loop decision workspaces containing context, evidence, uncertainty, safeguards, options, and consequences.
Human choices are authoritative rather than decorative. Codex waits for the response, uses it to shape the next artifact, and SceneBoard preserves the before-and-after states as immutable visual revisions.
How we built it
SceneBoard is an Apache-2.0 TypeScript monorepo with four main layers:
- a Next.js web application for boards, approvals, decisions, and revision navigation;
- a NestJS API for authentication, email verification, pairing, capability grants, boards, Human-in-the-Loop interactions, and history;
- a SceneBoard Codex plugin and Model Context Protocol server; and
- shared protocol schemas, SDKs, rendering components, and an isolated artifact runtime.
The pairing flow uses a short-lived SB-... code. The user approves the requested scopes, lifecycle permissions, target board, and duration before credentials are issued. Expressive HTML, Canvas, SVG, WebGL, and Three.js content runs in a separate sandboxed artifact origin with a constrained capability bridge.
GPT-5.6 operates inside Codex as the planning, implementation, explanation, and visualization engine. Through SceneBoard's tools, it expands fragmented context, chooses an appropriate visual medium, requests meaningful human decisions, consumes the recorded response, and revises the board. SceneBoard does not require users to expose OpenAI credentials to the board service.
Codex also drove the development workflow itself. In the primary build thread, it helped refine the product concept, plan the architecture, implement the frontend, backend, plugin, and protocol packages, create regression tests, diagnose browser and deployment failures, run end-to-end quality assurance, design the demo, and consolidate the project into a public monorepo. The human made the defining product and safety decisions—including the one-screen explanation principle, Human-in-the-Loop interaction model, permission boundaries, open-source direction, and final user experience—while Codex implemented and repeatedly validated them.
Challenges we ran into
The hardest challenge was combining expressive visual output with a trustworthy security boundary. A useful board must support interactive HTML and 3D content, but that content must never gain silent access to accounts, credentials, infrastructure, or unrelated board operations. We addressed this with explicit grants, closed protocol schemas, a separate artifact origin, sandboxing, and a narrow capability bridge.
We also had to make Human-in-the-Loop state durable. A request, its stable target, the human response, and the resulting revision must remain correlated across refreshes, reconnects, and historical navigation. Regression tests were added around unstable artifact and decision targets, response correlation, pairing recovery, and revision rendering.
Finally, a small approval modal contradicted the product's purpose. We redesigned long decisions as expandable workspaces where the title and response action remain visible while the detailed Markdown body scrolls independently.
Accomplishments that we're proud of
- A working end-to-end product at
sceneboard.dev, not just a visual prototype. - One-time pairing from Codex with explicit board and capability approval.
- Self-contained context expansion that translates internal references into human language.
- Human decisions that directly influence Codex's next visual output.
- Immutable visual revision history with Previous, Next, and Latest navigation.
- Documents, diagrams, charts, SVG, animated Canvas, interactive HTML, WebGL, and Three.js on one surface.
- A sandboxed artifact runtime that separates visual expressiveness from account authority.
- Multilingual product UI and a browser-tested desktop experience.
- An Apache-2.0 monorepo with public engineering, security, Git, and QA rules for contributors and AI agents.
What we learned
AI collaboration needs more than a better summary. People need context expansion: the rule, reason, consequence, evidence, uncertainty, and next decision assembled on the current screen.
We also learned that Human-in-the-Loop should be treated as a product surface, not a blocking dialog. When a choice is consequential, the interface must give the person enough room and information to decide responsibly.
Most importantly, a visual workspace changes the relationship between a coding agent and its user. The terminal remains the execution surface, while SceneBoard becomes shared visual working memory: a place where Codex explains and people retain control.
What's next for SceneBoard
We plan to publish the Codex plugin marketplace entry, expand reusable visualization and decision templates, add team sharing and administrative capacity controls, and make it easier for contributors to add domain-specific visual explanation patterns.
Our long-term goal is for SceneBoard to become an open visual expression layer for Codex—extensible by developers, useful across CLI and IDE workflows, and understandable to engineers, product teams, operators, and other decision-makers.
Log in or sign up for Devpost to join the conversation.