SynMod

Inspiration

OpenAI’s WebMCP showcase inspired me to explore what agent collaboration could look like inside a 3D modeling application. I wanted to build something tangible: a scene people could directly inspect, edit, and reshape while an agent works through the same interface.

What it does

SynMod is a local-first 3D modeling workspace where one person and a browser agent can edit the same open project. People can model through the visual editor, while an agent uses WebMCP tools registered by the page. Both can add primitives, adjust transforms and PBR materials, organize objects, perform non-destructive Boolean operations, and use the same undoable history.

Why WebMCP fits SynMod

3D modeling combines visual judgment with precise, structured operations. A person may decide that a rover’s cab looks too small, while an agent can translate that direction into exact changes to position, scale, geometry, or material. WebMCP connects those two ways of working.

SynMod registers its modeling capabilities directly with the browser through document.modelContext.registerTool(...). This lets ChatGPT inspect and modify the open scene using tools defined by the application. SynMod does not need an embedded chatbot or a separate OpenAI API key.

The person and agent operate on the same scene state. A person can select and reshape one component through the visual editor while the agent adds or refines another component. Together, people and agents can:

• Find objects by name and inspect their dimensions and transforms.
• Add boxes, spheres, cylinders, cones, and toruses.
• Move, rotate, scale, rename, recolor, duplicate, group, or hide objects.
• Adjust PBR properties such as roughness, metalness, opacity, and emission.
• Create non-destructive Boolean unions, subtractions, and intersections.
• Change scene lighting, exposure, shadows, and environment presets.

WebMCP is useful here because the agent does not need to guess where to click or manipulate the canvas indirectly. It receives focused modeling tools, stable object references, validation errors, scene revisions, and structured results. The person remains responsible for creative direction and visual evaluation, while the agent can handle precise or repetitive scene operations

How we built it

I developed SynMod with ChatGPT and Codex as coding collaborators. I supplied the product direction, visual references, feature priorities, and feedback while the agents helped implement, test, debug, and deploy the application. The editor uses React, Three.js, React Three Fiber, Drei, Zustand, IndexedDB, and WebMCP. Human controls and agent tools share one validated command layer so their edits follow the same rules for history, autosave, activity records, and undo/redo.

Challenges we ran into

One of the hardest parts was deciding which modeling features were important enough for the first release. I also needed to determine how to demonstrate real collaboration instead of showing an agent making isolated changes. Technical challenges included keeping human and agent operations consistent, handling edits made close together, validating multi-step transactions before committing them, and making dropdowns, panels, and 3D controls behave reliably across browsers.

Accomplishments that we're proud of

A person and a browser agent can work on different parts of the same live scene without using separate copies of the project. For example, I can adjust the rover’s cab through the inspector while the agent modifies its sensor rig or adds another component through WebMCP. Agent changes appear in the viewport, are identified in the activity history, and can be undone or redone by the person. Multi-part requests can also be applied as one validated transaction and one undo step. Simple modeling requests often require only one WebMCP tool call after the agent identifies the target.

What we learned

Exposing focused, well-described tools to a browser agent can make modeling requests more reliable than asking the agent to manipulate the interface indirectly. I also learned that total completion time depends on more than application performance. The scene operation itself can be fast, but agent reasoning and tool selection can vary even when prompts are similar. Designing concise tools reduces unnecessary reads and round trips, but it does not eliminate that variation.

What's next for SynMod

The next step is to test SynMod with people who already assemble models from primitives, such as students, prototypers, and technical designers. Their feedback will help determine whether agent assistance is most useful for repetitive scene setup, precise transformations, material exploration, or multi-object revisions. Near-term improvements will focus on clearer onboarding, stronger modeling constraints and measurement tools, reusable assemblies, and better performance in larger scenes.

Built With

• chatgpt