FixPilot X

A 3D assembly copilot that adapts to missing parts, compares rescue scenarios, and explains verified decisions through interactive MCP tools.

Comment

Inspiration

What happens when the manual is right, but reality goes wrong?

Traditional assembly manuals assume every part is available, every tool is ready, and every step happens in the expected order. But real life is different. A screw goes missing. A tool isn't available. A panel is installed incorrectly.

Suddenly, the manual stops being helpful.

We built FixPilot X — The manual that adapts to you to turn static instructions into an interactive, adaptive assembly experience.

Instead of merely telling people what to do, FixPilot X helps them understand what they can safely do next when their situation changes.

What it does

FixPilot X combines an interactive 3D workspace, a conversational copilot, and a dependency-aware assembly engine.

When a user reports that part B7 is missing, the system:

  1. Updates the actual inventory state.
  2. Identifies which assembly steps depend on B7.
  3. Blocks affected operations without losing completed work.
  4. Identifies independent steps that can still proceed.
  5. Updates the interactive 3D visualization.
  6. Explains the revised plan.
  7. Recalculates the workflow when B7 is recovered.

The same engine also handles missing tools, corrections to previously completed steps, and dependency-safe recovery.

Users can inspect components, rotate and zoom the 3D model, follow guided assembly steps, confirm completed operations, and save their progress.

FixPilot X doesn't simply display instructions. It adapts the workflow.

What makes it different

1. Adaptive physical workflows

A real dependency graph determines which operations are available, blocked, or completed. Missing components change the plan instead of merely triggering a warning message.

2. Interactive 3D guidance

The assembly visualization reflects the underlying state. Components, progress, and reported problems stay synchronized.

3. Explainable decisions

The Plan Inspector reveals how available steps and blockers change after an action, allowing users to understand the system's decisions.

4. Context-aware conversation

Supported conversational instructions operate on real project state and selected components. Ambiguous commands request clarification rather than inventing actions.

5. Reusable architecture

Two original assembly projects — an eight-step modular table and a six-step desk organizer — share the same planning engine while preserving independent progress.

How we built it

We developed FixPilot X using TypeScript, React, Vite, Three.js, Node.js, and the official Model Context Protocol SDK, with AI-assisted development using OpenAI Codex.

The architecture separates the assembly engine, conversational command router, 3D presentation, persistence, and MCP integration.

Our genuine local MCP server implements Streamable HTTP and exposes 18 tools for inspecting parts, modifying inventory, recalculating dependencies, confirming operations, handling recovery, and retrieving state.

The browser client can initialize an MCP session, discover tools, execute calls, and display real results.

The application also functions locally without a paid AI API.

Alexa+ integration

FixPilot X demonstrates a simulated Alexa+ experience, using our own conversational web interface.

The MCP implementation and assembly operations are real. We do not claim a native Alexa+ connection, Echo integration, or access to Amazon's private SDK.

The command interpreter uses bounded deterministic intent recognition rather than a runtime LLM. This approach keeps the core demonstration reproducible and available without paid inference services.

Challenges we ran into

Our biggest challenge was ensuring that an interrupted assembly never produced an invalid procedure.

We built and tested a dependency engine that distinguishes available operations from those blocked by missing parts, missing tools, or incomplete prerequisites.

Another challenge was keeping the conversation, saved progress, and 3D visualization consistent. Every meaningful operation needed to update shared state without creating contradictory results.

We also designed explicit confirmation and undo mechanisms. FixPilot X never treats an animation or button press as proof that physical assembly was performed correctly.

Accomplishments we're proud of

  • Two independent assemblies using one reusable planner.
  • Missing-part and missing-tool recovery.
  • Contextual commands and controlled undo.
  • Interactive 3D visualization and persistent progress.
  • Explainable planning through the Plan Inspector.
  • A genuine MCP server with 18 tools.
  • 164 automated unit/integration tests and nine browser scenarios reported as passed.
  • A reproducible build, public source repository, and documented limitations.

What we learned

An effective assistant needs more than conversation. It needs structured state, meaningful tools, validation, clear user control, and the ability to recover when the expected workflow

The Rescue Challenge — Our Latest Innovation

We extended FixPilot X with The Rescue Challenge, an interactive environment where judges can introduce unexpected assembly problems and explore their consequences in real time.

Users can combine missing parts, unavailable tools and recoveries, compare independent alternatives, visualize affected steps in 3D, inspect dependency graphs, and export reproducible evidence.

The project also includes a genuine 22-tool MCP server and an interactive MCP Apps implementation built with the official SDK, verified in a local reference host.

Verification results: 219 automated tests, 18 browser scenarios and 4,608 real MCP calculations passed.

The public Challenge runs the deterministic browser engine. Genuine MCP connections and MCP Apps require local setup. Alexa+ is simulated rather than natively integrated.

Judge Mode — A Faster Way to Experience FixPilot X

Our latest release introduces an optional guided Judge Mode that lets first-time visitors explore an unexpected missing part, inspect the resulting dependency changes, compare independent recovery scenarios, and approve a recovery using the original verified engine.

In the tested workflow, visible interactive controls were reduced from 57 to 28. The project also includes a simplified Windows launcher for the genuine 22-tool MCP server and MCP Apps reference host.

The public guided demo runs locally in the browser. Real MCP transport and MCP Apps require the documented local installation. Alexa+ remains simulated. Usability with first-time human participants has not yet been validated.

Built With

Share this project:

Updates

Submission history