GhostBuild

GhostBuild — Hands Busy. Mind Clear.

What if fixing something was as simple as having an expert standing beside you—without needing to hold a phone, search through a manual, or lose your place?

Inspiration

We've all been there: halfway through assembling furniture or repairing something, one hand holding a tool, the other holding a part, and somehow we're expected to scroll through a 40-page instruction manual.

The problem isn't that instructions don't exist. It's that they're rarely available in the moment, in the context, or in the form we actually need.

We built GhostBuild to bridge that gap: an intelligent, hands-free guide that brings the right information into your field of view, exactly when you need it.

What it does

GhostBuild transforms assembly and repair tasks into contextual, step-by-step guidance delivered through Even Realities G2 smart glasses.

Instead of navigating endless PDFs or watching entire tutorial videos, users can:

• Describe what they're trying to accomplish. GhostBuild helps identify the product, problem, and relevant task.
• Find grounded instructions. The system uses product documentation and other supporting sources to construct an actionable guide.
• Follow instructions through their glasses. Concise prompts, diagrams, and relevant part references appear on the G2 display.
• Navigate without putting down their tools. Touch gestures let users move through steps, inspect diagrams, and return to where they left off.
• Ask for help in context. Users can explore what a part is, where it belongs, or what an instruction means without abandoning their workflow.
• Stay in control. GhostBuild distinguishes suggested guidance from completed physical actions. It doesn't assume something has been assembled correctly just because the user advanced a screen.

Our primary demonstration uses the IKEA MICKE desk, including its manufacturer documentation, component identification, and assembly instructions.

GhostBuild isn't trying to put a computer screen in front of your eyes. It's trying to show you less information, but better information.

How we built it

GhostBuild combines a wearable interface, an intelligent guidance backend, and a companion phone experience.

The G2 application: Built with TypeScript, Vite, and the Even Hub SDK, our glasses application renders lightweight text and image cues optimized for the G2 display. It handles touch gestures, connection recovery, navigation, and source-linked diagrams.

The intelligence layer: A Python and FastAPI backend handles task state, product discovery, guided workflows, contextual questions, and manual-grounded instructions. The system uses structured playbooks rather than treating every user interaction as an isolated chatbot response.

Documentation and research: Our pipeline extracts and processes reference material, including manufacturer manuals, and turns it into steps with supporting evidence and visual references. We integrated AI-assisted research and guidance generation while preserving the distinction between source-backed information and user-confirmed actions.

The phone companion: The phone provides a richer interface for entering goals, inspecting sources, and asking questions. The glasses deliver the concise information needed during the physical task.

Resilience: WebSocket communication and an HTTP polling fallback help the G2 client recover from connection interruptions. Instructions can continue through touch and text when optional voice services are unavailable.

Challenges we ran into

Making information genuinely useful on a tiny display. A full manual page isn't helpful on smart glasses. We had to rethink instructions as small, context-sensitive cues and carefully prepare diagrams for a constrained display.

Turning documentation into dependable guidance. Product manuals contain diagrams, ambiguous part references, and steps that rely on information from previous pages. We had to preserve that context while making each instruction understandable on its own.

Getting real hardware to behave reliably. Developing for smart glasses meant working through packaging, touch gesture behavior, Bluetooth-connected rendering, local network configuration, and reconnection handling.

Knowing what the system actually knows. In a physical task, an incorrect claim of success matters. We designed the workflow around explicit user confirmation and clear distinctions between guidance, evidence, and completed actions.

Accomplishments we're proud of

• Built a real Even Realities G2 application and packaged it through Even Hub.
• Created a source-grounded IKEA assembly experience with step-by-step wearable guidance and diagrams.
• Connected wearable navigation to a persistent task-aware backend and phone companion.
• Implemented recovery paths for unreliable connections and unavailable optional services.
• Built automated tests across the backend, phone experience, and G2 application.

Most importantly, we demonstrated that AI assistance doesn't have to mean more screen time. It can mean putting the screen away and getting back to the real task.

What we learned

Building for augmented wearable experiences is fundamentally different from building another chatbot.

The hardest problem isn't generating an answer. It's delivering the right answer, at the right moment, in a form someone can use while doing something physical.

We learned to prioritize grounding, clear interaction design, graceful failure, and user control over flashy but unreliable automation.

What's next

We envision GhostBuild expanding beyond furniture assembly into appliance troubleshooting, bicycle maintenance, electronics repair, and other everyday tasks.

Future directions include more reliable conversational voice control, richer hardware-aware guidance, broader verified documentation coverage, and improved ways to understand task progress without interrupting the user's work.

GhostBuild's vision is simple: less looking up, more doing.

Built With

• anthropic-claude
• creao
• elevenlabs
• even-hub-sdk
• even-realities-g2
• fastapi
• javascript
• pillow
• pydantic
• pymupdf
• pytest
• python
• typescript
• vite
• websockets