Inspiration
When I was looking at rentals, I found it hard to imagine how I’d arrange my room before moving in. I didn’t have the design skills or patience to model it myself. I wanted to see whether my furniture would fit and try a few layouts before having to move anything.
What it does
Upload a floor plan, and optionally a video tour, and Refresh builds an editable 3D model of each room. Furniture from your video is separated into individual 3D pieces with fal.ai, so you start with what you already own.
Choose a room and move furniture around yourself, or ask the AI designer for a change, such as a more open layout, adding a desk or a better feng shui. It rearranges the room and can suggest real products to add, from a bundled IKEA library and live product search.
Every suggestion is checked for fit, door access and walking space before you see it. You review the proposed changes and decide whether to accept them, then walk around the room in first person to see how the layout feels.
How we built it
We designed the frontend in Figma through Claude Code using the Figma MCP. Working from a visual design gave Claude a clearer reference for the interface. We found this made the web design much better and helped avoid the generic styling and “AI slop” phrases that often creep into generated websites.
We built the editor with React, TypeScript and Three.js through React Three Fiber. Our Python backend uses FastAPI, with separate services for reconstruction, spatial checks and media generation.
The reconstruction pipeline combines floor-plan geometry with furniture extracted from video frames. We use fal.ai’s SAM 3 and SAM 3D models to isolate furniture and generate individual 3D meshes, which we scale for use in the editor.
An Anthropic-powered agent turns requests into proposed furniture changes. The backend checks those changes against the room geometry before showing a preview. We keep furniture dimensions and positions in a shared scene model so the editor and spatial checks agree.
Challenges we ran into
Getting reconstructed furniture into the right place, at the right scale and facing the right direction took more work than generating the meshes. Video frames contain overlapping objects and incomplete views, so we needed ways to flag uncertain results and correct them.
We also had to define what makes an agent’s suggestion acceptable. A layout could improve one score while blocking a door or moving something the user wanted to keep fixed. We added explicit geometry checks and a review step before applying changes.
Integrating the reconstruction pipeline with the editor was another challenge. Prepared scenes let us keep testing the editor and agent while the upload flow was still being connected.
Accomplishments that we’re proud of
We’re proud of connecting room editing, live spatial checks, agent suggestions and first-person exploration in one interface. You can ask for a change, inspect the proposed moves, accept them and walk through the result.
We also added room versions so users can compare ideas and return to an earlier layout. That makes experimenting with a room much less frustrating.
What we learned
We learned how much a useful room editor depends on consistent geometry. Units, furniture dimensions and coordinate conventions affect everything from rendering to collision checks.
We also learned to give the agent room to propose changes while checking the measurable parts in code. Keeping those responsibilities separate made suggestions easier to validate and problems easier to trace.
What’s next for Refresh
We’d also like to improve reconstruction quality to be photorealistic, so users can return to a home and keep trying layouts as their plans change.