Inspiration

Signing a lease or buying a home is one of the biggest decisions most people make, and it rests on a 20-minute tour and a set of listing photos picked to look good. The photos don't answer the questions you live with every day: Which room gets afternoon sun? Can I hear the highway from the bedroom at night? Will my bed and desk actually fit? Does this place work for all four of us?

We wanted you to be able to live in a place before you commit to it.

What it does

Ithaca turns a real property into an interactive, physically grounded simulation of the home.

  • Walk it. Explore a navigable 3D reconstruction of the actual interior.
  • Feel it over a day. Scrub through the day to see how sunlight moves through each room and hear how outdoor noise from traffic, animals, and insects reaches each room.
  • Bring your life with you. Take a photo of your own furniture (your desk, your bed, the weird lamp you love) and it becomes a 3D object at the right size, placed into a layout that actually fits.
  • Explore with an agent, not a chatbot. An AI guide moves you between rooms, changes the time of day, adjusts the simulation, and rearranges furniture through real tool calls on the scene. It also explains what living in the neighborhood would be like.
  • Decide together. Roommates or family members join from their own phones. Each person's agent keeps its own conversation and knows who already has plans for which room, so it raises a clash with you before anyone's plans get overwritten.

How we built it

Three data layers, one address. Each layer does a job the others can't:

  • Matterport gives us the inside: a ~46M-point scan and floor plan we rebuild into clean geometry (walls, real window and door openings, floors, ceilings) and relight ourselves.
  • Vexcel aerial survey gives us the outside: true north for the house, so we know which way each window faces, plus the neighboring houses and trees that shade those windows.
  • CoStar property data gives us the context: the listing joined to this exact Matterport model, with its facts and features, and nearby listings as comparables for the area profile.

On top of these we use OpenStreetMap, TxDOT traffic counts, NSRDB typical-year weather, Open-Meteo and iNaturalist. Together they let us simulate how the home behaves, not just how it looks.

Rebuilding the digital twin. Matterport's mesh is too coarse to live in. Up close it is wrinkled, the glass is shredded, and there are holes where furniture stood. We tried several ways to rebuild surfaces from the raw data: screened Poisson reconstruction from the point cloud (streaming the 1.9 GB cloud straight out of a remote zip), photo-guided depth from the 4K panoramas (Prompt Depth Anything, pinned to the scan with a guided filter and fused into a TSDF), and aligning a ~6M-Gaussian drone splat to the interior. None was clean enough to light.

What we ship rebuilds the house the way a builder would, and uses the scan only to measure:

  • Walls from the floor plan. Each wall centerline and thickness becomes a solid rectangle, joined with exact polygon booleans on a 0.1 mm grid. Window and door openings are cut where they really are.
  • Heights from the point cloud. Floors and ceilings are fitted as planes to the ~46 million points, so vaulted ceilings come out right. Doors stand at the angle the scan saw them.
  • Everything else from code. Stairs, sliders, cabinets, vanities, the pool deck and the roof. The roof is segmented into planar facets from Vexcel's elevation model. The whole house is about 17,000 triangles, every surface closed and flat.
  • True north from the air. Matterport's map placement was 6.5° off, enough to move a patch of sun half a meter across a floor. The roof slopes on Vexcel's surface model run square to the walls, which gives the real heading to within 0.07°.
  • No walking through walls. We rebuilt the scan's navigation graph by ray-casting a body-wide corridor between scan points.

Simulating light. We built two light systems: one you see and one you can measure.

  • What you see. Lighting is baked once, offline, into a 2048 × 2048 lightmap at 4 cm per pixel: skylight through the windows, ceiling lamps, and two colored bounces, with daylight and lamplight stored apart so each can be dimmed alone. The sun is the one light that moves. Its position comes from date, time and location, and a single 4096² shadow map covers the house and everything up to 80 m toward the sun, so the oak across the street decides when morning light reaches the bedroom. Each frame, rays from every window test how much sun gets past the eaves and trees, and that window's baked glow is scaled to match. One clock drives the color of every shader, from cool noon shade to golden hour to a 2700 K lamp-lit night.
  • What you can measure. Sun hours come from pvlib solar position, traced from floor, seated and standing points through each room's real windows. Daylight in lux uses the Perez all-weather sky on 145 Tregenza patches, with daylight coefficients precomputed per room, so any hour and any weather is a single matrix product. Inter-reflection uses the BRE split-flux method, weather is a typical year from NSRDB, and we checked the results against Radiance.

Simulating sound. Outside, we model road traffic with CNOSSOS-EU (OSM roads with TxDOT counts), rail with FTA 2018, and propagation with ISO 9613-2, using OSM buildings as barriers, per octave band, per hour, per outside wall. Birds (with a dawn chorus), cicadas, crickets and frogs are placed on a habitat grid from OSM, with species from iNaturalist and weather from Open-Meteo. Inside, statistical room acoustics becomes one symmetric linear solve: walls, floors, halls and stairwells couple the rooms, so you can also hear a TV in one room from another. The browser plays no recordings. It synthesizes everything: filtered noise set to the simulated levels, reverb from each room's RT60, and the birds, frogs and passing cars.

Turning photos into furniture. A vision-language model reads the photo and any printed dimensions. SAM 3 segments each piece and SAM 3D Objects reconstructs it as a 3D asset in about 30 seconds, for about two cents. We normalize every model to meters and a common orientation, and reject any model we can't bring within 2 cm of its stated size. Phone video works too: we take the sharpest frame from each stretch.

You can also pick from 9,506 real 3D models (Amazon Berkeley Objects, Poly Haven, Objaverse). We cleaned the catalog: fixed mislabeled classes ("a bed filed as a storage box"), rescaled about 130 models drawn in the wrong units, and refused models too heavy to stream. You can search by look, not just by words: an offline CLIP-family pass embeds each piece's image and title and extracts its colors in CIELAB, so "emerald velvet" or "quiet luxury" finds pieces whose titles never say it. We also trained a small "goes together" projection on pieces that were designed as sets, with hard negatives from the same category and maker so it can't cheat. It ships only because it beat the plain embeddings on held-out sets. A designer module then furnishes a room in layers, in one of 12 looks: the main piece, what makes it usable, the rug, light at three heights, the walls, then small things, each sized by designer rules of thumb.

Placing furniture with physics, not guesses. We check the dimensions, then a simulated-annealing solver searches over each piece's position and rotation for layouts that are physically feasible. It minimizes

$$E(L) = w_1 E_{\text{overlap}} + w_2 E_{\text{walls}} + w_3 E_{\text{door}} + w_4 E_{\text{walkway}} + w_5 E_{\text{pref}}$$

The hard terms (inside the room, no overlaps, clear floor in front of doors, nothing tall blocking a window, a 0.9 m walking path between doors) are weighted so heavily that they always dominate, and any layout that still breaks one is thrown out. The soft term scores the agent's wishes, like "sofa against the longest wall, facing the window." When a proposed change makes the layout worse, the solver still accepts it with probability

$$P = \exp\left(-\frac{E(L') - E(L)}{T}\right)$$

where the temperature T drops over time. Early on it explores freely; later it settles, so it doesn't get stuck in a bad layout. We run 14 restarts in parallel across 7 processes and return several layouts that look visibly different, or explain why it found fewer.

Agents that operate the world. We built a multi-agent system on LangChain deepagents and LangGraph with Claude:

  • Guide. Talks with you and acts on the scene through structured tools: navigate, set the viewpoint, set the time of day, toggle the light and sound layers, move furniture. Every argument is validated against the twin before anything reaches the screen. It reacts when you walk into a room, and you can interrupt it mid-sentence.
  • Background designer. Runs layout jobs asynchronously, several rooms at once, so the conversation never stops for it. Finished rooms are shown one at a time, and you decide on each before the next.
  • Observer. Learns your preferences from how you explore (where you linger, what you change), not just what you say.
  • Memory that stays where it belongs. Each family has its own memory, furniture and conversation, and it carries over between visits. Agents for different people don't read each other's conversations; they share only who has plans for which room. Furniture lives in a single register that is read fresh on every model call, so no agent answers from a stale memory of where the sofa is.

Phones, hands and voice.

  • Phone as a remote. Scan a QR code (a single-use token that expires in 10 minutes) and your phone gets a joystick, a look pad, a time-of-day dial, push-to-talk and a camera for furniture. A session hub delivers every event in order to every device and catches up anyone who joins late. When the guide asks you to choose, the options appear on your phone as pictures to tap.
  • Hand tracking. MediaPipe runs in a Web Worker and drops frames when busy, so rendering never stalls. A 1-euro filter with velocity prediction keeps the cursor smooth at display rate. Pinch to drag, two hands to zoom, sweep to turn, point to walk, and pick up a piece of furniture, turn it with your other hand, and let go to place it.
  • Voice. Deepgram streaming speech-to-text and text-to-speech, with the key kept on the server and the browser's speech engine as a fallback. The guide starts speaking while it is still writing.

Challenges and what we learned

A digital twin without a GPU. None of us had a GPU, and for most of Saturday we weren't sure a rendered twin was possible at all: reconstructing objects, recovering textures, keeping it interactive. What unblocked us was not rendering the scan at all. We rebuilt the house as simple geometry from the floor plan and baked the lighting once, offline, into lightmaps, so the browser only draws textures and one moving sun. The lesson: precompute everything that doesn't change, and simulate live only the one thing that does.

Grounding AI in physics. A language model can suggest where a couch should go, but it doesn't respect walls or door swings. So we split the work: the agent interprets intent ("I want my desk by the window"), and a constraint solver decides which layouts are actually possible. The LLM never outputs coordinates.

Making multimodal feel seamless. Voice, photos, hand tracking, persistent preferences, and a live 3D scene all have different latencies. Running the designer agent in the background and streaming results into the scene kept the experience from stalling. Phone microphones alone took a noise gate, a no-sound watchdog, an iOS audio context opened inside the tap, and capturing straight from the track on Android so the first words aren't lost.

Three people, one schema. We split into three streams (the app, the agents, the twin and simulations) and coordinated through written handoff notes. Every type that crosses the wire is one pydantic model, TypeScript types are generated from it, and a check fails the build when the two drift apart. Every API key is optional: without one, that part falls back (a scripted guide, browser speech, catalog furniture, recorded fixtures), so no one was ever blocked on someone else's service.

Accomplishments we're proud of

  • A navigable, relightable twin rebuilt from the floor plan and raw scan, not the stock mesh, with true north measured from the air to 0.07°
  • Light and sound simulated per room, for any hour of the year, from real location data and standard engineering models, checked against Radiance
  • Photo → correctly sized 3D furniture (within 2 cm) → feasible layout, end to end
  • Search across 9,506 real 3D models by style and color, not just by name
  • Several people, each with their own agent, planning one shared space from their own phones
  • About 58,000 lines of Python and TypeScript in about 30 hours, with about 200 tests

What's next for Ithaca

Search the whole market by what's inside the home: "quiet bedroom, morning light, fits a king bed and two desks." Ranked across every scanned listing in a city, by how the place actually lives, not by square footage.

Built With

Share this project:

Updates

Submission history