Inspiration
Have you ever seen a chair, desk, shelf, or literally anything you liked, found something similar online, and then had to ask: wait... will this even fit here? Usually that turns into opening five product pages, searching for dimensions, pulling out a tape measure, and then trying to imagine what a 29-inch-wide chair is supposed to look like in your room. I wanted to make that one interaction. Point your phone at something you like, find similar products, measure where you want it, and actually see whether it fits before buying it. That became Will it Fit?
What it does
Will it Fit? turns your phone into a spatial shopping assistant. You start by pointing the camera at something you like and capturing it. I analyze the image and search for real products that look similar. From there you can browse the results normally or just talk to the app: “Show me a cheaper one.”, “Only show black ones under $100.” or “Would the second one fit here?” The app keeps track of what you are talking about, so “the second one” and “that one” actually refer to the products currently on screen. When you want to check the size, you measure the available floor space directly with your phone using AR. I try to verify the selected product’s dimensions from its retailer listing. When trustworthy dimensions are available, I compare its footprint against the measured space; otherwise, the app clearly reports that fit could not be verified instead of guessing. Then comes my favorite part: View in my space. Most products online do not come with a usable 3D model, so I generate one from the actual shopping image with Stable Fast 3D. I use the product's real dimensions to scale it correctly, then place it into the room through ARCore. So instead of searching, guessing, ordering, and hoping, the flow becomes: see, find, measure, check, preview, then buy. I also added persistent shopping memory with Backboard, MongoDB for saved state, and optional ElevenLabs voice responses so the shopping agent can actually talk back. The shopping flow continues all the way into purchase intent. You can say things like “Buy the second one,” and the agent resolves the exact product, merchant, price, quantity, provider-backed variant identifiers when available, and fit result before asking for confirmation. It never purchases from the first command. After confirmation, the backend signs the purchase intent with a Visa TAP-aligned development flow and verifies it through a merchant-side verifier. Because Visa Intelligent Commerce itself was not publicly available during the hackathon, the final transaction step is clearly shown as a hackathon simulation rather than a real Visa payment.
How I built it
The app is native Android in Kotlin, with ARCore doing plane tracking, anchors, measurement, and real-scale placement. When you point the camera at something and capture, Qwen3-VL 8B runs locally through Ollama and figures out what it is looking at. I combine that with Google Shopping and Google Lens results through SerpApi, then use the vision model again to visually rerank the products instead of trusting text similarity alone. Finding the product was only half the problem. I also needed its actual size. Product pages are surprisingly inconsistent about dimensions, so I built a resolver that prefers structured product data and explicitly labeled specifications, then falls back to web extraction when needed. I never let the model just invent a width or depth because that would make the entire fit check pointless. For the physical space, ARCore gives me points in the real world. The user marks out the available footprint, and I build a rectangle from those measurements. Fit checking itself is deterministic: compare the product width and depth against the measured space, rotate it 90 degrees, and check again. For the AR preview, there usually isn't a 3D model hiding somewhere on the product page. I take the selected shopping image, send it to Stable Fast 3D running locally on an RTX 5080, get a GLB back, and scale that mesh using the dimensions I found online. The conversational side uses Qwen3 4B Instruct. Instead of letting the model directly mess with app state, it has to produce typed actions like search, select, compare, check fit, or show a product in AR. The backend executes the action and then gives the model the result to explain. I use MongoDB Atlas for persistent app data, Backboard for longer-term shopping memory, ElevenLabs for optional voice responses, and Vultr for the public-facing services behind willitfit.tech. For the Visa portion, I extended the same typed-action system with prepare_purchase, confirm_purchase, cancel_purchase, get_purchase_status, and open_checkout. Purchase state is kept server-side, confirmations are tied to the active purchase intent, and stale “yes” responses are rejected. The trust layer uses backend-only Ed25519 signing and an RFC 9421-style HTTP message signature around the purchase intent, with nonce and replay protection. A merchant verifier checks the signed request before the simulated commerce step runs. The original retailer URL is preserved so the user can still open the real product page.
Challenges I ran into
Getting a product onto the screen was the easy part. Getting the right product, with the right dimensions, at the right scale, into the right place was the hard part. The first product search was mostly text based, which meant taking a picture of a chair could return chairs that matched the description but looked nothing like it. I added Google Lens results and a local visual reranking step so appearance actually matters. Dimensions were worse. One site says 28" W × 31" D × 34" H, another hides the numbers inside a JavaScript-loaded specs tab, and another gives dimensions for the shipping box instead of the product. I ended up building a layered resolver and keeping the source with the dimensions so I know where each measurement came from. Then there was 3D. Shopping sites generally give you JPEGs, not GLBs. Stable Fast 3D solved the missing-model problem, but its output has no idea whether the chair is 20 cm or 2 meters tall. Separating shape reconstruction from real-world scale ended up being one of the most important design decisions in the project. I also hit a lot of state bugs once conversation entered the picture. If you search for something new while an old AI response is still running, that old response cannot suddenly select a product from your previous search. The same problem applies to dimensions and generated AR assets. I had to make sure stale responses could not overwrite the user's newer camera, product, or measurement state. And, of course, latency. My pipeline involves a vision model, shopping APIs, image downloads, dimension extraction, another model call, 3D reconstruction, GLB processing, and sending the result back to the phone, so I ended up caching basically everything worth caching. The Visa part had a different kind of constraint because the actual Visa Intelligent Commerce platform was not publicly available to hackathon teams. Instead of pretending I had access, I separated the flow into what I could really implement and what had to stay simulated. The signing, integrity checks, replay protection, and merchant verification run in my backend; Visa certification and transaction completion remain simulated or unavailable for the hackathon.
Accomplishments that I'm proud of
I'm proud that the demo is not just an AR cube with a shopping UI around it. The thing that appears in AR comes from the actual product you selected. Its appearance comes from that product's shopping image, and its scale comes from dimensions sourced from its listing. I'm also proud that all the pieces share the same state. If you photograph something, ask for a cheaper version, pick the second result, ask if it fits, and then say “show that one in my room,” every step still refers to the same product. And a surprising amount of the AI runs locally. The image understanding, visual reranking, conversational planning, and 3D reconstruction can all run on my own hardware instead of sending every part of the experience to another hosted model. Mostly, I'm proud that something which started as “wouldn't it be cool if your phone could tell you whether this chair fits?” somehow turned into a real end-to-end shopping flow. I also got the purchase flow to use the same product state as the rest of the app. “Buy the second one” does not create some new fake checkout object. It resolves the same product the user was already viewing, carries over its merchant, price, available variant information, and fit status, and requires an explicit confirmation before moving forward.
What I learned
I learned a lot. Giving an AI more tools is much easier than making sure all those tools agree on what the user is talking about. The product search can be correct and the experience can still fail if the dimensions are for another variant. The dimensions can be correct and the AR can still be wrong if the mesh is scaled incorrectly. The AR can be perfect and the conversation can still ruin everything by referring to a stale result. A lot of the work ended up being less about making the models smarter and more about giving them strict boundaries and keeping state consistent. I also learned that AR gets weird fast. A couple of points that look fine on a phone screen can represent a pretty bad physical measurement, and small errors matter when you're trying to answer something as binary as “does this fit?” And I learned that local models are a lot more usable than I expected. With smaller Qwen models and some caching, I could keep most of the intelligence on the machine running the backend instead of depending on one giant hosted model for everything. Building agentic commerce also made me think much more carefully about authorization. A shopping agent being able to understand “buy this” is very different from being allowed to act on it. I ended up treating purchase intent and purchase confirmation as separate states and invalidating stale confirmations whenever the user changes products or context.
What's next for Will it Fit?
First: make the 3D reconstruction better. It works, but a single shopping photo can only tell you so much about the back and sides of an object. I also want measurement to become almost invisible. Instead of manually marking the footprint, the phone should understand the empty space itself and tell you what products could fit there. After that, I'd like Will it Fit? to remember spaces, not just products. Imagine saving “my bedroom desk corner” or “the space beside my couch” and later asking: “Find me a bookshelf under $150 that fits next to my desk.” The same idea also goes way beyond furniture. TVs, appliances, luggage, car accessories, storage containers—there are a lot of things people buy online where the most important question is still: Will it fit?
Log in or sign up for Devpost to join the conversation.