Inspiration
When a contractor or DIYer snaps a photo of a renovation job — like resurfacing a worn 96-by-25-inch kitchen countertop — turning that photo into a verified, budget-compliant purchase across multiple suppliers takes hours. SpecToCart turns any project photo into a policy-verified PayPal order with a complete rejection log in one click.
What it does
SpecToCart takes a photograph of a project — a worn countertop, a bare wall, a broken fixture — and produces a complete, sourced, budget-enforced shopping cart paid through PayPal.
Gemini Vision reads the image and generates a structured Bill of Materials: 3–5 items, each with quantity, per-item price cap, and priority. A Python project-math engine then derives quantities from measurements — ceil(area × 1.10 waste / coverage) — instead of trusting the model to guess. Channel3's product graph sources each item at or under the per-item cap, scoring every candidate by relevance, requiring a head-noun match, and recording every rejection with an explicit reason (price_over_cap, currency_mismatch, insufficient_relevance, head_noun_missing, score_zero). A second Gemini call per item — the critic pass — asks whether the sourced product is plausibly the right category, catching cases word overlap cannot (a silicone seam cover is not a silicone sealant tube).
If sourcing exceeds the budget, a deterministic fallback drops Optional items in policy order (ascending relevance, descending line total) and records each drop. The cart becomes an itemized PayPal Sandbox order with idempotency keys and the photo's SHA-256 bound into custom_id. After capture, a two-column Reconciliation Certificate shows PayPal settlement beside the full Channel3 rejection audit trail — downloadable as JSON.
For returning builders, PayPal Vault v3 lets a second project settle in one click without a browser popup. A signature-verified webhook endpoint handles the asynchronous confirmation for buyer-not-present charges.
How we built it
- Gemini 3.6 Flash (3.8 fallback) via google-genai, with Pydantic schema enforcement for BOM extraction, step-plan generation, and the per-item critic pass. Two-tier retry chain for transient 503s.
- Python pipeline for all arithmetic: project math, budget fallback, relevance scoring. The model never guesses; the code computes.
- Channel3 product graph for live retail sourcing. Schema verified empirically before the pipeline was trusted to buy.
- Flask web app for the dashboard, PayPal JS SDK integration, policy gate, and reconciliation views.
- AG Grid Community v31.3.2 (Quartz Dark theme) for the interactive procurement manifest, custom currency formatting, and live spend-policy gate.
- PayPal Orders v2 with idempotency, Vault v3 for returning buyers, and signature-verified webhooks for asynchronous settlement.
Challenges we ran into
- Gemini transient failures. 503 UNAVAILABLE during a peak week, not a key or code problem. Built a two-tier retry chain with in-family fallback and exponential backoff.
- Relevance scoring for retail products. Cheapest is not correct. Fixed a spurious-match bug where possessive brand names tokenized as ["elmer", "s"] and single-letter "s" scored as a match. Added minimum-match thresholds scaled to query length and head-noun matching. Without these, "marble wall sticker" won "vinyl wrap" on two shared words.
- Sandbox merchant setup. Configured a US Business sandbox merchant and webhook listener in the Developer Dashboard to test browser-present checkouts and buyer-not-present vaulted captures; the buyer account can be any country.
- Semantic category mismatches. Token overlap alone scored a solid silicone seam cover as a winner for liquid silicone sealant. Rather than trusting raw keyword scores, we built a second-pass Gemini category critic and strict head-noun enforcement (
NoValidOfferError) that flags or blocks wrong categories before checkout rather than silently substituting a bad product.
Accomplishments that we're proud of
- Working end-to-end vision → BOM → sourcing → PayPal capture flow with committed proof
- 35 passing tests, including pure-function coverage of relevance scoring, budget fallback, project math, and idempotency
- Two independently verified PayPal proofs: browser-present capture and buyer-not-present vaulted capture
- Signature-verified webhooks with idempotent dedupe by event.id
- An audit receipt that lets a judge re-derive every decision without watching the video
What we learned
Deterministic pipelines beat autonomous loops for procurement. Fixed control flow makes the arithmetic testable and the receipts reproducible. The distinction that matters is not "shopping agent" — that space is crowded — but derivation layer: project math, rejection logs, and fallback policy that a replenishment agent structurally cannot produce. And strict rejection policies ("we flag and reject a category mismatch rather than silently substitute a wrong product") read as far more institutional than a blind auto-cart.
What's next for SpecToCart
- Single-store cart optimization (Channel3 raw schema does not yet expose cross-merchant grouping)
- Cheaper-Optional substitution (currently Optional items are dropped, never substituted)
- Multi-room renovation manifests (expanding project math to wall coverage, flooring, and electrical fixtures)
- Enterprise spend-policy: SKU substitution with spec-match scoring
Best Use of Channel3
SpecToCart uses the Channel3 product graph as its sourcing layer, not just its search layer. For each BOM item, SpecToCart queries Channel3 with a per-item price cap and stock filter, then evaluates every returned candidate against a relevance rule — minimum matched words scaled to query length, head-noun required — before selecting a winner. Every candidate Channel3 returned is recorded in the run receipt with its relevance score, matched words, and rejection reason. The receipt is not a success log; it is a full audit trail.
A per-item critic pass (a second Gemini call) then evaluates whether the sourced product is plausibly the right category, catching cases keyword scoring cannot distinguish — a silicone seam cover versus a silicone sealant tube, or a glue gun versus a heat gun.
Channel3's schema is verified empirically: id, title, description, offers[]{url, domain, price{price, currency}, availability}, category, variants. buy_url is offers[0].url, not a top-level field. The raw JSON was inspected with DEBUG_CHANNEL3=1 before the pipeline was trusted to buy anything, and the pipeline raises NoValidOfferError rather than substituting a wrong product when no candidate qualifies.
Best Use of AG Grid
The cart in SpecToCart is an AG Grid (ag-grid-community 31.3.2), and it is the window into the spend-control proof — not a static table. It runs the Quartz Dark theme (ag-theme-quartz-dark) with instant column sorting and resizing, a currency formatter on the Total column, and flexible column sizing (BOM Spec flex 1.3, Channel3 Winner flex 1.5) so the spec and the winning product get the most room.
Each row is a sourced BOM item. Expanding a row opens its rejection drawer: every Channel3 candidate that lost for that item, with its relevance score and rejection reason (price over cap, wrong category, not enough relevance). The critic verdict is inline on the row — the sealant row carries a red Critic REJECT — not buried in a log.
The grid is also the policy control. Flipping the budget toggle from $150 to $48 re-shapes the cart live in front of you: the Optional degreaser drops and the total falls from $49.79 to $45.54. The approval checkbox on the same view gates the PayPal button, so payment cannot proceed until the buyer approves the policy-checked cart. The Install Step Plan sits at the bottom of the same view.
A judge watching the video sees the audit trail and the $48 fallback on screen; a judge running the app can expand any row and test the policy themselves.
Built With
- ag-grid
- channel3
- flask
- gemini
- javascript
- paypal
- pydantic
- python
