EcoSwap

Inspiration

Most "sustainable shopping" tools either surface a generic list of eco-friendly products, or let an AI recommend alternatives with no way to verify the claim behind them. A product can call itself "eco-friendly" with no certification standing behind that word, and most tools have no mechanism to distinguish a substantiated claim from a marketing one.

EcoSwap was built around the opposite premise: treat every sustainability claim as something to verify, not something to repeat. Before recommending an alternative, it should confirm whether the claim actually holds up — real certifications and a reasoned comparison, not language alone.

What it does

Enter a product you're about to buy, or provide a photo of it, and EcoSwap:

Extracts structured attributes (category, material, packaging) from the input Searches for real alternatives and ranks them against a category-weighted rubric — durability is weighted heaviest for electronics, materials for clothing, since "more sustainable" means something different depending on the product
Runs a rule-based certification check, independent of the model — an alternative whose pitch relies entirely on vague marketing language ("eco-friendly," "natural") with no recognized certification behind it (B Corp, FSC, USDA Organic, etc.) is flagged rather than presented at face value
Supports sorting by score or price, and region-aware sourcing for local marketplaces
Can identify a product from a photo using two client-side computer vision models, with no server round-trip or added cost
Surfaces nearby donation or recycling options for the item being replaced, extending the problem beyond the purchase decision itself

How we built it

The core pipeline runs on the Claude Agent SDK, using web search as a tool so alternatives are drawn from live results rather than the model's training data. Scoring is handled by a separate, deterministic rubric layer in code — the model proposes subscores, but the final weighted ranking is computed independently, so the output isn't simply whatever the model reports about itself.

For photo input, we combined two client-side models: a pretrained MobileNet classifier for general object recognition, and a custom material classifier trained via transfer learning in Google's Teachable Machine, covering nine categories built from public datasets (waste-classification imagery and the Flickr Material Database for wood, fabric, leather, and stone). Both run entirely in-browser through TensorFlow.js, with no inference cost per scan.

For the nearby-places feature, Gemini — used separately from Claude to distribute API usage across providers — suggests candidate donation and recycling locations, but coordinates are sourced exclusively from OpenStreetMap's Overpass API. Location data never comes from a model-guessed address.

Challenges we ran into

A material classifier that was confidently wrong. Our first trained model had only five classes — plastic, metal, glass, paper, cardboard — so a photo of a fabric garment was forced into the nearest available bucket at 100% confidence. This was a class-coverage problem, not a training problem: the model had no option to express uncertainty. We addressed it by adding fabric, leather, stone, and wood as genuine classes using an established material-recognition dataset, and by introducing confidence thresholds so a low-confidence prediction is dropped entirely rather than surfaced as fact.

Nominatim could not precisely geocode informal businesses. The first version of the nearby-places feature asked an LLM to suggest a name and address, then geocoded that address. Results appeared reasonable until markers began landing over a kilometer from the actual location — a known limitation of free geocoding for informally tagged businesses, particularly outside major cities. Rather than continue tuning an approach with a structural reliability problem, we redesigned it: coordinates are plotted only when sourced from real, tagged OpenStreetMap data, and any suggestion that can't be matched to a verified location is shown as a manual search link instead of an estimated pin.

Model churn, twice within the same project. The model we initially integrated for this feature was deprecated for new API keys mid-build; its recommended replacement subsequently returned live 503 overload errors under normal usage. We built a fallback chain that attempts models in a defined order with exponential backoff, rather than depending on a single hardcoded model remaining available.

Accomplishments that we're proud of

A scoring system the model doesn't fully control. Category-weighted rubric scoring and certification verification both run independently in code, not inside the model's own output — the final ranking is computed and checked, not just generated.
Catching our own classifier's overconfidence before shipping it. A five-class material model returned 100% confidence on a photo it had no correct answer for. Diagnosing that as a class-coverage problem and fixing the root cause — not the symptom — is one of the strongest engineering calls in the project.
Choosing honesty over completeness in the nearby-places feature. Rather than keep an unreliable geocoding approach that occasionally produced pins over a kilometer off, we redesigned the feature so only verified locations get plotted, and everything else becomes a manual search link.
A pipeline that survived two rounds of live model deprecation without a rewrite. A model-fallback chain with retry logic meant the second mid-build deprecation required a config change, not new code.

What we learned

Confidence scores need a floor, or they lie. A classifier forced to choose among a fixed set of classes can return high confidence on a completely wrong answer, because "none of these" was never an option.
A second, independent check outweighs a better prompt. Prompt refinement had diminishing returns; adding separate, code-based verification — the certification check, the deterministic rubric, the Overpass-only coordinate rule — produced more reliable results than any prompt tuning did.
Scoping something down is often the harder, more defensible engineering decision. Dropping the geocoded-address fallback and hiding the map when nothing verified exists both meant shipping less than what was technically possible — and it was the right call each time.
Multi-provider orchestration is mostly about failure handling, not the happy path. Splitting work between Claude and Gemini was simple; making that split resilient — separate budget tracking, fallback chains, graceful degradation — was most of the real effort.

What's next for EcoSwap

Additional material classes, with the exception of materials such as Teflon coatings that aren't visually distinguishable from a photograph and require a non-visual identification path Broader OpenStreetMap tag coverage for informal recycling points (e.g. shop=scrap_yard), which we haven't fully incorporated yet
A cached set of real analyzed products as a fallback dataset, so a live demonstration doesn't depend on a live API call succeeding in real time
Expanded automated test coverage for the newer pipeline components (streaming, the Gemini fallback chain, photo-confidence gating), which currently have no dedicated tests beyond the core JSON parsing and scoring logic