herbtwin

• Live demo: https://herbtwin.kashifnazir.com
• Code: https://github.com/thekashifnazir/herbtwin (MIT)

In short

Track: Best Apps and Agents. One photo runs a multi-step workflow end to end, with no person in it:

Photo → the photo model reads it → the twin tests each choice on your next 2 weeks of weather → NVIDIA Nemotron 3 Super is asked three times, and how far the three asks agree feeds into how sure we are → one action.

It says the one thing to do now, including "leave it, this is normal for the season", and how sure it is ("our best read, about half the time", or "not sure, check first, about 1 in 3"). The twin is a small water-and-frost model of the plant. Signed in, it emails you when it is time to check, and the next photo asks "Did you do it?"

Works best for herbs grown outside. Outdoors, the twin tests every choice against your real 2-week forecast. Indoors there is less to go on and less you can change: no weather, and the twin can't model light, so an indoor answer is advice for room conditions with no forecast test, and it asks which window the plant is by.

• It holds back: on 150 expert questions, Nemotron 3 Super said "treat it" when the expert did not in 2% of answers; Nemotron 3 Ultra did in 12.7%. The photo check raised 0 false alarms on 149 healthy plants.
• NVIDIA and Nebius: Nemotron 3 Super decides, Nano talks and Ultra checks a trusted page, on Nebius Token Factory; reminder emails run as a Nebius AI Cloud Serverless Job.
• Also: species right 0.846 of the time; over 4,800 automated tests.
• Checked by people: a professional gardener's review, a former tree surgeon, and 11 strangers on Toloka (below), with what each changed.

Try it

Live demo: https://herbtwin.kashifnazir.com. No account needed: tap a sample photo on the home page, then "What does my plant need?"; the answer usually arrives in under a minute, with each step shown as it runs.

Judges: No code needed: each visitor gets 30 photo reads a day; the site-wide cap and a spending cap protect the credit. The five-minute tour and known limits are in JUDGES.md in the repo.

When it holds back

The costly mistake is doing too much: in about a fifth of expert answers the right advice is "leave it". So herbtwin says less when it knows less (the 2% and the 0 false alarms above), and:

• It says how sure it is, in words a gardener can use. "Not sure, check first" matched the gardener about 1 in 3 times on 624 held-out real gardening questions with answers from experienced gardeners, and says what would help.
• A poor photo whose name is unsure is never "our best read".
• A quote only when it is sure: the trusted-page check shows a gardening page's own sentence only when two checks pass (3 false agrees in 418), and quotes less often rather than wrongly.

Inspiration

A herb in an English garden looks bad in October: thirst, cold, or just the season? Most plant apps name the plant and list generic care; they rarely say "leave it", or how sure they are. The wrong answer costs a plant: watering a dormant herb, or spraying a tired one. It started in my own garden: this autumn the oregano died and the purple sage nearly did, and I could not tell what was normal for October.

It is for UK hobby growers with herbs outdoors: a third of UK adults (33%) grow their own fruit or vegetables (YouGov for the Horticultural Trades Association, December 2024).

I attended the Nebius × NVIDIA hackathon event in London in person, and received Nebius credits there (the first credits came through the AI Builder programme).

What it does

1. Send a photo and a place (a UK town, a postcode, or the photo's own location). Best outdoors, where each choice is tested on your local 2-week forecast; indoors the advice is for room conditions, and light comes from the window you pick (stretched, pale or leaning stems in the photo count too). Up to 3 photos of the same plant count as one read.
2. Watch it work. Each step appears as it runs and names its model.
3. The answer, what to do first. One main action, big, with a time tag and the plant's face; one line on where it came from ("Decided by NVIDIA Nemotron 3 Super on Nebius Token Factory"); a This week panel (Also do, Don't, Keep an eye on); and How sure we are. Under it, Your plant's twin shows every choice compared on the twin's chart. Which model did each step gives each step's model, seconds and cost against an all-Ultra estimate.
4. Ask. "What if I move it inside?" Nemotron 3 Nano runs the twin on that choice and searches trusted gardening sites with Tavily when it helps.
5. My plants, the reminder and "Did you do it?" Signed in, a plant keeps its answers; one email arrives partway through the advice's check window, and the next photo asks whether you did it and compares the two photos.

Built, then switched off: "What can I grow here?" for a bed (the RHS's monthly picks were in our top two lists 0.90 of the time), switched off for a more focused experience.

The plant model (the "twin")

A small world model of one plant, tried before herbtwin advises, the way a robot plans against a simulator: code rolls six choices (water, water lightly, shade, sun, bring in, do nothing) forward over 14 days of forecast weather. The pot is a water bucket: rain in, evaporation out, stress below a set level. Frost risk compares each night's low, 2.5 °C colder for a pot's roots, with the plant's own damage thresholds. Each action is scored against leaving the plant: better, same or worse.

Does the advisor use it? On 120 scripted cases, run with the twin on and off, Nemotron 3 Super took the twin's better choice on 42 risky cases in 84 of 84 answers with the twin and 78 of 84 without: Nemotron follows the forecast (not proof the twin is right).

How I built it

Python 3.12 and FastAPI. One adapter over Token Factory's OpenAI-compatible API carries every model call, so a model is one id in config. "Is it normal now?" comes from 360 cited month-by-month rows for 30 plants. Each stage is one file in src/herbtwin/stages/.

NVIDIA models: who does what

Step	Model (on Nebius Token Factory)	Why this one
Read the photo	GLM-5.3-Flash (backup DeepSeek V4.1 Flash)	Four NVIDIA vision models named the species less often (0.54 to 0.63, against GLM's 0.846 to 0.867)
Decide the advice, asked 3 times	NVIDIA Nemotron 3 Super	It over-treated least: 2%, against Ultra's 12.7%
Chat and what-ifs	NVIDIA Nemotron 3 Nano	10/10 on our what-if check, in about half Super's time
Check a trusted gardening page	NVIDIA Nemotron 3 Ultra	The largest, but it over-treats as the advisor; as the checker it met the false-agree bar in round 8, the round it shipped on (3 in 418)

Cost. One example read, run locally (#374): GLM-5.3-Flash read the photo in 17.4 s for $0.0016, and Nemotron 3 Super's three asks took 6.0 s for $0.0024, so $0.0040 in all, against about $0.0177 on Ultra (about 4.4x). That is one read, not an average, without the trusted-page check. The Ultra check costs under a cent an answer. Detail: docs/reference/nvidia-models.md.

How each service was used

Service	What it does in herbtwin	Proof
Nebius Token Factory	Every AI model call, through one adapter: GLM-5.3-Flash reads the photo; Nemotron 3 Super decides, Nano chats, Ultra checks a trusted page	Each answer's "Which model did each step"; the eval costs in docs/reference/results.md
Nebius AI Cloud	Reminder emails as a Serverless Job; public eval reports in Object Storage; the self-hosted Nano test	docs/REMINDERS.md; data/eval/reports/2026-10-19-nano-selfhost.md
Tavily	The chat's cited sources, the trusted-page check, shops for what the answer asks, and gardeners near a typed town	Chat right 25 vs 15 of 30 with and without search; trusted-page check 3 false agrees in 418
LangSmith	Traces every live model call (model, tokens, cost, time; photos, email addresses and locations removed); the saved evals as 3 datasets and 12 experiments	4,052 scored runs (#146); the workspace is private
Tendem	One professional gardener graded 3 answers	data/eval/reports/2026-10-19-expert-review.md ($47.86)
Toloka	A photo-reading pilot and a first-time-user study	data/eval/reports/2026-10-23-toloka-site-study.md

How Tavily is used

• The chat's search tool, limited to 14 trusted gardening sites such as the RHS and Kew. On 30 factual herb questions, the chat answered 25 correctly with Tavily search vs 15 without (+0.33, 95% CI +0.13 to +0.53, pre-registered). Re-checked on 22 Oct: 26 against 18.
• A trusted page that says the same. One Tavily search of trusted gardening sites per answer; Tavily Extract reads the first three pages whole; Nemotron 3 Ultra and plain code decide whether a sentence backs the action. If one does, the answer quotes it exactly, "found with Tavily". It never changes the advice. On the judges' samples it quotes the rosemary beetle advice from Gardeners' World.
• Shops and gardeners near you load after the answer and never change it.
• Full detail: docs/reference/evaluation-methods.md.

Checked by people

Expert review. I paid $47.86 through Tendem for a professional gardener (over 3 years' hands-on experience) to grade herbtwin's advice on 3 openly licensed sample photos. All three first actions were graded Acceptable, 3 out of 5 overall. Their points changed the rules (a light tidy in October; "Check first: Water only if the top of the compost is dry"). They called "7 in 10" overstated, so we measured it: on 624 real gardening questions with answers from experienced gardeners, it matched 51% of the time, and was retired (data/eval/reports/2026-10-19-expert-review.md).

A real user. A former tree surgeon tried it on the phone with me; most of what didn't land shipped in one round, such as a plain-word badge for how sure it is.

A gardener's question. A gardener friend asked how we tell light levels indoors. We didn't, so an indoor answer now asks which window the plant is by ("South window, close", "East or west window, close", "North window, or a metre or more back", "Away from any window") and looks in the photo for stems that are stretched, pale or leaning. It says plainly: "Indoor herbs get a simpler answer: there's no weather to test choices against."

A small Toloka pilot. To check our photo reading against people, on 11-12 Oct 15 beginner raters on Toloka looked at 5 openly licensed problem photos, each with and without a healthy reference photo beside it (30 ratings, $29.60). The raters' majority caught the problem the photo's source names in 4 of 5 without the reference (one more partly) and 3 of 5 with it; the reference seemed to make them over-call. A tiny sample; we publish aggregates only. Tendem and Toloka were paid from Nebius AI Builder programme credits.

A first-time-user study on Toloka. On 23 Oct, 11 strangers tapped the Rosemary beetles sample on the live site, not a plant of their own, against a plan committed before any data (data/eval/prereg/2026-10-22-toloka-site-study.md). All 11 got an answer, and 9 of 11 used the Ask box (bar 70%: met). But only 6 of 11 said the right thing to do (bar 70%: missed), and 6 of 11 found a source (bar 60%: missed). Asked for the page's big heading, 5 copied the plant's name; 3 gave a watering line instead; the twin's "Nothing beats leaving it" sat right under "Pick the beetles off by hand". So the action is now the biggest words on the page, and the twin says it only weighs water and frost. Median 15 minutes each, against 5 priced; aggregates only (data/eval/reports/2026-10-23-toloka-site-study.md).

How I know it works

What	Result
Problem check: 149 healthy / 102 problem photos	0 false alarms; 33 named right, 0 wrong
Says "treat it" when the expert did not (150 Stack Exchange questions, one ask each)	Super 2% (Ultra 12.7%, Nano 7.3%)
"Did you do it?" two-photo check	38 of 38 changes seen, 0 of 32 false
Species, top-1 (149 GBIF photos, 15 taxa)	0.846
Confidence ranks right advice above wrong (AUROC, 624 held-out gardening questions)	0.63
Main action matches the expert's (Super, 3 asks; 150 Stack Exchange and 200 MIRAGE questions)	0.43, 0.41 (ceiling about 0.8)

The expert questions are text only (no photo, place or weather), so they score Nemotron without the twin; a hand check put the ceiling at about 0.8. Intervals, methods and every report: docs/reference/results.md.

• Accessibility: axe-core (WCAG 2.0 to 2.2, A and AA) on the home, wait and answer pages, light and dark: one contrast miss, now fixed with a test; a focus ring on every keyboard stop; no sideways scroll at 320 px. Not yet done: a real screen reader.

What didn't work

We publish the failures; each has its report in the repo.

• Specialist advisors (season, soil and water, pests and health): pre-registered on 774 expert questions, they failed all three rules they were judged on, so the three asks stay plain.
• The trusted-page check failed rounds 1 to 4 and 7. Round 8 shipped: 3 false agrees in 418 (under the 3% bar), but it found only 10 of 26 good quotes against a 50% target.
• "7 in 10" matched the expert 51% of the time, so it was retired.
• Ultra as the advisor, and four NVIDIA vision models (see the models table).
• Nano self-hosted agreed with Token Factory on 81% of answers but matched the expert 4.8 points less often.
• Prices for the things to buy: every method failed a zero-wrong-price bar (Nano: 11 of 21 right, 9 wrong, 1 left blank; 4 of 6 shops block a direct fetch), so we show the item and the shop, not a price (data/eval/reports/2026-10-22-product-prices.md).
• The same photo can get a different read on another run. On site walk 2 (22 Oct), one openly licensed photo of flowering coriander was read as "looks healthy with no signs of stress" on one run and "has bolted and is flowering" on the next. A cropped downy-mildew basil got "Treat it." on one run and "Remove it. There is no cure for this." on another. That second pair is not a clean repeat, because the first run had a different place set. The answer says how sure it is, but one run is still one sample.

Not done yet

• A twin for indoor plants: indoors gets advice for room conditions, with no forecast test, and its light comes from the window you pick.
• A Nemotron photo reader: the first we will try again is Nemotron 3.5 Super VL, 5x faster than GLM but behind on species.
• A real screen-reader test, and a re-run of the Toloka study on the changed answer page.
• One server: the live site runs one instance with daily caps.

What I learned

• Cheap paid evals change how you build: 150 advice questions cost about $0.40, so I measured every model swap.
• The right answer is often to do less: of 774 expert answers, 150 say leave it and 66 say stop watering; only 37 say water. A bigger model is keener to treat.

What existed before the submission period (26 Aug 2026), and what was built during it

• Built during the period: everything in this project. The first commit is 2 Oct 2026; all app code, data, evals, the site and the reminder job date from then.
• The 2 Oct starting point was my own repo template: process files only (agent instructions, git hooks, PR template, changelog tool, licence).
• Reused ideas from an earlier eval project of mine: the model-adapter seam, the judge table, bootstrap statistics. Rewritten here, as docs/superpowers/specs/ records.
• Third-party work used as is: open-source libraries, public datasets and live APIs, each credited with its licence in NOTICE; the site's Credits page lists the photos, illustrations, data sources and models.

Built With

• cloudflare-workers
• deepseek-v4.1-flash
• fastapi
• glm-5.3-flash
• google-cloud-run
• jinja
• langsmith
• nebius-ai-cloud
• nebius-container-registry
• nebius-object-storage
• nebius-serverless-jobs
• nebius-token-factory
• nvidia-nemotron-3-nano
• nvidia-nemotron-3-super
• nvidia-nemotron-3-ultra
• open-meteo
• python
• qwen-image-edit
• resend
• supabase
• tavily