Inspiration

Some of our relatives look after gardens and small farms, and we watched them struggle with problems that turned out to be common. A spotted leaf would bring five opinions and five diagnoses. Asking an expert was hard: they are rarely available when you need them, and their time costs money. So we set out to build an app that anyone can use, for a plant's everyday needs as well as for diagnosing a problem and treating it.

What it does

You photograph a plant. GreenLens guides the shots like a camera app: the whole plant, a close-up of the leaf, the underside, the soil. Then you watch the analysis happen live, stage by stage, as findings land on the photo.

Vision on self-hosted GPUs. Models on our own GPUs find the leaves, name the species, rank the likely problems and look for signs such as spots, insects and soil condition.
When two problems look alike, it asks instead of guessing. For example: "Do the spots have rings, like a target?", with picture answers.
A diagnosis you can check, and a plan. GreenLens weighs the findings against curated disease profiles, the weather forecast, the soil and the plant's history. You get a diagnosis showing what it rests on, and a step-by-step plan where every step cites its source. There are no made-up confidence numbers.
An AI agent that knows your garden. Our Claude agent has 21 tools. It reads the forecast, your plants and your scans, remembers what you tell it ("no copper on anything") and acts for you, for example by adding frost-cover tasks for your tender plants. It can also search a farm's Slack channels and Google Drive records, and cites them.
Your garden over time. Insights show whether the plan worked, and a 3D view of the garden shows each plant's health, growth and frost risk.

How we built it

App: Expo SDK 57 and React Native for the iPhone, with Apple's own controls, plus Swift modules for LiDAR depth capture, a 3D garden in RealityKit and a Live Activity. Sign-in uses Clerk.
Backend: Rust (axum, tokio) as eight library crates and two binaries: the API and a worker. It exposes 76 API operations and streams each scan's progress to the app over SSE.
Data: SurrealDB 3.3.4 holds records, the graph, vectors and the job queue in one database, with gap-free event numbers so the app can resume a stream. SurrealDB Agent Memory keeps memories that everyone in a garden shares. RustFS stores the images privately behind signed URLs.
Self-hosted GPU infrastructure: three GPUs in teammates' machines. The workers pull their jobs, so no machine opens a port.
- An NVIDIA RTX 3070 runs the perception models through ONNX Runtime on CUDA: RF-DETR for leaves, SAM 2.1 for masks, BioCLIP 2 for species and our own disease head.
- An NVIDIA RTX 5070 Ti serves Qwen3.5-9B on vLLM to read the telltale signs.
- An AMD RX 9070 XT serves the same Qwen3.5-9B on vLLM's ROCm build, as a second vision-language worker.
- If no worker claims a job within 15 seconds, Claude takes it.
The agent: Claude Opus writes the chat answers and the plans. The plan writer returns strict JSON, every step needs a citation, and a dose check keeps amounts out of the text. Claude Haiku handles memory and the cheap calls.
Our principle: vision produces the facts, fixed rules set the risk, and the language model only writes the words.
Testing the farm connectors: Era by Eon's Slack and Google Drive simulators are our test bench, with a farm fixture that even plants a prompt injection. Across 3 runs of 18 questions, the agent got 54 of 54 answers and 51 of 51 citations right, and never followed the injection.
Built with Claude Code, guided by 184 project skills that cover Apple's design guidelines, our Rust conventions and Era.

Challenges we ran into

Coordinating self-hosted GPU workers across teammates' machines.
- Our GPUs aren't in one data centre. They're in teammates' machines, and each worker reaches the API over the internet.
- We wanted no open ports anywhere. So the workers pull their jobs: each one heartbeats, claims a job under a lease, and posts its results and events back over outbound HTTPS.
- The API reaches the internet through Tailscale Funnel. Each worker has its own token.
- We had to handle what happens when a connection drops mid-scan, or a worker dies holding a job. Its lease expires and the job goes back to the queue. If no worker claims it, Claude takes it: in our test, the fallback took over a killed worker's jobs in 47 seconds.
- We also killed the API in the middle of a scan. The stream resumed with every event in order, numbers 1 to 61 with no gap.
- Mixing NVIDIA (CUDA) and AMD (ROCm) cards meant one worker binary behind two vLLM builds, with different kernels, memory limits and device setups on each host.
Models that test well on their own data and worse on real gardens. Our disease head scored highly on its held-out split but much lower on handheld garden photos. Retraining with field photos improved it.
Datasets that leak. Published datasets hide flipped and rotated copies across their train and test splits. iNaturalist files diseases under the pathogen, so a "grape black rot" photo can show Boston ivy. We built an evaluation set checked against every training image and reviewed by two people.
A vision-language model that reads disease as insect damage. Qwen kept calling early-blight spots "leaf mines". Changing the prompt one line at a time showed how fragile the wording was, and we moved to a structural fix.
Honest uncertainty. In a full run on our GPUs, an early-blight photo came back as Septoria leaf spot, a look-alike. That is exactly why the app asks a question when two problems look alike, and why we don't show confidence percentages.
Licences. Every third-party model we ship is Apache-2.0 or MIT. We dropped good models and datasets whose licences didn't allow it.

Accomplishments that we're proud of

Our own computer-vision infrastructure.
- The perception models and the vision-language models run on three GPUs we host ourselves, two NVIDIA and one AMD, not on a rented API. A real scan's findings have run on each vision-language card.
- A full scan's perception (6 photos and 40 leaf crops) runs in 1.2 s on a single 8 GB RTX 3070. Qwen3.5-9B serves two jobs at once on the RTX 5070 Ti.
- An end-to-end scan ran on this infrastructure, from photo to a cited 8-step plan.
Every sponsor's technology in one product, each doing a real job.
- Expo builds the iPhone app, with our own Swift modules for LiDAR, the 3D garden and the Live Activity.
- Clerk handles sign-in and the tokens the API checks.
- SurrealDB holds records, the graph, vectors and the job queue in one database. SurrealDB Agent Memory gives a garden shared memories that the agent can use.
- Anthropic's Claude is the agent: 21 tools, plans that cite every step, and the fallback when a GPU worker is down.
- Era by Eon's Slack and Google Drive simulators test the farm connectors and the agent. Across 3 runs of 18 questions, it gotf 51 citations right, and never followed a planted prompt injection.
Models measured honestly. On held-out test photos scored only once, BioCLIP 2 has the right species in its top 3 for 93% of cases. Our disease head has the true problem in its top 3 for 84%, against 60% for guessing. In our benchmark, all 82 calls to the vision-language model returned valid structured output.
We wrote down the weaknesses with their numbers, next to the strengths.

What we learned

Only photos from new fields and gardens tell you how a model will really do.
Judge a model by its job: our disease head proposes candidates, so we judge it by its top 3.
Give every score its baseline, distrust published splits, and change prompts one line at a time.
Pull-based workers make self-hosting practical: machines on different networks only need outbound internet, and a lease that expires means a dead worker never strands a job.
Keep the language model for language: facts from vision and rules from data make the answers easier to trust.

What's next

Measure diagnosis accuracy end to end through the API.
Benchmark the RX 9070 XT and tune how many jobs it serves at once.
Add more GPU workers on more teammates' machines; the pull model makes each one a token and a container.
Connect real Slack and Google Drive workspaces.
Put GreenLens in the hands of the relatives who inspired it.