Landing Page of Talking Plant
Late night coding!
At MHacks
Plants Used for testing
Arduino and Sensor Connections

Inspiration

Most of us have killed a houseplant. Not out of neglect, really. The plant just had no way to tell us it was struggling until its leaves were already drooping. We wondered what would happen if it could speak up.

Kids especially grow up surrounded by screens and far from nature. Looking after something alive teaches patience, responsibility and empathy, but a silent pot of soil is hard for a six-year-old to care about. So we built Talking Plant: a real plant with sensors, a cartoon face and a voice, which tells a child when it's thirsty, answers their questions using its real readings, and thanks them when they water it. Each child names their own plant (ours is called Captain Leafy).

We entered the Sustainability track because sustainability starts with habits. A plant that asks for water only when its soil is actually dry, and warns you when you've given it too much, teaches kids to use water wisely without a lecture.

What it does

How we built it

Hardware. An Arduino UNO R4 reads Grove moisture, light and touch sensors and sends one JSON line per second over USB. The laptop sends the plant's mood back so an LED lights up when the plant is unhappy. A USB webcam watches the leaves.

Two layers: rules for the moment, AI for the bigger picture. We deliberately didn't put an AI model in the loop for every reading.

Live layer (every second, no AI). The raw moisture value $r$ becomes a relative percentage using two calibration points, the sensor dry ($r_{\text{dry}}$) and in water ($r_{\text{wet}}$): $$ m = \operatorname{clip}{[0,100]}!\left(100 \cdot \frac{r - r{\text{dry}}}{r_{\text{wet}} - r_{\text{dry}}}\right) $$

We smooth it with a median of the last three samples, $\tilde m_t = \operatorname{median}(m_{t-2}, m_{t-1}, m_t)$, so one bad sample can't flip the mood. To stop the plant flickering between moods at a boundary we use hysteresis: a houseplant becomes thirsty when $\tilde m \le 30%$ for 3 seconds, and only recovers at $\tilde m \ge 40%$. A watering is detected when moisture rises sharply and stays up:

$$ \tilde m_t - \min_{,t-30\text{s} ,\le, \tau ,\le, t} \tilde m_\tau ;\ge; 20 \quad \text{sustained for } 2\text{ s} $$

Thresholds change with the plant type: a succulent isn't thirsty until 15%, and is "soggy" from 70%. Real sunrise and sunset for the child's ZIP code decide whether darkness means sleepy (normal at night) or too dark (during the day).

Checkup layer (every hour, AI). A vision model running locally through Ollama describes the leaves in plain words. Outdoor air quality and weather come from Open-Meteo for the child's ZIP code. ASI:One reads all of it and picks one label from a fixed list of moods. Each night it labels the whole day, and the leaderboard score is simply $$ \text{score} = \frac{#{\text{happy days in the last } 7}}{7}. $$

Voice and conversation. ElevenLabs transcribes the child (speech-to-text) and gives the plant its voice. The plant's fixed lines are cached as audio so the core demo still talks when the Wi-Fi doesn't. Replies come from ASI:One with the plant's live readings and its last 24 hourly records in the prompt. A safety filter rejects anything too long, off-topic, unkind, or quoting a number the sensors didn't measure, and scripted lines take over if the AI is slow or offline.

Agent. Using Fetch.ai uAgents, the plant is also a chat agent registered on Agentverse, so you can talk to it from ASI:One and it answers from the same live context as the screen.

Data and app. A Python FastAPI backend ties it together. Neon serverless Postgres stores the plants, the 30-day care log, events and the leaderboard. A React app shows an animated character whose world changes with its mood: orange sky when thirsty, a rain cloud when soggy, stars when asleep, and a rainbow and confetti after a watering.

Challenges we ran into

Darkness that never happened. Our simulator reported light in lux while the plant's profile expected raw Arduino counts. The mood engine correctly ignores readings in the wrong unit, so simulated darkness silently did nothing. We made raw counts the default everywhere. Telling a watering from noise. Pulling the probe out and pushing it back in looked exactly like a watering and earned a fake "thank you". Requiring a sustained rise, and resetting continuity after gaps, fixed it. Real-world services. Neon's connection pooler rejected the query-timeout option our database library sent at connect time, so we set it per transaction instead. ElevenLabs' free tier can't use library voices through the API, so we chose a free default voice that sounds playful. A laptop too small for the model. Meta's Llama 3.2 Vision needs more memory than our 8 GB laptop had, so we fell back to the smaller moondream model. It called a golden pothos's natural yellow streaks "unhealthy". That's why the AI weighs the camera's description against the sensors instead of trusting it alone. Getting the AI to pick the right label. Given dry soil and drooping leaves, ASI:One first said unwell. Telling it which label wins when several apply (dry soil means thirsty, even if the leaves droop) took our test cases from 4/5 to 14/14 correct. Counting days once. Re-labelling a day left an old label on an earlier row, so the leaderboard could count one day twice. Labelling now clears the old one. Browsers vs. talking plants. Browsers keep sound and the microphone off until someone taps the page. We added a one-tap start and a kiosk launcher for the stage laptop. A test that only failed on Linux. Our serial-port test passed on Windows but raced on Linux CI: the fake board sent its only message before the reader started listening. We fixed it by making the fake board behave like the real one and keep sending every second.

Accomplishments that we're proud of

A real plant that really talks. It works end to end on actual hardware: soil sensors and a touch pad on an Arduino, a webcam on the leaves, and a cartoon face that reacts within a second when the soil dries out or gets watered. Fast enough for a conversation. A child's question is transcribed in about 1 second, ASI:One answers in about half a second, and ElevenLabs speaks the reply in about 1.5 seconds. That's short enough that a child doesn't lose interest. It never makes things up. Every answer is grounded in the plant's real readings ("my soil is 18% wet"), and a safety filter throws out anything invented, off-topic or unkind. On our test cases, ASI:One picked the correct mood label 14 out of 14 times. It works when things break. The second-by-second reactions are simple rules, not AI, so the plant still reacts with no internet. Its everyday lines are pre-recorded, and scripted answers take over if an AI service is slow or a key is missing. Every sponsor technology is live, not mocked. Our service check passes all 11 connections: ElevenLabs (speech-to-text and voice), ASI:One, the Fetch.ai agent registered on Agentverse, Neon Postgres, Ollama, and the weather and air-quality APIs. It knows its own plant. At sign-up, the webcam identifies the plant, such as an aloe vera, and picks the right watering thresholds for a succulent, a houseplant or a tree. A world that matches the mood. Orange sky when thirsty, a rain cloud when soggy, stars when asleep, and a rainbow with confetti after a watering. Kids understand how the plant feels before it says a word. Privacy and lean by design. Photos are described and then discarded, never stored. Raw sensor data is kept for 24 hours and the care log for 30 days. Built to be trusted. Over 90 automated tests cover the mood rules, the safety filter and the database. The database tests also pass against real Neon Postgres.

What we learned

Use rules where you need speed and trust, and AI where you need judgment. The second-by-second reactions are simple, testable rules that work offline. The AI adds the hourly picture and the conversation, always grounded in real readings. Speed is measurable. We benchmarked ASI:One before committing to a design: adding 24 hourly records to the prompt (about 800 extra tokens) kept the median reply at about $0.5,\text{s}$, so live chat could afford full context. Sensor numbers are relative. Our moisture "percent" is a calibrated scale between two reference points, not physical water content. Thresholds have to be measured per plant and pot, not looked up. Kids need forgiving design. That means big buttons, a voice instead of text, and a plant that never makes up problems, so a child is never scared or misled.

What's next for Talking Plant

Real thresholds for real plants. We'd measure dry and wet levels for common houseplants, instead of using three broad categories, and add a guided calibration at the pot. Into classrooms. We'd test with kids and teachers, then add a class garden, where every student's plant appears on one shared board, and a simple view for parents and teachers. No laptop needed. We'd move to a Wi-Fi board (the UNO R4 we already use has Wi-Fi) and an always-on server, so hourly checkups never miss a day and the plant can live on a windowsill. Show the water saved. We'd track how often the plant was watered only when it needed it, and turn that into a kid-friendly picture of good water habits. More voices and languages, so children can talk to their plant in their own language. Smarter eyes. A vision model trained on plant-health photos would spot yellowing, pests or sunburn more reliably than a general-purpose model.