ai-visual-plants-explorer

An intelligent, multilingual environmental guide powered by Nebius AI Cloud and NVIDIA, built to democratize botanical education globally.

Comment

Inspiration

Working as a national tour guide at the Quito Botanical Garden in Ecuador, I interact daily with tourists and local school groups fascinated by our biodiversity. Previously, I built a custom Convolutional Neural Network (CNN) that achieved 98% accuracy in identifying local plants. However, I quickly hit a wall: the CNN only gave scientific names, only worked for our specific garden, and required users to create accounts to save their progress. It couldn't tell stories, adapt to native languages, or truly teach. I realized that to democratize environmental education globally, I needed to evolve from mere image classification to intelligent, multilingual storytelling. This hackathon was the perfect catalyst.

What it does

AI Visual Explorer is a serverless web application that turns any smartphone into a global AI teaching assistant. Users take a photo of a plant or landmark, and the app instantly identifies it. Based on the user's selected persona (Educational, Touristic, or Explorer) and native language, the app generates a fascinating, structured lesson. It also features a "Discovery Passport" that saves all interactions locally on the device, eliminating the friction of user accounts while preserving the user's exploration history.

How we built it

The application relies heavily on the Nebius Token Factory for ultra-low latency inference, coordinating multiple state-of-the-art models:

  • Frontend: Vanilla JavaScript, HTML5, and CSS3 for a lightweight, fast, and accessible user interface.
  • Backend: Python with FastAPI, deployed on Render.
  • Vision & Classification: We use openbmb/MiniCPM-V-4_5 as our first line of defense to analyze the image and identify the primary subject.
  • Real-time Context: The Tavily Search API cross-references the visual diagnosis with the user's real-time GPS location to pull live environmental or historical context.
  • Reasoning & Synthesis: nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B acts as the core reasoning engine. It takes the visual data and web context, adopts the requested persona, and generates the final multilingual report.

Challenges we ran into

One of the biggest challenges was handling API timeouts and AI hallucinations when users uploaded blurry or unrecognizable photos. Initially, feeding a bad image into the 30B reasoning model caused the server to hang indefinitely. To solve this, I implemented a strict "circuit breaker" pattern. Now, if the MiniCPM vision model cannot clearly identify the subject, it immediately halts the execution pipeline. It bypasses Tavily and Nemotron entirely, returning a fast, friendly fallback message in the user's language. This saves massive compute time and stabilizes the app.

Accomplishments that we're proud of

I am incredibly proud of successfully transitioning from a localized, rigid CNN architecture to a flexible, serverless AI pipeline. Integrating Nebius and NVIDIA models allowed me to build a system that not only identifies species but contextualizes them in real-time across five different languages. Furthermore, building the offline-first "Discovery Passport" using local storage successfully removed all barriers to entry for young students.

What we learned

I deepened my understanding of chaining multiple LLMs and Vision models into a single, cohesive asynchronous workflow. Working with Nebius Token Factory taught me how to optimize API calls, handle strict timeouts, and inject real-time context (RAG) into a 30B parameter model without breaking the user experience.

What's next for AI Visual Explorer

The next step is to introduce gamification elements for school classrooms—such as specific botanical scavenger hunts—and to build a native mobile app using Flutter to leverage deeper offline capabilities for remote natural reserves without internet access.

Built With

Share this project:

Updates

Submission history