PlantPedia

• 

  Example of PlantPedia running with a common house plant

Inspiration

Geby (a team member) - "I was a member of my community service club called BreenBelt Rotary which does a community service act by building victory gardens - gardens similar to community gardens in communities dealing with food insecurity or conditions of not having enough or adequate food for a healthy life. After building the gardens, we realized the community was not able to sustain the garden since they didn't have enough knowledge of plant life and how to take care of plants in general. Moreover, farmers in Florida in the wake of multiple hurricanes have struggled to adequately prepare and care for their plants in the best way possible." Geby's experiences are the main reasons we decided to build PlantPedia.

What it does

PlantPedia is an AR Plant Doctor that leverages Plantnet, a plant identification API, Wikipedia, Google's Gemini LLM, custom neural networks, and custom logistic models to curate plant growth guidelines and ensure that your plants are growing healthily.

How we built it

We used Snap AR's Spectacles Interaction Kit along with Script Component integrations to build the AR frontend interacting with plants and users. We created a custom API that received a frame of the plant captured by the user, and ran several analyses on it. Firstly, it sends the picture of the plant through a plant-identifying API called plantnet-api, then scrapes Wikipedia and combines all this information to prompt Google's Gemini to give the user advice on how to grow their plants. Finally, we display the results in world-anchored containers on Snap AR's Spectacles.

Challenges we ran into

Snap AR’s Spectacles are relatively new, and the documentation is still evolving. This posed several challenges, as there were many gaps in the documentation that we had to work around. Despite this, we were able to interface with multiple APIs and even display custom visualizations on the Spectacles. Geby also developed custom algorithms and neural networks trained to improve plant health predictions.

Accomplishments that we're proud of

We successfully created an app that combines AR technology with sophisticated algorithms to help users better understand and care for their plants. This includes interfacing with various APIs, working around the limitations of early-stage AR hardware, and training custom neural networks for plant health assessment.

What we learned

We learned how to work with cutting-edge AR technology, navigate the challenges of limited documentation, and effectively combine multiple APIs to build a functional and creative app. We also gained a deeper understanding of plant care and how machine learning can assist in this field.

What's next for PlantPedia

We plan to expand PlantPedia’s functionality by integrating more plant care databases, enhancing the neural networks for more accurate health assessments, and improving the user interface on Snap AR's Spectacles. We also want to make the app more accessible to gardeners and farmers in regions dealing with climate-related challenges.

Built With

• api
• flask
• gemini
• javascript
• llm
• python
• tensorflow
• typescript