Flora 4 Future

Flora4Fauna
Screenshot of our Prototype
What we developed
Do you remember Jeff?
The educators 1
The educators 2
Tech Stack

Inspiration

As previously, our goal was to build something that combines current innovative technologies, create something fun and inspire people to think about climate-related issues. As Jeff Basil was quite popular in the previous hackathon we attended (https://devpost.com/software/jeff-basil-against-the-world), we now gathered all his friends and created something fascinating that primarily focuses on education!

What it does

In our application you can go for a stroll in your apartment with your mobile phone in your hand. You will see on the screen what your camera sees. Once climate-relevant objects appear in the view, Jeff's friends show up to educate you about the object's relevance for climate and how YOU can help to fight against the climate crisis. Jeff's friends spawn right next to the object and educate you on how to act more climate-friendly concerning that object. They stay where there are once they have been spawned and you can come back to them and you can even walk around them and look at them from different angles. We implemented an augmented reality (AR) app, that utilizes the phone's camera, analyzes the image, and provides augmented reality information.

A bit more detail

We continuously stream the phone's camera image to the screen of the phone and analyze the scene for climate-related objects. These images are processed by the Huawei ML Kit API, which allows performing AI-powered object detection in the scene. Each friend of Jeff (or Jeff himself) has his preferred object and topic to talk about, so once an object is identified, that a plant can talk about, we recognize the object, we position the respective plant at that position and it starts an education lesson to teach about climate-related issued concerning the detected object. After walking through your home you will end up having Jeff's friends all over the place.

This is ideal for children to learn about the climate in a fun and interactive way!

What our educators show

These are the message that Jeff's friends show upon detection of the object in the leftmost column.

Detected Object	Who	Hint
trash can	Jeff Basil	Did you know, that until 2050, plastic production will cause 52.2 Giga-tons of $CO_2$? Reduce your plastic!
radiator	Geogina Bush	Please turn off your radiators when opening windows or doors.
light switch, lamps	Jim Parsley	You can save a lot of energy by replacing classical light bulbs to newer technologies.
kitchen	Vroni Pepperoni	Instead of turning on heating in the winter you can just prepare very spicy food.
avocados, mangos, bananas	Giacomo Avocadio	The consumption of fruit from oversees should be reduced. Why not switch to regional products like apples, pears, or strawberries. These taste pretty awesome as well!
shower	Crocus Pocus	try to optimize you shower time, this will save a whole lot of hot water
bathtub	Crocus Pocus	make taking a bath an exceptional event and use the shower more often
car keys, cars	Elon Mush Room	Is it less than three miles? Just use the bike instead of your car. For longer trips switch to electric mobility

How we built it

We used:

Android Studio for app development
Google ARCore as augmented reality engine
Huawei ML Kit Api for AI-powered object detection
Kotlin for programming

Challenges we ran into

At first we tried to use the AR engine provided by Huawei Developers. However, we could not make it run as it is requiring some engine that can only be downloaded from Huawei AppGallery when connecting from China. We do not have Huawei phones, so we can not use the AppGallery on our android phones (without installing ugly extra apk's). We, therefore, switched to Google's ARCore AR engine which was working out of the box. Also, the Huawei ML Kit API is quite a challenge, would be cool if the documentation could be improved.

Accomplishments that we're proud of

We gathered all of Jeff's friends. Have you ever seen talking plants? Let us introduce you to Vroni Pepperoni, Giacomo Avocado, Jim Parsley, Crocus Pocus and his brother Lotus Pocus, Georgina Bush, Elon Mush Room

What we learned

We enjoyed doing hands-on app development a lot! We have used Kotlin for the first time and realized how cool the language is. Additionally, it was very pleasing how easy AR apps can be built with the ARCore engine.

Summary

We used an end-to-end system, where both ends were connected. Then we added the human back in the loop and of course, split both ends.

What's next for Flora 4 Future

Once the climate is saved, Jeff and his friends can finally enjoy their favorite leisure activities. Most importantly: Yeast loves Fortnite and pickles love football.