Earth Scanner

Earth Scanner educates us all about our environment in a fun and interactive way, making use of AR, ML, AI, Deep Learning, and more! Learn about the plants & animals around you without feeling bored!

Comment 1

Inspiration

During a particularly boring biology class, I thought, "Is this what education has come to? Monotonous learning and memorizing facts that don't seem to have any significance?" My thought processes were interrupted by my teacher, but it stuck with me. Me and my teammate have always wanted to make education more fun and interesting for children like us, but also for anyone in the world! The environment around us is such a captivating topic: You don't know anything about the plant next to you. Or the animals around you. Or maybe you do, but it was boring to learn about it! That's why we decided to make Earth Scanner, a solution to this problem!

What it does

Earth Scanner has 5 main features: Scan Animals, Scan Plants, Augmented Reality, Knowledge Hub, and Quiz Game. The scan features allow you to take a picture of an animal/plant and get all the information about it! The Augmented Reality, one of the coolest features, lets you see 3D animals floating around in your room thanks to AR technology! The Knowledge Hub provides information about dangers of plants & animals and taking care of them. Finally, the quiz game lets you test your wits with some fun questions!

How we built it

We built the main parts of our project with Python. The main file, Project.Py, connects to our other programs for each feature via the subprocess library.

Scan feature: The camera code uses cv2 to take a picture and save the frame of the image when the user presses the space key. The image is saved locally in the file 'object.jpg'. We decided to use Teachable Machine, a free software that allows us to train our own models, along with ml5js, a machine learning library that can read Teachable Machine's code, to recognize our animals/plants. Teachable Machine is easy and fast to use, which is why we chose it instead of something like Tensorflow, which would require changing our Python version, updating our PATH, etc. But since Teachable Machine works with javascript, we made a p5.js sketch file connected to an index.html which would send our image to Teachable Machine and get the results via ml5. Then, we used the webbrowser Python library to open our HTML file. Additionally, to avoid any CORS errors, we had to read the image uploaded on HTML from p5.js instead of just loading the image in function preload(). We used our problem-solving skills on an infinite level: There was little to no documentation to do all this! Finally, we connected a RapidAPI animal information API in p5.js on our own experimentation by converting Python to Javascript (as the API's code for retrieving information wasn't given for p5.js!) and got the information of the scanned animal/plant! It is a beautiful spiderweb of technologies!

Augmented Reality: We used WorldCAST.io and Sketchfab to create AR projects for animals and plants. We uploaded 3D models of plants and animals from Sketchfab to WorldCAST, and added the QR codes for our WorldCAST projects into our Python.

Knowledge Hub: We used a simple dropdown menu in Tkinter and added information from trusted and verified websites for the given subjects in the Knowledge Hub.

Quiz Game: Essentially, we created a JSON file with questions and answers, and used it in our Python code to provide the quiz. The Quiz class has various functions that set the questions and answers based on the JSON file, and it uses radio buttons to get the answers from the user. At the end, we used a Tkinter messagebox to display the user's results. We do not disclose the answers to make it fun for the user to find out themselves!

Challenges we ran into

It was an excruciatingly long, tedious, and ineffably difficult path to building our project. First of all: We used 3 languages!!! And they aren't often related with each other, let alone combined! We somehow connected p5js (Javascript), HTML, and Python in such a way that when you take a picture in Python, the picture is saved locally and loaded in HTML, from which p5js extracts the image and classifies it along with using an API to get more information, adds the information and classification on HTML, and then runs the HTML from Python! We inevitably ran into loads of errors whose solutions weren't easily (sometimes not even at all) available in documentation, or anywhere on the internet. So, we had to make our own workarounds and find a way through. We originally thought that we could get the classification from p5js and use it in Python to get the information and display it in a Tkinter window, but p5js didn't provide the functionality to save our classification in our own file. So, we had to completely turn around our plan and use p5js to get the information too, and keep the feature on HTML. That was one of the toughest challenges. Additionally, it was difficult to create the Augmented Reality, but as we had a tour of the HCL Innovation Labs, we had some insights on how to use 3D models in AR and stumbled upon the WorldCAST.io platform. This made our feature work like a charm! Making the quiz was a long process, as even though the technology was within Python, the logic was tricky to figure out. JSON saved the day! The Knowledge Hub feature was the easiest to make, but finding the real websites was hard. We did the work for our users!

Accomplishments that we're proud of

We'll split up our personal views on the accomplishments we have made in our journey: Vindhya: I coded and made the Knowledge Hub section and researched all the information on it, for example, Information on plants, animals, and insects to be watched out for, Caring for plants, Animals and safety, etc. I'm proud of learning so many technologies as I'm not a highly experienced coder. Tkinter, AI, ML, and even Python are relatively new to me, and I'm proud of figuring them out for the purposes of our program! Anika: I am so proud of Earth Scanner. I learned so much about technology, and even went as far as learning some basics about deep convolutional neural networks! Teachable Machine is a software I'm familiar with, but I'm not very advanced in Python, or p5.js, or HTML, or API usage in text languages. I learned a lot, like how to use radio buttons, classes in Python, running a Python file from ANOTHER Python file, using p5.js and HTML together, parsing JSON results, getting JSON properties and presenting them in a neat and readable format, using local files while avoiding CORS errors, making professional UIs with Tkinter, USING AUGMENTED REALITY, creating and downloading 3D models, the logic behind how AR works, extracting HTML objects from p5.js, AND EVEN. MORE. My greatest accomplishment is how much I've learned, and how I put everything together with such little help (other than my teammate, who always boosted my moral!) I'm especially proud of developing the entire project with Vindhya in the timeframe, along with school, exams, and little-to-no experience with most of the technology! All in all, this was an INEFFABLY rewarding experience!

What we learned

  • Image recognition
  • The technology behind image recognition (deep convolutional neural networks)
  • Using p5.js with HTML
  • Extracting HTML elements with p5.js
  • Getting results from APIs
  • Parsing JSON text
  • Showing JSON text in a presentable format
  • Using augmented reality
  • Creating and downloading 3D models
  • Making professional UIs with Tkinter
  • Saving with p5.js
  • Making classes in Python
  • Accessing other files with Python
  • The logic for making a quiz in Python
  • Using global vs. local variables
  • Using Teachable Machine models with p5.js
  • USING VSCODE

What's next for Earth Scanner

We plan to use better image recognition model technology than Teachable Machine to identify all animals/plants and their species in detail. We will add better functionality to the quiz by adding more questions and answers, along with personal feedback. We will make the AR more interactive and make the entire project available for phones and tablets as well. We will add more information in the Knowledge Hub and add a search bar to improve navigability. The project has a lot of potential and is highly feasible and practical: Schools can implement this with their students to ameliorate teaching methods, and we can even take it to a point where it can provide suggestions for taking care of individual plants or animals. Scanning plants and animals would become a whole new trend in the market, and we would never be left in the dark again.

Built With

Share this project:

Updates