The Covid 19 pandemic has left people at home in Lockdown across various countries. This has made finding help for jobs like assembling furniture, repairing appliances, etc. very difficult. With work from home becoming the norm people had to start using websites which they were not familiar with, facing a lot of issues navigating them. We felt a need to address these issues and came up with Do AR, an application to not just provide tutorials to complete repairs, build stuff but also help navigating through websites easier.
What it does
The application helps one to become more independent in performing various task at home. Using Augmented Reality to make the entire process more enriching and easier to follow. Do AR has 3 main features:-
1) DIY Helper
The DIY Helper guides users step by step in order to complete various activities like making a Book Shelf, repairing AC, etc. The steps are displayed in Augmented Reality to make the tutorials realistic and easy to follow. The progress through steps is recorded so a user can start the tutorial again from where they left.
2) Webpage Navigator
The Webpage Navigator helps the user to navigate through popular websites. The feature makes use of custom image targets which are defined by the user. A drop-down box contains the various activities which can be performed, selecting an activity highlights the button which performs the same.
The Visualiser provides various 3D models of objects (eg:- AC, hammer, etc) in exploded view with labels. This helps in identifying the various parts of an object.
There is also a Leader Board that keeps track of the progress made by various users in DIY Tasks. It helps motivate others to perform the same.
How I built it
1) Unity3D with C#- This served as the main layout and guided the workflow of the app. It was chosen one of the best tools out there for AR development.
2) Firebase- This served as the backend for our app for things like LoginAuth and Player Progress.
3) WatsonSDK- This served as our voiceover tool to create recorded audio and use voice commands making our job a lot simpler and not needing us to make a new voice recording for every thing.
4) Blender- The tool used to make all the 3D models, Labels and Animations.
5) echoAR- Our 3D Model storage online used for visualiser.
6) Vuforia - This was for AR support using Unity.
Challenges I ran into
1) As there were no pre-existing 3D animations of the DIY tutorials, we had to make the models and the animations from scratch by looking up the existing tutorials.
2) As the Webpage Navigator uses user defined image targets, calibrating them and adjusting their sensitivity was challenging as the quality of custom image targets are poor.
3) Adding voice control to our DIY tutorials was a challenge which was soved by using WatsonSDK to make speech to text.
4) We had to create labelled models from scratch for the visualiser.
5) Creating a Leaderboard required creating a DB to make it possible to track progress.
Accomplishments that I'm proud of
1) A completed login using Firebase.
2) A completed voice controlled DIY tutorial was created.
3) Completing a user defined AR Image Target for the WebPage Navigator
What I learned
1) How to add voice commands to unity.
2) Learned how to use echoAR for storing Models online.
3) Built AR app using Unity.
4) Using State Machines in Unity to and intermediate stages in animations.
What's next for Do AR
A tutorial marketplace can be introduced in the future to improve the quality, provide more and specialized tutorials for the users. A chatbot can be added to the tutorials to answer frequently asked questions. Various websites, software, and different interfaces(such as ATM machine, etc) can be added to the navigator to increase its utility. Further this can be used by different companies as an alternated for the traditional manual booklet.