ARcademy

Inspiration

Our team frequently uses web learning apps, such as Kognity and Khan Academy. Though these apps provide sufficient learning material on basic concepts, they tailor mostly to auditory and visual learners, using only text and images. Kinesthetic learners, however, may find it more difficult to retain information and visualize more complex concepts -- especially in engineering-based subjects, in which hands-on experience is essential to success. However, hands-on experience doesn't come cheap with material and tuition fees, and we were inspired to find a solution to this through our AR environment, which provides modifiable 3D-simulated lessons. Our solution is engaging, cheaper, and accessible anywhere, anytime.

What it does

Students can access a variety of courses from our easily accessible online library. Users can then plug in our Arduino hardware component and turn on our gesture recognition software to control 3D simulations in AR. We provide an AI ChatGPT-based assistant to aid students in their learning in real time.

How we built it

In this application we used WebXR for easy AR integration. We built the website using react and cite styled with tailwind CSS. The pose and gesture detection was built using Mobilenet-SSD model and deployed in react. We then used transfer learning to load our own custom gestures. We also used the Chatgpt API for our virtual chatbot, and Arduino was linked to React app for control of AR. 3D models were created from scratch using NomadSculpt, and imported into Echo3D to connect to the AR component.

Challenges we ran into

We had difficulties in training the gesture recognition on custom data. Additionally, we had trouble connecting the back-end to the front-end -- for example, connecting the ChatGPT assistant to display a React component, and also creating the finger mesh for detection from back-end to front-end. Getting the AR objects to change with gestures and learning to navigate AR environments proved to be a challenge as well.

Accomplishments that we're proud of

We are proud of creating a gesture recognition app that is capable of identifying a variety of gestures and connecting it to the front-end. We are also proud of being able to successfully implement the ChatGPT API into our React application. This was our first time touching web development, and thus take pride in our success in using the React router DOM to connect pages together and deploying our website on Vercel.

What we learned

In the creation of this project, we learned many valuable lessons on how to utilize gesture detection, ChatGPT API and designing custom components in React. It was also some of our first times attending a hackathon, and we got exposed and adapted quickly to the environment and workload while making use of our previously-learned coding skills.

What's next for ARcademy

We plan to improve our gesture recognition's accuracy by importing more training data. We also plan to expand to different curriculums and programs such as AP and IB, both which could benefit greatly from the hand-on experiences. Our team believes that not only engineering students can benefit from our AR environment, and thus we plan to expand to other topics such as History and Literature as well.

Built With

• arduino
• c++
• chatgpt
• echo3d
• javascript
• mobilenet-ssd
• nomadsculpt
• react
• tailwind
• transfer-learning
• webxr