Inspiration
The inspiration for our project, NounExist, came from the common struggles language learners face, particularly when trying to expand their vocabulary in practical settings. Traditional learning methods often feel disconnected from real-world use, and we wanted to bridge this gap using augmented reality (AR) and artificial intelligence (AI). By merging these technologies, we aimed to create a tool that not only supports vocabulary acquisition but also enhances it through interactive and contextual learning experiences.
What We Built
NounExist is an AR and AI-assisted educational application designed for Meta Quest 3. It enables users to identify objects in their environment and learn their names in various languages. Users perform a simple hand gesture to frame objects in their view, similar to taking a photo. The app then recognizes these objects using AI and displays their names in the selected language, effectively turning everyday moments into learning opportunities.
How We Built It
We developed NounExist using the following technologies and methodologies:
- Unity: Chosen for its strong support for AR development, facilitating the integration of complex graphical interfaces and real-time interactions.
- TensorFlow: Employed to train and deploy our custom object recognition model, ensuring the app can accurately identify a wide range of everyday items.
- Google Translate API: Integrated for quick and reliable translations across multiple languages, enhancing the app's educational value.
The development process involved rigorous testing in diverse environments to ensure the app's functionality under various lighting and background conditions.
Challenges We Faced
- Object Recognition in Variable Conditions: Ensuring the AI could reliably identify objects under different lighting and backgrounds was challenging.
- Gesture Recognition Accuracy: Developing a gesture recognition system that was both intuitive and consistently reliable across all user interactions.
- Performance Optimization: Achieving a seamless AR experience on a standalone VR device required significant fine-tuning of the app’s performance, balancing resource-intensive AI tasks with the need for a fluid user interface.
Accomplishments We're Proud Of
- Interactive Learning Tool: We successfully created a functional AR application that makes learning new vocabulary engaging and practical.
- Integration of Advanced Technologies: Our effective use of AR, AI, and real-time translation technologies in a cohesive application was a significant achievement.
- Positive User Feedback: Early testing has shown that users find the application both useful and enjoyable, which reaffirms our project's value.
What We Learned
This project enhanced our skills in several key areas:
- AR and AI Application Development: We gained deeper insights into developing for AR platforms, especially in integrating AI for real-time processing.
- Problem-Solving in Tech: We learned to navigate and resolve the complexities involved in combining multiple advanced technologies into a single application.
- User-Centered Design: Focusing on the user experience helped us refine our application to better meet the needs of language learners.
What's Next for NounExist
We plan to continue improving NounExist by:
- Expanding the Object Database: To include more objects and thus support a broader vocabulary.
- Adding More Languages: Increasing the number of languages supported by the app to reach a wider audience.
- Enhancing User Interaction: Further refining the gesture recognition system and exploring additional interactive features such as voice commands.
Log in or sign up for Devpost to join the conversation.