ARLingo

Welcome Page
Language Selection
Step2
Step4
Step3
Step1

Required Information

Ayush - A student who's pursuing Computer Science currently doing research in Human Computer Interaction and loves to play badminton! Leia - A Computer Science student @ UC Berkeley very passionate about XR. I aspire to create multiple tech companies that get us closer to a great future and empower diverse people from all around the globe. Manchen- An Electrical Engineering PhD student @ Stanford working on Optics and Photonics. Passionate about implementing XR Track - XREAL and XR Trailblazer Team Location - 016 Basement Huang Hardware - iPhone (LIDAR Sensor) Development Tools - xCode, OpenAI APIs, Swift, RealityKit, ARKit, and Sketch Fab (3d Models)

Inspiration

When growing up I learned English on my own by singing Justin Bieber’s baby.

From then on, I really enjoy learning new languages. However, when I visit a foreign place I cannot always remember the meaning of some objects. So, I need to pull out my phone and go to the Translate app, and it’s actually not that fast to use. I need to find the app, get into the selection tab, and it doesn’t recognize the language I wanna translate to. It’s just a mess.

So what if instead of that I would just open my camera and touch the screen to the object I wanna translate and voilà. I can even select a button to translate all items of a similar size around me, and best of all, it takes less than 3 seconds to do everything.

Do you know that feeling when you see an object (in the real world or your mind) but don’t remember the word in the language you’re trying to learn

What it does

Arlingo helps millions of people around the world to translate anything they want around them. We can help everyone who is in a new country, every tourist, every immigrant, even help the insert number of people with hands and hearing disabilities.

Arlingo allows you to save time in three simple steps: Open ARlingo app; Select the language to translate to. If it’s already selected, it will open the camera automatically. Touch the object on your screen and the translation will overlay it.

How we built it

We have used ARkit and SwiftUI to implement the frontend and backend of ARlingo. As well, we have used OpenAI’s API GPT and WHISPER to recognize the object, translate it on your screen and listen to it.

Challenges we ran into

Ideation took some time to figure out if we should use music in this first implementation of ARlingo, and we decided to rather focus on translating objects because it was much straightforward and more attractive for the user. Choosing the right AI model and API also to choose which one to use. Google Cloud Translation API seemed good, however the charging amount exceeded out budget, that’s why we chose to use OpenAI’s API for object detection and translation.

One of the first hurdles we encountered was ensuring that the chosen AI model and API could seamlessly integrate with ARKit and SwiftUI. The diversity of device specifications and operating systems among our potential user base meant we had to be meticulous in our selection to guarantee a smooth, universal user experience. Balancing these technical compatibilities while maintaining an intuitive user interface posed a significant challenge.

ARlingo aims to cater to a global audience, which brought forth the challenge of finding an AI model capable of recognizing a vast array of objects and accurately translating multiple languages. Testing and validating the AI's effectiveness across different languages and objects was an exhaustive yet necessary process to ensure inclusivity and functionality for all users. In an era where user privacy and data security are paramount, we faced the challenge of ensuring these aspects while leveraging powerful AI and AR technologies. Selecting APIs that complied with stringent privacy standards and implementing robust data protection measures were critical steps in our development process.

Accomplishments that we're proud of

We are immensely proud of several key achievements in the ARlingo project.

Especially for the amount of people we can help.

One of our major accomplishments is the innovative use of AR and AI, successfully integrating ARKit and advanced AI models to create a real-time, user-friendly app for object recognition and language translation. The global impact of ARlingo, with its ability to support multiple languages and recognize a wide range of objects, highlights our commitment to inclusivity and accessibility. We take particular pride in making language learning and translation accessible to people with disabilities, showcasing our dedication to creating technology that serves everyone. The seamless, efficient, and enjoyable user experience in ARlingo is a testament to our focus on user-centric design. Additionally, the successful collaboration among our team members, each contributing unique skills and perspectives, has been a rewarding and enriching experience.

What we learned

We learned how to use the ChatGPT’s API to translate the objects that show up in the user’s screen. We also got familiar with AR applications and gaming implementations. The journey of developing Arlingo has been as educational as it has been challenging. Throughout the project, we gained extensive knowledge in AR and AI technologies, especially in the integration of complex systems like ARKit, SwiftUI, and OpenAI's APIs. We encountered various challenges, from technical integration to addressing user privacy concerns, which honed our problem-solving and critical thinking skills. The project underscored the importance of keeping the end-user at the center of our design process, resulting in a more intuitive and effective product. It taught us the value of adaptability, being ready to learn, and pivot our strategies in response to new findings and feedback. Moreover, this project reinforced the value of teamwork, diverse perspectives, and collaborative problem-solving.

What's next for ARlingo

Looking ahead, we have exciting plans for Arlingo. We aim to expand our language options, focusing on including more less commonly spoken languages to broaden our app's reach. We're also working on enhancing the app's object recognition capabilities to include more complex and varied items. Improving accessibility features is another key focus, as we plan to introduce more features to aid users with different disabilities, making Arlingo even more inclusive. Actively incorporating user feedback is crucial for us, as it helps refine and improve the app's functionality and user experience. Exploring partnerships with educational institutions and language learning platforms is also on our agenda, as it will help expand our impact. Additionally, we're committed to staying abreast of advancements in AI and AR to continuously improve Arlingo's capabilities. Arlingo is more than just an app; it's a growing platform that bridges language barriers and connects cultures, and we are excited to see where it takes us next.