Inspiration
Some members of our team, namely Ryan and Gavin, have felt a disconnect between them and their mother tongue. The fact that they don’t know the language of their parents, nor grandparents, restricts them from getting to know them well, even if it's close family. Due to these restrictions that have been felt first-hand, the group felt that overall Beyond Language was a fantastic match, especially with new technologies and amazing movies like Project Hail Mary that have come out recently.
What It Does
This database contains many things. First, the main function: streaming. The database asks to access the user’s camera, and upon the user’s input/command to record, the camera has an AI model that identifies an object and uses any sort of recognizable or very obvious sound to associate a word from a foreign language to a noun (the object).
Then, the database shows the user when a word has been registered and continues to encourage the user to expand and grow their database to eventually translate back and forth between the foreign language and English. For example:
- Speak the alien/foreign language into the program, and the program speaks the message back to you in English.
- Speak in English, and the program translates it into the learned language.
How We Built It
The project was first divided into three sections:
- Frontend — worked on by first-time hackers Gavin and Omar to create a clean and functional UI.
- Backend — created by Chandan to process a webcam feed and identify objects.
- Server Infrastructure — a Raspberry Pi server created by Ryan.
Once all of these components were in a suitable state, they were merged together into a complete project. Communication was paramount to ensure that all features meshed well together. In the final stages of the project, troubleshooting and small bug fixes were underway, along with niche features like a phonetic alphabet system.
Challenges We Ran Into
Both new hackers had to spend a lot of time figuring out how to make a suitable and functional UI. Workshops that gave them experience were very helpful, but plenty of work was still ahead, with constant errors and fixes required.
On the other hand, the server section faced trouble because they didn’t have an HDMI cable and had to learn multiple terminal commands through trial and error to make everything work.
Finally, the backend had difficulties with the webcam. Even though the video feed worked, the microphone would not function correctly, which cost Chandan over an hour of debugging and troubleshooting.
Accomplishments That We’re Proud Of
The system integration between hardware components like the microphone, webcam, Raspberry Pi server, and image recognition AI are all brand-new technologies that we explored during this Hackathon.
While the webcam and microphone were plagued with problems, successfully integrating a live video feed into a web application was a huge step forward compared to simple text inputs in a debug console or even a game engine. Having real-life hardware features directly incorporated into software was a major accomplishment for the team.
What We Learned
We learned a lot about the Hackathon experience, especially with two rookies attending their first Hackathon and learning how to work together effectively while managing time efficiently.
As this was a two-day Hackathon, we learned the importance of utilizing our time wisely; otherwise, we would have had to work late into the night to finish the project.
What’s Next for ExtraTranslational
Right now, ExtraTranslational is entirely local, but we are looking forward to adding more sophisticated software capable of identifying actions and more abstract concepts.
AI integration with different models could greatly expand our scope and ensure more accurate assessments of situations and conversations.
Built With
- bcryptjs
- css
- express-validator
- filereader
- helmet
- html
- javascript-(single-page-app-in-client/index.html)-tensorflow.js-+-coco-ssd-(in-browser-object-detection
- jsonwebtoken
- localstorage-backend-/-tooling-node.js-+-express-(https-static-+-rest)-sqlite-via-better-sqlite3-dotenv
- mediarecorder
- morgan
- via-cdn)-web-apis:-webrtc-getusermedia
- web-audio-api
- web-speech-api-(where-supported)
Log in or sign up for Devpost to join the conversation.