Inspiration
您好！欢迎来到美国！Wilkomen! Schönen Tag! Willkommen in den USA! Confused? Frustrated? Understand a bit, but not completely? That's how we felt coming to the United States as international students.
We come here in pursuit of a better education and have incredibly hopes for our future. However, the language barrier that we face on a daily basis has been a crippling bottleneck towards our success. We want to get better at the language–but it's incredibly difficult to do so given our limited understanding of the language. As a result, our primary aim for this software is to allow students, who may be in a similar situation as us, to improve their language skills by identifying–and translating–items that may seem foreign to us.
We understand that mastering a language does not happen overnight. It's a long and grueling process. However, we also understand that if we are able to associate images, as well as items, to a certain language, we would be able to better remember a specific word in its respective language. As a result, this on-demand "point and shoot" object translation software would be incredibly beneficial to us–not only from a practice point of view but also from a "word/phrase retention" point of view.
This application is not only limited to users in the United States. This application is aimed at foreign students, business professionals, and even aspiring language learners on a global scale. This software has helped me greatly–as I am on my path towards learning the German language–and we all hope that this software would be able to benefit others as well.
What it does
Detect, identify and translate.
This application is a web application so it can be easily accessible to everyone. If a user sees an object, or word*, that seems foreign to them, the user can simply point their camera to that said object and the application will automatically provide the object's name (Button–up shirt --> Hemd, Tie --> Krawatte) in the user's desired language.
The translation occurs in real-time and DOES NOT require the user to snap an image of the item. However, if the user wishes to do so, they are more than welcome to save the results onto their mobile device for their future reference*.
*These are features that are currently being developed
How we built it
Our team utilized C++ and Python to develop this application. To extensively train our dataset, we used OpenCV's training software to help classify what would be considered a "positive" and "negative" representation of that image.
For our object detection software, we heavily relied on the use of Python and its libraries (opencv-python, Flask, Threads, numpy, googletrans) to develop this.
In order to make our software accessible, we developed this application into a web application and relied on the use of Flask to put this software on the web. At the current moment, this web application is still hosted on our local machines–so we could further iron out some stability issues and further increase the speed, and performance, of our application.
Finally, we chose Visual Studio Code as our primary IDE, GitHub as our version control software, OpenCV as our CV software, and Stack Overflow as our 24/7 support staff. We are incredibly grateful for everyone's support!
Challenges we ran into
Originally, our plan was to develop this application exclusively on Android. However, we ran into a ton of issues, notably OpenCV's tough initial setup process and needing to recompile OpenCV binaries again on our own. As a result, we ultimately switched to Python–and that worked like a charm!
A member of our team actually has access to a dedicated server at home–but had trouble connecting that server to the front-end portion of the web application. Worse, we were not able to utilize our device's onboard camera to take advantage of remote video processing. Thanks to Python's extensive network of libraries, we were able to link our Python object detection program to the web server (Flask) and our web application worked well from there!
Segue to web servers, it was incredibly difficult for us to choose which web server is the most suitable for our project. We knew that Apache was the industry standard–but the setup time was simply too long and complicated. As a result, we decided to pivot towards Flask. However, this transition was not without problems either. Flask has its own unique HTML "href" handling and syntax and that was quite the learning curve–as members of our team have very limited experience with website design and back-end front-end integration.
Accomplishments that we're proud of
Honestly, getting the web server to work. The fact that we were able to display an image on the web application was nothing short of a miracle. We spent an enormous amount of time learning the ins and outs of the Flask library (apparently, its HTML style of coding is incredibly different than what we're all used to!).
Understanding OpenCV. Computer Vision, specifically with OpenCV, was a new technology that we were all able to pick up. We are proud of what we learned and are more motivated to learn more about this awesome technology!
Finishing in time. As the previous item suggests, we were able to complete our code within a span of 24 hours–and didn't ask for an extension. This is awesome!
What we learned
Communication is incredibly important. Without communication, we will have no idea where everyone is and chaos will quickly ensue.
Being motivated and staying on track. We wish we could say that our project ran incredibly smoothly–but that statement could not have been further from the truth. We ran into issues literally everywhere and there were times when we wanted to give up. However, because we kept an eye on our end goal and were motivated to succeed in our project, we were able to complete our web application and we're super proud of that!
Getting rest...and Red Bull. Red Bull gives one wings–and a ton of energy. We don't drink Red Bull, but thanks to that energy drink, we were able to literally stay up for 24 hours. I think there isn't much to say more about that!
What's next for Smashing Barriers
We want to further develop this application and eventually launch this application on a broader basis. We are currently in the process of working on some exciting features–such as text recognition and translation and saving images from the real-time translation–and expect these features to be rolled out, and eventually launched sometime incredibly soon!
Log in or sign up for Devpost to join the conversation.