T(H)NKR

Inspiration

T(H)NKR was originally intended to be a simple object detection interface that would enable users to shop for items online by simply taking pictures of them. However, we realized that there are many companies out there such as Google, with it's Google Lenses and Google Goggles, and Samsung with Bixby vision.

We realized that our former project idea was no different from what was already available to consumers. We wanted T(H)NKR to be a unified solution to all object-detection-based e-commerce applications, not only for the hale and hearty but for the special needs community as well! To solve a problem that has been viewed as a challenging feat for many, we envisioned the need for a universal application that is suitable for all demographics, hence leading to the invention of the Special Needs Artificial Intelligence Library or S.N.A.I.L. for short.

The present

Today, applications that claim to be helping the special needs community have been successful in limited ways. We hope to amplify the effect by broadening the target audience spectrum. By engaging those with special needs, as well as those, who are able, we aim to make life a bit easier for both demographics. Special needs apps today are mostly restricted to one disability. At times, inaccuracy of commonly-used Machine Learning algorithms has led to the invention of more applications and software that boast high results.

What it does

The T(H)NKR can scan and identify consumer products such as shoes, bottles, and apparel. Using custom Object Detection algorithms, the app draws their attention to online stores that sell similar products at cheaper prices. The app facilitates communication between the abled and the less-abled to create a unique experience.

Sign Language is one of the most important methods of communication for the deaf and mute today. The T(H)NKR S.N.A.I.L. eases the communication boundary between the abled and the less-abled. It features a custom-built Sign Language to Speech/Text translation engine that processes the positions of the hand in real time. Words are displayed on a screen to verbalize the wearer's thoughts.

For the blind, we use object detection to identify obstacles in their path and the proximity to possible dangers. Common examples include road-crossings, where the application can alert the wearer about oncoming vehicles and traffic, thereby minimising the risks of accidents. Another common use case would be the presence of an elevation or demotion along their path. By identifying the obstacle before-hand, an alert is sent out which draws closer cautiousness towards the situation.

We envision the app to be part of a whole suite of multi-demographic applications available for download on smartphones.

How we built it

The backbone of the application is an Object Detection engine that is able to predict the presence of common day-to-day items. With an easy-to-use user interface, coupled with the straightforward functionality, the application sets itself apart from most applications that are aimed at aiding the less-abled.

The Machine Learning libraries involve a bare python script that runs a TensorFlow Computational Graph that takes in raw pixel values of its surroundings via webcam and predicts the probability of daily items existing in each state or frame. Functioning at 20fps, the real-time translation engine is quick in its predictions and is able to give accurate outputs irrespective of the lighting condition in its environment.

The user interface was designed and brought to life using Adobe Experience designer, a desktop app that is commonly used to design web and mobile applications through active wireframing and quick prototyping.

The Speech to Text engine uses the Google Voice Recognition API as training our own voice recognition machine learning model would be too computationally expensive and costly in terms of time.

Challenges we ran into

The problems faced while making the app were seen throughout the hackathon resulting in the absence of an actual working prototype. The Mchine Learning model we built, though only 200 MiB, was too large to be uploaded to Cloud GPU clusters on many commonly-used platforms such as Floydhub, Azure ML, Google Cloud and Paperspace. Our failure in submission lies in the services we chose to use and the problems that arose due to the inability of the model to be fully trained and tested during the development phase.

Accomplishments that we're proud of

Thought the odds were against us in terms of hardware provisioning and service dysfunctionality, we are amused at the tireless hours we put in in order to get the app up and running. Despite aimlessly running around for help, we were able to get most of the training done, though the full training and testing cycle was left incomplete.

We were able to reproduce our vision for the application using a brief wireframe that we created during the final moments of the hackathon.

What we learned

Despite weeks of planning and testing of new ideas, at times, some things are not meant to work out. We learned this the hard way by failing to meet the requirements for the finals-selection. Many arguments followed by hours of careful, tedious planning were met with serious competition from the other participating teams. This is just the beginning of what could potentially be a great application of Machine Learning in a real-world use-case.

The team has actively been learning new technologies that required them to use a different methodology of thinking.

We have met a lot of people across the span of two days. We have learned as much from them as they from us. We hope that this hackathon experience has brought out the more intuitive side of the participants!

T(H)NKR in the years to come...

As of now, we aim to complete the training of the application ML model. We hope to procure high accuracy from the model and we wish to see it come to fruition. There is still a long way to go and many exciting updates to follow. This project has taught us what it feels like to collaborate and share knowledge with one another. We hope to release the application to the public so as to better benefit the community - especially those who are less fortunate than most of us.

Regards, Team T(H)NKR

Built With

• adobe-xd
• floydhub
• google-cloud
• paperspace
• python
• tensorflow