Inspiration
During LauzHack opening, SBB presented their app "CFF inclusive" which helps people with sight problems. We were inspired by this idea and the fact that we could build on it to help train users achieve things that seem simple (finding a door or a button) but which are hard for them.
We decided to focus on option 2 of the SBB challenge: the implementation of the user experience.
What it does
In its current state, the app picks up the most prominent object on screen and circles it with a bounding box. Depending on the position of the bounding box on the screen (left/right/far/near), the user is given voice instructions at regular time intervals to direct it closer to the object. Once the object takes up a significant portion of the screen, the app considers that the user is close enough to hit the button and says so.
How we built it
We built it using an open ML framework for android: ML Kit: https://developers.google.com/ml-kit In its default state, the framework is set to detect the most prominent object on the phone screen.
We focused on the user interface implementation and left the ML part out of it. In a second step, the ML part could be wired to the SBB door recognition model so that the app would not simply pick up the most prominent object, but the train door and guide the user to it.
Challenges we ran into
- Running and adapting the existing ML framework on our phone was not simple.
- ML image output and phone screen have different resolutions so we had to take it into consideration when using bounding box position to guide the user.
Accomplishments that we're proud of
We are proud of having achieved a (little) something in such a short time. Although the project is complex (live image recognition), we managed to use simple geometry to guide a user with voice.
What we learned
Don't try to build everything from the ground up, use existing frameworks.
What's next for SBB door guidance
Plug it to the SBB door recognition model to effectively guide the user to the door!
How to run the code
Open the project with Android Studio, with root folder being "vision-quickstart": android/vision-quickstart . Then you can build and run it on an android phone that you connect to android studio via USB.
Log in or sign up for Devpost to join the conversation.