It is never easy to keep a high level of satisfaction for all kind of travelers, especially when they have a disability. This work presents a solution to help blind and visually impaired people to take public transport. The Swiss Federal Railways (SBB CFF FFS) is the national railway company of Switzerland. It is responsible for the safety and comfort of its travelers. In order to standardize the level of satisfaction of all travelers, they propose to elaborate an app to help visually impaired people to find their way around the railway station.

What it does

Our software automatically detects the doors, open doors, door handles, and door buttons of Swiss Federal trains in railway stations. The final purpose of this app is to guide the user to the closest door of the train and help him to reach the button to get inside the vagon. Optimally a voiceover or haptic system would finally give feedback to the impaired user. This functionality has however not been implemented yet.

How we built it

We adapted cutting edge models to train on the given dataset, we tested multiple possibilities such as EfficientDet and YoloV5. The final output has been obtained by combining multiple models output and adding test time augmentation to maximize the performances. Big up to Oracle Cloud! Without its computational power, we never would have been able to train such performant models. Besides, an Android application capable of helping the user with the camera stream of his phone has been developed.

Challenges we ran into

  • Deep learning is no easy task, especially when trading cutting-edge models deploying the latest versions from the edge, everything can go wrong. For unknown issues, debug might be difficult.
  • Among others, we faced issues for transporting our model to Android. In fact, ".pt" files are not directly implementable in an Android app and requires a careful compilation process of the source model. One workaround to avoid this problem would be to generate a ".tflite" model instead and then to implement it in Android using the TensorFlow API.

Accomplishments that we are proud of

An intuitive and responsive creative flow along with a varied and well-integrated ecosystem. Many state-of-the-art-models have been trained and evaluated on a custom dataset for impactful applications. Last but not least, we learned a huge amount of stuff truly mind-boggling, from custom object detection methods, to visualization tools for ML experiments!

What I learned

  • YoloV5
  • EfficientDet
  • Android development
  • Oracle cloud

What's next for BlindDetector

Being up for the competition, we focused our effort in delivering the best detection quality possible, training big models and combining their predictions. Smaller models can be ported to Android and integrated into the SBB app, in this case, we could be sincerely happy to help with such a strong cause.

Built With

Share this project: