-
Demo of Blink, describing images and answering questions.
Inspiration
For the 43 million blind individuals in the world, their lives are deprived of something so very crucial to the human experience: vision. They can’t see the faces of their loved ones, the signs on a street telling them where to go, and their connection with their beloved by embracing the world around them. Beautiful moments are unseen, experiences in life are never wholly felt, and context is censored, obstructing their ability to make fully informed decisions. Blink changes this.
What it does
Blink is an advanced mobile app that leverages cutting-edge AI technologies, including MiniGPT-4, to revolutionise image analysis and user interaction. Through a simple double-tap on the screen, users capture images that are then comprehensively analyzed by the AI to provide an auditory description. Tapping & Holding the screen enables users to ask follow-up questions about the image.
Blink empowers the visually impaired by offering them an independent means to perceive their environment.
How we built it
Blink's user-interface is simple and incredibly easy to use, catering to the visually impaired. The app is driven by Flutter and Dart on the front-end, and Python and Flask in the back-end. It leverages the Replicate library to curate an interaction history, and facilitates seamless communication with the MiniGPT-4 API. Articulate responses are vocalized from the AI model, and the app showcases on-screen outputs in real time.
Using this innovative blend of technologies, Blink redefines accessibility, exemplifying a holistic approach that marries innovation and usability for an inclusive user experience.
Challenges we ran into
MiniGPT-4 is a massive transformer model with over 13 billion parameters. Running such a huge model is not possible on a smartphone, thus we needed to use cloud computing to facilitate it, something that would be detrimental to the environment.
Instead, in order to align the model with green initiatives, we decided to leverage the Replicate Library, which hosts Nvidia A100 (40GB) GPUs, to run our model in an eco-friendly manner.
The other challenge we ran into was connecting the images taken by the phone to the image file needed by the back-end. We solved this problem by uploading images to a third party server, getting their URL, and giving the resulting URL to the MiniGPT-4.
Accomplishments that we're proud of
We take immense pride in taking one step towards a society in which visually impaired individuals can live and interact seamlessly. AI technologies have revolutionized many industries, and being able to use these new models to create a positive impact in society is exactly what stands for. Our project is a stepping stone in the journey of influential AI apps that will accommodate individuals with accessibility needs.
What we learned
We learned about using MiniGPT-4, which has been released for only 4 months. We also learned how we can work with different cloud technologies and create an app that is simple and intuitive.
What's next for Blink
Blink possesses immense potential to continue to assist individuals in improving their lives. Looking forward, we want to enhance the algorithm to decrease the app’s response time. We also want to find ways to enhance its usability for the general, non-visually impaired user, to expand our user base and improve the lives of many more.
Log in or sign up for Devpost to join the conversation.