Inspiration
We were thinking about an idea for this Hackathon that was useful for some people and that did something else that chatgpt could do on its own. By thinking we came across the idea of a Telegram bot that can help blind people with orientation and doubts only by taking pictures.
What it does
It is a telegram bot that a blind person can use to send pictures about its environment and it can ask questions so that it gets more information.
How we built it
We are using the openai model for the explanation about the image and the answer to our questions. We use a speech-to-text model for the messages sent by the blind person and a text-to-speech model for the replies. Lastly, we use a telegram bot to show the result, but it could be implemented in an independent app.
Challenges we ran into
Using a multimodal model and also implementing everything with the telegram bot as we had to retain the context from the previous messages.
Accomplishments that we're proud of
That it works.
What we learned
We learned how to use different models and combined them for a bigger project. We also learned how to use a telegram bot.
What's next for Team 8 - GuideBot
That a person can do a video 360 and the model explains the environment.
Log in or sign up for Devpost to join the conversation.