Inspiration
Mainly to help people who may not be able to use the web due to their visual impairment.
What it does
It helps blind individuals to understand what's on their screen by providing the AI, a screenshot of their screen .They can use it by pressing CTRL+R or by clicking start recording and it will start listening for their question, then their question will be passed and also the screenshot, which in turn will generate a response which will be narrated to the user.
How we built it
It was built using python. Gemini flash model for was used for generating the response by giving it both the question and the users screenshot.
Challenges we ran into
How to lower the delay between the user and the narrated response.
Accomplishments that we're proud of
It can be quite helpful to blind individuals.
What we learned
How to improve latency in this case by using streaming .
What's next for ScreenEye
It can be more beneficial if it would allow the user to talk in between the response of the model and the model then pivots itself to answer that question also.
Log in or sign up for Devpost to join the conversation.