BlindFindAR text to speech - the see for me snapchat lens

BlindFindAR A Lens to scan and SEE

Inspiration

Can a social visual app be more appealing to over a billion people with vision impairments.

What it does

Lets people use their phone with the snapchat app to SEE objects in their environment by scanning with text to speech feedback of what is recognized by snapchats scan action.

How we built it

In lens studio on the google cloud platforms compute engine virtual machines

Challenges we ran into

Having never used lens studio or snapchat there was a steep learning curve that had to quickly be climbed. A large hurdle was with only a low end laptop I had to get access to machine capable of running lens studio so had to figure out different ways to run lens studio in the cloud and the many subproblems that comes with such as gpu access, sharing audio over remote desktop.

Accomplishments that we're proud of

Rapid Prototyping BlindFindAR was started just 2 days before the deadline due to lack of hardware capable of running lens studio.

What we learned

Lens studio basics, incorporating assets into lens studio templates, basic testing processes for making lenses, creating basic UI in a lens, very simple scripting in lens studio , creating google cloud compute engine VMs with attached gpus to be able to run lens studio.

What's next for See for me Blind FindAR TTS Vision Scanner

Better Voice command activation prompts. Hot or cold feedback aka Directional guidance audio (is the object to the left or right of the phone, lower volume for distance and spoken indications like "hot" "closer" "further",etc "Scanned Screen" share, let others join with the person scanning to guide them to specific areas like https://www.bemyeyes.com/ . Get ready checklist: a premade list of the users belongings they need to find to start the day,etc ML object labels: to build user specific object recognition so object feedback like "clothing" is more specific as the built in asset household-objects-detection-model does not have labels. With funding accessibility user testing.