AudioReality

Inspiration

The word that stuck out to us in in SG Enable's prompt was 'enjoy'. We believe that conventionally, when people think about helping persons with disabilities out, most of the solutions gear towards improving their lives in a rather practical aspect.

For blind people, this entails navigation apps that merely point them safely in the direction they need to walk towards. Granted, navigating safely is immensely important. However, we would like to take it up a notch, by enabling blind people to enjoy scenery in their own unique way as well. Through artistic & poetic descriptions of their surroundings, we hope that the blind would be able to partake in 'sightseeing'. This way, even partaking in mundane activities like walking along the streets may seem less like a chore, and more of an immersive literary-stimulating experience.

What it does

AudioReality is a mobile app that reads the geolocation of an individual, determines what their surroundings look like, then outputs descriptions of their surroundings. Notably, these descriptions are tailored to give the user a pleasant literary experience, and are not 'practical' navigational information unlike other apps built for the blind.

Currently, we only have a minimum viable product that outputs location descriptions on our app. However, we hope to eventually integrate AudioReality into navigation apps, so blind users can navigate and 'sightsee' simultaneously.

How we built it

On backend, we used (1) Google's Streetview API, (2) Clip-interrogator API, and (3) Open AI's latest available GPT API, Davinci. For frontend, we programmed with Flutter, and utilized Flutter's geolocation identifier library.

First, we would obtain the user's current location with Flutter's geolocation library. Subsequently, we input current location in the Streetview API, to obtain an image of the user's surrounding location. Thereafter, we use Clip-interrogator API, which is an API that provides a very brief description of an image, to obtain keywords of the location, such as whether there are prominent landmarks, what the landscape generally looks like, etc. Lastly, we input these keywords into the GPT API with a query structure that requests GPT to output an artsy description of the location.

Challenges we ran into

This was the first time we programmed with Flutter, so it took us some time to understand and work with its syntax. Also, given that Open AI's GPT API feels somewhat like a blackbox, we spent some time manually 'chatting' with Chat GPT to ensure that our query structure would give the most accurate and literary-pleasing description. Lastly, given that we used 3 APIs for this, we needed some time to integrate all of them together with flutter.

Accomplishments that we're proud of

We're proud that we were able to come up with our minimum viable product in one week, given that we have never worked with the APIs we used and Flutter before. We're also proud of our idea, since we believe in the need to provide opportunities for the blind to enjoy sight, albeit in a different way. Currently, the software space is already saturated with apps that practically help the disabled. Hence, we believe in the necessity to balance this, by complementing practical ways of improving the well-being of disabled people with ways that allow them to simply have fun or feel fulfilled.

What we learned

We obtained experience working with the APIs that we used, and also learnt basics of how flutter works alongside dart syntax.

What's next for AudioReality

There are many features we plan to develop to improve upon AudioReality's minimum viable product.

(1 ) Conduct optimization on GPT's queries, so erroneous descriptions of locations will be reduced. Also conduct optimization to reduce loading time.

(2) Expand text-to-speech features, to improve blind users’ app navigation experience .

(3) Enable profile creation, so users can tailor the output of descriptions to their profile settings. For instance, ask for age, so we can query GPT to give descriptions that consist of less complicated words to children. Alternatively, create options for styled descriptions. For instance, "Victorian era poems" style, etc.

(4) Create a database of poems. We hope to collaborate with poets in Singapore and ask for copyright access, so we can integrate poems from poems.com.sg into our app. From there, we can programme our app to read poems about specific locations in Singapore if user happens to be close by.

(5) Automate. Currently, user needs to click "get location" for the app to output a description of the location. We hope to automate this process, so users can simply wear their earphones, walk around Singapore, and have poems / beautiful descriptions played to them as and when something relevant comes into their vicinity.

(6) Enable photo-taking. Currently, we provide location descriptions with google streetview. This works, but we'd like to improve upon it since google streeview may not necessarily reflect what the location currently looks like. We hope to enable photo-taking, or even video, so users can get the most current description of their surroundings.