Pathfinder

How might we enhance and improve the experience of using shopping mall directories with persons with disabilities? By helping to make the shopping experience a more pleasant one with the help of AI.

Comment

Inspiration

Our idea was birthed from the problem that we commonly faced when looking for directions in a shopping mall. Directories often only go so far as to provide a unit number to follow, leaving the directions up to the individual. More often than not, some stores also do not display their unit number in a prominent position, as such, we are left to infer based on their neighbours' labels.

Seeing the issues we faced, we wondered how much worse the experience is for people with disabilities. Would it be possible for us to give the humble shopping directory an upgrade?

What it does

We have come up with a two-part solution.

  1. AI-enabled Directory - The directory provides clear verbal instructions and animations for directions to stores based on what the user says to it. The user can ask for the location of specific stores or for more general recommendations based on their likings. The AI model behind the conversation is able to intelligently understand and contextualise the users' questions, leading to a more seamless experience.

  2. Floor Tactile Bumps - The next part of our solution solves the issue of mobility for the blind. Given that they are not able to use visual cues to find their direction, the AI will instead provide a separate set of instructions for them to follow, making use of tactile bumps embedded in the corridors of the mall. Users can then count the number of bumps they have crossed over to find their direction.

How we built it

  • The conversational AI is run on a React website front-end communicating with a node.js backend server. - The AI was built on OpenAI's embeddings and chatGPT APIs.
  • The 3D structure of the mall was created in Blender.
  • The pathfinding animation was created in Unity using A* Pathfinding to find the most optimised path to the shops.
  • The database was created using Notion and Google Sheets and data was gathered from Google Maps - and the Jem Directory Website.

Challenges we ran into

  • Arduino & Breadboard was working intermittently, because of connection issues
  • The website was not functioning consistently, voice detection did not function as expected
  • Database and videos had to be recorded manually
  • Unity project was laggy, some videos were more choppy

Accomplishments that we're proud of

Our website works with its primary functions such as the Learning Language Model for searching stores and the Speech to Text/Text to Speech Functions.

What's next for Pathfinder

  • To prevent blind people from getting lost as they may have miscounted the tiles, we have multiple of these directories at turning points to help blind people go to these junctions if they get lost.
  • Display menu and items the shops sell from users to view from the website.
  • Mobile integration so users can easily access the directory from the convenience of their phone
  • Sync the directions given in text to speech with the video

Built With

Share this project:

Updates