Polaris

Simulation of building configuration w/ undirected maps
Demo with the LEDs represending rooms or key features within the building, the color sensor as the wristband
Camera w/ YOLO
Arduino w/ buzzer & color sensor
Board representing locations of each node for color sensor
Wearable device
Photon not working
Photon working
Ball knowledge

Inspiration

Navigating buildings can be difficult, especially for people with visual impairments. Existing navigation systems like Google Maps can guide users to the right building, but they often cannot provide detailed directions inside. We wanted to make indoor navigation as simple as outdoor navigation. This inspired us to build Polaris, an AI-powered wristband that helps people navigate indoor spaces through voice guidance and color-based location detection, with the potential to use radio-based technology in the future.

What it does

Polaris is a wristband equipped with a buzzer, a color sensor, and a connection to an electronic device via an API. The program processes user voice commands, such as “Please take me to Room 100 in the Engineering building” or “Take me to the bathroom.”

The building is represented as an undirected graph, with each node representing a location, such as a room, bathroom, staircase, or elevator, and each edge representing a possible path between locations. Each node is assigned a different color for identification. For demonstration purposes, LEDs will be placed at designated locations throughout the building, and the color sensor on the wristband will detect these colors to determine the user's location. In the final product, we plan to explore using radio signals or Wi-Fi for indoor positioning.

The wristband provides haptic-free audio alerts through its buzzer: it beeps once when the user reaches the correct location and beeps continuously if the user deviates from the designated path.

The program uses pathfinding algorithms to calculate the optimal route to the user's destination. To account for congestion, existing cameras in the building can estimate pathway occupancy by measuring the percentage of each camera frame occupied by people. If occupancy exceeds 80%, the pathway is considered congested, and the algorithm prioritizes less crowded alternative routes when determining the best path.

Finally, the program communicates with the user's electronic device to provide real-time messages and voice guidance, with instructions such as “Go to the [Location]” and “You have arrived at the [Location].” This allows users to navigate indoor spaces efficiently and reach their destinations with greater ease.

How we built it

We built Polaris using a Flask server that processes the building map, tracks the user's location, and calculates the shortest route using Dijkstra's algorithm.

Voice Navigation: We used Grok to convert speech to text and ElevenLabs to convert text into spoken directions.

Building Map: We used Google Gemini to convert photos of drawn maps into undirected, weighted graphs. Nodes represent key locations, such as entrances, rooms, staircases, and bathrooms, and are assigned unique colors. Edges represent possible pathways between locations, with weights representing travel costs. The resulting graph configuration is saved in a .json file, which the Python program reads.

User Location Tracking: We connected Arduino UNO to a TCS34725 RGB color sensor and a Grove Buzzer v1.2, which are components of the wristband. The sensor detects color-coded markers near key locations (assumed to be already implemented in the building) and sends readings to the Flask server through the Arduino's serial connection. The server uses these readings to determine the user's location and provide navigation feedback through the buzzer.

Crowd Detection: We used OpenCV and YOLO to detect people in camera feeds. When the estimated occupancy of a camera frame exceeds 80%, the corresponding location is marked as congested on the graph. The routing algorithm then accounts for congestion when selecting a route.

Caretaker Notifications: We used Photon to send caregivers iMessage updates about the user's location and navigation status.

User Interface: We built a web interface served by the local Flask server that displays the building map, recommended route, and live navigation updates.

Challenges we ran into

We initially struggled to figure out how the wristband could identify the user's location within a building. During the early design stage, we considered using infrared lights but switched to an RGB color sensor when we couldn't find suitable infrared sensors.

At first, the sensor could only reliably detect red, green, and blue. The color sensor module's built-in light (used to illuminate objects) interfered with color detection. For example, the sensor would detect black as yellow because its light illuminated the black surface. As a result, our first iteration used only a red, green, and blue to represent different locations.

We later discovered that connecting the sensor's light-control pin to ground could turn off the built-in light. After modifying the code accordingly, we were able to reliably detect a variety of colors. We then implemented Dijkstra's algorithm and assigned different colors to building locations, allowing the wristband to identify locations and guide users along the calculated route.

Accomplishments that we're proud of

We are a team that formed on the first day of the event, and we had never met before. We are proud of how effectively we collaborated, divided tasks equally, and communicated our ideas. We are also proud of being able to turn our ideas into a working product by integrating hardware and software. In particular, we successfully implemented an iMessage feature via Photon that keeps caregivers of visually impaired individuals updated on their location and navigation status. Overall, we are proud to have worked together to turn our ideas into reality and take one step closer to solving a real-world problem.

What we learned

Arpey: How to create interactive 3D mockups of the product. How the Arduino terminal sends commands to the Python terminal. How to implement Dijkstra’s algorithm, a greedy algorithm for finding the shortest path in a weighted graph, to navigate indoor maps.

Henry: How to make API calls to services such as the Grok API, Photon for iMessage integration, and the ElevenLabs API. How to create local Flask servers to support user-facing visualizations. How to access computer cameras and webcams and use YOLO (You Only Look Once), an object detection model, to count the number of people in a camera frame.

Stephanie: How to implement speech-to-text and text-to-speech workflows using ElevenLabs, as well as how to create hidden .env files to store API keys securely. How to convert RGB readings from the color sensor into HSV (hue, saturation, and value) values to improve color classification and detect more colors in the color scheme.

What's next for Polaris

1. Upgrading hardware components: Replace the Arduino Uno with an Arduino Nano since we do not need as many ports. Replace the RGB color sensor with a radio- or Wi-Fi-based positioning system that works more reliably indoors and is less affected by environmental conditions. 2. Designing the wristband: Arrange and integrate the hardware components into a compact design that minimizes the wristband's overall size while maintaining functionality. 3. Expanding navigation capabilities: Enable the program to process building configuration inputs and generate undirected graphs for more complex indoor structures. Incorporate camera data to estimate occupancy and improve navigation based on the number of people in an area.