Pleiades

What the post filtering and edges look like to the computer
CAD Image of camera mount designed by Vincent
Camera mounted on user
Device Being Worn
Device Being Worn

Inspiration

Imagine this, being able to navigate through a world that you can "feel" without ever needing to open your eyes. That's the goal for Pleiades, to guide users through the troubles of the world using object detection and sending that information to the user through haptic feedback. We were inspired by the meta glasses and their capacity to impact others' lives, especially those with visual impairments.

What it does

In its end form, Pleiades will be in glasses similar to META glasses. The glasses will essentially be acting as a service animal or a white cane. Once the user places the glasses on for the first time, Pleiades uses a camera to take a video of whatever is in front of the user. Using that video as input, Pleiades processes each frame and can detect the path and its edges, and most importantly, whether or not the user is approaching the edge of the path.

If the user approaches the edge, feedback will be sent to the user using a haptic feedback system mounted as either a belt, necklace or maybe even on the glasses. The buzzers would trigger based on the direction the user is swaying from the path and with increasing intensity as the user gets closer to the defined path.

Essentially, Pleiades accepts visual information and turns it into a buzz/physical feeling anyone can understand. A tool like this empowers those with visual impairments to explore and interact with the world like never before.

But How does it ACTUALLY Work? Pleiades uses a camera to take videos that our code accepts as input. The quality of the camera isn't very important but the brightness/color outside is. The code processes each frame extensively, only a select part of each frame is used as extra objects in the background can cause incorrect path detection. The resulting frame is grayscaled, then a Gaussian blur is applied, then the image is filtered for specific lines. Horizontal lines are omitted as the user's path will never be horizontal relative to the camera.

The image filtering has been the most difficult portion of this project. Ideally, we should use machine learning to teach the program what to look for instead of using unreliable filtering methods. Nonetheless, the filtering works and the output of our code has two important things. The (x,y) coordinates of the edges of the path and the visual lines between those coordinates. The (x,y) coordinates are used to omit lines that are horizontal, and lines with a slope close to horizontal, and calculate the position of the user relative to the edges of the path. If the user strays too far from the center of the path the buzzer is triggered.

Challenges we ran into

Unlike software or web-based projects, we must deal with the hardware as well as the code (not to mention the path detection too).

On the hardware side of Pleiades: Luckily, we had nearly all the parts we needed from the beginning except for a case putting all of the parts (Raspberry Pi 4, USB camera, buzzers, and patience) together. Vincent modeled and 3D printed a container for the camera. The container for the Raspberry Pi was pulled from online (https://www.printables.com/model/369178-raspberry-pi-4-case ). However, there's still one more piece of hardware to complete the Pleiades guidance system: the buzzers (for haptic feedback).

The buzzers play the crucial role of giving the visually impaired information about the world around them. Whenever the camera detects the user getting close to the edges of the detected path, the buzzer closest to the edge the user is approaching buzzes.

Figuring out how to attach the RPI4 and camera containers to the user was a challenge. Ideally, the technology is small enough to fit in glasses, but we're not quite there yet. Until we get millions in funding, we've settled with the Raspberry Pi container being slid onto the user's belt and the camera is mounted on their chest.

Currently, the Raspberry Pi is too bulky and needs to be downsized. Mohamed has previously worked with the MAX7800 microprocessor that has more suitable capabilities for Pleiades. Also, the MAX7800 is a microcontroller that contains a neural accelerator which gives it Edge AI capabilities. It is specifically meant to be used for Machine Learning tasks in the physical world. Due to its relatively small size it can likely be fit into a pair of glasses. Just as important is its limited power consumption.

The coding issues were seemingly endless, horizontal lines popping up in the post-filtering video despite them being mathematically eliminated from the filtering process, the program stopping completely because an intercept or slope was over 0 producing an error, random lines being created due to filtering/lighting/unknown factors. Nonetheless, most of these coding problems were overcome, but there is definitely room for improvement.

Accomplishments that we're proud of

We're very proud that Pleiades can detect the edges reasonably well using just filtering. We're also very happy that we built the hardware and software and mushed them together successfully. In the beginning, we were unsure of how everything would fit together or if it even would due to the ambitiousness of our project. Our project is an excellent learning experience and a small taste of whats possible with the OpenCV library. On top of that, this project once fully completed would change the lives of those with visual impairments.

What we learned

Prior to this none of us had a lot of experience with OpenCV. In order to successfully complete this project we needed to familiarize ourselves with this technology. We had to learn more about how to utilize it and its plethora of functions. We all found its use of gray scaling to be incredibly fascinating.

We also learned about the experiences of visually impaired individuals. We found that using haptic responses or audio responses would likely be incredibly beneficial for their capacity to navigate the world. We also learned about the costs that go into having a service dog. Finally, we learned about the importance of machine learning models. If we had the time to train a machine learning model our products capacity would progress much further.

What's next for Pleiades

In the future, we hope to implement our product into actual glasses. This will require us to make the devices we are using to perform the necessary tasks much smaller. This includes using a very small and power-efficient microcontroller called the MAX78000. Its limited power consumption will be very important on a pair of glasses because the battery we can fit on there will be very small. It needs to be able to function for 8-12 hours for a good user experience. It also involves utilizing a far smaller camera. Beyond this, we'd like to use a more sophisticated haptic system to communicate to the user. Potentially, the haptic system could be sound-based and put in the glasses, or it could be a necklace or wristband. Most importantly an audio tutorial needs to be created explaining our product and its features to the visually impaired so they can properly utilize it.