SpotLight

Piece 2 Detection
Website Home Page
Instructions Manual

Inspiration

We wanted to work with computer vision, but we wanted to find a way for it to have an effect in the real world. In addition, for most computer vision projects, the computer is made to see what we can, so in this project we wanted to do the opposite and get a chance for the computer to show is what it can see.

What it does

SpotLight specializes assisting visually impaired users with assembling Ikea furniture. Users can toggle back and forth through the instruction manual, and SpotLight will shine the light on the piece you need.

How we built it

The backend was built around the OpenCV library for Python, which enabled object detection and image manipulation. The frontend was built using the Flask library for Python, which allowed for seamless integration with the Python backend. The webpages themselves were built around the fundamental HTML, JavaScript, and CSS. Additionally, the frontend was also used as the projector's image source.

Challenges we ran into

One of the main challenges that we ran into is transforming the camera's perspective of all the components to that of the projector. Although there were many standard transformation that are well documented, we struggled to get meaningful results from these transforms as they typically resulted in the new projected images to be outside the working area, or not pointing at the correct object.

Accomplishments that we're proud of

Despite all the difficulties we ran into, we managed to implement all our planned features in the end. There was no obstacle too big for us to overcome.

What we learned

Simple is best. As we implemented features and debugged issues, our project naturally grew in complexity. It was by questioning some core assumptions about how a library worked, how a feature should be implemented, or what we thought was causing a bug that we managed to simplify our code base and develop elegant solutions.

What's next for SpotLight

As it stands, SpotLight is designed for a highly specific task. We would like to broaden its scope by introducing more tasks. We will do so by using a greater set of object detection libraries, packaging the hardware into a more portable form, and giving our code an LLM-based understanding of its tasks for smarter highlighting.