Live Assistant

Inspiration

We wanted to explore an area where AI still hasn’t made a strong impact, which is 3D CAD design. CAD is a fundamental part of engineering across many fields, including mechanical engineering, aerospace, and even game design. It plays a critical role in how both physical and virtual products are created. However, unlike coding or text-based tasks, AI has not been effectively integrated into CAD workflows.

What it does

It analyses and gives advice based on the users screen.

How we built it

To do this, we built a system that mimics a live screen-sharing experience between the user and an AI assistant. First, we created a backend component that continuously captures screenshots of the user’s screen every few seconds. These screenshots are stored in a folder, where the most recent image is always saved under a consistent name. This allows other parts of the system to easily access the latest state of the user’s work. We also implemented a limit on how many screenshots are stored, so older ones are automatically deleted to keep the system efficient.

Next, we built a second component that takes the latest screenshot and sends it to an AI model. This allows the AI to understand what the user is currently doing in their CAD environment and respond based on that visual context.

On top of that, we created a simple frontend interface that allows the user to interact with the AI through a chat box. The user can ask questions such as what to do next or how to perform a specific action, and the AI responds with short, actionable steps based on what it sees on the screen.

Challenges we ran into

Once we were done we had a hard time merging our component that we worked on individually. We managed to overcome this by rewriting and recycling some components.