Basecase

Inspiration

Physical AI is limited by how its hardware was programmed, not by the hardware itself. One piece of hardware can be so much more than just what it was designed to do. We are attempting to utilize hardware to its limit.

What it does

It's a camera that runs a SOTA computer vision model, but it's more than that. Because a simple webcam can be designed for a certain purpose. Maybe as a security camera to scan people who walk by. Maybe as a key point model for a human athlete. Maybe we want the camera to scan for environmental indicators or scan a crowd. But what if we could do all of that, on one camera? Changing the camera's purpose is as simple as talking to it. It's a "compiler" that transforms the highest level of human instruction down to a physical change in what the hardware is executing in an instant. One device, infinite possibilities.

How we built it

The project consists of a web interface and a multimodal computer vision pipeline. The computer vision pipeline directly talks through the hardware through the linker, while it talks to the web interface through a web socket for responses and a complex orchestration agentic workflow for requests. The agent analyzes the natural language request, lexes it into tokens, and parses it into a sanitized JSON schema the pipeline can understand at the model level. However, the workflow can also take and learn from processed outputs from the pipeline to refine its own requests. The agent takes the processed JSON and makes a tool call to the pipeline. The pipeline then turns the JSON live, but the agent isn't sleeping either. The pipeline is constantly streaming back to the agent so changes can be processed in the orchestration layer and new changes can be pushed. The pipeline has a complex state machine to handle different kinds of request and changes.

Challenges we ran into

Basically the entire hackathon was a challenge. The truth is that originally, our idea was to use a robotic rover and get the robotic rover to track and perform different tasks on the fly using natural language. However, hardware quickly became our bottleneck and for 3 hours we were anxiously staring at the hardware request page, wondering what to do.

At the end we decided to pivot in the direction of complexifying the core state machine, stripping away the hardware and making the core more powerful, to the point I can say "find the yellow duck" then "track this human walking across the screen". If the human goes out of frame, there's even an arrow on the frame telling you where the human went.

Accomplishments that we're proud of

I'm proud that we currently have a partially working CV pipeline. Although I haven't built full functionality yet, we are almost done with integrating the agent orchestration layer to the actual perception API itself.

I'm super proud that our team did not get crushed by the failure of our first attempt. We were able to pivot and get back on track. It's hard to do this when I feel like so much is at stake.

What we learned

If we want to work in hardware, we should probably order parts early and form a basic idea before hacking starts.

What's next for Basecase

I plan on expanding computer vision to other forms of physical AI. Basecase becomes a middle man between AI and machine. It basically supercharges any device, giving it the ability to perform whatever you want (as long as the hardware supports it) just by talking. That's what I will demonstrate by the end of our project.

Built With

• openai
• pytorch
• qnx
• sam
• yoloe