The Council of Capability

Inspiration

We wanted to explore what happens when software is a consultant. Our inspiration came from the idea of a “council” comprised of multiple inputs, multiple perspectives, and a system that feels less like a single tool and more like a collaboration. We were especially motivated by how multimodal systems can help empower a user to make well-informed decisions.

What it does

The Council of Capability is a multimodal query system that allows users to interact with a centralized knowledge server. It accepts queries, performs fast full-text searches over product data, and returns relevant results in real time.

How we built it

We built a server using Python + FastAPI, backed by a SQLite3 database with FTS5 tables for fast product lookups. We implemented AI classes to with built-in prompts to ensure a role is played by each model. We connected XR glasses to the server for a seamless display of the output. Additionally we have a web application that the user can interact with.

Challenges we ran into

One of the biggest challenges was getting reliable communication between separate machines. We were becoming discouraged each time our server connections timed out but eventually we made a breakthrough and it felt so rewarding.

Another major challenge was coordinating the agents consistently, since each agent has a different role and a different view of the user’s context. This requires the orchestrator to reliably decide which agents to call and when. For more complex requests, the orchestrator also has to prompt each agent for the right piece of information and then merge their outputs into a single decision. That’s difficult because we don’t have much data, and combining too much context can make the prompt overly long, raising the risk of hallucinations or a muddled final answer if the model loses sight of the user’s main goal.

Accomplishments that we're proud of

We’re especially proud of successfully getting another machine to communicate with the server and perform real queries against the Grainger product database. That milestone was amazing as it showed that our networking setup and API design worked beyond a single local environment.