Symphony

Our local model neural architecture.

Inspiration

We were inspired by businesses that want to take advantage of AI capabilities, but are unable to use preexisting solutions due to sensitive data, lack of knowledge, or concerns over cost. The goal is to allow them to use AI to accelerate their business while keeping their data fully secure and proprietary.

What it does

We sell an all-in-one platform for businesses to deploy locally and inside their cloud environment that allows them to use AI to its full capabilities. We have locally run LLMs for simple queries and privately hosted LLMs inside AWS with tools, memory, and sandboxed execution environments for agents to execute tasks within. We provide the landing zone, you provide the value, and together we allow you to operate as efficiently as possible, with minimal friction.

How we built it

We used a Jetson Orin Nano as our base edge device. We trained our own decision head to route tasks to the local or remote language models based on the task difficulty. To save on VRAM and time, we used a Qwen3.5-2B shared backbone with an added multi-layer perceptron based on intermediate states. Because the prefill phase of the decision head is the same as the language model head, the computation to decide whether to send a request to the local model is nearly free, and the decision capability takes up very little memory.

Challenges we ran into

Acquiring cheap but powerful hardware to run models locally.
Getting compute for large instances to train ML models quickly.
Integrating on-prem and hybrid cloud solutions.
Creating a cost-efficient solution that is secure enough for use in industries with sensitive data.

Accomplishments that we're proud of

Developed a novel architecture to route queries to local and cloud LLMs efficiently.
Designed a robust cloud platform that puts security first, includes guardrails, governance controls, a safe landing zone, and allows for low-latency inference.
Created an elegant 3D-Printed case that aligns with our brand and marketing strategy

What we learned

Hybrid-cloud infrastructure design and implementation.
Security-first mindset to achieve privacy goals inside the cloud.
State-of-the-art research methods for model routing using edge devices.
Developing webapps that can run as websites and mobile applications.
Using AI assisted front-end development, in a way that does not look "vibe coded"
How the OpenCode SDK works, and can be integrated into projects.

What's next for Symphony

Improved pre-training of our routing model using task success data from queries to the direct models we are calling.
Federated learning to improve our routing model based on customer usage without compromising privacy.
Additional modes of communication with Maestro like text messaging, and email.
Improving platform stability and doing pilot tests with local small businesses.