Jarvis

Inspiration

Modern development environments offer enormous power, but accessing that power can be complicated. Building an application often means configuring tools, managing dependencies, switching between interfaces, and figuring out how everything connects.

AI agents can help, but they can also introduce more complexity. Different agents have different setups, strengths, and ways of working.

We wanted to bring those capabilities into one place. What if you could focus on what you wanted to achieve, while one assistant helped coordinate the tools needed to get there?

That question became Jarvis. Our ambition is to put the power to tackle a wide range of tasks into the user’s hands through one approachable application.

What it does

Jarvis is a desktop AI assistant that brings a conversational interface, a locally hosted language model, and multiple agents together.

At its core is an intermediary layer that uses a single local LLM to provide answers while communicating with other agents. Users interact through one application, while the system coordinates capabilities behind the scenes.

Jarvis is designed to help users move from describing a task to accomplishing it, without needing to manage every underlying tool themselves. Its capability resolver helps connect requests to available functionality, while research agents and generated tools provide routes for extending what the system can do.

Alongside the main chat, the workspace includes settings, memory, tools, and files. These sections give users a consistent place to manage the assistant and interact with its capabilities.

How we built it

We built Jarvis using Python and JavaScript, with React components written in JSX and SQLite queries written in SQL.

Backend and AI

Python powers the backend, including the FastAPI API, Jarvis core, capability resolver, generated tools, research agents, and SQLite access.

Our AI architecture uses a locally hosted model through llama.cpp, with an intermediary layer that communicates with multiple agents. This allows us to combine different capabilities behind a single user-facing experience.

We also developed a self-programming pipeline to explore how Jarvis could generate functionality for unfamiliar tasks. Improving the reliability and safeguards of that pipeline has been a major part of the work.

SQLite provides backend storage, with SQL queries handled through Python.

Frontend and desktop application

We built the interface with React, using JSX for components and pages such as Chat and Settings, and CSS for styling.

JavaScript handles frontend services, API calls, WebSocket communication, and the Electron main process. Vite supports frontend development and builds, with HTML providing the page structure in which React runs.

We used Electron to turn the web interface into a desktop application and Electron Forge to package it into a Windows installer.

Configuration and setup

We use JSON for manifests, configuration, API payloads, and tool schemas. PowerShell supports Windows setup and startup scripts, including environment-variable configuration.

Together, these pieces connect the desktop interface, backend services, local model, and agents.

Challenges we faced

Strengthening the self-programming pipeline

One of our hardest challenges was developing a pipeline that could generate functionality without damaging its own code, the application, or the user’s environment.

Generating code is only part of the problem. We also had to consider how generated tools should be checked, constrained, and integrated. Strengthening these safeguards remains an ongoing priority rather than a problem we consider fully solved.

Connecting the user experience

Setting up the sign-in page, integrating it into the web interface, and turning that interface into a dynamic desktop application each introduced different challenges.

We needed the screens to feel connected and behave consistently. For example, appearance changes initially affected only Settings. Moving theme management into the shared workspace made those changes apply across pages.

Balancing speed and capability

Choosing agents meant balancing responsiveness against the capability needed for demanding tasks. We had to consider which agents suited different requests and how their involvement affected the overall experience.

Making the agents communicate

Handling communication between agents and the locally hosted LLM in both directions was another significant challenge.

The intermediary layer needed to coordinate these interactions, while API calls and WebSocket handling connected the interface to backend activity. Making these parts communicate reliably was essential to presenting one coherent application.

Integrating separate branches and components

Different parts of the frontend and backend were developed across separate branches. Bringing them together required resolving differences in structure, styling, and expectations about how data would move through the application.

Making the combined project work coherently took more than simply merging files.

What we learned

We learned that an AI assistant’s usefulness depends on how well its parts work together. A capable model alone is not enough: communication, tool integration, safeguards, and clear user feedback all matter.

Building the intermediary layer helped us understand the challenges of coordinating multiple agents through a local model. It also showed us why speed and capability need to be considered together.

Working on the self-programming pipeline reinforced that generated code needs careful handling. Expanding what a system can do also increases the need for validation and controlled execution.

We gained experience connecting a Python and FastAPI backend to a React interface, handling API and WebSocket communication, working with SQLite, and packaging a web application for desktop use with Electron Forge.

On the frontend, we learned the value of reusable components, shared state, and a clear separation between the interface and its data service. Integrating separate branches also highlighted the importance of agreeing on interfaces and testing components together early.

Accomplishments we are proud of

We built an intermediary layer that uses a single locally hosted LLM to provide answers while integrating and communicating with multiple agents. This gives Jarvis a foundation for expanding beyond the capabilities of a single model.

We also created a futuristic, distinctive interface with a consistent visual identity. React and Electron provide an expandable foundation for new capabilities while keeping the experience familiar across the application.

We brought together the sign-in interface, chat, workspace pages, backend services, local AI, and agent communication into a shared desktop project.

Most importantly, we turned a broad idea into a prototype we can demonstrate, evaluate, and continue improving.

What is next for Jarvis

We plan to explore three versions of Jarvis: Standard, Max, and Elite. The capabilities and differences between these versions are still being developed.

Our other priorities include:

Connecting more agents to expand the range of tasks Jarvis can support.
Training our own models to explore more tailored behavior and better coordination.
Building more secure pipelines, especially around generated code and tool execution.
Improving speed and reliability across communication between the local model and other agents.
Refining the interface as the application gains new capabilities.

Jarvis is not yet device-agnostic, and users cannot currently access the same assistant from anywhere. Cross-device continuity remains part of our longer-term vision.

Our goal is to make increasingly powerful capabilities available through one app—so users can spend more time achieving their ideas and less time managing the tools behind them.