Vivianne

Vivianne: a local-first agentic harness with persistent semantic memory, multi-agent orchestration and cost-aware routing across NVIDIA Nemotron models on Nebius Token Factory.

Comment

Inspiration

Personal assistants that are actually useful need three things: memory, tools and time. Give them all three and they become expensive and opaque. You can't see what each call cost, you can't tell what a skill is about to do, and everything runs on one large frontier model, including work a small model would do just as well.

We wanted the opposite: an assistant whose state lives on your machine (its memory, skills, sign-ins and project files), whose reasoning runs on open NVIDIA Nemotron models through Nebius Token Factory, and whose spend is visible per model, per call.

What it does

Vivianne is an agentic harness for personal AI, shipped as a native desktop app (Tauri 2, Rust and React).

  • Long-horizon agent loop. Tool use, approvals and checkpoints. Long tasks survive rate limits, model fallbacks and app restarts.
  • Cost-aware model routing, powered by Laya. Every request is sized as easy, medium or hard before a model is picked, so a strong Nemotron model takes the planning and the hard work while a small one handles the rest. A savings meter compares that against sending everything to the big model.
  • Persistent semantic memory. Facts, preferences, and corrections are stored locally in SQLite with vector embeddings (a Nemotron or Qwen3 embedder) and recalled when relevant. You can list, edit, and delete everything it remembers.
  • Multi-agent orchestration. A project gets a lead agent with a written charter and hired specialists. Each specialist owns part of the codebase and works as a delegated subagent, with its own model, tool allowlist, and standing instructions.
  • Skills hub. Browse NVIDIA and Nebius skills, review a static scan and a commit-pinned manifest before installing, and get a warning if a skill changes after install.
  • Grounded research. Tavily search, with the sources listed under each reply, and answers rendered as structured summary cards.

How we built it

Runtime. The core is a Rust agent loop: one loop, driven by the chat, delegated subagents, team members, and scheduled routines. Tools are schema-described and gated per call by an approval policy. Programmatic tool calling lets the model write a short script that calls several tools in one round-trip instead of one tool per turn.

Retrieval. Memory recall fuses full-text (BM25) and vector search with reciprocal rank fusion:

$$ \text{RRF}(d) = \sum_{i \in {\text{bm25},\,\text{vec}}} \frac{1}{k + \text{rank}_i(d)}, \qquad k = 60 $$

Each hit is then weighted by an effective confidence that decays with time since last use, with a half-life $h$ that depends on the kind of memory (45 days for an ordinary fact, 365 for a correction you made), plus a small boost for memories used often:

$$ c_{\text{eff}} = \min!\Big(1,\; c \cdot e^{-\ln 2 \cdot \Delta t / h} \cdot \big(1 + 0.1 \ln(n + 1)\big)\Big) $$

where $\Delta t$ is the time since the memory was last accessed and $n$ is its access count. Recalling a memory resets its decay, so what you keep using stays sharp and the rest fades. New facts are reconciled against what is already stored (add, update, delete or skip) instead of piling up as duplicates.

Routing. The router is powered by Laya, an open decision engine (Apache-2.0) that answers typed questions about a piece of text in a single forward pass of a small encoder, with calibrated probabilities. For each request (a chat turn, a subagent's brief, a teammate's routine) Laya returns a difficulty, ${\text{easy}, \text{medium}, \text{hard}}$, plus whether the request asks for something to be built and whether it is only small talk. The routing tiers then map that level to a model.

Deterministic rules (length, how many things are asked, words that mean "think" or "just do") act as a floor that Laya can raise but never lower:

$$ \text{level} = \max\big(\text{level}{\text{Laya}},\ \text{level}{\text{rules}}\big) $$

When Laya is unavailable or uncertain, the router falls back to a hosted classifier if one is configured, then to the rules alone, so a routing decision never blocks a request. Every call records tokens and cost per model, which powers the usage view and the savings meter.

Teams. Specialists are plain Markdown files in the project (.viv/agents/*.md): YAML frontmatter for role, owned paths, model and tools, and the body for standing instructions. They travel with the repo and are readable by both the user and the agents. Team runs can split a task into assignments with dependencies, so dependent work starts only when its inputs are done and receives those reports.

Challenges

  • Getting a lead agent to actually delegate. Prompting alone wasn't enough: given a UI task, the lead would edit the files itself even with a UI engineer on the team. The fix was structural. Writes inside a path an active specialist owns are refused for anyone but that specialist, with a message that names the owner and how to hand the work over. Most multi-agent failures turned out to be organisational, not a matter of model capability.
  • Verification over claims. A specialist's report is a claim, not evidence. Team runs record which files each member actually touched (from its tool calls, not its report), run the project's own check command, and send failures back to the owning specialist.
  • Small models on real work. Routing only pays off if the small model is trusted with the right jobs. We kept it to bounded, checkable tasks and gave every caller a fallback when a cheap call fails.
  • Desktop constraints. Large transcripts (tens of MB), a Rust async runtime whose provider futures aren't Send, and WebView quirks all needed engineering work in the plumbing, not just in the prompts.

What we learned

1. Multi-agent failures are organisational, not intellectual. Our first lead agent had a UI engineer on its team and still edited the UI itself. The model was capable of delegating; nothing in its environment made delegating the path of least resistance. Research on multi-agent failures (MAST) points the same way: most breakdowns come from role confusion and ignored task boundaries, not weak reasoning. We stopped asking nicely and changed the environment: each specialist owns paths, and writes there are refused for everyone else, with a message naming the owner and how to hand the work over.

2. A rule in a prompt is a suggestion; the same rule in the tool gate is a guarantee. Plan mode refusing writes, specialists staying inside their paths, and payments or GPU jobs always asking for approval (even when everything else is on auto-approve) all live in the code that runs tool calls, not in the system prompt. The prompt explains the rule so the model plans around it. The gate makes sure it holds when the model doesn't.

3. A report is a claim, not evidence. Agents routinely say "done" about work they never ran. So team runs record which files each member touched from its actual tool calls, not from its summary. They run the project's own check command afterwards and send failures back to the specialist who owns the failing area, for at most two rounds before telling the user plainly.

4. Intent belongs in documents, not in context. Every action carries implicit decisions. If those decisions only live in a transcript, the next conversation contradicts them. Each project has a written charter, with a Decisions section and the reasoning behind each choice, and each specialist is a Markdown file in the repo. They survive restarts, model switches and day rollovers, and both the user and the agents can read and edit them.

5. Hybrid retrieval beats either half alone, and memory needs to forget. Keyword search finds exact names; embeddings find paraphrases. Reciprocal rank fusion combines them using only ranks, so no score calibration between the two is needed. Decay keyed to last use rather than creation keeps what you rely on and lets the rest fade. Reconciling each new fact against what's stored (add, update, delete or skip), instead of appending it, is what stops memory turning into a pile of near-duplicates.

6. Small models earn their place on bounded, checkable work. Routing pays off only when the small model gets jobs whose output can be validated: classification, summaries, structured JSON. Every cheap call has a fallback. Free-tier rate limits taught us to treat model failure as normal, with fallback chains rather than error screens.

7. Prompt caching is an architecture decision. The system prompt is ordered stable-first (directive, capabilities, workspace, charter), with what changes during a conversation (the goal, the relevant skills) at the end. The tool list keeps a fixed order so it serialises identically every time. Rarely used, schema-heavy tools (browser control alone is a fifth of all tool-schema text) load only when a conversation needs them.

8. One source of truth per tool. We once kept two schemas for the same tool, and they drifted: the dead copy documented a parameter the live one didn't expose, so the model couldn't discover it. Wherever two descriptions of one thing existed, one eventually lied.

9. Measure before optimising. Our lag wasn't where we guessed. It came from re-processing the whole conversation on every streamed token, from rewriting multi-megabyte transcripts after every turn, and from unoptimised debug builds, not from the model calls.

10. Treat outside content as data, never as instructions. Web pages, emails and texts reach the agent labelled as external. A message from outside never counts as the user asking for something, so anything it suggests still goes through the agent's approval rules.

What's next

  • Multiplayer and community. Today two people can already pair on a shared project. Next, projects become discoverable spaces, like a Discord server: anyone can find a project, request to join, and contribute with the owner's approval. Members can share custom agents, bots and skills, and see shared usage, so a team knows what its agents cost.
  • A hybrid SSM model for Vivianne. A small hybrid state-space model, mixing SSM and attention layers in the spirit of NVIDIA's Mamba-Transformer Nemotron-H, built to run locally. SSM layers keep memory flat as context grows, which suits an always-on assistant with long transcripts on consumer hardware.
  • A custom llama.cpp fork for local inference. Tuned for Vivianne's workload: long, stable system prompts, frequent tool calls, and background jobs running beside the chat, so local models are as usable as cloud ones.
  • One shared agent base. One tool definition, one profile and policy object for guardrails, and one way to spawn an agent, shared by chat, specialists and bots.
  • Real OS-level sandboxing for skills, and hosted always-on routines on Nebius AI Cloud.
Share this project:

Updates

Submission history