StewardAI

Steward enables AI agents to collaborate safely by sharing only the minimum information required, protecting sensitive data, enforcing approvals, and maintaining complete audit trails.

Comment

Inspiration

I run two AI agents. One has my work email and calendar, the other has my personal ones. They are far more useful when they can talk to each other, but today you either connect them fully or not at all.

Here is the moment that started it. My work agent is planning a client trip to Manchester and asks my personal agent when I'm free. A careless personal agent answers like this:

Wed 14 Oct, 2:15pm, cardiology follow-up with Dr Shah, Exeter Riverside Hospital

The work agent didn't need any of that. It only needed to know whether the time was available. Sharing details such as medical appointments, personal relationships, locations, travel plans, or other sensitive context creates unnecessary privacy and security risks. If an agent is compromised, attackers may use this information to build a profile of a person's life, contacts, routines, and movements. The appropriate response is:

Wed 14 Oct, 2pm to 4pm, busy, can't move

That gap is what Steward closes. Along the way I added the other things that worried me: a web page or an email that slips instructions to an agent, small answers that add up to something private, and two agents overwriting the same to-do list. The phishing screen came from real life: this week a phishing test landed in my work inbox, and it was convincing enough that I clicked the link.

What it does

Steward is a gatekeeper that sits between my agents and everything they can reach: the models, the tools, and each other. Agents can only ask each other a few fixed kinds of question (when am I free, my travel limits, my dietary and access needs, and an address stand-in for one booking). Code checks every question and trims every answer. Anything risky waits for me as an approval card on my phone.

This is one evening, played by the real agents:

  • 19:01 The work agent reads a station guide through Tavily. The page hides an instruction to send my calendar somewhere (HTML display set to None). The agent has also read my private data by now, so Steward holds anything that could send data out, including the train booking. I approve the train on my phone.
  • 19:02 The taxi needs my home address. Steward asks me first, then gives the agent a stand-in, [ADDR-1]. The real address only goes to the taxi company.
  • 21:15 A fake "verify your mailbox" email arrives. Steward spots the warning signs in code and withholds its link. The agent reports it as phishing.
  • 22:40 A lookalike client email hides an instruction in invisible text. Steward strips it before the agent reads it.
  • 23:30 A blood test appears in my personal calendar. The work agent only hears: Thursday, 9 to 10:30, now busy, can't move. Personal agent asks the work agent to move the train and taxi from that time, and those paid changes wait for me.
  • 01:30 The personal agent edits a shared to-do from an old copy. Steward notices the newer version and holds the edit.
  • 07:30 Nobody asked, but Steward writes me a morning brief: what happened overnight, plus one question. I tap Approve, and each change runs exactly once. Everything held overnight arrives as cards after the brief, and I deny the rest.

Every call lands in a hash-chained flight log, and the whole evening can be replayed from it. The replay makes the same 156 decisions, every time.

How I built it

Each agent runs in its own NVIDIA OpenShell sandbox. The network is closed except for one door: Steward. The agents never hold a model key; they hold a pass, and Steward holds the real Nebius key.

Who does what:

  • Work agent: Hermes on Nemotron 3.5 Super VL
  • Personal agent: OpenClaw on Nemotron 3.5 Lightning
  • Morning brief: Nemotron 3.5 Lightning writes the sentences from facts that code picks, then code checks every time, number and weekday before it is sent
  • Every decision (allow, hold, refuse, trim, phishing check, pause): plain code, no model
  • All model calls go through Steward to Nebius Token Factory

Steward itself is Python and FastAPI with an MCP gateway for tools, SQLite for the flight log, a React console, and Telegram for approval cards. Web search and page reading use Tavily: Steward wraps web_search and web_extract, marks every result as untrusted, and keeps the Tavily key to itself. The public demo is a separate, secret-free copy of Steward that replays the recorded evenings, behind Caddy and HTTPS on one AWS VM.

Challenges I ran into

  • My models were retired mid-build. Token Factory gave 12 days' notice that both models my agents used would leave on 9 November, in the middle of judging. I tested the recommended Nemotron 3.5 replacements on the VM (plain, streaming, tools, streaming with tools), they passed, and I switched both agents.
  • Deciding between OpenClaw, NemoClaw, and Docker. It took me a while to decide between them, spent days learning the architectures of them completely, and decided to go ahead with OpenClaw since its the best Agent-Sandboxing tool I could ever ask for, with built-in key management facilities that I leveraged for my Steward, and strictly policy constraint. Since I do model routing, key vaults, I found to stick with my plan as NemoClaw means I'd be encountering additional cost.
  • Deciding whether to store keys on OpenClaw or in Steward's vault. Since I am aiming towards a secure platform, I decided to go with agents performing two requests to get to the key instead of one with OpenClaw, and the path from Steward is only to the Nebius Token Factory. When agents requests the key, Openclaw would give it a placeholder and intercept enroute to replace it with pass in one request. The second request would take a bearer token from openclaw to Steward, where steward replaces the key and performs the call by itself to the Token factory, this way steward can manage everything in one place
  • The brief check was too strict. It rejected a correct brief because a lookalike sender's domain contained my client's name. The real fix was to stop handing the model any text that someone else wrote.
  • The real agent was too careful. Hermes kept reporting the attack emails instead of falling for them, so the "tricked agent" never appeared. To show what Steward does when an agent does fall for it, a scripted actor plays that part through the same door, and the log marks it as scripted.

Accomplishments that I'm proud of

  • To be able to research so much about the Agent security and related tools, such as OpenClaw, NemoClaw, Openclaw agent safety platform, Docker, risk factor for agents and security principles
  • A full night with the real agents where I made every decision from my phone: 4 approvals and 7 denials. The brief arrived on time, both paid changes ran once, and the replay matched all 156 decisions. The models cost 15 cents.
  • Nine live evenings played to the end, and every one replays with every decision matching.
  • 49 attack "traps" and 13 normal tasks played against a real Steward, all passing, and sandbox walls that hold: neither agent can reach Token Factory directly or read Steward's secrets.
  • A public demo anyone can try with no login, where nothing calls a model or changes anything.

What I learned

The big question was whether I could just ask a model to be the gatekeeper. So I measured it. I gave a Nemotron judge exactly the same rules in its prompt and ran the same 50 asks and replies through both, three times each:

  • Code stopped 24 of 24 attack asks. The judges stopped 23 and 21.
  • Code held all 33 asks that must wait for me. Both judges let some address requests through.
  • Both judges allowed a 28-day calendar sweep, even though the prompt says 14 days at most.
  • At temperature 0, the judges still changed their answer between identical runs on 3 and 10 of 50 cases. Code never did.
  • A check took 0.07 ms in code against 4.7 and 17 seconds for the judges.

To be fair to the judges, neither leaked a private detail in a reply. But a gatekeeper has to be right every time, so every decision in Steward stays in code, and a model only writes the brief.

Two smaller lessons: the words a model reads are part of the interface ("approved, done once" made the agent think the taxi was already booked), and trimming OpenClaw's built-in tools cut its prompt from about 31k tokens to about 4k.

What's next for StewardAI

The fixed question types are narrow on purpose, so more things end up as approval cards than I'd like. Next, I intend to do another ablation study:

  1. A translator that maps a free-text question to one of the fixed types, with code still checking every field. I'd also add a warning before an agent hits its spend limit, and an alert when an agent goes quiet.
  2. Two AI checkers inside the steward instead of fixed question types and deterministic code checks. A request checker that translates free-text agent requests into Steward's structured question types, extracts fields, and proposes a confidence score. A risk checker that reviews the proposed request and answer for privacy, security, prompt-injection, oversharing, and policy violations. Neither checker would be able to approve, deny, or execute anything.

Built With

Share this project:

Updates

Submission history