Inspiration

Notch has been on the App Store as a set-by-set workout coach: it runs a lifter through the session, logs every rep and plans the next weight. But a lot of coaching happens between workouts — a sore knee, a short Thursday, a question about food — and a chat that forgets you by tomorrow and can't touch your plan isn't a coach. We wanted one that knows you, does the small things for you, and keeps what it knows about you yours.

What it does

The Coach assistant is a new tab in Notch Coach, the hackathon build of the Notch app:

  • It remembers you. Every night, Nemotron distils short, typed facts from your chats and workouts — injuries, goals, schedule, equipment, preferences. What your coach remembers shows all of them, and deleting one means the coach stops using it.
  • It acts for you. It saves notes for your next workout and changes an exercise's sets, rep range, rest, intensity or warm-up — never the movement itself. Every change is stated exactly, recorded, and undoable for 24 hours.
  • It looks things up. Web search through Tavily, with sources you can tap; the coach's rules keep personal details out of the queries.
  • It checks in on its own. A daily message (06:00 UTC) about today's workout and one thing it remembers — or nothing, when there's nothing useful to say.
  • Its workout coach runs on Nemotron too. In this build, My Coach — the plan it writes for you and the coach inside each workout — runs on Nemotron Super through Token Factory, in the user's language.
  • It's private by construction. Every user has their own agent in their own sandbox; its memory, skills and sessions belong to that user alone. The agent never holds an API key and reaches training data only through tools that check every read and write — and it's hardened against web pages that try to speak for the user.

How we built it

  • An agent per user. Each user gets their own Hermes Agent in their own NVIDIA OpenShell sandbox, created on their first message, stopped after 10 idle minutes and deleted with their account. When the sandboxes that fit on our VM are all busy, a message waits its turn — users never share one.
  • Nemotron on Nebius Token Factory for every model call. Nemotron Super answers every message and check-in and drives Hermes' tool calls with a little thinking (reasoning_effort: "low"): with none, it promised actions it never took; with more, a reasoning model can spend its whole budget thinking and answer with nothing. Nemotron Ultra runs the nightly memory job with thinking on and a JSON schema; every operation it proposes is validated, and the facts' scores are computed in code. My Coach's plans and in-workout coach run on Super as well: the existing LangGraph coach now runs inside our Edge Functions with its model client pointed at Token Factory. Daily spend ceilings in our code bound what each can cost.
  • Keys the agent never sees. OpenShell's proxy adds the Token Factory key, the Tavily key and the user's own tool token to outgoing requests — each only for its own host. The sandbox's network policy lets Hermes reach exactly three places: Token Factory, Tavily's search and our tools.
  • Tools that act safely. notch-tools is our MCP server over the user's real training data: four read tools and three write tools, with an audit trail and undo. A tool never takes a user id from the model; the per-user token decides whose data it is.
  • Memory users can trust. Up to 15 facts per user, scored and evicted by rules in code, injected into every request as data rather than instructions. Hermes' own user-profile store is off, so a deleted fact can't live on inside the agent.
  • A channel with no inbound ports. The app writes to Supabase; a relay on the Nebius VM pulls jobs over signed requests, runs them in the user's sandbox and posts the reply back, with a push notification.
  • One Nebius VM, run by GitOps. k3s with Argo CD, which syncs the cluster from Git; CI builds the images after green tests. The whole deployment fits inside our hackathon credits.
  • Ours versus upstream. OpenShell, Hermes Agent and NemoClaw's Hermes sandbox image are the runtime, pulled at pinned versions under their own licenses. The relay, the sandbox manager, the tools, the memory pipeline, the coach's rules, the deployment and the app are ours.

Challenges we ran into

  • Isolation on a small VM. A sandbox per user is the right privacy model but costs memory. We start sandboxes on demand, stop idle ones, cap how many run at once and queue the rest in order — capacity runs out as waiting, never as sharing.
  • Reasoning models and tool calls. A reasoning model can spend its token budget thinking and return empty content, which a tool loop reads as "nothing to say", so we first turned thinking off. That had its own failure: once a conversation had history, the agent called the tool a turn needed in only 6 of 16 — it said "I'll pull that up now" and stopped, or answered "I've reverted your bench press" without undoing anything. A little thinking (low) fixed it: 16 of 16, and 40 of 40 across the whole tool check, for about the same cost. Our client fails loudly on an empty reply either way.
  • Memory the user controls. Deleting a fact has to really delete it. That ruled out letting the agent keep its own profile of the user, and made our facts — visible and deletable — the only memory about the person.
  • Secrets in a public repository. Keys live in the gateway, not in the agent or in Git, and the instant sign-in allowlist lives in a secret, with random addresses.
  • A model that fills the schema its own way. On the memory job's first real run, Nemotron Ultra put each new fact's wording in the field meant for a reference to an existing fact and left the category out, so our validator dropped all 24 operations and stored nothing — safe, but empty. The prompt now spells out every operation's fields, and the parser takes a wording from the wrong field when it clearly is one; the next run kept 25 facts and dropped none.
  • OpenShell on Kubernetes, the less traveled path. Its gateway strips Authorization on the way to a sandbox, so Hermes needed a patch to take its key from another header; a gate NemoClaw adds to Hermes waits forever on a Kubernetes volume; about one sandbox start in seven waits 2 min 16 s more for OpenShell 0.1.2's supervisor; and a gateway restart left running sandboxes out of reach for up to 13 s, which used up a message's retries until the relay learned to wait instead. Each one found on the real cluster, and each with a test now.
  • Any web page could speak as the user. Hermes tells the model that text after an [OUT-OF-BAND USER MESSAGE] marker is the user talking mid-turn, but never strips that marker from tool results. So a page Tavily returns, a file or an MCP response can impersonate the user; in a stock install with shell tools, that's a web page giving instructions with the user's authority. We found it while auditing the prompt Hermes sends (open upstream as #81828), proved it — 10 of 10 turns obeyed a forged marker — and closed it with a patch that relabels the marker in every tool result: 0 of 10, live in production, and submitted upstream as PR #143928.

Accomplishments that we're proud of

  • Closed a prompt-injection hole in Hermes Agent — and sent the fix upstream. Auditing what Hermes adds to the system prompt, we independently found that any tool result — a web page, a file, an MCP response — can carry the marker Hermes trusts as the user's own voice (tracked upstream as #81828, still open in every release). On our stack, a forged marker in a search result took over the reply in 10 of 10 turns. Our build-time patch takes it to 0 of 10 in production, and we contributed the fix, with 16 tests, as PR #143928.
  • Real per-user isolation that holds under load: waiting, never sharing — and checked from inside a sandbox: no keys, no other user's data, no way out but the three hosts the policy names.
  • An assistant that changes things but can always be undone.
  • A memory users can read and edit, with the rules for what's kept written in code, not left to the model.
  • Spend that can't run away, by construction.
  • Numbers from the live system: 900 turns over a 15-hour soak with none failed or empty (p95 7.9 s); a ten-turn conversation at p95 5 to 10 s and about $0.05; the tools called in 40 of 40 turns that needed one; six sandboxes running on a 2-vCPU, 8 GiB VM with 4 GiB to spare.
  • Memory measured, not assumed: on 12 chats with expected facts, recall 1.00 and precision 0.79 to 0.92 for about 3 cents a run; the nightly job about 6 s a user.
  • A workout coach on Nemotron that keeps up with a lifter: a new plan in about 5 s for 0.14 cents, a reply mid-workout in 1 to 2.5 s.

What we learned

  • Measure the agent with history, not just single questions: with thinking off and no conversation before it, Nemotron called the tool a turn needed in 36 of 37 turns; after an earlier exchange, in 6 of 16. The second number decided the setting.
  • Read what the framework adds to the prompt. Hermes' own guidance was about half of the system prompt, and some of it pulled against our coach — one part of it was the open marker issue above.
  • On a small VM, a sandbox per user is affordable if sandboxes stop when idle: Hermes levels off at about 300 MiB, so six fit in 8 GiB.
  • Keeping the model out of anything that must be exact — progression numbers, fact scores, which user a tool acts for — made the agent far easier to trust.
  • A validator that drops bad output needs a counter next to it. The memory job's summary counts dropped operations, and that number — 24 of 24 — is what showed the first real run had stored nothing.

What's next

Releasing the assistant to every Notch user, which needs a capacity plan for sandboxes at scale and privacy disclosures; continuing each user's Hermes session instead of resending the history as text, which keeps the real tool calls in context; replies in Hebrew, Arabic and Portuguese held to the same quality bar; and bringing the agent into the workout itself, so it can log sets and adjust the session live.

Built With

Share this project:

Updates

Submission history