Indie App Ops Agent — Nemotron Edition

• 

  We run three production apps as a solo developer — reviews, crashes, and revenue never sleep, even when we do.

• 

  Open source, Apache 2.0. Runs with nothing but a Nebius Token Factory key.

• 

  Not one model doing everything. Five NVIDIA Nemotrons, five different jobs, checking each other.

• 

  The generator never grades its own homework. An early draft guessed a cause with zero data behind it.

• 

  Multi-tenant, not a demo. Register your own apps — each account gets a structurally isolated workspace.

• 

  Every run, replayable. Tool calls, model calls, measured tokens, measured cost — five guards for under a cent.

• 

  This agent's real product is what it does not send — every suppressed interruption has a deterministic reason.

• 

  It drafts the reply. It never posts it. A second, independent Nemotron audits every draft before a human sees it.

• 

  We once shipped a paywall nobody could read. The vision model catches it: "light gray on dark navy, poor."

• 

  The console: seven items reached a human overnight. Everything else was classified, drafted, and filed silently.

• 

  A paying customer, a broken feature, a refund demand — escalated with confidence 3/3 from an independent vote.

Inspiration

We ship three production apps on Google Play as a one-person team — an AI diary, a queue-time predictor, and a bedtime-story generator. The apps make money while we sleep; the operations don't sleep at all. Reviews arrive in two languages, crash rates spike after releases, trial conversions sag quietly, and store screenshots ship with defects nobody notices until users do. We built the agent we needed: one that does the routine work end to end and interrupts a human only when a real decision is needed.

The Nemotron edition started as a port and became a rethink. Moving to Nebius Token Factory put a family of NVIDIA models behind one OpenAI-compatible API — so instead of one model doing everything, we redesigned the agent around five Nemotrons with five different jobs, checking each other.

What it does

Four guards plus a weekly report run unattended:

• Review guard — fetches new reviews, classifies them, drafts replies in the review's own language matching each app's tone guide. Interrupts only for ratings ≤2 or refund/policy risk.
• Revenue guard — tracks MRR, trials, conversion, churn per app. Interrupts only on anomalies vs. each metric's own baseline.
• Crash guard — watches crash rate and DAU, correlates spikes with releases.
• Store guard — a multimodal Nemotron looks at store screenshots: transcribes every text, rates its legibility, scans for truncation and policy-risk claims.
• Weekly report — the one run that always speaks.

Around them, a trust layer:

1. Independent auditor. Every reply draft is inspected by a different Nemotron (3.5 Lightning) with an adversarial brief: speculation about causes, invented contact info, wrong-platform instructions, language mismatch. It blocks-with-reasons; it never rewrites, never censors, and fails open — flagged drafts still reach the human.
2. Confidence panel. Two more independent reviewers vote on every escalation through different lenses (can the developer act today? does harm grow if this waits 48h?). Votes annotate — "confidence 3/3" — and never suppress.
3. Silence ledger. Every interruption the agent chose not to make is recorded with a deterministic, data-backed one-line reason. The console renders it: "N interruptions suppressed." A product whose value is silence has to prove the silence is judgment, not neglect.
4. Run traces & cost accounting. Every run is replayable — each tool call, each auxiliary model call, measured tokens, measured cost. The sidebar shows what the agent's own inference actually cost.

How we built it

• Strands Agents SDK (Python) for the agent loop; five thin, auditable tools; anomaly arithmetic in plain unit-tested Python — math in code, judgment in the model.
• Nebius Token Factory (OpenAI-compatible API) serving five NVIDIA models with distinct roles: Nemotron 3.5 Lightning (guards + auditor + panel), Nemotron 3 Nano Omni (vision), with per-guard overrides for Ultra 550B.
• FastAPI console — inbox with audit verdicts and panel votes, cross-app revenue report with sparklines, run timeline, silence log. Bilingual (EN/JA), Ctrl+K command palette.
• 36 unit tests; three modes (offline / demo / live) so judges can run the real agent with nothing but a Token Factory key.

The non-obvious parts

Making a vision model useful for contrast QA. Our worst production defect was a paywall whose hero copy was nearly invisible. First attempt: ask Omni to "find contrast defects" — it found none, because a VLM reads faint text effortlessly; model eyesight is superhuman. What worked: have the model transcribe every text and rate per-element legibility by comparing text color to background, then diff that in plain code against the copy the developer intended to be readable. The seeded low-contrast paywall (a reproduction of our real shipped bug) now escalates as poor: light gray text on dark navy background — while clean and borderline screens stay silent.

Measuring instead of assuming. indie-ops bench runs identical workloads across models and reports only measured numbers. Finding #1: smaller isn't faster — Nano 30B was the slowest model we tested (23k thinking tokens per run); Lightning won every guard at ~$0.001/run. Finding #2: reasoning models eat their own token budget — an audit call with max_tokens=400 returns empty JSON because the chain of thought consumed it first.

Designing for endpoint rotation. Mid-development, the Super 120B endpoint was removed from the public catalog with a day's notice, and our agent loop silently hung retrying 404s. The runner now preflights every model id against the live catalog and fails fast with the current model list. (The benchmark's winner turned out to be the survivor.)

Challenges we ran into

Getting a generator to stop grading its own homework (solved with a second model, not a bigger prompt); making silence auditable without burning tokens (solved with determinism, not generation); keeping every number in this submission a measured one.

Accomplishments we're proud of

The trust layer as an architecture — generation, audit, and interruption-calibration split across independent models — and a console we'd genuinely use for our own three apps. Total inference cost for the entire development and demo: under $1 on the trial credit.

What we learned

Multi-model design isn't about picking the best model; it's about giving models adversarial and complementary roles. And: measure everything, because half our assumptions (smaller = faster, VLMs see like humans) were wrong.

What's next

Live-mode rollout on our three production apps; a calibration dashboard that scores the confidence panel's votes against what the human actually did; store-guard support for screenshot diffs between app versions.

Built With

• fastapi
• nebius-token-factory
• nvidia-nemotron
• openai-sdk
• python
• strands-agents