POK-Agent

A Windows computer-use agent: a large Nemotron plans, a small Nemotron makes the quick on-screen decisions, and repeated tasks get cheaper. Researches with Tavily. Always on via Telegram.

Comment

Inspiration

Computer-use agents call a large model for every click, which is slow and expensive. People do not work that way: a fast system handles the routine and a slow one handles the unfamiliar. The track brief says to use Ultra or Super for serious reasoning and Nano for the fast, everyday calls. That is the System 1 / System 2 design we had been researching, so we put it on Nebius Token Factory.

What it does

POK-Agent drives Windows apps and the web.

  • Plans with Nemotron. nvidia/Nemotron-3.5-Super-VL (vision and tool calling) on Token Factory sees the screen and writes the plan.
  • Acts with a small Nemotron. Bounded steps ("press Equals", "is the page loaded?") go to a small model as System 1. In Settings, System 1, Any model, you pick a provider and model (for example Nemotron Nano on Nebius). POK-Agent asks it each typed question twice with the options shuffled and trusts the answer only when both agree. It is called straight from Rust: no sidecar, no extra software.
  • Rust owns safety. Redaction, confidence gates, approvals, no password fields, no elevated windows, observation freshness and post-action verification. The models never act directly.
  • Remembers. SQLite memory, reusable skills, and recorded "muscle memory" programs that System 1 replays without calling the planner.
  • Researches with Tavily. Tavily is the default web search as soon as a key is saved (details below).
  • Always on. A Telegram gateway with a default-deny allowlist and one-time pairing codes, permission prompts you answer from the chat, scheduled tasks, and a system tray so it keeps working with the window closed.

Best Use of Tavily

Tavily is part of the runtime, not an add-on. Save a key once (Settings, Web search) and Tavily becomes the default search:

  • web_search returns Tavily's ranked results plus its short synthesized answer (marked "verify"), and takes topic = news, time_range (day, week, month, year) and a deep mode. In the demo the agent chose topic: news, time_range: week on its own, checked the sources, and saved three headlines with their links to a file.
  • web_fetch falls back to Tavily Extract for pages that block ordinary requests or need JavaScript (never for local addresses).
  • If Tavily is unreachable, search falls back to DuckDuckGo and says so in the result.
  • The key is verified with a test search before it is stored in the Windows Credential Manager. Settings has a "Try it" box that runs one real search.

How we built it

A Rust workspace (core runtime, CLI, Windows adapter) plus a Tauri 2 and React dashboard. The planner and System 1 are plain OpenAI-compatible calls to https://api.tokenfactory.nebius.com/v1/, so one key serves both. We found Tauri-side bugs by driving the real app over DevTools: a deadlock in the gateway status command froze the window until we traced it.

Results (measured on Token Factory)

57 router questions from real screen captures, through the production request code (targets / done-checks / conditions, answers accepted but wrong, median latency):

  • Nemotron-3-super-120b-a12b: 19/19, 27/27, 10/11; 1 wrong; 1.2-2.7 s
  • Nemotron-3-Nano-30B-A3B: 18/19, 23/27, 10/11; 4 wrong; 1.0-2.1 s
  • Nemotron-3.5-Lightning: 18/19, 16/27, 10/11; 2 wrong; 0.8-1.6 s

Live run in the real desktop app (Nemotron-3.5-Super-VL planner, Nemotron Nano as System 1, "open Calculator and multiply 17 by 23"): 5 planner requests, 7 steps carried out by System 1, 28 completion checks, 75 s, and 391 verified on screen. The first attempt landed on the wrong number; the planner noticed, cleared and redid it. An earlier run of the same task needed 9 planner requests.

Carried over from earlier work (a different planner and System 1, not Nemotron): on Windows Agent Arena (154 tasks, clean Windows 11 VMs), on the 26 tasks where System 1 replayed a whole recorded program, the planner needed 28% fewer calls on average (20.6 to 14.8) at the same success. Across all tasks the average held flat at 23.0, because only a third had a recorded program after one pass.

Challenges

Reasoning models spent the whole token budget thinking and returned empty answers until thinking was switched off per request. Under load Token Factory answered slowly or with HTTP 529 (ten concurrent small calls took 10 to 20 s around midday, against about 1 s in the morning), so System 1 retries once and hands the decision back to the planner. Small models over-accept: Nano judged "done" wrongly more often than Super, which is why the confidence gates stay high.

Accomplishments that we're proud of

A hosted Nemotron System 1 that sits inside the agent loop without a sidecar, measured rather than assumed; an agent that verifies its own work and recovers from a wrong result; and an always-on gateway that is safe by default.

What we learned

Super is the accurate System 1 and Lightning the fast one. The confidence gates, not model size alone, decide whether a small model is safe to trust.

What's next for POK-Agent

Discord and WhatsApp adapters behind the same gateway, a Nemotron run of the full Windows Agent Arena, and fine-tuning a local System 1 from the router training log.

Try it

Needs Windows 11 x64 and your own Nebius Token Factory key (a free Tavily key is optional). Download the installer from the v0.1.12 release, open Settings, Model, choose Nebius, paste the key and pick nvidia/Nemotron-3.5-Super-VL. The build is unsigned, so Windows SmartScreen may warn. The README has the full setup, and the CLI also runs on Linux as a coding and research agent.

Built With

  • nebius-token-factory
  • nvidia-nemotron
  • openaiapi
  • react
  • rust
  • sqlite
  • tauri
  • tavily
  • telegram-bot-api
  • tokio
  • typescript
  • vite
  • webview2
  • windows-ui-automation
Share this project:

Updates

Submission history