-
-
-
-
-
Frame Journey
-
Inspiration
Deciding to do the thing was never the hard part. Keeping track is. Every habit tracker solves the easy half and then asks you to self-report the hard half, which is exactly the step a person skips on the day it matters most.
A camera can close that loop. The obvious way to use one is to send your day to a frontier vision model and ask it questions, so we priced that first. At 8 fps, one camera, it comes to about $421 a day. Nobody is running a habit tracker that costs twelve thousand dollars a month per room.
So the whole project became one question: how do you watch continuously, for almost nothing, and still be right?
What it does
You type a goal in plain English. Nemotron compiles it into skills, where a skill is one visual question with 1152 floats behind it. Every skill scores against a single shared frame embedding from NVIDIA C-RADIOv3-B, so asking fifty questions about a frame costs one forward pass plus some arithmetic. A state machine turns thousands of scores an hour into one event. When a moment is genuinely ambiguous, and only then, four frames go to Nemotron 3.5 Super VL. Anything a camera could never know, like the weight on a bar, it asks you, and your answer becomes a label that sharpens the detector.
Two models do all the perception. Four of the seven stages use no model at all.
The two cheapest decisions happen before any model runs. A still room is rejected in the browser, so it never becomes a request at all. And only state transitions travel back up to the ledger, so the control plane never holds a raw score.
It works on rooms and on shared screens, so "brush your teeth" and "study for an hour" run through the same pipeline. It takes feeds from a phone, a laptop, a screen share, or a thirty dollar ESP32 board over RTSP.
The one idea
C-RADIO is the vision encoder inside Nemotron 3.5 Super VL. That is not a coincidence worth mentioning, it is the reason the architecture holds together.
The cheap 24/7 index and the expensive cloud teacher share one visual representation. So when the teacher adjudicates a frame, its verdict lands as a label in the same 1152-dimensional space the skill's own weight vector lives in, and the skill can refit against it directly. A verdict from a model with a different encoder would be an opinion the skill has no way to absorb.
The models never change. C-RADIO is frozen and Nemotron is frozen. What improves is the 1152 floats sitting on top, which is why "it gets better" is something we built rather than something we hope for.
How we built it
Three processes, and the boundary between two of them is the whole point.
A Cloudflare Worker holds the goals, the skills and the ledger in two Durable Objects, and brokers the WebRTC handshake. Video itself never enters the Worker. A Python perception service does the high-frequency half: sample, gate, embed, score, run the state machine, escalate when stuck. It dials out over a single WebSocket, so it needs no inbound port, no TLS and no DNS, and the identical process runs on a laptop behind a home router or on a Nebius VM with nothing changed but one flag.
Where the NVIDIA models sit. nvidia/C-RADIOv3-B plus its siglip2 adaptor embeds every sampled frame. nvidia/nemotron-3-super-120b-a12b is the planner that compiles a sentence into a skill tree, and the voice assistant. nvidia/Nemotron-3.5-Super-VL is the teacher, called on roughly a tenth of a percent of frames.
Where Token Factory accelerated the build. It is OpenAI-compatible, so our client is a base URL, a bearer token and one method, and all of our effort went into the parts that are actually hard. More importantly, per-token pricing on a 120B model turns "call the expensive model on a tenth of a percent of frames" into an architecture instead of a wish. A camera at 8 fps is 28,800 frames an hour through the encoder, and the teacher sees four of them. At frontier-API rates we would have built something worse.
Nebius Serverless Endpoints run the one stateless piece. Everything else in perception holds state and belongs beside the cameras, but the encoder is a pure function, so we containerised it as pact_perception.serve behind a managed HTTPS endpoint. One image serves both roles because ENTRYPOINT is python -m and the module is the command, which is what Endpoints need given they let you set the arguments and not the entrypoint. It reports its own identity at /health, because a vector only means anything against a weight vector from the same encoder, and being wrong there is silent.
Nebius AI Cloud Compute VMs are what we actually ship. deploy.sh puts the whole watcher on one machine, and defaults to CPU deliberately.
Tavily fixes a consistency problem. Pact's claim is that it will not take your word for whether something happened, and that sits badly beside a target the system invented. "Do some squats every day" names no number, so one gets looked up and the goal keeps the citation. You then argue with the guidance instead of with Pact. It only searches when the sentence has no number, and it keeps a cited URL only if it was actually among the results, because a model asked for a citation will sometimes produce the shape of a reference instead.
Does it actually save anything
The always-on half runs on hardware you already own, and that is measured rather than projected. C-RADIO at 256px costs 68ms a frame on the demo laptop's own RTX 3050 Ti, which is 14.7 fps, comfortably past the 8 fps a rep counter needs. On the same machine's CPU it is 530ms. A gate samples at 0.5 Hz.
| per camera, per day | 0.5 fps | 8 fps | how we know |
|---|---|---|---|
| Every frame to the VLM | $26 | $421 | projected: $0.00061 × fps × 86400 |
| C-RADIO, on the laptop already in the room | $0 | $0 | measured: 14.7 fps on its RTX 3050 Ti |
| The teacher, on ambiguous moments only | ~$0.02 | ~$0.28 | per-call price measured, rate estimated |
The $0.00061 is what our teacher reports back on a real call, so the first row is arithmetic you can check rather than a figure we quote. We never ran the naive version. That row is what it would cost, not what it cost us.
The claim is about the shape of the curve. Naive is linear in frames times cameras times goals. This is one encode per frame and then arithmetic, so fifty skills add 57,600 multiply-adds to a single forward pass.
Challenges we ran into
Four of our assumptions about C-RADIO were wrong, and three came from authoritative sources. The adaptor is siglip2, not siglip2-g. Embeddings are 1152-d, not 768. AutoModel cannot do text at all, because that repo ships empty adaptor_configs and no adaptor tensors, so the weights only live inside the .pth.tar you reach through torch.hub. And image and text are two separate downloads.
The thresholds in our own design could never have fired. This is the one that would have sunk the project silently. sigmoid(w·e) with no scale or offset puts every score between 0.475 and 0.599, so an enter threshold of 0.75 is not badly tuned, it is unreachable, and the system would have reported nothing forever while every log line looked healthy. Borrowing SigLIP's own learned scale fails the other way and collapses the usable band into 0.013 of cosine, narrower than frame-to-frame noise. Every skill now carries its own bias and temperature, fitted from 60 frames of its own room, needing no labels.
A screen is not a room, and five assumptions break quietly. The motion gate rests on stillness meaning nothing is happening, which is true of a room and false of a screen, since reading for twenty minutes is motionless and is exactly the behaviour a goal wants to credit. Screens are never hard-gated. They also upload at 1280px and reach the teacher at 1024px, because at 512px a desktop's body text is a grey smear and the one component that could tell Python from JavaScript answers UNSURE forever.
On a screen the content is the noise. A gate like "is this a short-form video player" is answered by the shape, but the clip inside changes every few seconds and the encoder averages the whole frame, so the clip is what the score follows. So every skill declares a view. Measured on our test fixture, two frames of the same player with different clips sit 21.7 pixel levels apart raw and 0.73 apart under layout, a 29.8x collapse, while the panel still stands 110.7 levels above the ground beside it. The privacy falls out for free, because at that grid a page of text is a texture, so the embedding records that you were reading and not what.
Nemotron returns HTTP 200 with empty content when max_tokens is too low. It spends around a hundred tokens reasoning before answering, billed against the same budget. At the call site that is indistinguishable from a network failure, and it sent us debugging the wrong layer entirely. Our floor is 512 and we read reasoning_content separately.
A Cloudflare Worker cannot reach Nebius. It gets 522, while the identical request from api.github.com gets 200. So the Worker dispatches a GitHub Actions runner to start the encoder machine, and the runner's spin-up hides inside the ninety seconds the machine spends loading weights anyway.
os.cpu_count() reports the host, not your allocation. A 16-core endpoint encoded a frame in 208ms while a 4-thread laptop did it in 195ms, because torch started 192 threads on a 192-core host and they contended instead of working. More hardware, less throughput, and nothing in any log.
What we learned
After the second wrong assumption we stopped reasoning about hardware and started measuring it, and that habit is what caught the unreachable thresholds.
The same habit changed how we write prompts. rank_labels.py embeds candidate label phrasings and scores them against frames we actually captured, on separation over spread. That turned prompt engineering from taste into evidence, and it overturned the published advice: the recommended template for this model family, "This is a photo of {}.", bought about 5% more separation for 24% more spread on our frames, which makes it a worse detector for us. Those templates are tuned for cross-class ranking where only the argmax counts. We hold one question against a fixed threshold over time, and there a wobble is a false positive.
The third lesson is about where to put adaptation. Freezing both models and learning a 1152-float probe on top is what makes a frontier VLM's verdict reusable instead of merely consumed. Fine-tune the encoder and every vector already in the store becomes meaningless, so the system would forget everything it knew each time it got better.
What's next
Proper auth, because claim codes only stop a stranger who finds the join URL. Per-window redaction for screen shares. And the ESP32 vision layer, since the motion tracking already runs on real hardware.
Built with
nvidia/C-RADIOv3-B · nvidia/nemotron-3-super-120b-a12b · nvidia/Nemotron-3.5-Super-VL · Nebius Token Factory · Nebius AI Cloud · Nebius Serverless Endpoints · Tavily · Cloudflare Workers · Durable Objects · Cloudflare Realtime SFU · WebRTC · TypeScript · Python · PyTorch · numpy · SQLite · siglip2 · ESP-IDF · ESP32-S3 · GitHub Actions · Remotion
248 tests, all green. First commit 24 October 2026, so everything here was built during the submission period.
Log in or sign up for Devpost to join the conversation.