jaga

Type what to watch for. NVIDIA Cosmos-Embed1 watches a webcam on your laptop, and a cheap smart plug cuts power. No training, no cloud video.

Comment

Inspiration

A scenario — a composite, not a real person. Bu Sari leaves the iron on when she runs to the warung (the corner shop). Her CCTV stays silent, because an empty room with a hot iron in it has no motion.

Home cameras alert on movement, and many of the states that cause harm at home are still: an appliance on beside an empty ironing board, a fridge door left open, a door left open with nobody there. A plug timer cuts the iron in the middle of ironing, or leaves it on long after she has gone. What was missing is a watcher you can describe in a sentence and that acts on the sentence.

What it does

jaga turns one typed sentence into a tripwire on a camera feed that switches a plug. In real use the iron is plugged into a Wi-Fi smart plug (in our tests a desk lamp stands in as the load and the iron is an unplugged prop); you type "nobody is at the ironing board".

  • NVIDIA Nemotron 3 Super on Nebius Token Factory compiles the sentence, under a JSON schema, into a TripwireSpec: 3–5 short action-style English captions of what the camera sees when the sentence is true, 3–6 hard negatives of the same place when it is false (for example "someone ironing a shirt on the ironing board"), a dwell and an action. You see the captions and the negatives before arming, and choose what happens: cut the plug, or just notify.
  • NVIDIA Cosmos-Embed1-224p runs on the laptop. Every window of the last 8 frames, sampled at 1 frame per second, becomes one 256-d vector in the same space as the captions — a sentence becomes a classifier with no training.
  • Deterministic code decides. The margin between the best positive caption and the best negative caption has to hold for most windows across the whole dwell, including the newest. A 2-minute capture of the normal state at arming sets the reference. Only then does jaga call the plug over the home network and cut the power; the overlay shows OFF only after the plug confirms it. The model chooses where to look; the code decides when to act.
  • A Nebius Serverless Job recalibrates each tripwire's threshold from labelled margins (jaga calibrate): a per-tripwire ROC, the threshold at recall 0.9. The work is light; running it as a Job keeps calibration off the watcher's machine and leaves a receipt for every run (docs/RUNS.md). The threshold the pre-registered soaks use is fitted on the laptop from the labelled clips and frozen before them.

What leaves the laptop: the typed sentence (to Token Factory), labelled margins (numbers and labels, no vectors) to the Job through Object Storage, and — only if you turn it on — a one-line push. Video frames never leave the laptop.

Safety, enforced in code and tested (against a fake plug so far; the first real-plug run is still ahead). A trip can only cut power or notify; switching back on is a person's job (no ON call is reachable from a trip). One bad window cannot cut power. The overlay never claims OFF before the plug acknowledges it. A fridge or a freezer can never be cut. jaga is an extra guard, not a fire-safety device.

What is measured (and what is not yet). Measured, from scripts/bench.py into DEMO.md: Cosmos-Embed1 runs at p50 1.24 s per clip on Apple MPS and 3.09 s on CPU (fp32, M1 Max, 200 timed calls each; the laptop was busy, and DEMO prints the load), and the live loop's window period is p50 1.27 s. The 187 tests include an exhaustive check of the decision rule over all 49,152 above/below patterns against an independent oracle. Not run yet: the live loop on a real webcam with a real plug (so far it has run on recorded files, and the plug drivers only against fakes), clip accuracy (AUC and precision on labelled clips), the pre-registered soaks, and outside people's sentences — DEMO.md says "Not yet run" for each, and this page will not quote a value that DEMO.md does not print. Three events are planned: an ironing board left unattended (cuts power), a fridge door left open with nobody there (notify), and a door left open (notify, typed live in "fresh mode" with no labelled clips). One room, one person; not "any event".

How we built it

  • Perception — nvidia/Cosmos-Embed1-224p via transformers, fp32 on Apple MPS or CPU (jaga/embed.py): get_video_embeddings for each 8-frame window, get_text_embeddings for the captions. The model card says it was tested only with BF16 on Ampere and Hopper GPUs, on Linux; we pin the model revision and library version, read video with PyAV (decord has no macOS arm64 wheel) and resize every frame to one canonical size first so a live window and a recorded clip reach the model through identical pixels. Between CPU and MPS the six caption cosines on the card's own javelin example agree to 6 decimals (evidence/day0/).
  • Compiler — nvidia/nemotron-3-super-120b-a12b on Nebius Token Factory (jaga/compile.py): structured JSON output in the OpenAI {name, schema, strict} shape (the docs' shape is rejected, DX-REPORT item 6), max_tokens 4096 with reasoning_effort: low because Super is a reasoning model, and an empty reply is an error, never an invented spec. The request ID that produced each spec is stored with the tripwire, shown in the compile drawer and listed at /receipts. The three tripwires to be measured were each frozen from one compile made before any clip was recorded.
  • Calibration — Nebius Serverless Jobs + Object Storage (jobs/calibrate.py, standard library only): nebius ai job create on cpu-d3 / 4vcpu-16gb, stock python:3.11, no image build. The Job holds no account credential, only two single-object presigned URLs that expire after 2 hours; a result is adopted only if it matches the input that was sent (run ID, input hash, row count) and its threshold is finite and in range.
  • Watcher and overlay — Python 3.11: OpenCV capture at 1 fps, FastAPI + SSE on 127.0.0.1:8765 (never the LAN), SQLite, a single static overlay page (label tape, margin trace vs threshold, dwell bar, breaker toggle, schedule card, event log).
  • Actuator — tinytuya (Tuya "Smart Life" plug, local key) with a Shelly Gen3 local-RPC driver as the alternate (neither driver has run on a real plug yet); a fake LAN plug server exists only in tests and CI.
  • Verification: 187 pytest tests (safety invariants I1–I3, decision rule, guards, measurement rules; no model or key needed), scripts/verify_offline.py (an armed tripwire trips and a fake plug is switched OFF while every non-LAN connection is refused), ruff, pip-audit, gitleaks, a pre-registered soak protocol committed before any data.
  • New work. jaga was created during the submission period; its first commit is dated 2026-10-15.

Challenges we ran into

  • A CUDA/BF16-only model on a laptop. The Cosmos-Embed1 card's example hard-codes .to("cuda", dtype=torch.bfloat16) and decord. We ran the card's own example unchanged except for device, precision and the video reader, on CPU and MPS in fp32, and measured it ourselves. The speeds are ours, not NVIDIA's.
  • Making "nobody" visible. An unattended board has no motion, so the contrast has to come from the text side: Nemotron writes hard negatives ("someone ironing a shirt on the ironing board", "person standing at the ironing board folding clothes") that the margin is scored against. Whether that works is exactly what the labelled clips and the pre-registered soak measure — and it is not measured yet.
  • A reasoning model behind a structured-output endpoint. With a small max_tokens Super returns only reasoning, and at the recommended temperature the proposed action and dwell varied between repeat compiles of one sentence (the first fridge compile, before a prompt revision, proposed cutting the fridge's power). So the action and dwell are an explicit choice shown before arming, a code-side list can never cut a fridge, and K = 5 extra compiles per sentence are recorded (what that variance does to accuracy is not measured yet).
  • An honest measuring harness before any data. The soak protocol, the leave-one-out threshold, the baselines (a stillness timer, a HOG person detector, plug countdown timers) and the "real but bad" re-scoping clause were written and committed before a single clip existed, so a weak result cannot be hand-tuned into a good one.
  • The macOS camera permission. A detached process is refused the camera, so the live loop also accepts a recorded file through the same capture path.

Accomplishments that we're proud of

  • Cosmos-Embed1 and Nemotron 3 Super are the engine of one flow, not decoration: the sentence → captions → clip vector → margin → dwell → plug path runs end to end, with the Token Factory request ID stored with the tripwire and listed at /receipts.
  • The safety rules are tests, not prose: no ON call is reachable from a trip, and the decision rule is checked against an independent oracle over all 49,152 patterns; a mutation check shows which broken versions of the rule its tests catch (7 of 8 mutants; the 8th cannot change its behaviour at the shipped settings — evidence/runs/2026-10-17-decision-mutation.txt).
  • Nothing in the measurements is faked: the live loop is not mocked (the plug path has only run against fake plugs so far), the plug driver refuses to claim OFF without an acknowledgement, and every surface says what has and has not been measured.
  • A real Nebius Serverless Job with a receipt (aijob-e01e9n3qk4qq8zzk13, a labelled smoke run on a synthetic fixture — not a calibration) and a friction log (docs/DX-REPORT.md, 17 items) from building on all three platforms.
  • Accuracy and soak results: not run yet — this bullet is rewritten from DEMO.md after the measurements exist.

What we learned

  • A joint video–text embedder turns a sentence into a classifier, but only if the sentence is rewritten into the register the model was trained on; the card says so, and a language model is the right tool for that rewrite.
  • For an absence event the interesting evidence is the hard negative, not the positive: "nobody is at the board" is only meaningful against "someone is at the board but still".
  • Pre-registering the decision rule and the soak before collecting data changes how you build: the tools came first.
  • The model chooses where to look; the code must decide when to act. Anything that switches mains power should be a small, exhaustively tested rule, not a model output.

What's next

The 336p Cosmos variant at a measured cost; a per-tripwire nearest-neighbour from the user's three optional examples, only if an ablation justifies it; a Shelly-only, fully local kit; more than one room and more than one person.

Thank you for your time reviewing this project.

Built With

  • fastapi
  • hugging-face-transformers
  • nebius-object-storage
  • nebius-serverless-jobs
  • nebius-token-factory
  • nvidia-cosmos-embed1
  • nvidia-nemotron-3-super
  • opencv
  • pyav
  • pytest
  • python
  • pytorch
  • sqlite
  • tinytuya
  • vercel
Share this project:

Updates

Submission history