Permafrost

A $60 edge guardian for vaccine fridges: millisecond local reflexes, cited cloud-Qwen diagnosis, and a blackout-proof, tamper-evident cold-chain log.

Comment

Inspiration

A nurse in a two-room clinic opens the fridge on Monday and finds Friday's power blip silently cooked $8,000 of childhood vaccines. The monitor that could have warned her was unplugged months ago — because it screamed at every defrost cycle. That false-alarm fatigue is the real killer of the cold chain: a defrost spike (+3.2 °C) and a door-ajar failure (+3.4 °C) look nearly identical, threshold monitors alarm on both, and so staff stop trusting — and then stop plugging in — the very device meant to protect the stock. Permafrost exists to kill that fatigue: to earn its silence by reasoning about why a spike is benign instead of just yelling at it.

What it does

Permafrost is a $60 edge guardian for clinic vaccine fridges. A Raspberry Pi with two DS18B20 probes, a door reed switch, and a buzzer samples every 10 seconds and reacts in under 100 ms from local rules. Ambiguity — the last few hours of the temperature curve, ECIES-sealed, never raw streams — goes to a cloud Qwen brain that reasons like a refrigeration engineer and returns a structured ExcursionVerdict:

  • "Sawtooth every 6h = defrost cycle, benign" — no alarm, and it tells you why;
  • "Door-ajar signature; stock at risk in 22 minutes; alarm now" — each CRITICAL verdict citing cold-chain (CDC Vaccine Storage & Handling) guidance retrieved via text-embedding-v4 (invariant I4: every critical verdict carries ≥1 citation).

Verdicts execute as typed function calls (sound_alarm, notify, annotate_log, schedule_service), and a qwen3-tts-instruct-flash alert is instruction-controlled to sound "urgent but calm."

The clever loop: during calm periods qwen3.6-flash distills the cloud's own verdict history into a compact local IF/THEN ruleset, Ed25519-signs the bundle, and the edge verifies the signature before hot-swapping — refusing any unsigned, forged, or downgraded bundle. Pull the cable and the rules keep protecting the fridge; reconnect and the sealed event queue syncs into a gap-free hash chain. Compliance is cryptographic: every event extends a SHA-256 hash chain (WAL-crash-safe through blackouts), daily Merkle roots are Ed25519-signed, and verify-chain proves tamper-evidence. The numbers: classification accuracy 1.000 on the 4-class fixture set; reflex latency p95 < 0.01 ms against a 100 ms budget; ~$60 device vs $1–3k/yr commercial monitors.

How we built it

Python 3.12, with a FastAPI cloud brain and a Typer edge CLI. Edge state is SQLite in WAL mode (24h ring buffer + hash-chain log). Target hardware is a Raspberry Pi 3B+/4/5 with DS18B20 (1-Wire) probes, a reed switch, and a buzzer (guarded gpiozero / w1thermsensor imports; replay from seeds/*.csv is the supported, hardware-free judging path — judges need zero hardware).

The whole reasoning tier is built on Qwen Cloud: qwen3.7-plus + thinking for multi-step reasoning over slopes/periodicity/humidity co-signals into an ExcursionVerdict; structured output as the actuator contract; function calling for the typed tools; text-embedding-v4 for cited guidance retrieval; qwen3.6-flash for cheap rule distillation; qwen3-tts-instruct-flash for the spoken alert; and a Batch API plan for −50% weekly compliance sweeps. A transport switch picks FakeQwen (default, deterministic, fixture-backed, no key) vs LiveQwen (behind DASHSCOPE_API_KEY + PERMAFROST_LIVE=1, endpoint dashscope-intl.aliyuncs.com/compatible-mode/v1). The live path is wired end-to-end and was smoke-tested with a real DashScope call (a bogus key drives a real round-trip returning an authentic Model-Studio 401 with a real request_id — reproducible keylessly); the graded, deployed, and demo path deliberately runs the offline-deterministic FakeQwen engine so every byte is replayable without a key.

The FastAPI brain is deployed live on Alibaba Cloud Function Compute (managed python3.10 runtime) at https://permafrost-hoznckcsox.ap-southeast-1.fcapp.run — GET /health (liveness), GET /run (runs the door-ajar replay in the cloud on FakeQwen, returning verdicts + chain summary), and GET /verify (re-derives the hash chain and runs the offline-degradation checks in the cloud). The deployment is defined in infra/fc/ (handler.py FC entrypoint → the FastAPI app, plus s.yaml). Quality engineering: ruff + mypy, 331 pytest tests at 100% coverage (all offline, keyless), the I1–I4 invariant suite, a verify_offline.py proof that kills real sockets mid-replay, CodeQL SAST, Dependabot + pip-audit SCA, TruffleHog secret scanning, and a 5-stage GitHub Actions pipeline with semantic-release.

Challenges we ran into

  • Killing false alarms without going blind: a defrost spike and a door-ajar failure look nearly identical, so the edge reflex stays dumb-and-fast while the cloud brain does the discrimination on periodicity and co-signals and cites guidance so the "benign" call is defensible, not a guess.
  • Trusting rules the cloud wrote: letting a distilled ruleset hot-swap onto a device is a supply-chain risk, so the edge verifies an Ed25519 signature before activating any bundle — and we tested that it refuses forged and downgraded ones.
  • Surviving a blackout with an intact record: the hash chain is WAL-crash-safe, and verify_offline.py cuts real sockets mid-replay to prove the edge keeps alarming and queues sealed events with no gap, then reconnects and syncs a chain that still verifies.

Accomplishments that we're proud of

  • 1.000 classification accuracy on the fixture set and p95 < 0.01 ms reflex latency — anti-false-alarm without losing the save.
  • An agent that cryptographically ships its own offline fallback: cloud-authored rules, edge-verified signature, graceful degradation.
  • Tamper-evident cold-chain compliance (hash chain + signed daily Merkle roots) proven by 1-byte-tamper tests.
  • 331 tests at 100% coverage on a fully offline, deterministic replay path — zero keys, zero hardware.
  • A cloud brain deployed live on Alibaba Function Compute that reproduces the exact judging bytes, keylessly, from the cloud.

What we learned

The value of an edge agent isn't the alarm — it's the silence it earns. A monitor that reasons about periodicity and cites why a spike is benign is a monitor that stays plugged in. And the most elegant degradation story is one the cloud writes itself: when the smart tier is unreachable, the fallback isn't a hardcoded threshold, it's the cloud's own distilled, signed judgement running locally.

What's next for Permafrost

  • Archive daily Merkle roots to Alibaba OSS (the FastAPI brain is already deployed live on Alibaba Function Compute).
  • One full live keyed Qwen round-trip captured end-to-end — a whole permafrost replay graded through live Qwen with saved verdict text, a real qwen3-tts-instruct-flash alert, and a real Batch API weekly report (the live path is wired and smoke-tested; the graded/demo path stays on offline FakeQwen by design).
  • Run the physical GPIO rig (DS18B20 + reed + buzzer on a Pi), build the Watch/Ledger dashboard UI, and publish permafrost-edge to PyPI.

Built With

  • alibaba-cloud
  • codeql
  • dashscope
  • ds18b20
  • ecies
  • ed25519
  • fastapi
  • function-compute
  • github-actions
  • iot
  • mypy
  • pydantic
  • pynacl
  • pytest
  • python
  • qwen
  • qwen3-tts-instruct-flash
  • qwen3.6-flash
  • qwen3.7-plus
  • raspberry-pi
  • ruff
  • sqlite
  • text-embedding-v4
  • typer
  • x25519
Share this project:

Updates