Inspiration
Vision and hearing don't fail all at once - they fail so slowly that the person losing them adapts faster than they notice. Nobody wakes up and decides their hearing is worse; they turn the volume up one notch at a time until "louder than everyone else needs" feels normal. The same slow adaptation shows up in screen brightness, in comfortable text size, in reaching for captions on a video that used to play fine without them. None of that gets reported to anyone. It just becomes the new normal a phone quietly remembers and a person forgets.
We looked for prior art and found two dead ends. Active self-screening apps exist, but they require a deliberate test, real friction that most people try once and never repeat. Passive measurement research exists too, apps like OverSight that log real device signals for clinical study, but it stops at data collection; a researcher or clinician still has to interpret the export. Nothing sits between "took a test" and "handed a spreadsheet to a doctor." We wanted to know whether the settings someone already touches every day, with zero extra effort, could be turned into one honest sentence about a trend, one a person could actually read.
The harder question, and the one that shaped everything else: if that sentence is ever wrong, who notices? We decided the answer had to be "the code," not "a judge reading the pitch," and built accordingly.
What it does
Sensorium reads four passive Android signals — media volume, screen brightness, font scale, and caption toggles — the ones people already adjust without thinking, and turns weeks of that history into one honest, non-diagnostic report about how vision and hearing are trending. No test, no form, no new habit.
Under the hood it's an 11-node workflow, and three of those nodes touch no model at
all: a deterministic engine (SciPy linregress plus a Bonferroni-corrected change-point
test) computes every trend and every significance verdict, so a number's origin is
always one auditable place, never a language model's arithmetic. Two more nodes are the
core method: a pair of agents reason over the same week, one seeing only the numbers,
one seeing only the person's own words about how they're feeling, neither told the other
exists — so when they land on the same conclusion the agreement means something, and
when they don't, the disagreement is real rather than one prompt smoothing two facts
into a single narrative. A synthesis node grounds every claim back to that engine, a
retrieval node cites only sources actually fetched for that run, and a final node
compiles a report that opens with a fixed, non-negotiable refusal clause: never a
diagnosis, always "see a licensed professional" when the data says something's worth a
look.
It runs on a real phone. A small Android app reads the four signals with zero extra permission prompts and posts them to the exact workflow above — on a live Samsung device, it reported a 318.547% swing in volume settings and correctly refused to call it significant, while the model wrote "there is no detected change in volume settings" right above the number that would have fooled a simpler system.
How we built it
We built it in ten sequenced stages, and the discipline was that no stage was allowed to start until the previous one shipped its own acceptance-test suite. That constraint is why the system has 514 tests rather than a demo path: a stage wasn't "done" when it ran, it was done when there was a test that failed if it stopped working. The final codebase is 63 commits, each one a stage or a documented failure, and the whole thing is reproducible from a clean clone.
Order was the hardest design decision, and we got it right by starting with the constraints. JSON Schema contracts for every node's input and output were written before a single model was called. That felt like a detour at the time. It's what made everything downstream possible: a claim like "the model never invents a number" is unfalsifiable prose until every number has exactly one legal address to come from — and once it does, the evaluation harness is nearly free, because enforcing and measuring become the same code path.
The statistics are deliberately not the model's job. Node 3 is pure SciPy: a fitted
regression rather than an endpoint difference (with 6–8 samples, (last-first)/first is
hostage to whichever two readings sit at the window edges), and a nested-model F-test,
Bonferroni-corrected for having tried every split position, before any change point is
believed. The naive "did the error drop by a third" rule fired 15 times across 12
cases, including on the null cases that exist to catch exactly that. The F-test brings
it to one. This is the load-bearing decision in the whole project: significance is
computed where no prompt can reach it, so no amount of persuasive phrasing downstream can
promote noise into a finding.
Real infrastructure, not mocks. Featherless.AI routes each node to a Qwen2.5 model
sized to its stakes — 7B for narration, 32B for extraction and the blind agents, 72B for
synthesis. We pinned those ids against a live /v1/models response and then verified
each one with a real completion, because listed and callable turn out to be different
things: every meta-llama/* and google/gemma-* id returns 403 gated on our account,
so the models our design originally named were never actually available. One callable
model was rejected on merit — QwQ-32B emitted its JSON object twice in a single reply,
which is the reasoning-model failure mode structured output cannot absorb. Firecrawl
retrieves live health-authority pages so citations are checkable against what a run
actually fetched rather than what a model remembers. And SciPy replaced an
originally-specified Wolfram|One — recorded as a substitution rather than left for
someone to discover in the code.
Then we built the harness designed to beat us. Three arms, not two: a single prompt
alone, that same prompt handed our own validators and retrieved sources, and the full
workflow. The baseline runs on the largest model at temperature 0 with the union of
every input the pipeline splits across nodes — and the fairness of that setup is asserted
in tests/test_harness.py, so the suite fails if anyone quietly trims the baseline's
inputs or downgrades its model. Building the strongest possible opponent is the only way
a results table means anything, and it is why we can say precisely which row
decomposition wins instead of claiming it wins everywhere.
Finally, we built the thing the track didn't ask for. The brief scores a workflow, not an app. But the workflow consumes phone signals, and a workflow evaluated only on generated cases has an unanswered question sitting underneath it — do these signals exist outside the generator? So there's an Android client and a FastAPI service, with nothing reimplemented in Kotlin: the app is a client of the same nodes, prompts, routing and validators this repo measures, which means a guarantee in the results table is a guarantee about what the phone displays.
Challenges we ran into
Every one of these was a real failure from a live run, not a hypothetical:
- The model invented its own schema. Node 1's first prompt described behaviour, never a JSON shape, so the first live call returned a plain-prose question — and asked to repair it, the model guessed a schema by echoing its own input keys back, wrong field name and all. Fixed by generating the output contract directly from the JSON Schema and appending it to the prompt, so the text a model is held to can't drift from the check that holds it there.
- An agent's own rounding nearly broke "no invented numbers." One blind agent wrote "38.46%" for a figure whose engine value was 38.458 — technically present in "the input," if agent prose counted as input. We had to rule agent prose inadmissible as a numeric source; only the engine's own registry counts.
- A discarded p-value was quietly turning noise into headlines. The stats engine computed significance but was throwing the p-value away before the workflow ever saw it. Recovering that one value made abstention match the hidden ground truth exactly, on every test case.
- Our own independence check reported a leak on every single case — and was wrong every time. It was matching a rounded "1" from a figure against the literal string "minLength: 1" in a schema. A one-character match isn't a fingerprint; the check itself needed a test.
- A citation that would have scored perfectly was still useless. The best-scoring retrieved passage was a bare bibliography entry — real URL, real excerpt, completely unreadable as a suggestion. The citation checker had to learn to reject shape, not just provenance.
- A safety check failed on every report — because of line-wrapping, not safety. The refusal clause is hard-wrapped in its source file, so a byte-for-byte comparison was really testing whether a model matches its author's column width. Fixed the check, not the model, and safety adherence went to a clean pass on the first attempt, every time.
Accomplishments that we're proud of
- We published a row where our own architecture loses. One stability metric came in well below simpler baselines. Most submissions only show wins; we measured every metric against a fair baseline and kept the one that didn't go our way in the same table as the ones that did.
- Agent independence isn't asserted, it's proven, two-sidedly, on every test run. Our independence check re-derives from committed, real request payloads that no journal phrase reached the numbers-only agent and no trend figure reached the words-only agent — and the check itself is mutation-tested, so it can't quietly stop checking anything.
- The headline number is one the system refused to be impressed by. 318.547% is the kind of figure a demo would love to show off. Ours showed it off and labelled it noise, live, on a real phone, because significance comes from a deterministic engine and no amount of persuasive language downstream can overrule it.
- A full working app, not a mockup, in a track that didn't require one. Real Android client, real FastAPI backend, zero extra permission prompts, verified end to end on physical hardware.
- A large, fast test suite, and the parts that matter most are mutation-tested — we didn't just check that the independence proof and the numeric-fidelity guarantee pass; we broke them on purpose and confirmed the suite notices.
- Most of our submission documentation is generated from the code it describes, so the workflow diagram and reference docs can't drift from a system that changes after they're written — verified by adding a node to the routing table and confirming the build fails until it's drawn.
What we learned
The rule "never state a number that wasn't in the input" sounds airtight until an agent's own fluent prose starts looking like input — we learned to separate a source of numbers from commentary about numbers, and to trust only the first.
We learned that a check needs its own test as urgently as the code it's checking — our independence proof and our safety-clause check were each silently useless at one point (an empty fingerprint list, a whitespace-sensitive string compare) and passed brightly in green the whole time.
We learned a model can be correct in every individual sentence of a document and still produce something no one should read as a whole — a reply that stated a significant increase in one figure while also flagging the data as insufficient was never dishonest field by field; the document as a whole was.
And we learned to be precise about what the architecture actually buys: most of our guarantees came from the deterministic engine and the validators, and a single prompt handed the same tools reached the same place on almost every metric. Multi-agent decomposition earned exactly one measurable win — surfacing a conflict that a single, smoothing prompt missed — and that's the honest claim to make, not a bigger one.
What's next for Sensorium
These are in priority order, and the order is the point: the privacy work comes before any new capability, because every item below it makes the data more sensitive than it is today.
1. Encryption where we hold no key — before any new feature
This is sensory health data. It should be readable by the person it belongs to and by nobody else, including us. The target is a zero-knowledge design: keys derived and held on the device, data encrypted before it ever leaves, and a server that stores ciphertext it has no means of decrypting — so "we don't look at your data" stops being a privacy policy and becomes a property of the system, true even if we were compelled or breached.
We'd rather name the two hard parts now than discover them later. First, reasoning currently happens off-device, and a server cannot reason over data it cannot read — so this forces a real choice between on-device inference for the sensitive stages and a narrow, ephemeral decryption window that never touches disk. Second, key recovery: if only the user holds the key, losing the device means losing the history, and any "convenient" recovery path is usually a backdoor with better branding. Those trade-offs are the work. Today's architecture is a starting point rather than an answer — the phone never holds an API key and journal text crosses the boundary only when someone presses the button that says it will, but the data is not yet encrypted in a way that excludes us, and we're not going to describe it as though it is.
2. A wider passive signal set
Posture and viewing distance, screen time, and corrective-lens context. The four signals we
read today are the ones available without asking for anything. The natural next tier is
richer and more expensive: how close the device is held and at what neck angle, how long
the screen has actually been in front of someone (the app_foreground signal we
deliberately dropped rather than fake, because it needs special access), and whether the
person is wearing glasses or contacts — the last being a genuine confounder, since new
correction resets a font-scale trend and a system that doesn't know that will read the
reset as improvement.
Two commitments carry over unchanged. Capture stays deterministic and the model stays downstream of it — posture would be measured, not estimated by a language model, for the same reason Node 3 is SciPy: a signal a model can influence is a signal that can be argued into saying something. And every one of these costs a permission prompt, which is precisely the friction the current design exists to avoid, so they have to be opt-in and the report has to degrade gracefully without them. Our engine already treats a missing signal as missing rather than as zero, which is the behaviour this depends on.
3. Measurements we haven't earned the right to claim yet
A genuine adversarial red-team pass. Our safety score today reflects ordinary reports behaving correctly, not reports built specifically to break the refusal boundary. That's the next measurement, not a claim we're making yet.
Calibrated, not just relative, self-checks. The deliberate self-test currently reports a relative shift; a credit-card-width reference held to the screen plus a front-camera distance estimate would make an absolute measurement defensible.
4. Reach, once the foundation supports it
Real household-scale sync. The motivating case was always a shared device — the way a smoke detector protects a household without being "for" any one person. Today Sensorium is one profile per device, and household sync is the feature that most obviously must wait for the encryption work above rather than precede it.
iOS, deliberately deferred rather than skipped. Android's APIs for this exact signal set are meaningfully more permissive without extra permission prompts. iOS is a real next step once that trade-off is worth solving.
What doesn't change is the rule the whole system is built around: the measurement happens somewhere no prompt can reach, and the model only ever describes what the measurement found. Every item above adds signal, reach or reassurance — none of them get to move a verdict. A version of Sensorium that grew more capable and slightly less careful would be worse than the one we have now, and the tests are there to make sure we'd find out.
Log in or sign up for Devpost to join the conversation.