1. Inspiration
Every meeting ends with the same quiet dread. Someone says "I'll take that." Someone else nods. Three days later, nobody can say who owned what, or by when.
Meeting tools give you a summary, which is a memory aid and does not create accountability. We built the missing layer: the moment a person says "I'll own this by Friday," it becomes a tracked, owned, dated commitment, with nobody retyping it into a tracker.
Work software is usually a beige spreadsheet, so we made a supernatural case board. Open promises are dark case files pinned to a wall, and the unresolved ones are the things lurking in the Upside Down.
2. What it does
| Stage | What happens |
|---|---|
| Hear it | A locally run Chrome extension listens (walkie-talkie overlay on Teams/Zoom, plus email) only after the user opts in with Open Channel. |
| Understand it | Extraction fires only on explicit ownership language. Hedges ("we should", "maybe") are ignored. Every item is labelled by evidence quality. |
| Review it | New items land in an untriaged queue. Nothing reaches a destination tab until a human swipes it. |
| Act on it | Right-swipe and the AI plan runs: send a follow-up email, create a calendar reminder, or email a slide outline. Anything uncertain goes to human review. |
| Chase less | A manager digest sorts all commitments into four buckets and explains risk in plain words. Risk concerns work, never people. |
3. System architecture
The extension runs on the user's machine and is currently run locally, but the dashboard is live and deployed. The extension talks to the live web app over HTTPS. The server, SQLite store and AI/Google integrations live behind that single API.
| Layer | Runs | Role |
|---|---|---|
| Chrome extension | Your machine only | Listens after Open Channel, sends text to the live API |
| Board UI | Live website | Review queue, swipe, digest |
| Node API | Live website | Validation, routing, auth, origin checks |
| Rules extractor | Live website | Free, deterministic extraction |
| Gemini agents | Gemini API | Task Noter and Risk & Clarity, merged on top of rules |
| AI actions | Live website | Runs email, reminder or slide outline |
| SQLite store | Live website | Tasks, notes, edits, board state |
| Gmail, Calendar, ElevenLabs | External | Sending, scheduling, voice isolation |
Boundary notes
- The API only accepts writes from same-origin requests (the live host is allow-listed) and validates content type, size and IDs.
- Secrets (
GEMINI_API_KEY,ELEVENLABS_API_KEY,GOOGLE_*) are held server-side only. - The extension needs the live site's address in its host permissions to reach it (see section 13).
4. How a promise becomes a commitment
Rules run first, always, at zero cost. Gemini adds nuance on top and is merged in, never substituted. If Gemini is down, the product still works.
- Text is captured only after Open Channel.
- No explicit ownership, or a hedge ("we should", "maybe"): ignored, no task.
- Explicit ownership plus an exact date: Verified.
- Ownership with an approximate date: Review Needed.
- No date, or a thin deliverable: Clarification Required.
- Gemini's softer finds are merged and deduplicated, then everything waits in the untriaged queue for a human swipe.
5. The trust model
A commitment is the primary entity. Three facts about it are kept separate on purpose:
| Fact | Question it answers | Who can set it |
|---|---|---|
| Verification | Is the evidence good? | System (Verified / Review Needed / Clarification Required) |
| Acceptance | Did the owner agree? | Owner (local acknowledgement timestamp) |
| Completion | Is there a sign-off? | Human, with linked evidence |
A delivered file or an AI claim alone never marks work done, and contradictory sources stay visible until an authorised person decides. The UI never shows AI confidence percentages.
| From | Action | To |
|---|---|---|
| Untriaged | Swipe left | Pending review |
| Untriaged | Swipe down | Pending approval |
| Untriaged | Swipe right | AI in progress |
| Untriaged | Swipe up | Completed |
| AI in progress | Confident and succeeded | Completed |
| AI in progress | Unsure, failed or interrupted | Pending review |
| Pending review / approval | Human sign-off | Completed |
6. Swipe routing
Swiping happens only inside task details. Inbox cards stay still. Keyboard users get Shift + Arrow keys, and trackpad momentum is consumed once per gesture.
| Gesture | Destination | Meaning |
|---|---|---|
| ⬆ Up | Completed | Whole task done (approval-gated steps ask first) |
| ⬇ Down | Pending approval | Needs a sign-off |
| ⬅ Left | Pending review | Needs a human look |
| ➡ Right | AI actions in progress | Plan previews during the drag; a full swipe starts it |
7. AI that knows when to stop
The hard problem was never making the AI do more. It was making it know when not to. Every path fails safe.
- Google not connected: draft only, sent to Pending review.
- One Gemini call classifies the task as email, reminder, slides or other. "Other" with high confidence: draft only.
- Valid plan with high confidence: run it. Success means Completed; any error falls back to a draft.
- Otherwise a keyword fallback picks a template action sent only to the review address, and the task lands in Pending review.
Safeguards
- A recipient is used only if it appears verbatim in the task text. A hallucinated address is never used.
- Side effects (send, calendar) are always the last step.
- Fallback actions go only to the designated review address.
- Invalid JSON, low confidence, or a failed send means nothing happens and the task moves to human review.
- A server restart moves interrupted AI work to review with a retry message.
8. Data model and concurrency
| Table | Holds | Key fields |
|---|---|---|
live_tasks |
The commitment | id (CI-n), task, owner, due, verification, quote |
task_actions |
Board state | state, action_revision, workflow, ai_plan, ai_output |
notes |
Working note | body, revision |
commitment_details |
Owner/date/risk edits | owner, due_date, risk_level, revision |
acknowledgements |
Owner acceptance | accepted_at |
clarification_messages |
Simulated send history | channel, body, recorded_at |
users / session |
Sign-in | email, salted scrypt hash |
Concurrency you can trust. Notes, owner/date edits and swipe actions all carry an integer revision and use compare-and-set updates:
$$ \text{UPDATE}\ \dots\ \text{SET}\ \text{revision} = r + 1\ \ \text{WHERE}\ \ \text{revision} = r_{\text{client}} $$
If no row changes, the client was stale and receives HTTP 409 instead of silently overwriting someone's work. All SQL is parameterised, and passwords use per-user salted scrypt with constant-time comparison.
9. Measurement plan
No pilot observations have been collected yet, and targets below are illustrative. Fixture counts are never used as effectiveness metrics, and a zero denominator renders Not measured.
$$ \text{Extraction precision}=\frac{\text{correct extracted commitments}}{\text{all extracted commitments}}\qquad(\text{target}\ \ge 90\%) $$
$$ \text{Conflict precision}=\frac{\text{genuine flags}}{\text{all flags}}\qquad \text{Conflict recall}=\frac{\text{detected labelled conflicts}}{\text{all labelled conflicts}}\qquad(\text{targets}\ \ge 90\%) $$
$$ \text{Ownership completeness}=\frac{\text{accepted commitments with a named owner}}{\text{accepted commitments}} $$
$$\Delta T = \mathrm{median}(\mathrm{decision\ time}-\mathrm{detection\ time;\ pilot}) \le 0.75 \times \mathrm{median}(\mathrm{decision\ time}-\mathrm{detection\ time;\ baseline})$$
$$ \text{Notification burden}=\frac{\text{notifications}}{\text{commitments}}\ \ \text{(down, with missed follow-ups not up)} $$
| Metric | Target vs baseline |
|---|---|
| Extraction precision | 90% |
| Conflict precision / recall | 90% each |
| Ownership completeness | Improve |
| Resolution time | Reduce 25% |
| Status-chasing time | Reduce 20% |
| Notification burden | Reduce without more missed follow-ups |
Evaluation requires independently labelled source samples, adjudicated disagreements, held-out test data, false-negative review, comparable baseline and pilot windows, and reported sample sizes with medians.
10. Design system
The Upside Down Board is original: no franchise artwork, wordmarks, screenshots or likenesses. Fonts are Creepster (display headings only) and Special Elite (body), with system fallbacks.
| Role | Hex | Use |
|---|---|---|
| Canvas | #0A0A0A |
App background |
| File surface | #151211 |
Panels, dialogs, notes |
| Primary signal | #CC2200 |
Main action, borders |
| Ink | #F4EAE6 |
Primary text |
| Supporting ink | #C2ADAA |
Metadata, helper copy |
| Low risk / verified | #79C58B |
"Safe in Hawkins" |
| Moderate / review | #F0C35A |
"Something's Wrong" |
| High / clarification | #FF4A32 |
"The Demogorgon is Here" |
Risk is always spelled out in words as well as color. Nothing flickers or shakes, and reduced-motion preferences are respected.
11. Challenges we ran into
- Making a swipe feel physical. The AI plan previews during the drag, so it is inserted beside the moving card without re-rendering it and breaking pointer capture. Trackpad momentum fired several swipes until we limited each gesture to one card.
- Teaching AI restraint. See section 7.
- Meetings don't announce when they end. We combined stop-listening, recognised Leave/End buttons, tab close, and an explicit End meeting · review tasks control. The transcript stays saved locally if processing fails.
- A local extension talking to a live site. The extension stays on the user's machine, so the live host must be allow-listed on both sides: extension host permissions and the server's origin check.
- Theme without theft. Mood without copying anyone's IP: artwork, avatar, icon and wording are all original.
12. Accomplishments and lessons
Accomplishments
- A core principle of "don't claim what you can't prove."
- An end-to-end loop: listen → extract → review → swipe → AI action → email/calendar.
- Real HTTP and SQLite tests covering restart persistence, revision conflicts, origin checks and the meeting-to-untriaged-to-destination flow (20 passing at the last documented run; re-run
npm testafter the latest fixes). - Accessibility that survives the horror theme: words alongside color, a keyboard path for every swipe, and no flicker.
- A plain measurement plan instead of invented statistics.
What we learned
- Trust is a product feature. A tool that says "I'm not sure, please review" earns more confidence than one that is always certain.
- Hybrid beats pure LLM: free, explainable rules as the baseline, with the model adding nuance.
- The gesture matters as much as the model.
- Demos and production are different animals. Tenant auth, retention and audit logging come before any real client data.
13. Deployment reality
| Component | Where it runs | Status |
|---|---|---|
| Web app + API + SQLite | Live website | Deployed |
| Chrome extension | User's own Chrome, loaded unpacked | Local only, not deployed |
| Extension → app traffic | HTTPS to the live site | Requires the live host in extension host permissions |
Known gaps to close before real client data
- Storage on typical free hosting can be ephemeral, so SQLite needs a persistent disk.
- The app has a single shared local session and no multi-tenant authorisation.
- The origin check allows any
onrender.comhost; tighten it to the exact production hostname. - The extension must be reloaded in
chrome://extensionsafter every update.
14. What's next
- Real connectors. Permission-scoped Jira, Slack, Teams and Planner adapters, plus email and docs as evidence sources.
- A conflict service. When Jira says Friday and Teams says Monday, show both claims and let an authorised person decide. Never resolve with a weighted guess.
- A real pilot. Labelled transcripts, adjudicated disagreements, and honest numbers for the metrics in section 9.
- Production hardening. Multi-tenant auth, server-held secrets, retention and deletion controls, append-only audit history.
- Fewer pings, not more. Measure notifications per commitment against missed follow-ups.
Built With
- accessibility
- ai-agents
- chrome
- css3
- elevenlabs
- gemini-api
- gmail-api
- google-calendar-api
- google-gemini
- html5
- human-in-the-loop
- javascript
- microsoft-teams
- natural-language-processing
- node.js
- oauth2
- prompt-engineering
- rest-api
- speaker-diarization
- speech-to-text
- sqlite
- swipe-gestures
- web-speech-api
- zoom