Auto Daybook

Inspiration

We wanted an assistant that sits beside you all day and takes notes as you work. It notices that you've been debugging the same test for an hour, that a meeting just ended, that you drifted to YouTube with an assignment due tonight. You tell it what you want help with, like stretching after meetings, music that fits your work, or staying off YouTube before a deadline. It keeps watching, works out when that moment has come, and acts on it.

To take those notes, it has to see what you see, all day, and your screen has everything on it: messages, email, bank pages, the people in your calls. Most tools that work this way send your screen to a cloud model. We didn't want to make that trade, so Daybook never leaves the machine it's watching.

A year ago that would have meant a small, weak model or using a cloud model, which could potentially compromise your privacy. It doesn't anymore. Qwen3.8-Flash-Next is a 125B vision model, and NVIDIA's NVFP4 build stores most of its weights in 4 bits, which cuts it from about 360 GB to 133 GB with almost no loss: 91.5 vs 92.0 for FP8 on GPQA Diamond. It runs on a single ASUS GX10 next to everything else, reads screenshots in about 3 seconds, and is smart enough to write your skills for you.

What it does

Daybook watches your screen and keeps short text notes on what you're doing. What it does with those notes is up to you.

You make skills by asking for them. Type what you want in chat:

"Remind me to stand up every time I finish a meeting."
"If I'm on YouTube for 10 minutes while something's due today, nudge me back to work."
"When I open Reddit, warn me and close the tab."

Daybook writes the skill and shows you what it will do: the question it will ask about your screen, when it will act, and what it's allowed to do. Nothing runs until you approve it. You can turn skills on and off, or edit the file directly. If you edit it, it needs approval again.

A skill can react when something changes (you leave a meeting), when something lasts (50 minutes of focused work), or on a timer. It can show a full-screen reminder or a small speech bubble, play music, send a notification, switch apps, or close a browser tab after warning you. Daybook also keeps an Upcoming list of deadlines it has seen on your screen, and skills can use it. If you ask for something Daybook can't do, chat tells you instead of saving a skill that won't work.

Daybook comes with a few skills to start from. Music switches between focus and calm tracks as you move between code and reading. Break reminder covers the screen after 50 minutes of focused work. Instagram nudge pops up a bubble and closes the tab 5 seconds later.

The notes also make a timeline of your day, like "Fixing the journal test · coding · 25 min". You can ask about it: "what did I do in the last hour?" The answer links back to the timeline, and the durations come from the database, not the model.

Nothing leaves the GX10. Blocked apps like password managers never reach the model, the tray icon pauses capture, and Daybook keeps text notes, not images, which you can edit or delete.

How we built it

A skill is a short text file. Here's the built-in music skill:

id: music
name: Mood music
signals:
  mood:
    ask: Which kind of music fits what the user is doing right now?
    type: music.mood
rules:
  - when: mood updates
    do:
      - music.play_mood: { mood: $mood }
allow: [music.play_mood]

Skills never call the model. A skill asks typed questions about your screen and sets rules on the answers. Every 90 seconds, one model call answers every skill's questions at once, and plain code decides which rules fire. Adding a skill adds a question to that call, not another call, so you can keep adding skills without slowing anything down. Skills can only use the actions they list, from a fixed set. Skills made in chat can't run code.

When you ask for a skill, the model gets the skill format, the list of actions, the built-in skills as examples, and your recent notes. It returns a plan, code turns the plan into a skill file, and a validator checks every action, argument, and question before you see it.

To see the screen, Daybook checks every 5 seconds whether it's allowed to look: the screen isn't locked, you aren't away, and no blocked window is visible. When the screen changes, the model describes what's on it without guessing what you're doing. That guess happens in the 90-second call, and the timeline labels it as inferred.

Daybook is a Python app (FastAPI, SQLite) with a Qt tray and overlay, and mpv for music. The model is nvidia/Qwen3.8-Flash-Next-NVFP4 on vLLM. With its n-gram table read from SSD, it runs in about 104 GB of the GX10's 128 GB. Drafting a skill takes about 11 seconds, and on our recorded screen sessions it labeled 13 of 13 activities correctly.

Three of us built it in about a day, working through coding agents over SSH, testing on the same GX10. The agents claimed tasks on a shared tracker issue, worked in separate git worktrees, and opened 41 PRs, with Codex reviewing them. One lane built the backend and one built the UI. They all met at a single API contract.

Challenges we ran into

Turning a sentence into a safe skill. The model drafts a plan, code turns it into a skill file, and a validator checks every action, argument, and question. The review card then has to explain the skill in plain words so you know what you're approving.
Fitting the model. Even at 4 bits, the checkpoint is bigger than the machine's memory. Keeping the n-gram table on SSD made it fit.
One screen, many agents. Everyone developed on the machine Daybook watches. A lock file lets only one instance capture the real screen, and the rest run on recorded sessions.
Privacy edge cases. Blocking a password manager meant working out which part of every window is visible, checking before and after each grab, and discarding results that arrive after a pause or delete.
Time. Durations and deadlines are computed in code, never by the model. Even so, midnight, daylight saving, and gaps in capture each caused bugs.

Accomplishments that we're proud of

You can describe a habit in one sentence and get a working skill, without writing code.
Ten skills cost the same model load as one, and the same answers always trigger the same rules.
A 125B model runs locally on the same box as the screen it's watching.
795 automated tests, most of which run without a screen or a model.

What we learned

Keep the model out of decisions. Let it answer questions about the screen, and let code decide what happens. That's what made user-made skills safe and predictable.
Let code do the arithmetic. Durations, deadlines, and time windows all come from code.
Agree on the contract first. A mock server kept the UI and backend lanes from blocking each other.
Make swaps cheap, and test them on real data. We moved from qwen3.6:35b on Ollama to Qwen3.8 with a config change, then checked it against recorded screen sessions.