codex-work-buddy

• 

• 

• 

• 

• 

• 

• 

Inspiration

Most AI assistants wait for a prompt. But real work does not arrive as a clean prompt—it arrives as hundreds of messages, documents, meetings, deadlines, follow-ups, and half-finished ideas.

Even after agents became capable of researching, coding, and analyzing, people still had to decide what mattered, gather the context, assign the work, and track the result.

That inspired Codex Work Buddy, a local-first Personal Work OS built around a new Agent Loop:

Harness makes an agent capable of doing work. The Loop helps it decide when and what to work on.

What it does

Codex Work Buddy continuously turns fragmented work signals into an evidence-backed workflow:

1. Collect authorized, read-only work signals.
2. Preserve the original evidence locally.
3. Deduplicate, merge, and classify events.
4. Explain what happened and why it matters.
5. Create or update tasks in a trusted Task Ledger.
6. Dispatch eligible work to Codex Workers.
7. Collect results, execution logs, and evidence.
8. Learn from user confirmation without silently changing permanent rules.

Instead of another AI chat window, Work Buddy behaves like a small digital organization:

• Signal Producers discover possible work.
• Task Ledger keeps events, decisions, actions, and feedback traceable.
• Policy Layer controls priority, permissions, and execution boundaries.
• Codex Workers research, analyze, code, and produce deliverables.
• Buddy Meetings let users review work and teach boundaries through text or voice.
• Semi-automatic Optimization converts feedback into auditable candidate rules.

A single correction does not immediately become a permanent policy. Rules can remain a one-time adjustment or be scoped to a person or project, and only repeated or explicit confirmation promotes them into stable global behavior.

Work modes

The three work modes are execution-permission models:

• Dream pauses external collection and autonomous work while organizing local knowledge, rules, and task quality.
• Learn continues collecting and analyzing signals, but asks questions instead of automatically assigning Workers.
• Auto Work allows eligible tasks to be executed automatically within confirmed safety boundaries.

Person importance is only one input. Business impact, urgency, deadlines, dependencies, historical evidence, and confidence remain part of the final decision.

How we built it

The system is designed as a provider-neutral, local-first loop:

Lark messages and documents
          ↓
Canonical local evidence
          ↓
Signal Producer
          ↓
Task Ledger
          ↓
Policy and Dispatcher
          ↓
Codex Workers
          ↓
Result, evidence, and user feedback
          ↺

The main infrastructure includes:

• lark-cli for authorized, read-only Lark data access.
• Local SQLite as the authoritative source for evidence, tasks, policies, meetings, and Worker feedback.
• OpenViking as an optional memory provider for allowlisted document, code, and meeting summaries.
• Volcengine TTS for reading completed Buddy Meeting responses.
• Volcengine streaming ASR for live voice conversations.
• Codex Workers for research, coding, analysis, review, and artifact generation.
• A real interactive dashboard for signal buckets, task details, meetings, work modes, person priority, policy calibration, quick notes, and task archives.

Raw chats, relationship signals, private meetings, rules, and Worker feedback remain local. Work Buddy does not copy Lark credentials into its database or browser, and it does not silently write back to Lark.

Challenges we faced

The hardest problem was not connecting another model. It was making the Loop trustworthy.

First, every generated task needed an evidence chain:

Original event
→ Summary
→ Task judgment
→ Priority reason
→ Suggested action
→ Worker result
→ User feedback

Second, repeated polling had to remain idempotent. The system needed to distinguish a new event from an update, merge related signals, and avoid producing duplicate tasks.

Third, learning had to avoid overfitting. A single user correction may be an exception, so feedback becomes a reviewable candidate policy before it can affect future autonomous execution.

Finally, voice interaction required strict half-duplex behavior: ASR pauses while Buddy speaks, preventing its own TTS output from being recognized as another user instruction.

What we learned

The most important lesson was that agent capability is only one layer of an autonomous system.

A useful long-running agent also needs:

• A trusted source of truth.
• Explicit execution permissions.
• Evidence-backed prioritization.
• Reliable task and result state.
• Low-cost human calibration.
• Clear boundaries between local private context and exportable memory.

Codex Work Buddy does not directly train model parameters online. Instead, it continuously calibrates a personal work policy: who and what matters, which tasks may run automatically, when the user must be asked, and how work should be routed.

Accomplishments

We built a working product rather than a static concept:

• A real local dashboard with rich, differentiated work signals.
• Evidence-backed task creation and event bucketing.
• Worker assignment, execution state, and result review.
• Dream, Learn, and Auto Work permission modes.
• Person-priority calibration without overriding event severity.
• Buddy Meetings with TTS and live ASR.
• Semi-automatic policy optimization with rule scopes.
• Quick notes, task archiving, recovery, and complete local traceability.
• A safe mock mode for demonstrations using realistic but fully redacted data.

What’s next

Next, we want to strengthen the evaluation dataset, expand message-source adapters, improve first-time installation, and add more specialized Workers and Reviewers.

The long-term goal is simple:

You should not spend your entire day manually processing every message and task. You should spend your attention teaching, reviewing, and improving a digital team that increasingly understands how you work.

Built With

• asr
• lark-cli
• memory
• tts