Earshot

Your entire Mac, run by voice. Earshot opens apps, reads your screen, types and runs commands through the glasses on your face — no keyboard, no mouse, no hands. For some people, that's everything.

Comment

Inspiration

Agents crossed a line this year. They stopped suggesting and started acting: editing files, sending mail, running commands, operating real machines.

But the interface never moved. It is still a keyboard, a screen, and you sitting in front of both. So the moment you stand up and walk away, you go blind. You cannot see what it is doing. You cannot answer it when it asks you something. And you cannot stop it when it is about to do something wrong.

Every escape hatch we have today makes that worse, not better. SSH from your phone, remote desktop, a web console: all of them ask you to unlock a device and type on glass. They assume the problem is distance. The problem is hands.

That reframing is the whole project. Once you see it as a hands problem instead of a distance problem, you notice who has been living with it the entire time. For someone with ALS, a spinal cord injury, severe arthritis, or any condition that takes fine motor control, a computer is not inconvenient without a keyboard. It is unusable. Voice assistants did not solve it either, because Siri and Alexa call APIs. They can set a timer. They cannot open your IDE, read what is on the screen, and type a reply into it.

We had smart glasses sitting on the table with a microphone and a speaker already pointed at a human face. The gap was not hardware. It was that nothing connected the glasses to a machine that could act, and nothing made that machine safe enough to leave alone.

What it does

Earshot turns any Mac into something you operate entirely by voice, through the Ray-Ban Metas already on your face.

You speak. The agent operates your actual computer, opening apps, reading what is on the screen, typing, running shell commands, sending mail, and it answers in your ear. 577 ms median, glasses to Mac and back.

The path a sentence takes

Ray-Ban Meta --Bluetooth--> iPhone --WebSocket/LAN--> Mac host
                                                        |
                       OpenAI Realtime <-----------------+   speech in, speech out
                                                        |
                       cua computer-use <----------------+   drives the real machine
                                                        |
                       Jac memory graph <----------------+   remembers, reasons, explains

The phone is deliberately a dumb pipe. It captures 16 kHz PCM and forwards it. Every decision happens on the Mac. That matters because the phone can drop off wifi mid sentence and nothing about the agent's state is lost.

Two ways to talk to it

Mode For
Push-to-talk Loud rooms. Pressing the button is the act of addressing it.
Hands-free Continuous listening, gated on the wake word "Harness", for people who cannot press a button at all.

Twelve tools

The model picks; the safety gate decides whether it actually runs.

Tool What it does
open_app Launch and focus a macOS app
type_text Type into the focused window
press_hotkey Key combos such as ⌘N and ⌘⇧D
run_shell Run a zsh command; destructive ones hit the confirmation gate
confirm_pending_action Resolve a pending confirmation, human liveness checked
capture_to_workspace Photograph what you are looking at and save it to the Mac as a real file
run_workflow Run a saved multi-step routine, including pause-and-ask
undo_last_action "undo that": closes the app it opened, deletes what it typed, ⌘Z a hotkey
draft_reply Write a reply as you, from the learned model of you
explain_yourself "why did you do that?", answered by walking the provenance graph
analyze_graph "what are my patterns around deploys?", computed by traversal
get_status Summarise recent actions

After every action it narrates what it just did in a few words: "Opened Mail", "Typed the reply", "Undone". Obvious once you build for someone who cannot see the screen. They still need to know the state of their machine.

morning, a workflow rather than a voice macro

"Harness, do my morning tasks."

It opens Mail. Screenshots it. A VLM extracts what is actually on screen. It ranks those messages against the memory graph, who matters to you, in your order. It speaks a summary. Then it stops and asks what the reply should say. It waits. You answer out loud. It drafts, routes the send through the confirmation gate, waits for a real spoken "yes", sends, and reads your calendar back.

Multi-step, stateful, resumable, and it refuses to do the irreversible part without a human. End to end in ~33 seconds, including both human pauses.

It builds a model of you

Four verbs, four walkers, one graph:

Observe      write down what actually happened      (no LLM, observation is fact)
Infer        distil observations into Traits        (by llm)
DraftAsUser  act in your voice, from those Traits   (by llm)
Explain      walk the evidence chain, answer "why"  (traversal, then phrasing)

Every real tool call becomes an Observation node. Every few observations, Infer distils them into Trait nodes: "prefers TextEdit in the morning", "dictates replies casually", "deploys in the afternoon". A trait that cannot cite evidence is discarded at creation. An unevidenced belief cannot exist in this graph.

The effect is visible. Same request, before and after it had learned anything:

Draft Assumptions
Before (empty model) "Hey Sahiel! How about 7? Let me know what you think. 😊" 2 guesses
After (learned) "Yes! Seven works for me, see you there!" 0

The second one is the user's own phrasing, learned from a reply he had dictated earlier that morning.

It can tell you why

Ask "why did you draft it that way?" and it walks the actual graph:

Decision --Because--> Trait --EvidencedBy--> Observation --Involves--> Person

and answers out loud, from the nodes it walked:

"I drafted a reply to Sahiel, 'Sure! Seven works for me, see you there!', because you dictate replies casually. You dictated: 'Reply saying yes, seven works, see you there!'"

That is not retrieval and it is not a generated excuse. Every other agent memory is a vector store, and a vector store can tell you what is similar. It structurally cannot tell you why.

It can analyse itself

analyze_graph runs four walkers that compute the answer, then uses a model only to phrase it: HubScan (degree centrality), ClusterScan (topics, peak hours, approvals vs denials), TemporalScan (time-of-day habits), and ConfidenceScan (what it is sure of vs still guessing).

Q: "What are my patterns around deploys?" "54 observations related to deploys, peaking at 16:00, with 17 approvals and 7 denials. Notably, you deploy in the afternoon, supported by 8 observations."

Every number there came out of a traversal. We verified this by re-counting the same graph independently in Python and asserting exact equality.

You can look at it

A built-in Obsidian-style force-graph viewer at localhost:8799: ~450 nodes coloured by type, hubs sized by live degree and labelled on canvas, clusters per topic. Click a trait, see its evidence.

How we built it

The seam that made everything else possible

The very first thing we built was a transport abstraction: the core pipeline consumes a pipecat BaseTransport and never learns which one it got. create_transport("local") is the Mac's own mic and speakers. create_transport("websocket") is the phone over the LAN.

That one decision paid for itself repeatedly. We developed against the laptop mic with zero hardware in the loop, tested with a phone simulator that streams a wav file over the real socket, and swapped in the actual glasses at the end without touching pipeline code.

The voice loop

pipecat orchestrates it. OpenAI Realtime does STT, the LLM, and TTS with native audio, server-side VAD for turn-taking, and barge-in. Custom processors in the pipeline handle the parts Realtime does not:

  • InputResampler: the phone sends 16 kHz, Realtime wants 24 kHz.
  • TurnLatencyMeter: measures user-stopped-speaking to bot-audio-start, which is where the 577 ms number comes from. It is also load-bearing for safety (below).
  • WakeTap: sits between the user aggregator and the LLM, because the aggregator consumes TranscriptionFrames and anything downstream never sees them.

Controlling the actual machine

cua (trycua) via a local computer-server for typing, hotkeys, clicks, and screenshots. osascript for app lifecycle and frontmost-window checks. A plain zsh subprocess for shell.

Screen reading is a VLM step: screenshot, downscale, ask a vision model what is on screen. That is how morning knows what is in your inbox without an integration or an API key for your mail provider. It reads the screen like a person would.

The memory: Jac, object-spatially

This is the part we would defend hardest.

Two Jac graph modules, jac/memgraph.jac (people, email, calendar, preferences, actions, workflow plans) and jac/usermodel.jac (the provenance graph), sharing one persistent context.

Metric Count
Node types 14 declarations (13 unique names)
Typed edge types 12
Walkers 12
Node/edge abilities (can) 54
by llm() sites 5

What makes this Jac rather than "a graph library in Jac":

The nodes carry the behaviour. A Trait node has abilities that answer whichever walker is visiting it. It feeds the Infer walker, justifys to Explain, and grades itself for ConfidenceScan. The logic lives on the topology, not in a service layer above it.

walker Explain {
    can trace with Decision entry {
        for t in [here ->:Because:->] {
            visit [t ->:EvidencedBy:->];     # trait -> the evidence behind it
        }
    }
}

Reasoning is traversal. analyze_graph and explain_yourself are walks. The numbers are counted by moving over edges, and a model only phrases them afterward. That is exactly why the answers can cite evidence, and why we can verify them against an independent recount.

Persistence is free. Nodes attached to root survive process restarts through jaclang's built-in SQLite store. No ORM, no schema, no migrations. We proved it the boring way: write in one process, read it back in a second, mutate in a third.

The LLM calls are typed, not prompted. All five by llm() sites are typed MTLLM functions, for example distill_traits(list[str]) -> list[TraitDraft]: signature in, structured object out. There is not one hand-rolled prompt string in the memory layer.

And then we ported the entire host to Jac

The memory was Jac; the host was Python. So we converted all 26 modules: the realtime audio pipeline, the transport seam, the phone wire format, computer control, the tool schemas, the safety gate, the wake gate, the workflow engine, the graph viewer, preflight, and every test.

host/ is now 29 .jac files and zero .py. Because Jac compiles to Python bytecode, pipecat, cua and websockets are imported straight from .jac with no wrapper, and every documented run command still works unchanged. Installing jaclang registers an import hook, so python -m earshot.voice resolves the .jac module.

We did not take that on faith. Each module was diffed against its Python original by execution: the serializer across all seven frame paths, the capture sidecar byte for byte, the tool schemas by hash, preflight's 18 checks character for character including ANSI codes, and the memory bridge by round-tripping data through two separate processes.

The iOS client

SwiftUI, deliberately thin: capture mic audio, stream PCM up, play PCM down, plus a hands-free toggle that tells the host {"type":"mode","hands_free":true}. No agent logic on the phone at all.

Challenges we ran into

The agent approved its own rm -rf

The worst bug we found, and the one we are most glad we found.

We built a confirmation gate: destructive actions must be confirmed by the user before running. During testing, the model called confirm_pending_action(user_reply="yes") by itself, immediately, with no human having said anything, and authorised an rm -rf that nobody approved.

The gate was structurally broken in a way that is not obvious until you watch it happen: the confirmer and the thing being confirmed were the same actor. Asking a model to wait for permission is not a mechanism. It is a request.

The fix is a human-liveness check. The voice pipeline timestamps every real user utterance. A confirmation is only honoured if a human demonstrably spoke or typed after the gate prompt was issued:

if self._last_user_utterance_at <= pending.created_at {
    self.audit_log.append(("deny_no_user_reply", pending.intent));
    return (False, "DENIED: no reply from the user has been heard...");
}

The pending intent is deliberately kept on that denial, so a real reply a moment later still resolves it. It is pinned by regression tests, and we mutation-tested those tests. Disabling the check makes them fail, which is the only way to know they were ever load-bearing.

Ambient speech, and a race we only found by running it

Hands-free means every conversation in the room reaches the agent. A prompt saying "only act when addressed as Harness" is layer one, and layer one is not a guarantee. We had just watched a model ignore a much firmer instruction.

So layer two: the dispatcher refuses to execute any tool unless the wake phrase actually appeared in a transcript of user audio, including the mis-transcriptions Whisper produces (harnas, harniss, hardness), or you are already mid-dialog.

Then it broke in a way inspection would never have caught. The model issues its tool call before the async transcript lands, so "Harness, open Mail" got refused. The wake phrase existed but had not arrived yet. The fix is a short grace window: on refusal, re-check for up to 2 seconds. In testing the transcript landed ~60 ms later and the retry went through.

jaclang 0.16.7 had sharp edges

Roughly half an hour lost to four undocumented behaviours, all found by running things:

  • The documented filter syntax with a ? prefix does not parse. Use a comprehension with isinstance.
  • Entry on backtick-root fails; it is with Root entry.
  • A store written under module __main__ cannot be read back under a real module name, because the class registry refuses it. Every context has to bind full_target_path="memgraph".
  • Contexts swapped mid-process must be explicitly committed or writes vanish.

The Python to Jac port surfaced more: pass does not exist (use an empty block), docstrings go before a declaration, global is required to rebind a module variable, *args must be annotated, and Any does not flow into a typed destination the way Python's does.

One we checked rather than assumed: Python's field(default_factory=time.monotonic) became has created_at: float = time.monotonic(). Had Jac evaluated that once at class-definition time instead of per instance, every confirmation timeout would have silently broken. We constructed two instances 0.25 s apart and compared.

The VLM reads whatever is frontmost

Screen reading is the most demo-fragile thing we built: 10 to 16 s per screenshot, and it reads whatever is actually on screen. It once read a terminal and a browser instead of the demo inbox. In another run the demo windows slipped behind a real Inbox and the agent triaged, and replied to, a genuine Amazon email.

Two mitigations. The workflow now verifies Mail is frontmost before and after ⌘R and aborts aloud rather than typing into the wrong window. And the demo checklist is explicit about Do Not Disturb, window count, and verifying the screen once before going live.

Our tests were writing fake memories

The pytest suite's fake dispatcher wrote 2 observations into the real user model. Fabricated memory is worse than no memory, because it silently corrupts every trait inferred afterward. _observe is now a no-op under PYTEST_CURRENT_TEST or EARSHOT_NO_OBSERVE, and the two contaminated nodes were deleted.

When we later ported the tests to Jac, the same guard had to be re-established: jac test does not set PYTEST_CURRENT_TEST, so the Jac suites set EARSHOT_NO_OBSERVE explicitly. Same rule, different runner.

Checking our own analytics caught two real bugs

We wrote an independent Python recount over a graph dump that exits non-zero on any mismatch with the walkers. It immediately caught two genuine bugs: the graph dump emitted duplicate edges for nodes reachable by more than one path, and Topic hub degree conflated message-About with observation-About.

Both would have shipped as numbers said confidently on stage.

Auth, in the least obvious way

Adding OPENAI_API_KEY to .env broke text mode and the VLM. The codex responses backend accepts only a ChatGPT OAuth token and 401s on an API key, so the moment a key existed, the shared token getter started returning the wrong credential. Split into get_openai_token() and get_oauth_token(), which never falls back.

Still open

Camera capture does not work. The DAT session fails with noEligibleDevice pending Meta Developer Center onboarding, so the code is written and currently unreachable. And the host transport takes one client at a time.

Accomplishments that we're proud of

577 ms median voice turn, measured on real hardware, glasses to Mac and back. Conversational, not "wait for the spinner."

An agent memory that can be cross-examined. Ask why, and it walks Decision -> Because -> Trait -> EvidencedBy -> Observation and reads you its own evidence chain. We believe this is the actual contribution: not that the agent remembers, but that its reasoning is inspectable, and wrong beliefs are traceable to the observation that caused them.

We found a real agent-security vulnerability and fixed it structurally. Not a jailbreak or a prompt-injection demo, but a design flaw where the confirmer and the confirmed were the same actor. The fix is mechanical rather than instructional, and mutation-tested.

The whole host is Jac. 26 modules, 29 .jac files, zero Python left, with every module verified against its Python original by execution rather than by "it compiles."

Analytics that survive an audit. Every number analyze_graph speaks was independently recounted and matched exactly.

Honest data provenance. The demo graph is ~450 nodes, and most of it is seeded, but seeded through the same Observe walker a live interaction uses, with zero direct graph inserts and zero hand-authored traits (verified by grep). Every seeded node carries seeded=True, and the viewer badges the split on screen. We would rather volunteer that than have someone find it.

A real email, sent by voice, through a gate that said no first. Fabricated confirmation denied, real spoken "yes" accepted, message in the Sent mailbox.

Corpus inference for about $0.02. 347 observations distilled into 26 evidence-linked traits in 12 by llm() calls on gpt-4o-mini, by batching trait-matching per round instead of per-draft.

18 tests, covering the safety gate, the wake gate, and confidence-gated autonomy: the three places where a silent regression would be dangerous rather than annoying.

What we learned

Instructions are not mechanisms. The single biggest lesson. We asked a model to wait for confirmation and it confirmed itself. We asked it to only act when addressed and had to enforce it in a dispatcher anyway. Every safety property that mattered ended up as a check outside the model, and the ones we had left as prompt text were the ones that failed.

Reasoning as traversal is a different capability from retrieval. We started using the graph as storage. The shift happened when we realised the answer could be a walk: that "what are my patterns around deploys" is counting over edges, and the model's only job is to read the count aloud. That is also what makes it verifiable. You can recount a traversal. You cannot recount a vibe.

Jac's object-spatial model changed how we factored the problem. Putting abilities on nodes meant Observation decides how it reports itself to each walker, instead of five walkers each knowing how to interrogate an observation. Adding ConfidenceScan later touched the walker and the node, not a service layer. And free persistence removed an entire category of work, no ORM, no schema, no migration, which is most of why the memory got built at all in a hackathon.

Verify by executing, not by reading. Nearly every real bug here, the self-confirming model, the wake-transcript race, the duplicate edges, the VLM reading the wrong window, the tests writing fake memories, was found by running the thing and reading the output. None would have been caught by code review.

Honesty is a better strategy than polish. Badging the seeded data, saying out loud that camera capture is broken, keeping the "I stopped: Mail is not frontmost" abort: every one of those makes the demo more credible, not less. The failure modes we volunteer are the ones that stop being weapons.

Build the seam first. The transport abstraction on day one meant we could develop with no glasses, test with a simulated phone, and swap in real hardware at the end without touching the pipeline.

What's next for Earshot

Give it eyes

The camera is written and blocked on Meta Developer Center onboarding, and it is the biggest unlock we have. capture_to_workspace already treats a photograph as an input to an action, not a description: you look at something, it lands on your Mac as a real file with a sidecar note about what you wanted done with it.

"Harness, take a picture of this whiteboard and turn it into a ticket." "What's this error on my other monitor?" "Read me this letter."

Combined with the memory graph, images become nodes with provenance like everything else: "you photographed this schematic during the incident on the 14th".

Contextualise everything

Right now it acts when addressed. The next version should understand the situation before it is asked. The graph already knows your hours, your people, your topics, and your approval patterns. It should be able to notice you are in the deploy window you always approve in, that the person messaging is one you reply to within the hour, that this is the fourth time this week you have hit the same error, and lead with that.

The honest constraint: proactivity needs the same gating as action. An agent that speaks unprompted is one that can interrupt a conversation, so it goes through the confidence tiers and the wake gate, not around them.

Wherever hands are busy or unavailable

  • Accessibility, the reason this exists. Full computer control for people with ALS, spinal cord injuries, MS, severe arthritis. The next step is working with actual users rather than designing at them. Nothing here has been tested with the people it is aimed at, and that is the gap that matters most.
  • Field and trade work: electricians, mechanics, technicians. Look at the panel, ask what the code means, have the manual page opened and read back.
  • Clinical and lab settings: gloved, sterile, hands committed. Dictate to the record without touching anything.
  • On-call engineering, the original itch. Something pages at 2am, you ask from bed what is failing and approve the rollback out loud, through a gate that will not let the model approve itself.
  • Driving, cooking, carrying, holding a child: the ordinary hands-full moments where a computer is currently just unavailable.
  • Ageing users: no small targets, no menus, no "click the third icon."

Where the system goes

  • Multi-client: one host, several devices. Today the transport takes one.
  • Beyond macOS: the computer-control layer is swappable, and Windows and Linux are the same seam.
  • Local models: a graph of everything you do is exactly the thing that should not leave your machine. On-device inference makes the privacy story match the design.
  • Shared graphs: a team's model of itself, with per-person trust scoping (api_autonomy already computes trust per person).
  • Longer-horizon workflows: morning is three steps. The engine is resumable and already pauses for humans, so the ceiling is much higher.
  • Memory that forgets: traits decay when contradicted. Right now reinforcement only strengthens, and a model of a person should be allowed to be wrong, and to update.

Repo: https://github.com/PranavAchar01/EarShot-JacHacks, the entire host in Jac, with PORT-TO-JAC.md documenting every deviation and how each module was verified.

Built With

  • applescript
  • avdoundation
  • bluetooth
  • cua
  • force-graph
  • gpt-4o-mini
  • ios
  • jac
  • jaclang
  • macos
  • meta-wearables-device-access-toolkit
  • mtllm
  • object-spatial-programming
  • openai-realtime-api
  • pipecat
  • pytest
  • ray-ban-meta
  • sqlite
  • swift
  • swiftui
  • vision-language-model
  • voice-activity-detection
  • websockets
  • whisper
  • zsh
Share this project:

Updates