Axsail Voice Survey

Inspiration

A marine surveyor spends hours on a yacht inspecting it, then even longer at a desk turning handwritten notes and memory into a report. Surveyors and industry bodies (IIMS) consistently name report writing and report quality as the main pain point. In property restoration for instance, voice-to-structured-scope tools already ship in production and are the most wanted features in the industry. We wanted to test whether the same pattern works on a boat, with a wearable that keeps the surveyor's hands free.

What it does

The surveyor presses Start in the Axsail web app, then dictates into Amazon Bee on the Apple Watch while walking the boat.

Live findings: Axsail receives the transcript from Bee's stream as it is spoken. After every utterance, Claude rebuilds all findings from the whole transcript (for now, for the sake of simplifying): system, component, location, observation, reading, recommendation and severity. Corrections such as "scratch that" or "that seacock is on the starboard side" are resolved by context.
Checklist coverage: A panel of 13 systems and 96 checklist items shows what was inspected, what was skipped and why, and what nobody has mentioned yet.
Bee push notifications: When a finding lacks a detail the report needs, such as where the seized seacock is or the extinguisher's service date, Axsail sends the question to the watch as a Bee to-do with an alarm. The surveyor answers by voice, and the question closes on the next extraction.
Review: After Finish, the surveyor confirms, edits or deletes each finding and corrects the coverage. Editing a confirmed finding unconfirms it. Severity stays marked as an AI suggestion until confirmed.
Continue recording: If Finish was pressed by mistake, or the surveyor remembers something later, Continue recording reopens the inspection. Confirmed findings and typed fields stay as they are. When new dictation contradicts a finding the surveyor already confirmed, Axsail shows "New detail dictated" next to it, and the surveyor chooses which version to keep.
Report: On finalisation, a condition survey report is generated in the standard format: general information, scope and limitations, one section per system, findings by severity, sign-off and etc. Unconfirmed findings are listed as a warning. It prints or saves as PDF.

AI output stays a draft until the surveyor confirms it.

How we built it

Scope: Core flow, functional requirements, data model, and a report format taken from real survey reports.
1. Bee API: bee login, then a bee proxy that injects the credential, then the /v1/* REST endpoints, the /v1/stream SSE feed and /v1/todos.
2. Tests: 12 tests on a real Apple Watch: latency, session boundaries, offline capture, background noise, overlapping speakers, English, Russian and Greek speech, the assistant gesture, and the to-do alarm as an outbound channel.
3. Service: A Python FastAPI service in a hexagonal layout: domain, use cases, infrastructure. It holds an always-on subscription to the Bee stream and saves every new-utterance to Postgres. A scheduler runs extraction after each utterance with claude, with at most one run per inspection at a time. Updates reach the React UI over server-sent events.
4. Demos: A mock Bee stream replays a 15-minute scripted survey of a 62 ft motor yacht with Bee's real wire format and timing. A second scenario adds speech after Finish. Unit tests (116 so far) cover the review merge, the question rules and the lifecycle.
5. Real watch runs: The same stack runs against the real watch through the Bee proxy

What we learned

Utterances arrive on the stream $8$–$22$ s after speech. Extraction over the whole transcript takes $5$–$30$ s, growing with its length. Findings appear while the surveyor is still at the spot.
Re-extracting everything on each utterance keeps the logic simple. The LLM handles corrections and late details, with no state merging.
Tapping stop does not close a Bee conversation. Bee closes it about 6 minutes after the last speech, so separate recordings inside that window merge into one conversation.
start and end offsets reset with each recording, so we order the transcript by spoken_at.
The stream is at-most-once with no resume, and the server closed it after 80–107 minutes. If the service is down, utterances are lost.
Bee auto-deleted some conversations, and we couldn't identify the rule.
A recording made while the iPhone was offline was lost. Mobile data alone works.
There is no API for Bee to speak back. A to-do with alarm_at reaches the watch with haptics, and tapping it opens the recorder. That became our clarification channel.
In one real watch run, the speech dictated after the to-do alarm never reached Bee: not the live stream, nor Bee's own transcript. The conversation stayed "in progress" until it was closed by hand in the Bee app. It looks like a Bee bug; we have seen it once.
Overlapping speech corrupts findings. Background noise was fine. Russian and Greek languages were transcribed correctly;

Challenges

Asking at the right moment: Speech reaches us 8–22 s late, and surveyors often add the missing detail a sentence later. A question sent too early asks for something already said. Axsail sends a question only after a 60 s wait, and only when no extraction is running or pending, so every word already spoken has been read. To avoid constant buzzing, questions go out at most once every 2 minutes, up to 3 per to-do and 10 per inspection. Bee also rewords the to-do text on the watch and rejected non-ASCII text with HTTP 415. Questions are short ASCII and start with the component name, so they still make sense after Bee rewords them.
Findings have no stable identity: Every extraction rebuilds the list. To keep the surveyor's confirmations and edits when recording continues, we match findings by the utterance where each was first said, then by component name.
Private CA: The API uses a Bee-specific certificate authority, so the proxy was the reliable path.
Undocumented behaviour: Events such as tagged-utterance and new-utterance-chunks, the conversation-close timing and the auto-deletion all had to be discovered by testing.
Trust: A survey report carries legal and insurance weight. Every AI finding and severity needs the surveyor's confirmation, and new dictation never silently overwrites what the surveyor confirmed.
Auth on macOS: The CLI stored the token in Keychain but failed to read it back (error -25293). The workaround was BEE_FORCE_FILE_STORE=1.

What's next

Lower cost per survey: Every extraction re-reads the whole transcript, so a long survey gets more expensive with every sentence. We will switch on Claude's automatic prompt caching: setting in the request, and Claude caches the prompt itself and moves the cache point forward as the transcript grows. Each run then reads everything said before from the cache at about a tenth of the normal input price, or less, and pays full price only for the new sentences. The longer the survey, the bigger the saving. For a full day on a boat with breaks, we use the 1-hour cache, so a lunch break does not throw the cache away.
Photos with findings: A surveyor photographs every defect. Photos taken during a finding attach to it, and they go into the report next to the text.
Survey types: Condition surveys first, then pre-purchase, insurance and damage surveys. Each has its own checklist and report layout, built from real reports.
Vessel history: The next survey of the same boat starts from the last one: open defects, readings to compare, systems skipped last time.
Better questions on the watch: Ask about what matters most for the report: safety findings first, missing readings, systems not mentioned before the surveyor leaves the area.
Sign-off and delivery: A signed PDF, sent to the client straight from Axsail, with the report number and audit trail of what the surveyor confirmed and changed.