Wisper

Inspiration

My father runs a technical firm in northern Italy. I have been building software for its clients for the past few months, which is how I ended up watching this particular problem from close up.

His heating and photovoltaic technicians spend the day on site with gloves on and both hands busy. Then the report has to be filed: client, job, what they did, hours, kilometres, expenses, whether the work is finished.

Look at when it actually happens. Back at the office, near closing time, at the end of a day that started at seven, by someone who is already tired and wants to go home. It is a form, filled in by hand, one field after another, and nothing about it is automatic. So it gets rushed, or written from memory two days later, or forgotten altogether, and then the office is chasing a technician about a job he did last Tuesday.

The constraint was fixed before any code existed. Gloves, dirty hands, a ladder. Anything that needs a screen has already lost.

What it does

You say "Wisper". The phone can be in your pocket with the screen off. Then you just talk:

"Sono andato da Rossi per i tubi del fotovoltaico, tre ore e mezza, quaranta chilometri, dodici euro di raccordi, e ci devo ancora tornare."

(I was at Rossi's for the photovoltaic pipes, three and a half hours, forty kilometres, twelve euros of fittings, and I still have to go back.)

The fields fill in on screen while you speak. It answers out loud, asks for whatever is missing, and lets you fix anything by voice; say "no, aspetta, erano due ore" (no, hang on, it was two hours) and it changes. When you are done it reads the whole report back, you say "confermo", and the row lands in the company spreadsheet.

If you open it while standing on a site it goes first: "I see you are at Rossi's, and the only job open here is the Marostica photovoltaic. Is that right?" It proposes, it never assumes; you might be in the office filling in yesterday's report.

It also lists a client's open jobs on request, telling them apart by where they are, since a technician says "the one in via Roma" and not "M004". It creates new clients and jobs by voice and says so out loud when it does. It closes a job in the company sheet when you say the work is finished, so it stops being offered tomorrow. And you can cut it off mid sentence: say "Wisper" while it is talking and it stops and listens.

Why not just send a voice note, since the phone already records one hands free. Because a voice note does not file anything. It moves the typing from the technician to whoever plays it back at the office, and nobody can invoice against an audio file. Wisper produces the row itself, against the right job, and closes that job in the company sheet so tomorrow it is not offered again.

The phone is never touched. That is the whole project.

How I built it

How I work, stated up front. I ship products with AI; the product and design calls are mine, the code is Claude's. My job here was deciding what the app should do and what it should refuse to do, using it until something felt wrong, and directing the fix.

Several features exist because I used it and it annoyed me. It used to keep listening after I turned the screen off, and it picked up other people's conversations. It closed a job in the company sheet when I said "ho finito", meaning I had finished speaking. It read me a summary I already knew, with no way to cut it short. It held the microphone so tightly that nothing else on the phone could record. None of that came out of automated testing; it came out of being the person holding the phone.

What the app is made of. Native Android in Kotlin with Jetpack Compose, four layers, each replaceable. Vosk for the wake word, running entirely on the device, so until the word is heard no audio leaves the phone and no connection is needed. Android's own recogniser for the dictation. Gemini over REST for the reasoning, with a fixed response schema so the model has to return the report fields and cannot drift into prose; two models in cascade, because during development a brand new one answered 503 halfway through a test. System text to speech for the voice, with the Italian voice picked by measuring the ones the phone actually has instead of taking the default.

The whole loop lives in a foreground service, so it keeps listening with the app closed. A Google Apps Script bound to the spreadsheet is the bridge; the app posts a report, the script appends the row and hands back the client and job lists. The client and job list is also kept in a file on the phone, because the sheet takes seven seconds to answer from cold and the opening sentence cannot wait that long.

Two rules run through every file. One microphone owner at a time: the wake word and the recogniser alternate and never overlap, because two clients on one microphone do not throw an error on Android, they return silence, and you spend the afternoon debugging the network instead. And every turn ends with a spoken sentence, because when nobody is looking at the screen a silent failure is an invisible failure.

Challenges I ran into

The hard part was never the AI. It was honesty.

The model would announce data it had not stored. "Forty kilometres and twelve euros", with the kilometres field empty. On a work report that is the worst thing that can happen, because the technician is not looking at the screen; he hears it confirmed, he believes it, and the number does not exist. The fix was to take the job away from the model. The final read back is now built from the stored fields, so what you hear and what gets saved come from the same place and cannot disagree.

The same lesson came back three times. "Ho finito" in Italian means both "I have finished speaking" and "the job is finished", and the model kept closing jobs that had just been declared open. Prompting did not fix it. A rule in the code did: the confirmation turn cannot change the job status, whatever the model sends.

It came back again with the job itself. Told about a job that was not in the sheet, the model quietly picked the closest one that was, and then said out loud the name I had used rather than the one it had stored.

Then there was the microphone. Vosk and Android's recogniser both want it, and running them together produces no error at all, just silence. And once that was solved, a second version of the same problem: Wisper held the microphone so tightly that voice notes, calls and everything else on the phone recorded silence, and nobody would have blamed the app they had just installed.

Accomplishments that I'm proud of

It gives the microphone back. A wake word means holding the microphone open all day, and on Android whoever grabs it keeps it. For a while Wisper won every time and the rest of the phone went deaf. It now listens when the screen is off, or when it is the app you are looking at, and steps aside the moment you open something else, saying why in its notification. Nobody would ever notice this feature, which is the point.

What you hear is what gets saved. The report Wisper reads back before saving is built from the stored fields, not written by the model.

It refuses to guess which job you mean. The model now reports the job in the technician's own words, the code checks the two share a word, and anything else is opened as a new job and announced out loud.

It writes where the work already happens. The row lands in the spreadsheet the office has always used, against the right job, and the job is closed in that same sheet when the work is done. No new dashboard for anybody to learn or ignore.

And it survived contact with a real user who was not me. My father tried it at lunch and three of the things above exist because of what he said in fifteen minutes.

What I learned

That the most useful feedback of the whole build came from a fifteen minute conversation with my father, not from more code.

And that with a voice assistant, the things that must never be wrong do not get asked of a model politely; they belong in the code. Every time I settled for instructing the model to be careful, it was careful most of the time, and "most of the time" is not good enough for a number somebody is going to invoice.

What's next for Wisper

A spreadsheet is where this starts, not where it ends. The next step is writing into the system a firm already runs, an ERP like Odoo, so the report lands as a timesheet entry against the right job instead of a row somebody has to copy over by hand.

Then the two things a report is actually used for. Signatures: the surveyor's is signed once and kept in Drive, and the client signs on the phone there and then, offered only when the job is being closed, because a signature on unfinished work means nothing. And receipts: photograph it, read it with OCR, attach it to the expense. Right now an expense is a number somebody said out loud, and the accountant has to take it on trust.

Before any of that, an offline queue. The wake word already works with no connection, the reasoning and the save do not.

Things worth knowing

Single user. There is no login; the app writes to one spreadsheet, its owner's.

The spreadsheet endpoint is unprotected. It is a public Apps Script URL.

It needs a connection to understand you. The wake word is fully offline, the reasoning and the save are not. It does not fail silently: it says so out loud, and a half filled report is never thrown away, so saying "Wisper" again when the signal is back repeats the last question and carries on.

Noise degrades accuracy. Published Italian speech recognition benchmarks sit around 3% word error, measured on clean studio audio. On a building site expect meaningfully worse. This is the honest weak point of the whole idea.

There is no APK to download, and that is deliberate: the model key would travel inside it and anyone could pull it back out. The honest fix is a server that holds the key, which is not built yet, so the project ships as source and you build it with your own key.

Only two people have used it: me, and my father for fifteen minutes at lunch. No technician has run a whole day on it yet.

Built With

• android
• gemini
• google-apps-script
• google-sheets
• gradle
• jetpack-compose
• kotlin
• okhttp
• speech-recognition
• vosk