Beacon

• 

Beacon

Beacon is a voice first financial assistant for visually impaired users, answering complex finance questions by bypassing tedious screen readers which are unrealistic for deciphering bank statements.

Inspiration

Banking apps have become navigable with a screen reader. Two decades of litigation and regulation made logging in, reading a balance and paying a bill work for blind and low-vision users at most major banks. Understanding your own financial position never followed, because a screen reader can only read what is on the screen, and no screen in a banking app says "this month is unusual". That sentence does not exist until something computes it.

8,537,000 people in the US reported blindness or serious difficulty seeing even with glasses in 2024. Research by Griffith, Wentz and Lazar found blind participants took twice as long as sighted users on sites with poor accessibility, and five times as long when they had to call customer service to finish a task. That paper was cited in the Department of Justice's regulatory analysis for the ADA Title II web rule.

The study that pushed us to build this is a 2026 scoping review in JMIR covering 23 studies across 12 countries. It found that the evidence gaps are widest at the stage where someone lives with a financial product over time, and that no study has evaluated whether conversational interfaces reduce these barriers. We built the thing that review asked for.

What it does

You ask Beacon a question out loud and it answers in a sentence. Example from a test run:

What's unusual? In September 2026, your spending is normal except for three things: three new subscriptions you didn't have before, Paramount Plus, Audible, and CloudVault, totaling $47, a dining category outlier at $314.39, up 70.3% on your usual $184.63, and Corner Bistro charged you $38.50 twice, which may be a duplicate.

And last month? Nothing stands out in August 2026. Your spending looks normal.

The second answer shows two things that matter in speech. A follow-up keeps the subject of the previous question, so you never repeat yourself, and a question with a boring answer gets a short one.

Beacon handles scoped questions ("how much did I spend at Kroger in July?"), counts ("how many times did I eat out?"), superlatives ("what was my biggest purchase?"), lists ("what subscriptions do I have?"), income and savings ("am I saving money?"), and trends ("is my dining spending going up?"). A question it cannot answer from a transaction history, such as a bank balance or a transfer, is declined in one sentence. A request for advice is refused by a hardcoded path that never reaches the model.

Everything works by typing as well as by voice, and both give the same answer because both run through the same function. The interface is built for screen readers first, since people who use one already have text-to-speech and run it faster than any synthesized voice.

How we built it

A question travels through four stages. Nemotron reads it and returns a plan: what to measure, about what, over which period. pandas executes that plan against the ledger and produces the figures. Nemotron turns those figures into a sentence. Two guards check the sentence before ElevenLabs speaks it.

The model never does arithmetic. It also never sees a transaction. It receives precomputed aggregates: six months of totals, this month's categories and merchants, the recurring charges, and the anomalies our rules found. The app shows that exact payload on screen during the demo, so the claim can be read instead of believed.

The stack is Python, FastAPI and pandas on the backend, React and TypeScript on the front, Nemotron 3 Nano 4B served by vLLM on an NVIDIA L4 through Brev, and ElevenLabs Agents for speech. The ElevenLabs agent calls our own server as its language model, which is how the same answer reaches both the voice and the screen.

Challenges we ran into

The hard problem was not building the analysis. It was making the thing understand what a person actually says. We rebuilt that part four times, and each rebuild was forced by a measurement.

Four canned questions. The first version answered the four questions in our demo script. We wrote 41 questions a judge might ask and measured it: 7 correct. Three failures were worse than silence, because the answer was confident and about something else. "Did Netflix charge me?" returned a summary of the whole month.

Pattern matching, extended. We added entity extraction for categories, merchants and time ranges. That took it to 63 of 64 with no wrong answers, in 2 milliseconds. It looked finished.

The voice session that broke it. Speech is not typed text. "And how is my spending compared to the previous month?" was answered as a lookup for City Power and Light, because "to the previous month" was captured as a subject and the word "and" matched a token inside that merchant's name. Four of eleven spoken questions came back wrong, every one of them fluent. The test suite was green the whole time. What exposed it was a log of questions and answers we had added an hour earlier.

Letting the model route. We deleted the pattern router. Now Nemotron reads the question and returns a plan that pandas executes. Understanding came back to 62 of 64 with no misroutes, and the failures turned into honest refusals. It costs two to four seconds per new question, which we accepted.

Two mechanisms make free-form questions safe. The first is that the model gets aggregates and no transactions, so it can talk about the data without doing arithmetic on it. The second is the pair of guards. The numeric guard checks every number in the answer against the facts that were sent, allowing formatting differences, so $47 and forty-seven both pass while a number the model worked out itself fails. It caught the model claiming spending was "$189.60 more than last month", which is a subtraction of two real figures.

The name guard exists because of a failure the numeric guard cannot see. The model once said "For groceries in August 2026, you spent $2,803.34", where the figure was the correct total for the entire month and the label was wrong. Every number in that sentence was true and the sentence was still misleading. The name guard now checks every category and merchant the answer mentions against what the answer is actually about.

Across 145 checks, ten answers were blocked: eight attached a real figure to the wrong category, two contained a number the model had calculated. None of them were spoken. A blocked answer is retried once and then replaced by a deterministic sentence built from the same figures.

Infrastructure took its own share of the time. The GPU image had no C compiler and no CUDA compiler, so vLLM could not build its kernels or its sampler. The SSH tunnel to the GPU dropped silently, and because the fallback is a correct template sentence, nothing looked broken while the model was gone.

Accomplishments that we're proud of

Understanding 62 of 64 real questions, including paraphrases, speech recognition errors, follow-ups and curveballs, with no question answered as a different question.

Ten blocked answers out of 145 checks, and nothing wrong ever spoken. We can show the guard log and the turn log to back that up.

Voice and text that cannot disagree, because they are the same code path, and an interface that a screen reader user can drive without the voice mode being in the way. While the agent speaks, the live region stays silent so the screen reader does not talk over it.

What we learned

Reading the logs after a rehearsal found bugs that a green test suite hid. Three of our worst failures were invisible to tests, because the code did exactly what it was written to do.

A fluent answer to the wrong question is as harmful as a wrong number, and it is harder to catch. That realization produced the name guard and the read-back, where the answer states how the question was understood before it answers.

For a user who cannot glance at the screen to check a figure, the right failure mode is silence or a stiff correct sentence. We built for that from the start and it shaped every later decision.

What's next for Beacon

Full cloud deployment, so a demo does not depend on a laptop and a tunnel.

Local-only inference, tested end to end, so the transactions and the model can both stay on the user's machine. The payload we send is already small enough for a 4B model, which makes this a configuration change instead of a rewrite.

Offline mode with cached audio, and a full pass with a screen reader and the monitor off.

We are also ideating the most effective distribution for beacon to the target audience, and would like for it to become a venture. Today, insurance providers already fund assistive technology for members and have a direct interest in reducing the cost of dependence, which makes them a route to the people who need this. Beyond that, the analysis runs on a transaction export and replaces nothing a bank already does, so it can sit alongside an existing finance app or be built into one. This is a hackathon build, and that direction is a plan we have thought through, with no company behind it yet.

Built with

python, fastapi, pandas, react, typescript, vite, nvidia-nemotron, vllm, nvidia-brev, elevenlabs, ngrok, pytest, vitest, uv

Built With

• elevenlabs
• fastapi
• ngrok
• nvidia-brev
• nvidia-nemotron
• pandas
• python
• react
• typescript
• uv
• vite
• vllm