-
Overview of our Project
-
Home
-
Verbally Asking questions, shows arms movement.
-
-
Inspiration
We started by looking for what all of us had in common, and we realized we’re all partially visually impaired and need glasses to see clearly. That made us wonder how people with complete vision loss handle tasks that the rest of us take for granted. Around the same time, the theme had us thinking about the deep ocean, where sunlight never reaches, yet whales and dolphins find their way through sound and touch. Their world isn’t any smaller for lacking light; they just navigate it differently. That idea stuck with us. With that mindset, we researched areas where blind and visually impaired people face the most barriers, and finance stood out because trading is taught almost entirely through charts and visuals. We wanted to give people a way to dive into the markets the same way life thrives in the deep: by sound and by touch.
What it does
Sonar is a voice-first investing tutor for blind and low-vision learners. You hold the space bar to ask a question out loud, like "How did Apple do this month?" or "What if I'd put $1,000 into VOO a year ago?", and Sonar answers in a natural voice using real market data, never numbers from memory. Every price chart is felt instead of seen: a small robotic arm draws the line on your palm and taps at key moments like a peak or a finished trade, or the chart plays as rising and falling tones. Learners get $10,000 in practice money to trade stocks and ETFs, with a spoken read-back and a confirmation step before anything happens. Six short lessons with quizzes cover reading a graph, stocks versus ETFs and fees, a first trade, tokenized funds, backtesting, and options basics. To teach tokenization, Sonar uses PVOO, a practice token on Solana that mirrors one share of VOO, the S&P 500 ETF. Buying it mints a real token on devnet that settles in about two seconds instead of the next business day, and anyone can check that every token is backed on the public blockchain. No real money moves.
How we built it
The frontend is a single-screen Next.js app built for keyboard and screen-reader users. It has push-to-talk, numbered menus read aloud, a palm scene that animates the arm on screen, and sonification that turns chart height into pitch. ElevenLabs handles speech-to-text and the tutor's voice, with OpenAI text-to-speech as a fallback. The tutor runs on OpenAI's Codex model with function calling. It picks from 18 tools (quotes, charts, backtests, trades, lessons, token backing), and guardrails block any number that didn't come from a tool, any personal advice, and any claim of a trade the learner never confirmed. Trades go through a Python FastAPI engine that works as a practice brokerage. It pulls daily prices from Alpaca into Tiger Data, using TimescaleDB hypertables and continuous aggregates for weekly and monthly candles, and it records every trade as a single database transaction. PVOO is an SPL token we created on Solana devnet with @solana/web3.js: buys mint tokens, sells burn them, and a backing check compares the on-chain supply against our ledger. The palm arm is an Arduino with a three-joint arm that can draw on a palm. Its firmware does inverse kinematics and smooth, speed-limited motion with safety limits, and it receives the same drawing commands as the on-screen arm over Web Serial. We built fallbacks throughout, so the demo keeps working if the database, devnet, or the AI service goes down.
Challenges we ran into
Our biggest backend challenge was the AI itself. Our Gemini API key was blocked partway through the event, so we moved the whole tutor to OpenAI's Codex models using function calling, which meant rewriting how the conversation and tool calls were passed back and forth. Keeping the tutor honest was just as hard: we built guardrails so it can't say a price or return unless a tool actually returned it, but they sometimes blocked correct answers and forced slow retries until we tuned them. Solana devnet was another hurdle, since the faucet rate-limited our airdrops and we couldn't count on the network being up during a live demo. That pushed us to design fallbacks everywhere: if devnet is down, trades still complete and are clearly marked as simulated, and if the database goes down, the app keeps serving prices from backup files instead of crashing. Setup also ate time, with slow installs and packages that wouldn't build on one teammate's version of macOS.
Frontend’s biggest challenge was staying in sync with the backend. The palm arm bridge was written for an early prototype and spoke a different protocol than the finished firmware, so we had to rebuild the driver before charts reached the hand correctly. Building for blind users raised the bar further: every status had to be announced by a screen reader, every action had to work from the keyboard, and focus could never leave someone lost on the page. With Claude’s help, we wrote unit tests for the state machine, menus, trade flow, and chart drawing, and built a mock arm and fake serial port so the hardware path could be tested without the hardware. We also made the end-to-end suite run against a production build, the way a CI pipeline would, and in the end 200 unit tests and 130 end-to-end runs passed, ready to gate every merge.
For the hardware side, our first obstacle was getting anything printed at all. No 3D printers were open to UMD students that day except the ones at Startup Shell, which we didn’t have access to. We ended up hiring a third-party 3D printing service bureau, and someone on our team had to drive out to pick up the parts in person. By the time we had them, a two-hour print job had cost us a good five hours. Then the servo housings turned out to be too tight. We had designed them around one servo that had a small divot for tucking in its wire, and only after printing did we realize the other servos didn’t, so the wires got in the way and we spent a long time filing every part down to fit. On top of that, we had no screws, so we called around until we found a friend with an Arduino kit that had some, which cost more time. Once the arm was assembled, we moved on to the code, and a bug sent the lift joint’s servo the wrong way. It kept driving against a position it couldn’t reach until the servo was damaged inside, so we had to take the arm apart and rebuild it with a replacement. Eventually we fixed the code and got the arm working and integrated with our website.
Accomplishments that we're proud of
We’re proud that we built Sonar like we would run a research project, not a weekend experiment. We began with a rough draft of ideas, worked together to find a real problem worth solving, and kept that problem at the center of every decision. Before writing the real app, we used Claude to build a mock webpage, which let us test the experience and see how blind and low-vision users would move through it. From there, we added tutor tools that prevent it from making up prices or returns and guide users safely through practice trades. Most of all, we’re proud that this is our first hackathon project and that we finished it, from a draft on paper to a working app and a robotic arm that draws charts on your hand.
What we learned
Our biggest lesson was how to build under pressure. Almost nothing went as planned: our AI access was cut off partway through the event, the 3D printers we were counting on weren’t available, and each setback pushed our timeline further back. Rather than scaling back an ambitious idea, we changed our approach each time something broke and still finished on time. We also learned that accessibility can’t be added at the end; designing for people who can’t see the screen shaped every choice we made, from the voice tutor to the arm that draws on your hand.
What’s next for Sonar
Sonar’s next step is to move from practice to the real market. We plan to build an accessible way to trade with real money, starting with Solana and expanding to stocks, options,s and other assets. Every trade would keep the spoken read-back and two-step confirmation that protect learners today. We also want a smarter AI tutor that coaches each user on their own portfolio, builds lessons around their goals, and explains market news as it happens. The palm arm isn’t perfect yet and needs steadier, smoother movement, but we’re proud of what we built and see it as the beginning of a tool that brings blind and low-vision investors into the market.
Log in or sign up for Devpost to join the conversation.