Inspiration

Navigating through multiple screens just to send money or check a balance feels outdated. Inspired by the rise of voice assistants and conversational AI, I set out to create a banking experience that is intuitive, hands-free, and accessible to a wider audience, including those who may not be tech-savvy or prefer their native language.

What it does

FinVoice is a futuristic, voice-controlled banking assistant. Users can interact with their "NOVA CORE" AI assistant to manage their finances completely hands-free. You can say things like, "Transfer $50 to Aditi," or ask "Mera balance kya hai?" (What is my balance?), and the system understands and executes the request instantly. It also provides visual feedback through a dynamic spending analysis chart and transaction history.

How I built it

The project is a full-stack application. The React frontend uses AudioWorklet to capture high-quality audio from the user's microphone. This audio is streamed via WebSockets to a Flask backend. The backend acts as the brain, piping the audio to Google's Speech-to-Text API for transcription. The text is then sent to OpenAI's GPT-3.5 to determine the user's intent and generate a response. Finally, the response is converted to lifelike speech using ElevenLabs and streamed back to the user. All financial data and transactions are securely stored in a Supabase PostgreSQL database, which is updated in real-time.

Challenges I faced

The most significant hurdle was the "stale closure" problem in React, where the voice assistant would sometimes use outdated information (like an old balance). Solving this required a deep dive into React's useRef and hooks to ensure the WebSocket listener always had access to the latest state. Another challenge was coordinating the asynchronous nature of multiple API calls to ensure a snappy, conversational feel without long pauses.

What's next for FinVoice

I plan to add more advanced features like bill payments, savings goal tracking, and multi-factor authentication using voice biometrics to make it a complete financial assistant.

Built With

Share this project:

Updates

Submission history