CapitalVoice

Inspiration

Many users struggle with the complexity of digital banking, especially when performing transactions, checking balances, or managing services online. We wanted to create a natural, intuitive interface where users could interact with their finances as if speaking to a personal assistant. This led to the development of CapitalVoice, a multimodal AI platform combining voice, face, and emotion recognition to simplify banking operations.

What it does

CapitalVoice allows users to:

• Perform transactions, balance inquiries, and service management via voice commands.
  
• Authenticate facially in real-time.
  
• Detect emotional states to provide adaptive responses.
  
• Merge voice and visual inputs for a secure and personalized experience.
  

Mathematically, our interaction model can be expressed as:

$$ f(x_v, x_a) = \alpha \cdot \text{VoiceIntent}(x_a) + \beta \cdot \text{FaceIdentity}(x_v) + \gamma \cdot \text{Emotion}(x_v) $$

where (x_v) is the visual input (face image) and (x_a) is the audio input (voice command).

How we built it

• Frontend: React interface for live camera and microphone capture.
  
• Voice Layer: Node.js with Whisper API (speech-to-text) and ElevenLabs API (text-to-speech).
  
• AI Layer: FastAPI backend with DeepFace and OpenCV for face and emotion recognition.
  
• Data Flow: Asynchronous REST communication ensures low-latency responses between Node.js and FastAPI services.
  

Challenges we ran into

• Maintaining real-time performance while combining vision and speech models.
  
• Optimizing models to work efficiently on CPU hardware.
  
• Synchronizing image and audio data across Node.js and FastAPI.
  
• Stabilizing emotion detection for consistent results.
  

Accomplishments that we're proud of

• Successfully implemented a secure, multimodal AI system for banking.
  
• Achieved real-time recognition of faces and emotions.
  
• Built a working prototype that allows users to perform key banking operations naturally.
  

What we learned

• Combining language and vision models enhances accessibility and trust.
  
• Effective cross-service integration is possible with asynchronous endpoints.
  
• Human-centered AI design requires balancing security, usability, and performance.
  

What's next for CapitalVoice

• Integrate live Capital One API endpoints for real transactions.
  
• Deploy a cloud-based version with encryption and authentication.
  
• Extend capabilities with contextual memory and multilingual support.

Built With

• fastapi
• node.js
• vite