Sentience Finance

Real time emotional intelligence for financial decisions

Comment

Inspiration

I know many of my friends including myself struggle to manage expenses, we often use our expenses on the wrong product out of just looks good. I was inspired by a simple question: What if a financial system could observe, reason, and respond like a human advisor. Continuously and contextually instead of waiting for users to manually analyze their spending? Because financial tools today track expenses, generate charts, and provide static recommendations. But they do not understand users in real time. Financial stress comes not from lack of data, but from lack of timely guidance and behavioral awareness.

What it does

Sentience Finance is a Chrome extension paired with a FastAPI backend that watches for financial moments like checkout pages, banking sites, crypto platforms. And intervenes in real time through Gemini Live voice before impulsive decisions happen. The system:

  • First of all answers the question of this particular Hackathon. It's LIVE, barge-in, answers for a specific domain.
  • When you land on a checkout page, the extension detects it automatically. It pulls your emotional spending history from Firestore and opens a Gemini Live voice session, armed with your actual numbers, before you say a word.
  • Detects unhealthy spending trends before they become problems
  • Communicates naturally through voice interaction, detects emotions through camera after permission.
  • The system also calculates a Vulnerability Score (0–10) from three signals: how often you buy in a given emotional state, how much you typically spend, and how much you've regretted those purchases. In testing, interventions at scores ≥ 7.5 produced a 67% cart abandonment rate vs. 12% at scores ≤ 4.0 , a 5.6× difference.
  • Separate dashboard page to let user show their pattern dynamically which is just one tap away from the extension. The agent moves finance from tracking → understanding → guidance.

How I built it

It was made using the Google's ecosystem. From IDE to cloud everything. Google Stack Live voice + barge-in → gemini-2.5-flash-native-audio-preview-12-2025 Text + vision → gemini-2.5-flash SDK → google-genai >= 1.0.0 (v1alpha) Persistence → Firebase Firestore (Google Cloud) Deployment → Google Cloud Run (Dockerfile included) No third-party AI models. All inference is Google Gemini. The Vulnerability Score is calculated as:

$$V_{score} = \min!\left( \frac{F_{freq}}{10} \times 4 + \frac{A_{spend}}{250} \times 3 + \frac{R_{regret}}{10} \times 3,\ 10 \right)$$

where $\bar{x}$ is average spend in this emotional state and $\bar{r}$ is average regret score from past purchases. I focused heavily on building an agentic loop rather than a chatbot: observe → interpret → reason → advise → adapt.

Challenges I ran into

In this era of AI, there are various kind of agents everywhere. Every I thought about was already there. My idea did not come fully formed as usual. I implemented most of the feature one by one, had a hard time remaking the logic again and again, but I tried my best to implement as much feature as I can. While building the logic, since financial data is sensitive and ambiguous. Translating raw expenses into meaningful behavioral insights required multiple iterations of prompt architecture and structured data handling. Ensuring smooth interaction between conversational reasoning and system logic was complex. Finding the right balance between advice and user autonomy was a key design challenge.

Accomplishments that I'm proud of

  • The one I'm most proud of: the agent never waits to be asked.
  • Full respect to user privacy
  • Let user decide their mood manually or detecting through camera
  • I'm also proud of the Vulnerability Score. It's not a sentiment number pulled from nowhere — it's a formula grounded in the user's own behavioral data, and it demonstrably changes decisions.

And honestly: I'm proud that this is something I would actually use

What I learned

This was my first personal AI agent project, and the learning wasn't mostly technical. It was perspective.

Financial behavior is more psychological than mathematical. People don't overspend because they're bad at math — they overspend because they're human. Building for that means the AI has to be warm, specific, and non-judgmental. Getting the tone right in the system prompt mattered as much as getting the API calls right.

I also learned that the moment of intervention matters more than the quality of analysis. A perfect insight delivered a week later is useless. A good-enough insight delivered at the exact right second changes behavior.

What's next for Sentience Finance

  • Mood-Boosting Alternatives — When Gemini detects a stress spiral, it proposes a circuit-breaker: 5-minute guided breathing, 10-minute walk timer, or "sleep on it" wishlist save with next-morning reminder. Address the emotional root, not just block the symptom.
  • Predictive Intervention — Learn when a user is statistically most likely to make a regretted purchase and reach out proactively before they open a browser.
  • Regret Loop Closure — 48-hour follow-up: "How do you feel about that order now?" Closes the behavioral feedback loop.
  • Mobile app — Expand from a extension to a real time notifyer while making a purchase.
  • Collaboration — Combining with banks and other top financial tool to make a next level agent that will become a daily best friend for user.

Built With

  • audioworklet-api
  • chrome-extension-(manifest-v3)
  • fastapi
  • firebase-firestore
  • gemini-2.5-flash
  • google-cloud-run
  • google-gemini-live-api-(gemini-2.5-flash-native-audio-preview-12-2025)
  • google-genai-sdk-(v1alpha)
  • html/css
  • javascript
  • python
  • websocket
Share this project:

Updates