Allocentra AI

An AI agent that visually understands portfolio dashboards and autonomously executes allocation workflows.

Comment

Category

Allocentra AI fits the UI Navigator track. You can try it out here: https://allocentra-ai.vercel.app/.

It uses Gemini multimodal to interpret a dashboard screenshot, then returns structured, executable actions that the application can perform. Instead of a simple text chatbot, the agent observes a visual interface, reasons over what it sees, and helps the user act on that interface through quick actions and rebalance execution. The agent is hosted on Google Cloud Run, which satisfies the cloud hosting requirement.

Inspiration

Allocentra AI was inspired by a simple problem. Retail portfolio tools show data but they usually stop short of turning that data into actionable allocation decisions. The project pushes beyond passive dashboards by combining portfolio analytics, AI-generated insights, and one-click execution of suggested rebalancing moves.

What it does

Most portfolio tools stop at charts and tables. They show investors what happened, but they do not bridge the gap between visual understanding and action. Allocentra AI was built to explore a next-generation agent experience where an AI can look at a portfolio dashboard the way a human would, identify risk or concentration issues, and immediately suggest or trigger the next step.

How I built it

I built Allocentra AI in two layers:

  1. Portfolio dashboard layer: a Next.js frontend with API routes and a Neon Postgres database powers the portfolio dashboard, trade ledger, allocation views, and historical performance. A separate Google Cloud Run prices worker keeps the system updated with live market prices and historical closing data.
  2. Agent layer on Google Cloud: the multimodal agent is hosted on Google Cloud Run and uses Gemini through the Google GenAI SDK. The frontend captures the dashboard as a screenshot and sends it to the agent service for visual analysis. The backend then couples the model’s output with live portfolio data to enrich the AI response with exactly calculated rebalance trades, so the result is not just insight text but executable portfolio actions.

Challenges I ran into

  • Converting multimodal model output into strict, executable action schemas
  • Grounding AI recommendations in the actual portfolio state, not just what the screenshot appears to show
  • Reconstructing historical portfolio performance from trades plus time-series pricing
  • Making the agent useful as a UI-aware system, rather than just a text summarizer
  • Keeping the flow responsive across frontend capture, Cloud Run inference, database reads, and action execution

Accomplishments that I'm proud of

  • Built a multimodal UI agent with Gemini that acts like the user’s hands on screen: it observes the portfolio dashboard visually, interprets UI elements from a screenshot, and turns that understanding into executable actions rather than only returning text.
  • Connected the agent’s actions to real product behavior, including opening trade-ledger views, changing timeframes, and executing portfolio rebalances.
  • Hosted the agent on Google Cloud Run.
  • Unified portfolio analytics, AI reasoning, and action execution into a single end-to-end workflow.

What I learned

  • The jump from “AI insight” to “AI action” requires backend validation and normalization
  • Multimodal agentic AI requires strict action constraints.
  • Screenshot understanding is a strong interface for UI agents because it captures visual context clearly and doesn't rely on the DOM.

What's next for Allocentra AI

  • Expand the UI navigation layer so the agent can handle more complex portfolio workflows.
  • Evolve the experience into a more autonomous financial copilot that can gather insights from external websites and navigate across multiple pages of the application through screenshot-based interaction.
  • Add user accounts, authentication, and multi-portfolio support
  • Improve the AI analyst with richer context, longer memory, and personalized investing preferences
  • Add alerts, watchlists, scenario analysis, and more advanced performance/risk metrics

Built With

Share this project:

Updates