Pani Dumka (Пані Думка)

Pani Dumka: A multimodal AI orchestrator using Gemini & Vertex AI for background agent execution, state preservation, and zero-trust security on Google Cloud

Comment

🇺🇦 Пані Думка (Pani Dumka)

💡 Inspiration

The genesis of Пані Думка (Pani Dumka) is rooted in a profound personal and social mission. Founded by the creator of SmileAfterBurn — a project dedicated to the digitalization of Ukraine's social sphere after overcoming severe physical trauma, burnout, and a grueling recovery path — the goal was to design a sovereign, strategic digital co-founder.

Traditional single-threaded chat models are severely limited when dealing with multi-faceted enterprise workflows, OSINT, and complex system auditing. We envisioned a highly intelligent, culturally grounded Ukrainian AI Orchestrator that coordinates an elite fleet of 20 specialized sub-agents to act as a seamless extension of human capability — combining raw cognitive depth, strict mathematical determinism, and deep empathy.

⚙️ How We Built It

Pani Dumka is built as a production-grade, containerized full-stack ecosystem:

  • The Frontend: An ultra-premium, responsive React (Vite) interface styled with custom CSS and Tailwind, integrating bilingual support (Ukrainian/English) and voice synthesis (ElevenLabs).
  • The Orchestration Layer: Powered by Gemini 3.5 Pro via the Google Cloud Vertex AI enterprise endpoint. It acts as the "Brain," managing intent recognition, conversational memory, and state management.
  • The Fleet: Powered by Gemini 3.5 Flash for rapid, low-latency execution of specialized workflows (RAG, Data Analysis, Security Audits, OSINT).
  • A2A Protocol & Quality Gates: Inter-agent communication is structured using a formal A2A (Agent-to-Agent) JSON-RPC 2.0 protocol. To prevent hallucinations and enforce strict compliance, all agent outputs pass through deterministic validation gates (MathCore).
  • Zero-Copy Multimodal RAG: Directly references gs:// URIs from Google Cloud Storage to read large-context media (e.g., video transcripts and frames) natively on Google's internal network, preventing network overhead and security exposures.

📊 Mathematical Rigor (LaTeX Support)

Unlike typical LLM systems that rely purely on probabilistic output, Pani Dumka enforces strict mathematical verification for security, data quality, and profiling:

  1. Shannon Entropy ($H$) is computed by the Security Agent (Луцик) to detect encrypted payloads, obfuscated code, and potential keys within text streams: $$H(X) = -\sum_{i=1}^n P(x_i) \log_2 P(x_i)$$
  2. Pearson Correlation Coefficient ($r$) is used by the Auto-ML Agent to select relevant numerical features during time-series data analysis: $$r = \frac{\sum (x_i - \bar{x})(y_i - \bar{y})}{\sqrt{\sum (x_i - \bar{x})^2 \sum (y_i - \bar{y})^2}}$$
  3. Cosine Similarity handles semantic routing and vector matchmaking for personalization: $$\text{similarity} = \cos(\theta) = \frac{\mathbf{A} \cdot \mathbf{B}}{|\mathbf{A}| |\mathbf{B}|}$$

🧠 What We Learned

  • Context Optimization: How to orchestrate agent-to-agent state handovers without context bloat, transferring only the necessary structured state between active sub-agents.
  • Multimodal Efficiencies: Utilizing Gemini's 2M-token context window directly over GCS files drastically reduces pipeline complexity (no manual slicing, chunking, or frame extraction required).
  • Bridges to External Services: Implementing the Model Context Protocol (MCP) securely using sandboxed tooling preserves a zero-trust architecture while granting agents access to AlloyDB, BigQuery, and Google Workspace.

🚧 Challenges We Faced

  1. Deterministic LLM Validation: Enforcing structured JSON schemas across a 20-agent fleet. We resolved this by building a recursive validator that sanitizes and parses outputs, forcing retries if standard schemas fail.
  2. Authentication & Git Overrides: During local development, Git Credential Manager (GCM) repeatedly interfered with our HTTPS Personal Access Tokens, triggering persistent 403 Forbidden rejections. To bypass this, we implemented our own OAuth 2.0 Device Authorization Grant (RFC 8628) flow inside a Node/TypeScript script, allowing secure, interactive user auth and token retrieval directly on-device.
  3. State Synchronization: Keeping the real-time Voice UI (using WebSockets) synchronized with the active cognitive agent state without lagging.

Built With

Share this project:

Updates

Submission history