Ecostream Monitor

ESM which is an admin-side project monitors data from bigquery + vertex ai (gemini) + other google cloud product account and on datadog for improving customer experience with need for local search

Comment

Inspiration

Living in Lagos, the gap between local, sustainable farmers and digital consumers is massive. While we want to support SDG 12 (Responsible Consumption), finding low-carbon food sources is difficult, and mobile networks can be unreliable.

We realized that building an AI app isn't enough—it has to be "Emission Critical." If the AI lags or hallucinates, the user leaves, and the sustainable choice is lost. We were inspired to build EcoStream Monitor not just as a food finder, but as a demonstration of how Professional-Grade Observability (Datadog) ensures that AI-driven sustainability efforts are measurable, reliable, and fast, even in emerging markets.

Hardware requirements

We included a json called datadog_monitor and ecostream_monitor for datagog category. Furhtermore we also included our env file in the additional file

What it does

EcoStream Monitor is a voice-activated, observability-first AI guide that connects users to local sustainable markets.

  1. Voice Interaction: Users ask (via voice) for produce (e.g., "Where can I get fresh Yam in Ikoyi?").
  2. AI Intelligence: The system uses Google Vertex AI (Gemini 1.5 Flash) to interpret intent, locate markets, and calculate the carbon footprint saved by shopping locally.
  3. Real-Time Feedback: It responds with a human-like voice using ElevenLabs, ensuring accessibility for all literacy levels.
  4. The "Magic" (Observability): Behind the scenes, the system acts as a telemetry bridge. It doesn't just logs requests; it treats "Carbon Saved" as a technical metric. If a user finds a market, Datadog Case Management automatically triggers a business workflow for admins to verify the impact.

How we built it

We architected a unified telemetry pipeline using Python (Flask):

  • The Brain (Google Cloud): We utilized Vertex AI (Gemini 1.5 Flash) for low-latency reasoning. We instrumented the model using ddtrace to capture P95 latency and token counts. Data persistence for long-term impact reporting is handled by BigQuery.
  • The Eyes (Datadog): We deployed the Datadog Agent (US5 Region) to monitor the full stack.
    • APM: traces every request from OAuth login to AI generation.
    • Case Management: We configured a specific Monitor that listens for the event Sustainable Market Found. When triggered, it opens a Case (e.g., EUTS-1) in Datadog, assigning it to an admin for impact verification.
    • Dashboards: We built a custom dashboard correlating Technical Performance (Latency) with Business Impact (Total CO2 Saved).
  • The Nervous System (Confluent): We used Confluent Cloud (Kafka) to stream market-activity events in real-time, allowing for future scalability where other services can subscribe to market demand without hitting the primary database.
  • The Voice (ElevenLabs): We integrated the ElevenLabs Python SDK to generate localized, natural-sounding audio responses, making the tech feel human.

Challenges we ran into

  • Latency vs. Observability: We needed to ensure that adding heavy instrumentation (Kafka producers, BigQuery inserts, and Datadog traces) didn't slow down the user experience. We solved this by using asynchronous patterns and efficient SDK implementations.
  • Datadog Case Management Integration: figuring out how to programmatically trigger a "Case" from a code event required deep diving into the Datadog API. We successfully implemented a flow where a Python api.Event.create() triggers a Monitor, which then opens the Case.
  • Region Configuration: We initially had issues connecting to the Datadog Agent until we explicitly configured the environment to us5.datadoghq.com to match our account region.

Accomplishments that we're proud of

  • End-to-End Observability: We can see the exact millisecond latency of a Gemini thought process in Datadog.
  • Automated Impact Tracking: We moved beyond "viewing logs" to Actionable Intelligence. The fact that a carbon-saving transaction automatically opens a Case file for an admin is a workflow we are very proud of.
  • Validating the Stack: Successfully getting Google OAuth, Vertex AI, Confluent Kafka, and ElevenLabs to all talk to each other within a single trace view.

What we learned

  • AI Needs Monitoring: "It works on my machine" doesn't apply to LLMs. Variable latency is real. Seeing the "Gemini Thinking" span in Datadog gave us insights into how prompt complexity affects speed.
  • Data in Motion: Using Confluent taught us that sustainability data shouldn't be static; streaming it allows for immediate reaction to supply/demand changes.
  • Observability is Sustainability: You cannot improve what you cannot measure. By treating "Carbon Saved" as a metric just like "CPU Usage," we made sustainability an engineering KPI.

What's next for Ecostream Monitor

  • Predictive Analytics: Using Confluent to stream data into a forecasting model to predict food shortages before they happen.
  • IoT Integration: Adding sensors to the farmers' stalls to push real-time inventory levels into the Confluent stream.
  • Smart Alerts: Using Datadog Watchdog to detect anomalies in carbon savings data (e.g., if a specific region suddenly stops saving carbon, alerting us to potential supply chain issues).

Built With

Share this project:

Updates

Submission history