SynErgi

SynErgi is a self-evolving grid optimizer combining multi-agent planning and GRPO reinforcement learning to cut costs, emissions, and risk—learning cheaper, cleaner energy decisions every run.

Comment

SynErgi — Self-Evolving Grid Optimization

Multi-agent orchestration + GRPO RL + prompt self-improvement for power grid planning
Repo: github.com/imruchi/weavehacks2

Inspiration

Analysts managing modern power grids are overloaded by telemetry. When humans can’t scan everything, predictions degrade and resources are misallocated.
The outcome: avoidable curtailment of clean energy, higher costs, and reliability risks.

We built SynErgi so agents can analyze, plan, act, and improve themselves on every loop — turning data overload into better grid decisions.

What SynErgi Does

Closed loop:
Analysis → Planning → Action → Critique → Mutation (self-improve) → RL update

  • Analyst Agent — tool-using reasoning over a digital twin (spatial clustering, storage strategy, baseline comparison, cascading risk).
  • Planner Agent — converts insights into executable actions: per-node MW adjustments with a strict, parser-friendly format.
  • Actuator Agent — executes against the simulator/twin (charge/discharge, demand response, line reconfig) and updates state.
  • Critique/Gate — evaluates deficit, cost, emissions, risk, fairness and decides approve/revise.
  • Mutator — evolves the Planner’s system prompt (ε-greedy bandit + critique feedback). Records prompt before/after snapshots.
  • GRPO RL Learner (Serverless on OpenPipe) — streams trajectories (state, plan, reward) to GRPO, which learns a policy that biases future planning toward cheaper, lower-risk actions.

Two improvement engines

  1. GRPO RL (policy learning): learns from reward signals to improve planning behavior.
  2. Prompt self-improvement (instruction learning): critique-driven mutations refine system prompts without retraining.

Telemetry: full reward breakdown + traces via Weights & Biases Weave.
Bench outputs saved as .jsonl for regression tracking.

End-to-End Learning Flow

Energy Grid → Agentic System → RL (GRPO) → Prompt Self-Learning

  1. Energy Grid / Digital Twin – Simulated grid produces state (demand/supply per node, SOC, risks).
  2. Agentic System (LangGraph) – Analyst → Planner → Actuator → Critique → reward computed.
  3. RL on Serverless (OpenPipe) with GRPO – Trajectories streamed to GRPO to train a lightweight planning policy.
  4. Prompt Self-Learning (ART-style) – Mutator performs prompt nudges (spread vs focus, caps, normalization) with ε-greedy updates.

Result: Behavior improves on both policy learning and instruction learning, feeding back into the next cycle.

Architecture Overview

Digital Twin / Simulator
│
▼
Analyst Agent ──► (clustering, storage, baseline, risk)
│
▼
Planner Agent ──► (simulate, cost/risk, policy vault)
│
▼
Actuator Agent ──► (charge/discharge/DR/reconfig) → updates twin
│
▼
Critique/Gate ──► approve / revise + metrics (deficit, $ cost, CO₂, risk)
│
├─ approve/revise → Mutator (prompt nudge + bandit update)
│
└─ trajectories → GRPO on OpenPipe (serverless RL) ──► updated policy

How We Built It

  • Agents & Orchestration: langgraph + LangChain messages/tools in a deterministic state machine.
  • Tools: spatial clustering, storage strategy, baseline comparison, cascading risk, simulation, cost/risk, and policy vault.
  • Rulebooks + Bandit: multiple operational rulebooks (safety thresholds, caps, priorities); ε-greedy picks policies updated via rewards.
  • Prompt Mutation (ART-style): Mutator appends targeted nudges and logs prompt_before, prompt_after, and snapshots per iteration.
  • RL with GRPO (Serverless): trajectories streamed to GRPO training, biasing Planner behavior toward better actions.
  • Simulator/Digital Twin: tick-based grid generator + reward calculator (deficit, $ cost, emissions, risk, fairness).
  • Observability: full Weave traces, bench/e2e JSONL results.
  • Retrieval Utilities: used Tavily for prototyping during analysis.

Repository Structure


weavehacks2/
├─ agents/
│ ├─ prompts/ # Planner prompt variants (YAML)
│ └─ tools/ # Analysis, planning, and actuation utilities
├─ data/
│ ├─ simulator.py # Grid simulator + reward calc
│ └─ bench/ # Scenarios + outputs (jsonl)
├─ evals/
│ └─ run_planner_prompt_bench.py # Bench runner (scenarios × planner prompts)
├─ multi_agent_system.py # LangGraph pipeline (includes Critique + Mutator)
└─ README.md # ← this file

Quick Start

Requires Python 3.10+, install dependencies via pip install -r requirements.txt
(Optional) Set WANDB_API_KEY for Weave tracing.

# 1. Install dependencies
pip install -r requirements.txt

# 2. (Optional) enable tracing
export WANDB_API_KEY=YOUR_KEY

# 3. Run one optimization loop
python multi_agent_system.py

What you’ll see

  • Analyst/Planner/Actuator/Critique logs in console.
  • Reward breakdown (deficit, cost $, emissions, risk, fairness).
  • If approved/revised: Mutator applies a prompt nudge and loops (max_turns / max_iters bounded).

Run the Bench (scenarios × prompt variants)

python evals/run_planner_prompt_bench.py

Outputs saved to

  • data/bench/e2e_runs.jsonl — end-to-end runs (analysis → plan → act → critique → mutate), with:

    • analysis_text, plan_text, actions_parsed
    • reward_data { cost_usd, deficit_before_mw, deficit_after_mw, emissions_kg_co2, risk_*, fairness_* }
    • critique_text, decision
    • history per iteration: prompt_before, prompt_after, system_prompts_snapshot, metrics, bandit policy

  • data/bench/planner_runs.jsonl — (if used) planner-only prompt variant sweeps.

Tip: If you see AttributeError: 'int' object has no attribute 'strip', ensure planner prompt overrides from YAML are strings, and defensively cast overrides to str(...) before .strip().

Challenges

  • PPO infra mismatch: Weave didn’t fit our original PPO stack; we pivoted to GRPO on OpenPipe serverless RL, which integrated cleanly with tracing.
  • Planner output drift vs parser strictness: LLMs like JSON/prose; actuators need strict block format. We tightened the planner contract and hardened parsing (verb/key normalization, numeric coercion).
  • Type brittleness (YAML/bench): Non-string prompt overrides caused crashes; fixed via defensive casting + YAML hygiene.
  • Tradeoff management: Early plans slashed deficits but cost more; Critique+Mutator now steer toward cheaper follow-ups.

Accomplishments

  • Agents × RL, together: practical loop where learned policy (GRPO) and prompt evolution (ART-style) reinforce each other.
  • Self-evolving prompts: measurable before/after planner system prompt tied directly to cost/deficit improvements.
  • End-to-end telemetry: full traceability—plan, actions, reward, critique decision, and prompt snapshots—per iteration.

What We Learned

  • Training and wiring GRPO inside a multi-agent, tool-using workflow.
  • Deep Weave tracing for multi-agent systems (turning black-box behavior into inspectable timelines).
  • Practical Tavily usage during analysis prototyping.
  • The importance of format contracts when converting LLM output into real actuator commands.

What’s Next for SynErgi

  • Real-time integration: streaming telemetry & SCADA adapters.
  • Constraint-aware planning: MILP/CP or a verifier tool the Planner must pass before Actuator.
  • Richer rewards: dynamic fuel/DR pricing, emissions pricing, customer impact.
  • Human-in-the-loop ops: one-click approvals, rollbacks, and cost vs risk diffs.
  • Production hardening: containerized micro-agents, tool authN/Z, and CI-gated evaluation benches.
  • Tighter GRPO ↔️ Mutator coupling: use policy gradients to target prompt mutations, not just nudge them.

Reproducing Key Results (cheat-sheet)

# Single run (see one full loop and reward):
python multi_agent_system.py

# Batch bench (see scenario × prompt variants, with mutations & cost deltas):
python evals/run_planner_prompt_bench.py

# Inspect results (unix examples):
tail -n +1 data/bench/*.jsonl | less
jq '.' data/bench/e2e_runs.jsonl | less

Look for entries with:

  • deficit_before_mw → deficit_after_mw (big ↓ is good)
  • cost_usd (trending ↓ across iterations is good)
  • decision: "approve" vs "revise_*"
  • history[].prompt_before vs history[].prompt_after
  • history[].system_prompts_snapshot to audit the full agent context

Credits

Built with LangGraph, LangChain, Weave, OpenPipe (Serverless RL), GRPO, and a lot of grid-nerd love.

Built With

  • agents
  • apis
  • art
  • asyncio
  • built-with-what-languages
  • cloud-services
  • databases
  • digital-twin
  • frameworks
  • grpo
  • json
  • langchain
  • langgraph
  • multi-agent
  • openai
  • openpipe
  • pathlib
  • platforms
  • pydantic
  • python
  • python-dotenv
  • qwen
  • qwen2.5-14b-instruct
  • reward
  • rl
  • rl-engine
  • w&b
  • wandb
  • weave
  • weights
Share this project:

Updates

Submission history