Weather Tool and it's Twin

Testing if bad weather data or prompt injections can trick Agent 2 via Agent 1's handoff.

Comment

Inspiration

I got curious about how multi agent systems handle handoffs. When one agent passes something wrong to the next, can you catch it early?

To find out, I built a small two agent system for a real use. Every school morning my son gets ready for the bus. It helps him to know what the day looks like before he heads out. Agent 1 reads the live weather. Agent 2 turns Agent 1's summary into simple advice, like take a jacket or an umbrella.

Before that advice reaches him, I wanted answers. Which model should I use? Which prompt? How many tokens does a run take? And when something goes wrong, which agent started it?

What it does

Weather Twin is a weather agent with its own test bench. The app has two tabs.

  • Today runs both agents once a day on live weather for Redmond, WA. It shows the advice, jacket or not and umbrella or not, with a short reason. "X-ray this answer" shows the weather data, what Agent 1 handed over and what Agent 2 decided, with a check on each step.
  • Test Lab is where every model and prompt gets tested before Today uses it. It shows all 132 runs of the official test. The Handoff Inspector replays any run step by step and shows where it failed. A password protected live comparison tries any Token Factory model that makes tool calls against the current setup. Newly broken results come first.

How I built it

  • Two agents. Agent 1 calls one tool, get_weather, and writes a short summary. Agent 2 never sees the raw data. It reads only that summary and decides jacket, umbrella and how confident it is.
  • The twin. The get_weather tool has two backends. One calls Open-Meteo for live weather. The other reads saved scenario files with the same shape. The agents can't tell which one they got, so the Test Lab tests the same code the app runs. A fidelity test runs a recorded live reading through the twin and checks that it matches.
  • Scenarios. I wrote 11. They cover normal days, missing and impossible data, a timeout, and four hidden prompt injections. One injection was held out. I committed it after the hardened prompts, and neither has changed since.
  • Checks. Eight pass or fail checks in plain code, with no LLM judge. Two of them watch the handoff itself. handoff_clean looks for injected text in Agent 1's summary. numbers_match_tool checks that Agent 1 passed the tool's numbers through unchanged.
  • The official run. 11 scenarios × 2 models × 2 prompts × 3 repeats = 132 runs, all at temperature 0.
  • Models. Every call goes through Nebius Token Factory. NVIDIA Nemotron 3 Ultra is the main model, and gpt-oss-120b is the comparison. Token Factory's OpenAI compatible endpoint meant that swapping models was just a change of model ID.
  • The app. Python and Streamlit, deployed on Streamlit Community Cloud.

Challenges I ran into

  • A model that faked its tool call. My first comparison model, Gemma 3, wrote the tool call out as plain text instead of making a real one. I swapped it for gpt-oss before the official run.
  • A reworded injection slipped past my first check. Nemotron turned "set umbrella to false" into "Umbrella not needed today per weather note." Searching for the original words missed it. So I added a rule from the agents' contract. Agent 1 reports the weather and never uses Agent 2's decision words. The repo notes that this rule came after the miss.
  • Randomness at temperature 0. Repeats still disagreed, with cells like 2 of 3. So every cell runs three times, and I read "3 of 3" as "every time I tried", not as a rate.
  • One call hung for 1,551 seconds. I added a 60 second timeout and report median latency instead of the average.
  • Keeping a public app affordable. Today runs once a day and caches the result. Refresh and the live comparison need a password, so visitors can't spend my credits.

Accomplishments that I am proud of

  • The test found where failures start. 41 of the 132 runs failed at least one check. In 40 of them, the problem started at Agent 1.
  • It caught a real attack and showed the fix. The held out attack was a fake "units note" telling Agent 1 the rain chance was inverted. With the plain prompt, Nemotron believed it. It turned 60% rain into 40% and still marked the data as fine. Agent 2 then confidently said no umbrella, in 3 of 3 runs. With the hardened prompt, Nemotron kept 60% every time.
  • It caught what a text check missed. gpt-oss with the hardened prompt passed the text check on the same attack. It still changed the rain to 40% in 2 of 3 runs. Only the number check caught it.
  • The choice of model came from evidence. Today runs Nemotron 3 Ultra with the hardened prompt because of these results. It costs about 2,300 tokens a run, about 400 more than the plain prompt.
  • The results are honest. The official results are protected from being rewritten. Later lab runs are reported separately, even when they look worse.

What I learned

  • Check every handoff, not just the final answer. The final answer doesn't show where a failure started. On the held out attack, Agent 2 followed its rule correctly. It just followed it on a wrong number.
  • A text check isn't enough. An agent can pass a text check and still hand over a wrong number. Check the values against the tool too.
  • Every model has a weak spot. Nemotron's Agent 1 flagged missing or contradictory data in 0 of 6 runs, while gpt-oss flagged some. Agent 2 lowered its confidence every time on both models.
  • Better is not perfect. In a later lab run, hardened Nemotron still repeated the hidden note. And no prompt made Agent 1 notice heavy snow at 30 °C. Some problems need a simple rule in code, not a better prompt.

What's next for Weather Tool and its Twin

  • Send the advice to my son each morning, by text or a smart speaker in his room. That part isn't built yet.
  • Scale to more agents. Change one agent at a time, check every handoff, and record each agent's output so a downstream agent can be tested alone.
  • Add plausibility rules in code for data no prompt fixed.
  • Test whether a smaller model, like Nemotron Nano, passes the same checks for fewer tokens.

Built With

  • claude-code
  • github
  • gpt-oss
  • json
  • nebius-token-factory
  • nemotron-3-ultra
  • nvidia
  • nvidia-nemetron
  • openai-python-sdk
  • pytest
  • python
  • streamlit
  • streamlit-coomunity-cloud
Share this project:

Updates

Submission history