Kizashi

A background agent that watches the public IRS record for the nonprofits you fund, and says nothing until one of them is about to lose its status

Comment

Inspiration

A nonprofit that misses its annual IRS filing three years in a row loses its tax exemption automatically, on the due date of the third return, with no hearing and no warning letter beforehand. The IRS posts the revocation months later. The organizations this happens to are the small all-volunteer ones with rotating treasurers, which is exactly who community foundations, United Ways and fiscal sponsors fund by the hundred. The funder finds out at regrant time, when the money is supposed to move. Congress has a bill to make the IRS send a warning. Nobody sends one today.

The date is computable from public data in year one, so the agent computes it.

What it does

Kizashi takes a portfolio of EINs, reads three public IRS bulk files (the Business Master File, the Form 990-N e-Postcard download, and the Automatic Revocation List), and classifies every organization: current, one period missed, two periods missed, already revoked, reinstated, or excluded for a named reason (group-ruling subordinate, church, terminated, not a 990-N filer). It writes one ledger row per organization with the reason it stayed quiet. For the few it surfaces, a Strands agent writes a brief and an outreach note from the evidence, and adds a review note only when the record itself suggests a person should check first. A second agent dispatches: send_alert when there is no note, hold_for_review when there is one. A deterministic hook gates send_alert: wrong class, wrong date, already alerted or dismissed, and the call is cancelled with the reason written to the gate feed. send_alert delivers through Amazon SES when three environment variables opt in and records a dry-run delivery id otherwise; the feed carries the channel and id either way.

The agent is scored twice against the IRS's own record. The IRS publishes every revocation it has already made with its effective date. Running the date formula backwards over the 152,249 revocations from 2021 to 2026 that have a 990-N postcard on record reproduces the IRS's own date exactly for 133,844 of them (87.9%), and the app says how much of the window that covers (155,946 revocations have no postcard and are not scored). Then the classifier is rerun as of September 2024 over 130,601 filers with every later revocation hidden from it: of the 135 it would have surfaced then, 105 were revoked in the two years since (77.8%), and the outcome of every other class is in the same table. Both numbers are computed by scripts in the repository and shown in the app.

How we built it

  • Strands Agents SDK: GraphBuilder with a custom MultiAgentBase node that runs the deterministic classifier and never calls a model, then a brief node (pydantic structured output, batched) and a dispatch node whose Agent owns the send_alert tool.
  • BeforeToolCallEvent hook as the gate: it reads the ledger class, the predicted date and the alert history, sets cancel_tool with a reason when the call is not allowed, records a hold_for_review call as a third decision, and logs every one. AfterToolCallEvent records sends with their SES message id or dry-run id.
  • Model: Amazon Bedrock via the Mantle endpoint, OpenAIModel(bedrock_mantle_config=...), Gemma 4 31B. Strands mints a short-term Bedrock API key per request from the AWS session.
  • Background: an EventBridge Scheduler schedule launches an ECS Fargate task on the 7th of each month that pulls the alert history from S3, fetches the three IRS files, runs the full pipeline with the model and publishes the report and the history back to S3. One scheduled launch completed end to end in under five minutes of container time. A BedrockAgentCoreApp entrypoint runs the same pipeline on /invocations locally; the hosted AgentCore step is blocked by an account quota.
  • Data: pure-Python loaders for the three IRS files, a classifier over dates, a backtest script, a silence scorer, a FastAPI API, and a React app (landing plus dashboard) deployed as static assets. 64 tests.

Challenges we ran into

  • The 990-N bulk file holds only the latest filing per organization, so "two consecutive years missing" is a column compare, but a 990-EZ filed later is invisible in it. The Business Master File's tax-period field is where that later return shows up, so the classifier takes the later of the two sources and records which one it used.
  • The revocation list's reinstatement column is only populated for some reinstated organizations; the rest are detectable only because they filed again after the revocation date. Reinstatement is inferred from either signal.
  • The backtest residual has two real spikes, +12 months and beyond -24 months, which turned into ledger reasons rather than being dropped.
  • Structured output from an open-weight model over 38 organizations needed batching, validation, one retry and a deterministic fallback path so a run always completes. In the committed run all 38 briefs came from the model, and it raised a review note on two of them, a PBA local and a base council, which the dispatcher held instead of sending.
  • Scoring the silence honestly meant naming the bias: the postcard file is latest-only, so an organization that filed again after a revocation looks current today. The score reports those separately instead of folding them into recall.

Accomplishments

  • A measured accuracy claim against published ground truth.
  • A gate that is exercised through the model, on camera, with allowed, held, cancelled and already-alerted outcomes in the same feed, each send carrying a delivery id.
  • A second score that tests the agent's judgement rather than the IRS's arithmetic: what happened, two years later, to everything it would and would not have surfaced.
  • A ledger that makes the silence legible: 848 of 886 organizations in the demo county produced nothing to send, each with a reason.

What we learned

What the agent does not send is the product, so every silence has to be checkable. Every organization the agent did not warn about is a row someone can check, and every alert it tried to send passed a hook that the model cannot talk its way around.

What's next

Turning KIZASHI_SES_SEND on for a real grants inbox, a national run across all four BMF regions, and a portfolio upload in the app so a funder can drop in its own grantee list.

Built With

Share this project:

Updates

Submission history