IDEMARK

Replay-safe effect finality for event-driven production accounting.

Comment

Inspiration

IDEMARK came from a simple but dangerous distributed-systems failure: a Kafka message can be delivered again even after the business effect it triggered has already succeeded.

For production accounting, a replay is not just another message. It can become a duplicate settlement.

We wanted to solve the boundary between reliable event delivery and unreliable external effects—not by adding more agents, but by giving the business action a durable identity and making its final outcome independently verifiable.

What it does

IDEMARK makes event-driven settlement actions replay-safe.

It uses Gemini as a semantic sensor to extract accounting facts and source evidence from unstructured events. A deterministic kernel verifies the evidence, canonicalizes the facts, derives a stable action_id, and applies deterministic settlement logic.

Confluent Kafka handles the event and transactional intent boundary. IDEMARK's effect gateway handles business-effect finality with idempotency, execution fencing, payload-conflict detection, and UNKNOWN reconciliation.

When the same event is replayed, IDEMARK recognizes the same business action and returns the existing receipt instead of creating another committed effect.

How we built it

The system is deliberately split into deterministic and probabilistic layers.

  • Gemini + Google ADK — semantic extraction only
  • Provenance verifier — validates extracted facts against the original source
  • Canonicalizer — creates stable SHA-256 action identities
  • Settlement kernel — deterministic integer-based calculations
  • Confluent Kafka — real event streaming, consumer groups, transactions, and rebalance handling
  • IDEMARK Effect Gateway — idempotent execution, fencing, identity-conflict detection, and UNKNOWN reconciliation
  • IBM BOB — For audit, Logs and General verifications and checks
  • Independent verifier — audits persisted evidence without Gemini or live application state
  • Chaos harness — deliberately injects crashes, replays, timeouts, tampering, and rebalance conditions
  • Next.js + FastAPI — real-time proof console and API surface

We validated the mechanism using a three-arm comparison: naive processing, Kafka transactions alone, and full IDEMARK.

Challenges we ran into

The hardest challenge was not getting Kafka or Gemini working. It was defining exactly what each system could legitimately guarantee.

Kafka can provide strong transactional guarantees within its own boundary, but an external effect lives in another failure domain. That created the central crash window we had to reproduce and close.

We also had to handle ambiguous downstream outcomes safely. Instead of blindly retrying an uncertain operation, IDEMARK records UNKNOWN and requires reconciliation before reaching a final state.

Finally, we had to make the proof independently auditable rather than relying on dashboard output alone.

Accomplishments that we're proud of

We are most proud that IDEMARK is built around a discovered failure mechanism rather than a feature list.

In the latest real-infrastructure run:

  • 200 source events
  • 6 observed consumer-group rebalances
  • 37 redeliveries
  • 0 duplicate committed effects
  • 0 lost events
  • 4 ambiguous effects safely reconciled
  • 2 identity conflicts detected
  • 20/20 automated tests passing
  • Independent cryptographic verification: PASS

Most importantly, we deliberately broke the processing path after an effect succeeded and before the source offset was finalized, then verified that replaying the same business action did not create a second committed effect.

What we learned

We learned that reliable delivery and reliable execution are different problems.

A message ID is not a business identity. A Kafka offset is not an authorization to execute an effect. And a successful network response does not always mean the caller knows whether the external operation is committed.

The strongest systems therefore separate semantic identity, execution authority, effect finality, and verification.

We also learned that the most convincing technical demos are not the ones that claim a system cannot fail—they are the ones that deliberately break it and show what survives.

What's next for IDEMARK

IDEMARK can expand beyond studio royalty accounting into any event-driven workflow where replay can create a costly external effect: licensing, participation, settlements, payouts, fulfillment, and other operational commitments.

The next step is to turn the prototype's deterministic effect boundary into a reusable infrastructure layer with broader downstream adapters, stronger reconciliation contracts, richer audit tooling, and production-grade authorization and compliance controls.

Built With

  • confluent-cloud
  • confluent-kafka
  • confluent-kafka-python
  • docker
  • fastapi
  • google-adk
  • google-cloud
  • google-gemini
  • google-genai
  • ibm-bob
  • next.js-14
  • playwright
  • postgresql
  • pytest
  • python-3.12
  • remotion
  • render
  • sha-256
  • sqlite
  • tailwindcss
  • vercel
Share this project:

Updates

Submission history