Takegraph

TAKEGRAPH is a self-healing build system for generative media that regenerates only what changed, recovers from failures, and ships every asset with verifiable lineage.

Comment

Inspiration

Four words can invalidate an entire generative-media production.

A sixteen-second product film may contain eighteen connected assets: a creative brief, product reference, shot plan, four keyframes, four video clips, marketing copy, narration, music, an end card, a poster, and the final composed master.

Now imagine that the legal team changes four words the day before launch.

The honest answer to:

“What needs to be regenerated?”

is four assets.

But most generative pipelines regenerate all eighteen.

Not because the other work is wrong, but because the system does not understand what the change actually affected.

Software engineering solved this problem decades ago. When one source file changes, a compiler does not rebuild an entire project. It understands the dependency graph and rebuilds only what is invalid.

Generative-media pipelines do not have that intelligence.

They repeat expensive provider calls, waste time when deadlines are closest, and discard outputs that a human already reviewed and approved.

The deeper problem is trust.

Providers time out. Models return malformed responses. Workers die midway through jobs. Teams often cannot tell whether a failed request was billed, which attempt produced the final asset, or whether the published file is still the exact file that passed review.

When someone asks:

“Which model produced this frame, what inputs did it use, and did it pass review?”

the answer should not live in a Slack thread.

That was the problem I wanted to solve.

Then, while preparing the demo, I discovered something that made the problem personal.

A one-word legal change rebuilt 16 of 18 nodes instead of four.

More than 470 tests were passing. No exception was thrown. The interface looked healthy.

But the central promise of the product was silently false.

The worker and the impact engine were using two different—but individually reasonable—definitions of a source hash. Because those hashes could never match, valid work could never satisfy the reuse proof.

That moment changed the project.

TAKEGRAPH could not merely claim that an asset was reusable, recovered, reviewed, or verified. Every important claim had to be independently demonstrable.

TAKEGRAPH became a build system for generative media: rebuild only what changed, recover without a human, and prove every byte.

What it does

TAKEGRAPH represents a production as a dependency graph.

Each node describes one piece of work: its recipe, provider, model, parameters, inputs, outputs, review state, and recovery history.

Every output is content-addressed in Backblaze B2 and fingerprinted from:

  1. its own canonicalized recipe; and
  2. the selected output hashes of its upstream dependencies.

This allows TAKEGRAPH to determine exactly which work remains valid after a change.

One edit, four rebuilds

I changed one line of legal copy in a real eighteen-node production.

TAKEGRAPH produced this impact plan:

>>> 4 REBUILD / 14 REUSE   (2 provider calls) <<<

rebuild  copy.pack                  NODE_SPEC_CHANGED
rebuild  audio.narration            UPSTREAM_FINGERPRINT_CHANGED
rebuild  graphic.end_card           UPSTREAM_FINGERPRINT_CHANGED
rebuild  compose.delivery_package   UPSTREAM_FINGERPRINT_CHANGED

The result:

  • 4 nodes rebuilt
  • 14 nodes reused
  • 2 provider calls made instead of 12
  • 10 provider calls avoided
  • Every decision accompanied by an auditable reason code

TAKEGRAPH does not simply display a green “cached” badge.

For a node to be reused, the system proves that its specification is unchanged, its upstream fingerprints still match, its selected output remains valid, and the stored bytes still hash to the expected value.

A rebuild is never unexplained, and reuse is never assumed.

It heals itself

Generative providers fail for many different reasons, and not every failure should be treated the same way.

TAKEGRAPH uses typed provider error classifications to decide whether it should:

  • retry the same provider;
  • switch to a fallback provider;
  • request a bounded retake;
  • wait for human intervention; or
  • stop permanently.

To demonstrate this, I injected a provider timeout into a video-generation node during a real build.

Attempt Recovery mechanism Model Outcome
1 PRIMARY pixverse-v6-i2v FAILED — TEST_FAULT_PROVIDER_TIMEOUT, class TRANSIENT
2 SAME_PROVIDER_RETRY pixverse-v6-i2v SUCCEEDED — linked through parent_attempt_id

The node completed with a PASSED state and the reason:

“Transient failure; retrying the same model (1 of 2).”

The complete eighteen-node build finished in 13 minutes and 1 second without human intervention.

Both attempts remain permanently visible in the node’s lineage.

The injected failure is also impossible to confuse with a real provider incident. Fault injection requires both an environment-level flag and a demo-scoped project. Every rule is single-use, expires automatically, and renders a visible red TEST FAULT badge.

Every published release is independently verifiable

A TAKEGRAPH release contains the SHA-256 hash of every asset it ships.

When someone clicks Verify, the system does not trust a status stored in PostgreSQL.

It downloads every released object from B2 again, recomputes its hash from the actual bytes, and compares that result with the published manifest.

In the demonstrated release:

  • 8 assets were independently re-downloaded
  • 8 hashes matched
  • verification completed in 6.7 seconds

The verification endpoint is deliberately available to guests.

A verification claim is not meaningful if only the person who created the project is allowed to test it.

How we built it

Backblaze B2 is part of the algorithm

Backblaze B2 is not simply the location where TAKEGRAPH uploads finished files.

It is the mechanism that makes reliable incremental rebuilding possible.

Every asset is stored under a content-addressed key:

tenants/{org}/cas/sha256/aa/bb/{hash}.{ext}

The integration uses genblaze_s3.S3StorageBackend.

Because the key is derived from the asset’s content hash, TAKEGRAPH gains three essential properties.

Automatic deduplication

Two builds that produce identical bytes converge on the same object.

This makes “14 nodes reused” a real storage and compute saving rather than a database label.

Provable reuse

The reuse proof does not trust PostgreSQL merely because a record says an asset exists.

TAKEGRAPH reads the object back from B2 and hashes the bytes again before accepting the asset as reusable.

Third-party release verification

A release manifest is a collection of expected hashes.

Anyone with access to the verification endpoint can retrieve the release objects and independently reproduce the verdict.

Deeper B2 integration

The B2 integration extends beyond basic upload and download operations.

  • Separate work and release buckets: Temporary build artifacts and published releases use different buckets with separate least-privilege application keys.
  • Permission-boundary testing: An automated test confirms that the work-bucket key is rejected when used against the release bucket.
  • No dangerous stored master key: The master key is used once during setup. scripts/b2_setup.py raises an error rather than writing any key with bypassGovernance.
  • HMAC-signed event notifications: Upload events feed a reconciler, allowing the system to discover and index an upload even if a worker was offline when it completed.
  • Short-lived presigned URLs: PUT and GET URLs are created only after authorization and are never persisted.
  • Honest Object Lock reporting: Retention configuration is read directly from B2 and displayed as NOT_CONFIGURED when it has not been enabled.

Genblaze powers the generation layer

TAKEGRAPH uses genblaze_core for the fundamental generation abstractions:

  • Pipeline
  • Run
  • RunBuilder
  • Step
  • Manifest
  • Asset
  • Modality
  • RunStatus
  • StepStatus
  • observability events
  • ObjectStorageSink
  • KeyStrategy

ObjectStorageSink and KeyStrategy allow TAKEGRAPH’s content-addressed storage scheme to connect directly to B2.

Provider adapters include:

  • GMICloudImageProvider
  • GMICloudVideoProvider
  • ElevenLabsTTSProvider

The recovery system depends on genblaze_core.ProviderErrorCode.

Retry decisions are based on typed error classes rather than brittle string matching against provider error messages.

The eighteen-node ORBIT production

The demonstrated ORBIT build spans five provider and processing categories.

Node type Count Provider or processor
SOURCE_TEXT / SOURCE_IMAGE 2 Resolved from the project revision
STRUCTURED_PLAN / STRUCTURED_TEXT 2 Anthropic claude-sonnet-4-6
IMAGE_GENERATION 4 GMI Cloud seedream-5.0-pro
VIDEO_GENERATION 4 GMI Cloud pixverse-v6-i2v
AUDIO_GENERATION 2 ElevenLabs eleven_multilingual_v2 and music_v2
Transformation and composition 4 Local Pillow and FFmpeg

System architecture

The domain layer is a pure package with zero network or database I/O.

It contains:

  • canonicalization;
  • content fingerprints;
  • the reuse proof;
  • the impact engine;
  • five state machines; and
  • the recovery ladder.

Keeping these decisions inside a pure domain package makes the core logic testable without provider credentials, object storage, or a running database.

The surrounding system includes:

  • FastAPI for transport and persistence;
  • a separate worker service, which is the only process allowed to call providers or write generated bytes;
  • PostgreSQL 16 as a durable work queue using FOR UPDATE SKIP LOCKED, leases, and heartbeats;
  • Next.js 16, React 19, and Tailwind CSS v4 for the production workspace;
  • Pillow and FFmpeg for local transformation and composition.

Fingerprints are canonicalized using JCS, RFC 8785, and hashed with SHA-256.

The domain implements canonicalization locally to preserve its zero-I/O boundary, while a conformance test confirms that its output matches genblaze_core.canonical_json byte for byte.

Challenges we ran into

The bug that silently disproved the product

The worker stored a source node’s output using the JCS hash of:

{"brief_text": "..."}

The impact engine expected the hash of the whitespace-normalized text itself.

Both definitions were reasonable. Both were independently tested.

But they could never produce the same value.

As a result, every resolved source node reported CACHE_ASSET_MISSING, invalidating its direct dependents and everything downstream from them.

A one-word legal change rebuilt sixteen of eighteen nodes.

The system did not crash. More than 470 tests continued to pass.

I found the defect only because I tested the product’s headline claim against a live build before recording the demonstration.

The fingerprint definition now exists in one shared function, and a conformance test ensures every caller uses the same contract.

This was the most important lesson of the project:

A system can be confidently wrong while every visible indicator remains green.

One recovery path worked; four did not

The GMI handler had a correct recovery branch, but four other handlers did not.

A failed Anthropic or ElevenLabs node could be scheduled for recovery and then terminate with:

attempt is in unsupported state

No handler correctly understood a human-requested retake.

The reason was simple: no node had ever reached a second attempt in the test suite.

Investigating that single untested region exposed several related defects:

  • recovery was missing in four of five handlers;
  • retakes were unsupported in all five;
  • a repeated retry could reuse an idempotency key;
  • the retry denylist behaved like an allowlist; and
  • a provider-specific prompt limit was incorrect.

Untested paths rarely fail alone. They cluster.

The provider’s real limit differed from our assumption

The music-generation prompt originally embedded the complete shot-plan JSON and was guarded at 5,000 characters.

ElevenLabs rejected the request at 4,100 characters.

A 4,551-character prompt passed TAKEGRAPH’s validation but returned a terminal 422 response from the provider.

The build could not recover because retries cannot fix an invalid request.

The prompt also contained camera bodies and focal lengths—details a music model could not meaningfully use.

The new music prompt contains only:

  • the creative brief;
  • target duration; and
  • the relevant shot beats.

It was reduced to approximately 410 characters.

A correct dashboard looked broken

The storyboard originally loaded full-resolution originals into small preview tiles.

Some assets were as large as 1.6 MB.

The result was:

  • approximately fourteen seconds before all previews appeared;
  • eighteen B2 reads on each page view;
  • roughly 6 MB transferred; and
  • broken-image icons whenever B2 returned an XML error response after a request limit was reached.

The fix was a content-addressed WebP poster cache.

The first poster request for an asset performs one B2 read. Every later request uses the local cache.

Performance improved from:

14 seconds → 1.16 seconds
18 requests → 2 requests
~6 MB → ~174 KB

Video posters are extracted slightly after the opening frame because several generated clips begin nearly black. Using frame zero made valid videos look like failed outputs.

Sequential verification appeared to be a server failure

Release verification originally checked eight assets one after another.

On a degraded connection, the combined network round trips took 78 seconds.

The reverse proxy disconnected before the request finished, so the browser displayed an internal-server error even though the API was still correctly verifying the release.

The checks are independent; only the final verdict needs to be combined.

Running them concurrently reduced verification time from:

78 seconds → 6.7 seconds

Tests were touching the development database

The test suite could fall back to the development DATABASE_URL.

Tests truncated tables between cases, wiping active demonstration builds and producing inconsistent failures based on leftover state.

The failing tests appeared random because the underlying problem was shared state.

One failure could permanently strand a project

Impact preview requires a successful baseline for the selected base revision.

A failed build could leave the current revision without a valid baseline, while attempting to rebuild from an older revision was rejected as stale.

Together, those safeguards created a state from which the project could never build again.

The recovery flow was redesigned so safety checks cannot trap the project permanently.

Accomplishments that we are proud of

The central claim is measured, not asserted

On a real production using real providers:

  • 4 nodes rebuilt
  • 14 nodes reused
  • 2 provider calls were made instead of 12
  • 10 provider calls were avoided
  • every node received an auditable reason code

Self-healing works end to end

The product records both the failed and successful attempts, links them through lineage, classifies the failure, explains the recovery decision, and completes the build without human intervention.

Fault injection cannot misrepresent reality

A simulated failure requires two independent authorization conditions.

Every injected rule is single-use, expires automatically, and is visibly marked as TEST FAULT.

A failure created by TAKEGRAPH cannot be presented as a genuine provider outage.

Published releases can be verified by guests

The verification result is computed from the bytes stored in B2 rather than a database status.

The proof remains useful even outside the account that created the production.

The core system is extensively tested

TAKEGRAPH includes 485 tests.

The core domain contains no I/O and defines five state machines through explicit transition tables.

The missing transitions are as important as the valid ones.

For example, there is no direct:

RUNNING → PASSED

A node cannot skip storage, hashing, and verification before becoming successful.

The demo uses the real product

The demonstration is not a mockup or a manually animated prototype.

Puppeteer controls the live application, ElevenLabs generates the narration, and FFmpeg assembles the final video.

What we learned

Content addressing is a correctness feature

I originally treated B2 as the place where generated files would be stored.

Building the reuse proof changed that understanding.

When object keys are content hashes, these questions become equivalent:

“Is this the same work?”

and:

“Are these the same bytes?”

That relationship makes deduplication, reuse validation, release integrity, and third-party verification part of one consistent model.

B2 became load-bearing for TAKEGRAPH’s central algorithm rather than a replaceable storage detail.

Passing tests do not automatically prove the product’s promise

The source-hash defect demonstrated that two individually tested components can still disagree at their boundary.

The tests proved that each hash function behaved consistently.

They did not prove that the worker and impact engine meant the same thing when they used the word “fingerprint.”

The definition now exists once, and every headline claim is checked against a live system before it is presented.

Untested paths form clusters

No node had ever reached a second attempt.

Therefore almost every feature depending on second attempts was broken at the same time:

  • recovery in four handlers;
  • retakes in all handlers;
  • retry idempotency;
  • retry classification; and
  • fallback behavior.

Finding one defect in an untested region should increase suspicion about the entire region.

The installed SDK is the source of truth

The actual Genblaze modules are:

  • genblaze_core
  • genblaze_s3
  • genblaze_gmicloud

genblaze_core exposes no top-level exports.

Inspecting the installed packages early prevented the architecture from being built around incorrect assumptions.

The same lesson applied to ElevenLabs’ real prompt limit.

Honest unknowns are better than impressive guesses

TAKEGRAPH does not fabricate numbers to make the interface appear complete.

When pricing information is unavailable, the estimate is displayed as UNKNOWN.

When Object Lock is not configured, the release reports NOT_CONFIGURED.

The REAL BASELINE badge appears only when the baseline actually ran against real providers.

Trust is not created by making every field look successful.

It is created by ensuring every field means exactly what it says.

What’s next for TAKEGRAPH

Production deployment

TAKEGRAPH currently runs locally against real providers and real Backblaze B2 infrastructure.

The frontend is Vercel-ready. The API and worker require deployment to infrastructure that supports long-running processes.

End-to-end cross-provider fallback

The recovery ladder and fallback logic are implemented and unit-tested.

However, the injected transient failure recovered during the same-provider retry stage, so a real cross-vendor failover has not yet completed end to end.

TAKEGRAPH will not claim that demonstration until it has happened.

Pricing-aware impact previews

The impact engine should translate dependency changes into the number a producer ultimately needs:

This revision costs $0.42 instead of $4.90.

This requires reliable ingestion of provider pricing data.

Until then, the system reports pricing as unknown rather than presenting an unsupported estimate.

Agent-assisted repair

When a quality gate fails, TAKEGRAPH should propose a small, bounded parameter adjustment for human approval instead of blindly regenerating the same request.

Object Lock for published releases

The release bucket will use Backblaze Object Lock for genuine retention guarantees, with the active retention policy surfaced directly on the public proof page.

Partial-node reuse

Today, if one shot inside a four-shot plan changes, the complete shot-plan node may invalidate all four clips.

The next stage is finer-grained dependencies, allowing TAKEGRAPH to rebuild only the specific shot that changed.

Closing

Generative media does not need another interface that hides a collection of provider calls behind a progress bar.

It needs infrastructure that understands what changed, preserves what remains valid, survives failure, and proves what was ultimately published.

TAKEGRAPH brings software-style build intelligence to generative production.

Change one sentence. Rebuild four assets. Reuse fourteen. Recover from failure. Verify the release.

Rebuild only what changed. Recover without a human. Prove every byte.

Built With

Share this project:

Updates