BabyChain

Design image-and-video model chains on a visual canvas, then run the exact same workflow as a durable API on Vercel. Every step persists in AWS Aurora, so long media runs survive serverless limits.

Comment

Inspiration

The first sentence was the architecture: every output becomes the next input.

That is how real generative media work behaves. A useful workflow is rarely one model call. It is a generated image becoming the conditioning frame for an image-to-video model, sometimes with a refine step in the middle and a video-to-video modification at the end. The creative object is not the single prompt; it is the chain.

Tools like ComfyUI made that node-by-node mental model feel natural, and Comfy Cloud now exposes an API for submitting workflow JSON, tracking async jobs, and retrieving outputs. The gap BabyChain addresses is different: not "can a graph run remotely?", but "can a product expose one normalized image/video chain contract that is authenticated, retryable, resumable, callback-ready, and durable across serverless invocations?" In our own product work, we kept rewriting visual workflows as backend glue because the graph format, provider credentials, run state, callbacks, and product API contract lived in different places.

BabyChain closes that gap by making the canvas and API one product surface. A creator designs a chain visually; a developer calls the same chain through POST /api/v1/chains/runs. Every run, ordered step, checkpoint, output, failure, and callback state is persisted in AWS Aurora, so Vercel functions can stay stateless and disappear between provider calls without losing the workflow.

What it does

BabyChain is a self-hosted visual studio, Agentic Workflow planner, and durable HTTP API for image and video model chains. The studio runs on Vercel, durable state lives in AWS Aurora, and both the UI and public API share one schema and execution contract.

  • Multi-flow canvas studio. Compose image → refine → image-to-video → video-to-video chains across 54 public BabyChain model entries from Alibaba Cloud, Black Forest Labs, BytePlus, Google, OpenAI, and Runway, generating 58,752 valid chain templates. Many independent flows can live side by side on one workspace, with autosave, duplication, a flow-level move handle, run controls, and a saved results Library.
  • Three chain runners. Run a flow in Self Control when the human writes every prompt and parameter; Agentic Copilot when Amazon Nova proposes the next step for review; or Agentic Autopilot when Amazon Nova plans and BabyChain applies each downstream step automatically.
  • Schema-true cards and API validation. Every node card is generated from the selected model's normalized generation_* schema: fields, enum options, defaults, ranges, required flags, placeholders, and model role constraints. The UI and API fail for the same reasons, at the same paths, before provider credits are spent.
  • Durable API contract. POST /api/v1/chains/runs creates the same run the canvas uses. GET /api/v1/chains/get/{runId} advances or reads the durable run. POST /api/v1/chains/continue/{runId} approves Copilot checkpoints. POST /api/v1/chains/cancel/{runId} cancels the run and attempts provider cancellation.
  • AWS Aurora-backed execution memory. Runs, ordered steps, agent checkpoints, canvases, API keys, audit events, callback deliveries, and BabySea webhook deliveries live in the private babychain_private schema. The browser can close, the Vercel invocation can end, and a later poll or cron pass can still resume the chain.
  • Optional durable media storage. Completed outputs can be copied into AWS S3 (with CloudFront/custom domain support) or Vercel Blob before the step is marked succeeded, so API responses, canvas previews, downstream handoffs, and Agentic Workflow checkpoints can use stable media URLs instead of short-lived provider links.

The important schema detail: both execution modes use the generation_* contract. In BabySea mode, the BabySea SDK already exposes normalized generation_* inputs. In BYOK mode, BabyChain talks directly to provider APIs, but still presents the same normalized generation_* contract to the canvas and API. That is why we built Semantic Lady as a separate local SDK: it resolves model metadata, provider model ids, UI names, workflow roles, field definitions, enum values, defaults, and constraints without storing credentials or submitting generations. BabyChain uses Semantic Lady for schema discovery and validation, then provider adapters translate the normalized input into provider-specific payloads.

How we built it

We built BabyChain with Next.js App Router, React Flow, Vercel, AWS Aurora Serverless v2, Semantic Lady, Amazon Nova (Amazon Bedrock), optional AWS S3 / Vercel Blob storage, and provider adapters for Alibaba Cloud, Black Forest Labs, BytePlus, Google, OpenAI, Runway, and BabySea.

The durable store is a private AWS Aurora schema, babychain_private, with eight tables:

Table Owns
chain_run Run lifecycle, idempotency, terminal output, error state, callback intent
chain_step Ordered step params, provider ids, output files, timing, and failure details
chain_agent_checkpoint Copilot/Autopilot suggestions, selected prompts, validation state, token usage
canvas Owner-scoped workspace graphs and saved Library cards as jsonb
api_key Hashed caller API keys and scopes
audit_event Append-only operational trail
callback_delivery Signed terminal callback attempts and bounded retry state
babysea_webhook_delivery Inbound BabySea webhook bookkeeping

Why AWS Aurora, and not DynamoDB or Aurora DSQL: a chain run is a small, correctness-critical relational transaction, not a high-throughput key-value access pattern. Creating a run inserts one chain_run and its ordered chain_step rows in a single Aurora transaction, so a half-written chain can never exist; ON DELETE CASCADE foreign keys keep steps, checkpoints, and callbacks consistent; a partial unique index enforces caller-scoped idempotency in the database rather than in application code; and jsonb holds the flexible canvas graph and provider payloads beside that relational spine. The durable runner also leans on PostgreSQL primitives the other two designated engines do not give us as directly, SELECT … FOR UPDATE SKIP LOCKED run-leasing and guarded UPDATE … WHERE status = … single-winner transitions, while Aurora Serverless v2 scales to a low floor as the self-hosted control plane idles between bursts. We did not need single-digit-millisecond key-value reads; we needed transactions, row-level leasing, and referential integrity. That is the relational case for Aurora.

The runner is deliberately boring. Each invocation advances at most one action: submit a provider step, poll a provider step, apply an approved Copilot checkpoint, finalize a run, or deliver the terminal callback. A run can be advanced by create-run, by GET /api/v1/chains/get/{runId}, by the recovery cron, or by an inbound BabySea webhook. No function instance has to live for the whole chain.

Concurrency is handled as part of the product, not as a best-effort background task. Idempotency-Key hashes are unique per principal so client retries replay the same stored run. Provider submit keys are deterministic per run, step, and chain version. Active runs are claimed with FOR UPDATE SKIP LOCKED so overlapping pollers do not double-spend. The watchdog cancels and fails a step that never reaches a terminal provider state. Callback delivery is attempt-capped. Audit/callback/webhook histories are pruned on retention windows. The database pool sets statement_timeout and idle_in_transaction_session_timeout so a stuck query cannot pin a connection forever.

Outbound calls are treated as a trust boundary. Caller webhook URLs and any caller-supplied media URLs must be HTTPS without embedded credentials, and are DNS-resolved and rejected if they point at loopback, private, link-local, or cloud-metadata addresses, so a run cannot be turned into a request against internal services. Terminal callbacks are HMAC-signed (X-BabyChain-Signature) and inbound BabySea webhooks are signature-verified and recorded idempotently before they touch a run. API keys are stored hashed with explicit scopes, every state change appends an audit_event, and server and browser errors flow to Sentry. None of this is novel on its own; it is the baseline a self-hosted control plane needs before it spends real provider credits.

The Agentic Workflow is not a sidecar bolted onto the UI. It is another execution mode on the same durable runner. Amazon Nova (Amazon Bedrock) is called through the Converse API. The creative pass uses Amazon Nova reasoning mode with a schema-grounded prompt; the repair pass runs greedily with reasoning off when the first response is malformed or fails validation. Every proposed downstream step is completed against Semantic Lady defaults, validated against the model's actual generation_* fields, persisted as a checkpoint, and then submitted like any other step. Provider-native prompt enhancement is pinned off by default for planner-authored steps so the prompt the planner wrote is the prompt that runs.

The current gate is 357 tests plus typecheck, lint, formatting, package checks, deployment preflight scripts, and production builds. The tests cover templates, model catalog generation, provider adapters, Semantic Lady schema translation, runner transitions, Agentic Workflow validation, AWS Aurora migrations, API auth, callback behavior, storage, network safety, video trimming, and error guidance.

Challenges we ran into

Serverless time limits vs. minutes-long video steps. The tempting solution is to stretch timeouts. The durable solution is to never wait for a whole video chain inside one invocation. BabyChain persists state to AWS Aurora after each action, then lets the next request, webhook, or cron tick continue the run.

Two execution worlds, one request vocabulary. BabySea mode already gives a normalized generation_* contract. BYOK mode does not. Alibaba Cloud, BFL, BytePlus, Google, OpenAI, and Runway all name fields, ratios, media inputs, defaults, and output formats differently. Semantic Lady became the missing layer: a local schema SDK that lets BabyChain present one schema vocabulary while adapters handle provider payload translation.

Provider APIs disagree about sizes and roles. The same "16:9 image" can be a string enum, a width/height pair, a provider-specific size token, or a pixel-budget calculation. The worst live case was Alibaba Cloud: qwen-image accepts a small snapped-size set; z-image-turbo has dimension caps; Wan families enforce per-model pixel budgets. For budgeted models, we fit the requested ratio (w:h) into a model-specific P_max:

$$s = \sqrt{\frac{P_{\max}}{w h}}, \qquad (W, H) = \Big(16\Big\lfloor\tfrac{s w}{16}\Big\rfloor,\; 16\Big\lfloor\tfrac{s h}{16}\Big\rfloor\Big)$$

Models that allow no freedom get explicit snapped tables. Similar work went into Runway endpoint ratios, Google duration enums, BFL output expiry, and OpenAI quota errors that look like transient rate limits but are not.

Agentic does not excuse non-determinism. A planner that can write prompts is useful only if it is inspectable. We had to make Amazon Nova suggestions schema-grounded, persist every checkpoint, force planner outputs through the same validator as human-authored inputs, disable silent provider prompt rewrites by default, and repair malformed JSON without losing the creative plan.

Fail-fast chains and cancellation. When a step fails, downstream queued steps can never receive their input. They must be skipped server-side immediately, the run must go terminal, and the canvas must surface the provider's real error. Cancellation also has to propagate outward: the canvas Cancel button, cancel API, and watchdog all attempt provider cancellation so a locally canceled run does not keep spending at the provider.

Autosave that actually survives the browser. Debounced autosave silently dropped the last burst of edits before reload. We replaced it with a dirty flag, a steady flush loop, and a sendBeacon final flush on tab close. AWS Aurora wake-ups are handled with a 30s connection timeout and explicit TLS handling for RDS URLs.

Accomplishments that we're proud of

  • One AWS Aurora-backed contract powering the studio, public API, Library, callbacks, and Agentic Workflow checkpoints.
  • A visual canvas that is not a second implementation: it submits the same POST /api/v1/chains/runs payload product backends use.
  • 54 public model entries across six providers, producing 58,752 valid chain templates without letting invalid role combinations leak into the UI.
  • Semantic Lady extracted as its own SDK instead of being buried inside BabyChain, making BYOK schema normalization reusable outside this starter.
  • Agentic Copilot and Autopilot implemented as durable workflow modes, not UI-only prompt helpers.
  • Provider-safe cancellation, idempotent run creation, deterministic per-step submit keys, signed callbacks, output archival, audit trails, and bounded recovery behavior in an inspectable starter.
  • A 357-test gate that let us keep changing live-provider behavior during the submission period without losing confidence.
  • Multiple deployment paths around the same runtime model: Vercel, AWS CloudFormation, AWS EC2, Coolify, Docker, Fly.io, and Google Cloud Run.

What we learned

  • Statelessness is a design choice. Once every fact about a run lives in AWS Aurora, serverless limits and instance churn become normal operating conditions instead of existential threats.
  • A schema is a product boundary. Hand-built forms drift. A durable API and a canvas editor need one field contract, not two copies of the same idea.
  • Both modes can share generation_*, but only if BYOK has a schema brain. BabySea mode gets that vocabulary from the BabySea SDK. BYOK mode needed Semantic Lady because direct provider APIs are not naturally normalized.
  • Agentic workflows need receipts. If an agent writes the next step, the system must store what it suggested, what was selected, what schema it saw, what validation happened, and which output resulted.
  • Provider docs are a starting point; the live API is the truth. Size rules, quota behavior, expiry windows, duration enums, and content-filter shapes all had to be verified against real provider responses.

What's next for BabyChain

  • Branching chains: one source image feeding several video treatments inside the same durable run.
  • Team workspaces: multi-user accounts, scoped API keys, and quotas on top of the existing api_key model.
  • Run economics: estimated and actual provider spend from data already stored in AWS Aurora.
  • Deeper agency: let the planner choose topology, not only downstream prompts, while keeping the same schema validation and checkpoint discipline.

Built With

Share this project:

Updates