MuJoCo Physics Simulation of Two VLA models
P2P Communication via learned representations [Excalidraw]
Manim animation of latent space transformations
SpacetimeDB Implementation [Excalidraw]

Robots are getting their own brains. Relay lets those brains work as one team.

Project repository • Dashboard demo • Learned messages • Shared-space experiment

Relay

Different robots. One shared task.

Relay connects robot models so they can share information, coordinate tasks and report progress in one workspace. We built and tested it in simulation, with a two-arm folding cell and a five-robot factory.

Inspiration

One robot can follow a plan but two robots need to agree on what is happening to get anything done.

That gets difficult when each robot runs a different AI model, sees a different part of the environment and receives information at a different time. Developers still have to build that connection when they have more than one robot as to who owns the next step, what each robot needs to communicate, how long an observation remains useful and what happens when a worker disconnects.

Different reasoning models also have different internal thoughts. The pooled features selected by our adaptors contain 960 components for SmolVLA and 1,024 for X-VLA, which are two open-source VLA models.

We started with a specific question: when one arm sees a hazard and the other cannot, can a learned message change the second arm's action?

Relay was a project that we built around a learned communication channel, a durable coordination backend and a dashboard that makes the entire process transparent.

What it does

Register models and robots. Describe the model configuration and runtime capabilities. Managers will approve exact configurations for use while employees work with approved releases.
Assign a shared objective. Create a task with a tangible objective and assign it to connected workers.
Choose a communication approach. Research workflows that support LLM mediation, orchestration, learned latent messages and hybrid communication.
Inspect the exchange. Follow messages, task revisions, reported outcomes, latency and attention notices through the dashboard as a whole.
Maintain the workspace automatically. SpacetimeDB runs connection checks, deadline checks, analytics aggregation and retention jobs.

Each worker owns its inference and execution handler. Qwen mediation and learned adapters are explicit integrations; selecting a dashboard mode does not automatically load an arbitrary model.

Our folding experiments give the arms different observations. An obstacle can be visible to one arm while the other's image and local measurements remain unchanged. That makes peer information necessary for the hidden-hazard decision.

How we built it

Model translation. Frozen SmolVLA and X-VLA backbones extract features from each arm's own image and instruction. Small learned senders encode messages, receivers interpret them, and replacement action heads combine peer context with local information.
Coordination infrastructure. SpacetimeDB stores the shared objectives, ordered action and feedback records, peer messages, resource reservations and runtime presence. TypeScript reducers validate updates and enforce coordination rules, while scheduled retention jobs clean up expired records.
Operational control. Relay Studio provides tasks, model approvals, scoped worker credentials, audit history, communication inspection and analytics. A Spacetime-backed deployment profile helps preserve workspace state across web-server restarts.
Simulation and evaluation. We built a deformable T-shirt workcell with two six-joint arms and a 271-vertex flex cloth, plus a 24 × 14 m factory with five robots and shared stations. Qwen3 runs locally through MLX for phase review in two of the communication pipelines.

Inside the learned communication channel

Each arm receives its own RGB observation and instruction. Its frozen backbone produces a feature vector, and a trainable sender maps that vector into a bounded 64-dimensional message.

ARM A → ARM B

A's own image + instruction
          |
          v
Frozen SmolVLA features: 960D
          |
          v
Sender A:
LayerNorm → Linear(960, 96) → GELU
          → Linear(96, 64) → tanh
          |
          v
64D learned message
          |
          v
SpacetimeDB
scope + identity + timing + committed packet
          |
          v
Receiver B:
Linear(64, 64) → GELU → Linear(64, 32) → tanh
          |
          v
32D peer context
          |
          +---- B's own features and measured state
          |
          v
B's replacement action head
          |
          v
Cartesian motion + gripper + coordinated hold


ARM B → ARM A

B's own image + instruction
          |
          v
Frozen X-VLA features: 1024D
          |
          v
Sender B: 1024 → 96 → 64
          |
          v
SpacetimeDB → Receiver A: 64 → 64 → 32
          |
          +---- A's own features and measured state
          |
          v
A's replacement action head

What reaches the action head?

Input	Dimensions	Purpose
Local feature projection	32	Visual and language context from the arm's own frozen backbone
Local measurements and phase	39	23 measured facts and a 16-dimensional phase encoding
Decoded peer context	32	Information extracted from the received learned message
Peer-fact slots	23	Mode-dependent structured information
Validity and age	2	Whether peer context is available and how old it is
Total	128	Input to the replacement action head

The action head maps 128 → 96 → 64 → 5. Its five outputs are three Cartesian motion values, a gripper logit and a coordinated-hold logit. Predicted displacement is bounded to 1.8 cm and constrained to the workcell.

When the required peer context is missing, the policy masks both the message and decoded context. An explicit availability guard holds the arm in place before communication can actually be established. Masking after decoding also prevents decoder biases from inventing information when no valid message exists.

The pairwise adapter/head bundle has 324,874 trainable parameters.

Four communication conditions

Condition	Information available from the peer
No communication	Peer context, facts, validity and age are zeroed. A separate checkpoint learns the baseline.
Structured text	Canonical JSON containing 23 measured peer facts, parsed into the fact slots.
Embeddings	A learned 64D message derived from the peer's RGB and language features.
Hybrid	The learned message plus 11 exact safety fields.

These conditions use different information sources, so the comparison evaluates communication mechanisms rather than only encodings of identical inputs. Qwen does not provide a hidden peer-information route in the controlled latent diagnostic.

Adapter implementation • Training and evaluation contract

Training a shared latent space

A shared message width is only an interface. Two senders can each produce 64-value messages but assign completely different meanings to them.

Our first pairwise adapters learned directional codes. The shared-space experiment adds two constraints that help with training the learned adapters:

One shared receiver: both arms reuse the same message-decoding module.
One training-time fact anchor: a shared Linear(64, 23) decoder must reconstruct each sender's own normalized local measurements from its message.

                 FROZEN FEATURE CACHES
                  /                 \
          SmolVLA: 960D         X-VLA: 1024D
                |                    |
                v                    v
          Trainable sender A   Trainable sender B
                |                    |
                +------ 64D ---------+
                         |
             +-----------+-----------+
             |                       |
             v                       v
      SHARED RECEIVER          SHARED FACT ANCHOR
        64 → 64 → 32             Linear(64, 23)
             |                       |
             v                       v
     Peer context + local       Predict sender's
     state and features         own measured facts
             |                       |
             v                       v
     Replacement action heads   Auxiliary MSE loss
             |
             v
     Motion + gripper + hold loss

TRAINING:
Recipient action loss updates its head and receiver
through the peer's sender. Backbones stay frozen.

RUNTIME:
The fact anchor is removed from the inference path.
Embedding-only messages carry no extra fact payload.

The action objective combines Cartesian mean-squared error, 0.3 × gripper binary cross-entropy, and hold binary cross-entropy. The shared variant adds 0.5 × fact-anchor mean-squared error.

We cache features with pinned model and extraction identities, split data by whole episode, and select checkpoints using development loss within a 400-epoch budget. Adapter training uses the cached features; its CPU training time excludes foundation-model feature extraction.

For the transfer test, the existing system stays frozen while a new sender encoder trains against one arm. We then test whether the other arm understands it without updating that receiver. The shared variant achieved 96.8% other-arm hold accuracy versus 48% for pairwise adapters, averaged over five training seeds.

This result covers two known VLA families, two cameras and a small held-out diagnostic. It does not establish arbitrary-model compatibility or complete learned-policy folds.

Shared-space results and limits • Training implementation

How we use SpacetimeDB

Sponsor integration: Best App Using SpacetimeDB.

SpacetimeDB is the preferred shared coordination backend for Relay. We use its private tables, transactional TypeScript reducers, live subscriptions and native scheduled reducers to connect model communication with an operational workspace.

1. Durable objectives and ordered execution

Spacetime stores objectives, success criteria, assigned workers, claims, revisions and reported outcomes. In the simulation runners, reducers commit the plan, prepared action batch and observed feedback in order. The runner confirms the required committed state before advancing the next allowed physics step.

Identical database retries can recover an existing result. Conflicting duplicate writes are rejected. Retrying a database operation does not authorize repeating a robot movement.

2. Scoped text and latent-message transport

Packets carry their run, sender, recipient, observation timing, expiry, codec and adapter identity.

Exact readback confirms the committed batch. The Python receiver rejects expired, superseded or out-of-order observations and records the reason. Learned translation and vector scoring run in the application; Spacetime supplies durable, synchronized message state.

3. Shared factory resources

The factory coordination module records robot capabilities, command batches, shared-resource reservations, acknowledgements and outcomes. Reducers check prerequisites and resource ownership so robots can coordinate station access and handoffs explicitly.

4. Live subscriptions

A trusted Node SDK bridge subscribes to private task, message, presence, attention and analytics tables. The Python backend forwards authorized change notifications through server-sent events. Dashboards and workers then fetch their current permitted snapshot through the application API.

Notifications do not carry robot commands or database credentials. Authorization is rechecked while streams remain open. Reconnecting clients reload state, and polling provides a fallback when live delivery is interrupted.

5. Automatic schedulers and cron-style jobs

Job	Trigger	What happens
Runtime presence watchdog	Workers heartbeat every 10 seconds and renew a 45-second lease.	A scheduled reducer marks an expired runtime stale and creates an attention notice. Generation checks stop old timers from expiring renewed leases.
Task deadline watchdog	30 minutes by default; configurable from one minute to 24 hours.	The reducer checks current task state before raising an overdue notice. Completion and cancellation resolve it; superseded revisions cannot overwrite newer state.
Health reconciliation	Hourly UTC by default.	Reconciles expired runtime leases and overdue task notices.
Analytics rollups	Hourly UTC by default.	Aggregates deduplicated numeric samples into count, sum, minimum and maximum buckets. The dashboard can show means with sample counts.
Retention cleanup	Daily at 00:00 UTC by default, with seven-day retention.	Removes eligible old coordination metadata while retaining authoritative task history, revocation tombstones and unaggregated samples.

These jobs use native schedule tables and scheduled reducers. Spacetime invokes the due callback, which checks current state before updating it transactionally. Jobs can continue while the Studio web process is offline, provided the database remains running.

Managers can pause jobs, choose hourly or daily UTC schedules, configure retention from 1–90 days, or run a job just once. The dashboard exposes next and last execution times, execution counts and processed-record counts.

Rollups process up to 256 samples per batch and cleanup deletes up to 256 eligible records per batch. A remaining backlog schedules a continuation one second later. Delayed calendar jobs advance to the next future boundary instead of replaying every missed interval.

6. Workspace recovery and stale-writer protection

In the Spacetime-backed deployment profile, a private versioned checkpoint preserves accounts, configuration, model approvals, scoped worker credentials, audit records, telemetry and the coordination outbox. Local SQLite files act as disposable caches restored before requests are served.

Changed API requests return success only after confirmed checkpoint commits. Database leases and revision checks prevent stale application instances from committing outdated workspace or task updates.

Startup reconciliation repairs pending coordination metadata without replaying robot actions. The supported profile is one private workspace per customer with one intended application replica; inference and execution remain on agent hosts.

7. Measured activity becomes inspectable analytics

Recorded samples include exchange latency, database latency, LLM latency and packet size. Scheduled aggregation turns those observations into summaries with sample counts. The dashboard distinguishes measured results from estimates instead of treating assumed labor savings as observed performance.

Coordination and scheduling • Subscription bridge • Persistence and recovery

System architecture

                     OPERATOR
                Goal + success criteria
                         |
                         v
                    Relay Studio
       Roles · model releases · scoped workers
                         |
                         v
                    SPACETIMEDB
       +---------------------------------------+
       | Private task and peer-message tables  |
       | Transactional TypeScript reducers     |
       | Claims, revisions and resource owners |
       | Plans, actions and observed outcomes  |
       +---------------------------------------+
             |                  |
             |                  +-----------------------+
             v                                          v
      LIVE SUBSCRIPTIONS                       NATIVE SCHEDULERS
      Trusted Node bridge                      Presence watchdogs
             |                                 Task deadlines
             v                                 Hourly health checks
      Authorized SSE notices                   Analytics rollups
             |                                 Daily retention
             v
      Scoped snapshot fetch through Studio API
             |
             v
         AGENT RUNTIMES
      Local model inference
      Qwen / trained-adapter integration
      Simulator or customer execution handler
             |
             v
        OBSERVED OUTCOMES
             |
             +---- committed feedback ----> SPACETIMEDB
                                                |
                                                v
                                      Next permitted decision

HOSTED WORKSPACE:
Private checkpoints + lease/revision fencing
Restore caches + reconcile coordination outbox

Spacetime owns the committed coordination state and timers. Agent hosts own model inference and execution. The generic customer worker exchanges structured reports; learned channels and Qwen are explicit integrations in supported workflows or customer handlers.

Challenges we ran into

Learning meaning across model spaces. Matching dimensions did not align the representations. Pairwise adapters could solve their training task while inventing incompatible private codes. The shared receiver and fact anchor addressed that failure in the tested model/camera pairings.

Making partial observation real. We rendered counterfactual views at the same physical state and removed peer-side obstacle geometry and shadows from the recipient's view. We checked that its own image and local facts remained unchanged when only the peer's hazard changed.

Handling stale information. Adding 2–18 tick delays and 15% packet drops erased the fresh-message advantage. Freshness checks make the transport contract explicit, but the policy still needs training on actual historical observations.

Recovering distributed state. Renewed heartbeats, cancelled tasks, revoked credentials, stale writers and retried requests each needed explicit behavior. A working happy path was only one part of the implementation.

Separating demonstrations from evidence. The completed dashboard fold uses task-trained controllers. The learned-adapter experiments demonstrate one-step information use and transfer; completing a reliable fold with those adapters remains unfinished.

Accomplishments that we're proud of

The receiver uses its peer's information. Learned messages raised hidden-hazard hold accuracy from 50% to 100% on the recorded held-out test. Zeroing messages with the same checkpoint reduced accuracy to 50%; shuffling reduced it to 25%.
Transfer improved across the tested model/camera pairings. The shared-space variant achieved 96.8% other-arm hold accuracy versus 48% for pairwise adapters, averaged over five training seeds.
A complete task has an auditable history. A fixed-seed dashboard fold with task-trained controllers recorded 282 durable frames and 564 packets, with database auditing and independent action replay.
Communication helps the controlled factory task. Matched runs completed 12/12 with private messages versus 2/12 without, using the deterministic scheduler and in-memory coordination. Separate Qwen/Spacetime runs are reported independently.
The coordination stack is operable. Model approvals, scoped workers, subscriptions, watchdogs, maintenance jobs, audit history and durable recovery connect the research channel to a usable dashboard.

Results

The learned-message diagnostics use one held-out cloth episode with 20 physical states. The controlled factory comparison uses 12 seeds. These are small simulation evaluations with explicitly different experimental scopes.

Measurement	Result
Hidden-hazard hold accuracy: learned messages vs no communication	100% vs 50%
Same trained checkpoint: zeroed / shuffled messages	50% / 25%
One-step action error: learned messages vs no communication	0.89 mm vs 4.68 mm
Other-arm encoder transfer: shared-space vs pairwise	96.8% vs 48%; mean across five seeds
Swapped senders: shared-space vs pairwise	98.5% vs 51%
Controlled factory completion: messages vs none	12/12 vs 2/12; deterministic scheduler and in-memory coordination
Packaged pairwise adapter/head bundle	324,874 trainable parameters
Local SpacetimeDB synchronous commit-and-readback	Approximately 4–6 ms mean across recorded channel runs; model inference excluded
Hold accuracy with delayed and dropped messages	66.3%; no improvement over the no-communication condition

Structured text also reached 100% hidden-hazard hold accuracy and 12/12 controlled factory completions, with smaller packets in the reported comparisons. The learned channel has not established an advantage over that baseline on these tasks.

Learned-channel evidence • Transfer evidence • Completed-fold audit

What we learned

Useful coordination requires both meaning and timing. The recipient must understand a message, and the message must still describe the world when it arrives.

A common message size does not create a common vocabulary. Shared training constraints were necessary to make the tested senders more interchangeable.

Structured text is a strong baseline. Held-out episodes, matched observations and zero/shuffle interventions helped us distinguish real information use from an attractive visualization.

Infrastructure is part of the model system. Claims, revisions, deadlines, subscriptions and recovery determine whether useful model outputs can become dependable shared work.

What's next for Relay

Delay-aware training: train with actual earlier peer observations, variable latency and packet loss.
Closed-loop learned control: turn the one-step coordination results into completed folds scored end to end.
Additional model families: test whether the shared-space mechanism transfers beyond the two families already represented.
Broader tasks and evaluations: expand the number of episodes, environments and coordination failures.
Customer validation: measure setup effort, task completion, waiting time and operator intervention with independent users.

The goal stays practical: different robots should share what they know, adapt when conditions change and make their collective decisions inspectable.

Research grounding

Relay's learned channel is CoLA-inspired and informed by VLA-Fabric, Latent Telepathy and Interlat. Our implementation combines mixed-family feature experiments, small state-assisted replacement action heads, a shared-space training variant and an operational coordination stack. It is not a reproduction of those papers.