Posecode: Spatial DSL for LLMs

A human-readable kinematic DSL that gives AI spatial reasoning. It enables LLMs to describe, validate, and render complex 3D biomechanical movements via an MCP server and WebGL.

Comment

Inspiration

LLMs are excellent at generating text and code, but they still struggle to describe precise movement in 3D space.

A prompt like “raise the left arm, rotate the torso, and bend the knees” is understandable to a human, but too ambiguous for a renderer or simulation system. It does not define exact joints, angles, timing, coordinate frames, or physical limits.

That inspired Posecode: a human-readable spatial DSL that gives AI a structured way to describe movement.

What it does

Posecode sits between natural language and low-level 3D motion data.

An LLM can translate a movement request into structured Posecode, which can then be:

  • validated,
  • scored for biomechanical fidelity,
  • rendered in the browser,
  • and used by AI agents through an MCP server.

The project includes:

  • a human-readable kinematic DSL,
  • a client-side WebGL renderer,
  • an MCP server for agent workflows,
  • and a biomechanical fidelity scorecard.

The scorecard evaluates factors such as joint limits, balance, continuity, and spatial consistency.

Conceptually, the final score is:

$$ F = \sum_{i=1}^{n} w_i s_i $$

where (s_i) is an individual biomechanical check and (w_i) is its weight.

How I built it

The system follows a simple pipeline:

  1. A user describes a movement in natural language.
  2. GPT-5.6 converts it into Posecode.
  3. The parser creates a structured motion representation.
  4. The validator checks syntax, joint references, and movement constraints.
  5. The scorecard evaluates biomechanical plausibility.
  6. The WebGL renderer displays the motion in 3D.

The MCP server exposes these capabilities to AI agents, allowing them to generate, validate, inspect, and improve movements programmatically.

I kept the language, validation engine, renderer, and MCP layer separate so each component can evolve independently.

How I used GPT-5.6 and Codex

GPT-5.6 acts as the reasoning layer between natural-language instructions and structured movement.

It interprets anatomy, direction, sequencing, joint relationships, and timing before generating valid Posecode.

Codex helped me:

  • design and refine the DSL grammar,
  • implement the parser and validation logic,
  • connect Posecode to the WebGL skeleton,
  • build the MCP tools,
  • create tests,
  • and debug coordinate-system and transformation issues.

I used Codex as an engineering collaborator while making the key product, architecture, and language-design decisions myself.

Challenges

The hardest challenge was translating ambiguous language into exact geometry.

Words such as “forward,” “slightly,” or “turn” depend on the coordinate system, body orientation, active joint, and movement range.

Another challenge was designing a DSL that is readable for humans but constrained enough for LLMs to generate reliably.

Rendering was also difficult because human motion is hierarchical. Rotating a shoulder affects the elbow, wrist, and hand, so joints cannot be treated independently.

Finally, syntax validation alone was not enough. A pose can be valid code but still be physically unrealistic, which led to the biomechanical scorecard.

What I learned

The biggest lesson was that spatial reasoning needs an explicit representation.

Prompting alone cannot fully remove ambiguity. A structured DSL creates a shared contract between the LLM, developer, validator, and renderer.

I also learned that validation should explain more than whether something failed. It should show what failed, why it matters, and how the movement could be improved.

This enables an agent loop:

  1. generate,
  2. validate,
  3. inspect the score,
  4. revise,
  5. render.

Potential impact

Posecode could be useful for:

  • game and character animation,
  • fitness and exercise visualization,
  • dance and choreography,
  • education and anatomy,
  • rehabilitation demonstrations,
  • sports analysis,
  • robotics,
  • and synthetic motion generation.

The long-term goal is to create a shared spatial interface between language models and systems that operate in the physical world.

LLMs already have languages for software, data, and interfaces.

Posecode gives them a language for movement.

Built With

Share this project:

Updates