Human Procedure → Machine Procedure

Inspiration

People responsible for physical processes face a practical question: what should they change, and will that change be worth the effort and cost?

A repetitive task might benefit from a robot arm, a humanoid or a simple piece of custom tooling. Choosing among them requires understanding the work, the required output, operator involvement and the evidence behind the proposal. We wanted to make that investigation easier for lab managers, workshop operators and operations teams.

What it does

Human Procedure → Machine Procedure organizes the decision into three steps: see the task, review the change and compare costs.

The current workspace offers two prepared examples: tightening gearbox screws and placing a planet gear. Users can watch the human task and initiate an analysis that researches sources with Tavily, then uses a GMI reasoning agent to assess SO-101, Unitree G1 and custom-tooling alternatives. The assessment records the proposed intervention, rationale, references and unresolved requirements.

Prepared machine animations make the alternatives understandable. A comparison screen places the human task beside the selected concept and displays conditional cost estimates. Saved analyses can be reopened, and the decision and supporting evidence can be exported for review.

The wider workbench connects procedure extraction with structured operations, apparatus requirements, CAD and scene workflows. Its purpose is to help users identify a promising experiment and the measurements needed to evaluate it.

How we built it

The backend is written in Python and uses typed, validated records to connect procedure inputs, model outputs, research evidence, machine concepts and process accounting.

Nebius Token Factory provides the NVIDIA inference integration. In a documented live verification, nvidia/Nemotron-3_5-Lightning extracted seven operations from a source procedure, which were then consumed by the protocol compiler. Requested and served model identities matched, and the output passed strict validation. That result establishes the text-to-protocol path; it does not establish general video-understanding accuracy.

Tavily supplies research sources for the candidate-assessment workflow. GMI supplies reasoning for that workflow. Forma/OpenCAD and OpenIndustries support machine-design and scene work. The latest three-screen experience uses prepared robot and tooling animations rather than generating new motion during each analysis.

We also documented a separate local humanoid diagnostic: a compatible GR00T checkpoint produced one action chunk, which SONIC consumed in MuJoCo for 40 control steps, or 0.8 simulated seconds. This used a Cosmos backbone and retained model revisions and execution evidence. It was a bounded integration test, with task success unevaluated. It was not a physical robot run or a Nebius-hosted GPU job.

Challenges we ran into

Connecting the components was only part of the challenge. We also needed a clear user journey and consistent boundaries between observations, model proposals, visualizations and measured results.

Structured output needed precise prompts and strict validation. One live probe returned a JSON boolean when the application required an object. We corrected the prompt, retained the failure evidence and kept the validator strict.

Robot integration introduced checkpoint compatibility, memory and controller-interface constraints. Economic comparison required equal attention: a video duration is not a measured production cycle, and a plausible machine animation does not establish yield, labor requirements or savings.

Accomplishments that we're proud of

We reduced a broad engineering workflow to a task-first interface that makes the proposed change and its economic assumptions easier to inspect.

We documented successful NVIDIA text inference through Nebius Token Factory with downstream compilation, created task-matched human/machine comparisons, and retained a real local GR00T-to-SONIC diagnostic. Sources, assumptions and unresolved requirements remain attached to the decision rather than disappearing behind a recommendation.

What we learned

Start with the complete user journey, then integrate each technical component into it. A useful automation proposal needs a clear task boundary, an understandable mechanism and a comparison that identifies what still needs measurement.

We also learned to separate successful software execution from successful physical work. Both matter, but they answer different questions.

What's next for Human Procedure → Machine Procedure

Complete live validation of the latest automatic candidate-assessment flow, package the required demo assets for reproducible judging, and improve progress feedback during analysis. Then measure accepted output, operator attention, cycle time and rework in matched trials, and use those observations to revise the proposed intervention and its economics.

Built With

• nebius
• vercel