Inspiration

Telling a robot "put the small yellow gear in the bin" sounds simple, but language models are good at words, not pixels, and robots should never act on a guess. We wanted to see how far Apertus, a fully open Swiss LLM, could go as the language brain of a pick-and-place system while keeping every robot decision checkable and safe.

What it does

See It, Say It, Pick It takes one workcell image and one English instruction and returns a checked robot plan with a first-pick pixel. If the instruction is ambiguous, it asks for clarification (ask_user). If the task is unsafe or impossible, it refuses (refuse).

How we built it

Each model has one bounded responsibility, and a deterministic resolver makes the final call:

Observe: Qwen2.5-VL lists objects and bounding boxes; SAM 2.1 produces masks.
Recover: Grounding DINO + SAM search again only for references that are still unresolved.
Parse: Apertus v1.5 8B converts the instruction into a small typed task intent instead of a free-form robot action. We split coordinated instructions into clauses, constrain the output schema, verify that every qualifier (color, destination, etc.) is preserved, and allow one bounded semantic repair.
Resolve: Deterministic constraints check identity, spatial relations, uniqueness, feasibility and safety.
Ground: Mask geometry selects an interior first-pick pixel and emits the plan.

Apertus never invents pixels and never overrides the safety checks.

Challenges we ran into

Compositional instructions: short literal commands were reliable, but nested relations, coordinated clauses, negation and scope were less stable.
Structured output: Apertus sometimes returned null instead of an empty list, dropped half of a relation pair, or lost a qualifier.
Language–perception mismatch: Apertus correctly says "gear", but the vision inventory may label it sun or graspable_object, so exact matching fails.
Relations amplify detection errors: one duplicate or missed object can make a whole nested reference ambiguous.

Accomplishments that we're proud of

On the official public development set we scored 24.0 / 65, with an unsafe-command catch rate of 0.90. The system almost never executes unsafe motion, and every refusal is traceable to an explicit check.

What we learned

Keep LLM intent typed: Apertus is most useful proposing a constrained intent, not a raw action.
Keep final decisions deterministic: explicit ambiguity, feasibility and safety checks make behavior predictable.
Our system is safe but conservative: a false-alarm rate of 0.50 means it still rejects too many valid tasks.

What's next

Model-backed semantic matching between language and open-vocabulary visual labels, without a hand-maintained alias list.
Duplicate suppression and conflict filtering before relation solving.
Scale-aware grasp and container feasibility checks, plus testing on more held-out-style instructions.