See It, Say It, Pick It

Inspiration

Robots need to connect what they see with what we mean. We wanted to explore how Apertus and open vision models could bridge that gap without task-specific training.

What it does

See It, Say It, Pick It turns a workcell image and an English instruction into a grounded robot action plan - or asks for clarification or refuses when appropriate.

How we built it

Apertus interprets instructions, Qwen identifies objects and stacking relationships, and SAM segments each object. Deterministic geometry builds a scene graph that Apertus uses to plan actions, followed by format and mechanical checks. Everything runs locally on a four-GPU CSCS Clariden node.

Challenges we ran into

Distinguishing similar objects, interpreting physical support relationships, and keeping model outputs consistent required careful coordination between perception, geometry, and planning.

Accomplishments that we're proud of

Our pipeline scored 54.9/65 on the public evaluation. We reproduced that score in a rebuilt container, with all 50 answers matching the reference run exactly.

What we learned

Language models become more useful for robot planning when they receive explicit spatial evidence and clear constraints. Reliable integration matters as much as model capability.

What's next

We want to improve perception in cluttered scenes, evaluate generalization on held-out data, and test the generated plans on a physical robot.

Built With

• apertus
• python
• qwen
• sam
• segmentation