Inspiration

Our journey began with a simple yet powerful idea: for robots to be truly helpful partners in the real world, they need to earn our trust by demonstrating reliable performance not just in average cases, but in the full range of situations they may encounter. This insight led us to pioneer a new approach to robot skill licensing based on rigorous real-world testing and continuous monitoring.

We were particularly inspired by the potential for robotics to enable flexible automation of picking and placing tasks. The ability to reliably grasp and manipulate a wide variety of objects is a critical capability for many industries, but one that remains challenging for robots due to the diversity of object shapes, materials, and environmental conditions.

What it does

Learner's Permit is a framework for licensing robot skills based on demonstrated performance in real-world conditions. It enables robots to gradually acquire new capabilities while providing guarantees about their reliability and safety.

At the core of the framework is a gating mechanism based on contextual multi-armed bandits that grants "permits" for specific skills only after a robot has completed a sufficient number of successful trials under diverse conditions. Each permit is associated with a learned skill policy (implemented using GR-ConvNet) and a set of formal conditions under which the policy is licensed to operate.

Once a permit is granted, the system continuously monitors the robot's performance using metrics like grasp success rate and pose estimation accuracy. If performance degrades, permits can be dynamically revoked to prevent further failures. This allows robots to safely explore the boundaries of their capabilities while mitigating risks.

The initial focus of Learner's Permit is on pick-and-place tasks in tabletop settings, with a modular perception stack based on the GR00T N1.7-DROID model for object manipulation.

How we built it

We developed Learner's Permit as a modular framework around several key components:

A gating mechanism based on contextual multi-armed bandits for skill licensing
GR00T N1.7-DROID, a vision-language-action (VLA) policy that drives a simulated Franka arm in NVIDIA Isaac Sim through RoboLab
A formal language for specifying permit conditions and safety constraints
A distributed architecture using Redis and FastAPI for real-time performance monitoring and permit management

To train and evaluate our policies, we leveraged NVIDIA Isaac Sim to generate large-scale simulated datasets and perform rigorous testing across a wide range of objects, poses, lighting conditions, and other variables. By sampling extensively from this "task space" during testing, we were able to characterize the reliability of each skill and define robust permit conditions.

Key to our approach was the development of a comprehensive suite of tools for running experiments on NVIDIA GPUs in the cloud, including:

Scripts for provisioning powerful RTX-series GPU VMs on Nebius AI Cloud with NVIDIA Isaac Sim and RoboLab pre-installed
A Serverless AI Job pipeline for running large-scale policy evaluations that can checkpoint progress and resume after preemption
Monitoring and dashboarding tools for visualizing live robot performance and permit status

We also developed a novel vision-based scene change detector using the Nemotron-3.5-Super-VL model served on Nebius Token Factory. By comparing live camera frames to reference images from the original permit-granting trials, this component allows us to automatically revoke permits when the environment changes in ways that may invalidate our performance guarantees.

On the crop disease classification front, we demonstrated the flexibility of our framework by licensing both few-shot (Nemotron-3.5-Super-VL) and fully-trained (ResNet-18) models through the same trust gate, using a dataset of leaf images from the PlantVillage corpus.

Challenges we ran into

Defining meaningful and verifiable permit conditions posed a significant challenge. We wanted our conditions to be expressive enough to capture the nuances of real-world tasks, yet precise enough to be reliably testable. Striking this balance required close collaboration between our robotics, machine learning, and formal methods teams.

Sim-to-real transfer also proved more difficult than anticipated for certain object categories. While our synthetic training pipeline using Isaac Sim was effective for many rigid objects, generalizing to deformable or translucent objects required additional real-world fine-tuning and the development of specialized augmentation techniques.

On the systems side, building a scalable and fault-tolerant architecture for distributed robot monitoring and control pushed us to innovate. We used Redis for real-time data streaming and FastAPI for building modular microservices, but ensuring seamless interoperability between these components and our robot platform required deep dives into the intricacies of different APIs and communication protocols.

Accomplishments that we're proud of

Looking back, we take pride in several key milestones:

Demonstrating reliable grasping of over 100 novel objects in cluttered tabletop environments
Achieving sub-millimeter precision in pose estimation for industrial kitting tasks
Maintaining grasp success rates above 95% under varying lighting conditions
Substantially reducing training time and data requirements through sim-to-real transfer using Isaac Sim
Deploying our framework on the Franka Emika Panda robot and validating its performance in real-world trials

But perhaps our proudest achievement is developing a principled framework for continuous testing and incremental deployment of robot skills. By providing formal guarantees about the reliability and safety of robot behaviors, we believe Learner's Permit represents a significant step towards trustworthy robot autonomy.

We were also thrilled to contribute a new dataset of real-world grasping interactions to the robotics community, which we collected as part of our testing and evaluation process.

What we learned

Through this project, we gained a deep appreciation for the challenges of reliable robot autonomy in unstructured environments. Some key lessons:

Formal specifications and rigorous testing are critical for building trust in robot systems. Informal notions of reliability are insufficient.
Sim-to-real transfer learning using tools like NVIDIA Isaac Sim can greatly accelerate training and reduce data collection burdens, but additional fine-tuning is often necessary.
Modular system design around microservices and event-driven architectures using tools like Redis and FastAPI is key to building scalable and adaptable robot frameworks.
The most impressive AI is not always the most useful. Simple, reliable models that earn trust are often more valuable than brittle state-of-the-art approaches.

On the technical side, we deepened our expertise in sim-to-real transfer learning, Bayesian robotics, and the intricacies of various perception algorithms and robot control schemes. But perhaps most valuably, we honed our skills in combining diverse tools and techniques into an integrated system for real-world robot autonomy.

What's next for Learner's Permit

We're excited to expand the Learner's Permit framework to new task domains and robot platforms. From assembly to inspection to mobile manipulation, we believe our approach of incremental licensing based on real-world validation has vast potential.

Integrating more advanced perception and reasoning capabilities is a particularly exciting frontier. By combining our framework with emerging techniques in 3D scene understanding, task and motion planning, and natural language grounding, we see a path to enabling more flexible and adaptive robot skills.

We're also keen to engage further with the robotics community to standardize benchmarks and share best practices around robot skill testing and validation. We plan to release key components of our framework as open-source modules to accelerate progress.

Ultimately, our goal is to empower roboticists and industry practitioners with tools to build robot systems that are not only capable, but truly trustworthy. By decomposing complex skills into testable components and continuously validating performance, we hope to accelerate the path to reliable, scalable robot autonomy across domains.