-
-
Login screen (Nvidia Edition)
-
Issac Sim
-
Simulator
-
Adapter configuration
-
Issac sim running
-
Simulator
-
Task View
-
Real time joint control
Inspiration
As a team who build robots as a hobby and work as developers, we wanted to develop a dedicated IDE that allows users to work with both high level and low level functions, while being modular enough to run with anything. When we checked the market, we found out that IDEs are all geared towards pure software tasks. So, as robot developers, we are left with very few reliable options. This lead us to develop RoboDE.
What it does
The core system is an IDE that can run either as a cloud based website, or completely locally. The IDE combines AI into a product that uses AI as an assistant, rather than as the decision maker. The system is also built in such a way that users can customise it according to their hardware specifications and not require high end compute to run the system. Every component is modular, including LLM adapters and choice of simulators. Since Issac Sim requires compute to run, we built an in-house basic simulator that shows the users what they are doing. However, if available, the users can also simulate in Issac sim in real-time, or use a recording. We have also tested the IDE with multiple LLMs, including Nemotron (multiple variants), Phi4:14b and Gemini Flash variants and have documented the results. Modularity and ease of use is at the core of the system. The core idea reason behind adapter modularity is that simulation should not be different from the real world; just changing adapters should allow the robots to work as expected in the real world. A companion app has also been developed partly that aims to serve non-technical users.
How we built it
RoboDE is built with Electron and React (TypeScript), backed by a Python FastAPI engine. The same app runs on the desktop, as a shared team server in the browser, and as a phone-friendly Business App. Two-stage AI planning: a Planner LLM turns plain English into robot tasks, and a separate Task LLM converts each task into robot commands. Every model plugs in through one adapter interface, so NVIDIA Nemotron (via Nebius Token Factory, the NVIDIA API, Ollama or OpenRouter) works without code changes. Safety checks: inverse kinematics, joint limits, collision detection and postcondition checks run before anything moves. LLM output is never executed unchecked. NVIDIA Isaac Sim: a GPU worker runs Isaac Sim 6.2 in the cloud. The IDE controls robots live, runs validated plans and records Isaac’s camera for playback. 3D and editing: three.js for the scene view, Monaco for code.
Challenges we ran into
Deployment in a cloud platform was truly challenging as we kept hitting server limits. This lead us to develop the local simulator and the modular configurable cloud simulator. Moreover, running Issac sim requires significant compute which we did not have, and assumed that this would be a common problem for developers. So, we built the in-house simulator that later links to other simulators.
Accomplishments that we're proud of
We are proud that we are able to split the system into multiple modular components, allowing users with all sorts of compute to run it. We are also extremely proud of the UI and are hopeful that it will simplify development processes. The unified interface can also allow robot developers to quickly prototype and then carry over that prototype to the real world. Moreover, the business app aims to simplify command and control for non-robot developers, while still ensuring safety.
What we learned
Smaller LLMs are powerful. A large LLM is not required for a lot of tasks, and that is why the modularity of the system works. Users can use large LLMs for the complex planning tasks while smaller versions can be used for the conversion tasks. This can assist in token economics and also make the system more accessible. However, large amounts of testing is required to determine the optimum solution.
What's next for RoboDE
We want to extend RoboDE to become a platform that unifies robot developers and robot users. Integrating the business app with technology like Nvidia NuRec to allow reconstruction of room scene and then apply robots there. This can allow faster adoption of robots in both personal spaces and business areas. Moreover, we want to develop the planner engine further to account for larger, more complex environments as well. We also want to integrate it deeper with Nvidia Tech and try it out with higher tier Nemotron models to achieve path planning and robot motion in complex environments.
NOTE: There is a special Nvidia version that is functionally the same as RoboDE, but it uses slight Nvidia accents for colouring and auto-selects Issac Sim and Nebius Token Factory as default selections. Functionally, it is the same.
