About the Project
Jupiter stems from an idea that is deeply important to me: making computers genuinely usable for people with limited mobility or difficulty operating a PC, so they can simply express what they want to accomplish and use the full power of ChatGPT to achieve it.
Jupiter is an AI agent for Windows desktops that transforms natural-language requests into safe, visible, and verifiable actions across the operating system, browser, files, and applications.
Unlike an agent that directly controls the computer based solely on model output, Jupiter separates reasoning from authority:
Codex reasons and proposes. Jupiter validates, authorizes, executes, verifies, and stops.
This architecture keeps the user in control throughout the entire task.
Inspiration
Jupiter was inspired by a question that was both technical and deeply personal:
What if using a computer did not depend on knowing how to operate a keyboard and mouse?
I wanted to make computers more accessible to people with limited mobility or difficulty using a PC. Instead of learning every interface, navigating menus, or coordinating precise movements, a person should be able to express the desired outcome and let an intelligent assistant handle the mechanical work.
At the same time, giving an AI access to a desktop raises serious security and trust challenges. It is not enough for an agent to be capable; users must be able to see what it plans to do, approve sensitive actions, interrupt it instantly, and understand what actually happened.
This gave rise to Jupiter’s central idea: an intelligent desktop agent that behaves more like supervised cruise control than unrestricted autopilot.
What It Does
The user sends Jupiter a request through text or push-to-talk voice. Jupiter asks Codex to transform that objective into a structured plan, validates the plan, and then executes each approved step through governed capabilities.
Jupiter can:
- Create and display a structured plan before acting.
- Open and focus allowlisted Windows applications.
- Search and navigate the web through a structured, isolated browser.
- Open links in the user’s regular browser without reading its cookies, history, or session.
- Create verified UTF-8 text files and open them safely in Notepad.
- Complete bounded workflows involving applications such as Paint.
- Inspect, copy, move, organize, and send approved files to the Recycle Bin.
- Arrange managed windows into focus or split-screen layouts.
- Evaluate the risk of each action and request confirmation at the exact moment it is needed.
- Pause when it detects meaningful physical keyboard or mouse input.
- Cancel active tools and ignore late results after a task has stopped.
- Verify every step before continuing.
- Maintain a sanitized local audit trail.
- Operate in English or Spanish through a compact, translucent companion interface.
For example, a user can ask Jupiter to research a topic, gather sourced results, arrange Chrome and Notepad side by side, create a summary, and report the verified outcome. During this workflow, touching the keyboard or mouse immediately returns control to the user, and Jupiter never resumes without an explicit instruction.
How We Built It
Jupiter was developed as a native Windows-oriented desktop system using:
- Electron, React, and TypeScript for the desktop companion and operational interface.
- Codex App Server as a persistent local reasoning service communicating through
stdioand JSONL. - C# and .NET for the native Windows supervisor.
- Structured Chrome automation for bounded web research.
- SQLite for sanitized local auditing.
- Typed, runtime-validated IPC contracts between Electron’s main, preload, and renderer processes.
- Vitest, ESLint, TypeScript checks, native self-tests, smoke tests, and end-to-end workflows for verification.
The system is divided into clearly defined trust boundaries. Codex proposes a structured, versioned plan, but Jupiter owns the operational state.
A Plan Validator checks the proposed capabilities and arguments. The Scheduler controls dependencies and execution leases. The Policy Gateway evaluates permissions and risk, while the Tool Gateway is the only access point to tools with real-world effects. Finally, deterministic verifiers confirm the outcome before the plan can continue.
The main execution flow is:
User request
↓
Codex proposes a structured plan
↓
Jupiter validates capabilities, arguments, permissions, and risk
↓
The Scheduler dispatches an authorized step
↓
The Tool Gateway invokes a controlled adapter
↓
Jupiter verifies the real result
↓
Continue, retry, replan, or stop
↓
Report the verified outcome to the user
Throughout development, we followed a Specification-Driven Development process. Every capability was connected to a functional or non-functional requirement, an architectural decision, acceptance criteria, and reproducible verification evidence.
The implementation was divided into 23 approved stages covering feasibility, the desktop foundation, Codex integration, orchestration, permissions, physical takeover, voice, browser automation, verification, security, packaging, localization, and governed application workflows.
Challenges We Ran Into
The greatest challenge was balancing autonomy with safety. Making an agent perform actions is relatively straightforward; making every action attributable, interruptible, scoped, and verifiable is much harder.
Some of the most significant challenges were:
- Keeping the operational plan outside the model so execution remained deterministic and observable.
- Preventing unknown or malformed model capabilities from reaching the operating system.
- Ensuring the planner schema, executable capability registry, and policy rules could never silently diverge.
- Propagating cancellation through the planner, scheduler, policy system, and active tools.
- Preventing late results from cancelled operations from advancing an old plan.
- Distinguishing meaningful physical takeover from minor mouse jitter or synthetic input generated by Jupiter.
- Separating isolated browser automation from the user’s personal signed-in browser session.
- Creating files atomically without overwriting existing data or allowing path traversal.
- Preserving Electron’s main/renderer isolation while exposing the necessary native Windows functionality.
- Supporting different application installation methods, including traditional paths and Windows execution aliases.
- Packaging Codex and the native .NET supervisor into a reproducible Windows release.
- Supporting English and Spanish consistently across the operational interface.
- Building, testing, and documenting a large native application with limited development credits and resources.
Desktop automation also introduced many unpredictable conditions: applications can disappear or change state, windows can move, browser content can change, and cancellation can happen at any moment. These realities forced us to design Jupiter to fail safely instead of assuming that every workflow would succeed.
Accomplishments That We’re Proud Of
We are especially proud that Jupiter became more than a visual prototype. All 23 approved development stages have been implemented, with 21 stages fully verified and the remaining stages awaiting external or physical verification evidence.
Some accomplishments that stand out include:
- A real, persistent Codex App Server integration with structured planning.
- A deny-by-default Capability Registry and governed Tool Gateway.
- Exact, single-use approvals for sensitive actions.
- Native physical-input takeover with a target response time of:
$$ p95 \leq 150\text{ ms} $$
- Execution leases that prevent stale results from continuing a cancelled task.
- Structured, sourced web research without exposing the user’s personal browser session.
- Local voice recognition that sends only the final editable transcript across IPC—not audio or partial hypotheses.
- Atomic, create-new-only UTF-8 file creation with path, size, and hash verification.
- A compact bilingual companion window with persistent operational controls.
- Sanitized recovery and auditing that can reconstruct a workflow without storing secrets.
- Reproducible Windows installer and portable release pipelines.
- Security and end-to-end tests covering takeover, prompt injection, external failures, and unexpected application states.
Most importantly, Jupiter demonstrates that useful desktop agency does not require surrendering control to the model.
What We Learned
We learned that the quality of an agent depends as much on the deterministic system surrounding the model as it does on the model itself.
Structured output was essential, but receiving valid JSON was not enough. The plan schema, Capability Registry, policy rules, runtime arguments, and verifiers all needed to agree. We therefore added automated checks that fail when the planner’s contract diverges from what Jupiter can safely execute.
We also learned to prefer structured integrations over visual automation. Native APIs, application-specific tools, browser protocols, and accessibility interfaces are generally more reliable and easier to verify than screenshots and fixed coordinates. Visual interaction remains valuable, but it should be a controlled fallback rather than the default.
Cancellation proved to be a system-wide property, not just a button. Stopping safely requires interrupting the active model turn, revoking the execution lease, cancelling the active tool, blocking new actions, and rejecting any result that arrives later.
Another important lesson was that verification must be part of execution rather than an afterthought. A tool reporting success does not necessarily mean the expected result exists. Jupiter therefore verifies relevant postconditions before advancing the plan.
Finally, Specification-Driven Development helped us manage the project’s complexity. Writing requirements and acceptance criteria before implementation made security decisions explicit and allowed us to evaluate behavior through evidence instead of assumptions.
What’s Next for Jupiter
The immediate next step is to complete the remaining physical verification of the packaged Notepad workflow and finish the external submission evidence.
After that, the goals are to:
- Improve the interface and make Jupiter’s explanations clearer and more natural.
- Test a much wider variety of natural-language requests and Windows environments.
- Expand the ecosystem of allowlisted applications and MCP integrations.
- Conduct accessibility testing with real users.
- Strengthen recovery from application, browser, network, and model failures.
- Add richer—but still bounded and privacy-preserving—session continuity.
- Improve the way Jupiter communicates progress, uncertainty, and partial results.
- Explore a mobile companion capable of securely sending requests to the computer.
- Eventually allow Jupiter to be invoked from phones, watches, or smart glasses.
The long-term vision is for Jupiter to become an unobtrusive interface between human intent and the computer: available whenever it is needed, transparent about every action, and always subordinate to the person using it.
Log in or sign up for Devpost to join the conversation.