Nimbus

Point, ask, learn: Your on-screen tutor — hold a key, ask about anything, and it explains it out loud and shows you where. Studying or stuck in an app, learn without leaving your screen.

Comment

Nimbus: An AI tutor that can see your screen

Nimbus started from a problem I kept running into whenever I was learning new software.

Most people don't get stuck because they don't understand the concept. They get stuck because they can't find the next button to click.

It happens everywhere. A student understands the Excel formula but can't find the option their teacher mentioned. Someone learning VS Code knows what they're supposed to do but can't locate the right panel. A teacher helping a student remotely has to describe an interface using words alone. Most existing help pulls you away from what you're doing. You end up searching Google, opening YouTube, reading documentation, then switching back and trying to apply instructions that weren't written for the screen you're actually looking at.

I wanted something that felt like having a tutor sitting next to you.

Nimbus is a voice-controlled, screen-aware Windows assistant. Hold a push-to-talk hotkey, ask about what's on your screen, and Nimbus answers out loud. If your question is about using an application, it can point directly at the button, menu, or field you're looking for. In Tutor Mode, it goes a step further by drawing circles, arrows, labels, and underlines directly on the screen so the explanation feels visual instead of verbal.

The education side of the project is what matters most to me. Nimbus helps students learn software while they're actually using it instead of forcing them to switch between tutorials and the application. It also makes remote teaching much easier by letting instructors guide students visually instead of relying on long descriptions. Schools and organizations can add their own Markdown knowledge files for internal software, workflows, or learning systems, and Nimbus keeps local memory for each application so future help can build on previous conversations instead of starting from scratch every time.

How I built it

Nimbus is a Windows desktop application built with Python and PyQt6.

The interaction is intentionally simple:

Hold a configurable push-to-talk hotkey. Nimbus records your voice and captures the current screen. It sends the transcript, screenshot, local per-application memory, and optional knowledge-base context through the configured AI pipeline. Nimbus speaks the response and, if needed, returns the coordinates of the relevant UI element. A transparent, click-through overlay displays the pointer without interrupting the application underneath.

The overlay system ended up being one of the most challenging parts of the project.

Screenshots use physical pixels, while Qt works in logical coordinates, and Windows allows every monitor to have different DPI scaling. To make the pointer reliable, Nimbus creates a separate overlay for each physical display and converts coordinates for the correct monitor before drawing anything. That keeps the pointer accurate even on mixed-DPI, multi-monitor setups.

For locating interface elements, Nimbus supports two approaches. The preferred method is receiving direct [POINT:x,y] coordinates from the vision model. When a provider can't return exact coordinates, Nimbus falls back to a two-stage numbered grid system that first identifies a larger region before narrowing it down. For very small interface elements, or anything close to the cursor, it can perform an additional cropped verification pass before showing the pointer.

Tutor Mode builds on top of this by accepting annotation data for circles, arrows, labels, and underlines. The overlay renders these temporarily with simple animations, making walkthroughs feel much closer to having someone point things out on your own screen.

I also wanted users to stay in control of their data. Nimbus supports bring-your-own API keys stored in Windows Credential Manager, local Markdown memory for each application, a local Knowledge Folder, session-history export, optional diagnostics with configurable retention, and a clear-all-local-data option.

The speech system is modular as well. Users can run local Faster-Whisper speech recognition together with Kokoro text-to-speech for a more private or offline-focused setup, while cloud providers are also supported.

Challenges I faced

The biggest challenge wasn't getting the AI to answer questions. It was making the whole experience feel dependable.

It's easy for a model to say, "Click the button in the top-right corner." It's much harder to consistently place a pointer on the exact UI element, on the correct monitor, at the correct scale. Mixed-DPI Windows displays made that especially difficult, and solving it meant treating screenshot coordinates and display coordinates as completely different coordinate spaces.

Packaging was another major hurdle.

Nimbus bundles a desktop UI, audio input and output, local speech models, PyAV, ONNX Runtime, Kokoro, phonemizer support, and several native Windows dependencies. Getting everything working inside a PyInstaller build meant tracking down missing DLLs, plugins, hidden imports, model files, and runtime dependencies. To make deployment easier to validate, I added a --selftest mode that checks runtime imports without requiring a microphone, internet connection, or visible interface.

There were plenty of smaller issues too. I had to fix a Qt DPI-awareness warning during startup, resolve Knowledge Folder path problems on managed Windows profiles, tighten hotkey validation, protect debug logging from failing before interactions began, and generally make failures feel less disruptive. Those improvements eventually became features like first-run onboarding, tray pause and resume, audio feedback cues, and non-blocking overlay error notifications.

What I learned

One thing this project reinforced is that building a good AI product isn't just about choosing a capable model. Everything around the model matters just as much.

For Nimbus, that meant bringing together voice input, visual pointing, screen annotations, spoken feedback, memory, privacy controls, and graceful error handling into a single experience.

Direct coordinate grounding turned out to be much more effective than relying only on grid selection, but having a verification pass and fallback system made the overall experience much more reliable. Likewise, the per-monitor overlay architecture wasn't just a nice improvement—it became essential for making Nimbus work properly on real Windows setups instead of only in ideal demo environments.

I also gained a much deeper appreciation for testing desktop applications. Nimbus includes an extensive automated test suite covering hotkeys, coordinate conversion, overlays, memory, settings, provider routing, diagnostics, update checks, packaging behaviour, and startup paths. GitHub Actions runs the test suite on Windows with Python 3.13, and the release workflow automatically builds and publishes installers.

Nimbus was developed with OpenAI Codex powered by GPT-5.6. Codex helped implement features across multiple files, investigate real stack traces, improve packaging, write tests, set up CI and release automation, and think through architectural decisions. That development workflow is completely separate from Nimbus's runtime AI, which uses a configurable vision model to understand the user's screen and provide grounded assistance during actual interactions.

Built With

Share this project:

Updates