Nimbus

Point, ask, learn. Tutorials tell you where to click. Nimbus takes you there. Hold a key, ask about anything, and it explains out loud and shows you where. Studying or stuck, learn in place.

Comment

Nimbus

Try It out: asknimbus.org

Inspiration

There is a gesture people make when software defeats them.

They stop typing. One hand comes off the keyboard and hovers near the screen, index finger out, not quite touching the glass. And they say: "this thing."

Not the name of it. Just "this thing, here."

I have watched people do this for years without thinking about it, and then one day I did think about it, and I could not stop. Because that gesture is the entire problem. If you cannot name the button, you cannot search for it. You cannot describe it to a chatbot either, since describing it is precisely the thing you are unable to do. The answer is out there. It is just that the door has a lock on it and the key is the answer itself.

So what happens instead? A tutorial video gets opened. It gets scrubbed. Two minutes go by, then five, then the person gives up and does the task some longer way, or does not do it at all. And the thing that beat them was never the concept. They understood the concept fine. They lost to vocabulary.

The premise came from seeing HeyClicky, which had the right idea on what I thought was the wrong platform. A browser extension cannot see the application people actually get stuck in. Think about which software genuinely ruins someone's afternoon. It is never the popular thing with the good tutorials. It is a university's own admissions portal. A lab instrument's control panel. A video editor with three hundred buttons and no labels. An accounting package a small shop has run since 2009 and nobody has ever written a single page of documentation for.

All of that is a native Windows window. A web product cannot help with any of it, not because of effort, but because of architecture.

I am in Pakistan. Practically every student machine here is a Windows laptop, often shared, often not new. So the platform decision and the market decision were the same decision, and both of them pointed at the hard option: Win32, DPI-aware overlays, a frozen installer, all of it.

And then there was one line I wrote in my notes on about day three, which turned out to be the whole product:

"Click the publish button, top right" is advice. A pointer landing on the publish button is an answer.

Everything after that was just me trying to earn that sentence.

What it does

You hold Ctrl + Alt + Space. You ask out loud, in normal words, the way you would ask a person sitting beside you. You let go.

There is a small chime. Then, about a second and a half later, a voice starts answering. And while it is still talking, your actual mouse cursor lifts off and flies across the screen to the thing it is describing.

That is the moment. The first time I got it working I made it point at the same button maybe forty times in a row, like an idiot, just to watch it happen.

It is not only a finder, and I want to be clear about that, because "AI that finds buttons" undersells it badly. Turn on teaching mode and it draws on your screen: a box around a control, the rest of the display dimmed so one region becomes unmissable, numbered badges when a task has an order to it, so a nine step export becomes nine markers in sequence instead of a paragraph you have to keep re-reading.

Point it at work you made and ask why it is wrong, and this is where it stops feeling like a utility. It circles the exact step where you dropped the chain rule and draws an arrow to where the correction belongs. It finds the cell whose formula is pointing at the wrong range. The line of code mutating the value everything downstream depends on. The node in an editing graph that is why your export came out looking flat and grey.

You can drop your own PDFs into a folder named after a program, and it becomes competent at software no model on earth has ever seen. You can say "quiz me" and days later it will ask you to show it where something is, then grade where you actually point, on your actual screen.

And it will never click for you. Not once. That was not a limitation I ran out of time to fix. That was the point, and I will come back to it.

How I built it

Python, with PyQt6 driving a click-through per-monitor overlay, mss for DPI-aware capture, Win32 for working out which application is in front, and Gemini through the google-genai SDK. The business side is a Next.js app on Vercel: accounts, Stripe, EasyPaisa, and licences signed with Ed25519 so they verify with no network at all, because a tool that dies on a flight is worse than one that gets pirated.

Everything in the pipeline is built around one number I refused to negotiate on: a second and a half from releasing the key to hearing the first word. So nothing waits its turn. Speech recognition finalises while the screen is being captured. The answer streams in, and I flush at sentence boundaries so the voice starts on sentence one rather than politely waiting for the paragraph to finish.

The piece I am quietly proudest of is how a coordinate travels. It does not come back as a [POINT:x,y] tag buried in prose that I then parse with a regex and pray over. It comes back as a declared function call:

point_at(y=412, x=1180, label="export button")

Note that \(y\) and \(x\) are separate typed integers rather than an array, specifically so the model cannot quietly swap them on me. They arrive normalised to \([0, 1000]\), with \(y\) first, and I convert to pixels:

$$x_{px} = \frac{x_{norm}}{1000} \cdot W, \qquad y_{px} = \frac{y_{norm}}{1000} \cdot H$$

That reads like a footnote. It is not. It is the difference between a contract and a convention, and it is the reason the pointer lands on the button instead of somewhere in its neighbourhood.

Challenges I ran into

The silence. This one nearly broke me, and it happened at the worst possible time: right after everything started working.

I had the pointer. I had the voice. I put them together and the product got worse. The cursor would fly beautifully to exactly the right control, and Nimbus would say nothing at all. Total silence. Someone holds a key, asks a question with their actual mouth, and gets a cursor twitching across a mute screen.

I assumed it was my bug. It was not. I sat down and measured it properly, across thinking budgets of \(0, 64, 128, 256\) and \(512\), and the pattern was perfect and merciless: whenever the model chose to call the function, it emitted zero text. Gemini returns prose or a function call in a turn. Never both.

I remember the specific feeling of that, which was not excitement at a discovery. It was dread, because it looked like I had to choose between the two halves of my product, and either choice made it pointless.

What broke the deadlock was giving up on it being one request:

# Two calls, fired at the same instant, on different threads.
speech   = generate(contents, tools=None)          # no tools -> prose is forced
geometry = generate(contents, tools=[point_at],    # tools only -> coordinates
                    thinking_budget=0)

One declares no tools at all, which forces prose, and streams straight into the voice. The other declares tools only, runs at a minimal reasoning budget, and lives on its own thread. So the cost is not the sum of two calls:

$$T = \max(T_{\text{speech}},\; T_{\text{geometry}})$$

The geometry resolves while the first sentence is still being spoken. It is the best architectural decision in the project and I did not design it. I was forced into it by a measurement that disagreed with me, which I have come to think is the only reliable way I ever improve anything.

Nimbus kept photographing itself. The overlay has to vanish before every screenshot, wait for the Windows compositor to actually catch up, let the capture happen, then reappear. Miss that and the model sees Nimbus's own cursor and starts helpfully pointing at it.

That cycle costs about \(55\) ms, which out of a \(1500\) ms budget feels like a personal insult. I spent the better part of a day trying to delete it using Windows capture exclusion. It cannot be done. Exclusion fails on a layered window, and the overlay has to be layered to be translucent at all. Two mutually exclusive requirements, no clever way through.

I wrote the failure down in my design notes, in detail, including why it fails, so that a version of me three weeks later would not cheerfully waste another day on it. That document became one of the more valuable things in the repository. Then I found the \(55\) ms somewhere else entirely.

Three coordinate spaces, one of which changes per monitor. Physical pixels across the whole virtual desktop. Qt's logical units, which differ per screen because DPI differs per screen. And whatever resolution the model believes it was looking at.

A single silent transposition anywhere in that chain does not throw an error. It just offsets every point Nimbus will ever make, forever, by an amount that looks almost right. I lost more hours to that than to anything else, and the fix that helped most was not clever code. It was naming things so pedantically that the mistake becomes hard to type.

The single largest accuracy improvement in the entire project, in the end, was getting the aspect ratio of the screenshot right. Not a better prompt. Not a better model. Arithmetic.

The small humiliations. A freshly issued 7-day trial displayed "6 days left." The token expires in \(6.9999\) days and I floored it. It took me an hour to find and about two seconds to lose all confidence in myself, because nobody trusts anything else on your screen after they catch you being wrong about six.

I once ran a complete round of manual testing, well over an hour, on a build that did not contain the change I was testing. Everything behaved exactly as it had before, and I could not work out why, and the answer was that I was testing yesterday's executable. Searching the packaged file for a string proves nothing, since the modules are compressed inside it. So now there is a verification step that opens the shipped binary and reads its own code objects to answer the question "is my change actually in here." That tool exists purely as a monument to one wasted afternoon.

Two microphones fighting each other, where the speech-to-speech path and the speech recogniser both grab the input device and neither works, with no error message worth reading. A model name an agent recommended with total confidence that simply does not exist, which failed at connect and taught me to check the live model list instead of trusting anything that merely looks plausible. A frozen bundle that was 1.1 GB until I worked out which scientific libraries were being dragged in by accident.

None of that is glamorous. All of it is what building this actually was.

Accomplishments that I'm proud of

The pointer lands. After all the DPI arithmetic and the coordinate spaces and the transposition paranoia, somebody asks where a thing is and the cursor goes there. I have shown this to people who have used computers for twenty years and watched them go quiet for a second.

2,030 tests, and the interesting ones are strange. They do not test behaviour. They test intentions. That no emoji ever appears in the interface. That one module never imports cryptography. That no private key exists anywhere in the repository. That a base prompt is only ever appended to, never replaced. Agents wrote most of this code, and what an agent does most dangerously is not write a bug. It is quietly reverse a decision you made for a good reason, and then tell you it succeeded.

The chat panel is invisible to the model and visible to you. Verified at zero of 120,000 marker pixels.

My entire Gemini bill for 90 days was $16.89, building a product that sends screenshots on literally every interaction. That is not luck, it is reasoning budgets set to zero for perception questions and knowledge base content cached against the model instead of resent. On a full-size knowledge base, 10,008 of 10,013 prompt tokens came from cache.

Privacy that is counted, not promised. With a password manager in front, Nimbus answers without taking a screenshot at all, and shows you how many times it has refused. I deliberately did not let a model make that call, because a privacy guarantee that holds most of the time is not a guarantee, it is a hope.

And then the part that has nothing to do with code. Two teachers I know well liked this enough to tell their own classes about it. That was it. No ads, no launch, no list. Twenty-three students then chose to send money by EasyPaisa, type in a transaction reference from their receipt, and wait for me to confirm it by hand before their licence arrived.

My brother was the first person to pay. It was a test transaction, honestly, to check the checkout worked end to end, and I have declared it as such. But I still have the notification.

Nobody clears that much friction for something they were mildly curious about. That is the number I care about, and it is not really a number.

What I learned

Measure. Do not ask. Every figure in this submission came from running the call and writing down what happened. The two-call architecture, the reasoning budgets, the capture cycle I could not delete. All of it came from being wrong out loud and finding out quickly.

Negative results are results, and you should ship them honestly. I built Google Search grounding, measured it properly, and it made answers worse under my own prompt. It ships turned off, with a tooltip that says so. Every failed attempt I bothered to write down has saved me more time later than the successes did.

Restraint is a feature. I could have built the agent that clicks for you. It demos better. It is easier. And it teaches nothing, because it makes people dependent on the assistant instead of competent with the tool. The hand on the mouse has to be the student's own, since that is where the learning actually gets stored. Every time I have been tempted to relax that, it has been for my benefit and not theirs.

Distribution is a person, not a channel. I had no audience, no following and no money for ads. I had two teachers. And a teacher will not put their reputation behind something that embarrasses them in front of thirty people, which makes their word worth more than any amount I could have spent.

AI wrote most of the lines and none of the decisions. The genuinely interesting engineering was building the checks that make the first half of that sentence safe to say.

What's next for Nimbus

In two weeks I walk into a room of about 100 students and demonstrate this myself, on their software, for the first time. Not through a teacher vouching for me. Just me and a room that has never heard of me.

I am nervous about it in a way I have not been about any bug in this codebase, because it tests something no test suite can. I am handing out free subscriptions to part of the room, and what I actually need is not the sign-ups. It is the two numbers I do not have: how many pay when the free month runs out, and the full list of everything that goes wrong on machines and software I have never touched. The pointer landing two buttons off. The question the model misheard. The app it has never seen. The moment somebody gives up and reaches for the mouse.

After that, in order: code signing, because an unsigned installer makes Windows warn strangers and that quietly costs real installs. An edition where inference runs on my own infrastructure so a student never has to see the words "API key," which is the single biggest thing standing between this product and people who are not developers. Urdu, since the model already handles it and my students already switch language mid-sentence without noticing. Institutional seats with a console for the teacher. And knowledge packs that teachers can write themselves and earn a share of, because I am never going to author authoritative material on three hundred pieces of software, and the people who have taught them for a decade already can.

But mostly I want to keep sitting in rooms and watching for the gesture. The hand coming off the keyboard, the finger hovering near the glass, "this thing, here."

I want to get to the point where nobody has to make it.

Built With

Share this project:

Updates