Glance

Inspiration

Assistants still feel disconnected from what we are actually doing. They can answer questions, but we have to copy text, paste screenshots, explain context, and switch windows.

Glance was inspired by a simple idea: what if your assistant could understand whatever is on your screen? Studying, coding, reading, planning dinner, shopping, or doing chores should start with the same interaction: select it, circle it, or ask.

What it does

Glance is a screen-aware personal assistant for studying, work, and daily life. It lives as a floating desktop orb and understands selected text, screenshots, circled regions, annotations, and the app you are using.

It can solve math with LaTeX, turn text into Mermaid diagrams, explain or fix code, summarize content, generate images, create recipes and grocery lists, open maps, compare prices, monitor price drops, and help book restaurants.

For real-world actions, Glance uses Fetch.ai APIs and Agentverse agents to search nearby restaurants, reserve tables, compare marketplace prices, order food or products, debate topics, and track price drops.

How we built it

Glance is built with Electron, React, and a local FastAPI backend. The desktop app handles the floating orb, hotkeys, selected-text capture, screenshots, region capture, freehand annotations, and artifact rendering.

The backend uses a structured /artifact pipeline that routes each request to the right response type: math, Mermaid, recipe, image, map, booking, price comparison, code fix, and more.

We integrated OpenAI/xAI providers for model responses and image generation, plus Fetch.ai Agentverse agents through a bridge for agentic workflows like restaurant booking, food ordering, price comparison, and price monitoring.

Challenges we ran into

The biggest challenge was making Glance feel native instead of like another chatbot. We had to reliably capture selected text, screenshots, and regions across apps while handling macOS Accessibility and Screen Recording permissions.

Routing was also challenging because the same context can mean different things. A recipe could mean “explain it,” “generate an image,” “make a grocery list,” or “find restaurants near me.” We built an artifact router so the user’s intent chooses the right output.

Accomplishments that we're proud of

We are proud that Glance feels fast and natural: open the orb, select text, circle something, and get the right artifact.

We are also proud of the artifact system. Instead of one generic chat response, Glance returns useful formats like LaTeX math, Mermaid diagrams, generated images, restaurant cards, price comparisons, maps, and booking flows.

The Fetch.ai integration is another highlight because it lets Glance move from understanding the screen to taking real-world actions through agents.

What we learned

We learned that context is the missing piece in most assistants. The model matters, but the experience depends on how easily the assistant understands what the user is doing.

We also learned that structured outputs are much more useful than plain text. A math problem should look like math, a process should become a diagram, and a restaurant search should become actionable options.

What's next for Glance

Next, we want to expand Glance into a fuller desktop assistant with deeper calendar, email, browser, shopping, maps, and delivery integrations.

We also want to improve the agent layer so users can approve actions, track progress, and chain tasks together, like turning a recipe into a grocery order and a scheduled cooking plan.