widget mode

FRIDAY AI — An Agentic Vision Assistant

Inspiration

FRIDAY was inspired by the idea of building an AI assistant that does more than answer questions. I wanted to create a system that can observe what is happening on a computer screen, identify potential problems, reason about what might have gone wrong, and help resolve issues safely.

Many AI assistants focus on generating responses, but I wanted to explore a different approach: connecting computer vision with reasoning, decision-making, and action verification.

This idea led to FRIDAY's agentic vision workflow, which aims to transform visual information into useful, controlled actions.

What It Does

FRIDAY is an AI assistant project that combines voice-enabled interaction with computer vision and an experimental agentic workflow for screen troubleshooting.

Its vision capabilities use OpenCV-based processing to work with visual inputs, including screen content, images, and video. OCR support helps extract text from visual information.

One of the project's key features is its agentic vision loop. Instead of stopping after identifying a possible problem, the workflow is designed to:

Observe: Examine screen information and look for potential errors or unexpected states.
Diagnose: Analyse the available evidence to determine what may have gone wrong.
Propose: Suggest a possible corrective action.
Request approval: Allow the user to review and approve an action before it is executed.
Act: Carry out the approved action through the supported interaction mechanisms.
Verify: Check the resulting state to determine whether the issue appears to have been resolved.

This observe–diagnose–act–verify approach is the central idea behind FRIDAY's vision workflow. It aims to make AI-assisted troubleshooting more systematic, transparent, and safety-conscious.

How I Built It

I developed FRIDAY as a modular AI assistant, with a desktop interface and a dedicated Vision page for its visual capabilities.

The project's technical approach brings together several components:

Python: The foundation for implementing and coordinating the assistant's functionality.
OpenCV: Used for computer-vision processing and visual analysis.
OCR: Supports the extraction of text from visual content.
AI reasoning: Helps interpret detected problems and develop possible solutions.
Desktop interaction mechanisms: Support controlled interaction with the computer where the required integration is available.
Agentic workflow: Connects observation, diagnosis, proposed actions, approval, and verification into a structured process.

A major design consideration was separating the detection of a possible problem from the decision to take action. This helps avoid treating every visual anomaly as a confirmed error or executing a proposed fix without user oversight.

The project also includes a simulated-desktop benchmark intended to evaluate aspects of the workflow, including detection, verification, recovery, and safety.

Challenges I Faced

One of the main challenges was coordinating multiple stages of an AI workflow. Detecting something unusual on a screen is not the same as understanding its cause, and proposing a solution does not guarantee that the solution will work.

Each stage needs to provide useful information to the next. Otherwise, an incorrect diagnosis can lead to an ineffective action or an inaccurate conclusion about whether the problem has been resolved.

Another challenge was safety. An AI assistant that can interact with a computer must distinguish between suggesting an action and actually carrying it out. FRIDAY's workflow therefore incorporates user approval and restrictions around destructive actions.

Reliability was also an important consideration. Visual systems can produce false detections, so the assistant needs to account for uncertainty rather than assuming every detection is correct.

Finally, evaluating the complete workflow requires more than checking whether individual components run successfully. The process must also be assessed as a whole, from the initial observation to the final verification.

What I Learned

Building FRIDAY has taught me that useful AI systems depend on more than the intelligence of an individual model. Their effectiveness also depends on architecture, integration, safety, and the quality of the feedback loop.

I learned to think of computer vision as more than image processing. Visual information can become part of a larger decision-making process when it is connected to reasoning and carefully controlled actions.

I also learned the importance of verification. An assistant should not simply assume that an attempted fix worked; it should examine the resulting state and assess whether the intended outcome was achieved.

The project has strengthened my understanding of modular development, AI workflow design, and the challenges of making autonomous behaviour reliable and safe.

What's Next?

FRIDAY is an evolving project, and its agentic vision workflow is an important step towards more capable AI-assisted computer interaction.

Future development will focus on testing the workflow in real desktop environments, improving the accuracy of visual error detection, strengthening diagnosis and recovery, and making verification more reliable.

I also want to continue improving how FRIDAY coordinates perception, reasoning, voice interaction, and task execution while maintaining appropriate user control.

The broader goal is to build an assistant that does not merely describe a problem, but can help investigate it, propose a solution, and check the result.

FRIDAY explores a simple but powerful idea: computer vision becomes more useful when an AI system can connect what it sees with what it reasons, what it does, and what it verifies.