Piwi

• 

  Piwi

Inspiration

Piwi started from a simple idea: instead of using an AI only to generate text or answer questions, I wanted an AI that could actually do things on a computer.

A natural-language instruction such as creating a file, finding information online, modifying an existing project, or combining several operations should be turned into real actions performed on the user's machine.

The goal behind Piwi is therefore to move from a conversational AI to a more autonomous desktop agent.

What it does

Piwi is an autonomous AI desktop agent designed to execute tasks directly on a computer from natural-language instructions.

Depending on the request, Piwi can:

• understand the user's objective and determine the actions required;
• use tools to search and retrieve information from the web;
• read, create and modify files;
• inspect directories and workspaces;
• use file and directory selectors when the target is ambiguous;
• execute several actions sequentially as part of the same task;
• keep track of the execution context and tool results;
• operate inside a selected sandbox to limit where file operations can take place.

For example, a user can ask Piwi to create a file and fill it with information retrieved from the web. Piwi can retrieve the information, process the result, and then create the requested file rather than simply explaining how the user could do it manually.

Gemini is integrated into Piwi as an LLM backend and is used for tool-based reasoning and decision making.

How we built it

Piwi is built as a modular Python application with a desktop interface and an agent-oriented execution architecture.

The system is divided into several components, including:

• an LLM layer supporting Gemini;
• a tool catalog defining the capabilities available to the agent;
• a tool agent responsible for executing multi-step tool interactions;
• an action system that converts tool requests into typed executable actions;
• a plugin architecture for operations such as file manipulation, directory inspection and HTTP requests;
• a sandbox and context system used to maintain the active workspace and execution state;
• an agent loop responsible for managing execution, blocking states, user interaction and resumption;
• a desktop GUI for interacting with Piwi and monitoring execution.

The Gemini integration uses the Gemini API through Google's OpenAI-compatible endpoint. Piwi also preserves Gemini tool-calling metadata such as Gemini 3 thought signatures when continuing a tool conversation.

A major part of the development work has been making the agent reliable when moving between reasoning, tool execution, user interaction and execution resumption.

Challenges we ran into

One of the biggest challenges was making tool calling reliable across multiple turns.

A tool-based agent needs to preserve the conversation state correctly between an LLM response, the execution of a tool, and the next LLM request. With Gemini, this also required correctly preserving the thought_signature returned with tool calls.

Another challenge was handling ambiguous targets.

For example, when a user asks Piwi to modify "my todo list" without giving an exact filename, Piwi should not blindly guess. The system can instead ask the user to select the appropriate project or file and then continue execution using that selection.

We also had to deal with execution state and resume behavior. A selector temporarily blocks execution, and Piwi must wait for the user, receive the selected resource, restore the correct context and continue the original task without opening the selector again.

Finally, API limitations and quota errors highlighted another important aspect of building an autonomous agent: failures from the underlying model must not silently turn into false claims of success.

Accomplishments that we're proud of

We are proud that Piwi has evolved beyond a simple chatbot into a system capable of connecting:

natural-language instructions → reasoning → tool selection → real execution → context → continuation

The project now has a modular tool architecture, sandboxed file operations, user selectors, execution state management and Gemini-based tool calling.

We are particularly proud of the work required to make Piwi continue a task after a user interaction instead of treating each interaction as a completely new request.

What we learned

Building Piwi taught us that the difficult part of an AI agent is not only the LLM itself.

A useful autonomous agent also needs reliable tool interfaces, strict action validation, execution state, context management, error handling and a way to recover from interruptions.

We also learned that tool calling introduces an entirely different set of engineering problems compared with a traditional conversational application. Preserving state between model calls, validating actions, handling ambiguous targets and preventing execution loops are all critical to making an agent dependable.

Working with Gemini also showed how important provider-specific details can be when building multi-turn tool interactions.

What's next for Piwi

The next step for Piwi is to make the agent increasingly reliable and capable across a wider range of real desktop workflows.

The long-term goal is to move toward an assistant that can take a high-level objective, plan the necessary steps, use the appropriate tools, recover from failures, interact with the user only when necessary, and complete the task end-to-end.

Piwi is being developed as a general-purpose foundation for autonomous computer interaction rather than as a single-purpose application.

Built With

• agenticai
• aiagent
• autonomousagent
• desktopautomation
• gemini
• generativeai
• google
• llm
• openaiapi
• pyqt
• python
• toolcalling
• webautomation