Generic Macro

Inspiration

I wanted to build a fine tuned agent (post-training). This is trivial for a specific task when using skills or tools. The issue with tools and skills is that they aren't general. What a user needs for a task to be completed is for the agent to figure out how to do that in a replicable way. Tools can do this once, but tool code is not formatted with simplicity in mind. For a computer use agent you don't care about the exact implementation and just care about interactions. Using a macro means the agent is essentially making its own tool, but it allows for further composition with other macros to complete more complex tasks without struggling on smaller tasks along the way.

What it does

Run a command from your terminal in natural language and a macro is created. Macros are similar to agent tools, but meant to be more human readable (similar to WoW macros) and defined compositionally so long horizon workflows can be broken down into simpler tasks. When workflows need to be repeated their structure is already in its simplest form. If there is only a small change in the workflow then the macro can be modified just in that way by the agent to perform the correct task. This means a task is only ever difficult once. By being human readable users can modify a macro if they believe it could be improved without needing substantial coding knowledge. Just as agents abstract away code, we also need a way to abstract away agent interactions that involves minimal code.

How we built it

Antigravity all the way. No, but seriously I used GPT-4o-mini for LLM calls since it is cheap and responsive with multimodal inputs. The LLM interprets a natural language query (the macro to be created) and is also given context about the user's computer so it can be used as a computer use agent. OCR is used to help the agent navigate the screen before interacting using the macro.

Challenges we ran into

The biggest struggle was for the agent to remember that it needs to enter a specific environment before an action. For instance, you could use this agent as a coding agent even though it doesn't have terminal access and it would write correct code but forget that it needed to navigate into a file before updating a line.

Accomplishments that we're proud of

I was excited to think about tools in a new way. As I've experimented more with agents and tried to take the perspective that LLMs are a way to translate natural language to machine code I've realized that we really don't know how to use agent and existing implementations are just our best guess, not the end product.

What we learned

I learned how to use the OpenAI API. Normally I use local models or NRP hosted models.

What's next for Generic Macro

Training models to perform general tasks should be a lot more effective, it just requires good training examples. Generic Macro is not trying to mimic a human like some computer use agents. Instead the goal is to perform tasks that humans already perform.

Built With

• agents
• ocr
• openai
• powershell
• python