LoopdeLoop - Computer Use Agent SOTA Attempt

• 

  hud page showing parallel processing of my agent for different tasks

Inspiration

I was inspired to build loopdeloop because I love how cool agents are. I really like that agents are basically an artificial human and you can teach it / raise it to do anything.

What it does

LoopdeLoop is a computer use agent built using cua and evaluated using hud. Essentially, loopdeloop can take any prompt, and take control of your computer's gui in order to accomplish the task laid out by that prompt.

How I built it

LoopdeLoop builds on the CUA framework but layers in new engineering around callbacks, multi-model composition, and config-driven design. It has a custom recovery system that learns from mistakes, tuned prompting for OSWorld benchmarks, and HUD integration for evaluation. All of this combines into a more resilient, production-ready architecture.

Challenges we ran into

The biggest hurdle was GNOME’s Activities overlay, which kept trapping the agent. I fixed it with programmatic hot corner prevention. Getting reliable element detection also took work, and I had to add fallback strategies and trajectory learning to keep workflows on track. Resource tradeoffs and framework integration pushed me to build config-based routing and abstraction layers.

Accomplishments that we're proud of

I’m proud that LoopdeLoop hit competitive results on OSWorld, especially with its callback system and recovery design. Solving the GNOME overlay problem felt especially novel, since it’s an issue every Linux-based agent runs into.

What I learned

Building agents taught me that environment quirks can completely break performance if you don’t engineer around them. I also saw how important it is to combine vision and language models carefully, and how learning from failure patterns is just as powerful as optimizing for success.

What's next for LoopdeLoop - Computer Use Agent SOTA Attempt

Next, I want to boost visual grounding with stronger models, extend cross-platform support, and add reinforcement learning so it can improve autonomously. I also see value in building domain-specific modules, enabling multi-agent collaboration, and eventually open-sourcing the callback and recovery system so others can use and extend it.

Built With

• agent
• cua
• hud
• osworld
• python