Inspiration

A lot of everyday computer problems already have solutions online, but following those solutions isn't always easy. We were thinking especially about older family members, like a grandparent who finds a forum post explaining how to fix their computer's audio but still needs someone to walk them through unfamiliar settings.

We wanted to make technical tasks easier by letting people find the instructions they need and have Stride carry them out automatically, without having to work through the steps themselves.

What it does

Stride is an AI-powered macOS assistant that reads instructions from websites, forums, and documents and carries them out directly on your computer. It can navigate applications, change settings, and execute commands while following the provided steps.

Unlike a general-purpose computer agent, Stride focuses on executing existing instructions rather than independently coming up with a solution. It also checks for potentially unsafe actions, asks for approval when necessary, and verifies results where possible. Users can see exactly what was completed, what failed, and what couldn't be verified.

Our goal is to make technical tasks accessible to people who wouldn't otherwise feel comfortable performing them on their own.

How we built it

We built Stride primarily with TypeScript and Electron, using macOS Accessibility APIs and screen capture to interact with applications.

Along the way, we use Mana to design a prototype potential ui/ux models and functionalities regarding the app. Ultimately, through prompting its agent and reviewing designs we settled on the most convenient and welcoming prototype.

link

We integrated Anthropic's Claude to interpret unstructured instructions, break them into actionable steps, and evaluate the results. For visual computer interaction, we used Jev for fast, structured decisions about on-screen actions, with Claude as a fallback.

We also integrated Factory's Droid SDK into our execution system. When a user approves a task, Droid can execute approved commands through a permission-controlled interface. We built safeguards that restrict it to the specific command being executed, preventing unauthorized actions or additional commands.

Alongside this, we developed our own safety rules and verification system. Rather than relying on the AI to determine whether something worked, Stride uses actual checks whenever possible and provides a record of the actions taken and their results.

Challenges we ran into

One of our biggest challenges was latency. Computer-use agents often need to repeatedly analyze the screen, decide what to do, and wait for the interface to respond. This made some relatively simple tasks take much longer than expected. We spent a lot of time experimenting with different approaches to make these interactions faster while maintaining reliability.

Another challenge was making the agent safe and predictable. Instructions from the internet aren't always correct, current, or trustworthy. We needed to carefully control what the agent could execute and distinguish between successfully performing an action and actually achieving the intended result.

Balancing automation, speed, and safety turned out to be much harder than we initially expected.

Accomplishments that we're proud of

We're proud of building a working desktop assistant that can carry out real actions across macOS applications rather than simply explaining what the user should do. We also put significant effort into making those actions controlled and verifiable, with clear results that users can understand.

Another accomplishment was how much we improved the project through feedback. Throughout the hackathon, we spoke with mentors and judges, reconsidered parts of our approach, and used their suggestions to refine the product.

Most importantly, we built something we genuinely see ourselves and our families using beyond the hackathon.

What we learned

We learned how much engineering goes into building a reliable agentic system. Getting an AI model to suggest the next action is one thing, but making sure that action is appropriate, executes correctly, and actually accomplishes something is much more complicated.

We also learned the importance of choosing different tools for different parts of the system. Larger models are useful for understanding complex instructions, while specialized models can help make frequent computer interactions faster.

One of our biggest takeaways was that reliability and transparency matter just as much as the AI's capabilities, especially when building for people who might not recognize when something has gone wrong.

What's next for Stride

Our next priority is improving Stride's speed and reliability across a wider range of real-world instructions and applications.

We also want to test it with more people, particularly older adults and users who aren't technically experienced. This would help us identify where they still need assistance and measure how often Stride can successfully complete tasks, how long those tasks take, and when human intervention is necessary.

Beyond that, we want to improve the installation and permissions experience, expand application support, and eventually explore bringing Stride to other platforms, including iPad.

Our long-term goal is to make everyday computer tasks easier to complete independently, without sacrificing safety or control.