Inspiration
We interact with computers through countless applications, windows, files, and browser tabs, yet getting work done still requires constant switching, clicking, and typing. While AI assistants have become remarkably capable, connecting conversational intelligence to the real desktop remains a meaningful challenge.
keyboard.wtf started with a simple idea: your computer should adapt to how you communicate, rather than forcing you to adapt to it.
With this project, we explored how that vision could extend to Alexa+. What if a conversational assistant could understand a user's desktop context and invoke real, permission-controlled computer actions instead of being limited to conversation alone?
What We Built
keyboard.wtf × Alexa+ is an integration prototype that explores connecting an Alexa+-style conversational experience to a Windows desktop agent through the Model Context Protocol (MCP).
The architecture separates the conversational client from the desktop capabilities. An MCP server exposes selected tools through Streamable HTTP, allowing the simulated client to discover available capabilities, invoke actions, and receive structured results.
Through keyboard.wtf's existing desktop automation foundation, the project explores workflows involving application launching, desktop context, selected text, file discovery, and reusable routines.
Rather than duplicating the Windows automation logic inside the conversational interface, the integration is designed around a clear tool contract. This separation makes the architecture easier to test, maintain, and adapt to another compatible MCP client.
Because access to the official Alexa+ developer environment is restricted, our hackathon experience uses an independent simulator. It demonstrates the intended integration architecture without claiming to be connected to Amazon's private production systems.
How We Built It
The project builds on keyboard.wtf's Windows desktop application and automation architecture, with C#/.NET, MCP, Streamable HTTP, and an AI-powered conversational layer.
We focused on separating three responsibilities:
- Conversation and reasoning: interpreting natural-language requests and deciding which capabilities to invoke.
- MCP integration: exposing discoverable tools through a standard protocol rather than coupling desktop functionality to one assistant.
- Desktop execution: delegating actions to the existing Windows automation layer, with appropriate permissions and truthful execution results.
This approach keeps the integration model-independent: the desktop capabilities should not depend on Gemini being the conversational client. A compatible future Alexa+ client could use the MCP interface without requiring the desktop automation layer to be rewritten.
Challenges and What We Learned
The central challenge was not simply adding voice to desktop automation. It was designing a reliable boundary between an AI assistant and a real operating system.
Desktop actions can fail, applications may not be available, and a requested operation may require clarification or user confirmation. Reporting that an action was attempted is not the same as verifying that it succeeded.
The project reinforced the importance of clear tool schemas, protocol compatibility, permission boundaries, cancellation, error handling, and separating simulated interfaces from real integrations.
We also learned that a local prototype and a production Alexa+ integration are different engineering milestones. A future deployment would require the appropriate Amazon developer access, secure remote connectivity, authentication, and end-to-end compatibility testing.
What's Next
Our goal is to make desktop capabilities accessible through a clean, well-documented MCP interface that another compatible assistant can discover and use.
The longer-term vision is a computer that is not merely something an AI talks about, but an environment it can understand and help operate—with user control, transparency, and security built into the interaction.
Log in or sign up for Devpost to join the conversation.