Inspiration
"The first- level digital divide shifts from inequalities in physical access to inequalities in material access" - Van Deursen & Van Djik
This made me think of how every technological breakthrough is followed by a new form of digital divide associated with it. From computers to World Wide Web to Smartphones. While AI Chatbots have become prevelant among a diverse demographic, AI agents despite their diverse functionalities and capabilities still are consolidated among a cream of tech enthusiasts. The major reason for this is the complexity involved with creating such Agents.
What it does
SIMA is a modular computer use agent that is meant to be accessible, adaptable, interactive and FUN!
SIMA uses a modular architecture that focuses on local machine integration without necessitating API integrations and VM deployment. SIMA registers as a Second Profile on the user's device and stores all secrets through Apple's Keychain. To increase accessibility, A VNC is established to monitor and cowork on the same profile.
SIMA skips overhead API costs and related complexity by using an MCP architecture on top of the user's coding agent,eg: Claude Code, Codex. Not only are such agents becoming more prevelant, but their tokens are often underutilized by most. SIMA builds on these trends and runs the agent headlessly to maximize abstraction
3.SIMA supports distant chat capability with agent using Photon's Spectrum framework, allowing real time task allocation from afar.
How we built it
The menu bar app was built completely with Swift And the backend was built with typescript and compiled to ES modules
Challenges we ran into
-We chose a project too complex for the timeframe -Handling binary data setting buffers, parsing through them were all time intensive and required heavy planning, understanding and logical complexity.
- From Diffie-Hellman key exchange to VNC handshake, they were all really complex to implement despite having prior theoretical knowledge about them.
- For a person who sparsely commented on code, the complexity of code made commenting integral to track the workflow.
- Few hours into the hackathon we realized we stepped into a well to deep
Accomplishments that we're proud of
- Completing the project. This is probably my favorite and most personal project for the challenge it posed and the concept of modularizing agents is still fascinating to me.
- Producing an mvp, especially because my first time I wasn't able to turn in one.
What we learned
-Apple Authentication using Diffie-Hellman key exchange -Setting up VNC client using RFB protocol -Complexity of binary handling -Parsing bytes for authentication and encoding
What's next for SIMA- Session Isolated Modular Agent
build more front end and backend parts to support customization, for different themes of AI providers. We wish to build an open source community around it where people can easily contribute and configure their agents.
Log in or sign up for Devpost to join the conversation.