ClawPhone

The 24/7 AI assistant that lives in your pocket. No laptop always on is needed at home, or cloud servers. Capture everything, the phone will understand, summarize, classify and schedule for you.

Comment

Inspiration

Every day I try all the new tech available from the first moment, when I tried Clawbot/Hermes, blow my mind, but every time I explain this to all my friends and colleagues, the first stopper was to have a laptop 24/7 at home on. In that moment I was thinking, where can everyone have a device always, on... of course, their phones.

What it does

So in that moment I had this crazy idea to create an app that you can tell everything, like I was doing with my agents, and this app summarize, schedule, keep track, of everything in my life. Clawbot is doing exactly that. Logging EVERYTHING, where we don't wait time finding different apps to log something, for example to keep tracking our daily meals, just a pic and log, for a meeting, just record and save. For financial tracking, just upload your statements or log your spending. All this integrated with Siri.

How we built it

All this logs will be processed, categorized, and summarized with a new local model in the iPhone called Apple Foundation Models, who also have vision and audio capabilities, who will extract and analyze your text, audios or photos and categorize in different ways, and "vaults", where we can also create more custom vaults and explain the model what is about the new vault so the model will know when should use that new model. The default models will be

  • Health
  • FInance
  • Personal
  • Work

Challenges we ran into

First challenge I got was to make this app run and debug, because the AI Models in local you can run only in a device, not in the simulator, so with the help of ChatGPT, I was able to enable this deployment into my phone and continue the project working there.

Second challenge was the slowness of the model with the cold start, where ChatGPT also helped me to handle that in a different way where I was able to start the models and have it ready for the moment the user start using it.

Third challenge was the design, I ran into different design rounds with ChatGPT because the app have too much data, a little overwhelming for the user, but we finally got a good one.

Accomplishments that we're proud of

The first accomplishment that I found and really impressed me was the first big shot. I created a very huge first prompt to develop this app, and from all the time that I have been working with different models, never was able to run a swift project under the 10th prompt, because there is always an issue with the code or project file for Xcode, when I saw the app running after hours running the first prompt, I got very excited to do this hackathon.

The way that I handled the compilation of the vaults when have so much data and need to summarize or compress the data, made me very proud, because in phones the context is very short so I really need to keep it simple to make it work.

What we learned

This is my first time building an app end to end with ChatGPT, and im really impressed, the GPTImage is a tool that have no limits, from creating all the visual not just with vectors, the diagrams, thumbnails, and the way ChatGPT showed me different designs, is a game changer for me.

I learned that Codex is way more than "Code". I used it to clear my ideas, create documents, images, diagrams, and everything that I needed to make this project in the short time I had to make it.

What's next for ClawPhone

ClawPhone will keep improving everyday to turn into the All in One assistant that I wanted to build with server agents weeks ago. The roadmap is clear, better understanding, Soul customization, MCP connections, fetch online data, and more model improvements.

Built With

Share this project:

Updates