Inspiration
I always wanted to build a personal assistant specifically built for my need. I was experimenting with multiple LLMs and context length was a limiting factor, as the conversations will be continuous and will be handling files as well. With Gemini's 1 million token it was made easy.
What it does
This a digital assistant that sits in your desktop with access to your local system and files. It has some basic inbuilt functionalities like managing todos/reminder, web browsing, file search (vector search) and analyse image. Added to that users can define their own functionality into the assistant giving infinite possibilities. The assistant develops a personality overtime based on the conversation it has with the user.
How we built it
This a built with Gemini 1.5. Both Google AI and Vertex AI is supported as of now. It heavily relies on the Function Calling feature of gemini. The todos are stored in a vectorized format in a local database. Vector file search support image and text only for now. Both these are accomplished with embedding option available with Google AI/Vertex AI. Users can analyse photos, which is done with multimodal capabilities of Gemini.
Challenges we ran into
The first issue I ran into is the 2 RPM limit of Gemini 1.5 in Google AI. Due to which I have to switch to Vertex AI for development. Hopefully when pay as you go is released this will be solved.
Since Vertex Gemini 1.5 is still in preview I faced a lot of stability issues where I got BLOCK_LIST for simple texts or got responses without candidate. Hopefully these will be solved when it is out of preview.
Accomplishments that we're proud of
I was able to complete this POC in a short period of time. This is something I wanted to build for a long time and happy to finally start doing it.
What's next for My Assist
- Support Gemini 1.0 for a low cost support.
- Also build a version with finetuned Gemma for an offline assistant (which is my ultimate goal).
- Add voice support.
Log in or sign up for Devpost to join the conversation.