Inspiration
After OpenAI's GPT-4o demo, I was talking to a family friend who envisioned a hands-free, screenless tool which could read and respond to emails on his drive home from work. We wanted to create this tool and allow for people to manage their inbox without looking at a screen.
What it does
Gmail assistant takes a voice input and can summarize, write, and send emails for you, entirely through voice input and output.
How we built it
We utilized functions from the Google Cloud Gmail and Speech-to-Text/Text-to-Speech APIs. We prompted an OpenAI model to provide assistance and format certain responses in a specific way which would access Gmail. It's a locally run program in its current form, due to hiccups when developing a web app with microphone access. We also have a web version that only takes web input.
Challenges we ran into
- The Gmail API was clumsy to use when logging new users in. We had to create an interface that allowed users to give us permissions to read and send their emails.
- We ran out of time to get our web app up and running how we wanted it. We were able to develop a version that worked with text input, but the obstacle we weren't able to get around was the microphone access.
Accomplishments that we're proud of
- Neither of us had ever used Google cloud, and we're proud to have implemented a pretty smooth user experience and integration on the web.
- Our program works really well if you can get around the UX.
- This was our first hackathon! Getting a working product was a big deal to us.
What we learned
- How to use Google cloud
- How to use the OpenAI API
- How to built a basic web app
- It's really important to shoot for an minimum viable product. We got bogged down by trying to do too much at once. When we took a step back, learned the fundamentals, and then applied what we were trying to do, we made a lot more progress.
What's next for Gmail Assist
We want to have this experience be as frictionless as possible. To do this, we can
- Make a web interface with seamless audio input and output
- Fine-tune our default prompts so the output is more consistent
- Implement interruptions to the TTS so the interactions feel more conversational
- Make a mobile app interface
Log in or sign up for Devpost to join the conversation.