Salyn - Attachment Buddy using ChatGPT and Deeplabv3 + Nylas

Uses AI to search through attachments on a Chrome New Tab Interface. Makes it simple to search through large piles of images like scanned documents, partial text pdfs or photos without subject lines.

Comment

Inspiration

As an independent agency we need to go through multiple scanned documents in a single day. As people with photo crazy families, we cannot count the number of times we've been sent emails without a proper subject line, but with a large number of photo attachments that are then deleted from devices that sent them. This is a problem statement that can only be solved through AI, and luckily the Nylas API provided the absolute correct answer for us.

What it does

Simply put, using ChatGPT for the frontend allows for multiple tag generation to enable fuzzy search of images and other attachments on the backend. This, combined with a fast running custom DeeplabV3 model, allows images to be tagged and stored in a frontend DB like SQLite to be approx searched later.

How we built it

We used Nylas Authentication and Attachment APIs to login to the user's email and then fetch his attachments on a rolling bases. This was then indexed and stored into a local DB, and inferenced on Tensorflow JS version of a Deeplab V3 model trained on ImageNet and Coco Stuff datasets. At search time we used ChatGPT in order to expand the search query into a set of terms that could be fuzzy matched on the local DB, this is done quickly and efficiently by using Nylas API.

Challenges we ran into

The Nylas API was excellent for this purpose, and the major challenge was the OpenAI api to figure out what prompt engineering could be done to enrich a search term with several fuzzy matchable generic strings.

Accomplishments that we're proud of

Integrations that we accomplished using two diverse sets of APIs and Tensorflow JS on a desktop Chrome environment.

What we learned

AI might be here to stay, but at least for practical use cases they're not the easiest to implement in.

What's next for Salyn - Attachment Buddy using ChatGPT and Deeplabv3 + Nylas

Implement the Deeplab model on more platforms, utilize more of the API

Built With

Share this project:

Updates