The Daily Pickle

Inspiration

I've seen a lot of AI/ML/SWE related daily digests, so I wanted to, initially, create one for the field I came from, in neurobiology. Then I realized it would be even better if I could create one that's customizable, so anyone can type in their field or topic of interest and receive a tailor-made email every morning on the latest news on this topic.

What it does

The user enters their email and their topic of interest. Every 24 hours, they will receive a daily digest of the latest news regarding their topic of interest. (For testing purposes, there is no email since it AWS SES requires verification; however, email functionality is implemented and recorded in the demo video).

How we built it

When the user submits their email and interest, it is stored into a table in DynamoDB through a Lambda function. Every 8AM PDT, EventBridge triggers a Lambda function to go through every user and generate a daily digest based on their topic of interest. Bedrock is used as the LLM provider, with Llama 3.1 80B as the model of choice. The LLM analyzes the user’s topic of interest and generates keywords optimized for news search. The first 5 keywords are used to query the News API’s /everything endpoint, and top 10 articles retrieved for each keyword are stored in a list. These articles are then scored by counting keyword occurances (with different weightings depending on whether the keyword appears in the title, description or article). The top 10 articles are selected and fed back to the LLM for summarization. The LLM then transforms these articles into a personalized daily digest following a structured format. These digests are stored in a separate DynamoDB table with a “ready_to_send” status. At 9AM PDT, another EventBridge event triggers a Lambda function to send the pending digests using Amazing SES.

Challenges we ran into

A big challenge was being able to get the most up to date news, without accidentally picking up noise. For example, we'd often get random articles containing keywords that are not related. That's why we came up with a weighted scoring method to only include the most important articles. There are also situations where keywords were too complicated. For example, it was hard to find articles containing "artificial intelligence legislation", exactly. We fixed this by some prompt engineering to ask the LLM to come up with optimized search terms instead of specific, technical jargon.

Accomplishments that we're proud of

This project started out very small, using just Lambda and Bedrock, and I'm very proud of how we were able to grow this to use so many other AWS services, while staying true to the core. I'm proud of coming up with the weighted scoring algorithm, as well as getting SES to work and learning everything DNS-related last minute to implement the emailing feature.

What we learned

A big thing I learned is that we don't need a server for everything! I had no idea Lambda functions could be so convenient, and how easy it was to combine it with other services.

What's next for The Daily Pickle

So much more. The Lambda functions could be much more optimized. At the moment, all digest generation and article scraping is done using one Lambda function. The current version of the program cannot handle a lot of users, as the function will just time out. We can fix this by linking multiple, smaller-scope Lambda functions together using EventBridge, and having more error handling. We're also trusting the LLM to stick to the formatting guidelines correctly, without accounting for hallucination. We can address this by adding more robust error handling, and doing multiple checks (or even some AI agent shenanigans :D). We'd like to create a website for people to sign up, and add functionality for people to be able to unsubscribe or stay subscribed for as long as they want. Multi-topic subscriptions, sentiment analysis for the articles, allowing users to customize digest delivery time, "share this pickle", group subscriptions, etc etc, are all functionalities we'd love to explore and develop.

Built With

• apigateway
• bedrock
• dynamodb
• eventbridge
• lambda
• newsapi
• python
• ses