We were suffering from newsletters filling our mailboxes. We love the newsletter and how they are carefully hand picked. We thought of the idea of this application to restore sanity to our mailboxes but moving the newsletters to our platform.
What it does
We subscribe to newsletters on behalf of the users and we extract the important links from it. The users get to see a list of newsfeed containing these links where they can read it, mark is as read, add it to favorites, add notes and more.
How we built it
- We have an SMTP server that will listen on the 25 port, and receive the incoming data, then parse it as a Mail Messages according to the standard SMTP protocol, then extract the html part and crawl all the links in it, after that it get each link meta info and organize all the found data then send it to a predefined webhook, so we can continue processing it. We followed the concept of "Microservices", each service will do a job, our first service is the SMTP listener with the built in crawler, Golang was used to make use of its high concurrency features and the cross-compilation that makes distributing it very easy.
- The scrapper sends our application the links that it fetched through a webhook which triggers spreading the newsletter to the users that subscribed to.
- We built the front-end component using react and react-router
Challenges we ran into
- Some of the links returned by the tool that read the email are not useful like un-subscription links for example. The challenge was to identify these links programmatically and exclude them from the users list. We also provided the users with ways to report these links to exclude them in the future.
- Setting up the mailbox was really difficult
Accomplishments that we're proud of
- The application was really done in less than 48 hours and it was our first time to use react so we're so proud of the results!
What we learned
- This is the first time we use react on the web.
- We should really plan better ahead and give ourselves more time to work on side projects and competitions :)
What's next for Newsletter To You
- Adding more features to the application that allows it to grow and create a community. Stuff like public discussions arounds links, publishing statistics about newsletters usage.
- Building browser extensions that automatically allows you to register to these newsletter through our platform
- Machine learning to identify the un-important links.
To see data in the application immediately, you need to login with this account: email: firstname.lastname@example.org password: secret
The reason is that when a new user sign up, new user still have to wait till the system receive new newsletters so he can see it.