Inspiration

Chat UIs are the new browser, bots make it possible for content providers to use it as a primary source of content broadcasting in a informal and personal way.

Using chat bots to deliver content means that users can receive the latest news or desired content into their favorite chat application (telegram only for now).

I just wanted a way to receive news/notifications in my mobile phone/computer every time a new article was published from my favorite sources.

What it does

It accepts input from the user to a chat bot (telegram only for now) about the website(s) the user reads and keeps sending a message every time a new content is available (sends title + short description + image + link).

This way the user is always up to date with their favorite news channel without the need to access the website all the time.

It supports all xml based (including rss, atom and html) and json based documents since it uses xpath/json path to query the content data.

How I built it

Started as a small cli tool I built for myself based om YAML files for crawler configuration and spawned a never ending process to crawl web content every N minutes (defined in the yaml file) written in python.

As the time passed I started to modify/improve it into small services (queue, local content cache, etc);

XPath make it possible to receive notifications about anything, be it a news website article, a system warning that uses json api and so on.

Challenges I ran into

A way for end users to define the xpath query with no technical knowledge (still not perfect).

Also crawling content without being blocked by content providers ;)

Accomplishments that I'm proud of

In this project, using tor proxy to crawl data ;)

Also using yaml files to configure crawlers so it is easy for anyone to configure one in their own machine.

What I learned

Chat UIs + bots can replace the browser (at least partially).

What's next for xvier

Sounds stupid but.. make it available to the public, I have been using myself and two other friends as lab rats :)

I also want to opensource it (all components).

Last notes

Sorry, no video, no images.. it is my hobby project, but I got really excited when I saw this challenge announcement in my inbox! - I am terrible in doing those anyway (I am alone in the project)

Built With

Share this project:
×

Updates