As technologies evolve, standards raise, and expectations grow, there is a greater need for data to be instant and real-time. Traditional REST APIs might not be able do that. That's where event-driven platforms such as Confluent Kafka comes to place.
What it does
MyFluent lets users type in what topic they want to stream, then it will listen to Twitter and pipeline real-time tweets into Confluent Cloud. The tweets are then consumed by displaying them dynamically on the website. MyFluent is especially useful to developers who would like to use rapidly changing datasets, as millions of people uses Twitter every day and these data reflects current trends. This can further be used to build a dashboard, or train machine learning models.
How I built it
I use Confluent Kafka, Python Flask, and Twitter API to build this project.
Challenges I ran into
Being completely new to Confluent Cloud is the biggest challenge throughout the weekend. I have to read their documentations, set up everything from scratch, and ask mentors for help. I also ran into some issues with threading and web sockets, where the server cannot communicate with the client. That took away some good hours :-(
Accomplishments that I'm proud of
I am proud that this project can run successfully within the time constraint.
What's next for MyFluent
I really want to dive deeper into using Confluent connectors and their other features, and try to build a better-looking tweets dashboard.