-
-
Adding the Topics into the Kafka Brokers
Inspiration
We wanted to build something like a Bloomberg Terminal and its Publisher/Subscriber style architecture.
What it does
Each user has the option to subscribe/unsubscribe to multiple stock prices that the user wants to follow.
How we built it
We developed Publisher scripts that are responsible for pulling data from each API source using REST APIs. These Publishers are managed by multiple Kafka Brokers with a replication factor of 3. We finally connect these brokers to a front-end webpage developed using Flask Framework. We utilize MongoDB for store User login details and Subscription information.
Challenges we ran into
Dockerizing the whole system took a while. We were also stuck while setting up Kafka manager dashboard, which hindered our progress. We ended up downgrading to a lower version and discarded the plan to install Kafka manager. We added all the topics manually into the Kafka servers for time being.
Accomplishments that we're proud of
Implementing Kafka into the architecture and Dockerizing the whole system.
What we learned
Docker, Kafka and the whole Pub/Sub Architecture and their challenges in implementing them.
What's next for Real-time Stock Market Dashboard
A more streamlined and interactive UI, more efficiency and reliability while adding new topics. A notification system for the subscribed topics.
Log in or sign up for Devpost to join the conversation.