We were excited to work with open source projects and we liked the idea of Apache Eagle where we pretty much understood the working of a Storm-based application.
What it does
Eagle uses Storm for streaming data and Kafka for messaging. The idea is to develop a security event correlation platform for user to easily onboard new metric (topic) and model metric correlation by sensing the changes in groups of topics dynamically.
How we built it
We have built a new Storm topology and a new API to fetch the events. We created the routing logic in Spout and dispatcher logic in Bolt for the system to enable correlation of topics belonging to a group in a bolt.
Challenges we ran into
We had hard time setting up kafka and zookeeper environment on our machines. Also, as Apache Eagle was a giant code base involving HBase and dynamic policy frameworks, we had to design a simpler and feasible one for the hackathon. Also, we spent more time understanding the flow of the system.
Accomplishments that we are proud of
We managed to add some pieces of code to Eagle, which is a storm and kafka-based application. We are happy that we set our footprints on an Apache project.
What we learned
We learnt the end-end working and implementation of a distributed system and our mentor was really helpful and patient enough to answer our dumb questions throughour the hackathon. He gave us constant feedback about our design, which was really motivating.