Inspiration
Search Hadoop
What it does
This project provides the ability to search, filter, and correlate logs for all of the Hadoop services in your Ambari cluster, adding tremendous value for cluster operations to quickly triage issues. It provides Log Stash and Solr as Services in Ambari, with configs and commands to stop/start/restart. We also added an Ambari View that's an iframe to a custom UI.
How I built it
We added service definitions for Log Stash and Solr in python, that mange the lifecycle of the logstash and solr processes. Since logstash requires configurations for each service, we pulled the log locations from each service's configs for ease of use.
Challenges I ran into
Collaborating with my peers since all of us were sharing the same source code and literally had to commit and pull every couple of minutes.
Accomplishments that I'm proud of
The project works end-to-end, has a highly valuable UI that makes it easy for operations to triage, and make it extremely easy to configure and update the log locations dynamically.
What I learned
How to add a custom service in Ambari, and a simple View.
What's next for Hadoop Log Search
Make it production ready, and secure.
Log in or sign up for Devpost to join the conversation.