Search Hadoop

What it does

This project provides the ability to search, filter, and correlate logs for all of the Hadoop services in your Ambari cluster, adding tremendous value for cluster operations to quickly triage issues. It provides Log Stash and Solr as Services in Ambari, with configs and commands to stop/start/restart. We also added an Ambari View that's an iframe to a custom UI.

How I built it

We added service definitions for Log Stash and Solr in python, that mange the lifecycle of the logstash and solr processes. Since logstash requires configurations for each service, we pulled the log locations from each service's configs for ease of use.

Challenges I ran into

Collaborating with my peers since all of us were sharing the same source code and literally had to commit and pull every couple of minutes.

Accomplishments that I'm proud of

The project works end-to-end, has a highly valuable UI that makes it easy for operations to triage, and make it extremely easy to configure and update the log locations dynamically.

What I learned

How to add a custom service in Ambari, and a simple View.

What's next for Hadoop Log Search

Make it production ready, and secure.

Built With

Share this project: