- Challenges associated with traditional log management and the evolution of the cyberattack landscape which demanded a Centralised Log Management solution to easily build pipelines to collect and stream logs.
- Democratizing the ability to create log pipelines to even users with low technical skills.
- Overcome the bandwidth issues associated with multiple agent installation.
- Operational overhead of management of multiple log gateways/agents.
What it does
A self-service GUI based platform for building and maintaining log collection, filtering, masking, and redirection pipelines facilitating log streaming from multiple sources to multiple destinations including Azure Sentinel. In simple terms, it aggregates logs from disjointed systems to advanced platforms like Sentinel. Kubernetes native CLM platform enabling deployment to any environment of choice*.
It also helps to mutate sensitive data and additional actions (such as deleting a record from a log), some of the supported sensitive data examples are below
- Personally identifiable data.
- Health Data
- Financial Data (like credit card numbers)
- IP addresses may be considered sensitive, especially when in combination with personally identifiable data
How we built it
- We built it as a Kubernetes native app.
- We leveraged k8s APIs and we chose golang to build the platform.
- Various open-source tooling was integrated to be able to seamlessly collect, stream, monitor logs.
Challenges we ran into
- Encrypting logs in transit
- Building efficient monitoring dashboard for sanity/quality check
- Connecting k8s cluster from multiple providers
Accomplishments that we are proud of
- One agent to collect logs from multiples sources and send that to multiple destinations
- Reduced total bandwidth consumption for log data traffic
- Inbuilt monitoring and alerting capability for health check of log-collector
- Agentless log collection
- K8s native multi-environment deployment
- Work's well with a hybrid environment
What we learned
- Logging best practices
- Multiple log formats(CEF, LEEF, Syslog, Json, XML etc.)
- How CLM can be build to complement SIEM for better threat analysis
What's next for Ops_brew
- Multi K8s environment support
- Capturing of network flow logs
- Enabling more sources(IT/OT)
- Creating more workbooks for all sources
Ops_brew with Azure Sentinel scenarios
Here we are categorizing the scenarios on how Ops_brew can accelerate the adoption of Azure Sentinel.
For customers on the lookout to invest in a new SIEM solution or renew an existing license, Azure Sentinel is a future proof solution considering the fact that it natively supports cloud workloads and not to mention its easier to implement as does not have the heavy CAPEX investment associated with traditional SIEM. SIEM solutions need logs from source systems to be effective, Ops_brew enables quick creation and deployment of pipelines from hybrid sources to Azure Sentinel.
For customers who have already procured Enterprise SIEM, generally have limited use-cases built and do not have integrations for cloud workloads. Ops_brew can direct logs from the cloud and on-prem sources to Azure Sentinel and other SIEM solutions.
Managed Security Providers would have ideally made significant investments in tools and processes to offer SoC services to multiple customers. Ops_brew can enable such a team to redirect logs to their existing tools and to Azure Sentinel, leveraging it as a secondary SIEM.