-
-
MadLava: Feel The Lava
Inspiration
MadLava started with a slow local Spark workload. Spark showed me jobs, stages, tasks and execution plans, but I wanted to understand what was happening deeper inside the JVM: slow or excessively invoked methods, silently created exceptions, serialization costs and driver-side pressure.
What it does
MadLava is a lightweight Java profiling agent that selectively observes JVM and Spark driver behaviour, produces bounded JSONL reports and presents the results through an offline report viewer.
How I built it
I built MadLava incrementally through small, independently testable iterations. Each feature follows the same core pipeline:
Instrumentation → protected callbacks → bounded aggregation → asynchronous reporting
Every iteration produced a runnable and distributable increment while preserving the functionality and tests completed previously.
Challenges I ran into
(apart from running out of Codex credits last week)
The hardest challenges were preserving bytecode semantics, instrumenting constructors safely, handling class loaders and Java modules, preventing recursive profiling, controlling overhead and reporting measurements without implying false precision or causality.
A profiler must not only collect data—it must preserve the behaviour of the application it observes.
Accomplishments that I'm proud of
I created a distributable Java agent with selective profiling, JVM diagnostics, Spark insights, privacy-conscious reporting and an offline viewer, without requiring an external backend.
It is certainly not perfect, but I am proud of how much I managed to design, build, test and document by myself in only a few days.
What I (re)learned
Building a profiler is mostly an exercise in restraint. A trustworthy tool must explain what it observed, where the data came from, what may be missing and whether each value is exact, sampled, estimated or partial.
Most importantly, I was reminded to follow the KISS principle—Keep It Small and Simple—especially when building the first version of an MVP.
What's next for MadLava
The next steps are deeper real-world certification across supported Java and Spark versions, broader performance and overhead validation, improved Spark and PySpark correlation, and continued refinement of the offline report viewer.
The full technical journey, including the architecture, design decisions, implementation challenges and lessons learned, is available in MadLava: Feel the Lava.
Log in or sign up for Devpost to join the conversation.