Smart energy consumption prediction & recommendation system

Inspiration

Every month, millions of Indian households open their electricity bill and are surprised — sometimes pleasantly, usually not. By the time the bill arrives, the consumption has already happened; there's no way to act on it. We kept asking ourselves: what if a household could see next month's bill coming, understand why it's shaped the way it is, and get a concrete, personalized way to bring it down — before the meter has even finished counting?

That question became the seed of our project. We didn't want to build "just another ML prediction model." We wanted something that combined real engineering (appliance load calculations), real machine learning (weather-responsive forecasting), and real transparency (explainable AI) into one coherent system that a non-technical person could actually use and trust.

What We Learned

This project taught us as much about honest engineering as it did about machine learning.

Data isn't neutral. We started with a well-documented, Kaggle-available household energy dataset from France. It looked perfect on paper. But when we dug into the correlation between temperature and consumption, we found it was negative — colder days meant higher usage, the signature of a heating-dominated climate. For India, where air conditioning and cooling drive demand, that relationship runs the opposite way. Using it would have meant training a model that had learned the wrong physics for our use case. We discarded it and rebuilt our pipeline around genuine Indian smart-meter data (CEEW's Bareilly and Mathura datasets), merged with NASA POWER's historical weather API.

The "best" model depends on how you ask the question. When we compared five algorithms — Decision Tree, Random Forest, XGBoost, LightGBM, and CatBoost — a single train-test split suggested Random Forest was strongest ($R^2 = 0.873$). But a single split can be a matter of luck. We ran 5-fold cross-validation instead:

$$ \text{Mean CV } R^2 = \frac{1}{5}\sum_{i=1}^{5} R^2_i $$

Under this more rigorous lens, LightGBM came out ahead (mean $R^2 = 0.563$), with the lowest overfitting gap among the boosting models. We learned, concretely, why cross-validation — not a single lucky split — has to be the basis for model selection in any serious evaluation.

One explanation method is a claim; four agreeing is evidence. We didn't stop at SHAP. We cross-verified its findings against LIME, Permutation Feature Importance, and ELI5 — four mathematically distinct approaches. When all four independently ranked minimum and average temperature as the dominant predictive features, that convergence gave us something a single method never could: confidence that the pattern was real, not an artifact of one technique's assumptions.

How We Built It

The system layers several independent pieces, each solving one part of the problem:

Appliance-based baseline — standard wattage formulas convert a user's appliance inventory (AC tonnage, fan type, geyser usage, etc.) into an estimated daily consumption: $$ \text{Units (kWh)} = \frac{\text{Wattage} \times \text{Hours Used}}{1000} $$
Calibration against reality — users enter last month's actual bill, and we scale the baseline to match it, capped within a sane range so one bad data point can't distort the whole estimate: $$ \text{Calibration Factor} = \text{clip}\left(\frac{\text{Actual Bill Units}}{\text{Baseline Units}},\ 0.5,\ 2.0\right) $$
Weather-aware ML adjustment — a LightGBM model, trained on real consumption and NASA POWER weather data, predicts how today's weather should shift usage relative to a typical day, applied only to weather-sensitive appliances like ACs and coolers — not to fridges or lighting, which don't respond to temperature.
Explainability layer — SHAP's TreeExplainer breaks down each prediction into the factors driving it, so the output is never a black box.
Tariff-accurate billing — real MERC (Maharashtra Electricity Regulatory Commission) slab rates convert predicted units into an actual rupee estimate, with a full breakdown of which slab, rate, and charges apply.
Rule-based, percentage-driven recommendations — tips are generated from each appliance's share of total consumption, not the fluctuating final prediction, so the advice stays stable and meaningful even as weather shifts.

The stack: Python and Flask (later rebuilt in Streamlit), scikit-learn/XGBoost/LightGBM/CatBoost for modeling, SHAP/LIME/ELI5 for explainability, and SQLite for persistence — deployed on Streamlit Community Cloud.

Challenges We Faced

The single hardest problem wasn't modeling — it was getting SHAP to run in production without crashing. On free-tier cloud hosting (512MB RAM), our app kept dying with out-of-memory errors. We debugged it like a real engineering problem, not by guessing:

Reduced the model from 100 to 30 estimators — still crashed.
Simplified every surrounding module (explanations, recommendations) — still crashed.
Finally, isolated the issue by disabling SHAP alone, with everything else untouched — the crash disappeared.

That systematic elimination told us, with certainty, that SHAP's TreeExplainer initialization — not model size, not our own code — was the actual memory bottleneck. It reframed the problem from "something is broken" to "we need more memory headroom," which led us to Streamlit Community Cloud (roughly 1GB RAM), where the full system — SHAP and all — finally ran cleanly, with zero compromise on the model or explanation quality.

We also had to confront subtler issues: a chronological train/test split that silently introduced seasonal bias, a live-weather API that returns different temperature ranges depending on what time of day you query it (fewer remaining forecast slots late at night), and the genuine, honest limitation that no dataset of this size can fully explain household-level consumption variance — which is precisely why we designed the ML layer as a capped adjustment on top of a user-grounded baseline, rather than the system's sole source of truth.