Inspiration
With how prevalent AI has become today, it makes sense that it is also changing the energy landscape with how much electricity computing actually consumes. Data centers used about 415 TWh in 2024, roughly 1.5% of the world's electricity, and it is expected to more than double by 2030. Building clean power plants takes years. But we all know that noticed that energy hungry work doesn't care when it runs: a model training run, an overnight batch job, an EV plugged in until morning.
Despite that, the grid's carbon intensity changes hour to hour. Solar comes up at midday, wind rises and falls, and coal and oil "peaker" plants switch on for the evening rush. This means that depending on when this computing is performed, the same kilowatt-hour can be noticeably cleaner. Finding that cleaner time is what GridShift is for: turning that flexibility into real emissions cuts using only software, with no new hardware.
What it does
GridShift forecasts the carbon intensity of the electricity grid (in grams of CO₂ per kWh) for the next 24 hours and tells you the cleanest time to run a flexible job.
You get provided with a 24-hour prediction with an 80% confidence band, shown next to the last 7 days of actual grid data. Based on the provided predictions, you can schedule jobs by picking a preset (ML training node, EV charger, 250 kW data-centre batch job) or by entering your own power draw, duration and deadlines. GridShift then finds the start time with the lowest average carbon intensity that still finishes before your specified deadline.
GridShift provides a comparison between starting now against the best window and shows the kg of CO₂ avoided, a driving-distance equivalent, and what that adds up to if 10,000 jobs shifted every day for a year.
Saved jobs: scheduled jobs are stored with a running total of CO₂ saved. Under the hood: a live panel benchmarks our continuous aggregates against raw data and shows compression stats. GridShift works for any US grid region the EIA covers. It is built around PJM, the largest grid operator in the US.
How we built it
An ingest pipeline pulls hourly electricity generation by fuel type from the EIA v2 API, handling pagination. It converts each fuel's output into CO₂ using emission factors (coal ~1000, oil ~800, gas ~430 kg/MWh; nuclear and renewables 0). For testing without an API key, I built a realistic synthetic generator: nuclear baseload, a solar curve that varies with cloudiness, slowly drifting wind, and coal and gas filling the rest, with dirtier evening peaks. Data is bulk-loaded with Postgres COPY and upserted, so re-running ingest never creates duplicates.
Tiger Data (TimescaleDB):
grid_readings is a hypertable split into 7-day chunks. Columnar compression, grouped by region and fuel, is applied automatically to chunks older than 7 days. An hourly continuous aggregate computes gCO₂/kWh and renewable share. A hierarchical daily continuous aggregate is built on top of the hourly one, never touching raw rows. Background policies keep everything refreshed and compressed. Model output goes into a forecasts hypertable, and scheduled jobs live in a regular relational jobs table alongside the time series. Machine learning:
scikit-learn's HistGradientBoostingRegressor uses hour of day, day of week, day length (for season), and the intensity at the same hour 24, 48 and 168 hours earlier. Because every input is at least 24 hours old, all 24 future hours are predicted in one shot. There's no feeding predictions back in, so errors don't compound. Separate quantile models produce the 10th–90th percentile band, calibrated with conformal prediction. It is evaluated on the last 14 days of held-out data against a naive "same hour yesterday" baseline.
The app itself was built with Streamlit and Plotly, with configuration in a .env file.
Challenges we ran into
The first model lost to the baseline, making predictions about 30% worse than simply predicting "same as this hour yesterday". The cause was the season features: season was encoded as day-of-year, but with only a few months of history, the test period always fell on calendar days the model had never seen, and tree models can't extrapolate. I tested alternatives on several synthetic datasets and settled on three changes: day length as the season feature, predicting the correction to yesterday's value rather than the raw level, and stronger regularization. The model went from losing to winning by 24–35%. Another challenge we faced was overconfident uncertainty bands. The "80%" band only contained 49% of actual values. Raw quantile models are overconfident on data they weren't trained on. Conformational calibration was then added: hold back part of the training data, measure how often reality falls outside the band, and widen it to match. Coverage rose to 72–76%. I also faced issues with benchmarks hidden by network lag. Timing queries from the app mostly measured the round trip to the cloud, which hid the database's own speed-up. We switched to the server-side execution time reported by EXPLAIN ANALYZE. Finally, there were some issues with database setup details. Some TimescaleDB commands, such as refreshing a continuous aggregate, can't run inside a transaction. The newest EIA hours are often only partly reported, which made the grid look suddenly clean, so those hours were filtered out at ingest.
Accomplishments that we're proud of
A forecast that measurably beats a strong naive baseline on held-out data: about 10.6 vs 15.1 gCO₂/kWh average error (24–35% better) on our synthetic test sets, with uncertainty bands calibrated to their stated confidence level. Every major TimescaleDB feature working together: hypertables, compression, continuous aggregates, a hierarchical aggregate, automatic policies, and relational tables in one database. On even our small demo dataset, the continuous aggregate answers the daily-intensity query about 3× faster than raw data, and the gap grows with every row ingested. A clear answer for users: "start at 2 AM, save X kg of CO₂". It's built on an honest comparison, not a vague green score. Testing the whole pipeline offline before touching the database. That's how the losing model was caught.
What I learned
Always compare against a naive baseline. Our first model looked reasonable on a chart and was still worse than "same as yesterday". Without the baseline I'd never have known. Feature choices depend on how much history you have. Features that make sense with years of data can hurt with months. Uncertainty estimates need checking. A model that says "80% confident" should be right about 80% of the time, and calibration is cheap to add. Time-series databases change the architecture. Continuous aggregates let the app and the model read pre-computed results instead of re-adding up raw data on every request. Compression and retention become a single policy line instead of a cron job. Carbon timing is a real, untapped lever. The cleanest hour of the day varies with weather and demand, so a schedule that's fixed once is not enough. A forecast is needed.
What's next for GridShift
Marginal emissions: currently GridShift uses the grid's average intensity. Marginal intensity (which plant actually ramps up for the next kWh) is a better signal for load shifting, and it would be great if we could add it from sources like WattTime or Electricity Maps. Plugging into real schedulers: a REST API, plus integrations for Kubernetes, Slurm and Airflow so batch jobs and training runs are timed automatically, and smart-charging APIs for EV fleets. Shifting location as well as time: cloud workloads can move between regions. Comparing forecasts across grid regions would let a job run where and when the grid is cleanest. Scoring our own forecasts: every forecast run is already stored with a timestamp. Next is a continuous aggregate that tracks forecast error over time and triggers retraining when accuracy drifts. Richer models: weather forecasts (wind speed, cloud cover, temperature) as inputs, and longer training history for real seasonal patterns. Lifecycle emission factors in place of direct stack emissions only.
Built With
- eiaopendata
- git
- numpy
- pandas
- plotly
- postgresql
- python
- scikit-learn
- sql
- streamlit
- tigerdata
