Inspiration
The challenge was to buy 100,000 tonnes of CO2e credits from 4,355 projects as cheaply as possible, with a $1M budget. Early on we noticed that money isn't the constraint, since $1M is about ten times what you need. The real problem is risk. Projects fail, and related projects (same country, developer, registry or project type) tend to fail together.
What it does
The most obvious answer is to buy the single best-value project. That costs $115,295, but it fails to deliver about one time in six. We wanted a portfolio that is cheap and still holds up when things go wrong. That led to our feature: if failures can come from shocks, we wanted to throw shocks at the portfolio ourselves and watch it recover.
To Build the portfolio, It finds the cheapest set of credits that delivers 100,000 tonnes with a chosen confidence (99% by default). Our submitted portfolio costs $178,924, holds 27 projects across 8 countries and 3 registries, and delivers in 98.9% of simulated futures it was never optimised on. Shows the price of safety. The same method gives $149,684 at 90% confidence, $158,491 at 95%, $165,134 at 97% and $178,924 at 99%.
Runs a disaster lab. You choose a hazard (wildfire, flood/cyclone, earthquake, drought, or scandal/regulatory crackdown), aim it at a region, country, developer, registry or project type, set a severity and strike. The strike destroys part of the affected holdings and raises the failure risk of everything tied to that target.
How we built it
Risk model: Each project's failure probability comes from its rating (AAA 1% up to CCC 35%, unrated 15%). It is multiplied by 1.5 if the project has had a reversal. Projects with a buffer pool lose only half their tonnes on failure. We simulate 3,000 possible futures. In each one, every project draws a random "luck" value made of its own noise plus shared shocks for its country, developer, registry and project type. A project fails when its luck falls below the cut-off implied by its failure probability. Because of the shared shocks, related projects fail together.
Optimiser: One linear programme, solved with SciPy's HiGHS solver, minimises cost subject to a CVaR constraint: averaged over the worst share of futures, delivery must still reach 100,000 tonnes. Purchases are limited by each project's available tonnes and the budget. To keep it fast, the candidate list is the 300 best-value projects plus the 2 best in every country. We never hard-coded diversification; it emerges because concentrated portfolios do badly in the worst futures. Honest validation: The optimiser plans against one set of futures, and we measure the result on two fresh sets (6,000 futures, at two correlation strengths). A six-step binary search tunes how cautious the CVaR constraint is, so the fresh-futures confidence lands on target without overpaying. App. The interface is built in Streamlit, with the logic in engine.py and the interface in app.py. We used NumPy and pandas for the simulation and Altair for the confidence timeline.
Challenges we ran into
Overfitting. Our first optimiser was an exact integer programme. It reported 99% confidence on its own scenarios but only 87–90% on fresh ones. We replaced it with the linear programme and fresh-scenario tuning described above. An unknown correlation model. The organisers didn't publish how failures are correlated, so we planned at a pessimistic correlation strength and validated at two others. No locations in the data. Our first disaster map placed projects on a pixel grid, but the dataset has no coordinates, so the map was guesswork. We switched to disasters that target real fields in the data: region, country, developer, registry and project type. A misleading chart. Confidence sometimes appeared to rise after a strike. The chart was sorting its labels alphabetically, which put "start" on the right and scrambled events after the ninth. We fixed the ordering and confirmed across 150 test strikes that a strike can never raise confidence. A version mismatch. The app crashed on a laptop running Python 3.9, whose Streamlit version didn't support a newer chart option. We added a fallback and tested on both old and new Streamlit.
Accomplishments that we're proud of
98.9% confidence for $178,924, compared with 84% for the $115,295 naive answer. 15% cheaper than our own earlier greedy version, which needed $211k for 99%. Results that hold up on futures the optimiser never saw. An extra check on 10,000 new futures gave 98.8%. A disaster lab that finds the portfolio's weakest point and recovers from a 22-point confidence drop for under $40,000. Testing our own limits: more scenarios, more candidates and finer tuning saved at most about 1% and cost reliability, which tells us we're close to the best this method can do.
What we learned
Optimising for the average is a trap when failures are correlated; what matters is the bad tail. An optimiser's view of its own results is not evidence. It has to be tested on data it never saw. CVaR lets you express "survive the bad cases" as a linear constraint, so a hard risk problem solves in seconds. A safety level has a measurable price, and choosing it is a decision, not a technical detail.
What's next for Optiver Challenge | FN
Better risk estimates. Adjust the noisy ratings using signals in the data, such as project status (e.g. "Late to verify", "On Hold") and vintage. Removals. Add an option to favour carbon removals, since today's cheapest mix is mostly renewable-energy reductions (wind, hydro, solar). Matching the official scoring. Tune the confidence level to however the organisers score "survives the unexpected". Richer disasters. Model disasters by real geography if location data becomes available, and allow several disasters in a row to stress-test the rebalancing further.
