TailDaam

TailDaam: governed Pakistani fuel-price data, a baseline-beating volatility model, and an insight agent that refuses when the record does not support an answer.

Comment

TailDaam — Project Story

Inspiration

Petrol and diesel prices are among the most politically sensitive numbers in Pakistan. Every fortnight the price changes, everyone has a theory, and the evidence is scattered across weekly PBS reports, city-level annexes and global benchmark feeds. Most fuel-price tools either repeat a headline or generate an answer without showing where it came from. TailDaam was built to answer a narrower question well: what does the record actually support, and what does it not?

What it does

TailDaam is an end-to-end data product for Pakistani petrol and diesel prices:

  • A governed warehouse with 91,672 clean rows from 62 raw files and 3 independent publishers.
  • A deterministic data-quality agent that logs every FIX / FLAG / FAIL decision.
  • A baseline-beating model for next-month Brent realised volatility, not the administered domestic price.
  • A refusal-first insight agent that quotes gold-table facts, returns citation tags and row ids, and structurally refuses unsupported questions.
  • An hourly Prefect pipeline, a live FastAPI backend and a free static Hugging Face Space.

How I built it

  1. Acquisition and provenance. Every raw file is downloaded from its original publisher, SHA-256 hashed and recorded in data/manifest.csv with URL, publisher, fetch time, licence and bytes. Sources: Pakistan Bureau of Statistics (41 weekly SPI reports + 9 monthly city annexes), U.S. EIA (11 daily petroleum spot series) and the World Bank Pink Sheet.
  2. Cleaning and quality. Raw files are normalised into typed long-format tables with stable row ids. A quality agent runs 14 deterministic rules covering schema, duplicates, ranges, daily/weekly jumps, cadence gaps and freshness. Warnings are kept, not hidden.
  3. Gold layer. Only the gold tables may be quoted. Every gold row stores row_ids, a citation_tag, period, entity, value and unit.
  4. Model. The declared baseline is persistence. On next-month Brent realised volatility, a pre-registered HAR ridge model improves MAE from 17.8044 to 16.5102, a 7.269% reduction, with 89.25% empirical coverage for the nominal 90% interval. The price-level attempt is reported as a non-win rather than hidden.
  5. Agent. Retrieval runs over 2,417 gold facts. Scope, time, product and city filters run before generation. The answer is assembled from retrieved facts; post-hoc verification rejects unsupported numbers or unknown citations.
  6. Delivery. FastAPI serves the API and a hand-written single-page app. A static Hugging Face Space calls the FastAPI Cloud backend, so the demo requires no login.

Challenges

  • PBS publication gaps are real. The weekly series has genuine gaps, and the quality agent flags them instead of filling them.
  • The April 2026 HSD value is a real outlier. 522.29 PKR/L against 380.56 petrol triggers a weekly-jump flag; it is kept and annotated, not silently removed.
  • City data was not one rolling file. PBS keeps each month under a different file name; the first build accidentally ingested only September. I corrected this by adding January–September 2026 and redeploying.
  • Price level is close to a random walk. I tried the obvious target first and reported the negative result instead of presenting a spurious win.
  • Free-tier deployment. Docker Spaces required payment, so the UI became a static Space calling the FastAPI Cloud API.

What I learned

  • A smaller, honest answer beats a larger generated one.
  • Refusal must be a code path, not a prompt request.
  • Baselines and negative results are part of the product, not footnotes.
  • Provenance is not paperwork; it is what makes an answer defensible.

Key result

MAE improved from 17.8044 to 16.5102, a 7.269% reduction versus the persistence baseline.

Lower MAE than persistence on a chronological 80/20 split, with published group errors and an honestly reported price-level non-win.

Try it out

Built with

Python, FastAPI, Prefect, pandas, scikit-learn, Chroma, fpdf2, Pydantic, requests, openpyxl, Hugging Face Spaces, FastAPI Cloud, GitHub, Pakistan Bureau of Statistics, U.S. EIA, World Bank Pink Sheet.

Built With

Share this project:

Updates

Submission history