BI Guardian

An AI quality gate for Power BI projects—local, deterministic audits with evidence-backed GPT-5.6 remediation plans.

Comment

Inspiration

Power BI teams often inherit semantic models that are difficult to understand, inconsistently documented, and risky to change. A proper review can require hours of manually inspecting tables, relationships, measures, calculated columns, and model metadata.

The problem becomes even harder in organizations where sensitive semantic-model definitions, DAX expressions, file paths, and business logic cannot be uploaded to public AI services.

We built BI Guardian around one question:

Can AI help teams improve a Power BI project without receiving the project itself or becoming the source of truth?

What it does

BI Guardian is an AI quality gate for Power BI projects.

A user uploads a complete PBIP/TMDL project as a ZIP file. The archive is validated, inspected, and parsed locally in the browser. BI Guardian then:

  • Separates user-authored objects from Power BI-generated metadata.
  • Creates an inventory of tables, columns, measures, and relationships.
  • Generates local semantic-model diagrams.
  • Runs a deterministic ruleset against supported metadata.
  • Produces a calibrated health score with transparent deductions.
  • Creates evidence-backed findings with classification, severity, prevalence, and affected scope.
  • Exports an audit report, remediation report, technical manual, diagrams, and engineering backlog artifacts.

AI planning is optional and explicit. When the user requests it, BI Guardian sends only sanitized aggregate context and a limited set of representative finding summaries.

Raw DAX, calculated expressions, source paths, line numbers, project files, and local evidence are not sent to the model.

GPT-5.6 organizes the verified findings into prioritized actions, engineering work items, quick wins, assumptions, and limitations. The complete deterministic scope and local evidence are then reattached inside the browser.

How we built it

BI Guardian was built with Next.js, React, and TypeScript.

The application contains three intentionally separated layers:

  1. Local inspection and parsing
    The browser validates the archive and selectively parses supported PBIP and TMDL metadata.

  2. Deterministic audit and scoring
    A versioned ruleset identifies supported metadata conditions, review recommendations, and documentation improvements. The score is calculated in code, not by AI.

  3. GPT-5.6 remediation planning
    A server-only OpenAI route receives a sanitized, size-limited payload. A strict structured-output contract ensures that the response only references known deterministic findings.

Codex was used throughout development to accelerate implementation, review architecture decisions, create and improve automated tests, investigate edge cases, and validate export and security behavior.

The production application is deployed on Vercel. The OpenAI API key remains server-side and is never exposed to the browser.

Challenges we ran into

One of the hardest problems was distinguishing user-authored semantic-model objects from artifacts automatically generated by Power BI.

Treating generated date tables, template objects, or generated relationships as authored design decisions would create misleading findings and unfair score deductions.

Another challenge was keeping the system conservative. Static metadata cannot prove business intent, runtime correctness, or performance impact. BI Guardian therefore avoids claiming that a model is wrong when the available evidence only supports a recommendation for review.

We also had to maintain a strict privacy boundary while still giving GPT-5.6 enough context to create a useful engineering plan.

Finally, the scoring system required careful calibration so that repeated object-level observations would not create disproportionate penalties.

Accomplishments that we're proud of

  • A complete production deployment with local PBIP/TMDL inspection.
  • A deterministic audit engine with transparent scoring and rule-level deduction caps.
  • Clear separation between authored, generated, and unknown-origin metadata.
  • Local SVG and PNG semantic-model diagrams.
  • Evidence-backed findings that remain authoritative after AI planning.
  • A strict GPT-5.6 remediation contract with validated finding references.
  • Exportable audit reports and a semantic-model technical manual.
  • Successful testing with both a synthetic project and a real Microsoft Power BI sample.
  • 218 automated tests passing before deployment.
  • No raw DAX, project archive, source path, or local evidence sent to OpenAI.

What we learned

We learned that AI becomes more trustworthy when it is placed after deterministic analysis rather than before it.

GPT-5.6 is most useful here as a planning layer: it can organize verified findings into actionable work without inventing the evidence, calculating the score, or modifying the semantic model.

We also learned that privacy architecture is not only a security requirement. It can be a core product capability for enterprise AI adoption.

Finally, separating Power BI-generated metadata from user-authored design decisions dramatically improves the fairness and usefulness of an automated audit.

What's next for BI Guardian

The next steps include:

  • Auditing changes between two PBIP versions.
  • CI/CD quality gates for semantic-model repositories.
  • Organization-specific rule packs and scoring policies.
  • GitHub and Azure DevOps work-item integrations.
  • Broader PBIP and TMDL metadata coverage.
  • Enterprise deployment options with private AI infrastructure.
  • Historical audit tracking and remediation progress.

BI Guardian will remain evidence-first. The goal is not to automatically rewrite a Power BI model, but to help engineers understand it, review it, document it, and improve it safely.

Built With

Share this project:

Updates