Memora

Memora turns legacy and modern data platforms into a trusted AI knowledge layer for lineage, operations, onboarding, and migration.

Comment

Inspiration

Every enterprise data platform contains critical knowledge, but that knowledge is fragmented across source code, database objects, pipelines, runtime history, architecture documents, and the memories of a few experienced engineers.

This creates familiar problems:

  • New engineers spend weeks discovering how systems work.
  • Support teams open multiple tools to investigate one failed workload.
  • Impact analysis depends on tribal knowledge.
  • Modernization teams repeatedly reverse-engineer legacy behavior.
  • AI assistants can produce convincing answers without understanding the real system.

We built Memora around one principle:

AI should explain verified evidence. It should never invent the system.

Our goal was not to build another SQL Server documentation tool or a chatbot over uploaded documents. We wanted a trusted knowledge and operations layer that could work across legacy and modern data platforms.

SQL Server, SQL Server Agent, and SSIS became our first proof point because they represent a difficult, real-world legacy environment. They are the starting connector, not the boundary of the product.

What it does

Memora transforms technical artifacts, live metadata, operational evidence, and approved architecture into a connected and explainable knowledge layer.

It helps teams:

Connect and approve

Memora reads files that a team has explicitly approved and performs controlled, read-only discovery against connected platforms.

It discovers databases, schemas, tables, procedures, orchestration jobs, job steps, packages, dependencies, and deployment targets. Uncertain mappings remain inactive until a person reviews them.

Build a canonical knowledge model

Platform-specific metadata is converted into a common model of:

  • Business workloads
  • Orchestration jobs and tasks
  • Data sources and destinations
  • Procedures and transformations
  • Runtime evidence
  • Downstream dependencies
  • Architecture decisions
  • Modernization targets

This means new connectors can contribute to the same knowledge layer without changing how users ask questions or explore lineage.

Explain end-to-end lineage

A user can follow a workload from its business purpose to its ordered job steps, SSIS packages, SQL procedures, source tables, target tables, reports, and downstream impact.

Every relationship links back to technical evidence.

Ask Memora

Users can ask questions such as:

  • What does this pipeline do?
  • Where does this data come from?
  • What could be affected if this procedure changes?
  • Why did this workload fail?
  • Which reports depend on this table?
  • What should a new engineer understand before supporting this process?
  • How could this workload be migrated to the approved target platform?

Memora preserves the original question, expands domain vocabulary, classifies intent, rewrites the retrieval query, performs semantic and keyword retrieval, reranks the evidence, and then uses GPT-5.6 to generate a grounded explanation.

Answers include confidence, citations, verified lineage, and explicit limitations.

Support operations

When something fails, capability-aware collectors first inspect the platform version, available services, and permissions.

Memora then retrieves approved runtime evidence and connects the failure to the relevant task, package, database objects, and downstream consumers.

Memora does not execute AI-generated SQL against the enterprise platform.

Accelerate modernization

Teams can provide an approved target architecture for Databricks, Microsoft Fabric, Snowflake, another cloud platform, or a different internal standard.

Using the trusted knowledge of current behavior, Memora prepares reviewable:

  • Source-to-target mappings
  • Transformation scaffolds
  • Reconciliation tests
  • Workflow drafts
  • Migration runbooks
  • Architecture decision records
  • Data contracts

These remain drafts until a person approves them.

How we built it

Memora uses a modular enterprise architecture.

The backend is implemented in Python and FastAPI, with Pydantic models and a connector framework for platform-specific discovery.

The user interface is built with React, TypeScript, and Vite. It includes guided onboarding, connection validation, discovery, workload selection, mapping review, activation, lineage, operations, modernization, and Ask Memora.

For our first enterprise connector, Memora uses pyODBC and Windows Authentication to perform read-only discovery across SQL Server, SQL Server Agent, and SSIS. Approved source folders provide a low-impact knowledge baseline, while live collectors provide operational and runtime evidence when required.

The enterprise storage architecture supports:

  • PostgreSQL for control data and governance
  • Qdrant for vector retrieval
  • sentence-transformers for embeddings
  • BM25 for keyword retrieval
  • NetworkX for lineage and dependency relationships
  • A file-based mode for restricted pilot installations

The LLM layer is abstracted through LiteLLM, with GPT-5.6 as the default reasoning model.

The question pipeline contains eight separate stages:

  1. Preserve the original question
  2. Apply light normalization
  3. Expand domain vocabulary
  4. Classify intent
  5. Rewrite the retrieval query
  6. Perform semantic and keyword retrieval
  7. Rerank the strongest evidence
  8. Generate a grounded answer

We added PowerShell installation and startup scripts for controlled Windows environments, automated activation reports, fail-closed deployment controls, and pytest-based validation.

How OpenAI powers Memora

OpenAI is used where reasoning adds value, while deterministic collectors remain responsible for establishing system truth.

GPT-5.6 explains retrieved catalog, lineage, runtime, and architecture evidence. It does not independently invent relationships or execute generated SQL against connected systems.

Codex helped us:

  • Audit the existing legacy implementation
  • Define the modular eight-stage knowledge pipeline
  • Refactor the connector architecture incrementally
  • Build capability-aware SQL Server discovery
  • Diagnose version-specific metadata failures
  • Design the guided enterprise onboarding experience
  • Add validation and deployment controls
  • Create the product demonstration and test workflow

This separation between deterministic evidence collection and AI explanation is central to Memora's enterprise design.

Challenges we faced

Platform versions and permissions

Enterprise environments do not expose identical metadata. A query that works on one SQL Server version may fail on another because a system view or column is unavailable.

We solved this by introducing capability discovery. Memora detects the server version, available services, permissions, and supported metadata before selecting an approved collector.

Mapping orchestration to deployed packages

A SQL Agent master job may contain T-SQL tasks, SSIS steps, nested jobs, protected package references, or targets that no longer exist in the exported source folder.

Memora separates verified mappings from unresolved items and requires human approval before uncertain relationships become trusted knowledge.

Preventing hallucinated lineage

A language model can produce a plausible explanation even when evidence is incomplete.

We therefore build lineage deterministically, keep evidence references with every relationship, rerank retrieved context, expose limitations, and prevent the model from silently converting assumptions into facts.

Corporate deployment constraints

The pilot needed to work inside a controlled Windows environment with Windows Authentication, limited database access, optional infrastructure dependencies, and no requirement to send the complete data estate outside the organization.

We created a guided setup flow, file-based pilot mode, read-only connectors, local catalogs, and environment-specific installation scripts.

Accomplishments that we are proud of

  • Compiled 10,132 technical objects into a usable pilot knowledge catalog
  • Reconstructed a master workload containing 32 ordered operational steps
  • Connected SQL Agent jobs, SSIS packages, T-SQL tasks, databases, and dependencies
  • Built a human-reviewed onboarding and activation workflow
  • Created evidence-backed Ask Memora answers with citations and limitations
  • Added capability-aware, read-only operational collectors
  • Built modernization artifact generation from approved target architectures
  • Created a reproducible Windows enterprise installer and pilot validation gate
  • Designed Memora around a platform-neutral canonical knowledge model

What we learned

The hardest part of enterprise AI is not calling the model. It is building trustworthy context.

We learned that:

  • Human approval is not friction; it is part of the trust model.
  • File artifacts and live metadata are complementary knowledge sources.
  • Capability detection is safer than assuming every platform behaves the same way.
  • Retrieval quality must be separated from answer generation.
  • Modernization works best when current verified behavior is combined with an approved target architecture.
  • A good enterprise AI product must clearly show what it knows, why it believes it, and what remains uncertain.

What's next

Next, we plan to:

  • Expand connectors across Databricks, Microsoft Fabric, Snowflake, BigQuery, lakehouse, transformation, and orchestration platforms
  • Add continuous metadata and lineage drift detection
  • Strengthen SSO, RBAC, audit trails, and approval workflows
  • Connect additional runtime monitoring and incident-management sources
  • Add more governed modernization artifact generators
  • Evaluate retrieval and answer quality against enterprise test sets
  • Move from the first enterprise pilot toward a secure multi-platform production release

Memora makes complex data systems understandable, operable, and ready to modernize.

Built With

Share this project:

Updates