SORTIFY

R&D and Product Strategy Managers are tasked with producing Research Journals, Comparative Paper Tables, Competitive Site Data Snapshot, and Product Roadmap Decisions.

Comment

Inspiration

Every week, enterprise AI research directors, CTOs, and product managers are tasked with making multi-million-dollar decisions: Which inference stack should we deploy? Which quantization format preserves accuracy? How do competing models compare in tokens per second per dollar?

To answer these questions, teams spend dozens of hours manually reading dense arXiv preprints, scraping documentation, and testing benchmarks. When they turn to existing LLM agents for help, they run into what we call the $4.2B Hallucination Tax:

  1. Unconditioned Comparisons (Apples-to-Oranges): An agent confidently reports that "Model A achieves 1,450 tokens/sec while Model B achieves 850 tokens/sec"—completely omitting that Model A was measured on an 8x H100 SXM5 cluster in FP8 with batch size 64, while Model B was evaluated on an A100 in FP16 with batch size 1.
  2. Silent Claim Contradictions: Vendor marketing pages routinely make speed claims that contradict peer-reviewed arXiv papers. When standard chat agents ingest both, they silently overwrite or average out the numbers rather than identifying and auditing the conflict.
  3. Fragile Scrapers: The moment a competitor updates their documentation DOM or pricing layout, static scraping scripts fail, burning token budgets on error loops.
  4. Lost Lineage: Context window compression destroys the exact character-level citations needed for engineering audits.

We asked ourselves: What if an AI research platform was built not as a chat window, but as an event-driven, self-improving epistemic runtime? A system where no table cell is ever generated freehand, every claim is anchored to exact character offsets and hardware tuples, and the agent autonomously patches its own code when websites change.

That inspiration became SORTIFY.

What it does

SORTIFY is the first self-improving AI R&D and product strategy runtime that enforces Zero Freehand Factual Generation. Accessible live at https://sortify.my.id, it automates scientific and competitive diligence from raw web ingestion to boardroom-ready deliverables:

  • R&D Goal Wizard with Invariant Locks: Users configure target research schemas, set hard token/USD budget ceilings, and activate mandatory condition-binding locks (Hardware + Precision + Batch Size).
  • Grounded Web & arXiv Ingestion: Automatically searches and extracts full-fidelity markdown from arXiv preprints and competitor documentation via the Tavily Grounding API.
  • Reactive Evidence Graph: Records every extracted metric as a typed node (Source, Claim, Condition, Decision) with exact character offsets (char_offset_start, char_offset_end), verbatim source quotes, and cryptographic provenance edges.
  • Autonomous Scientific Refereeing: Real-time Cypher pattern rules detect contradictions (PAT_CONTRADICTORY_CLAIM). The event bus spawns an auditor behavior that invokes containerized NVIDIA NIM Nemotron-Ultra-253B to audit experimental methodologies and assign visual epistemic state markers:
    • [✓] Verified Homogeneous Baseline
    • [≈] Scaled / Normalized Condition
    • [!] Contested Claim under active referee audit
    • [—] Not Reported (N/R) — never fabricated or interpolated.
  • Deterministic Artifact Compiler: Transforms verified subgraphs directly into publication-grade Booktabs LaTeX, academic research journals, and structured CSV/JSON with zero LLM text interpolation. Every cell in the compiled table is interactive, opening the exact source sentence and character offset in one click.
  • Competitive Site Data & DOM Drift Monitor: Continuously monitors competitor documentation and pricing pages, tracking structural changes with full-length SHA-256 DOM hashes and visual side-by-side diff drawers.
  • Autonomous Meta-Harness: When an extraction worker encounters an updated website layout or schema failure, the system captures the AST and stack trace, prompts Nebius-hosted Qwen2.5-Coder to synthesize a unified Git diff patch, validates it through two strict security gates, and hot-reloads the code without downtime.
  • Stripe Enterprise UI Suite: An 8-page production cockpit built to the visual standard of Stripe Enterprise (#635BFF Blurple, #0A2540 Midnight Navy, #00D4FF Electric Cyan, #F6F9FC Pearl canvas).

How we built it

SORTIFY is architected around an asynchronous, typed event-driven core connecting specialized foundation models and infrastructure:

1. The Tripartite Technology Engine

  • Nebius Token Factory (https://api.tokenfactory.nebius.com/v1/):
    • meta-llama/Llama-3.3-70B-Instruct: Handles high-throughput entity extraction, parsing raw markdown from arXiv into typed Pydantic claims with exact character offsets. Also runs background graph compaction during idle windows.
    • deepseek-ai/DeepSeek-R1 & DeepSeek-V3: Drives deep-reasoning hypothesis planning, research decomposition, and inference budget allocation.
    • Qwen/Qwen2.5-Coder-32B-Instruct: Generates unified Git diff code patches when extraction workers fail.
  • Tavily Grounding API (https://api.tavily.com):
    • TavilyClient.search: Conducts domain-filtered arXiv and competitor research.
    • TavilyClient.extract: Extracts clean, full-fidelity markdown and raw DOM snapshots from complex documentation layouts.
  • NVIDIA NIM Microservices (http://localhost:8000/v1):
    • nvidia/Llama-3.1-Nemotron-Ultra-253B-v1: Deployed in a self-hosted container on 8x H100/H200 nodes; acts as an impartial scientific methodology referee, evaluating experimental rigor and resolving contradictory claims.
    • Multi-LoRA PEFT Runtime (NIM_PEFT_SOURCE): Hot-loads specialized LoRA adapters for domain-specific evaluation rubrics.

2. Event-Sourced Epistemic Architecture

  • Typed Asynchronous Event Bus (AsyncEventBus): Built in Python 3.11 with non-blocking AsyncIO, dispatching typed events across structured namespaces (Lifecycle:*, Graph:*, Patterns:*, Behaviors:*, LLM:*, Patches:*, Approvals:*).
  • Reactive Property Graph: Maintains nodes and provenance edges (EXTRACTED_FROM, CONDITIONED_ON, CONTRADICTS). An automated Cypher pattern engine monitors every commit for epistemic anomalies (PAT_UNCONDITIONED_TABLE_METRIC, PAT_CONTRADICTORY_CLAIM).
  • Two-Stage Approval Gates (Gate A/B):
    • Gate A (Deterministic AST Analysis): Uses Python's native ast parser to inspect patch diffs. It strictly bans dangerous calls (os.system, subprocess, eval, exec, arbitrary network sockets) and prevents hardcoded fixture hacks.
    • Gate B (NIM Referee & Isolated Pytest Sandbox): NVIDIA NIM evaluates code logic and safety, followed by an automated test runner requiring a 100% regression pass rate in an isolated Docker sandbox.
  • Deterministic Artifact Compiler: Built with Jinja2 and pure Python, this engine compiles verified subgraphs directly into LaTeX Booktabs code and Markdown tables without allowing an LLM to generate or interpolate numbers.
  • Frontend Design System: Built to Stripe Enterprise standards using Tailwind CSS, Lucide icons, and modern ES2024 JavaScript, featuring synchronized two-pane scrolling, interactive SVG graph visualization, and real-time token telemetry meters.

Challenges we ran into

  1. Eliminating Unconditioned Metrics at the Schema Level: Early worker iterations frequently extracted throughput numbers without binding the underlying GPU model or precision format. We resolved this by building Pydantic v2 validation contracts that enforce mandatory condition binding, paired with reactive Cypher triggers (PAT_UNCONDITIONED_TABLE_METRIC) that reject unconditioned commits and prompt targeted extraction.
  2. Preserving Exact Character Offsets Across Document Transformations: Mapping an extracted fact back to its exact character range in an arXiv preprint is notoriously difficult after HTML stripping and markdown normalization. We built a dual-index coordinate tracker that preserves char_offset_start and char_offset_end relative to the raw source stream, allowing the UI to highlight the verbatim source sentence with cryptographic certainty.
  3. Preventing Autonomous Code Patches from Overfitting: When our Meta-Harness began writing Git diff patches to fix broken parsers, the coding model initially tried to take shortcuts—such as hardcoding expected fixture return values to satisfy unit tests. We engineered Gate A AST inspections to detect static return statements and enforce generalized regex and DOM selector patterns.
  4. Balancing 253B Referee Rigor with Token Efficiency: Invoking an ultra-large model like Nemotron-Ultra-253B for every extraction would quickly exhaust compute budgets. We designed a tiered routing architecture: Nebius Llama-3.3-70B handles 85% of high-speed extraction, reserving the 253B NIM referee strictly for when a Cypher contradiction pattern is triggered.

Accomplishments that we're proud of

  • True Zero Freehand Factual Generation: We successfully proved that AI agents can generate complex, academic-grade comparative benchmark tables without hallucinating a single figure. Every metric is provably anchored in verified graph nodes.
  • Working Autonomous Self-Patching Closed Loop: When a competitor changes their website DOM, SORTIFY autonomously intercepts the error, generates a Git diff patch, passes both Gate A (AST) and Gate B (NIM & Pytest), and hot-reloads the skill pack without restarting the runtime.
  • Sub-50-Cent Enterprise R&D Runs: By pairing Nebius Token Factory's extraordinary inference throughput with hard budget controllers, SORTIFY compiles verified multi-paper research tables for between $0.14 and $0.42 USD in compute.
  • Production-Grade Stripe Enterprise Design: Rather than presenting a bare-bones terminal demo, we delivered eight fully realized, interactive UI surfaces live at https://sortify.my.id.

What we learned

  • Structured Epistemic Graphs Beat Massive Context Windows: Dumping 30 research PDFs into a million-token context window produces fuzzy, unconditioned summaries with high hallucination rates. Structuring intelligence as typed graph nodes with mandatory condition binding produces provable, repeatable truth.
  • Defense-in-Depth for Self-Improving Agents: Autonomous code synthesis requires rigorous safety boundaries. Enforcing deterministic AST checks (Gate A) before LLM refereeing (Gate B) ensures that self-healing systems remain secure, robust, and free from regression.
  • Synergy of Specialized Foundation Models: No single AI model excels at every task. Nebius Token Factory provides unparalleled extraction and coding throughput; Tavily provides grounded real-time web ground truth; and NVIDIA NIM delivers high-parameter scientific methodology refereeing. In concert, they form a self-improving epistemic engine that is far greater than the sum of its parts.

What's next for SORTIFY

  • Autonomous Multi-Agent Tournament Evaluation: Expanding the Meta-Harness to host autonomous Elo-rated tournaments between fine-tuned LoRA adapters and model checkpoints.
  • Direct GitHub PR & Enterprise Jira Synchronization: Automatically converting compiled strategy matrices into production GitHub pull requests, architecture decision records (ADRs), and Jira epics.
  • Native Equation-Level LaTeX Verification: Ingesting arXiv LaTeX source packages to mathematically verify mathematical proofs, loss equations, and gradient derivations.
  • Federated Epistemic Networks: Enabling enterprise research consortiums to share verified benchmark subgraphs over decentralized ledgers while protecting proprietary model weights and internal benchmarks.

Built With

  • asyncio
  • fastapi
  • langchain-nebius
  • nebius
  • neo4j
  • networkx
  • nvidianim
  • pydantic
  • talivy
  • tokenfactory
Share this project:

Updates

Submission history