NexusFlow AI

Inspiration

NVIDIA's open-weight Nemotron family—spanning from the efficient 30B MoE up to the 550B Ultra frontier reasoning model—represents one of the most capable foundations for long-horizon agentic workflows. Nebius Token Factory brings these models to life with serverless, autoscaling, OpenAI-compatible endpoints and per-token pricing.

However, moving from a raw LLM API endpoint to an enterprise-grade agent product introduces critical operational hurdles:

• Security vulnerabilities: Exposing provider keys to frontend bundles.
• Cost runaway: Lack of quota controls, budget ceilings, and token spend telemetry.
• Workflow bottlenecks: Managing long Server-Sent Events (SSE) streams and concurrent multi-step document pipelines without drops.

We engineered NexusFlow AI to bridge this gap: an open-source, resilient control plane designed specifically for NVIDIA Nemotron models hosted on Nebius Token Factory.

What it does

NexusFlow AI delivers a full-stack control plane and copilot for running autonomous workflows:

1. Zero-Trust Provider Architecture: Only the FastAPI backend holds NEBIUS_API_KEY. The browser interacts solely through an authenticated, audited, and rate-limited gateway—eliminating credential leak risks.
2. Multi-Tier Nemotron Model Routing:
   • Fast Tier (nvidia/Nemotron-3_5-Lightning & Nemotron-3-Nano-30B-A3B): 1M context, ultra-low cost ($0.06/1M input tokens) for high-throughput tasks, summarization, and quick agent steps.
   • Balanced Tier (nvidia/nemotron-3-super-120b-a12b): 120B MoE for structured extraction, verified JSON generation, and multi-turn workflows.
   • Frontier Tier (nvidia/Nemotron-3-Ultra-550b-a55b): 550B MoE with 1M context for complex reasoning, planning, and code synthesis.
   • Vision Tier (nvidia/Nemotron-Nano-V2-12b): Multimodal document and visual reasoning.
3. Streaming Agent Copilot: Low-latency Server-Sent Events (SSE) streaming with live Time-To-First-Token (TTFT) and token throughput telemetry.
4. Document Ingestion Engine: Streams PDF, DOCX, TXT, and CSV uploads with magic-byte validation, EICAR security checks, semantic chunking, and embedding generation via Qwen/Qwen3-Embedding-8B.
5. Resilient Background Jobs: Async worker pool with heartbeat monitoring, exponential backoff retries, and graceful queue drains.

How we built it

• Frontend: Next.js 16 (App Router), React 19, TypeScript, and Tailwind CSS v4 providing a responsive streaming interface and metrics dashboard.
• Backend: Python 3.12, FastAPI, Pydantic v2, and SQLAlchemy 2.0 (async) with custom request tracing and automatic log credential masking.
• Inference & Models: Direct integration with Nebius Token Factory (https://api.tokenfactory.nebius.com/v1) using retry policies, jitter, and automated fallback to curated model registries.
• Storage & Portability: Dual-database architecture supporting PostgreSQL 17 via asyncpg connection pooling in production, with seamless zero-config SQLite + aiosqlite support for local development and CI testing.
• Containerization: Multi-stage Docker Compose orchestration for database, migrations, API, and UI.

Challenges we ran into

• Resilient SSE Streaming: Buffering long-context responses from Nebius Token Factory through FastAPI to Next.js clients without chunk latency or connection drops.
• Cross-Database Type Portability: Ensuring SQLAlchemy models maintain native JSONB performance and UUID indexing on PostgreSQL 17 while retaining full compatibility with SQLite for fast automated test runs.
• Key Security & Redaction: Building strict middleware sanitizers to ensure that sensitive tokens and authorization headers never touch application logs or browser bundles.

Accomplishments that we're proud of

• Monorepo architecture that starts up with clean separation of concerns and production-ready safety rails.
• Intelligent tier-based routing that pairs the right Nemotron model with the right task complexity, optimizing both speed and token expenditure.
• Over 60+ automated unit and integration tests verifying schema compatibility, error handling, and worker resilience.

What we learned

• The exceptional reasoning performance and high-throughput benefits of NVIDIA Nemotron 3.5 Lightning and Super 120B on Nebius Token Factory.
• Designing async job queues that survive worker crashes through heartbeat-based stale recovery.

What's next for NexusFlow AI

• Visual node-based workflow builder for chaining Nemotron agents into multi-step DAGs.
• Multi-tenant enterprise workspaces with granular RBAC and hard budget spend caps.
• Production connectors for Slack, GitHub, Linear, and Notion.

Built With

• docker
• fastapi
• nebius-token-factory
• nextjs
• nvidia-nemotron
• postgresql
• pydantic
• python
• react
• sqlalchemy
• tailwindcss
• typescript