darelm

Darelm is an autonomous AI Analyst. Powered by Qwen, it writes and executes Python to uncover complex trends and instantly generates production-ready BI dashboards from raw data.

Comment

Inspiration

I hate data analysis. Not the insights — the process.

Every time I needed to run an ML experiment, I faced the same tedious ritual: load the data, inspect it, handle nulls, encode categoricals, split the dataset, pick a model, interpret the results. By the time I got to the interesting part, I'd already spent an hour on boilerplate work I could do in my sleep.

I wanted a system that handled the entire pipeline the moment I described my goal. Not an autocomplete tool. Not a Jupyter notebook assistant. An autonomous system that thinks through the problem, executes real code, evaluates its own outputs, and hands me a finished, interactive report.

There was also a more personal motivation. I've always been mildly irritated by how many "AI Data Tools" are just thin wrappers around a chat prompt. I wanted to build something that demonstrated what real software engineering looks like underneath an AI tool: secure execution sandboxes, stateful autonomous reasoning loops, decoupled microservice architectures, and a premium frontend. Data analysis as the interface; hardcore software engineering as the foundation.

That tension became Darelm.

What It Does

Darelm is a Qwen-powered data intelligence platform with three autonomous agents:

Agent 01 — Conversational Analyst lets you ask questions about your data in plain language. It reasons, writes Python, executes it in a secure cloud sandbox, observes the result, and reasons again until it finds the answer. Every number it reports was actually computed — never hallucinated.

Agent 02 — Autopilot Analyst takes a single goal and runs a complete, hands-off analysis. A Planner decomposes your goal into ordered steps. An Executor runs each step sequentially, feeding findings forward. A Synthesizer compiles everything into a structured, interactive report with charts, tables, and plain-language conclusions — without you touching it again after the initial prompt.

Agent 03 — ML Experimenter is the Data Scientist. Give it a target variable and a modeling goal, and it autonomously handles feature engineering, trains a classifier or regressor, evaluates performance, and extracts feature importance — all within a strict self-imposed time budget. If it can't finish in time, it returns whatever it completed honestly rather than failing silently.

All three agents support CSV, Excel, and live database connections. Every report is downloadable as PDF, CSV, or JSON.

How We Built It

All three agents share the same execution foundation:

  • Backend: Stateless FastAPI, with full conversation history reconstructed from PostgreSQL on every request — horizontally scalable by design
  • AI: Qwen models via the Qwen Cloud API, using native OpenAI-spec function calling — no regex parsing, guaranteed schema adherence
  • Sandboxing: E2B for secure, remote Python execution — the AI writes code, E2B runs it in an isolated container, the result comes back as a structured observation
  • Streaming: Server-Sent Events (SSE) for real-time thought streaming — the user sees the agent working, not a spinner
  • Frontend: React + Vite, Zustand for state, Tailwind for styling, Recharts for dynamic interactive visualizations
  • Infrastructure: Dockerized backend deployed on Alibaba Cloud ECS, with Alibaba OSS for dataset storage

The charting system deserves a specific mention. Rather than having the AI generate matplotlib images — which look cheap and break the UI — the agents are prompted to output structured JSON arrays representing data points. The React frontend intercepts this JSON and renders native, interactive Recharts components directly in the DOM. The user gets a live, hoverable chart. The agent never touches image generation.

Challenges We Ran Into

Dynamic chart rendering vs. static images Getting the LLM to reliably output structured chart JSON instead of writing matplotlib code required careful prompt engineering and strict output schema enforcement. The architecture had to change before the prompts could work.

The Ghost in the Machine: UnboundLocalError: cannot access local variable 'json' Agent 03 kept crashing with an error that made no logical sense. After significant debugging, the culprit was a rogue import json statement buried inside the backend's streaming function. In Python, a local import shadows a global import for the entire function block — breaking the code running above it. A brutal lesson in Python scoping that cost more time than it had any right to.

Docker ate the dataset Uploaded a Kaggle dataset, the database logged it perfectly, the UI showed it — then docker-compose down was run. The file vanished. Without an explicit volume mount (./uploads:/app/uploads), Docker wipes the container filesystem completely on restart. The database remembered a file the hard drive had already forgotten.

The sneaky background race condition A background FastAPI task was compressing uploaded datasets into .gz files and deleting the raw CSVs to save disk space — a deliberate optimization. But Agent 03 started throwing FileNotFound errors. The infrastructure was so fast it had already compressed and deleted the raw file before the agent could spin up its sandbox to read it. The system was working against itself. The fix required the sandbox loader to automatically fall back to the .gz version if the raw file no longer exists.

The zombie sandbox When a user refreshed the browser mid-experiment, the SSE stream connection dropped. The frontend had no session to reconnect to — but the E2B sandbox kept running, executing code, consuming credits, producing results nobody was receiving. This forced explicit sandbox lifecycle management: every sandbox ID is persisted to the database, orphaned sandboxes are detected on reconnect, and a cleanup routine kills any sandbox whose session has been inactive beyond a threshold.

Accomplishments That We're Proud Of

Autonomous charting Agent 02 proactively decides when data needs visual context, selects the correct chart type, and structures the data for the frontend to render as interactive Recharts components. The user doesn't have to ask for a plot. The agent decides.

Seamless agent handoff After reading a complete Autopilot report, clicking Chat with Agent 01 passes the entire report context to the conversational agent instantly. The conversation picks up exactly where the analysis left off — no copy-pasting, no re-explaining.

The two-timer architecture Agent 03 believes it has 5 minutes. The backend enforces a hard 6-minute kill switch independently. The 60-second gap lets the model wrap up gracefully rather than getting killed mid-sentence. It's a small detail that took real thought to design correctly.

Security by design Darelm executes AI-generated Python code on real user data. That forced a serious security architecture: SSRF protection on database connections, path traversal validation on file access, connection string encryption at rest, per-user rate limiting, httpOnly JWT cookies, and cross-user data isolation enforced at every query level.

The frontend Dark, monochromatic, no emojis, no purple gradients, no AI smell. It looks like something you would actually pay for.

What We Learned

AI is a component, not a product. Wrapping an LLM API call in a UI is not a product. The LLM is the engine — you still have to build the car. The execution environment, state management, UI renderer, data pipelines, security guardrails — none of that comes from the model. The AI only works because of the engineering surrounding it.

You cannot prompt your way out of a structural limitation. When the AI couldn't render matplotlib charts in a headless environment, tweaking the prompt wouldn't fix it. The architecture had to change: force the AI to output structured JSON, build a React charting engine to render it. The lesson is to constrain the AI structurally rather than begging it textually.

Defensive engineering is a mindset shift. The moment you build a tool that writes and executes arbitrary Python code on a server, your thinking changes from "How do I make this work?" to "How do I prevent this from destroying everything?" SSRF protection, strict sandboxing, encrypted credentials, hard timeouts — these aren't features you add at the end. They're decisions you make at the start.

Distributed systems don't behave sequentially. Managing state across a FastAPI backend, PostgreSQL, background workers, and E2B sandboxes simultaneously taught a hard lesson: you cannot assume things happen in order or complete successfully. The race condition and the zombie sandbox both came from that assumption. Explicit lifecycle management is not optional.

Think like a user, not an engineer. The original vision was a chat interface. But most users don't know what questions to ask their data. Agent 02 — the Autopilot — exists because the real user experience is handing someone a finished report without making them do the work of interrogating the system. The engineering insight is interesting. The user doesn't care. Just give them the answer.

What's Next for Darelm

Agent 04 — Data Engineer A dedicated "Data Janitor" agent. Upload three messy, unlinked datasets and Agent 04 autonomously identifies foreign keys, joins them, handles nulls, normalizes dates, and outputs a clean unified table for Agents 01–03 to work with. Darelm currently assumes data is at least somewhat workable. Agent 04 removes that assumption entirely.

Scheduled Autopilots Agent 02 is powerful, but you still have to trigger it manually. The next step is cron-based scheduling: "Run this Autopilot report every Monday at 8AM on our live PostgreSQL database and email the PDF to the executive team." Darelm stops being a tool you use and becomes an employee that works in the background.

1-Click MLOps Agent 03 currently trains a model and gives you a .pkl file. The next evolution is deployment: take the model Agent 03 just built and automatically spin up a micro FastAPI endpoint hosting it, giving the user a live prediction API URL they can plug directly into their own software.

Data Warehouse & SaaS Integrations CSV, Excel, and PostgreSQL are the foundation. Adding Snowflake, BigQuery, Stripe, Shopify, and HubSpot transforms Darelm from a data science tool into a universal business intelligence platform.

Multiplayer Workspaces Full team collaboration: shared dashboards, in-line annotations on Agent 02 reports, and the ability to tag a teammate to ask Agent 01 a follow-up question directly on a specific chart — without leaving the report.

Built With

Share this project:

Updates