-
-
This screenshot shows all the details of the analysis of the dataset uploaded by the user!
-
There is an integrated AI assistant model that will prompt and decode data for the user.
Inspiration As a solo data scientist I often repeat the same pattern: load a CSV, run EDA, search for target leaks, handle missing values, build a baseline model, tune it a bit… and explain the same things again and again to non-technical stakeholders. I wanted a tool that feels like a mid-level DS sitting next to me: it looks at the dataset, explains what’s going on, trains a reasonable baseline model, and tells me “here are the next 3 things you should try”. That’s how DataPrism (Nikita DS Agent) started. What it does DataPrism turns raw tabular data into a structured DS report and baseline model in a few clicks: Upload a CSV / Excel / JSON / Parquet file and optionally specify the target column. Automatic EDA: shape and basic statistics missing values and high-cardinality features simple checks for class imbalance and data quality issues Auto task detection (classification vs regression) and training of a baseline model using scikit-learn (RandomForest / GradientBoosting / Logistic/Linear Regression). Dataset health score (0–100) with human-readable reasons: too many missing values, strong imbalance, too many unique categories, etc. Feature importance visualization and suggestions for new features / preprocessing. Experiment plan: a prioritized list of “Do now / Next / Later” actions to improve the model. Run history: every upload is saved as a run_id and can be reopened in the UI. Inference mode: choose a previous run, upload a new file, get predictions + a small preview table. DS-agent chat: ask questions like “How can I improve this model?” or “What’s wrong with this dataset?” and get an LLM-based answer grounded in the current run. How we built it Backend Python + FastAPI (served via Uvicorn). pandas / NumPy for data loading and EDA. scikit-learn for modeling (pipelines, preprocessing, baseline models, metrics). A simple “run manager” that saves metadata (EDA summary, model type, metrics, plots, feature importance) on disk and exposes /runs, /runs/{id}, /runs/{id}/predict, /upload, /ask. Frontend Pure HTML/CSS/JavaScript (no heavy framework) with a custom dark dashboard UI. Left panel: upload + inference. Middle panel: EDA text, model card, metrics, health, recommendations, feature engineering hints, experiment plan, predictions preview, plots. Right panel: list of recent runs and the DS-agent chat. LLM integration /ask endpoint takes run_id + question, builds a compact summary of the run (health, metrics, top features, recommendations) and sends it to an LLM to generate DS-style answers. Challenges we ran into Generalizing EDA for “any” tabular dataset Designing heuristics that work for sports, finance, marketing and other domains without manual tuning. Performance vs. detail Large files can be heavy; we had to limit rows, keep computations reasonable and still provide meaningful insights. Explaining results, not just printing numbers Translating metrics and EDA findings into short, actionable recommendations is surprisingly hard. LLM alignment Making the DS-agent answers consistent with the actual metrics and not hallucinate things that are not in the run metadata. UI layout Packing a lot of information (EDA text, health score, feature importance, experiment plan, plots, chat) into one screen without overwhelming the user.
Accomplishments that we're proud of We have a working end-to-end prototype: upload → EDA → baseline model → health and recommendations → saved run → inference → DS-agent Q&A. The app works on real datasets (including 20–30k+ rows) and automatically adapts to different schemas. We implemented a clean run history with run_id and a simple inference API, which already feels close to a real DS assistant. The dark dashboard UI is minimal but expressive: health score, feature importance bars, experiment plan and plots are all readable in one place. The DS-agent is already helpful for quick “What should I do next?” questions.
What we learned A lot of DS “magic” can be automated with solid heuristics and good defaults, even before full AutoML. UX matters: data scientists want transparency and control, not just a black box that prints a single score. It’s important to design the system around runs (experiment tracking mindset) from day one if you want to be production-ready later. LLMs are powerful as copilots, but they must be grounded in real metrics and metadata to be reliable.
What's next for DataPrism Smarter auto-modeling: cross-validation, automatic model selection, basic hyperparameter tuning and better handling of time-series splits. Richer experiment tracking: store configs, model artifacts and metrics, and expose a /runs/{id}/report endpoint for exporting human-readable reports. Stronger DS-agent: more structured reasoning over runs, code-ready suggestions, and explanations tailored to different skill levels (junior DS vs. business user). Multi-user & cloud-ready version: authentication, separate projects, and deployment as a small SaaS service. Integrations with external storage and BI tools so DataPrism can fit into existing data workflows in real companies.
Log in or sign up for Devpost to join the conversation.