What Inspired Me
The inspiration for this project stemmed from a common frustration: data analysis can be incredibly tedious for those who don't have a strong grasp of Excel, Python, SQL, or Polars/Pandas. I noticed that business analysts and non-technical team members often struggle to extract simple insights from CSV files without relying on data engineers.
With the rapid advancements in Large Language Models (LLMs), specifically GPT 5.6 and Codex, I realized there was a massive opportunity to democratize data science. I wanted to build a bridge between complex data manipulation and plain English. The goal was simple: allow anyone to drag and drop their CSV files and ask questions like, "Merge the sales and customers datasets on 'CustomerID' and show me the top 5 regions," without writing a single line of code.
What I Learned
Building this project was a massive learning curve that touched on full-stack development, prompt engineering, and data processing.
Advanced Prompt Engineering: I learned how to structure prompts so that GPT 5.6 and Codex consistently output raw, executable Python code rather than conversational text. I learned the importance of injecting the exact CSV schema (headers and data types) into the context window to prevent the AI from hallucinating column names.
Backend Architecture: I gained deep experience with FastAPI, learning how to handle asynchronous file uploads and manage multiple Polars DataFrames in memory simultaneously.
Statistical Abstraction: I learned how complex mathematical and statistical operations can be abstracted away from the user. For example, calculating the standard deviation of a dataset relies on statistical formula
How I Built the Project:
I built the application using a modern, lightweight tech stack:
Backend: Python with FastAPI for high-performance, asynchronous API endpoints. Frontend: JavaScript, HTML, and CSS. Data Engine: Polars for heavy data manipulation and in-memory table operations. AI Integration: GPT 5.6 and Codex.
The Architecture Workflow:
A user uploads CSV files via the frontend interface. FastAPI receives the files and loads them into Polars DataFrames, extracting the column headers and basic statistics. When the user submits a natural language query, the backend constructs a highly specific prompt containing the user's request and the structural schema of the DataFrames. The prompt is sent to GPT 5.6 model through api, which generates the exact Polars code required to fulfill the request. The backend safely evaluates this generated code against the loaded DataFrames, formats the resulting table, and sends rows back to the frontend.
The Challenges I Faced
I initially used Pandas for data processing, but it became slow and memory-intensive when working with large datasets. Switching to Polars significantly improved processing speed and memory efficiency, making the application more scalable and reliable.
Built With
- codex
- polars
- python
Log in or sign up for Devpost to join the conversation.