SQL LLM AI Agent: Chat with Your Database

Inspiration

Valuable datasets are often locked behind SQL databases, making them inaccessible to people without technical expertise. Researchers, students, journalists, and organizations frequently struggle to retrieve insights because they cannot write complex SQL queries. We wanted to remove this barrier by allowing anyone to interact with databases using natural language.

What it does

Our system converts natural language queries into executable SQL, executes them against a connected database (e.g., a .db file), and returns the results. Instead of relying on predefined schemas, the AI dynamically identifies the relevant tables and extracts their schema at runtime, allowing it to work seamlessly across databases with varying structures and multiple tables. Query results are stored in conversational memory, enabling follow-up questions and context-aware analysis of previously retrieved data.

How We Built It

Our pipeline uses Qwen2.5-7B as the orchestrator.

1. Intent Detection: Every user query is first passed to Qwen2.5-7B to determine whether it is: a general conversational query, or a SQL/database query. General conversations are answered directly by Qwen2.5-7B without invoking the database.

2. Relevant Table Selection: For SQL-related queries, the same Qwen2.5-7B model identifies only the tables required to answer the question. We improve this step using few-shot prompting, providing examples such as:

User: Get the latest GDP per capita of the country with the lowest literacy rate.
Output:
GDP_per_capita
Literacy

User: Which country has the highest unemployment but the lowest poverty?
Output:
Unemployment
Poverty

These examples teach the model to select only the relevant tables from the database, minimizing unnecessary context.

3. Schema Extraction: Once the relevant tables are selected, we extract only their schemas and pass them—along with the original user query—to the Arctic Text2SQL-7B model.

4. SQL Generation & Execution: Arctic Text2SQL-7B generates the SQL query, which is then executed against the database to retrieve the requested data.

5. Conversational Memory: The retrieved results are stored in the chat memory. This allows Qwen2.5-7B to retain awareness of previous queries and extracted data, enabling natural follow-up questions, multi-turn reasoning, and conversational data analysis without requiring the user to repeat context.

This is Whole visual of it Arch

Nature of Data

The dataset used in this demonstration consists of 25 real-world tables extracted from World Bank data, making it representative of a production-scale Text-to-SQL use case. Each table contains a specific socioeconomic or demographic metric, such as GDP, birth rate, death rate, fertility rate, and literacy rate, for approximately 195 countries, spanning the years 1960–2025.

Each table contains roughly 12,675 records (195 countries × 65 years), resulting in a multi-table relational database with hundreds of thousands of records. All 25 metric tables are stored in a SQL database and connected to our Text-to-SQL agent.

This setup enables the system to answer complex, multi-table analytical queries. For example:

Give the name of the country with the highest fertility rate in 2022, and retrieve the death rate of that country in 1977.

To answer such queries, the pipeline first identifies the relevant tables (Fertility Rate and Death Rate) using the few-shot table selection model. The extracted schemas are then passed to Arctic Text2SQL-7B, which generates the appropriate SQL query. The query is executed against the database, and the retrieved results are returned to the user in a conversational format.

List of files

Challenges we ran into

One of the biggest challenges was selecting the right Text-to-SQL model. We initially used Qwen2.5-Coder 7B, which performed reasonably well but began hallucinating when generating queries over schemas with 25+ tables, making it unsuitable for our use case. We then migrated to Snowflake Arctic-Text2SQL 7B, which significantly improved query generation accuracy and reliability. Initially, we deployed the Q8 quantized model (~8.1 GB), but our target hardware included GPUs with only 4 GB VRAM. To improve deployment efficiency, we switched to the Q4_K_S quantized variant (~4.77 GB) that model is not present in ollama so we use llama.cpp backend for this. using GGUF extension compressed models for llama.cpp. Despite its much smaller memory footprint, it exhibited negligible accuracy degradation compared to Q8, making Q4_K_S the optimal performance-to-resource trade-off for our application.
Another challenge was fine-tuning the prompts for the Text-to-SQL model, general LLM, and table selection pipeline. While less significant than model selection, prompt engineering was critical to minimizing hallucinations and ensuring consistent outputs. Through iterative experimentation and leveraging LLM-assisted prompt refinement, we developed prompts tailored to our use case, resulting in improved accuracy and reliability across the pipeline.

Accomplishments that we're proud of

We're proud of the entire engineering effort, from infrastructure to model optimization. We set up llama.cpp with CUDA acceleration, use quantized GGUF format for efficient inference, engineered high-quality prompts for accurate SQL generation, built reliable query-to-table retrieval, and made numerous low-level engineering decisions that collectively improved performance, accuracy, and efficiency.

What we learned

This project strengthened our understanding of LLM quantization, prompt engineering, and practical LLM deployment. We also expanded our knowledge of specialized models, particularly Snowflake Arctic-Text2SQL 7B, more revision on techniques of few shot learnings, hands on llama.cpp, and gained valuable experience in building an end-to-end AI system. Beyond the technical learnings, developing this project was both challenging and enjoyable.

What's next for SQL LLM AI Agent: Chat with Your Database

We have not finalized the next steps yet. However, we plan to continue exploring projects in the ML systems and AI infrastructure space, focusing on building efficient, production-ready machine learning applications.