Inspiration

Finding the right dataset for research can take hours. Researchers often have to search multiple websites, understand unfamiliar data, and figure out whether it's actually useful.

We also noticed another problem: research gets disconnected. You might have explored something similar before, but there's no easy way to connect your previous questions and findings.

That's why we built DataOPedia. We wanted to make it easier to find, understand, and connect research in one place.

What it does

DataOPedia is an AI-powered research workspace that helps researchers discover datasets, evaluate their quality, and explore connections between research questions.

Here's how it works:

  1. Ask a question: Enter a research question in plain English.
  2. Discover datasets: Gemini understands the question and helps find relevant datasets from real public repositories.
  3. Analyze the data: Pandas and NumPy calculate statistics, identify missing values, and summarize the dataset.
  4. Evaluate suitability: Gemini uses verified data and statistics to explain how useful a dataset might be for the research question.
  5. Connect research: An interactive knowledge graph helps researchers explore relationships between questions, datasets, and previous findings.

Instead of just giving researchers a list of links, DataOPedia helps them understand what the data contains and how it connects to their work.

How we built it

We used Python and FastAPI for the backend and Google Gemini to understand research questions and evaluate datasets.

The main technologies include:

  • Gemini API: Understands questions and provides research assessments.
  • Pandas and NumPy: Perform statistical analysis on real datasets.
  • Vector embeddings (384 dimensions): Represent research questions numerically so we can compare their meaning.
  • Cosine similarity: Identifies related questions even when they use different words.
  • Knowledge graph: Visualizes connections between research questions and datasets.
  • Database and caching: Store research history and reuse previous analysis results.

One important decision was to never let Gemini invent datasets or statistics. We retrieve real data and calculate the statistics ourselves. Gemini helps interpret the results.

Challenges we ran into

1. Gemini API limits and reliability

We ran into API rate limits, timeouts, and failed requests. Repeated AI calls also made the app slower.

We tested different models, improved error handling, and introduced caching to reduce unnecessary requests.

2. Connecting related research questions

Two questions might mean the same thing without using the same words.

We explored vector embeddings and cosine similarity to identify these relationships. We also learned that similar questions don't always require the same datasets.

3. Working with different datasets

Public datasets have different formats, variable names, and missing values. We built a profiling pipeline to analyze them consistently.

Accomplishments we're proud of

  • Built a workflow that goes from a research question to real dataset discovery and statistical analysis.
  • Combined Gemini with verified dataset information instead of relying on AI-generated facts.
  • Developed structured research suitability assessments.
  • Designed a semantic search approach using vector embeddings.
  • Built the foundation for an interactive research knowledge graph.
  • Introduced caching to reduce repeated processing and API usage.

We're especially proud that DataOPedia goes beyond a typical AI chatbot. It focuses on helping researchers work with actual data.

What we learned

Our biggest lesson was that AI works best when it's combined with reliable data and traditional computation.

We learned how to use Gemini for interpretation while letting Python handle statistical calculations.

We also learned about vector embeddings, semantic similarity, knowledge graphs, caching, and handling unreliable APIs.

Most importantly, we realized that researchers don't need AI to do all their thinking. They need tools that make the repetitive parts of research easier.

What's next?

We want to continue improving DataOPedia by:

  • Making the interactive knowledge graph more powerful and customizable.
  • Supporting more public dataset repositories.
  • Improving semantic connections between research questions.
  • Adding deeper dataset comparisons and visualizations.
  • Supporting collaborative research workspaces.

Our goal is simple: help researchers spend less time finding data and more time understanding it.

Built With

Share this project:

Updates

Submission history