• 

  Architecture-Diagram

Skill Gap India 🎯

Inspiration

60% of Indian engineering graduates are unemployable — not because they lack intelligence, but because colleges teach outdated curricula while industry moves fast. We kept seeing our peers graduate and struggle to get jobs despite 4 years of hard work. Nobody had measured this gap at scale. We decided to build the tool that does.

What We Built

A full data intelligence platform on Databricks that:

• Ingests 5000 real-world job postings
• Processes them through a Bronze → Silver → Gold Delta Lake pipeline
• Calculates a Gap Score for each skill domain using Apache Spark
• Recommends free SWAYAM/NPTEL courses via FAISS vector search
• Tracks all experiments with MLflow
• Displays insights on a live Databricks Dashboard

How We Built It

We used the Databricks Lakehouse architecture as our core:

Bronze Layer — Raw job postings ingested as-is into Delta Lake

Silver Layer — PySpark cleans, normalises, and extracts skill tags from raw data

Gold Layer — Aggregated Gap Scores calculated per skill domain:

$$\text{Gap Score} = \frac{\text{Industry Demand} - \text{College Supply}}{\text{Industry Demand}} \times 100$$

FAISS Recommender — Sentence embeddings map each skill gap to the most relevant SWAYAM/NPTEL course semantically

MLflow — Every pipeline run tracked with parameters and metrics

Challenges We Faced

• Data quality — Real job postings had inconsistent skill naming. PySpark normalisation was critical to make gap scores meaningful
• FAISS on Databricks Free Edition — Setting up vector search within free tier compute constraints required careful memory management
• Time — Built end-to-end in a single day at a campus hackathon. Every architectural decision had to be fast and deliberate

What We Learned

• How to design a real Medallion Architecture on Databricks
• How semantic search differs from keyword matching and why it matters for recommendations
• That India's employability crisis is measurable — and solvable with the right data infrastructure

What's Next

• Real-time job scraping pipeline replacing static dataset
• College curriculum ingestion module for institution-level gap scoring
• Student-facing web app using Databricks Apps
• Expansion beyond engineering to MBA, medical, and law graduates

Built With

• databricks
• pyspark
• python
• sql