The Gap Engine

proof demo mode
graph built

Inspiration

I kept thinking about how science actually moves forward. We celebrate the answers but the real bottleneck is always the questions. Someone had to ask "what if network topology affects how disease spreads" before epidemiology changed forever. That question existed in the structure of the knowledge graph years before anyone asked it. So I built something to find those questions automatically.

What it does

The Gap Engine ingests scientific papers from Semantic Scholar, builds a knowledge graph where nodes are concepts and edges are co-occurrence/citation strength, and then algorithmically finds the gaps. Two types: structural gaps (concept pairs strongly connected through a bridge node but with zero direct research linking them) and cross-domain gaps (research communities from different fields with high conceptual similarity but no cross-pollination). Every gap gets a leverage score based on betweenness centrality, community reach, and citation momentum. The highest-scoring gaps get turned into specific research questions by an LLM. There's also a historical validation mode where you feed it papers from 2005 and it surfaces the network science/epidemiology gap that actually got closed later. That's the proof of concept.

How we built it

Python + FastAPI backend. NetworkX for the graph. Semantic Scholar's public API for paper data. Groq (llama-3.3-70b) for question generation because it's fast enough to run live. React + TypeScript frontend with D3 force graph for the visualization. The gap detection is pure graph theory: Type A gaps score on $\text{score}(A,C) = w_{AB} \cdot w_{BC}$ for indirect paths with no direct edge, and Type B uses cosine similarity between community concept vectors across different fields of study. Leverage scoring is a weighted sum: $0.35 \cdot \text{betweenness} + 0.30 \cdot \text{community reach} + 0.25 \cdot \text{citation momentum} + 0.10 \cdot \text{cross domain bonus}$.indirect paths with no direct edge, and Type B uses cosine similarity between community concept vectors across different fields of study. Leverage scoring is a weighted sum: $0.35 \cdot \text{betweenness} + 0.30 \cdot \text{community reach} + 0.25 \cdot \text{citation momentum} + 0.10 \cdot \text{cross domain bonus}$.

Challenges we ran into

Getting the gap detector to surface actually interesting gaps was way harder than expected. First version kept returning boring intra-CS pairs like "pattern recognition to data science" which is not a gap, that's just the field. Had to bias the demo graph toward cross-domain inputs and raise the cosine similarity threshold on cross-domain detection significantly. Also the field labels were broken, nodes were showing up as "Political Science" when they were clearly epidemiology. Ended up writing a canonical field normalizer in the ingestion layer to fix that. The historical validation mode was also initially cheating by hardcoding the answer, had to refactor it to run the generic detector first and only label the match afterward.

Accomplishments that we're proud of

The historical proof actually works without being rigged. Feed it a 2005 corpus and the network science/epidemiology gap surfaces organically. The leverage scorer is doing real graph math (betweenness centrality, community detection) not just fake confidence bars. The inversion detector is a feature I haven't seen anywhere else: it finds studied phenomena and surfaces their unstudied inverses, verb-aware so it reads "X inhibits Y, does Y inhibit X?" instead of something generic.

What we learned

The hardest part of building a gap detector is defining what "interesting" means mathematically. Bridging concepts that sound profound are usually just philosophical noise words like "theory" or "system" so I had to build a stopword filter specifically for meta-concepts. Also learned that graph topology alone isn't enough, you need citation momentum to separate gaps that matter now from gaps that nobody is working on because they're genuinely not productive.

What's next for The Gap Engine

Plug in real-time queries so anyone can type a topic and get the nearest high-leverage gap in seconds. Cross-reference gaps with NSF/NIH funding data to find questions that are both scientifically important and underfunded. Build alerts so you can subscribe to a gap and get notified when a paper starts edging toward closing it. Long term this should be a tool every PhD student opens before choosing a dissertation topic.