TechSight

Inspiration

Spotting breakthrough technologies before they become mainstream is critical for researchers, investors, and R&D organizations. However, technological signals are siloed across disparate datasets: cutting-edge preprints, peer-reviewed journals, and international patent filings. Manually tracking thousands of weekly publications across multiple platforms with different schemas and delays is nearly impossible. We built TechSight to automate this intelligence gathering using unsupervised Natural Language Processing (NLP) and predictive machine learning.

What it does

TechSight is an end-to-end technology trend aggregation and forecasting platform:

Multi-Source Data Synthesis: Ingests and standardizes research literature (OpenAlex, arXiv) and patent registries (USPTO/EPO) into a unified relational schema.
Unsupervised NLP Topic Discovery: Applies TF-IDF vectorization and KMeans machine learning clustering ($k=12$) to automatically extract emerging technical themes without manual human taxonomies.
Explainable AI Momentum Index: Computes a transparent 5-factor composite trend score (0–100) combining CAGR, year-over-year consistency, acceleration (inflection point detection), corpus volume, and citation momentum.
Predictive Time-Series Forecasting: Deploys Linear Regression and Moving Average estimators with 80% confidence bands to forecast research and patent volume up to 2027.
Executive Analytics Dashboard: An interactive, multi-page Streamlit interface with real-time domain filtering, radar factor breakdowns, and data exploration.

How we built it

Data Ingestion & Integration: Python connectors with automated retries, rate-limiting, and cursor pagination querying OpenAlex Works and arXiv APIs.
AI & NLP Pipeline: Scikit-Learn pipeline performing n-gram TF-IDF feature extraction, cosine similarity calculation, and KMeans centroid clustering validated with Silhouette metrics.
Predictive Modeling: Supervised time-series models evaluated against historical trajectories using standard validation metrics (MAE, RMSE, MAPE).
Backend & Frontend: High-performance FastAPI REST API paired with an executive Streamlit dashboard styled using Plotly interactive visualizations.

Challenges we faced

API Limitations & Delays: Patent data suffers from publication delays and strict federal API authentication. We engineered a resilient fallback caching layer to ensure continuous data availability without pipeline crashes.
Schema Normalization: Unifying heterogeneous document schemas (inverted abstract indices from OpenAlex, XML from arXiv, and CPC classifications from patents) into a single analytical vector format.
*Algorithmic Explainability: Balancing sophisticated predictive modeling with explainable scoring so domain scouts understand *why a technology received a specific momentum index.

Accomplishments that we're proud of

End-to-End Pipeline: A fully automated 6-stage pipeline from raw API data ingestion to machine learning forecasting and interactive visualization.
Rigorous Validation: 100% test pass rate across 12 pytest suites covering data cleaning, clustering convergence, and forecast bounds.
Executive UX: Designed a dedicated, clean multi-page executive platform with sub-second responsiveness.

What we learned

The nuances of unsupervised clustering on domain-specific scientific text.
How multi-source data synthesis (pairing research velocity with intellectual property filings) offers a far more accurate adoption signal than either source alone.
Best practices in building resilient, modular AI data pipelines with graceful API degradation.

What's next for TechSight

Integrating transformer-based embeddings (e.g., SciBERT) for deeper semantic clustering.
Adding real-time alert notifications for technology inflections and rapid citation surges.
Expanding coverage to European (EPO) and World Intellectual Property Organization (WIPO) global registries.