I wanted to find the next big thing like Bitcoin. I saw it once happen with Binance on the Cryptocurrency Exchange market via a Python Package Repo. Is there a way to systematically discover the next big thing?
It boils down to getting access to asymmetric information. Asymmetric information is a relatively new concept being explored in economic theory. It allows individuals to make more informed decisions because they have access to more information than the other party.
What it does
This project allows you to get access to asymmetric information. It analyzes thousands of potential data entries from various different data feeds to identify hot topics. It essentially is a custom "trends" finder.
How I built it
It finds trends by scraping the package registry of Python, Ruby, and JS packages. It also analyzes Arxiv Research Papers. It then creates a custom dashboard for the user to consume the content on these registries.
Challenges I ran into
Performing keyword extraction proved to be difficult without much knowledge of NLP. Additionally, building a set of scrapers robust to API changes over millions of scraped data was difficult.
Accomplishments that I'm proud of
- Coming up with the non-trivial and non-obvious data feeds such as Python Package Registry
- Discovering Asymmetric information as a theoretical underpinning.
- Using an auto-generated code framework through a clever repurposing of a framework (Jinja)
- Generating millions of data entries after only writing a few lines of code for each data source.
What I learned
- How building resilient APIs is difficult.
- Why asymmetric information captures the problem well.
- How important interfaces and inheritance can be in removing code duplication.
What's next for Asymmetry: Towards finding the next big thing like Bitcoin
To build the last step. A data analysis technique with ML or NNs for analyzing the raw data. Things include keyword extraction, line charts for trends over time, and word clouds.