Connectiki main page
We wanted to quickly find suggestions for things we make like, specifically with movies, music, and TV shows. We also wanted to do it without having to use heavy machine learning algorithms, training sets, or data collection.
What it does
Attempts to find new things that a user may like given their current interests. For each interest, Connectiki finds all related interests from the Wikipedia article of the interest (ie what does the article link to?).
It then presents the user with a list of all relatated interests that are shared between the user's current interest. In short, Connectiki uses Wikipedia as a relationship-graph
How we built it
We made heavy use of Python in order to find Wikipedia articles, scrape their contents, and find all links referenced by each article.
We also used Python, Flask, Jinja, and Thrift to create the server infrastructure necessary to display data to the user.
Challenges we ran into
The first major challenge we ran into was Hackathon "idea block". We came into the Hackathon with an ambitious star gazing project. However, upon learning that we did not have the time to ramp up on mapping technologies (GIS, Mapping, huge NASA datasets), we decided to abandon the project and start anew on day 2 of the hackathon.
This forced us to keep our project extremely limited in scope, avoiding any sort of "feature crawl".
In terms of project-specific difficulties, it was challenging to scrape Wikipedia pages to find relevant links (each article has wayyy too many links) and store this data in a way that would not slow down the user experience. To solve this, we made heavy use of in-memory caching and filtering out unimportant wikipedia links (Who cares about references, anyway?!)
Accomplishments that we're proud of
We made a working service that we would use. That's all that matters.
What we learned
- Research the difficulty of a project before attempting to execute it (Way too much wasted time)
- The structure and layout of Wikipedia URLs and articles
- How to quickly deploy a Python service
What's next for Connectiki
We are planning a few more features
- Also include backreferences in the Wikipedia dataset so that each article may have more references
- Include "context" so that the user knows exactly how items are related
- Include "category specific" searches (eg "I'm looking for new music")