open street map data
initial crime street map
initial cluster for graphing
nodes that represent crime in philadelphia area
network of nodes that represent crime in temple area
geoJSON open street map data
We created this application with temple students in mind. A lot of us live offcampus and safety is something thats really important to us and our friends so we created an application that would use crime data in order to keep students safe on-campus and off.
What it does
Hindsight is a web application that uses millions of lines of data taken from openstreetmap's api in order to predict the safest path students should take to their destination depending on the time of day.
How we built it
Challenges we ran into
One of the biggest challenges we ran into was dealing with millions and millions of lines of data from the open street map api. Some of us have had experience with big data sets but nothing of this scale and capacity. One of the things we really had to look out for when implement different graph algos was our runtime complexity if our algo was more than O(n^2) it would take forever to compile. There was also a lot of data cleanup we had to do for those millions of lines in order to parse it into our custom data structures there was over 42 different types of crime in our initial data but we had to parse out for ones that were the most important to use for example assault and theft were highly weighted in our graph while gambling was not. Another thing that was a challenge was trying to create a machine learning model for our application. Another challenge was transforming and dealing matricies with numpy for our data visualization models. Learning react native was challenge for a lot of us for our front end it was a new technology but we really wanted to challenge ourselves.
Accomplishments that we're proud of
We're proud of being able to tackle and parse such a large dataset. Our initial dataset of crime in the temple area from 2015-2019 was around 2million rows long. We were able to parse through this data, create an accurate depiction of what it means, and build a tool that takes advantage of everything we learned. For a lot of this was also the first time we've worked with technologies like firebase, react.js, numpy, and geojson.
What we learned
- how to parse millions of lines of data efficiently
- how to apply machine learning models large datasets
- how to use react.js for front end with a node.js backend
- how to use firebase for application that require a lot of data
- how to effectively view a streetmap and manage graph data with million of points
What's next for Hindsight
We hope to expand our application and make it more accurate. We didnt have a lot of time during the 12 hackathon to accurately create routes for our data and properly create a machine learning model for future crime prediction