The project required to approximate the shortest path between two points along a graph network for large sparse graph. This posed good challenges in algorithms, data structures, data handling and was thus a captivating challenge.
Challenges we ran into
The main challenge in the project was data handling. The map images consists of 4.6 billion pixels which is a really huge amount of data to handle. Another challenge was the execution time and computation power (resource limitations). Therefore the project required innovative techniques to handle the data and keep within the available resources.
Accomplishments that we're proud of
We successfully handled the huge sparse dataset through a exhaustive use of data handling, scaling and algorithms.
What we learned
We had to perform various optimization on system level (calculating bits and bytes) to perform the task within available resource. Also, extensive use to data structures improved our command in that area. Above all, data handling was the major learning through this project.
Our second challenge is the Conocco Phillips challenge. The details of the challenge can be found here: https://medium.com/@dhruvsandesara/db58f86000cf