Proper digital earth reconstruction is a must if we want to keep improving the many of the services we already use. This is especially true for Open Source services such as Open Street Map (OSM), where we face many challenges. OSM issues refer to wrong/missing descriptions of the features at some locations. Fixing these issues in an integral task of digital earth reconstruction. Thankfully, thanks to powerful and carefully tuned Machine Learning models, we can, at least, partially automate the process of repairing the issues. Given an image, such a model can correct the label of an OSM issue, by for example deciding whether there is a footway or a highway at a given location. But where we still need the help of the community is in the collection of quality images from issue-location. And that's where you come in! With our game ConquerShot, we motivate the user through a bit of healthy competition to go out and about to issue locations and to carry out several tasks, helping us in fixing difficult OSM issues. In the end, this is nothing, but one large collaboration, to help improve the world, by improving the digital services that push it forward. May the quickest and most motivated Player build the largest empire and reign in the land of Conquershot!
What ConquerShot does ...
Players can create and enlarge their own empire in the real world by uploading pictures to fix issues and conquer areas. When the player fixes an issue, their kingdom's influence propagates and puts pressure on the neighboring kingdoms. By allowing different players to fix the same issue, they can compete for territories. Yet the first player gets the most score points! As an intended side effect, our model gets multiple pictures for better classification. The faster you fix issues, the better for your score and territory! The uploaded images will then be evaluated by two different self-developed Machine Learning models that are working on fixing the issue and classifying the displayed object into either primary, footway, or neither. All in all - the user, as well as OpenStreetMap benefit from the application equally.
How we built it
- In the backend, we built a machine learning pipeline that mainly consists of two image classification models, which are based on a pre-trained ResNet-18 backbone from torchvision library and fine-tuned afterwards with the data provided by the sponsor. For the first model, we additionally sampled data from MS-COCO dataset. The trained model is stored in pth. format and the evaluation results on test data are saved in the csv. files. Furthermore, the web-app is built on top of a Flask application, that handles all the logic of the game and communicates efficiently with the front-end.
Challenges we ran into
There existed some edge cases in the dataset where the feature "footway/primary" is even difficult to determine manually. Such cases in the dataset could mislead the classifier to make the predictions. Also the scale of the test dataset originally is not large enough to make us fully confident on the model performance. Computing and propagating the kingdoms' influence throughout the map, especially at competing tiles. To overcome this, we created an algorithm that computes a diminishing score radiating from a fixed issue's tile, and the player with the highest score for a given tile conquers it until another player triumphs with a higher score. Yet the biggest challenge was creating a game world overlaying the OSM view and syncing with zoom and panning, as well as translating between real world coordinates and OSM's grid system. This was overcome by combining two different zoom levels through projecting the coordinate system from one onto the other.
Accomplishments that we're proud of
- The training of the ML model did not take long and the accuracy was really good (~94.4% acc.)
- The general idea for our game strategy was innotative with displaying the territories by using an influence-based system on OSM tiles
- Making use of discord channel enables an instant communication with the advisors from the sponsor
What we learned
The project provided us with an amazing opportunity to dive into new technologies and to solve difficult problems. Our Data Science and Machine Learning experts were able to further their skills in computer vision and image classification, while our frontend and backend developers got the chance to learn and improve their React and Flask skills. Furthermore, we feel confident know with the leaflet API.
What's next for ConquerShot
- Extension to more OSM features: So far, we only deal with the "highway" feature with a simple binary classifier. The model can be directly re-used for other binary classified features. And additionally, it can also be easily extended to a multi-classifier to handle more complicated issues. The game will directly benefit from more issues as it would make it more interesting and further competition.
- Route planning from the player's location to issues
- Smart notifications that inform the player when he/she is close to an issue