Conquistadores

Inspiration

Team work makes the dream work! We believe this to be true also about the communities we live in. Therefore, we designed and implemented an Android-based intelligent app that offers to users the possibility to contribute to OpenStreetsMap in a fun way. Moreover, it enables users to discover the city and “claim” the area of their guild! Represent your crew by getting to the flag before your enemies, win experience and cool prizes. Conquistadores -- conquer the city with your friends!

What it does

Let’s look at the general use case:

Bob is walking through the city. Bob loves helping other people. What Bob loves even more is defeating his friends at any game he can (as we all do). Therefore, Bob authorized our app to track his location.

Bob passes now by an issue area -- a perfect flag to be conquered! -- as his pleasant, non-intrusive notification tells him. He wants to get from Mate to Crew Captain faster than his pirate friends, so he takes a photo and sends it through our app to the magical ML cloud. Once processed, the photo gives Bob rewards and experience, depending on how hard the flag is, how long it has been there, current events and Bob’s level! All factors will be thought so as to keep the challenge between effort and reward and keep Bob playing with his friends 🙂.

Game mechanisms:

basic idea: accumulate points and experience to progress in the game through photos that will finally contribute to OpenStreetMap
a higher level gives you more points for the same issue points can transform in prizes (apart from being great ego boosters)
the longer an issue stays unsolved, the higher the rewards (points and XP) for it
the higher the level, the more XP you need to advance
prizes can be coupons, real objects or even virtual prizes
achievements : first photo, 100th photo, first of your guild to get to a level, etc.
the code is customizable to have more teams (we can even go franchise-based teams, my young Padawan)
be active every day for XP and points multipliers, hidden events and flags (issues)

Another use case to help with labelling data: Fun 90-seconds mini games: categorize as many issues to win extra experience and points! Swipe left (for footway) and right (for primary) among as many photos as you can. Is the photo weird? Blurry? Downright wrong? Then swipe up and let us know! The more labels you swipe correctly, the more points you get. The correctness of a swipe will be decided after a short period of time (e.g. 2 weeks). The label of a photo will be decided by mutual anonymous vote of multiple users. After e.g. 100 users label a photo through this mini game, the majority decides and that photo automatically gets labeled. Human-helped AI. If the label is different than the one given by the ML, it could signal an issue to the devs: are the humans inaccurate or should the model be improved?

How we built it

Brainstorming what we would like to see in such an app and how other gamification apps managed to capture or attention and keep us playing
Designed in Figma most components
Chose the tech stack most advantageous to us and our purposes
While Front and Backend coded, ML coded and then trained the model
Integrated the ML model
Debug, debug, debug

Challenges we ran into

[ML] we tried using Transfer Learning from VGG19 and ResNet, but it was difficult to work with, slow to train and did not perform as expected
[ML] some data augmentation techniques, such as hue modification, are counterproductive
[FE] integrate OpenStreetMaps with the react-native-maps package
[BE] optimizing over the latency of the ML model and using caching mechanisms

Accomplishments that we're proud of

We implemented more features than expected
The accuracy of the ML classifier on unseen data is 85%
Developing from scratch to a final product on the mobile phone and integrating ML in it
The minimalistic and realistic frontend

What we learned

things might not go according to plan => a contingency plan is always good
KISS is always easier said than done
manually looking into a dataset, seeing how well a human understands it, is a great “waste” of time model callbacks are great to not miss the sweetest spot during training small data sets → easy to fall into overfitting

What's next for Conquistadores

Make the UI more user-friendly and stylish
Make the UX more gamified:
reminders (“100 XP more until you become Pirate Crew Master / Sir Lancelot!”)
unlockable areas if certain issues are solved
colourful, dynamic screens to attract users
week summary: who improved, who got lazy, best photos among friends etc.
if the user sends hard-to-classify photos (distinguished by ML’s classification numbers) => points withdrawn. This part assumes a well-tuned ML model, this is why it is considered future work.
For the ML side: We want to use image segmentation, more data augmentation and other computer vision ideas to have a more robust classifier. Methods like k-fold cross validation might have also helped with the reduced-size data set.
The current classifier has not been tuned and the architecture was not varied for performance. There could be better
Encourage sustainability! We will determine how a user got to a flag based on his speed, accelerometer and other gps data. An eco-friendly user (who uses the bike, rather than the car) will always get better rewards
Partnerships for the prizes