Inspiration
Sometimes nothing matters but collecting as much information as possible about something - information that might not be indexed or exist at all. Google processed 3.5 billion searches a day in 2018 and 'googling' is the dominant verb for searching, yet there exists a dearth of tools for massive internet-assisted searches, or Human Search Engines.
A brief overview
The future of search will require expanding beyond indexing, and focusing on searching socially and algorithmically by both recruiting people and giving them the tools to help search for what they value most - whether it's a missing child or a priceless possession. The search3 paradigm contributes four main ideas: (1) the use of a blockchain to host a pre-trained machine learning model that users can use out of the box, (2) a blockchain-based social referral system that rewards participants for spreading the search even if they aren't in the immediate vicinity, (3) the utilization of a proxy re-encryption network to enable trust-less market of lead information, (4) a two stage 'peek-reveal' pipeline for filtering spam responses.
As an effect, social mobilization, aided by computer vision tools, can increase participation in time critical searches, and filter the ensuing amount of reports and leads.
iOS loading screen https://www.youtube.com/watch?v=3f4T66sMNdQ
iOS flow https://youtu.be/3f4T66sMNdQ
Here's a video of the drone platform in action: https://www.youtube.com/watch?v=1PaJ2KyyMMY
Technical Flow
Users are socially recruited into a referral system (implemented as an Ethereum dApp), that rewards users for submitting leads, as well as referring people that submit leads. Ethreum's contract transparency, decentralized uptime, and ease of international programmatic payments make it an ideal platform for this.
Each individual participant can download a pre-trained model that allows them to augment their ability to visually recognize an object, for example a missing child. A iOS mobile client allows users to perform recognition on device using coreML, and upload when sufficiently confident. A autonomous sensor client allows users to hook in a variety of devices with video streams (such as drones) and upload any sightings.
After sighting and recording important findings, an information escrow mechanism, powered by nucypher's pyUmbral, allows participants to disclose sightings and accuracy ratings, but ensure that they are compensation for their lead before (cryptographically) revealing the full information they have.
How we built it (tech stack)
Autonomous Drone Platform => DJI Tello, Go, dlib
Edge Compute iOS client => Tensorflow, CoreML
Query Incentive Network dApp => Solidity/Ethereum/DAI
Face Recognition ML => dlib, tensorflow, IPFS
Webapp => Javascript, JQuery, Web3
Information Exchange and Escrow => IPFS, pyUmbral from nuCypher, heroku for hosting shim
Challenges we ran into
All being said, combining all the pieces takes a lot of glue!
More information is available in our github organization! https://github.com/search3
Built With
- coreml
- crowdsourcing
- dai
- dlib
- drones
- go
- javascript
- nucypher
- python
- tensorflow
Log in or sign up for Devpost to join the conversation.