Sometimes nothing matters but collecting as much information as possible about something - information that might not be indexed or exist at all. Google processed 3.5 billion searches a day in 2018 and 'googling' is the dominant verb for searching, yet there exists a dearth of tools for massive internet-assisted searches, or Human Search Engines.

A brief overview

The future of search will require expanding beyond indexing, and focusing on searching socially and algorithmically by both recruiting people and giving them the tools to help search for what they value most - whether it's a missing child or a priceless possession. The search3 paradigm contributes four main ideas: (1) the use of a blockchain to host a pre-trained machine learning model that users can use out of the box, (2) a blockchain-based social referral system that rewards participants for spreading the search even if they aren't in the immediate vicinity, (3) the utilization of a proxy re-encryption network to enable trust-less market of lead information, (4) a two stage 'peek-reveal' pipeline for filtering spam responses.

As an effect, social mobilization, aided by computer vision tools, can increase participation in time critical searches, and filter the ensuing amount of reports and leads.

iOS loading screen

iOS flow

Here's a video of the drone platform in action:

Technical Flow

Users are socially recruited into a referral system (implemented as an Ethereum dApp), that rewards users for submitting leads, as well as referring people that submit leads. Ethreum's contract transparency, decentralized uptime, and ease of international programmatic payments make it an ideal platform for this.

Each individual participant can download a pre-trained model that allows them to augment their ability to visually recognize an object, for example a missing child. A iOS mobile client allows users to perform recognition on device using coreML, and upload when sufficiently confident. A autonomous sensor client allows users to hook in a variety of devices with video streams (such as drones) and upload any sightings.

After sighting and recording important findings, an information escrow mechanism, powered by nucypher's pyUmbral, allows participants to disclose sightings and accuracy ratings, but ensure that they are compensation for their lead before (cryptographically) revealing the full information they have.

How we built it (tech stack)

Autonomous Drone Platform => DJI Tello, Go, dlib

Edge Compute iOS client => Tensorflow, CoreML

Query Incentive Network dApp => Solidity/Ethereum/DAI

Face Recognition ML => dlib, tensorflow, IPFS

Webapp => Javascript, JQuery, Web3

Information Exchange and Escrow => IPFS, pyUmbral from nuCypher, heroku for hosting shim

Challenges we ran into

All being said, combining all the pieces takes a lot of glue!

More information is available in our github organization!

Built With

Share this project: