IpfsML ( Decentralized WEB3 DataHub )

Inspiration

Problem

1. Data scientist lack on getting different web3 datasets based on there required problem statements.
2. Uploading and retrieving machine learning or deep learning models they build was really hard and model management quiet difficult.
3. Need to go different data source to grab the required datasets and still hard to search needed data.
4. Data and model personalised storage are hard.

What it does

Solutions

1. WEB3 DATA HUB is a common solution, using that people can easily pull required datasets (easily access via PIP packages)
2. Easy one line cmd to store and pull the models are data as required.
3. Easy one line search cmd to get any data as per required
4. Personalised encrypted data and model storage.

How we built it

IPFS

1. NFT.storage and Web3.storage are used to store the data
2. Multiple cron jobs that fetch data daily from various platforms
3. Data cleaning and data validation jobs that clean each datasets before push.
4. Support encryption if needs.
5. personalised support to direct upload to nft storage or web3 storage via function calls.

Challenges we ran into

Getting datasets and models are very hard. keep improving pipelines and model updates.

What's next for IpfsML ( Decentralized WEB3 DataHub )

1. SDK (js and go)
2. Pipeline improvements for tons of data handling

Built With

• ipfs
• machine-learning
• pip
• python