We’ve enjoyed working with state-of-the-art tech for the past few years. While exploring, we came upon the AI art space, around the same time we got interested in the Internet Computer. Always wanting to work towards more tech equality by providing access to the latest AI advances to everyone, we realized the IC provided the perfect platform to allow for more democratic access to these algorithms by solving some logistical and infrastructure issues. We decided to explore combining text-to-image generators with NFTs which can be hosted by the IC, thus coming up with the idea of this NFT generator. At the same time, we make this functionality accessible to everyone via a simple UI, which we host on the Internet Computer.

What it does

It takes the user's text input, relays it to a state-of-the-art text-to-image generator and updates the canister with the generated image such that it shows up as an NFT in the user's wallet.

How we built it

The UI is hosted on the IC as an asset canister and calls the backend canister (an NFT store) when a user provides a text input as a prompt to generate an image. The backend canister stores the data and keeps a queue of generation jobs to be done. This queue is polled by an AWS lambda function (this can be replaced once canisters can make http request to external services) which retrieves the generation jobs to be done and starts the text-to-image generator (VQGAN-CLIP, hosted on AWS SageMaker, sadly IC machine learning is too expensive for now) with the corresponding user input. After the generation, another lambda calls the backend canister and updates the generation job with the image such that the NFT is complete. The UI allows logging in with Stoic Wallet such that the requested NFT shows up there once completed.

Challenges we ran into

  • Canisters being able to make http requests to e.g. AWS to trigger the image generator would be nice but the polling-based approach works for now
  • Integration with stoic wallet (no documentation, had to search obscure sources till we found a single code repo from an existing NFT collection)
  • Testing and debugging the canisters deployed to the IC network is not straightforward (testing on the local network is easy but limited for this project as it cannot be integrated with the AWS services)

Accomplishments that we're proud of

  • Everyone can generate image-based NFTs for themselves based on simple text inputs they provide
  • End-to-end pipeline for user-provided text input to generated image to NFT on the IC
  • Complete software system mostly hosted on the IC (uses AWS for machine learning)

What we learned

  • further experience with Motoko and building on the IC
  • own NFT collection canister hosted on the IC
  • working with the text-to-image algorithm and deploying it
  • integrating different modules hosted on Web3 (IC) and Web2 (AWS) infrastructure
  • offering Stoic Wallet as a login option + NFT integration

What's next for Infekted

  • Proper UI design
  • More UI functionality (e.g. display my NFTs there, potentially display all generated NFTs, explore generated NFTs by category or search)
  • fine-tune generation algorithm & make generated images high-resolution
  • look into hosting generation algorithm on IC to make the pipeline completely decentralized
  • integrate payments to make the project sustainable
  • potentially add tokenomics (e.g. make certain words or terms being used frequently more expensive for the user to provide as input)
  • adapt API and functionality such that it can be used by any other canister/service (e.g. for a game wanting to integrate custom-generated NFTs)

Built With

Share this project: