-
-
Waveform and Spectogram
-
Proof of Creation
-
Metadata Inputs
-
Annotating the full track
Inspiration
The advent of digitization affected no file format and distribution mechanism more rapidly than music. The convergence of the easily shared MP3 file format, with the networking scale of Web 1.0 and the sheer demand for music led to the onset of peer-to-peer filesharing. The industry was blindsided by upstarts like Napster that leveraged the emerging technologies of the day to deliver listeners unfettered access to each others music collections.
Peer-to-peer filesharing solution Napster redefined the way we consume music, with ramifications to this day
Though music streaming has gone some way to offer a superior user experience to raw peer-to-peer filesharing and proved that listeners will pay for this, the economics of music streaming are problematic. Market leaders such as Spotify struggle with profitability, and are the subject of frequent criticism from key stakeholders in the music community due to the rates received for every stream. Streaming has not fixed structural issues such as a music file lacking authoritative metadata or tracking whether the granular components or samples inside a song have been derived from existing works. Such features would lead to a more equitable distribution of income to creators, and enhanced opportunities for revenue generation.
With the emergence of Web 3.0 we can now address the structural issues of file format and distribution to ensure that music intellectual property is encoded into the decentralized internet ecosystem, harnessing the security of blockchain. NUSIC conceives a music file, not as a single track, but as a binary tree that protects multi-channel segments of audio down to the individual beats and bars, and a distribution mechanism that encrypts the media files in decentralized storage.
A self-referencing binary tree, one measure split into 4 stems. At the atomic level a single beat, syllable, note or bass stab is a piece of intellectual property worthy of protection. This could include all channels in a musical production.
NUSIC Layer 1 is designed to secure the creation metadata behind every part of every song, provide encrypted distribution to give creators full control over their music assets, and include the audio fingerprint of every component as part of an open database, so that sounds can be identified and tracked in the wild. Such a layer 1 is not without precedent, but it is only today with interoperable blockchains and new modes of media consumption, that the prospect becomes realizable...
What it does
Layer 1
A music creator loads the multi-channel audio files of their music as Proof of Creation to instantiate the creation-metadata to a base layer that refers to the encrypted upload of said multi-channel audio files on IPFS. Creator annotates track to identify sections, such as Intro, Verse, Chorus etc so specific parts of the composition can be identified within the dataset.
Once the metadata is instantiated to the base layer and the multichannel audio files are uploaded to IPFS, users may access the music according to the creators preferences. A creator may choose to make their music available permissionlessly, whereby anyone can access and freely use the music, or encrypt it, restricting access through an NFT license pass.
NFT License Pass
NFT license passes enable users to access the multichannel content according to the license the creator grants as outlined above. In order to facilitate access to the music data according to the creator's wishes, the NFT license pass may only be made available to whitelisted individuals, that may require KYC certification, zero knowledge proofs or other accreditation.
Metaverse
The first NUSIC Metaverse is conceived as a music festival featuring music from some of the greatest Music NFTs across 3 stages and an audio-visual installation of a permissioned NFT license pass as a three-dimensional metaversal artifact. In order to simulate the real world popularity of the musicians, each stage features a virtual audience based on the popularity of the artists behind the NFT, drawing from their Spotify monthly listeners, YouTube subscribers and TikTok followers.
To access the music content that the NFT license pass protects users must complete a fetch-quest to get whitelisted. KYC is currently available at the Dox Station kiosk, once the user has provided documentation the music can be accessed.
How we built it
Parachain
After investigating the architecture, we built a substrate node and then moved to a parachain template, testing it out on Rococo by running multiple nodes and a collator. All of the music files are stored on Web3Storage and the CryptoJS library is used to encrypt the files, the polkadot{.js} API library is used to connect to the parachain and make transactions.
Upload and Annotate
For the upload page the front-end is built in React and uses the wavesurfer-js library so that the full track file can be annotated. Once the full track has been annotated and the creator has uploaded their stems every stem is automatically organized into a binary tree data structure with the track explorer which visualizes the data structure with react-d3-tree.
ERC-3643N
KYC takes place at the Dox Station and is handled through Onfido, the wallet address gets registered with OnchainID
The NFT License Pass is permissioned and based on the ERC-3643 fungible token standard by Tokeny, the functions freeze and forcedTransfer coupled with users being whitelisted in tokenIdentityRegistry mean that this same NFT contract could be used as a security token and therefore be registered with the SEC or issued under an exemption such as Reg A, Reg A+, Reg D, Reg S or Reg CF enabling issuers to raise varying levels of capital from accredited, non-accredited and international investors alike in a compliant manner according to the specific regulation.
As NFTs are all unique tokens rather than a set of interchangeable coins, the unit256 balances in the transfer functions had to change to the specific token ID rather than just the amount of tokens:
function forcedTransfer(
address _from,
address _to,
uint256 _amount
) public override onlyAgent returns (bool) {
uint256 freeBalance = balanceOf(_from) - (frozenTokens[_from]);
if (_amount > freeBalance) {
uint256 tokensToUnfreeze = _amount - (freeBalance);
frozenTokens[_from] = frozenTokens[_from] - (tokensToUnfreeze);
emit TokensUnfrozen(_from, tokensToUnfreeze);
}
if (tokenIdentityRegistry.isVerified(_to)) {
tokenCompliance.transferred(_from, _to, _amount);
_transfer(_from, _to, _amount);
return true;
}
revert('Transfer not possible');
}
Original T-REX function
function forcedTransfer(
address _from,
address _to,
uint256 _tokenId
) public override onlyAgent returns (bool) {
bool tokenFrozen = frozenTokens[_from][_tokenId];
if (tokenFrozen) {
frozenTokens[_from][_tokenId] = false;
emit TokensUnfrozen(_from, _tokenId);
}
if (tokenIdentityRegistry.isVerified(_to)) {
tokenCompliance.transferred(_from, _to, _tokenId);
_transfer(_from, _to, _tokenId);
return true;
}
revert('Transfer not possible');
}
Function updated to refer to tokenId
In the freeze function, again rather than specifying the balance to be frozen the list of tokenIds is looped through to freeze each token one-by-one:
function freezePartialTokens(address _userAddress, uint256 _amount) public override onlyAgent {
uint256 balance = balanceOf(_userAddress);
require(balance >= frozenTokens[_userAddress] + _amount, 'Amount exceeds available balance');
frozenTokens[_userAddress] = frozenTokens[_userAddress] + (_amount);
emit TokensFrozen(_userAddress, _amount);
}
Original T-REX function
function freezePartialTokens(address _userAddress, uint256[] memory _tokenIds) public override onlyAgent {
for (uint256 i = 0; i < _tokenIds.length; i++) {
require(_exists(_tokenIds[i]), "NFTToken: operator query for nonexistent token");
frozenTokens[_userAddress][_tokenIds[i]] = true;
emit TokensFrozen(_userAddress, _tokenIds[i]);
}
Function updated to refer to tokenId
KYC verification is handled through the Onfio API and subsequently the wallet address of the verified user will be registered in IdentityRegistry using registerIdentity function. We used OnchainID's SDK to create the instance of the Identity contract, and this is then passed to the registerIdentity function of IdentityRegistry.
Festiverse
The festival environment & 3D assets were built with Blender, and are designed to convey an organic spacey aesthetic
The Moralis NFT API was used with the Unity Boilerplate to authenticate the user so they can interact in the environment with their wallet, it calls Nethereum and the Unity Video Player to stream the audio and visual content from the NFTs. The gameobjects next to the NUSIC signage react to the music in range of the specific artists area by extracting the audio source from the NFT and having the amplitude of the music affect their size.
Bringing web 2.0 metrics onchain with Chainlink
In order to bring the Spotify, YouTube and TikTok numbers onchain for 3 artists with music NFTs featured at the festival, we called the Chainlink Chartmetric external adaptor which is now set-up on a LinkPool node and currently running on Kovan.
.@RAC https://t.co/kxWIhFgcwE pic.twitter.com/V3fnSfWYRH
— LinkPool (@linkpoolio) April 28, 2022
The music hedge-nodes represent artist popularity data retrieved from the Chartmetric external adaptor on Chainlink
Challenges we ran into
As a multidisciplinary team working across different core competencies inevitably there were issues in threading the various components of the project together into a coherent whole. We couldn't resist the opportunity to start building out the Metaverse, though this meant we had to use C# to interface with the external adaptor due to the Moralis Unity engine. When the artist numbers were first ran into Unity there were too many NPCs being rendered in the crowd (e.g. Steve Aoki has ~17.3M Spotify listeners) and crashed the project, so each of the NPCs represents 100K listeners/subscribers/followers - this could be adjusted to a logarithmic scale in the future to be more favorable to smaller or up-and-coming artists.
Designing the runtime storage for the music metadata and designing the binary tree structure so that it makes musical sense, as well as indexing the transactions with a view to both a machine and human readable identifiers are important problems to solve, and what is included in this hackathon is far from perfect. Tracks which are not recorded to a click or don't maintain a metronomic tempo are much more difficult to organize into a data structure like this. Without validating beat tracking with Recurrent Neural Networks and Dynamic Bayesian Networks à la madmom the whole data annotation system is vulnerable to human error and a high standard of music authentication will be required to prevent rampant piracy!
Accomplishments that we're proud of
Before we started work on the hackathon none of us would have believed that by this time we would have a fully operational layer one demo designed to musical standards, a brand new permissioned NFT contract, and a functioning music festival Metaverse which anyone with the internet can try. We've enjoyed working with two talented new team members who we only met a month ago, everyone maintained their commitment to see the project through to delivery.
What we learned
Before identifying the Moralis Unity SDK we explored a series of different options for the Metaverse, including getting hold of some land in The Sandbox, applying to build in oncyber and looked at Decentraland as an another Metaverse to build in. However there was a surprising amount of friction to build an experience in any of these Metaverses, even with land in The Sandbox there are currently limited options for development, the typeform we filled in for oncyber received no response and without land in Decentraland the only choice we had was to build out the NUSIC Metaverse experience from scratch.
Fortunately this proved to be the right decision and is significantly expanding our options for go-to-market, where better to offer experimental new musical experiences that leverage the NUSIC Layer 1 in collaboration with artists as prospective network validators, and where else to build-up community engagement for the project than in an open music Metaverse?
What's next for NUSIC: Layer 1 for Music
Layer(s)
The next step for this blockchain is likely a private rights layer, so that confidential and commercially sensitive ownership data can remain private within the ecosystem; this may be necessary to unlock broad industry adoption. In order to facilitate interaction between the creation and rights layers Polkadot's XCM Format could be utilized. By enabling metadata to be loaded onchain at the point of creation (i.e. through DAWs such as Pro Tools or Logic Pro) and for the files to be encrypted for distribution we improve market efficiencies and ensure the maximum revenue goes to the creators. This creates a dynamic feedback loop as the same DAWs access the music assets at the consumption layer and producers draw from an ever-growing catalog of music samples. Streaming services themselves are essentially utilities and while the current dominant market players may adopt such a system, more likely is that new market entrants create improved musical user experiences that leverage the granularity of the music information stored onchain and on IPFS, delivering deeper engagement and interactivity with listeners than was possible with fully mastered tracks as the only saleable unit.
Music Information Retrieval
Including the audio fingerprint of not just a track, but the entire data structure is foundational to a next generation open music dataset, super-charging existing open source music projects such as AcoustID and Music Brainz with crypto economics and communities can potentially accelerate this development. Use cases include advancements to ContentID systems, algorithmic mix engines, AI DJs as well as tools for creativity that lower the barrier to entry around music creation.
NFT
Our permissioned ERC-3643N contract, that was built as a result of this hackathon, can be submitted as an EIP in the near future and this will be useful for the emerging market for security NFTs. We intend to run a Security Token Offering for NUSIC DAO using this contract as well as offering NFT Music Bonds to the marketplace in a regulatory compliant manner.
We had previously faced pushback against our NFT Bonds for passing the Howey Test, as they are conceived as financial instruments for musicians, rather than utilities or otherwise; this drove our enquiry into security NFTs and spearheading this new NFT contract. In the future, under the qualifying legal parameters these can finally be launched as crypto securities.
Metaverse(s)
To encourage an open music Metaverse we do not intend to charge for land, but instead work towards permissionless building and collaboration within the spirit of a music festival as the ultimate creative ecosystem. Many of the ground rules around the open Metaverse have yet to be laid, we encourage leaders, builders, creators, artists, DJs and musicians to join us on the ground floor in disrupting the space and defining together what a decentralized music Metaverse can really be.
Log in or sign up for Devpost to join the conversation.