Open Driving Monitor

dNFT platform for physical goods that changes in real time according to Sensor and AI data.

Comment 1

Inspiration:

Approximately a month or so ago we saw in chainlinks official X.com a post where there was an idea for dynamic NFTs that could serve as a way to valuate property or physical goods such as sneakers, homes or cars that employed IoT sensors that fed data through Chainlink oracles and functions to those NFTs.

Solution:

We built a platform powered by Chainlink and AI models that is able to continuously monitor the environment of vehicles and with this information generate a dNFT sales token on the Polygon network.

In addition to a hardware system that is responsible for running the AI models and obtaining data from the sensors.

Materials and software:

https://github.com/altaga/Open-Driving-Monitor-Chainlink/blob/main/README.md#materials

Connection Diagram:

This general connection diagram shows how through a camera we can obtain images of the driver or those of the streets to later obtain relevant data on the driver's alertness, his state of mind and the objects around the car. All fed back by our internal screen and our online web map.

  • On-Chain:

    • Chainlink Functions: The new chainlink functions are used to be able to make API Calls to our Off-Chain API and update the metadata of each of the dNFTs when required.
    • Polygon: This network is used as it is one of the networks compatible with Chainlink Functions and its low gas fees.
    • IPFS: We use the NFT.Storage IPFS services to store the metadata of the NFTs and images.

  • Off-Chain:

    • AWS DynamoDB: In this database we update the values of the sensors and the data obtained from the AI services continuously.
    • AWS IoT: We use this service to communicate securely between the sensors and the AI module to the cloud.
    • AWS API Gateway: We use this service as the main API to be called by Chainlink Functions.

  • Sensors:

    • NXP Rapid IoT Prototyping: This sensor kit allows us to measure many aspects of the car and with that data we can generate a better report on its conditions and its value over time.

  • AI Module:

    • Eye State Detection: Through preprocessing in OpenCV haarcascades, OpenCV DNN and a frozen graph inference model (Tensor Flow), we obtain the driver's state of attention and drowsiness.. Details
    • Emotions Identification: Through preprocessing in OpenCV haarcascades, OpenCV DNN and a frozen graph inference model (Tensor Flow), we obtain the driver's mood.Details
    • YoloV3: Using OpenCV DNN and the famous network YoloV3 from Darknet We carry out the identification of vehicles and pedestrians in the blind spot of the car. Details

  • Online Platforms:

    • Open Driving Monitor: Using a board enabled with OpenCV DNN, we created a system that can run the 3 AI models and also provide vehicle GPS information at all times. The selected board will be shown later. Details
    • Open Driving Navigator: Using the NextJS, Open Layers and Vercel framework, we create a map that allows us to display the cars that are on our platform in real time and their states.. Details

Chainlink Functions:

For our project it is essential that the data of each of the cars registered as dNFTs is updated, however performing this update is a complicated technical problem. For this, thanks to the Chainlink functions and its new Decentralized Oracle Network (DON) they allow computing API calls and providing the smart contracts with that information through a subscription model on the ChainLink platform.

Contract:

You can check the whole process on our Github:

https://github.com/altaga/Open-Driving-Monitor-Chainlink/blob/main/README.md#contract

Chainlink Functions Platform:

Now that all the functions have been implemented correctly, all that remains is to deploy the contract. But something else was missing, we had to make our contract able to consume the function on Chainlink, so we had to create a subscription at https://functions.chain.link.

Although it is a very simple process, we found the following message when trying to fund our subscription.

This was a surprise for us because we already had LINK in our wallet, however in the wallet section we found the problem.

So all that was necessary to do was perform the LINK swap on the page https://pegswap.chain.link/.

Once the swap was made it was possible to fund our subscription.

And finally we just had to add our contract as a consumer of the subscription.

Once this was done and we called the sendRequest function from our contract, we could see how chainlink received the request and provided us with the data in the contract a few seconds later.

AI Models and Sensors:

The purpose of all these models and sensors in the project is to provide added value by being able to measure the car in real time, making it possible to make a better assessment of the continuous use of the vehicle.

  • Emotion: Thanks to obtaining the driver's emotion it is possible for us to predict careless handling of the vehicle.
  • Drowsiness: With this result it is possible for us to assess if the car had a crash, what was the reason for it.
  • Environment Sensors: With this data it is possible for us to know under what conditions the driver exposes the vehicle, as well as measurements of accelerations suffered by the vehicle, as well as automatic crash detection.

More on this section of our Github:

https://github.com/altaga/Open-Driving-Monitor-Chainlink/blob/main/README.md#ai-models-and-sensors

The Final Product:

Complete System:

In-car system:

Commentary and final words:

We think we accomplished what we wanted to do wwhich was bringing the dNFT idea we saw, into a real dNFT platform with real sensors and the blockchain backend with Chainlink functions that we wanted to try out. Of course this is just a functional PoC, but it serves as a blueprint of how these kind of solutions would be built. We still need to improve the front end of it to make it more comprehensive, but all the backend functionality works better than expected even with the AI aspects of it. Hopefully you liked the project, thanks for reading.

Built With

Share this project:

Updates