DRCoordinator 1.0.0

The previous "A framework that enables Consumer to pay NodeOp only as much LINK is required to cover the gas costs incurred by the data delivery, plus some profit margin set by NodeOp" but better!

Comment 2

Direct Request Coordinator 1.0.0 (DRCoordinator)

A framework that enables dynamic LINK payments on Direct Request (Any API), syncing the price with the network gas and token conditions. It targets node operators that seek being competitive on their Direct Request operations.

Video demo slide deck

(New) Try it out with How To 01: DRCoordinator Basic Tutorial

See DRCoordinator 0.1.0 submission for the Chainlink Hackaton Spring 2022

Inspiration

The first version of DRCoordinator presented at the Spring 2022 hackaton was rushed and felt like a PoC (in fact there was lots of prototype testing!). Since then, I really wanted to revamp it with production standards and give it closure. With OCR2DR in the works the place of DRCoordinator on Direct Request production integrations is uncertain. Nevertheless, DRCoordinator 1.0.0 will help well anyone who wants to learn Web3 development on Ethereum with Solidity & Hardhat and few Chainlink products in depth (Direct Request and Price Feeds).

These are the new on-chain features (contracts):

  • Adopted fulfillData() as fulfillment method instead of fallback() (which has been removed).
  • Standardised and improved custom errors and removed unused ones.
  • Standardised and improved events.
  • Added Spec.paymentType which enables REQUEST LINK payment as a percentage (as permiryad), apart from the flat payment type already supported. A percentage REQUEST LINK payment is more beneficial from the Operator point of view and allows setting minContractPaymentLinkJuels as 0 Juels in all DRCoordinator TOML job specs (simplifying and standardising them all).
  • Added support for whitelisting consumers on-chain (authorised consumers) per Spec, as DRCoordinator TOML job specs must use the requesters field to protect themselves from spamming attacks (due to low minContractPaymentLinkJuels).
  • Added a refund mode. If SPOT LINK payment is less than REQUEST LINK payment DRCoordinator refunds Consumer the difference.
  • Added consumerMaxPayment, which allows Consumer to set a maximum LINK amount willing to pay per request.
  • Added multi Price Feed support (2-hop mode). DRCoordinator can calculate the wei units of GASTKN per unit of LINK (weiPerUnitLink) using two price feeds: GASTKN / TKN (priceFeed1) and LINK / TKN (priceFeed2). This mode allows to deploy DRCoordinator on networks where the LINK / GASTKN Price Feed is not available, e.g. Gnosis Chain, Moonriver, Moonbeam, Metis, etc.
  • Replaced the L2 Sequencer Offline Flag logic with L2 Sequencer Uptime Status Feeds to check L2 Sequencer availability on Arbitrum, Metis and Optimism.
  • Added public lock in DRCoordinator.sol as per Read-only Reentrancy. It may be useful for Consumer devs as DRCoordinator has methods like availableFunds() (which its result varies if read during requestData() and fulfillData() execution).
  • Added permiryadFactor, which allows tuning the fee percentage limits.
  • Improved interfaces and contracts inheritance.
  • Simplified DRCoordinator.cancelRequest() by loading FulfillConfig.expiration and FulfillConfig.payment.
  • Improved the Consumer libraries (contracts), e.g. DRCoordinatorClient.sol, ChainlinkExternalFulfillmentCompatible.sol.
  • Removed sha1 logic for syncing (CUD) JSON specs in DRCoordinator storage.
  • Removed minConfirmations logic after understanding that the Consumer plays no role on it (see Chainlink release v1.5.0 and Adjusting Minimum Outgoing Confirmations for high throughput jobs). Also that it is still not possible setting minConfirmations from a job pipeline variable.
  • Applied Chainlink's Solidity Style Guide with few exceptions like not grouping by visibility, removing the leading _, or using callWithExactGas (which I believe it is not needed thanks to the ethtx task).
  • Added NatSpec.
  • Upgraded to Solidity v0.8.17.

These are the new off-chain features (Hardhat repository):

  • Fixed bugs in utils, tasks, methods, etc.
  • Improved transaction overrides (from ethers.js) options.
  • Improved Web3 provider and signer management.
  • Extended the network support (e.g. Optimism / Arbitrum Goerli, Klaytn Baobab, etc.) with regards to the Chainlink framework (e.g. LinkToken, Price Feeds), contract deployment & verification, etc.
  • Improved test suite and added GitHub Actions CI.
  • Improved project folder structure.
  • Improved tasks documentation.
  • Updated dependencies.

How it works

This is a high level overview of the Direct Request Model with DRCoordinator:

DRCoordinator Flow

1. Deploying a DRCoordinator

NodeOps have to deploy and set up first a DRCoordinator:

  • Deploy, set up and verify a DRCoordinator using the drcoordinator:deploy Hardhat task.
    • NB: By default it will attempt to fetch the LINK / TKN Price Feed on the network and it will error if it is not found. In this case NodeOps will require to deploy in Multi Price Feed mode (See Price Feed Contract Addresses for choosing the right Price Feeds).
  • Amend any non-immutable config after deployment using the drcoordinator:set-config Hardhat task.
  • NodeOps can check the DRCoordiantor storage detail using the drcoordinator:detail Hardhat task.

2. Adding the job on the Chainlink node

NodeOps have to add a DRCoordinator-friendly TOML job spec (image no 1), which only requires to:

  • Set the minContractPaymentLinkJuels field to 0 Juels. Make sure to set first the node env var MINIMUM_CONTRACT_PAYMENT_LINK_JUELS to 0 as well.
  • Add the DRCoordinator address in requesters to prevent the job being spammed (due to 0 Juels payment).
  • Add an extra data encode as (bytes32 requestId, bytes data) (via ethabiencode or ethabiencode2 tasks) before encoding the data for the fulfillOracleRequest2 tx.

3. Making the job requestable

NodeOps have to:

  1. Create the Spec (see SpecLibrary.sol) of the TOML spec added above (image no 2 & 3) and upload it in the DRCoordinator storage via DRCoordinator.setSpec() (image no 4).
  • NodeOps should create the equivalent JSON Spec and upload it using the drcoordinator:import-file Hardhat task.
  1. Use DRCoordinator.addSpecAuthorizedConsumers() if on-chain whitelisting of consumers is desired.
  2. Share/communicate the Spec details (via its key) so the Consumer devs can monitor the Spec and act upon any change on it, e.g. fee, payment, etc.

4. Implementing the Consumer

Devs have to:

  • Make Consumer inherit from DRCoordinatorClient.sol (an equivalent of ChainlinkClient.sol for DRCoordinator requests). This library only builds the Chainlink.Request and then sends it to DRCoordinator (via DRCoordinator.requestData()), which is responsible for extending it and ultimately sending it to Operator.
  • Request a Spec by passing the Operator address, the maximum amount of gas willing to spend, the maximum amount of LINK willing to pay and the Chainlink.Request (which includes the Spec.specId as id and the request parameters CBOR encoded) (image no 5).

Devs can time the request with any of these strategies if gas prices are a concern:

  • Call DRCoordinator.calculateMaxPaymentAmount().
  • Call DRCoordinator.calculateSpotPaymentAmount().
  • Call DRCoordinator.getFeedData().

5. Requesting the job spec

NB: Make sure Consumer has LINK balance in DRCoordinator.

When Consumer calls DRCoordinator.requestData() DRCoordinator does (image no 5):

  1. Validates the arguments.
  2. Calculates MAX LINK payment amount, which is the amount of LINK Consumer would pay if all the callbackGasLimit was used fulfilling the request (tx gasLimit) (image no 6).
  3. Checks that the Consumer balance can afford MAX LINK payment and that Consumer is willing to pay the amount.
  4. Calculates the LINK payment amount (REQUEST LINK payment) to be hold in escrow by Operator. The payment can be either a flat amount or a percentage (permiryad) of MAX LINK payment. The paymentType and payment are set in the Spec by NodeOp.
  5. Updates Consumer balancee.
  6. Stores essential data from Consumer, Chainlink.Request and Spec in a FulfillConfig (by request ID) struct to be used upon fulfillment.
  7. Extends the Consumer Chainlink.Request and sends it to Operator (paying the REQUEST LINK amount) (image no 7), which emits the OracleRequest event (image no 8).

6. Requesting the Data Provider(s) API(s), processing the response(s) and submitting the result on-chain

NB: all these steps follow the standard Chainlink Direct Request Model.

  1. The Chainlink node subscribed to the event triggers a directrequest job run.
  2. The OracleRequest event data is decoded and the log and request parameters are processed and (9) used to request the Data Povider(s) API(s) (image no 9).
  3. The API(s) response(s) (image no 10) are processed and the result is submitted on-chain back to DRCoordinator via Operator.fulfillOracleRequest2() (image no 11 & 12).
  • NB: forwarding the response twice (i.e. Operator -> DRCoordinator -> Consumer) requires to encode the result as bytes twice (via ethabiencode or ethabiencode2)./
  • NB: the gasLimit parameter of the ethtx task has set the amount defined by Consumer when called DRCoordinator.requestData() plus GAS_AFTER_PAYMENT_CALCULATION (50_000 gas units).

7. Fulfilling the request

  1. Validates the request and its caller.
  2. Loads the request configuration (FulfillConfig) and attempts to fulfill the request by calling the Consumer callback method passing the response data (image no 13 & 14).
  3. Calculates SPOT LINK payment, which is the equivalent gas amount used fulfilling the request in LINK, minus the REQUEST LINK payment, plus the fulfillment fee (image no 15). The fee can be either a flat amount of a percentage (permiryad) of SPOT LINK payment. The feeType and fee are set in the Spec by NodeOp.
  4. Checks that the Consumer balance can afford SPOT LINK payment and that Consumer is willing to pay the amount. It is worth mentioning that DRCoordinator can refund Consumer if REQUEST LINK payment was greater than SPOT LINK payment and DRCoordinator's balance is greater or equal than SPOT payment. Tuning the Spec.payment and Spec.fee should make this particular case very rare.
  5. Updates Consumer and DRCoordinator balances.

Challenges I ran into

The current DRCoordinator design makes it work as a "forwarder", as it forwards requests and responses between the Consumer & Operator contracts. I call this design DRCoordinator-Cooperator. The first task I addressed for this hackaton was prototyping a DRCoordinator-Operator, which would have been a DRCoordinator that inherited from Operator and extended its functionality for a new kind of requests using any of these approaches:

  • Implement internal LINK balances (do not use LINK.transferAndCall()) and just emit an OracleRequest event (which is required to trigger directrequest jobs, but it does not require to be preceded by a LINK payment).
  • Use LINK.transferAndCall() and implement a new DRCOORDINATOR_REQUEST_SELECTOR.

None of them came to fruition due to the Operator.sol subclassing limitations; the key methods have private visibility. Despite either modifying Operator.sol or getting rid of it was an option, I wanted to stick to the Chainlink standards and do not increase the risk of NodeOps (Operator.sol has been audited & widely tested).

Accomplishments that I'm proud of

Having the willpower of revamping the version presented at the previous Chainlink Hackaton knowing OCR2DR is in the works. DRCoordinator 1.0.0 is a more mature product and I've achieved many of the "What's Next?" bullet points listed on the previous hackaton.

Also the multi Price Feed mode (2-hops mode), and running lots of experiments with directrequest jobs trying the new built-in core tasks, e.g. ethabiencode2.

What's next for DRCoordinator

Contract improvements:

  • Factor in the L1 fees when having to calculate MAX / SPOT LINK payment amount on L2s. The Chainlink KeeperRegistryBase1_3 does it for Arbitrum and Optimism.
  • Fuzz testing.
  • Consider extending Spec (or a related mapping) with job spec metadata, e.g. an optional IPFS CID that points to the integration docs.
  • See if there is a way to improve the Etherscan experience without sacrificing the contracts design. DRCoordinator.sol NatSpec is not well displayed (is it because of inheriting it from the interfaces?) and custom errors revert reasons are not well displayed either.

Tasks & tools improvements:

  • Add any missing task and/or tool based on users' feedback, for instance "add funds".
  • Incorporate some of the DRAFT tasks & concepts that help troubleshooting requests, e.g. decoding fulfillment transactions.
  • Improve the README.md and How To guides. For instance include examples of different DRCoordinator configs (e.g. multi price feed, L2 Sequencer Uptime), TOML/JSON specs, showcase the refund mode, etc.
  • Keep track of all the above introducing a changelog.

But also:

  • Support any NodeOp that wants to give it a try. For instance an improved ethabiencode2 task would allow NodeOps not having to list, price & manage N Get jobs (e.g. (uint256), (uint256,uint256), (uint256, uint256, uint256), ...) cause each Consumer could enforce its desired ABI per request.
  • Try to understand DRCoordinator's place in the ecosystem once OCR2DR is released.

What I learned

An even deeper dive into Chainlink Direct Request.

Built With

Share this project:

Updates