Inspiration

Hydra is inspired by TheGraph -- an Ethereum-focused project allowing anyone to deploy a query node exposing historical smart contract data. DApp developers no longer need to build a data pipeline ingesting and indexing historical smart contract data, focusing on the core features of the product.

Hydra solves a lot of common problems dApp developers face when trying to interact directly with a Substrate chain via RPC calls:

  • Slow fetching
  • Poor support or absence of filtering, pagination and search capabilities
  • Running costly archival nodes.

What it does

Hydra is a query node for Substrate-based blockchains. A query node ingests historical events and extrinsics and provides rich, domain-specific, and highly customizable access to the underlying data, far beyond the scope of direct RPC calls.

For example, expired Kusama Treasury spending proposals are pruned from the state of the Kusama blockchain, so querying, say, one-year-old proposals becomes problematic. Indeed, one has to track the evolution of the state by sequentially applying the Treasury events and extrinsics in each historical block.

That's where Hydra gets you covered. Define your data model and the Hydra indexer will get it in sync with the chain. On top of that, you get a batteries-included GraphQL server with comprehensive filtering, pagination, and even full-text search capabilities.

The whole set-up requires just a declarative data model definition in the form of a GraphQL schema file, and a set of "mappings", which tell the indexer how the historical events and extrinsics should be handled. The rest is done by Hydra, which outputs deployable Docker images for the indexer and the GraphQL server. A minimal, but fully functional GraphQL API for historical Kusama proposals can be built and deployed in just a few minutes.

To get a feeling of how a deployed query node looks like, go to the playground and e.g. query rejected proposals with bond value greater than 800000000000000:

query {
proposals(where: { bond_gt: "10000", value_gt: "800000000000000", status_eq: REJECTED }, limit: 5) {
    proposer
    version
    value
    status
    beneficiary
    bond
  }
}

How we built it

The project initially started out of the need to better index and expose Joystream on-chain data. For Hakusama, we've reworked it into a general-purpose query node tool codenamed Hydra.

We designed a dialect of the GraphQL schema language to support the rich type system of Substrate modules written in Rust. The schema file is parsed and used to generate the database schemas and the source code for the Substrate indexer and the GraphQL server.

The source code generation is a mesh-up of the Mustache templating combined with heavy use of the Warthog library, an opinionated data-driven tool for building GraphQL APIs.

The Hydra indexer is a Substrate event ingesting pipeline with pluggable event handlers ("mappings"). The data is indexed and stored in a Postgres database, which is then used by the GraphQL server to resolve API requests.

Challenges I ran into

One of the biggest challenges, still being addressed, is a great variety of Substrate chains we aim to support. Even for a single chain Runtime upgrades and the rich system of modules make it challenging to properly extract and index the extrinsic data in a uniform way.

Another challenge was to keep the workflow of building a query node with Hydra reasonably streamlined, yet make the output as customizable as it gets. For example, we had to find a balance between type safety and expressiveness of the GraphQL schema definition dialect we introduced. We decided to reduce the initial setup and configuration overheads by introducing a scaffold command to the tool. The scaffolder generates a minimal fully functional project, which then can be iteratively customized.

Accomplishments that I'm proud of

  • Expressive GraphQL schema dialect for defining the data model
  • Hacking with Substrate API
  • Mustache templates hacking

What I learned

  • New tools for building GraphQL APIs and schemas
  • A whole zoo of Kusama extrinsics
  • Trade-offs between UX, the supported features and reasonable timelines

What's next for Hydra

  • Add reference mappings for common Substrate modules: balances, staking, to name a few
  • Significantly improve dev UX by enforcing type safety and further decoupling indexing and the GraphQL API
  • Significantly improve performance and synchronization times
  • Introduce GraphQL authentication and limit quotas

Built With

+ 18 more
Share this project:

Updates