Kwil DB Administrator

Enabling anyone to easily deploy a decentralised database network with permanent backup and recovery made possible the Arweave network.

Comment 2

Inspiration

I've been working on a project called Embrace Community for around a year now. It is centred on empowering Creators to setup their own sovereign communities and to take ownership of their creative endeavours. One of the issues that has often come up is the use of a reliable and decentralised DB provider. I have worked with Ceramic and Polybase but neither fit my needs well.

With the introduction of Kwil DB I have a solution that meets my development requirements e.g. checking on-chain permissions before allowing write & read access of database content / being able to easy decentralise the data between a number of nodes.

This hackathon is an exploration of the tools I would need to use a decentralised DB provider like Kwil in a production app, and the ways extensions / tools could make development easier for people to build web3 applications.

What it does & How we built it

The Kwil Admin UI is written in Next.Js & communicates with the local Kwil API deployed within the Docker network. It is a DB Admin tool to enable management of DB schemas on the Kwil Db Server.

The DB Sync tool enables multiple servers to keep in sync with the same DB. When the DB Sync extension is initialised it communicates with the Sync Server to restore any actions that exist on Arweave for that DB ID. It does this using the Bundlr GraphQL API and the server executes any found actions on the local DB. This enables actions to be re-played when a server has been taken down or when the schema is re-deployed on another server.

Once the initial actions have been restored, the Sync Server will continue to listen for transactions on the GQL API. This enables another Provider to run on a different server and submit actions to the same DB. By running 2 or more Providers with this service, specific databases can remain synced across many nodes. If all Providers were to be destroyed, there is still the option to fully restore the DB, given the permanent storage of actions on Arweave.

The DB Backup extension enables the Admin to run a full schema and data backup of the database to Arweave. The schema is built by connecting to the Kwil server and then the we use the schema info to get all the data for each table in the DB schema.

The helper extension provides helper methods to the actions to simplify important functions such as creating unique Ids and dates. Additional methods can also be added to extend the functions available, but for now only 2 exist.

Challenges we ran into

The Bundlr GraphQL service does not have a WebSocket endpoint which means I have to poll the server every X seconds to check for any database updates. In an ideal world I would connect to a WS endpoint and receive updates. To get around this I used node-cron so I have a workaround for now :)

I also had a few issues with understanding ports in Docker. There are services that are accessed from outside the Docker network as well as inside there were a few times were this challenged me.

It isn't possible to write to a Kwil DB table unless you use an action. As I was syncing actions that included data relevant to the original address caller I had to be inventive with how to sync data which was a challenge.

My wish was to develop an IDE inside of the Admin UI but there were complications getting this setup so late in the hackathon. For example getting a custom language model (Kuneiform) to work with Monaco VSC Editor.

Accomplishments that we're proud of

I've wanted to integrate a DB with Arweave for sometime, making backups of data to ensure that there is a decentralised copy even if the server itself is centralised. With the DB Sync Service I am really happy that I can run my own decentralised DB network without any reliance on any particular company / centralised authority.

The Kwil Admin UI was a last minute addition to the hackathon that I decided to make in the final few days as I had managed to get the syncing service working through Arweave. I think it makes for a really smooth developer experience and I'm really excited to continue building DB schema's using this tool.

What we learned

I've never used gRPC before or Proto3, however after seeing how Kwil extension work I wanted to explore this. I am also happy to write an app in the new version of Next (13) which has shipped quite a few changes. It's also been a while since I've used Docker so I've been happy to tap into the recent changes that Docker have made.

What's next for Kwil DB Administrator

There are many possibilities after this and I have many ideas of how to take the project to the next stages. These include:

Integrate IDE within Admin UI.

Access Control for Admin UI.

Encrypting data saved to Arweave.

Smart Contract Verification of Providers / Staking.

Visual table editor functionality.

Create Automated Test Suite.

Documentation / Tutorial Videos.

I'm hoping to get support with being able to focus on building out these features. I would like to make tools so that any developer can very quickly start creating Web 3 applications using a decentralised DB.

Built With

Share this project:

Updates

Submission history