Starport Plugin Service

Inspiration

I was inspired to take place in the Starport challenge, and specifically in the Plugin System Challenge, because I had used Starport before and was interested in expanding its functionality. The opportunity to expand on this tool which had helped me "scaffold" my journey into blockchain was too attractive to pass!

What it does

My Plugin Service is able to list, download, build, extract, and inject configured plugins into the running Starport process. It supports both command and hook plugins. Command plugins can add custom commands under a specified parent command. Hooks can add pre/post run behavior to a specified parent command.

How we built it

I broke the process into the steps the plugin would go through.

First, the user would have to download the plugins. This posed the first question: where can we put them? I looked into where Starport stores its local configuration. I saw that in the .starport directory, individual files of each known chain were stored, under the local-chains directory. I decided to put plugins in a plugins directory under the configured plugin ID.
As a second step, the user would want to build the plugins. In correspondence with the go-plugin docs, building plugins meant compiling them to a plain binary. I decided to store these in the output directory under the configured plugin ID.
Next, the user would want to use them alongside their project. To properly add their functionality to the Starport binary, an extraction process would have to occur. In this process, which is run before every suitable command, the plugin listens on RPC, and gives the proper information regarding the data of the plugin. The data is stored in an application-friendly data structure which allowed future processes to easily access the functionalities of the plugin. Specifically, the service uses a pattern which allows the implemented function to be called after the extraction client is killed. This involved creating a new client inside the running function with configuration from the initial client. This made the injection process much easier.
To inject the plugins, the service uses functions provided by the cobra package, which made a breeze to mount hooks and add a custom command.

Challenges we ran into

Implementing the wrong tech at first

Initially, I set off on the goal of making a plugin system that would use the native plugin module of Go. There were some signs of progress, such as the completion of the download and build phase. Specifically, it was able to run plugins, but had extreme limitations when it came to dependency versioning. Articles such as Plugins In Go by Eli Bendersky further enforced the drawbacks I was seeing when testing this solution.

I had seen other potential solutions, such as go-plugin by Hashicorp but had not fully understood it the first time around. At a roadblock, I decided to take a look in the Discord, and sure enough, others had the same issues, and İlker made the suggestion to look into go-plugin again. With no other choice, I decided to look at it again, this time fully understanding what solutions it provided to the obvious limitations of native Go plugins. This was the first step in fixing the problem that I had assumed would present a problem.

Thinking through how to properly execute functions later

After extracting the data and functions from a plugin, it's information would have to be stored somewhere, to use later. First of all, it would need a way to get custom named plugins. This required a Mapper type to be implemented in the plugin, which gets the names of the custom plugins. This would be killed, and a new plugin, with knowledge of these custom plugins, starts. It then extracts all information about each of these plugins, including the information required to run them later. This required a lot of thinking about design, specifically because of the introduction of another map to the go-plugin map of plugins seemed over the top.

Accomplishments that we're proud of

I am extremely proud of:

Diligence when researching go-plugin, a project that revolved around a communication protocol I had little experience with.
Critical thinking when debugging unexpected behaviors
Breaking down plugin process into understandable components

What we learned

Through the challenges I faced in this project, I have learned more about RPC's and how to use them in Go. It has given me more of an idea of how exactly they play a part in scaled systems, and especially distributed systems. RPC's are widely used in the blockchain for efficient information retrieval, and practically applying a subset of the very widely used communication protocol.

What's next for Starport Plugin Service

I hope to get this feature implemented in the next Starport release, so many can use it to expand to their individual needs with its modular nature. I plan to internalize feedback from the community to continue making improvements to the system. Future expansions that include a marketplace for plugins, that can be sold, could be possible. Expansions to this system could be possible, enabling a developer community around Starport, and ultimately aiding developers of all experience in their Starport workflow.