DST scaffolder

We used AST based semantic analysis to improve code scaffolding of starport by adding utils to generate code reliably. We demonstrated the usage in our code and removed most placeholders.

Comment

DST Scaffolder

Introduction

We made a package for scaffolding starport artifacts with semantic checks.

Inspiration

Starport has scaffold commands to create and modify artifacts like chain, module oracle etc. Right now the code uses placeholder strings to patch code, making it dangerous and prone to issues from lint tools, formatters, etc

What it does

Our package astutils help in patching code reliably anywhere inside the code and add snippets of code.

Placeholders and line number based utils are problematic for developers who may be left in a limbo once the code gets into a bad state. To protect developers from such issues, we can target the issue either at

  1. lex level
  2. parse level
  3. or even down further(basically doing compile checks which are done by compiler, to know ahead, of issues like type checks and name collisions of variables)

We chose to solve the code generation issue at parse level(for now). Running vet-like analysis is possible on each change made to the AST.

Powerful code generators should check all three levels for errors. Simple patching of code is probably fine for IDEs where the generated code is short and local for developers to fix any issues. But when too many files are being touched by a single command, even genny cannot figure out all the issues it created.

Our plan

Our plan was to:

  1. Parse the target file to AST
  2. Parse the snippet code to be injected
  3. Add the snippet code to the AST
  4. Run analyzers that vet runs(See go/analysis) after each file's ast is updated.
  5. Fail and send back errors on any failure

We could only achieve the goals partially. As opposed to simple line number based code gen, our goals proved to be too ambitious for the time period.

How we built it

  1. We build an AST for the code file
  2. We used a package called dst instead of ast to preserve comments and their order as decorators, which is particularly hard with simple ast based formatting.
  3. We walk through the AST and find the location to inject code at. As most code snippets in the repo are currently text, we did not use the full power of semantic checks possible.
  4. We also parse the text to be injected into code and verify that its valid.
  5. We add the code to be injected, into the AST.
  6. We finally send genny the content to write back to the file.

Challenges we ran into

  1. go/ast does not preserve comments position on files. So we had to use dst
  2. Once we use dst, we had run into issues around node manipulation(dst has to finally use go/ast for writing to nodes and it gave rise to issues around duplicate nodes). This prevented us from writing cleaner code Visitors to walk through the AST and write code. After discovering the clone method in dst, we now think the code can be refactored better based on Walk method instead. That would give us a neat chain of methods we can share between code gens that shares path in AST traversal.
  3. Parser tools do not let us parse text into arbitrary node types into AST. So we had to resort to some clever ways to get the vector code into AST and inject it in place.

Accomplishments that we're proud of

  1. We got neat and clean code generated with the utils. Once the basic helpers were in place, the code is easy to use anywhere.
  2. We could atleast prove that a semantic analysis based code gen is possible without losing expressiveness(albeit our current code is still lacking in this respect)

What we learned

We got to revisit compile concepts from school. We learned about go's ecosystem for analysis tools and quirks of each of them.

What's next for DST scaffolder

  1. We need to refactor code, use dst walker instead of a list of node mappers. This will give us a chain of node visitors with beautiful code(considering go's type system)
  2. We can extract out utils as a separate project for code injection in general with better semantic checks.
  3. We need to add similar functionality for proto files.
  4. We need better error messaging for issues like an existing token already having same name etc.
  5. We can add vet-like analyzer passes on the AST we build for bullet proof code checks(Currently I believe this happens with genny runner dryrun). This allows to suggest developer any specific changes they have to do for a successful scaffold. We don't need to lose sleep over what issues code gen can bring.

Built With

Share this project:

Updates