Working with Beocat often requires repeated and complicated steps that require hardcoded shells scripts. These shells scripts are not very generic and are difficult to reuse or modify for different projects. The pipeline project intends to fix that by making easy-to-use GUIs that build pipelines out of reusable blocks.
What it does
The pipeline works in three layers: The first layers is a program block config file that defines how to run a specific tool, such as running hisat2 to take in DNA sequence files and output a single aligned scaffold file. The program blocks can be chained together so that outputs from one can be passed to another block's input. After building a chain in the GUI, a pipeline.JSON file can be exported then compiled with the third layer-- the compiling takes a pipeline.JSON and creates shell scripts to run the programs and pass data between them, automatically. The GUIs make the pipeline easy to use, but no performance is sacrificed. The shells script generation and program blocks can be run in parallel and on different threads, with all of the same flexibility that Beocat offers.
Challenges we ran into
There were a lot of tedious aspects of tkinter's dynamic GUI system. Compiling config blocks to shell scripts posed some challenges in regards to the flow of the overall pipeline, as shell script is a more difficult language to debug, especially on Beocat. Dynamically writing code that is able to synchronize parallel jobs was more difficult than we expected.
Accomplishments that we're proud of
We wrote a compiler that took JSON config files and turned them into run-able shell script pipelines. We made an intuitive GUI that a newcomer to Beocat would be able to swiftly pick up.
What we learned
Shell script has a lot of quirks and nuances that were sometimes diffucult to overcome, and so we have become better at the language of shell script
What's next for Kitten Mittens: A Beocat Pipeline Compiler
We will next flesh out the GUIs and make it even more intuitive, add more program blocks, and polish the shell script compiler to ensure that it is bug free in all scenarios.