STSS_Speed

What it does

STSS_Speed is a reproducible offline workflow for running large-scale self-targeting spacer searches with the original STSS software. The goal is simple: spend less time waiting on downloads, fixing broken runs, or wondering whether an analysis is complete. The workflow takes local FNA files and matching GBFF annotations, validates them, builds a reusable per-contig cache, and then runs STSS in isolated workers with checkpoints. It keeps the analysis offline during the STSS stage by blocking Entrez, CDD, PHASTER, and other HTTP calls. This makes large jobs more stable and easier to audit. On a 6,000-sample run, the STSS stage processed about 2.306 genomes per second, or 138.334 genomes per minute. Compared with an earlier offline pilot, the observed throughput was about 9.11× higher. This is a practical cross-run comparison, not a universal benchmark claim.

How we built it

STSS_Speed started from a practical idea: large-scale bioinformatics work should not waste time repeating avoidable failures. I designed the workflow around a few rules:

• Keep FNA inputs read-only and record them in a deterministic manifest.
• Download and validate GBFF annotations before starting STSS.
• Split validated GBFF files into a shared per-contig GenBank cache.
• Require complete cache coverage before computation begins.
• Run STSS in separate workers with CPU affinity, so tools such as Clustal Omega do not compete for every CPU thread.
• Break long jobs into checkpoints, so a failed batch can be investigated and retried without discarding completed work.
• Recheck sample counts, worker status, result rows, cache status, and offline-network counters during aggregation. STSS_Speed does not redistribute custom HMMs, private classifier rules, genome files, GBFF files, caches, or research results. It bootstraps a pinned original STSS version locally instead.

Challenges we ran into

The difficult part was not only making the workflow faster. It was making it reliable when real-world conditions were messy. Network downloads could fail because of connection resets, TLS problems, timeouts, or incomplete GBFF packages. Some annotations needed to be checked carefully before they could be trusted as cache inputs. WSL mounts could disappear during long jobs, and unrestricted parallel execution could make performance worse because multiple tools competed for the same CPU resources. Another challenge was distinguishing a real zero-hit result from a failed analysis. A run with no self-targeting hits is still meaningful, but only if the input, cache, worker exit status, and offline-network checks all pass. Finally, reproducibility had to be balanced with data protection. The workflow is public, but private models, classifier updates, genome metadata, and result-level information are not.

Accomplishments that we're proud of

We turned a single-run STSS workflow into a checkpointed, recoverable, and auditable pipeline for large-scale analysis. A few things matter most:

• A 6,000-sample STSS run completed at 2.306 genomes per second.
• Observed throughput reached 138.334 genomes per minute.
• The workflow achieved a 9.11× observed throughput improvement over an earlier offline pilot.
• Each run can verify cache completeness, worker success, result consistency, and blocked network access.
• Failed stages are preserved instead of overwritten, making troubleshooting possible without losing completed work.
• The public repository remains usable without exposing private HMMs, updated classifiers, genome information, or result files.

What we learned

Speed is not just about adding more workers. For large bioinformatics workflows, the biggest gains often come from reducing avoidable work: preventing duplicate downloads, validating files before computation, reusing trustworthy caches, isolating failures, and assigning CPU resources carefully. We also learned that reproducibility needs to be built into the workflow itself. If a result cannot be traced back to a manifest, a validated annotation cache, successful workers, and consistent outputs, then finishing quickly is not enough. Most importantly, a robust workflow should make the correct path easier than the risky path.

What's next for STSS_Speed

The next step is to make STSS_Speed easier for more researchers to use without weakening its validation rules. Planned directions include:

• More standardized benchmark reports across different dataset sizes and hardware.
• Easier setup through improved environment checks and optional containers.
• Clearer progress dashboards for long-running checkpoint jobs.
• Better summaries for failed downloads and cache-validation problems.
• Additional tests for scaling worker counts while avoiding CPU oversubscription.
• Continued support for original STSS assets while keeping private models and updated classifier rules outside the public repository.

Built With

• bioinformatics
• biology
• biopython
• blast+
• computational
• genbank
• stss