Locksmith

About the Project

Inspiration

Modern package managers make it easy to install code that a team has not fully reviewed. A dependency update may introduce lifecycle scripts, unexpected transitive packages, network access, file-system access, or other risky behavior. Yet most security tools answer a generic question: “Is this package suspicious?”

I wanted to answer a more practical question:

Is this exact dependency state safe for this repository and its review policy?

That idea inspired Locksmith—a dependency safety reviewer that puts package changes “on trial” before they enter a project’s lockfile.

How I Built It

Locksmith is a Next.js and TypeScript application powered by Qwen through Alibaba Cloud Model Studio. Users can submit a public GitHub repository for review or use the guarded npm installation flow locally.

The system retrieves dependency files and computes a deterministic dependency-state fingerprint. It then collects real package evidence from npm and PyPI, including metadata, manifests, lifecycle scripts, entrypoints, tarballs, and selected source files.

Six Qwen agents examine the evidence:

Baseline Agent identifies what changed and what evidence is available.
Manifest Agent reviews package configuration and lifecycle behavior.
Static Agent searches source files for risky patterns.
Behavior Agent infers possible install and runtime behavior.
Skeptic Agent challenges unsupported or context-free findings.
Judge Agent resolves the evidence into an Allow, Review, or Block decision.

The final report includes cited evidence, uncertainty, findings, and remediation guidance. For local npm projects, Locksmith can review a candidate lockfile before allowing installation to proceed.

The application is deployed on Alibaba Cloud ECS, with Qwen handling the agent panel and PostgreSQL provisioned through ApsaraDB RDS for durable persistence.

What I Learned

I learned that multi-agent systems are most useful when each agent has a clearly bounded responsibility. Simply asking multiple agents for opinions does not automatically improve reliability. The process becomes more valuable when agents inspect different evidence, challenge each other, and pass structured findings to a final decision-maker.

I also learned that confidence is not evidence. A suspicious-looking pattern may be completely normal for a package’s purpose, so Locksmith distinguishes between direct observations and inferred behavior. This makes uncertainty visible instead of hiding it behind a single risk score.

Challenges

One major challenge was retrieving enough real package evidence without making reviews too slow or expensive. We had to limit inspection coverage, select representative source files, and handle npm and PyPI packages through different artifact formats.

Another challenge was preventing hallucinated security findings. I designed prompts that require concrete references to package names, files, scripts, or observed metadata. The Skeptic Agent was added specifically to question claims that were unsupported or taken out of context.

I also had to ensure that previous approvals could not be mistaken for approval of a changed dependency state. Locksmith therefore fingerprints dependency files and treats every changed state as a new review.

Finally, true behavior analysis is difficult without executing packages in a sandbox. The current system labels behavior findings as inferred unless they are directly supported by retrieved evidence. A future version can add isolated execution and deeper runtime observation.

Locksmith is my attempt to make dependency approval more evidence-based, explainable, and safer for everyday development teams.