SkillPulse

Zero-dependency static analyzer, MCP 2025-11-25 contract validator, auto-fixer, and visual topology dashboard that scores and heals AI Agent Skills in under 200ms.

Comment

Inspiration

What it does

How we built it

Challenges we ran into

Accomplishments that we're proud of

Inspiration

While installing and running the Devpost Learn Skill Pack (challengepost/learn-ai-basics) for this very hackathon, we became fascinated by how modular SKILL.md files and Model Context Protocol (MCP 2025-11-25) tool servers work together to guide coding agents. However, we also noticed a silent reliability pain point across the agentic ecosystem: broken cross-boundary contracts.

When a developer writes a SKILL.md prompt instructing an agent to invoke a tool (e.g., delete_production_db), but that tool is missing or renamed in mcp_tools.json—a Ghost Tool—the agent silently hallucinates or crashes at runtime. Similarly, unreferenced Orphan Tools and bloated markdown files quietly burn thousands of tokens of context window budget on every prompt turn. We wanted to build eslint + pytest for the AI agent stack.

What it does

SkillPulse is a unified, zero-dependency static analyzer, MCP contract validator, deterministic auto-fixer, and visual HTML diagnostics dashboard that audits AI agent workspaces in under 200 milliseconds:

  • Agent Skill Static Linter (SKILL001–SKILL006): Validates YAML frontmatter delimiter integrity, naming regex rules, mandatory description attributes, prompt trigger cues, broken relative markdown links on disk, and BPE token footprint estimates ((\approx 1.3\text{ tokens/word})).

  • MCP 2025-11-25 Contract Validator (MCP001–MCP005): Validates JSON Schema definitions (inputSchema, properties, required) and cross-references every tool call in SKILL.md against mcp_tools.json to detect:

    • Ghost Tools (MCP003, (-20\text{ pts})): Tools referenced in skill prompts but missing from MCP manifests.
    • Orphan Tools (MCP004, (-5\text{ pts})): Tools declared in MCP manifests but never used by any skill.
    • Unimplemented Schema Stubs (MCP005, (-5\text{ pts})): Auto-generated placeholder schemas that satisfy contract topology but still require backend implementation.

  • Deterministic Ecosystem Health Score (0–100, Grade A–F):

[ \text{Score} = \max(0, 100 - (20 \times \text{Errors}) - (5 \times \text{Warnings})) ]

  • Deterministic Contract Auto-Fixer (skillpulse fix): Automatically repairs missing frontmatter descriptions, injects compliant MCP 2025-11-25 object schema skeletons, and stubs missing Ghost Tools—honestly classifying generated placeholders as MCP005: STUB / UNIMPLEMENTED ((-5\text{ pts})) so a broken workspace jumps from 25% (Grade F) to 80% (Grade B) without faking backend verification.

  • Self-Contained Obsidian HTML Dashboard & CI Exports: Generates a zero-dependency offline dark-mode report (skillpulse-report.html) featuring an interactive Skill ↔ MCP Contract Topology Map, a Model Context Window Budget Meter (8k Skill Budget vs. GPT-4o 128k / Claude 200k / Gemini 1M), client-side severity filtering, 1-click Copy Fix buttons, and --format json / --format markdown exports for CI/CD pull requests.

How we built it

We started in an empty directory and installed the Devpost Learn Skill Pack (npx skills add challengepost/learn-ai-basics --all -y). Running 1-start initiated a flipped interaction where our coding agent interviewed us before writing a single line of application code:

  1. Scope & Architecture (scope.md, prd.md, spec.md): The skill interview helped us cut sprawling ideas (like spinning up live headless LLM servers) and lock in a razor-sharp, deterministic proof of concept built strictly with the Python 3.10+ standard library (pathlib, json, re, argparse, dataclasses, html, shutil).

  2. Modular Implementation: Guided by spec.md, we built strongly typed dataclasses with full JSON serialization/rehydration (models.py), a custom zero-dependency YAML parser and skill scanner (linter.py), the MCP 2025-11-25 schema and cross-reference engine (mcp_tester.py), the weighted health scorer (scorer.py), the atomic auto-fixer (fixer.py), the single-file HTML dashboard generator (reporter.py), and the unified CLI (cli.py).

  3. Verification: We built reference healthy_workspace (100% Grade A) and broken_workspace (25% Grade F) fixtures alongside 45 automated pytest unit and integration tests that execute in 0.35s.

Challenges we ran into

  1. Zero-Dependency YAML & Schema Parsing: To ensure SkillPulse runs instantly in any local or CI environment without pip dependency conflicts or C-compilers, we engineered SimpleYAMLParser and our MCP JSON Schema validator from scratch using only Python standard library modules.

  2. Scoped Tool Extraction Without False Positives: Extracting tool names from markdown risked flagging regular inline code blocks or sibling skill names as Ghost Tools. We scoped extraction specifically to tools: / mcp_tools: frontmatter lists and # MCP Tools sections, adding regression tests to guarantee zero false positives.

  3. Honest Auto-Fix Classification (MCP005): When skillpulse fix generated a valid MCP schema stub for a missing Ghost Tool (delete_production_db), our dashboard initially labeled the link as a ✓ Verified Contract. During our acceptance review, we realized this could mislead developers into thinking backend code existed. We introduced Rule MCP005 ("x_skillpulse_stub": true), which downgrades the fatal (-20\text{ pt}) Ghost Tool error into an explicit (-5\text{ pt}) ⚠ Schema Stub (Unimplemented) warning.

  4. Offline file:// Clipboard Security: Inline JavaScript string interpolation in onclick handlers broke when remediation snippets contained nested JSON quotes, and browsers restrict navigator.clipboard on local file:// URLs. We refactored the dashboard to use safe DOM payload lookups and a zero-dependency clipboard fallback with toast notifications.

Accomplishments that we're proud of

  • Sub-200ms Execution & 45 Passing Tests: The entire 45-test suite runs in 0.35s with zero external runtime dependencies.
  • Visual Contract Topology Map: Seeing Ghost Calls light up in pulsing red, Orphan Tools in amber, Schema Stubs in warning badges, and Verified Contracts in emerald green makes debugging agent workspaces intuitive.
  • 100% Spec Fulfillment: Every single requirement in scope.md, prd.md, and spec.md—plus JSON result rehydration (--from-json) and CI Markdown PR summaries—is fully implemented and tested.

What we learned

Letting the AI interview us first via the Devpost Learn Skill Pack completely changed how we build with coding agents. Instead of prompting the agent piecemeal and watching it forget context ten prompts later, having scope.md, prd.md, and spec.md anchored in the repository gave the coding agent persistent architectural memory across every implementation and testing step.

What's next for SkillPulse

  • GitHub Action & GitLab CI Templates: Packaging skillpulse scan --format markdown as a 1-line CI step that automatically posts Contract Topology summaries on Merge Requests.
  • IDE Diagnostics Extension: Surfacing SKILL001–006 and MCP001–005 squiggly-line diagnostics live inside VS Code and Cursor as developers edit SKILL.md files. ## What we learned

What's next for SkillPulse

Share this project:

Updates

Submission history