CircuitScope: See Inside AI Models

Open-source tool that reveals how AI models make decisions, making black-box models transparent and debuggable

Comment

Inspiration

We were frustrated by a fundamental problem in AI: nobody really knows how neural networks make decisions.

When ChatGPT hallucinates or refuses a legitimate request, even its creators can't definitively explain why. This "black box" problem isn't just academic—it's blocking AI deployment in healthcare, finance, legal systems, and other critical sectors where explainability is mandatory. Imagine a doctor unable to explain why an AI recommended a specific diagnosis, or a bank unable to justify an AI's loan rejection. This opacity creates a trust crisis.

Anthropic's groundbreaking "Attribution Graphs" paper offered a solution—a method to trace exactly how AI models make decisions. But there was a massive catch: their approach requires days of GPU computation, billions of training tokens, and complex infrastructure that puts it out of reach for most researchers and organizations.

We thought: What if we could democratize AI interpretability? Why not create a lightweight baseline that implements all the core techniques but runs on a small transformer in under 15 minutes on CPU?

What it does

CircuitScope acts as an "X-ray for AI" that reveals the hidden decision-making process inside neural networks. It transforms opaque models into glass boxes where every decision can be traced and understood.

Core Capabilities:

🔍 Decision Path Tracing
Maps the complete journey from input tokens through network features to final outputs. Like following a GPS route through the model's "thought process."

🧩 Feature Discovery
Identifies interpretable computational units the model develops internally—"acronym detectors," "number magnitude sensors," "syntax validators"—the building blocks of AI reasoning.

📊 Attribution Graph Building
Creates visual influence maps with quantified scores showing exactly which features caused which outputs. Every connection has a measurable impact score.

🐛 Model Debugging
When a model hallucinates or fails, CircuitScope pinpoints the exact features that misfired. No more guessing why the AI went wrong.

⚡ Efficient Execution
Achieves in minutes on CPU what traditionally requires days on GPU clusters by condensing interpretability on a baseline model, making mechanistic interpretability accessible to everyone running any hardware.

How we built it

Core Technologies:

  • PyTorch - Deep learning framework for model implementation
  • NumPy - Numerical computations and array operations
  • Python 3.11+ - Primary programming language

☁️ Cloud Platform

  • Snowflake Integration: Full deployment on Snowflake notebooks with CPU compute
  • Stage Architecture: Direct connection to The Pile dataset via Snowflake Stages

ML/AI Components:

  • Custom JumpReLU - Sparse activation function with learnable thresholds
  • Cross-Layer Transcoders - Multi-layer feature extraction architecture
  • Attribution Graphs - Causal graph construction algorithms

Data & Training:

  • The Pile Dataset (EleutherAI) - Training data via Hugging Face
  • AdamW Optimizer - With custom learning rate scaling
  • Token-level shuffling - For efficient training on limited compute

Key Algorithms Implemented:

  • Frozen LayerNorm - For linear attribution paths
  • Backward Jacobian - Computing gradients through model
  • Neumann Series - For indirect influence computation
  • Two-step Graph Pruning - Node and edge pruning algorithms
  • TWERA - Time-Weighted Expected Residual Attribution
  • Virtual Weights - Cross-layer feature communication

Optimization Techniques:

  • CPU-optimized batching - For memory-efficient execution
  • Sparse matrix operations - For graph computations
  • Stop-gradient computation - For maintaining linearity

Challenges we ran into

1. 🧮 Mathematical Density

The original paper's mathematics is extraordinarily complex. Understanding frozen denominators, virtual weights, and proper Jacobian attribution required deep expertise in:

  • Transformer internals and attention mechanisms
  • Differentiation theory through deep networks
  • Linear algebra for graph computations
  • Numerical stability in gradient flows

2. ❄️ The Freezing Complexity

We discovered that neural networks normalize data at multiple stages (like standardizing test scores). If ANY normalization changes during analysis, it breaks the causal chain and makes decisions untraceable. We had to freeze values at EVERY stage—missing even one destroyed attribution accuracy.

3. 💻 Computational Optimization

The original method requires GPU-days and billions of tokens. We achieved CPU-minutes through:

  • Token-level instead of document-level shuffling
  • Simplified attention freezing while preserving mathematical validity
  • Reduced feature dimensions maintaining interpretability
  • Stop-gradient operations for linear attribution paths
  • Bandwidth-controlled gradient flow for stability

4. 🎯 Numerical Stability vs Mathematical Correctness

The gap between "mathematically correct" and "numerically stable" proved massive. We spent weeks debugging epsilon-protected divisions, normalized reconstruction targets, and adaptive learning rate scaling to prevent gradient explosion while maintaining attribution accuracy.

Accomplishments that we're proud of

We implemented every critical component from the paper: frozen denominators, virtual weights, indirect influence, two-step pruning, and supernode clustering. Our system successfully runs on a snowflake notebook, with The Pile dataset connected via Stage, and Catalog and Schema built.

What we learned

The difference between "mathematically correct" and "numerically stable" is huge in deep learning Attribution is fundamentally about counterfactuals: "what would change if this feature wasn't active?"

What's next for CircuitScope: See Inside AI Models

  • Scale to other open source models where layers are also open sourced: (GPT-2(124M-1.5B), GPT-Neo (125M-2.7B), GPT-J-6B, Pythia (70M-12B), OPT (125M-66B), BLOOM (560M-176B), T5 (60M-11B), BERT/DistilBERT, StableLM (3B-7B), Cerebras-GPT (111M-13B), LLaMA 2 (7B-70B), Mistral-7B, Mixtral-8x7B, Phi-2 (2.7B), Phi-3 (3.8B-14B), Qwen (0.5B-72B), Falcon (7B-40B), MPT (7B-30B), FLAN-T5, CodeGen (350M-16B), GALACTICA (125M-30B), OpenLLaMA (3B-13B))

📈 Model Scaling Roadmap

Phase 1 - Small Models (Q1 2025)

  • GPT-2 (124M), DistilBERT (66M), Phi-2 (2.7B)
  • Focus on educational demonstrations

Phase 2 - Medium Models (Q2 2025)

  • GPT-J (6B), LLaMA 2 (7B), Mistral-7B
  • Target research applications
  • Build dataset of "circuit patterns" documenting common computational motifs
  • Create educational materials showing how different prompts activate different circuits

Built With

Share this project:

Updates

Submission history