cognicheck

A lightweight memory consistency layer for LLM-based NPCs that detects and resolves contradictions while reducing LLM costs by up to 80%

Comment

CogniCheck: Memory Contradiction Control for LLM NPCs

Research Prototype — "CogniCheck: A Lightweight Memory Consistency Layer for LLM-Based Game NPCs"

Overview

CogniCheck is a modular, production-style system that detects and resolves memory contradictions in LLM-driven game NPCs. It dramatically reduces LLM API costs (target: 60–80% reduction) while maintaining high memory consistency across multiple NPCs.

Key Features

Feature Description
🧠 Hybrid Detection Semantic embeddings + deterministic rules
⚡ Cost Control LLM called only at high confidence (≥ 0.75)
📊 Full Metrics CRR, MCS, F1, FRR, latency, LLM reduction
🔁 Multi-Seed Runs 5 seeds with mean ± std reporting
🔌 REST API FastAPI endpoint for real-time integration
🧪 Unit Tests ≥80% coverage on core modules
🔬 Ablation Study 3 variants to prove component contributions

Project Structure

cognicheck/
├── core/
│   ├── memory.py          # SQLite-backed MemoryStore
│   ├── detector.py        # Contradiction Detector (semantic + rules)
│   ├── resolver.py        # Resolution Engine (4 strategies)
│   └── llm_gateway.py     # LLM cost gate + retry + logging
├── simulation/
│   ├── event_generator.py # Scripted + probabilistic events
│   └── npc_sim.py         # Multi-NPC simulation runner
├── evaluation/
│   ├── metrics.py         # CRR, MCS, P/R/F1, FRR, latency
│   ├── benchmark.py       # Per-seed benchmark runner
│   └── ablation.py        # Ablation study runner
├── api/
│   ├── main.py            # FastAPI app
│   └── routes.py          # /process_event endpoint
├── config/
│   ├── config.yaml        # All tunable parameters
│   └── settings.py        # Pydantic config loader
├── tests/
│   ├── test_memory.py
│   ├── test_detector.py
│   └── test_resolver.py
├── data/
│   ├── simulation_events.json
│   └── ground_truth_contradictions.json
├── results/               # Auto-generated outputs
├── main_cli.py            # CLI entry point
├── run_all_seeds.py       # Run all 5 seeds + ablation
└── requirements.txt

Quick Start

1. Install Dependencies

cd cognicheck
pip install -r requirements.txt

2. Run Full Simulation (All Seeds)

python run_all_seeds.py

Or with the CLI directly:

python main_cli.py --seeds 42 123 456 --ablation

Offline mode (no LLM API needed — uses local heuristic):

python main_cli.py --no-llm --ablation

3. Run Unit Tests

pytest tests/ -v --cov=cognicheck/core --cov-report=term-missing

4. Start the REST API

uvicorn cognicheck.api.main:app --host 0.0.0.0 --port 8000

Send an event:

curl -X POST http://localhost:8000/process_event \
  -H "Content-Type: application/json" \
  -d '{
    "npc_id": "npc_1",
    "event_text": "The player betrayed me.",
    "sentiment": -0.9,
    "entities": ["player"],
    "relation_change": "trust_decreased"
  }'

Configuration (config/config.yaml)

detection:
  similarity_threshold: 0.7        # Semantic threshold for contradiction
  retrieval_min_similarity: 0.5    # Min similarity for candidate retrieval
  confidence_low: 0.4              # Below → keep both + uncertain flag
  confidence_high: 0.75            # Above → call LLM
  rule_boost: 0.3                  # Added to score when a rule fires

resolution:
  enable_llm: true
  llm_model: "local"               # "local" | "openai" | "ollama"

Evaluation Outputs

After running, results/ contains:

File Description
evaluation_report_{seed}.json Full metrics per seed
final_memory_state_{seed}.json All NPC memories after simulation
results_summary.csv Aggregate table (all seeds)
llm_calls_log.json Every LLM call with latency
cognicheck.log Full simulation log

Resolution Strategies

Strategy Description
Naive Always store, never resolve
Recency New memory replaces old (audit trail kept)
Full LLM Every event triggers an LLM call (expensive baseline)
Hybrid ✅ Low conf → keep both; mid → recency; high → LLM

LLM Backend Options

llm_model Description
local Deterministic heuristic (offline, no API key)
openai OpenAI GPT-4o-mini (OPENAI_API_KEY env var needed)
ollama Local Ollama with llama3.2

Reproducibility

All simulations use fixed random seeds (random, numpy). Scripted events in data/simulation_events.json ensure deterministic base scenarios. Probabilistic extensions use seeded RNGs.

Built With

Share this project:

Updates