Cortex: Cognitive Perception for AI Agents

A neuroscience-inspired perception framework that filters 92% of sensor noise before your AI agent reasons — saving API calls, reducing latency, and enabling embodied intelligence.

Comment

Inspiration

AI agents today are drowning in data. A single security camera generates thousands of motion events per day — most of them routine (pets walking by, shadows shifting, repeated movement). Without filtering, agents waste 92% of their reasoning on noise.

Humans solved this problem millions of years ago. Your brain doesn't consciously process every photon hitting your retina. Instead, subcortical structures like the thalamus filter sensory input before it reaches conscious awareness. We built Cortex to give AI agents the same capability.

What it does

Cortex sits between sensors and reasoning:

Sensors/APIs → Cortex (perception filter) → Your LLM/memory layer

It applies neuroscience-inspired mechanisms to decide what matters:

  • Habituation (sensory adaptation): Repeated stimuli raise the threshold — you stop noticing the clock ticking
  • Orienting Response (Sokolov reflex): Abnormally large stimuli always fire — a sudden loud noise gets attention
  • Circadian Rhythm: Time-of-day awareness adjusts vigilance levels
  • Decision Engine: Routes events by priority to appropriate actions
  • Notification Queue: Manages background-to-agent communication

It also includes bridges for Gemini, Elasticsearch, local VLM inference (Cosmos bridge with Qwen3-VL-2B), and an MCP Server with 11 tools.

How we built it

Pure Python, zero external dependencies. 7 cognitive modules inspired by Thompson & Spencer (1966) for habituation, Sokolov (1963) for orienting response, and Borbely (1982) for circadian rhythms.

We validated against 96+ hours of live sensor data from a home security camera system (3,052 events across two cameras). The Cosmos VLM bridge enables egocentric reasoning — the agent interprets camera feeds from its own first-person perspective using a local 2B parameter model running on a Mac mini in 1.2-2.4 seconds per image.

Cortex also integrates with ReachyMini, a tabletop robot with camera, 4-mic array, and IMU — enabling embodied AI that sees, thinks, and physically responds. Perception maps to physical actions: novel stimulus triggers head tracking, repeated stimulus gets habituated, high-priority events trigger alert animations.

Challenges we ran into

  1. Cosmos-Reason2-8B was too large for our M2 8GB Mac mini (OOM at 8.1GB). We switched to Qwen3-VL-2B (1.0GB) which runs at 2.3s per image.
  2. Kitchen camera pet noise: Cats and dogs triggered constant false alerts. We solved this with strict_mode (AND condition: diff>=60 AND changed_ratio>6.68%) plus YOLO classification to distinguish humans from animals.
  3. Tuning habituation thresholds: Too aggressive filtering missed real events; too permissive defeated the purpose. We settled on a dynamic system that adapts based on time-of-day and location.

Accomplishments that we're proud of

  • 92% cognitive load reduction validated on real-world data (3,052 events → 224 conscious events)
  • 201 tests all passing across Python 3.10/3.11/3.12
  • Zero dependencies — pure stdlib Python, pip install and go
  • Local VLM inference running egocentric reasoning at 1.2-2.4s on consumer hardware
  • 11 MCP tools that give any Claude Code session real-time perception
  • Published on PyPI as cortex-agent — anyone can pip install and try it

What we learned

  • 92% of sensor events are noise. A perception layer before reasoning is not optional — it's essential for any agent working with real-world data.
  • Cognitive science provides battle-tested algorithms. Habituation, orienting response, and circadian rhythms are millions of years of evolution solving the exact same problem AI agents face today.
  • Local VLM inference is practical. A 2B parameter model on a consumer Mac provides useful egocentric reasoning in under 3 seconds.

What's next for Cortex

  • Multilingual embedding support for better cross-language memory retrieval
  • Spreading activation over co-occurrence graphs for associative recall
  • ReachyMini physical integration — connecting Cortex to a real robot body for embodied cognition demos
  • Episode chunking for automatic scene segmentation of continuous sensor streams

Built With

Share this project:

Updates