AViCA - Adaptive Visual Cognitive Architecture

An adaptive computer vision system that transforms live visual input into structured scene understanding through perception, spatial reasoning, visual memory, and predictive cognition.

Comment

AViCA—Adaptive Visual Cognitive Architecture

Inspiration

Traditional computer vision systems often process every frame using a fixed sequence of operations. However, visual understanding becomes more challenging when targets move, disappear behind obstacles, lighting changes, or multiple objects look similar.

I wanted to investigate a different approach: Can a vision system decide what to observe next, remember what it has already seen, predict what might happen, and use uncertainty to guide its decisions?

This question inspired AViCA (Adaptive Visual Cognitive Architecture), a research-oriented computer vision system that combines perception, visual working memory, uncertainty estimation, predictive perception, spatial reasoning, and active observation.

Rather than treating each image as an isolated input, I designed AViCA to maintain state across observations and use that state to decide which visual evidence would be most useful next.

What It Does

AViCA implements an adaptive perception loop that transforms raw visual input into evidence, maintains an evolving representation of the scene, and selects subsequent observation actions.

Its main capabilities include:

  • OpenCV-based perception: Processes image pixels using computer vision operations, including grayscale conversion, Sobel gradients, contour and candidate-region analysis, and targeted region-of-interest inspection.
  • Uncertainty estimation: Uses Shannon entropy and missing-evidence signals to represent ambiguity in the current perceptual state.
  • Active perception: Evaluates candidate observation actions using expected information gain and computational cost.
  • Visual working memory: Maintains a bounded set of entities, their appearance, confidence, and temporal state, including transitions through tracking, occlusion, recovery, and forgetting.
  • Predictive perception: Projects candidate target positions across multiple future time steps and evaluates predictions against subsequent visual observations.
  • Spatial world modeling: Represents directional, topological, and relative depth relationships without claiming calibrated metric 3D reconstruction.
  • Learned visual attention: Implements a tabular Q-learning policy alongside a utility-based observation policy to investigate adaptive action selection.
  • Scientific evaluation: Includes benchmark comparisons, ablation experiments, policy evaluation, tracking metrics, and local hardware profiling tools.

I also developed an interactive interface that makes the architecture's internal behavior inspectable through visualizations of the scene, cognitive state, working memory, prediction horizon, action traces, and experimental results.

How I Built It

I built AViCA as a modular TypeScript application using React and Vite, with OpenCV.js/WebAssembly for browser-based image processing.

I organized the cognitive engine into independent modules for perception, uncertainty, memory, spatial reasoning, prediction, tracking, and active observation.

The core processing cycle is

$$ \text{Observe} \rightarrow \text{Estimate Uncertainty} \rightarrow \text {Select Action} \rightarrow \text{Gather Evidence} $$

$$ \rightarrow \text{Update Memory} \rightarrow \text{Predict} \rightarrow \text{Evaluate} $$

The system first extracts visual evidence from raw image data. It then estimates uncertainty and selects an observation action, such as inspecting a region of interest, analyzing edges, or tracking motion. The resulting evidence updates the internal scene representation and informs subsequent predictions and decisions.

1. Perception and Uncertainty

I implemented a perception pipeline that extracts visual features and candidate regions. The uncertainty engine uses Shannon entropy to characterize ambiguity:

$$ H(p)=-\sum_{i=1}^{n}p_i\log_2(p_i) $$

Here, (p_i) represents a probability assigned to a candidate interpretation. The resulting uncertainty signal helps the architecture reason about whether additional evidence may be valuable.

2. Active Observation

I designed AViCA to compare observation actions according to their expected information gain and computational cost. Its utility-based policy is complemented by a learned Q-learning policy that estimates the value of actions in discretized perceptual states.

My objective was to investigate how selective observation can balance information gathering with computation, rather than assuming every observation is equally valuable.

3. Memory and Prediction

I developed a visual working memory that maintains a bounded collection of entities and updates their confidence over time. Its temporal decay model is:

$$ c_t=c_0e^{-\lambda\Delta t} $$

where (c_0) is the initial confidence, (\lambda) is the decay rate, and (\Delta t) is the elapsed time.

The predictive perception module projects candidate trajectories across four future time steps, from (t+1) through (t+4). These projections provide hypotheses for subsequent observations, including situations where a target becomes temporarily occluded.

4. Evaluation and Experimentation

I developed a research interface for comparing passive vision, attention-based vision, AViCA's greedy policy, and AViCA's learned policy. I also included ablation experiments to investigate the contribution of individual architectural components.

The repository includes automated tests for perception, temporal tracking, predictive perception, spatial modeling, learned policies, and evaluation infrastructure.

I also included AWS deployment configurations and benchmarking scripts. Cloud benchmark results remain pending until actual AWS execution is performed.

Challenges I Faced

Integrating Perception with Decision-Making

A conventional detection pipeline produces visual outputs, but an adaptive architecture must also decide what to inspect next. I had to coordinate perception, uncertainty estimation, action selection, and evidence integration within a consistent processing loop.

Maintaining Identity Over Time

Objects may move, become occluded, or reappear in different positions. I addressed this challenge through visual working memory, appearance information, optical-flow estimation, tracking, and predictive hypotheses.

Making Uncertainty Useful

Representing uncertainty is not enough on its own; it must influence the next action. I incorporated uncertainty and expected information gain into the observation-selection process while accounting for the computational cost of additional operations.

Evaluating a Complex Architecture

An integrated system can contain many interacting components, making it difficult to determine which mechanisms contribute to its behavior. I included benchmark comparisons, policy evaluation, and ablation experiments to provide a framework for investigating those contributions.

I also made a distinction between measured results and planned hardware evaluations. My AWS deployment configurations provide a basis for reproducible infrastructure, but they do not establish completed cloud benchmarks.

What I Learned

Building AViCA taught me that visual intelligence involves more than recognizing objects. Temporal continuity, uncertainty, memory, prediction, and action selection are interconnected parts of the perception process.

I also learned the value of modular architecture and transparent evaluation. Keeping components separate makes them easier to test, compare, and improve, while visualizing internal state makes complex behavior easier to understand.

Developing this project independently also challenged me to think across multiple engineering layers, from image-processing algorithms and state representations to reinforcement learning, interface design, testing, and deployment.

Most importantly, AViCA helped me explore how active perception can turn computer vision from a fixed sequence of image-processing operations into a stateful, decision-driven process.

What's Next

My next steps include evaluating the architecture on broader real-world datasets, improving tracking under difficult visual conditions, exploring richer learned observation policies, and validating the deployment configurations on actual AWS hardware.

I also plan to strengthen quantitative comparisons between passive and active observation strategies using reproducible experimental conditions and clearly documented measurements.

My vision for AViCA is simple: build computer vision systems that do not just see but decide what to look at, remember what matters, anticipate what comes next, and adapt when the available evidence is uncertain.

Built With

Share this project:

Updates

Submission history