Kanoon

Overall description of our project.

The moment that inspired it

Every time I've been in a car weaving through Indian city traffic, I've watched the driver guess: what's behind the truck in front? Is that scooter about to cut in from the blind side of the auto-rickshaw? Autonomous vehicles face the same problem, except worse — a single camera and LiDAR can only see what isn't occluded. Humans cope by making eye contact and honking. Cars need a different channel.

V2V Perception is my attempt at that channel: let cars share a compressed view of what they see, so each vehicle can "see through" the one in front of it and react to hazards that are still hidden from its own sensors.

What it does

The project is a three-stage pipeline that takes raw camera frames and turns them into actionable, bandwidth-efficient safety alerts that can be transmitted between nearby vehicles.

Part 1 — Segment & Store. A MobileNetV3-based DeepLabV3 model semantically segments each frame (road, vehicle, pedestrian, etc.) and writes the binary mask + metadata to a local SQLite store keyed by frame ID and timestamp.
Part 2 — XOR Delta Encoding. Instead of transmitting full masks between vehicles, I compute the pixel-wise XOR between consecutive masks to isolate only what changed — a newly visible pedestrian, a car that just pulled into the lane — and map those changes to a 3×3 spatial grid relative to the ego vehicle. This is what makes the bandwidth story work.
Part 3 — Autonomous Decision. A lightweight ~48K-parameter CNN takes the zone report and outputs an urgency-scored driving decision (e.g., pedestrian < 15m → CRITICAL). When no training data is available, a rule-based fallback keeps the system safe instead of silent.

The core idea, in one equation

Given two consecutive segmentation masks $ M_{t-1} $ and $ M_t $, the transmitted delta is:

$$ D_t = M_t \oplus M_{t-1} $$

where $ \oplus $ is the bitwise XOR. Because consecutive urban-driving frames are highly correlated, $ D_t $ is sparse — most of its bits are zero. Run-length encoding this sparse delta lets one vehicle tell another what changed in its field of view at a tiny fraction of the raw mask size. The receiving vehicle can reconstruct a useful perception of occluded space with:

$$ \hat{M}t = M{t-1} \oplus D_t $$

That XOR-delta step is the "impact" part of HACK4IMPACT for me — cooperative perception is only useful at the city scale if the radio link doesn't melt.

How I built it

The pipeline is deliberately modular — each stage runs independently so I could iterate on segmentation without retraining the decision model, and vice versa.

Stage 1 uses pretrained DeepLabV3 with a MobileNetV3 backbone so it runs on edge hardware. Masks are stored as binary blobs in SQLite (frame_id, timestamp, H, W, mask, class_labels) so the rest of the pipeline treats storage like a queue.
Stage 2 computes XOR deltas and projects them into a 3×3 ego-centric grid. Each grid cell reports which object classes changed and by how much. That projection is what lets Stage 3 reason in "zones" instead of pixels.
Stage 3 is a small custom CNN with ~48K parameters that takes the zone report and classifies the correct driving response, with an urgency score on top. A rule-based fallback kicks in whenever confidence is low or training data is missing, which kept the safety behaviour predictable during hackathon-week development when the training set was incomplete.
The whole thing is structured around Cityscapes-style directory conventions so I can swap datasets without refactoring.

What I learned

Why compression is a perception problem, not just a networking problem. Pushing raw masks is infeasible; thinking in deltas changed the entire architecture.
How to keep a deep-learning system honest. The rule-based fallback forced me to write explicit safety invariants, which ended up being more useful than the learned model in edge cases.
That modularity pays for itself fast. Splitting into three independently runnable scripts meant I could debug the XOR stage without re-running segmentation on every iteration.
Practical PyTorch deployment on a small parameter budget, SQLite as a poor-man's message bus, and the difference between a demo that "sort of works" and one that degrades gracefully.

Challenges I ran into

Bandwidth vs. fidelity. Too aggressive a delta threshold and the receiving vehicle misses a real change; too lenient and you're back to sending whole masks. Tuning this on real-looking frames took most of a day.
Class-conditional urgency. A pedestrian at 15m is critical; a car at 15m in the same lane as you is normal traffic. Encoding that without a giant rules file was harder than expected; I ended up parameterising thresholds per class.
Training data gaps. Cityscapes doesn't label the exact scenarios I cared about (occlusion-revealed hazards). The rule-based fallback was originally a hack and became a first-class safety feature.
Keeping the decision CNN tiny. Every time I added capacity to fix an edge case, I lost the edge-deployment story. Capping the model at ~48K parameters forced better feature engineering at the zone-report stage.

What's next

Real vehicle-to-vehicle transport (UDP multicast / DSRC / C-V2X sidelink) instead of local SQLite hand-off.
A CARLA-based simulation harness so two ego vehicles can actually share deltas in a virtual city.
Adversarial robustness — what happens when a malicious vehicle broadcasts a fake "clear road" delta?