• 

• 

  PlaneFuse Live

• 

• 

• 

• 

PlaneFuse

PlaneFuse makes on-device mobile camera AI faster by letting a compatible vision model work directly from native camera data instead of first building a full RGB image.

On mobile Apple platforms, camera video commonly arrives as NV12, not RGB. A typical vision pipeline converts that data to RGB, normalizes it, builds a full tensor, and only then starts the learned part of the model.

PlaneFuse asks:

Why have mobile AI build a full RGB image just to immediately transform it again?

The result is a Metal + Core ML pipeline, a live camera app, and a reproducible benchmark showing 11.8% lower matched p50 latency than the strongest matched materialized-RGB baseline.

The short version

Conventional path

Camera NV12 → RGB → normalize → full RGB tensor → first learned convolution → MobileNetV2

PlaneFuse

Camera NV12 → native Y + UV planes → transformed source-reuse stem → same MobileNetV2 tail

The model is not retrained.

PlaneFuse changes how the model is reached, not what the model learned.

What I optimized

Speed: lower matched input-to-result latency.

Representation efficiency: remove the full Float32 RGB intermediate.

Arm execution: run the optimized stem in Metal on Apple Silicon arm64.

Developer experience: make the work easy to verify, reproduce, and integrate.

Camera capture, preprocessing, Metal execution, and Core ML inference all stay on-device.

The core idea

Color conversion and normalization are fixed operations, while the first convolution is linear before activation.

PlaneFuse composes those operations ahead of time so the first learned layer can work from values derived directly from the camera's Y and UV planes.

In plain English:

Instead of building RGB and then running the first convolution, PlaneFuse folds compatible RGB conversion and normalization into that first learned step.

The rest of MobileNetV2 stays unchanged.

The three final paths I compared

B2 is the strongest matched conventional baseline. It takes the same NV12 camera input, builds a full normalized Float32 RGB representation, then runs MobileNetV2's first learned stem and the unchanged tail.

C1 is the first PlaneFuse path. It removes the full RGB intermediate and computes the first learned activation directly from the native Y and UV camera planes.

C1-SR is the final PlaneFuse path. It keeps C1's native-plane approach, then adds source reuse by staging small Y and UV tiles in Metal threadgroup memory and reusing them across multiple output channels.

B2 = conventional RGB path

C1 = remove RGB

C1-SR = remove RGB + restore efficient source reuse

The part that surprised me

Removing RGB was only half the problem.

C1 already removed the RGB intermediate and was faster than the matched RGB baseline. Then I tried a more aggressive direct camera-space fusion path.

It was slower.

The missing piece was reuse. The representation I removed had also been helping later work reuse the same source data efficiently.

That led to C1-SR. It stages a small tile of native Y and UV data in Metal threadgroup memory and reuses it while computing multiple output channels.

The final path keeps both benefits: no full RGB intermediate, while preserving useful source reuse.

Final benchmark result

The final workload is Apple MobileNetV2 ImageNet classification running locally on Apple Silicon.

B2: 1.737875 ms

C1: 1.633458 ms

C1-SR: 1.532583 ms

C1-SR measured 11.8% lower matched p50 latency than B2.

It was also 6.2% lower than C1.

The paired B2 minus C1-SR median 95% confidence interval for the absolute latency difference was [0.180250, 0.198792] ms.

What changed at the representation boundary

The B2 baseline creates a full normalized Float32 RGB representation before the learned stem.

For the 224 × 224 × 3 input, that resource has a 606,208-byte Metal allocation.

PlaneFuse C1-SR uses 0 bytes for that full RGB intermediate.

That is not a total-memory claim. Both paths still use model weights, source textures, activations, and Core ML resources.

B2: NV12 → full normalized RGB → learned stem

C1-SR: NV12 → transformed native-plane stem

Both use the same pretrained MobileNetV2 tail. No retraining.

Making sure faster did not mean different

For the final C1-SR vs C1 comparison across the fixed 64-input validation corpus:

Top-1 agreement: 100%

Top-5 set agreement: 100%

Top-5 ranking agreement: 100%

Maximum activation error: 5.960464e-6

The earlier B2/C1 comparison retained two real-image top-5 differences, while top-1 stayed unchanged under the declared quality contract.

How I benchmarked it

I wanted the result to survive more than one lucky run.

The final protocol used Release builds on Apple Silicon arm64, the same 64-input corpus, the same pretrained tail, and the same activation boundary.

It used 5 independent benchmark processes, with 20 warmup triples and 240 measured triples per process. All six B2 / C1 / C1-SR execution orders were balanced, and the paired analysis used a deterministic 10,000-replicate block bootstrap.

The timing boundary is matched from the same NV12/model entry point through the same model tail.

Clean-clone reproduction

I also reproduced the project from a fresh public GitHub clone.

B2: 1.5899 ms

C1: 1.5026 ms

C1-SR: 1.4355 ms

The ordering stayed C1-SR < C1 < B2.

The fresh PlaneFuse result was approximately 9.7% lower p50 than B2, with quality and parity checks passing again.

The reviewed 11.8% experiment remains the headline result.

Source reuse scaling

I built a separate stem-only experiment to check whether C1-SR really behaved like a reuse optimization.

At narrow channel widths, staging overhead made C1-SR slower. As more channels shared the same staged source tile, the paths moved toward parity.

At the full 48-channel MobileNetV2 stem, C1-SR was 4.97% faster than C1.

That helped confirm why the final optimization worked instead of treating the kernel as a black box.

What did not work

Several ideas did not survive end-to-end testing.

Float16 failed the quality gate.

A Metal 4 route was not suitable for the stable release path.

A polyphase compiler was mathematically correct but produced no stable end-to-end win.

The more aggressive direct camera-space fusion was slower because it removed useful reuse.

That last failure directly led to C1-SR.

PlaneFuse Live

PlaneFuse Live is a real macOS camera application running locally on Apple Silicon.

It shows the live camera feed, MobileNetV2 predictions, B2 and C1-SR paths, parity state, runtime metrics, and the reviewed benchmark result.

It is an ImageNet classifier, not an object detector.

After the initial model setup, inference runs locally without requiring a network connection.

Built around Arm client execution

PlaneFuse uses Swift, Metal, Core ML, AVFoundation, Core Video, Apple Silicon arm64, and MobileNetV2.

The real camera path is:

AVCaptureVideoDataOutput → CVPixelBuffer NV12 → CVMetalTextureCache → Y + UV Metal textures → C1-SR stem → persistent activation → unchanged Core ML tail → local classification

The optimization happens directly on the Arm-powered client.

Developer experience

I also wanted another developer to be able to understand and reuse the work.

The repository includes a compile-checked runtime, a camera integration example, model compatibility documentation, reproducibility tooling, benchmark evidence, and a working live app.

The workflow is:

inspect → compatibility check → compile → verify → benchmark → integrate

The current implementation fully verifies MobileNetV2. I do not claim arbitrary-model support.

Run it yourself

PlaneFuse currently targets an Apple Silicon Mac.

1. Clone

git clone https://github.com/Barap1/PlaneFuse.git

cd PlaneFuse

2. Check the environment

./pf doctor

3. Prepare MobileNetV2

./pf setup mobilenetv2

4. Run the live app

./pf live --app

5. Run a quick verification

./pf reproduce quick

./pf evidence --check

6. Run the full benchmark

./pf reproduce final

Fresh reproduction output is written under artifacts/reproduction/<timestamp>/ and does not overwrite the reviewed evidence.

What I learned

The biggest lesson from PlaneFuse is that AI optimization does not have to start inside the neural network.

Sometimes the interesting work is at the boundary around the model.

I also learned not to trust an optimization just because the diagram looks simpler. The aggressive fusion path removed more work and ran slower because it removed useful reuse.

That failure changed the direction of the project and led to the final design.

Why PlaneFuse matters

PlaneFuse improves a pretrained model without retraining it, replacing it, or moving inference to a server.

For the verified MobileNetV2 workload, it removes a temporary full-RGB representation, keeps the same pretrained tail, preserves the declared output quality, measures 11.8% lower matched p50 latency, reproduces from a clean public clone, and ships as a working local camera app with reusable tooling.

The same representation-boundary idea could be explored for compatible camera models used in accessibility, robotics, industrial inspection, AR, smart cameras, and other latency-sensitive local vision applications.

Sometimes the fastest representation is the one you never create.

What's next

The current project deliberately stays narrow:

one model, one camera contract, one Arm client platform, one measured optimization.

The next step is to test the compatibility workflow on additional vision architectures and Arm client devices, and to add direct energy measurements.

Source code: https://github.com/Barap1/PlaneFuse

Built With

• applesilicon
• avfoundation
• coreml
• corevideo
• macos
• metal
• mobilenetv2
• python
• swift
• xcode