Doing preprocessing like "zero mean unit variance" normalization and augmentations like random rotations of images on the CPU takes a considerable amount of time and thus slows down the training of the model on the GPU.
What it does
Instead of doing this on the CPU both preprocessing and augmentations can be offloaded to the GPU by using pytorch layers that do the same thing, work on the GPU (+DataParallel) and are way faster!
How I built it
Define a structure that can take a finite amount of preprocessing and augmentation tasks. When defining the NN model, this structure can then be used as first part of the model but with
no_grad=True for the non-model layers.
What's next for [GPU] Preprocessing & Augmentations as Part of the Model
Draft a first sketch of the classes required to implement this and define a set of preprocessing´s and augmentations frequently used, which later can be implemented.