PatchMix: Evolving Image Compositions for Feature Representation Learning

Evolving Image Compositions for
Feature Representation Learning

In this work, we propose PatchMix, a data augmentation method that creates new samples by composing patches from pairs of images in a grid-like pattern.

We also explore ways to find better image interpolation strategies, and propose a guiding strategy to discover optimal grid-like patterns and image pairing jointly via evolutionary search , which we call Guided PatchMix.

Our method outperforms a base model on CIFAR-10 (+1.91), CIFAR-100 (+5.31), Tiny Imagenet (+3.52), and ImageNet (+1.16), and also shows performance gains over previous approaches when evaluating the Transfer Learning capacity and Robustness of a model trained with Guided PatchMix configuration discoveries.

Training Workflow:

Step 1:
Train Random PatchMix (Θ)

We train Random PatchMix to define our fitness criteria f_T by optimizing L_T over a dataset of images and their
corresponding labels.
Step 2:
Sample Random Population

Our population is initialized with individuals A_i that are sampled from random pairing between classes and random binary values in each mask M_a.
Step 3:
Search for Configurations

Use genetic search to find the set of masks M_i,j and category pairs (c_i, c_j) that correspond to discovered class combi-
nations by using f_T as the fitness criteria.
Step 4:
Eureka! - Configurations Found

We use the best set of masks M_i,j discovered in Step 3 to create informative augmented training samples based on the class combinations (c_i, c_j).
Step 5:
Training Guided PatchMix

We train a final prediction network f_O using the original training set, random pairs as described in Step 1, and the discovered configurations of Step 4.

Genetic Search Findings

The figure below shows the evolution of class-pairs selections
on CIFAR-100 over 250 generations.

As some class combinations get discarded systematically in
the search process, some interesting choices emerge.

For instance, the model initially selects the pairs (plain, seal),
(chimpanzee, mushroom), but those are discarded after 50 generations
and more informative combinations are selected. After 100 generations,
the model has discovered many of the class combinations that it will use,
such as (chimpanzee, raccoon), (road, tractor), and (seal, shark).

PatchMix Model

@InProceedings{PatchMix_2021_BMVC,
title = {Evolving Image Compositions for Feature Representation Learning},
author = {Paola Cascante-Bonilla and Arshdeep Sekhon and Yanjun Qi and Vicente Ordonez},
booktitle = {British Machine Vision Conference (BMVC)},
month = {November},
year= {2021} }