Abstract

The capacity to learn incrementally from an online stream of data is an envied trait of human learners, as deep neural networks typically suffer from catastrophic forgetting and stability-plasticity dilemma. Several works have previously explored incremental few-shot learning, a task with greater challenges due to data constraint, mostly in classification setting with mild success. In this work, we study the underrepresented task of generative incremental few-shot learning. To effectively handle the inherent challenges of incremental learning and few-shot learning, we propose a novel framework named ConPro that leverages the two-player nature of GANs. Specifically, we design a conservative generator that preserves past knowledge in parameter and compute efficient manner, and a progressive discriminator that learns to reason semantic distances between past and present task samples, minimizing overfitting with few data points and pursuing good forward transfer. We present experiments to validate the effectiveness of ConPro.

Incremental Few-shot Image Synthesis

There are two grave challenges in incremental few-shot training of a generative model. The fist is catastrophic forgetting that generally occurs in continual learning, and the second is limited sample diversity caused by strong overfitting. In ths work, we propose a systematic role allocation between the two players of GAN to overcome both at once.

Conservative Generator

We observe that in continual training of a GAN, the responsibility to retain past task knowledge only applies to the generator. Hence, we design Adaptive Factorized Modulation (AFM) that efficiently expands generator parameters upon encountering a new task. This way, generator can perfectly maintain its past generative capacity, which is in turn used to train a progressive discriminator.

Progressive Discriminator

As discriminator is fundamentally free from the threat of forgetting, we mainly encourage it to combat overfitting by incorporating distance learning. Specifically, we leverage the conservative generator as the generative replay, and additionally train our discriminator with supervised contrastive objective (SupCon) to prevent unnatural memorization and extreme gradient signals. Note that our SupCon loss relies on the generator's ability to produce past task samples of good quality. As our discriminator is trained to evaluate semantic distances between samples of different classes, we further employ this quality to initialize the modulation parameters of the generator for an incoming task with the most relevant past task weights.

Qualitative Results