Learning from Many Tasks in Meta-learning

6 Jan 2023 | 10.00am (Singapore Time)

In many machine learning applications, only a limited number of training samples are available. A successful approach to alleviate this problem is through meta-learning, which tries to extract meta-knowledge from similar historical tasks. Evidently, the larger the number of tasks to learn from, the more meta-knowledge could be learned. However, popular meta-learning algorithms like MAML only learn a globally-shared meta-model. This could be problematic when the task environment is complex, and a single meta-model is not sufficient to capture the diversity of meta-knowledge. Moreover, with a large number of tasks, accessing all task gradients in each training iteration may not be feasible. The sampling of tasks in each iteration also increases variance in the stochastic gradient, resulting in slow convergence. 

In this talk, Prof James Kwok from Hong Kong University of Science and Technology will propose to address these problems by structuring the task model parameters into multiple subspaces, where each subspace represents one type of meta-knowledge. Moreover, he will also introduce a scalable solver with theoretical optimality guarantees based on the improvement function. Variance reduction is also incorporated into meta-learning to achieve fast convergence. Prof Kwok will conclude the talk with experiments on various meta-learning tasks, demonstrating its effectiveness over state-of-the-art algorithms.