[CFAR Outstanding PhD Student Seminar Series]Reducing the CO2 Emission of Training Deep-learning Models: Towards Efficient Large-Scale Dataset Distillation and Pruning by Wang Kai
20 Apr 2023 | 4.00pm (Singapore Time)
Ultra-large-scale datasets could help deep learning (DL) achieve many remarkable results in computer vision (CV) and natural language processing (NLP) areas. However, training on such datasets is costly and results in heavy CO2 emissions. Dataset distillation and pruning may reduce training costs by generating (selecting) small but informative datasets with little information loss while utilising fewer samples and achieving comparable results as the original dataset.
In this talk, Wang Kai from the National University of Singapore (NUS) will introduce three recent works: (1) Designing efficient matching strategies to reduce the iterations of previous works, where the team proposed a novel matching strategy named Dataset Distillation by REpresentAtive Matching (DREAM) which only selects representative original images for matching. DREAM could also be plugged easily into popular dataset distillation frameworks and may reduce matching iterations by 10 times without performance drop. He will then explain his second work on (2) Using the generative model as information container, where they introduced a novel distillation scheme to Distill information of large train sets into generative Models (DiM), which uses a generative model instead of small and informative images to store information of the target dataset. This scheme minimises the differences in logits predicted by a models pool between real and generated images. At the deployment stage, the generative model synthesises various training samples from random noises on the fly. Lastly, Wang will discuss about iii) Learning to reduce the number of forward, where the team proposed InfoBatch, a novel framework aiming to achieve lossless training acceleration through unbiased dynamic data pruning. Specifically, InfoBatch randomly prunes a portion of less informative samples based on the loss distribution and rescales the gradients of the remaining samples.
SPEAKER Wang Kai is a Ph.D. student from the National University of Singapore, under the supervision of Prof. You Yang. His research area includes dataset distillation, pruning, denoising, privacy and more. Through his research, he aims to improve the efficiency of training deep learning by reducing computational cost, using less data and annotations, from data to algorithms. From 2020 to 2023, Wang published papers from ICLR, NeurlPS, TIP, as well as 8 papers from CVPR and 2 from ECCV. He was awarded the AISG Ph.D. Fellowship when he enrolled in NUS.