.png?sfvrsn=d3a97fa3_9)
Learning with Deep Learning: Optimiser and Network Architecture
[CFAR Rising Star Lecture Series]
Learning with Deep Learning: Optimiser and Network Architecture by Dr Zhou Pan
Deep learning has achieved great success in many applications. While it is computationally challenging to train deep networks on increasingly large-scale datasets, network architectures have also shown their importance in applications.
In this talk, Dr Zhou Pan will explore two main important parts of deep learning – deep optimiser and network architecture design. Dr Zhou will first cover network optimisation, where he will introduce the followings: 1) A faster optimiser, i.e. ADAptive Nesterov momentum algorithm (Adan) which surpasses the SoTA optimisers across many tasks in CV, NLP and RL fields; and 2) A general weight-decay-integrated Nesterov acceleration technique to speed up the convergence of previous network optimisers (e.g. SGD, Adam). He will then talk about architecture design where MetaFormer reveals the important parts of the vision transformer while iFormer designs special architecture to learn both high- and low-frequency in data respectively.
Learning with Deep Learning: Optimiser and Network Architecture by Dr Zhou Pan
23 Feb 2023 | 3.00pm (Singapore Time)
Deep learning has achieved great success in many applications. While it is computationally challenging to train deep networks on increasingly large-scale datasets, network architectures have also shown their importance in applications.
In this talk, Dr Zhou Pan will explore two main important parts of deep learning – deep optimiser and network architecture design. Dr Zhou will first cover network optimisation, where he will introduce the followings: 1) A faster optimiser, i.e. ADAptive Nesterov momentum algorithm (Adan) which surpasses the SoTA optimisers across many tasks in CV, NLP and RL fields; and 2) A general weight-decay-integrated Nesterov acceleration technique to speed up the convergence of previous network optimisers (e.g. SGD, Adam). He will then talk about architecture design where MetaFormer reveals the important parts of the vision transformer while iFormer designs special architecture to learn both high- and low-frequency in data respectively.
SPEAKER
Dr Zhou Pan
Senior Research Scientist
Sea AI Lab (SAIL), Sea Limited
Senior Research Scientist
Sea AI Lab (SAIL), Sea Limited
Dr Zhou Pan obtained his Ph.D. Degree in Computer Science from the National University of Singapore (NUS) in 2019 and had previously graduated from Peking University in 2016. Prior to his current role as a Senior Research Scientist at Sea AI Lab (SAIL), he worked as a research scientist at Salesforce AI Lab from 2019 to 2021. Dr Zhou’s research interests include computer vision, machine learning and optimisation. He has published about 50 papers and was the winner of the Microsoft Research Asia Fellowship in 2018.
A*STAR celebrates International Women's Day
From groundbreaking discoveries to cutting-edge research, our researchers are empowering the next generation of female science, technology, engineering and mathematics (STEM) leaders.
Get inspired by our #WomeninSTEM