Optimization-based learning theory

Optimization-based learning theory is a recent trend in deep learning theory community because the optimization method decides the generalization performance of overparameterized models. A stochastic gradient descent (SGD) is known to perform well for training neural networks and often outperforms other learning methodologies. Therefore, a fundamental question is why SGD performs well for overparameterized models. Given this, we theoretically work on the following two questions:
(Global convergence): Why SGD converges to an optimal solution of the empirical loss despite its nonconvex loss landscape.
(Implicit bias): Why SGD converges to a better solution in generalization despite various solutions.
Analyzing diffusion models adn LLMs are also in this scope.