Machine Intelligence

Recent advancements in 3D deep learning have led to significant progress in improving accuracy and reducing processing time, with applications spanning various domains such as medical imaging, robotics, and autonomous vehicle navigation for identifying and segmenting different structures.

To allow deep learning on 3D data to be adopted by companies, we addressg the difficulties in data annotation, representation learning and deployment challenges in real-world scenarios.

Learning continually for new knowledge accumulation is one of the prominent features of human intelligence. On the other hand, deep learning-based AI models are mostly static with an  assumption that all the distributions are known apriori and are pre-defined in the training data.

The project aims to bridge the gap by developing Self-Aware Continuous Learning Models that are continuously:

-       Aware of the historical distributions,

-       Detect unknown novel distributions,

-       Estimate the complexity of the unknown distributions,

-       Choose relevant data to represent the unknown distributions, and

-       Adapt to represent new distribution without (catastrophically) forgetting any historical distribution.

It also improves AI models’ generalisation capability through representational adaptations with new data. The project will build AI models that learn throughout life, to enable critical applications that warrant dynamic and up-to-date decision making, in diverse application domains (more information).

The world is not stationary. To thrive in evolving environments, humans are capable of continual acquisition and transfer of new knowledge, from a continuous stream of highly correlated visual stimuli over multiple tasks, while still retaining previously-learned experiences.

In contrast to human learning, most AI systems focus on recognizing a pre-defined number of object classes. During training, these systems are often presented with a stationary pool of carefully shuffled, balanced, and homogenised images. In most cases, these systems struggle to recognise new objects when presented with a video stream of new object classes that are incrementally available over time. In addition, they also fail to retain good performance on earlier learned classes without repeated passes over the data, known as catastrophic forgetting.

Dr Zhang Mengmi and Dr Sun Ying aim to design a continual learning framework that recognises incrementally available object classes on video streams without repeated visits of the old data in a memory-efficient manner.

With a multi-disciplinary approach spanning neuroscience, cognitive science, and AI, we hope to gain a better understanding of the following scientific questions and provide insights about replay mechanisms in biological and artificial intelligence:

1. How AIs can make use of compositional structure and representations in the memory to learn to recognize new objects faster;
2. What levels of abstract compositional representations from the old object classes are good for replay, as well as how many times of replay required and in what sequence to replay;
3. How far our current AI continual learning systems are from human continual learning;
4. How future generations of continual learning algorithms can be inspired from human continual learning behaviors; and
5. How the strategies in AI systems assist human continual learning.
   

More information can be found here.

The amount of compute and data points used to train AI systems has increased dramatically. GPT-4 training costs over $100 million. LlaMa, a large language model developed by Meta, was trained on around one billion data points—a more than 160-million fold increase from Perceptron Mark 1, the first artificial neural network.

This has led to challenges in terms of:
    - High costs to train models due to higher volumes of data labelling required.
    - Edge devices with limited resources struggle to execute AI workloads.
    - Anomaly detection use cases may struggle with the lack of anomalies data given that those are rare events.

To improve data and computation efficiency in machine learning, we work on the following research topics:

Learn with Less Data:
    - Achieve efficient learning from a small amount of labelled data or even no labelled data. 
    - We devote to explore on semi-supervised learning/weekly supervised learning, self-supervised learning, transfer learning and unsupervised learning to reduce the data annotation requirement.

Computation efficiency: 
    - Reduce the computation cost of deep AI models, especially on edge devices. 
    - We focus on constraint-aware model compression and energy-efficient learning to compress heavy deep model into a lighter one to meet the requirement with constraints on memory and power consumption.