AI for Human-Robot Collaboration​ Workshop

Speaker: Fang Fen

This work aims to solve the fine-grained visual recognition (FGVR) problem in a localise-to-classify fashion, which recognises visual objects from subordinate categories by first localising the discriminative region in images. The discriminative region can be localised as either a single part of an image containing the entire object (e.g., a bird), or multiple parts of an image corresponding to different semantic parts of the object (e.g., heads, torsos and tails). We consider the former because the latter is ill-defined, especially when some semantic parts are inherently invisible in the image. Localising the discriminative region as a single part usually is achieved by predicting a bounding box around the object. However, the bounding box may still contain interference information, or miss out useful context information. This work aims to learn an adaptive box for different images in a self-driven manner with deep reinforcement learning (DRL). Specifically, our approach consists of a discriminative knowledge distillation module and a DRL discriminative adapter learning module. The knowledge distillation module improves model performance by distilling multiple weak classifiers with different pre-defined coarse adaptive boxes into a single strong classifier, and then the DRL adapter learning module predicts the adaptive box for localising the discriminative region adaptively, using the guidance from the strong distilled classifier.

Speaker: Huang Xin, Shi Wei

We will talk about two novel approaches to enhance human-robot conversational interactions. Our first approach, Situated video Q&A, compared various approaches and different fusion strategies for situated reasoning of video Q&A which enable a robot to answer questions about a situation. In our talk, we show that by leveraging a pretrained vision & language model, the generation method is better than a classification approach for all types in the STAR challenge and we achieved the best model award in the STAR 2022 challenge. The method can also be easily adapted to gearbox domain. Our second approach, Robot-initiated Exception handling Dialogue System, aims to improve the robustness of the robot arm in gearbox domain by allowing them to initiate dialogue when they encounter a task they cannot perform or an error in their behavior. The system uses the deep learning technique so that it can handle exceptions and finally towards a successful resolution of the issue. These approaches have the potential to enhance the usability of conversational robots in various domains including manufacturing and healthcare.