Multi-modal self-adaptation during object recognition in an artificial cognitive system

David Miralles, Guillem Garrofé, Carlota Parés, Alejandro González, Gerard Serra, Alberto Soto, Xavier Sevillano, Hans Op de Beeck, Haemy Lee Masson

Producció científica: Article en revista indexadaArticleAvaluat per experts


The cognitive connection between the senses of touch and vision is probably the best-known case of multimodality. Recent discoveries suggest that the mapping between both senses is learned rather than innate. This evidence opens the door to a dynamic multimodality that allows individuals to adaptively develop within their environment. By mimicking this aspect of human learning, we propose a new multimodal mechanism that allows artificial cognitive systems (ACS) to quickly adapt to unforeseen perceptual anomalies generated by the environment or by the system itself. In this context, visual recognition systems have advanced remarkably in recent years thanks to the creation of large-scale datasets together with the advent of deep learning algorithms. However, this has not been the case for the haptic modality, where the lack of two-handed dexterous datasets has limited the ability of learning systems to process the tactile information of human object exploration. This data imbalance hinders the creation of synchronized datasets that would enable the development of multimodality in ACS during object exploration. In this work, we use a multimodal dataset recently generated from tactile sensors placed on a collection of objects that capture haptic data from human manipulation, together with the corresponding visual counterpart. Using this data, we create a multimodal learning transfer mechanism capable of both detecting sudden and permanent anomalies in the visual channel and maintaining visual object recognition performance by retraining the visual mode for a few minutes using haptic information. Our proposal for perceptual awareness and self-adaptation is of noteworthy relevance as can be applied by any system that satisfies two very generic conditions: it can classify each mode independently and is provided with a synchronized multimodal data set.

Idioma originalAnglès
Número d’article3772
RevistaScientific Reports
Estat de la publicacióPublicada - de des. 2022


Navegar pels temes de recerca de 'Multi-modal self-adaptation during object recognition in an artificial cognitive system'. Junts formen un fingerprint únic.

Com citar-ho