Towards an Embodied Artificial Intelligence Through Sensory Substitution: Research through Embodied Design, Body-Driven Experimentation, and Multimodal Data for Sensorimotor AI

Project Website

TEMBAI-HCI is part of a coordinated project whose aim is to develop a new approach to AI based on embodied and ecological principles applied to natural cognitive processes such as perception and action through a groundbreaking design of sensory substitution devices.

In this new approach, sensory inputs are intimately dependent on the exploratory movement of the human agent, generating the so-called perception and action cycles.

Three subprojects form part of this coordinated project working towards an Embodied Artificial Intelligence Through Sensory Substitution, each focusing on different aspects:

1. Sensory Substitution Dynamics for an Embodied Artificial Intelligence (TEMBAI-PSI), focused on psychology and philosophy, at Universidad Autonoma de Madrid.
2. Sensory substitution devices (TEMBAI-DEV), at Universidad Politécnica de Madrid, and
3. Research through Embodied Design, Body-Driven Experimentation, and Multimodal Data for Sensorimotor AI (TEMBAI-HCI), at Universidad Carlos III de Madrid (UC3M).

Objectives:

TEMBAI-HCI, led by UC3M, brings together experts in engineering, Human-Computer Interaction (HCI), Interaction Design (IxD), acoustics, AI, cognitive neuroscience, and experimental psychology to contribute to the design of SSD solutions, the development of a sensorimotor database for AI algorithms, and the embodied cognition model developed in the coordinated project.

We will apply Research through Embodied Design (RtED), conducting embodied design explorations to guide design and research; and test emerging knowledge on body perception, action, and sensory substitution devices (SSDs) through experimental studies.

Key contributions include:

1. The development of design guidelines for SSDs and a testing framework, and protocols for experimental studies for real-world tasks, and grounded in an ecological embodied approach.
2. Results from experimental studies investigating the influence of body perception on affordance-based action using SSDs.
3. A proof of concept for our proposed approach to sensorimotor AI in the form of a data competition using the data collected in the studies in this project —including sensory inputs, sensor data, and task outcome metrics.

These results will generate insights that contribute to the development of a theoretical and practical framework for ecological embodied AI.