Perception, Action and Cognition, PACO-Plus

The design and construction of cognitive systems that can perceive and interact with their
environment is an extremely challenging task. The goal of the PACO-PLUS project is to
show that such a construction is feasible with a design relying on a theoretical and measurable
basis which on the one hand applies to humans and on the other hand to artificial systems.
More precisely, PACO-PLUS aims at the design of cognitive robots capable of developing
perceptual, behavioural and cognitive categories in a measurable way and of communicating
and sharing these with humans and other artificial agents. To achieve this, the project brings
together a consortium of robotics researchers, engineers, computer vision scientists, linguists,
theoretical neuroscientists and cognitive psychologists.
The systems we aim at are supposed to interact and function together with humans. They are
meant to be able to cooperate and to enter a dialogue communicating with the human.
Therefore they need to understand both, what they perceive and what they do. Our hypothesis
is that such understanding can only be attained if we consider perception and action together.
In this process the artificial system needs to learn and adapt to the momentarily existing
situation to be able to act and react appropriately.
Central to the approach are three almost axiomatic assumptions which are linked to each other
and which are the building blocks of a new approach required to create cognitive artificial
agents:
• Objects and Actions are inseparably intertwined; these so-called Object-Action
Complexes are the building blocks of cognition.
• Cognition is based on recurrent processes involving nested feedback loops operating
on, contextualizing and reinterpreting object-action complexes. This is done through
actively closing the perception-action cycle.
• A unified measure of success and progress can be obtained through minimization of
contingencies which an artificial cognitive system experiences while interacting with
the environment or other agents, given the drives of the system.
To demonstrate the feasibility of our approach we will build robot systems that step by step
get increasingly advanced cognitive capabilities. They will operate in real-world scenarios
and be able to learn to interact and perform basic communication with humans.
During the first years we will do this on existing robots that can manipulate simple things in
well-defined tasks while developing an understanding of how these things should be regarded
as objects belonging to different categories on the basis of the actions they can perform on
them and the measure of success. Both far and near perception by vision and haptics is
included, as well as recognition and classification of actions that these systems and other
agents perform.
Later during the project we will have developed more advanced robot systems with more
human-like abilities. Then a larger repertoire of actions can be considered and the systems
will therefore be used in more complex situations. Moreover, more complex interaction
between the systems and humans will be possible with an increasing need for communication
between agents, leading to steps towards language.
The feasibility of our approach will be demonstrated in two real-world scenarios: 1) A robot
system to augment human action and aid humans, and 2) a robot system able to explore and
manipulate objects in an unconstrained environment.