COGNIRON - The Cognitive Robot Companion

.MPG

The overall objectives of this project are to study the perceptual, representational, reasoning and learning capabilities of embodied robots in human centred environments. The project will develop methods and technologies for the construction of such cognitive robots able to evolve and grow their capacities in close interaction with humans in an open-ended fashion.

The project is structured along 6 Research Activities (RAs) and three Key Experiments (KES).
The latest project results are illustrated for each part of the project below, through the most recent videos.
More detailed information can be found on our pages under www.cogniron.org .

RA1

RA2

RA3

RA4

RA5

RA6

RA7

KE1

KE2

KE3

RESEARCH AREA 1 - Multimodal Dialogues (Top)

(lead partner: University of Bielefeld)

Using language to communicate with others is one of the most important cognitive abilities of humans. Enabling dialogue capability is, therefore, essential for a cognitive robot companion interacting with humans . Since a robot is embodied
and situated in the real environment, its dialogue system has to handle more complex interactions than in human computer interaction.
One of the crucial aspects is the handling of multi-modality because in embodied communication human interlocutors make heavy use of gestures and other non-verbal signals and make references to the shared environment. Building a
flexible dialogue system with the ability to handle multi-modal information and continuously evaluating the system during the different development cycles are the focus of this research activity.

		.MOV	RA1_HelloHuman_Run21#12BADE.mov (Univ. of Bielefeld)
		.MOV	RA1_ILostYou2.mov (Univ. of Bielefeld)
		.MOV	RA1_WhatColorItIs_Ru#12BADC.mov (Univ. of Bielefeld)

RESEARCH AREA 2 - Detection and Understanding of Human Activity (Top)

(lead partner: University of Karlsruhe)

Detection and understanding of human activities is a basic capability of a robot acting in close cooperation with humans. This research activity deals with visual detection and tracking of human faces, which is one important component of this capability. Another studied component is 3D modeling of human body for detection and recognition of postures, and the interpretation of human activities based on gesture, postures, attitudes, and motions.

	.AVI	ActiRec138MB.avi (Univ. of Karlsruhe)
	.MPG	croisement.mpg (LAAS-CNRS)
	.MPG	fondencombreFIM2.mpg (LAAS-CNRS)
	.MPG	Fusion10MB.mpg (LAAS-CNRS)
	.MPG	GROUPE2.MPG (LAAS-CNRS)
	.MPG	main-config-encombre-F.mpg (LAAS-CNRS)
	.MPG	main-config-encombre-FC.mpg (LAAS-CNRS)
	.MPG	main-dynamique.mpg (LAAS-CNRS)
	.MPG	occultationciblestatique-ICOND.mpg (LAAS-CNRS)
	.MPG	occultation-ICOND2.mpg (LAAS-CNRS)
	.MPG	proche-suiviautreindividu-FIDM.mpg (LAAS-CNRS)
	.MPG	proche-suiviavecsaut-FIDM.mpg (LAAS-CNRS)
	.MPG	proche-suivichgapp.mpg (LAAS-CNRS)
	.MPG	proche-suivinormal.mpg (LAAS-CNRS)
	.MPG	proche-suivioccultation-FIDM.mpg (LAAS-CNRS)
	.MPG	proche-suivivariation.mpg (LAAS-CNRS)
	.AVI	run1display.avi (Univ. of Karlsruhe)
	.AVI	run1displayHuman2.avi (Univ. of Karlsruhe)
	.AVI	run1images.avi (Univ. of Karlsruhe)
	.AVI	Spülmaschine_2_normale Geschwindigkeit.avi (Univ. of Karlsruhe)
	.MPG	suiviarret-P.mpg (LAAS-CNRS)
	.MPG	suivi-ICOND.mpg (LAAS-CNRS)
	.WMV	TUSSI.WMV (Univ. of Karlsruhe)

RESEARCH AREA 3 - Social Behaviour and Embodied Interaction (Top)

(lead partner: University of Hertfordshire)

In the context of HRI social behaviour and embodied interaction is an important area of research that involves numerous issues of e.g.
verbal, non-verbal and affective interaction. This RA is concerned with social spaces, gestures, postures and body movements occurring in
human-robot interactions and their role in research on robot motion planning, navigation and recognition of human-activities. This work aims
at providing scientific insights based on experimental data on socially acceptable, primarily non-verbal behaviour. Verbal communication
(dialogue) and affective factors (user comfort) are considered. Experimental data is derived from HRI user studies as well as simulation and robotics testbeds. RA3 also aims at implementing motion planners and reactive motion execution schemes, derived from user studies.

	.MP4	2Hum1Rob2.mp4 (LAAS-CNRS)
	.MPG	2HumWall.mpg (LAAS-CNRS)
	.MPG	HALL.MPG (LAAS-CNRS)
	.MPG	LAASVid2.mpg (LAAS-CNRS)

RESEARCH AREA 4 -Skill and Task Learning (Top)

(lead partner: EPFL)

Learning Skills and Tasks is fundamental to the development of cognitive robot companion. For the companion to show adaptive, life-long learning behaviour, it must be capable of acquiring new skills when required (e.g. change of workplace or of habit on the user’s part). It must be capable of reuse (in the sense of bootstrapping knowledge) and incremental acquisition of skills through the learning of complete tasks. Imitation learning from humans is one of the main tracks investigated in this Research Area.

.MPG
hoap2-calinon-epfl-mpeg4.avi (EPFL)

.MPG
newCognironAnimation.avi (EPFL)

RESEARCH AREA 5 -Spatial Cognition and multi-modal situation awareness (Top)

(lead partner: University of Amsterdam)

The objective of this Research Activity is to understand how an embodied system can come to a conceptualisation of sensory and sensorymotor data for acting, moving manipulating in typical home settings. This addresses fundamental questions of scene understanding, which include object recognition, and extraction of relationships between objects including their temporal properties. The ability to interpret
situations, i.e., states of the environment and relationships between components of the environment that are static or evolving over time,
using different sensing modalities, is essential for a cognitive system to assess its own state and decide its actions. For addressing openendedness the representations are learned by means of an exploratory learning process in which human feedback plays a role.

.AVI
Bielefeld_showing.avi (Univ. of Bielefeld)

.AVI
Bielefeld_testing.avi (Univ. of Bielefeld)

.MPG
IPA_objects.mpg (Fraunhofer IPA)

.MPG
LAAS_integration_views.mpg (LAAS-CNRS)

RESEARCH AREA 6 -Intentionality and Initiative (Top)

(lead partner: LAAS-CNRS)

In addition to understanding its environment, to learning and to interacting with people, making decisions, be it for autonomous deliberation and task achievement, or for human-robot collaborative problem solving is a fundamental capability of a cognitive robot. This Research Area studies decision making abilities in uncertain and varying environments, as well as cognitive architectures for embodied robots, that integrates together perception, action, learning, decision-making and interaction to enable a consistent behaviour.
When interacting with a robot, people tend to attribute intentions to it according to its behaviour and other factors. Studies on intentionality attribution and expression are also conducted within this RA.

.MPG
subjective.mpg (LAAS-CNRS)

RESEARCH AREA 7 -Systems Level Integration and Evaluation (Top)

(lead partner: Fraunhofer IPA)

The integration, demonstration and validation of COGNIRON’s research activities is achieved in three Key Experiments with concrete implementations on real robots in realistic settings, each focusing on one or more fundamental abilities of a cognitive robot. The demonstrations serve as integration platforms for research results and reflect research progress towards a cognitive robot companion. They provide a means for scientific evaluation.

KEY EXPERIMENT 1 - The Robot Home Tour (Top)

(lead partner: University of Bielefeld)

Scenario: A robot is shown to the home environment of its owner. Dialog between the owner and the
robot defines the objects and places, and disambiguates interpretations.
This Key Experiment stresses informational human-robot interaction and the acquisition and learning of scenes and situations.

.AVI

Home tour video [24.8 MB] (LAAS-CNRS)

KEY EXPERIMENT 2 - The Curious Robot (Top)

(lead partner: LAAS-CNRS)

Scenario: A robot interprets a person's attitude and interacts with him to understand his needs, then fetches a requested object and hands it to the person. Unknown objects encountered by the robot are modeled autonomously and named by the person.
This Key Experiment stresses object learning and recognition, human activity understanding, and close physical human-robot interaction.

	.MPG	jido_boit_obst_above.mpg 837 ko (LAAS-CNRS)
	.MPG	jido_boit_side.mpg 350 ko (LAAS-CNRS)
	.MPG	jido-grasp-glass.mpg 856 ko (LAAS-CNRS)
	.MPEG	frame3.mpeg 1.5 MB (LAAS-CNRS)
	.MPG	join2convobs.mpg 1.15 MB (LAAS-CNRS)
	.MOV	manip_nhp.mov 267 kb (LAAS-CNRS)
	.MOV	run1.mov 21.2 MB LAAS-CNRS)
	.MOV	Trajectory_withNHP.mov 5.09 MB (LAAS-CNRS)
	.MOV	Trajectory_withVSTP.mov 4.5 MB(LAAS-CNRS)
	.AVI	VideoGrasping1.avi 18.4 MB (LAAS-CNRS)
	.AVI	VideoGrasping2.avi 21 MB (LAAS-CNRS)

KEY EXPERIMENT 3 - Learning Skills and Tasks (Top)

(lead partner: University of Karlsruhe)

Scenario: A robot is shown by a person how to achieve a task such as arranging objects on a table. The robot learns how to achieve this
task by imitating her, and reproduces the task.
This Key Experiment stresses task learning and reasoning about tasks to acquire knowledge about their goals and achievements.

.WMV KE3_video .wmv (Univ. Karlsruhe / Fraunhofer IPA)