Section: Overall Objectives

Introduction

The synthetic definition of the research area of the Bunraku Project is: Perception, decision and action of real and virtual humans in virtual environments and impact on real environments . The main objective of the Bunraku project is to develop cross fertilization of researches in the fields of virtual reality and virtual human. Our challenge is to allow real and virtual humans to naturally interact in a shared virtual environment. This objective is very ambitious as it requires developing and federating several research fields. However, it is fundamental for several areas such as taking into account the human activity in manufacturing, the individual or collective training process, or the human study in cognitive sciences. We have the chance in the team to gather competencies in complementary research areas that allow us to address most of the problems to be solved. Concerning other domains, we are developing strong collaborations with well known research labs in their respective fields.

One of the main concerns of virtual reality is how real users can interact with objects of the virtual world. More generally, interaction can be the result of an individual interaction of one user with one object, but also a resulting interaction between objects in a chain reaction, a common interaction of several users on the same object, and can also be between real and virtual humans. User interaction with objects of the world should be both physical and cognitive: body and brain should be both part of the interaction by the concurrent use of gestures, haptic, gaze, and speech. To allow this multimodal interaction with objects within the world, we will have to develop a generic multilevel model of the objects of the world, and corresponding multimodal rendering (visual, haptic, audio, cognitive) and acting (language, gesture, mind).

Another key objective of Bunraku concerns the interaction between real and virtual humans with the objective to allow them to cooperate and communicate together, but also to be interchanged in a dedicated applicative context. To reach this ambitious objective, we have to develop expressive autonomous human-like characters able to perform in real-time complex and believable actions. Concerning the motion control of virtual humans, we have to increase the introduction of dynamic laws and physical capabilities inside our models to produce more complex and credible motions. In the same time, we have also to take into account two very significant constraints: real-time and controllability. Concerning multi-modal rendering, we have to develop a solution to synchronize in a reactive way gestures of a virtual human with other modalities such as speech (co-verbal gestures) and gaze, in a non predictive context.

To combine autonomy and believability, we are also working on a unified architecture to model individual and collective human behaviors. This architecture includes reactive, cognitive, rational and social skills and manage individual and collective behaviors. To tackle the embodiment of cognitive symbols we have to develop a complex hierarchy of perception decision action loops, by managing the bidirectional exchanges of information between the different levels. We are also continuing to develop our model of behavior coordination, by integrating it in an audio-visual attentive coordination, integrating the management of human memory, the filtering of attention and the cognitive load.

Moreover, to study the real human activity or to train them in the context of a virtual reality application, it is important to control the evolution of the virtual world and in particular the activity of the autonomous characters: this is the purpose of the scenario to supervise this evolution. Orchestrating an interactive session is useful to take partially the control of autonomous characters populating the virtual world, but also to control the impact of the user interaction.

Our objectives are decomposed into three complementary research themes:

• Multimodal interaction with objects within the world;

• Expressive autonomous characters;

• Interactive scenario languages.

To reach all these objectives, it is necessary to develop complementary research activities. In the past years, we have worked independently on most of these topics. In the Bunraku research group, in complement of the individual evolution of the research activity on each field, we want to reach a new stage concerning their integration into a common and federative research program.