Team siames

Overall Objectives
Scientific Foundations
Application Domains
New Results
Contracts and Grants with Industry
Other Grants and Activities

Section: New Results

Synoptic objects allowing autonomous agents to interact with a virtual environment

Participants : Marwan Badawi, Stéphane Donikian [ contact ] .

We propose the STARFISH (Synoptic-objects for Tracking Actions Received From Interactive Surfaces and Humanoids) architecture that uses Synoptic Objects to allow real-time object manipulation by autonomous agents in an informed environment. We define a minimal set of primitive Basic Actions which are used to build Complex Actions. We then assign these actions to Interactive Surfaces which are the parts of an object's geometry that are concerned by the action. The agent then uses these Interactive Surfaces to get the data specific to the object when it wants to manipulate it and to adapt its behavior accordingly.

STARFISHstands for Synoptic-objects for Tracking Actions Received From Interactive Surfaces and Humanoids. It is a new system which allows the easy definition of interactions between autonomous agents and the Synoptic Objects in the environment.

Synoptic Objectsare objects designed to offer to the autonomous agent a summary, or synopsis, of what interactions they afford. When an agent queries such an object, it knows what actions it can perform, where it should position itself, where it should place its hands, what state the object is in, whether it is allowed to perform the action, etc... All these indications, are given through the use of STARFISH Actions and Interactive Surfaces. An autonomous agent is itself considered as a Synoptic Object which allows interaction between agents (such as shaking hands) without any additional special considerations.

Figure 30. The complete set of Basic Actions.
TransferTransfer location of an object ( go, carry )
MoveMove own body part ( kick, reach )
GraspGrab an object ( grasp, let go )
IngestTake an object into own body ( eat )
SpeakProduce sound ( say, sing )
AttendFocus sense organ ( listen, look at )

STARFISH Actionsconsist of a group of simple atomic actions, the Basic Actions which are the building blocks used to create Complex Actions. Through the Basic Actions described in Figure  30 , we can build more complex and varied actions. These Basic Actions are inspired by the Conceptual Dependency Theory (CDT), which seeks to represent complex natural language input using simple basic action verbs. We took CDT and used it the other way around: to generate complex behaviors using simple basic actions as building blocks. Note that for the remainder of this section, whenever we refer to a Basic Action, its name will be typeset in bold faceto differentiate it from the corresponding verb.

For example, the Open Door Complex Action (illustrated in figure  31 (a)) can be easily decomposed into its Basic Actions:

These Complex Actions are managed using state machines through the HPTS++ architecture (cf paragraph 5.4 ). A very simplified model of the Open Door Complex Action can be seen in figure  31 (b).

Figure 31. The Open Door Complex Action.
(a) An agent pulling open a door  (b) The simplified state machine

Interactive Surfacesare then used to model and describe the affordances of an object. They are generally parts of the object's geometry that act as hotspots when a certain action is to be accomplished, such as where to place the agent's hand when Grasping the doorknob or where to stand in order to open that door(Figure  32 ). Each Synoptic Object can have as many Interactive Surfaces as needed, depending on how much interaction it offers. Furthermore, many Interactive Surfaces of a single object can be associated to the same Basic Action, and the choice of the Interactive Surface is determined by the Complex Action the agent is doing. For example, in order to carry a suitcase, the agent has to Grabit by the handle, but in order to open the same suitcase, the agent has to Grabit by the lock.

Figure 32. Interactive Surfaces for the door modeled in the STARFISH Editor.

Seeing how the Interactive Surfaces and STARFISH Actions are two completely independent concepts, we do not have to go through with them both when creating a new Synoptic Object. When an object with a new shape but with the same functionality has to be introduced into the simulation, it is only necessary to model its appropriate Interactive Surfaces without having to modify their associated behaviors.


Logo Inria