- A5.4. Computer vision
- A5.5. Computer graphics
- A5.5.1. Geometrical modeling
- A5.5.3. Computational photography
- A5.5.4. Animation
- A5.6. Virtual reality, augmented reality
- A9.1. Knowledge
- A9.2. Machine learning
- A9.3. Signal analysis
- B2. Health
- B2.2. Physiology and diseases
- B5.7. 3D printing
- B9.1. Education
- B9.2.2. Cinema, Television
- B9.2.3. Video games
- B9.2.4. Theater
- B9.6.6. Archeology, History
- B9.6.10. Digital humanities
1 Team members, visitors, external collaborators
- Remi Ronfard [Team leader, Inria, Senior Researcher, HDR]
- Melina Skouras [Inria, Researcher]
- Stefanie Hahmann [Institut polytechnique de Grenoble, Professor, HDR]
- Olivier Palombi [Univ Grenoble Alpes, Professor, HDR]
- Nachwa Abou Bakr [Inria, until Aug 2021]
- Sandrine Barbois [Univ de Provence]
- Qianqian Fu [Univ Grenoble Alpes, until Nov 2021]
- Vaishnavi Ameya Murukutla [Univ Grenoble Alpes]
- Emmanuel Rodriguez [Inria]
- Manon Vialle [Univ Grenoble Alpes]
- Pierre Ecormier-Nocca [Inria, Engineer, Jan 2021]
Interns and Apprentices
- Karthik Subramanyam Chakka [Inria, from Feb 2021 until Jul 2021]
- Valentin Gomelet–Richard [Inria, from Jul 2021 until Aug 2021]
- Camelia Guerraoui [Inria, from Jun 2021 until Aug 2021]
- Hugo Kraft [Inria, from Feb 2021 until Jul 2021]
- Yidi Zhu [Inria, from Feb 2021 until Jul 2021]
- Geraldine Christin [Inria, until Aug 2021]
- David Jourdan [Inria, from Oct 2021 until Nov 2021]
2 Overall objectives
ANIMA focuses on developing computer tools for authoring and directing animated movies, interactive games and mixed-reality applications, using virtual sets, actors, cameras and lights. This raises several scientific challenges. Firstly, we need to build a representation of the story that the user/director has in mind, and this requires dedicated user interfaces for communicating the story. Secondly, we need to offer tools for authoring the necessary shapes and motions for communicating the story visually, and this requires a combination of high-level geometric, physical and semantic models that can be manipulated in real-time under the user’s artistic control. Thirdly, we need to offer tools for directing the story, and this requires new interaction models for controlling the virtual actors and cameras to communicate the desired story while maintaining the coherence of the story world.
2.1 Understanding stories
Stories can come in many forms. An anatomy lesson is a story. A cooking recipe is a story. A geological sketch is a story. Many paintings and sculptures are stories. Stories can be told with words, but also with drawings and gestures. For the purpose of creating animated story worlds, we are particularly interested in communicating the story with words in the form of a screenplay or with pictures in the form of a storyboard. We also foresee the possibility of communicating the story in space using spatial gestures. The first scientific challenge for the ANIMA team is to propose new computational models and representations for screenplays and storyboards, and practical methods for parsing and interpreting screenplays and storyboards from multimodal user input. To do this, we reverse engineer existing screenplays and storyboards, which are well suited for generating animation in traditional formats. We also explore new representations for communicating stories with a combination of speech commands, 3D sketches and 3D gestures, which promise to be more suited for communicating stories in new media including virtual reality, augmented reality and mixed reality.
2.2 Authoring story worlds
Telling stories visually creates additional challenges not found in traditional, text-based storytelling. Even the simplest story requires a large vocabulary of shapes and animations to be told visually. This is a major bottleneck for all narrative animation synthesis systems. The second scientific challenge for the ANIMA team is to propose methods for quickly authoring shapes and animations that can be used to tell stories visually. We devise new methods for generating shapes and shape families, understanding their functions, styles, material properties and affordances, authoring animations for a large repertoire of actions, and printing and fabricating articulated and deformable shapes suitable for creating physical story worlds with tangible interaction.
2.3 Directing story worlds
Lastly, we develop methods for controlling virtual actors and cameras in virtual worlds and editing them into movies in a variety of situations ranging from 2D and 3D professional animation, to virtual reality movies and real-time video games. Starting from the well-established tradition of the storyboard, we create new tools for directing movies in 3D animation, where the user is really the director, and the computer is in charge of its technical execution using a library of film idioms. We also explore new areas, including the automatic generation of storyboards from movie scripts for use by domain experts, rather than graphic artists.
3 Research program
The four research themes pursued by ANIMA are (i) the geometry of story worlds; (ii) the physics of story worlds; (iii) the semantics of story worlds; and (iv) the aesthetics of story worlds.
In each theme, significant advances in the state of the art are needed to propose computational models of stories, and build the necessary tools for translating stories to 3D graphics and animation.
3.1 Geometric modeling of story worlds
Scientist in charge: Stefanie Hahmann
Other participants: Rémi Ronfard, Mélina Skouras
We aim to create intuitive tools for designing 3D shapes and animations which can be used to populate interactive, animated story worlds, rather than inert and static virtual worlds. In many different application scenarios such as preparing a product design review, teaching human anatomy with a MOOC, composing a theatre play, directing a movie, showing a sports event, 3D shapes must be modeled for the specific requirements of the animation and interaction scenarios (stories) of the application.
We will need to invent novel shape modelling methods to support the necessary affordances for interaction and maintain consistency and plausibility of the shape appearances and behaviors during animation and interaction. Compared to our previous work, we will therefore focus increasingly on designing shapes and motions simultaneously, rather than separately, based on the requirements of the stories to be told.
Previous work in the IMAGINE team has emphasized the usefulness of space-time constructions for sketching and sculpting animation both in 2D and 3D. Future work in the ANIMA team will further develop this line of research, with the long-term goal of choreographing complex multi-character animation and providing full authorial and directorial control to the user.
3.1.1 Space-time modeling
The first new direction of research in this theme is an investigation of space-time geometric modeling, i.e. the simultaneous creation of shapes and their motions. This is in continuity with our previous work on "responsive shapes", i.e. making 3D shapes respond in an intuitive way during both design and animation.
3.1.2 Spatial interaction
A second new direction of research of the ANIMA team will be the extension of sketching and sculpting tools to the case of spatial 3D interaction using virtual reality headsets, sensors and trackers.
Even though 3D modeling can be regarded as an ideal application for Virtual Reality, it is known to suffer from the lack of control for freehand drawing. Our insight is to exploit the expressiveness of hand (controller) motion and simple geometric primitives in order to form an approximated 3D shape. The goal is not to generate a final well shaped product, but to provide a 3D sketching tool for creating early design shapes, kind of scaffolds, and for rough idea exploration. Standard 3D modeling systems can then take over to generate more complex shape details.
Research directions to be explored include (i) direct interaction using VR; (ii) applications to form a 3D shape from rough design ideas; (iii) applications to modify existing objects during design review sessions; and (iv) provide tools to ease communications about imagined shapes.
3.2 Physical modeling of story worlds
Scientist in charge: Mélina Skouras
Other participants: Stefanie Hahmann, Rémi Ronfard
When authoring and directing story worlds, physics is important to obtain believable and realistic behaviors, e.g. to determine how a garment should deform when a character moves, or how the branches of a tree bend when the wind start to blow. In practice, while deformation rules could be defined a priori (e.g. procedurally), relying on physics-based simulation is more efficient in many cases as this means that we do not need to think in advance about all possible scenarii. In ANIMA, we want to go a step further. Not only do we want to be able to predict how the shape of deformable objects will change, but we also want to be able to control their deformation. In short, we are interested in solving inverse problems where we adjust some parameters of the simulation, yet to be defined so that the output of the simulation matches what the user wants.
By optimizing design parameters, we can get realistic results on input scenarii, but we can also extrapolate to new settings. For example, solving inverse problems corresponding to static cases can be useful to obtain realistic behaviors when looking at dynamics. E.g. if we can optimize the cloth material and the shape of the patterns of a dress such that it matches what an artist designed for the first frame of an animation, then we can use the same parameters for the rest of the animation. Of course, matching dynamics is also one of our goals.
Compared to more traditional approaches, this raises several challenges. It is not clear what the best way is for the user to specify constraints, i.e. how to define what she wants (we do not necessarily want to specify the positions of all nodes of the meshes for all frames, for example). We want the shape to deform according to physical laws, but also according to what the user specified, which means that the objectives may conflict and that the problem can be over-constrained or under-constrained.
Physics may not be satisfied exactly in all story worlds i.e. input may be cartoonish, for example. In such cases, we may need to adapt the laws of physics or even to invent new ones. In computational fabrication, the designer may want to design an object that cannot be fabricated using traditional materials for example. But in this case, we cannot cheat with the physics. One idea is to extend the range of things that we can do by creating new materials (meta-materials), creating 3D shapes from flat patterns, increasing the extensibility of materials, etc.
To achieve these goals, we will need to find effective metrics (how to define objective functions that we can minimize); develop efficient models (that can be inverted); find suitable parameterizations; and develop efficient numerical optimization schemes (that can account for our specific constraints).
3.2.1 Computational design of articulated and deformable objects
We would like to extend sketch-based modeling to the design of physical objects, where material and geometric properties both contribute to the desired behaviors. Our goal in this task will be to provide efficient and easy-to-use physics-aware design tools. Instead of using a single 3D idealized model as input, we would like to use sketches, photos, videos together with semantic annotations relating to materials and motions. This will require the conceptualization of physical storyboards. This implies controlling the matter and includes the computational design of meso-scale materials that can be locally assigned to the objects; the optimization of the assignment of these materials such that the objects behave as expected; the optimization of the actuation of the object (related to the point below). Furthermore, the design of the meta-materials/objects can take into account other properties in addition to the mechanical aspects. Aesthetics, in particular, might be important.
3.2.2 Physical storyboarding
Story-boards in the context of physical animation can be seen as a concept to explain how an object/character is supposed to move or to be used (a way to describe the high-level objective). Furthermore, they can be used to represent the same object from different views, in different scales, even at different times and in different situations, to better communicate the desired behavior. Finally, they can be used to represent different objects behaving "similarly".
Using storyboards as an input to physical animation raises several scientific challenges. If one shape is to be optimized; we need to make sure that the deformed shape can be reached (i.e. that there is a continuous path from the initial shape to the final shape) - e.g. deployable structures. We will need to explore different types of inputs: full target animations, key-frames, annotations (arrows), curves, multi-modal inputs. Other types of high-level goals, which implies that the object should be moving/deforming in a certain way (to be optimized), e.g locomotion, dressing-up a character.
3.3 Semantic modeling of story worlds
Scientist in charge: Oliver Palombi
Other participants: Rémi Ronfard, Nicolas Szilas
Beyond geometry and physics, we aim at representing the semantics of story worlds. We use ontologies to organize story worlds into entities described by well defined concepts and relations between them. Especially important to us is the ability to "depict" story world objects and their properties during the design process 13 while their geometric and material properties are not yet defined. Another important aspect of this research direction is to make it possible to quickly create interactive 3D scenes and movies by assembling existing geometric objects and animations. This requires a conceptual model for semantic annotations, and high level query languages where the result of a semantic query can be a 3D scene or 3D movie.
One important application area for this research direction is the teaching of human anatomy. The Phd thesis of Ameya Murukutla focuses on automatic generation of augmented reality lessons and exercises for teaching anatomy to medical students and sports students using the prose storyboard language which we introduced during Vineet Gandhi's PhD thesis 28. By specializing to this particular area, we are hoping to obtain a formal validation of the proposed methods before we attempt to generalize them to other domains such as interactive storytelling and computer games.
3.3.1 Story world ontologies
We will extend our previous work on ontology modeling of anatomy 26, 30 in two main directions. Firstly, we will add procedural representations of anatomic functions that make it possible to create animations. This requires work in semantic modeling of 3D processes, including anatomic functions in the teaching of anatomy. This needs to be generalized to actor skills and object affordances in the more general setting of role playing games and storytelling. Secondly, we will generalize the approach to other storytelling domains. We are starting to design an ontology of dramatic functions, entities and 3D models. In storytelling, dramatic functions are actions and events. Dramatic entities are places, characters and objects of the story. 3D models are virtual actors, sets and props, together with their necessary skills and affordances. In both cases, story world generation is the problem of linking 3D models with semantic entities and functions, in such a way that a semantic query (in natural language or pseudo natural language) can be used to create a 3D scene or 3D animation.
3.3.2 Story world scenarios
While our research team is primarily focused on providing authoring and directing tools to artists, there are cases where we also would like to propose methods for generating 3D content automatically. The main motivation for this research direction is virtual reality, where artists attempt to create story worlds that respond to the audience actions. An important application is the emerging field of immersive augmented reality theatre 21, 32, 24, 25, 20, 15, 22. In those cases, new research work must be devoted to create plausible interactions between human and virtual actors based on an executable representation of a shared scenario.
3.4 Aesthetic modeling of story worlds
Scientist in charge:Rémi Ronfard
Other participants: Stefanie Hahmann, Mélina Skouras, François Garnier
Data-driven methods for shape modeling and animation are becoming increasingly popular in computer graphics, due to the recent success of deep learning methods. In the context of the ANIMA team, we are particularly interested in methods that can help us capture artistic styles from examples and transfer them to new content. This has important implications in authoring and directing story worlds because it is important to offer artistic control to the author or director, and to maintain a stylistic coherence while generating new content. Ideally, we would like to learn models of our user's authoring and directing styles, and create contents that matches those styles.
3.4.1 Learning and transferring shape styles
We want to better understand shape aesthetics and styles, with the long-term goal of creating complex 3D scenes with a large number of shapes with consistent styles. We will also investigate methods for style transfer, allowing to re-use existing shapes in novel situations by adapting their style and aesthetics 23.
In the past exhaustive research has been done on aesthetic shape design in the sense of fairness, visual pleasing shapes using e.g. bending energy minimization and visual continuity. Note, that these aspects are still a challenge in motion design (see next section). In shape design, we now go one step further by focusing on style. Whereas fairness is general, style is more related to application contexts, which we would like to formalize.
3.4.2 Learning and transferring motion styles
While the aesthetics of spatial curves and surfaces has been extensively studied in the past, resulting in a large vocabulary of spline curves and surfaces with suitable control parameters, the aesthetics of temporal curves and surfaces is still poorly understood. Fundamental work is needed to better understand which geometric features are important in the perception of the aesthetic qualities of motions and to design interpolation methods that preserve them. Furthermore, we would like to transfer the learned motion styles to new animations. This is a very challenging problem, which we started to investigate in previous work in the limited domains of audiovisual speech animation 11 and hand puppeteering 19.
3.4.3 Learning and transfering film styles
In recent years, we have proposed new methods for automatically composing cinematographic shots in live action video 18 or 3D animation 16 and to edit them together into aesthetically pleasing movies 17. In future work, we plan to apply similar techniques for the new use case of immersive virtual reality. This raises interesting new issues because spatial and temporal discontinuities must be computed in real time in reaction to the user's movements. We have established a strong collaboration with the Spatial Media team at ENSADLAB to investigate those issues. We also plan to transfer the styles of famous movie directors to the generated movies by learning generative models of their composition and film editing styles, therefore extending the previous work of Thomas 29 from photographic style to cinematographic style. The pioneering work of Cutting and colleagues 14 used a valuable collection of 150 movies covering the years 1935 to 2010, mostly from classical Hollywood cinema. A more diverse dataset including European cinema in different genres and styles will be a valuable contribution to the field. Towards this goal, we are building a dataset of movie scenes aligned with their screenplays and storyboards.
4 Application domains
The research goals of the ANIMA team are applicable to many application domains which use computer graphics and are in demand of more intuitive and accessible authoring and directing tools for creating animated story worlds. This includes arts and entertainment, education and industrial design.
Arts and entertainement
Animated story worlds are central to the industries of 3D animation and video games, which are very strong in France. Designing 3D shapes and motions from storyboards is a worthwhile research goal for those industries, where it is expected to reduce production costs while at the same time increasing artistic control, which are two critical issues in those domains. Furthermore, story is becoming increasingly important in video games and new authoring and directing tools are needed for creating credible interactive story worlds, which is a challenge to many video game companies. Traditional live action cinematography is another application domain where the ANIMA team is hoping to have an impact with its research in storyboarding, virtual cinematography and film editing.
Performance art, including dance and theater, is an emergent application domain with a strong need for dedicated authoring and directing tools allowing to incorporate advanced computer graphics in live performances. This is a challenging application domain, where computer-generated scenography and animation need to interact with human actors in real-time. As a result, we are hoping that the theater stage becomes an experimental playground for our most exploratory research themes. To promote this new application domain, we are organizing the first international workshop on computer theater in Grenoble in February 2020, under the name Journées d’Informatique Théâtrale (JIT). The workshop will assemble theater researchers, artists and computer engineers whose practice incorporates computer graphics as a means of expression and/or a creative tool. With this workshop, our goal is to create a new research discipline that could be termed “computer theater”, following the model of computer music, which is now a well established discipline.
Teaching of Anatomy is a suitable domain for research. As professor of Anatomy, Olivier Palombi gives us the opportunity to experiment in the field. The formalization of anatomical knowledge in our ontology called My Corporis Fabrica (MyCF) is already operational. One challenge for us is to formalize the way anatomy is taught or more exactly the way anatomical knowledge is transmitted to the students using interactive 3D scenes.
Museography is another related application domain for our research, with a high demand for novel tools allowing to populate and animate virtual reconstructions of art works into stories that make sense to museum audiences of all ages. Our research is also applicable to scientific museography, where animated story worlds can be used to illustrate and explain complex scientific concepts and theories.
Our research in designing shapes and motions from storyboards is also relevant to industrial design, with applications in the fashion industry, the automotive industry and in architecture. Those three industries are also in high demand for tools exploiting spatial interaction in virtual reality. Our new research program in physical modeling is also applicable to those industries. We have established strong partnerships in the past with PSA and Vuitton, and we will seek to extend them to architectural design as well in the future.
5 Social and environmental responsibility
5.1 Footprint of research activities
ANIMA is a small team of four permanent researchers and three PhD students, so our footprint is limited. We estimate that we run approximately twelve computers, including laptops, desktops and shared servers, at any given time.
Research in computer graphics is not (yet) data intensive. We mostly devise procedural algorithms, which require limited amounts of data. One notable exception is our work on computational editing of live performances, which produces large amounts of ultra high definition video files. We have opted for a centralized video server architecture (KinoAI) so that at least the videos are never duplicated and always reside on a single server.
On the other hand, our research requires prowerful graphics processing units (GPU) which significantly increase the power consumption of our most powerful desktops.
The COVID19 crisis has changed our working habits heavily. We have learned to work from home most of the week, and to abandon international travel entirely. This holds the promise of a vastly reduced footprint. But it is too early to say whether this is sustainable. While the team as a whole has been able to function in adverse conditions, it has become increasingly difficult to welcome interns and Masters students.
5.2 Impact of research results
Our research does not directly address social and environmental issues. Our work on 3D printing may have positive effects by allowing the more efficient use of materials in the production of prototypes. Our work on virtual medical simulation and training may provide an alternative in some cases to animal experiments and costly robotic simulations.
Our most important application domain is arts and culture, including computer animation and computer games. Globally, those sectors are creating jobs, rather than destroying them, in France and in Europe. The impact of our discipline is therefore positive at least in this respect.
We are more concerned with the impact of the software industry as a whole, i.e. private companies who implement our research papers and include them in their products. We note a tendency to increase the memory requirements of software. While much effort in software engineering is devoted to improving the execution speed of graphics programs, there is not enough effort in optimizing their footprint. This is an area that may be worth investigating in our future work.
6 Highlights of the year
6.1 PhD thesis
Ameya Murukutla defended her PhD thesis on "Text-to-movie authoring of anatomy lessons" on December 7, 2021.
The jury was composed of Marie-Christine Rousset (UGA), Philippe Joly (IRIT, Univ. Toulouse), Pierre-Antoine Champin (LIRIS, Univ. Lyon, ERCIM), Nady Hoyek (Univ. Lyon) and Marc Braun (doyen de la faculté de médecine de l’Université de Lorraine).
Mélina Skouras received the Young Researcher Award 2021 from Shape Modeling International.
7 New results
7.1 Film directing for Computer Games and Animation
Participants: Rémi Ronfard.Over the last forty years, researchers in computer graphics have proposed a large variety of theoretical models and computer implementations of a virtual film director, capable of creating movies from minimal input such as a screenplay or storyboard. The underlying film directing techniques are also in high demand to assist and automate the generation of movies in computer games and animation.
In this comprehensive survey, we present a historical and up-to-date summary of research in algorithmic film directing, and identify promising avenues and hot topics for future research.
The survey was presented as a state-of-the-art report (STAR) during the online Eurographics conference in May 2021, and constitutes the first chapter of the final report for the Montage Spatial joint project between ANIMA and ENSADLAB.
7.2 Fashion Transfer: Dressing 3D Characters from Stylized Fashion Sketches
Participant: Stefanie Hahmann.
An intuitive way to facilitate 3D creation is to use sketches, because everyone knows how to draw. However, reconstructing a 3D digital object from a drawing is a complex challenge that sketch-based modeling methods have been trying to address for the last decade. Indeed, a 2D image of an object can represent an infinity of 3D objects, since the depth is unknown. It is common to make assumptions about the object or to formulate geometrical constraints to restrict the space of solutions. Our novel approach is to take advantage of developable surfaces, not only for reconstruction from a single image of the object but also to allow an extension to fashion designs. Built on the differential geometry properties of developable surfaces, we have developed a sketch-based modeling system for stylized fashion sketches 6. This method not only preserves the style when reconstructing but also when transferring to characters with different morphology. This work is the result of a collaboration with Adrien Bousseau (GraphDeco INRIA) and Damien Rohmer, Marie-Paule Cani (LIX Ecole Polytechnique).
7.3 Fine Wrinkling on Coarsely-Meshed Thin Shells
Participant: Melina Skouras.
In this work, we propose a new model and algorithm to capture the high-definition statics of thin shells via coarse meshes 5. This model predicts global, fine-scale wrinkling at frequencies much higher than the resolution of the coarse mesh; moreover, it is grounded in the geometric analysis of elasticity, and does not require manual guidance, a corpus of training examples, nor tuning of ad-hoc parameters. We first approximate the coarse shape of the shell using tension field theory, in which material forces do not resist compression. We then augment this base mesh with wrinkles, parameterized by an amplitude and phase field that we solve for over the base mesh, which together characterize the geometry of the wrinkles. We validate our approach against both physical experiments and numerical simulations, and show that our algorithm produces wrinkles qualitatively similar to those predicted by traditional shell solvers requiring orders of magnitude more degrees of freedom. This work is the result of a collaboration with Etienne Vouga's group (University of Texas at Austin) and Danny Kaufman (Adobe Research).
7.4 Geometric construction of auxetic metamaterials
Participant: Stefanie Hahmann.
Recent advances in digital manufacturing, where computational design, materials science and engineering meet, offer whole new perspectives for tailoring mechanical properties and fabrication of new meta-materials with applications as diverse as product design, architecture, engineering and art. A meta-material is a material whose microstructure can be controlled to achieve the desired macroscopic deformation behavior.
We focussed on a category of metamaterials called auxetic structures, or auxetic networks. Auxetic materials are characterized by a negative Poisson's ratio. They do not behave like usual materials, because when they are stretched in one direction, they expand in the perpendicular direction.Whereas regular auxetic networks are well studied, our focus is on disordered auxetic networks. In particular, we are exploring geometrical strategies to generate 2-dimensional random auxetic porous meta-materials. Starting from a dense irregular network, we seek to reduce the Poisson's ratio, by pruning bonds (edges) based solely on geometric criteria. To this end, we first deduce some prominent geometric features from regular auxetic networks and then introduce a strategy combining a pure geometric pruning algorithm followed by a physics-based testing phase to determine the resulting Poisson's ratio of our networks 4. We provide numerical results and statistical validation. We also show physical tests with both laser-cut rubber networks and 3D-printed networks showing auxetic behaviour, see Figure.
7.5 Feature Lines Modification Based on As-Stiff-As-Needed Surface Deformation
Participant: Stefanie Hahmann.
Shape creation for the automotive industry can be thought of as exploring the "mental shape space" of designers. To improve communication between designers during a project review in an immersive environment, we introduced a new physics-based method to intuitively deform manufactured shapes. Standard methods such as ARAP shape deformation produce overly organic shapes, inappropriate in our context. With the goal of being able to control the material stiffness and prevent undesirable organic effects, we first introduced in 2020 virtual anisotropic materials for shape, associated with membrane-like structural behavior 2. We then developed in 2021 a deformation method guided by shape features 7 which gives the user a powerful tool for exploring a new shape space. The CIFRE thesis of Youna le Vaou co-supervised with JC Léon was a fruitful collaboration with the Virtual Reality center of PSA (S. Masfrand and M. Mika).
8 Partnerships and cooperations
8.1 European initiatives
8.1.1 FP7 & H2020 projects
Participants: Hahmann Stefanie, Mélina Skouras, Bonneau Georges-Pierre (Maverick team).
- Title: Analysis, Design, And Manufacturing using Microstructures
- H2020 FET OPEN RIA, contract no. 862025.
- BCAM Bilbao, Spain (coordinator)
- Technion, Israel
- INRIA Grenoble, France
- EPFL, Switzerland
- TU Wien, Austria
- Universidad del Pais Vasco UPV/EHU, Spain
- Seoul National University
- Stratasys LTD
- Trimek SA
- Hutchinson SA
- Date/Duration: 2020-2023
Research of the ANIMA team within the ADAM2 project combines user-guided shape modelling using microstructures, followed by validation and structural optimization using physical process simulation, and finalized by physical realizations via additive and hybrid manufacturing. As a first contribution, we developped a geometric algorithm for the construction of a 2D random porous auxetic metamaterial. We validated the approach by fabricating some real exemplars using additive manufacturing (3D printing with flex filament) and laser-cutting rubber material 4.
The contract is funding the PhD thesis of Emmanuel Rodriguez. The goal is here to design a laser-cut metamaterial able to fit a given shape. An inverse modeling approach is used to compute the optimal laser-cut pattern.
8.2 National initiatives
8.2.1 ANR ONTO-SIDES
Participants: Olivier Palombi.
SIDES is a learning digital platform common to all French medical schools, used for official exams (tests) in faculties and for the training of students for the National Ranking Exam (ECN) which is fully computerized since 2016 (ECNi).
As part of this platerform, Olivier Palombi was the coordinator of the research projects SIDES 3.0 which ended successfully in June 2021.
A continuation of the project, SIDES LAB, is planned to start in June 2022, under the leadership of Franck Ramus, Laboratoire de Sciences Cognitives et Psycholinguistique, ENS, Paris.
8.2.2 ANR E-ROMA
Participants: Stefanie Hahmann, Rémi Ronfard, Nachwa Aboubakr, Mélina Skouras.
E-ROMA is a collaborative project funded by ANR with LIRIS (Raphaëlle Chaine, Julie Digne): Musée Gallo-Romain (Hugues Savay-Guerraz, Maria-Pia Darblade-Audoin) and Université Paris-Sorbonne (Emmanuelle Rosso).
The project was funding the PhD thesis of Pierre Casati, which was interrupted in June 2020.
Nachwa Aboubakr (PhD in June 2020) started a one-year postdoc contract in September 2020 and we are investigating new research directions with her, including end-to-end reconstruction of 3D scenes from bas reliefs and animation of the reconstructed 3D scenes using deep learning methods.
8.2.3 ANR ANATOMY2020
Participants: Olivier Palombi, Rémi Ronfard, Ameya Murukutla.
ANATOMY2020 was a collaborative projet funded by ANR with TIMC, Anatoscope and EHCI/LIG. The project was completed successfully in October 2021.
The project was funding the PhD thesis of Ameya Murukutla (May 2021) : Text-to-movie authoring of anatomy lessons.
During this PhD we extended our « prose storyboard language » with Petri net semantics and used it to generate short animated "movie lessons" useful for teaching anatomy to medical students. This opens promising new directions for future work to investigate the applicability of the method for authoring interactive games (exercises, quizzes, tests, exams) and virtual reality « experiences » in AR / VR.
8.2.4 MONTAGE SPATIAL
Participants: Rémi Ronfard.
Montage Spatial was a contract between ANIMA and ENSADLAB funded by the Ministery of Culture and Communication for reviewing the state of the art in computational directing tools for virtual reality, focusing on the concept of montage.
The project was successfully concluded in June 2021, with the publication of three reports : (i) a state of the art report (STAR) presented at the Eurographics conference 8; (ii) a critical discussion of the state of the art with respect to the new application domain of immersive, 6 degree-of-freedom virtual reality; and a proposal for future directions that need to be investigated for directing movies in virtual reality.
Participants in the project included Rémi Ronfard, external collaborator Francois Garnier (ENSADLAB) and SACRE-PSL PhD student Rémi Sagot-Duvauroux, who also built a working implementation of a virtual "spatial montage" table at ENSADLAB during the project.
We are preparing an INRIA exploratory action to continue and amplify this line of research in collaboration with ENSADLAB on the more general topic of spatial cinema, including spatial decoupage, mise-en-scène and montage.
8.2.5 LIVING ARCHIVE OF ISADORA
Participants: Manon Vialle, Rémi Ronfard, Mélina Skouras.
The Living Archive of Isadora is a collaboration between artist Elisabeth Rémy-Schwartz, Inria project team EXSITU in Saclay and ANIMA. The project was funded by Centre National de la Danse in 2020 and allowed us to acquire a unique motion capture dataset of Isadora Duncan choreographies perfomed by Elizabeth Schwartz.
ANIMA PhD Student Manon Vialle is now using this dataset to create interactive dance lessons in augmented reality. The lessons were initially tested with professional dancers during four workshops at Centre National de la Danse in Pantin between February and July 2021.
This work was presented to a professional audience at the CND in Lyon in October 2021.
8.3 Regional initiatives
8.3.1 PERFORMANCE LAB
Participants: Rémi Ronfard.
ANIMA was a member of the cross-disciplinary project (CDP) Performance Lab (2018-2021) which was funded by IDEX UGA. An active collaboration was established with the Litt&Arts team at Univ. Grenoble Alpes on the topic of "digital dramaturgies", with several internships shared between the two laboratories and co-directed by Rémi Ronfard and Julie Valero at Litt&Arts 31, 12, 27. This activity was helpful in shaping the new research agenda of the ANIMA team, towards authoring and directing virtual story worlds. The project was completed successfully in December 2021.
Participant: Stéfanie Hahmann, Olivier Palombi, Rémi Ronfard, Mélina Skouras.
9.1 Promoting scientific activities
9.1.1 Scientific events: organisation
Rémi Ronfard is a member of the steering committee for the intelligent cinematography and editing workshop (WICED).
Mélina Skouras co-organizes (with Céline Coutrix) the first French Human Computer Interaction - Computer Graphics - Virtual Reality (HCI-CG-VR) Seminar whose annual theme is the fabrication of tangible artifacts, June 7, 2021, and co-organizes (with Benoit Roman, Kostas Danas and Florence Bertails-Descoubes) the workshop "From Computational Fabrication to Material Design" of the GDR MePhy, June 22, 2021.
General chair, scientific chair
- Stefanie Hahmann was general co-chair of the international conference Shape Modeling International (SMI 2021), November 14-16, 2021.
9.1.2 Scientific events: selection
Chair of conference program committees
- Mélina Skouras is the chair of the posters program at ACM SIGGRAPH.
Member of the conference program committees
- Rémi Ronfard is a program committee member for the Motion in Game (MIG), and interactive storytelling (ICIDS) conferences.
- Mélina Skouras is a program committee member for the ACM SIGGRAPH, Eurographics, Pacific Graphics and Shape Modeling International (SMI) conferences.
- Stefanie Hahmann was a program committee member of the four conferences SGP21, SPM21, SMI21, SIAM GD21.
- Rémi Ronfard is a reviewer for the SIGGRAPH and SIGGRAPH ASIA conferences.
- Mélina Skouras is a reviewer for the SIGGRAPH ASIA, CHI and Shape Modeling International (SMI) conferences.
Member of the editorial boards
- Rémi Ronfard is an associate editor for the Computer Animation and Virtual Worlds journal.
- Stefanie Hahmann is an Associate Editor of CAG (Computers and Graphics, Elsevier) and CAD (Computer Aided Design, Elsevier).
Reviewer - reviewing activities
- Rémi Ronfard is a reviewer for the ACM Transactions on Graphics and IEEE Transactions on Visualization and Computer Graphics journals.
- Mélina Skouras is a reviewer for the ACM Transactions on Graphics and Computers & Graphics journals.
- Stefanie Hahmann was a reviewer for IEEE Computer Graphics & Applications, Computer Graphics Forum, and Computers & Graphics.
9.1.4 Invited talks
- Mélina Skouras gives keynote talks at the Shape Modeling International (SMI) conference (2021-11-15) and at the MePhy Workshop "From Computational Fabrication to Material Design" of the GDR MePhy (2021-06-22) and invited talks at the Toronto Geometry Colloquium (2021-04-28) and the MePhy Workshop "Elasticity + Geometry" (2021-03-24).
- Stefanie Hahmann was invited to participate in the Dagstuhl Seminar on Geometric Modeling, November 21-26, 2021. She gave an invited talk on Geometric construction and fabrication of ramdom porous metamaterials. LZI Schloß Dagstuhl, Allemagne.
- Stefanie Hahmann gave an invited talk at the TECHNION CGGC seminar, Israel, March 2021.
9.1.5 Leadership within the scientific community
- Stefanie Hahmann is Workpackage Leader and Principal Investigator for INRIA in the European FET OPEN Horizon 2020 project ADAM2 (Analysis, Design and Manufacturing of Microstructures, contract no. 862025).
- Stefanie Hahmann is member of the SMI (Shape Modeling International Association) steering committee.
- Olivier Palombi is the national director for heath at the Digital University in Health and Sport (Uness.fr).
- Olivier Palombi is a member of the board of directors of the International Conference of Deans of Medicine and Faculties of French Expression (CIDMEF).
9.1.6 Scientific expertise
- Stefanie Hahmann was an expert for US-Isreal bi-national science foundadtion.
- Stefanie Hahmann was an expert for the Einstein Stiftung Berlin (Germany).
9.1.7 Research administration
- Mélina Skouras is a member of the Conseil de Laboratoire of the Laboratoire Jean Kuntzmann, and of the Comité des Emplois Scientifiques (CES) at Inria Grenoble Rhône-Alpes.
- Stefanie Hahmann is an elected member of the European Association for Computer Graphics – chapitre français (EGFR) and serves as secretary in the streering committee.
- Stefanie Hahmann is an elected member of the council of Association Française d’Informatique Graphique (AFIG).
- Stefanie Hahmann is an elected member of the Conseil Scientifique of Grenoble INP.
- Stefanie Hahmann is a member of the Comité d’Etudes Doctorales (CED) at Inria Grenoble.
- Stefanie Hahmann is Responsable Scientifique (Maths-Info) at the Grenoble Doctorate School MSTII.
9.2 Teaching - Supervision - Juries
- Master: Mélina Skouras, Surface modeling, 13.5 HETD, M2, Ensimag-Grenoble INP.
- Master: Mélina Skouras, Computer Graphics II, 18 HETD, M2, Ensimag-Grenoble INP (MoSIG).
- In 2021 Stefanie Hahmann had a teaching load of 209h HETD. She is responsible of 3 classes at Ensimag-Grenoble INP: Numerical Methods (240 students, 3rd year Bachelor level, 42h), Geometric Modeling (70 students, Master 1st year, 58h) and Surface Modeling (45 students, Master 2nd year, 52h). She was president of the jury over 30 Masters (PFE) thesis defences.
- Stefanie Hahmann is co-responsible of the department MMIS (Images and Applied Maths) at Grenoble INP with 120 students (level M1, M2).
Rémi Ronfard and Olivier Palombi co-supervised the PhD thesis of Ameya Murukutla, which was defended in Grenoble in December 2021.
Rémi Ronfard and Mélina Skouras are co-supervising the PhD thesis of Manon Vialle, which was started in October 2020.
Rémi Ronfard and Olivier Palombi are co-supervising the PhD thesis of Sandrine Barbois, which was started in April 2020.
Rémi Ronfard co-supervises (with Francois Garnier at ENSADLAB and Guillaume Souluez at Univ. Paris 3) the PhD thesis of Rémi Sagot-Duvauroux, which was started in October 2020, as part of the SACRE PHD program at University PSL in Paris.
Mélina Skouras co-supervised (with Florence Bertails-Descoubes) the PhD thesis of Mickaël Ly, which was started in October 2017 and which was defended in September 2021.
Mélina Skouras co-supervises (with Adrien Bousseau) the PhD thesis of David Jourdan, which was started in October 2018.
Mélina Skouras co-supervises (with Florence Bertails-Descoubes and Thibaut Métivet) the PhD thesis of Nicolas Parent, which was started in October 2020.
Mélina Skouras co-supervises (with Florence Bertails-Descoubes and Thibaut Métivet) the PhD thesis of Alexandre Teixeira da Silva, which was started in February 2021.
Stefanie Hahmann, Mélina Skouras and Georges-Pierre Bonneau co-supervise the PhD thesis of Emmanuel Rodriguez, which was started in October 2020.
Rémi Ronfard supervised two Master's theses in 2021:
- Yidi Zhu, M2 MOSIG and ENSIMAG from February to June 2021, Directing virtual stage performances using voice and gesture;
- Karthik Chakka, M2 MOSIG, from February to June 2021, Sketching cinematography: from storyboard to layout animation.
Rémi Ronfard also supervised two Master's internships
- Camelia Guerraoui, M1 INSA Lyon from June to August 2021, Blocking Notation, a tool for annotating and directing theater;
- Valentin Gomelet-Richard, ESRA Bretagne, Reproduction de deux scènes de films en animation 3D dans le moteur de jeu UNITY 3D.
Stefanie Hahmann supervised a Master internship of Hugo Kraft, UGA, from Februrary to June 2021 on Polycube mesh parameterization in Virtual Reality sketching.
- Mélina Skouras was a member of the Jury for the Concours CRCN/ISFP of Inria Rennes - Bretagne Altantique.
- Stefanie Hahmann was a member of the Comité de Sélection PR 27 at University of Burgundy, and Comité de Sélection MC 27 at INSA Lyon.
- Stefanie Hahmann was PhD. jury member of Camile Brunel (Univerité de Bordeaux, INRIA) and PhD. reviewer of Tong Fu (Université Claude Bernard, Lyon) and Manon Jubert (Université Aix-Marseille, ENSAM).
Rémi Ronfard presented his research project « Archive interactive Isadora Duncan » with Élisabeth Schwartz and Sarah Fdili Alaoui at Centre National de la Danse in Pantin, as part of the "Exposés de recherche et de notation en ligne", on January 20, 2021.
Rémi Ronfard and Manon Vialle presented their research work at Centre National de la Danse in Lyon, during a four-hour workshop entitled "Conversation art et science autour du mouvement d’Isadora Duncan" on October 9, 2021.
Olivier Palombi was a speaker at the symposium "Centre de Soutien aux Innovations Pédagogiques I-Site MUSE : Apprendre de l’Avenir" in Montpellier in October 2021 ; and the seminar on "Intelligence artificielle (IA) en santé", in December 2021.
10 Scientific production
10.1 Major publications
- 1 inproceedingsRecognition of Laban Effort Qualities from Hand Motion.MOCO'20 - 7th International Conference on Movement and ComputingArticle No.: 8Jersey City/ Virtual, United StatesJuly 2020, 1-8
- 2 articleAs-Stiff-As-Needed Surface Deformation Combining ARAP Energy with an Anisotropic Material.Computer-Aided Design121April 2020, 1-15
- 3 articleKinoAI and the audiovisual notebook : a plural solution for rehearsal studies.European Journal of Theatre and Performance2June 2020, 334-375
10.2 Publications of the year
International peer-reviewed conferences
Conferences without proceedings
10.3 Cited publications
- 11 articleA Generative Audio-Visual Prosodic Model for Virtual Actors.IEEE Computer Graphics and Applications376November 2017, 40-51
- 12 inproceedingsRéalité augmentée sur scène.Journées d'Informatique ThéâtralePerformance Lab, Univ. Grenoble AlpesGrenoble, FranceFebruary 2020
- 13 articleDepicting as a method of communication.Psychological Review1232016, 324–347
- 14 articleAttention and the Evolution of Hollywood Film.Psychological Science2132010, 432-439
- 15 inproceedingsdAIrector: Automatic Story Beat Generation through Knowledge Synthesis.Proceedings of the Joint Workshop on Intelligent Narrative Technologies and Workshop on Intelligent Cinematography and Editing2018
- 16 inproceedingsCamera-on-rails: Automated Computation of Constrained Camera Paths.ACM SIGGRAPH Conference on Motion in GamesParis, FranceACMNovember 2015, 151-157
- 17 inproceedingsContinuity Editing for 3D Animation.AAAI Conference on Artificial IntelligenceAustin, Texas, United StatesAAAI PressJanuary 2015, 753-761
- 18 inproceedingsMulti-Clip Video Editing from a Single Viewpoint.European Conference on Visual Media ProductionLondon, United KingdomACMNovember 2014
- 19 inproceedingsSpatial Motion Doodles: Sketching Animation in VR Using Hand Gestures and Laban Motion Analysis.Motion, Interaction and GamesOctober 2019
- 20 inproceedingsHolojam in Wonderland: Immersive Mixed Reality Theater.ACM SIGGRAPH 2018 Art GallerySIGGRAPH '182018
- 21 inproceedingsImprovisational puppets, actors, and avatars.Proc Comp Game Dev Conf1996
- 22 inproceedingsCave: Making Collective Virtual Narrative.ACM SIGGRAPH 2019 Art GallerySIGGRAPH '192019
- 23 articleFunctionality Preserving Shape Style Transfer.ACM Transactions on Graphics3562016
- 24 articleAn empirical study of cognition and theatrical improvisation.Creativity and Cognition2009
- 25 bookHamlet on the Holodeck, The Future of Narrative in Cyberspace, Updated Edition.MIT Press2017
- 26 inproceedingsMy Corporis Fabrica: A Unified Ontological, Geometrical and Mechanical View of Human Anatomy.Proceedings of the 2009 International Conference on Modelling the Physiological Human3DPH'092009, 209--219
- 27 inproceedingsEtude des répétitions 2.0 : KINOAI, logiciel web.Journées d'Informatique ThéâtralePerformance Lab, Univ. Grenoble AlpesGrenoble, FranceFebruary 2020
- 28 articleThe Prose Storyboard Language: A Tool for Annotating and Directing Movies.CoRRarXiv:1508.07593v32019
- 29 inproceedingsSeeing Behind the Camera: Identifying the Authorship of a Photograph.2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)2016, 3494-3502
- 30 articleCombining 3D Models and Functions through Ontologies to Describe Man-made Products and Virtual Humans: Toward a Common Framework.Computer-Aided Design and Applications1222015, 166-180
- 31 inproceedings Pourquoi l'informatique théâtrale ? Journées d'Informatique Théâtrale Performance Lab, Univ. Grenoble Alpes Grenoble, France February 2020
- 32 articleThespian : An architecture for interactive pedagogical drama.Artificial Intelligence in Education1252005, 595–602