Team, Visitors, External Collaborators
Overall Objectives
Research Program
Application Domains
Highlights of the Year
New Software and Platforms
New Results
Bilateral Contracts and Grants with Industry
Partnerships and Cooperations
Dissemination
Bibliography
XML PDF e-pub
PDF e-Pub


Section: New Results

Physics-based Deep Neural Network for Augmented Reality

Participants : Jean-Nicolas Brunet, Andrea Mendizabal, Antoine Petit, Nicolas Golse, Eric Vibert, St├ęphane Cotin.

Figure 5. The U-Mesh framework allows for extremely fast simulations of soft tissues accounting for large non linear deformations.
IMG/GraphicalAbstractMICCAI2019.png

We propose an approach combining a finite element method and a deep neural network to learn complex elastic deformations with the objective of providing augmented reality during hepatic surgery. Derived from the U-Net architecture, our network is built entirely from physics-based simulations of a preoperative segmentation of the organ (see figure 5). These simulations are performed using an immersed-boundary method, which offers several numerical and practical benefits, such as not requiring boundary-conforming volume elements. We perform a quantitative assessment of the method using synthetic and ex vivo patient data. Results show that the network is capable of solving the deformed state of the organ using only a sparse partial surface displacement data and achieve similar accuracy as a FEM solution, while being about 100x faster. When applied to an ex vivo liver example, we achieve the registration in only 3ms with a mean target registration error (TRE) of 2.9mm. This results were presented at MICCAI 2019 [22].