Team clime

Overall Objectives
Scientific Foundations
Application Domains
New Results
Contracts and Grants with Industry
Other Grants and Activities

Section: New Results

Image assimilation

Sequences of images display structures evolving in time. This information is recognized of major interest by, for instance, meteorological forecasters. However, the satellite acquisitions are mostly assimilated in geophysical models on a point-wise basis, discarding the space-time coherence visualized by the evolution of structures. Assimilating images is then becoming of major interest and the problem should be considered in two ways:

In both cases, image information is assimilated within models, raising a number of theoretical and experimental questions.

Impact of the regularization term

Participants : Etienne Huot, Isabelle Herlin.

The objective is to infer the dynamics from a sequence of satellite images. The application concerns the estimation of surface velocity from Sea Surface Temperature (SST) acquisitions. We define an Image Model (IM ) describing the evolution of the surface temperature and velocity. SST observations are then assimilated in the IM by minimizing a cost function, including the measure of the discrepancy between observations and simulations and a regularization term. Two regularization constraints have been compared and tested (see Figure 7 ): (i ) the smoothness constraint, based on the gradients of the velocity components, and (ii ) the second-order div-curl constraint, based on gradients of irrotational and vorticity components. For quantitative evaluation, synthetic data (computed by an ocean simulation model developped in the MOISE project-team during the ADDISA ANR project) are used. In this context, the second-order div-curl regularization is better adapted (4.5% improvement of results compared to the smoothness constraint).

Figure 7. Top. Three simulations of SST data. Middle. Left: velocity computed by the model (ground thruth). Center: with smoothness constraint. Right: second-order div-curl regularization. Bottom. Same fields displayed using an HSV representation.

Requirements on the evolution equation

Participants : Etienne Huot, Isabelle Herlin, Gennady Korotaev [ Marine Hydrophysical Institute, Ukraine ] .

In the context of ocean surface velocity estimation, an Image Model (IM ) is used to express the evolution of the temperature and the dynamics of the velocity. Sea Surface Temperature acquisitions are assimilated in this IM , to drive pseudo-observations of velocity, which are further assimilated in the oceanic forecasting model.

Two Image Models have been proposed. The Simple Image Model (SIM ) is based on a simplification of the advection-diffusion equation governing the transport of temperature and on the stationarity hypothesis of the velocity field, i.e. , it considers that the surface velocity varies much slower than the temperature. Even if this heuristic is often verified, the main drawback is its lack of physical origin to express the dynamics. Hence, an Extended Image Model is defined using the same evolution equation for temperature and modeling the velocity through a shallow-water approximation: the evolution of the two components of velocity are linked by the water layer thickness. Results are then compared using first synthetic data, demonstrating the quantitative improvement obtained with the EIM (see Figure 8 ).

Figure 8. Top: Three simulations of SST data. Middle: ground truth (left), compared to the motion estimated with SIM (center) and with EIM (right). Bottom: same fields displayed using an HSV representation.

Solving ill-posed image processing problem using data assimilation

Participants : Dominique Béréziat [ UPMC / LIP6 ] , Isabelle Herlin, Nicolas Mercier, Jean-Paul Berroir.

Most image processing problems are ill-posed in the sense that the image equation , modeling the links between the image and the quantity (named the state vector) to be computed is not invertible. A unique solution can however be obtained using a Tikhonov regularization technique. If an evolution equation , describing the dynamics of the state vector, is available, it becomes possible to obtain a unique solution, without any regularization, by integrating the evolution equation from the initial condition. Data assimilation offers the mathematical framework to solve simultaneously the image and the evolution equations. We proposed a method to transform, in a generic way, an ill-posed Image Processing problem into a 4D-var formulation. First, state and observation vectors have to be defined. Second, the evolution equation must be exhibited. For some applications, this equation is inferred using physical considerations. However, the dynamics is often unknown and generic models are considered, expressing a temporal regularity of the state vector. Third, model errors associated to the image and evolution equations must be defined. These errors are fully described by their covariance matrices and we studied some generic choices and their impacts on the result. Covariance matrices can also used to process noisy data by discarding the contribution of observations in the computation of the state vector (see Figure 9 ). Last, an initial condition should be provided. It can be obtained using the traditional approaches: with the image equation and the Tikhonov regularization.

For allowing a generic transformation of an ill-posed image processing problem into a 4D-var formulation, the evolution and observation models are expressed as two operators involved in the evolution and observation equations. These models are discretized and Automatic Differentiation (AD) tools are then used to compute the discretized differentials and adjoints. This enhances the generic aspect of the 4D-var formulation as only observation and evolution models need to be implemented. Moreover, as complex evolution models can lead to unstable numerical schemes, an elegant solution to enhance stability is to split the evolution operator in several simple sub-operators: AD computes differential and adjoint sub-operators.

Figure 9. Example of optical flow determination in case of missing data. Left: Data Assimilation. Right: Spatial regularization.

Impact of covariances in 4D-var formulations

Participants : Dominique Béréziat [ UPMC / LIP6 ] , Isabelle Herlin, Sahar Syassi.

A general data assimilation algorithm solves, with respect to the state vector, three equations: an evolution equation, an observation equation and an initial condition. Each equation is weighted by a covariance matrix in the functional to be minimized in the variational formulation. The aim of data assimilation is to determine a solution which is a compromise between the observations and the evolution model, given the initial condition. If observations are noisy, they are discarded from the process by imposing high values of the observation error's covariance matrix. 

The situation is slightly different in image processing, due to the low confidence in the evolution equation: the image dynamics is usually unknown and only approximated. Consequently, the contribution of that equation in the determination of the state vector has to be lowered. Two problems are then arising.

First, it is no more possible to compute a solution from the observation equation as it is generally ill-posed. The solution is then to add a regularization term, expressed within the observation equation.

Second, the evolution equation errors must be located in time-space. This is achieved by measuring the discrepancy between a solution computed by the evolution equation and a solution computed by the observation equation including the regularization term. This distance is used to specify the covariance associated to the evolution equation error.

Determining optical flow with large displacement

Participants : Dominique Béréziat [ UPMC / LIP6 ] , Isabelle Herlin.

This addresses video sequences for which the image dynamics is totally unknown. A velocity fields transport of velocity by itself is considered. It is well suited to impose a temporal regularity of the velocity fields. The standard OFCE (Optical Flow Constraint Equation), modeling the image brightness transport by velocity, is applied as observation equation. If large displacements, and therefore high velocities, occur, the OFCE is however no more valid: this PDE is only standing for infinitesimal displacements. The transport of image brightness I by velocity Im3 $\#119856 $ between two dates can however be expressed in the following form: Im4 ${I(\#119857 +\#119856 \#948 t,t+\#948 t)=I(\#119857 )}$ . This equation is non linear but differentiable. This property is sufficient to apply 4D-var as the algorithm does not need to inverse the observation equation to compute the solution. Successful tests have been performed on synthetic data and video sequences.

A posteriori guaranteed motion estimation

Participants : Sergey Zhuk, Isabelle Herlin.

In this study, we focus on the application of the minimax state estimation framework for the motion estimation from an image sequence, using the optical flow equation:

Im5 ${\mstyle \mfrac \#8706 {\#8706 t}I+{\#9001 v,\mstyle \mfrac \#8706 {\#8706 x}I\#9002 }+\#956 \#9653 I=f{(t,x)},I{(t_0,x)}=G{(x)},x\#8712 P}$

First this equation is rewritten as an observation equation:

y(t) = H(t)v(t) + f(t), t0$ \le$t$ \le$T(1)

where Im6 ${y{(t)}\#8614 \mstyle \mfrac \#8706 {\#8706 t}I{(t,x)}+\#956 \#9653 I{(t,x)}}$ , and Im7 ${H{(t)}v{(t)}\#8614 -\#9001 v{(t,x)},\mstyle \mfrac \#8706 {\#8706 x}I{(t,x)}\#9002 }$ and we consider the evolution equation:

Im8 ${\mstyle \mfrac d{dt}v{(t)}+L{(v)}=Bg{(t)},v{(t_0)}=v_0,t_0\#8804 t\#8804 T}$(2)

describing the dynamics of the motion field v(t, x) . The objective is to estimate v provided that:

Im9 ${G{(v_0,f,g)}{:=\#8741 }Q_0^\mfrac 12v_0{\#8741 }^2+\#8747 _{t_0}^T{\#8741 }R^\mfrac 12{(t)}f{(t)}{\#8741 }^2+{\#8741 Q^\mfrac 12{(t)}g{(t)}\#8741 }^2dt\#8804 1}$(3)

where Q0, Q, R are self-adjoint positive definite bounded linear operators with bounded inverses.

We are looking for the estimation among all functions solving (2 ) for some v0, g and verifying:

Im10 ${\#119985 {(t,v{(t)})}\#8804 1,t_0\#8804 t\#8804 T}$(4)

where Im11 $\#119985 $ denotes the value-function:

Im12 $\mtable{...}$(5)

and min is taken over all g(·) .

We investigate the conditions on the "shape" of the model operator L so that the value function Im11 $\#119985 $ verify in a weak sense some Hamilton-Jacobi-Bellman (HJB) equation. We study the problem of solution approximations of the resulting HJB by finite-dimensional HJBs.


Logo Inria