Section: New Results
Image assimilation
Sequences of images display structures evolving in time. This information is recognized of major interest by, for instance, meteorological forecasters. However, the satellite acquisitions are mostly assimilated in geophysical models on a pointwise basis, discarding the spacetime coherence visualized by the evolution of structures. Assimilating images is then becoming of major interest and the problem should be considered in two ways:

from the model's viewpoint, the problem is to control the location of structures using the observations,

from the image's viewpoint, a model of the dynamics and structures has to be built from the observations.
In both cases, image information is assimilated within models, raising a number of theoretical and experimental questions.
Impact of the regularization term
Participants : Etienne Huot, Isabelle Herlin.
The objective is to infer the dynamics from a sequence of satellite images. The application concerns the estimation of surface velocity from Sea Surface Temperature (SST) acquisitions. We define an Image Model (IM ) describing the evolution of the surface temperature and velocity. SST observations are then assimilated in the IM by minimizing a cost function, including the measure of the discrepancy between observations and simulations and a regularization term. Two regularization constraints have been compared and tested (see Figure 7 ): (i ) the smoothness constraint, based on the gradients of the velocity components, and (ii ) the secondorder divcurl constraint, based on gradients of irrotational and vorticity components. For quantitative evaluation, synthetic data (computed by an ocean simulation model developped in the MOISE projectteam during the ADDISA ANR project) are used. In this context, the secondorder divcurl regularization is better adapted (4.5% improvement of results compared to the smoothness constraint).
Requirements on the evolution equation
Participants : Etienne Huot, Isabelle Herlin, Gennady Korotaev [ Marine Hydrophysical Institute, Ukraine ] .
In the context of ocean surface velocity estimation, an Image Model (IM ) is used to express the evolution of the temperature and the dynamics of the velocity. Sea Surface Temperature acquisitions are assimilated in this IM , to drive pseudoobservations of velocity, which are further assimilated in the oceanic forecasting model.
Two Image Models have been proposed. The Simple Image Model (SIM ) is based on a simplification of the advectiondiffusion equation governing the transport of temperature and on the stationarity hypothesis of the velocity field, i.e. , it considers that the surface velocity varies much slower than the temperature. Even if this heuristic is often verified, the main drawback is its lack of physical origin to express the dynamics. Hence, an Extended Image Model is defined using the same evolution equation for temperature and modeling the velocity through a shallowwater approximation: the evolution of the two components of velocity are linked by the water layer thickness. Results are then compared using first synthetic data, demonstrating the quantitative improvement obtained with the EIM (see Figure 8 ).
Solving illposed image processing problem using data assimilation
Participants : Dominique Béréziat [ UPMC / LIP6 ] , Isabelle Herlin, Nicolas Mercier, JeanPaul Berroir.
Most image processing problems are illposed in the sense that the image equation , modeling the links between the image and the quantity (named the state vector) to be computed is not invertible. A unique solution can however be obtained using a Tikhonov regularization technique. If an evolution equation , describing the dynamics of the state vector, is available, it becomes possible to obtain a unique solution, without any regularization, by integrating the evolution equation from the initial condition. Data assimilation offers the mathematical framework to solve simultaneously the image and the evolution equations. We proposed a method to transform, in a generic way, an illposed Image Processing problem into a 4Dvar formulation. First, state and observation vectors have to be defined. Second, the evolution equation must be exhibited. For some applications, this equation is inferred using physical considerations. However, the dynamics is often unknown and generic models are considered, expressing a temporal regularity of the state vector. Third, model errors associated to the image and evolution equations must be defined. These errors are fully described by their covariance matrices and we studied some generic choices and their impacts on the result. Covariance matrices can also used to process noisy data by discarding the contribution of observations in the computation of the state vector (see Figure 9 ). Last, an initial condition should be provided. It can be obtained using the traditional approaches: with the image equation and the Tikhonov regularization.
For allowing a generic transformation of an illposed image processing problem into a 4Dvar formulation, the evolution and observation models are expressed as two operators involved in the evolution and observation equations. These models are discretized and Automatic Differentiation (AD) tools are then used to compute the discretized differentials and adjoints. This enhances the generic aspect of the 4Dvar formulation as only observation and evolution models need to be implemented. Moreover, as complex evolution models can lead to unstable numerical schemes, an elegant solution to enhance stability is to split the evolution operator in several simple suboperators: AD computes differential and adjoint suboperators.
Impact of covariances in 4Dvar formulations
Participants : Dominique Béréziat [ UPMC / LIP6 ] , Isabelle Herlin, Sahar Syassi.
A general data assimilation algorithm solves, with respect to the state vector, three equations: an evolution equation, an observation equation and an initial condition. Each equation is weighted by a covariance matrix in the functional to be minimized in the variational formulation. The aim of data assimilation is to determine a solution which is a compromise between the observations and the evolution model, given the initial condition. If observations are noisy, they are discarded from the process by imposing high values of the observation error's covariance matrix.
The situation is slightly different in image processing, due to the low confidence in the evolution equation: the image dynamics is usually unknown and only approximated. Consequently, the contribution of that equation in the determination of the state vector has to be lowered. Two problems are then arising.
First, it is no more possible to compute a solution from the observation equation as it is generally illposed. The solution is then to add a regularization term, expressed within the observation equation.
Second, the evolution equation errors must be located in timespace. This is achieved by measuring the discrepancy between a solution computed by the evolution equation and a solution computed by the observation equation including the regularization term. This distance is used to specify the covariance associated to the evolution equation error.
Determining optical flow with large displacement
Participants : Dominique Béréziat [ UPMC / LIP6 ] , Isabelle Herlin.
This addresses video sequences for which the image dynamics is totally unknown. A velocity fields transport of velocity by itself is considered. It is well suited to impose a temporal regularity of the velocity fields. The standard OFCE (Optical Flow Constraint Equation), modeling the image brightness transport by velocity, is applied as observation equation. If large displacements, and therefore high velocities, occur, the OFCE is however no more valid: this PDE is only standing for infinitesimal displacements. The transport of image brightness I by velocity between two dates can however be expressed in the following form: . This equation is non linear but differentiable. This property is sufficient to apply 4Dvar as the algorithm does not need to inverse the observation equation to compute the solution. Successful tests have been performed on synthetic data and video sequences.
A posteriori guaranteed motion estimation
Participants : Sergey Zhuk, Isabelle Herlin.
In this study, we focus on the application of the minimax state estimation framework for the motion estimation from an image sequence, using the optical flow equation:
First this equation is rewritten as an observation equation:
where , and and we consider the evolution equation:
describing the dynamics of the motion field v(t, x) . The objective is to estimate v provided that:
where Q_{0}, Q, R are selfadjoint positive definite bounded linear operators with bounded inverses.
We are looking for the estimation among all functions solving (2 ) for some v_{0}, g and verifying:
where denotes the valuefunction:
and min is taken over all g(·) .
We investigate the conditions on the "shape" of the model operator L so that the value function verify in a weak sense some HamiltonJacobiBellman (HJB) equation. We study the problem of solution approximations of the resulting HJB by finitedimensional HJBs.