# Keywords

- A6.2.1. Numerical analysis of PDE and ODE
- A6.2.6. Optimization
- A6.2.7. High performance computing
- A6.3.2. Data assimilation
- A6.4.1. Deterministic control
- A6.4.2. Stochastic control

- B4.4. Energy delivery
- B4.4.1. Smart grids
- B7.1.2. Road traffic
- B7.1.3. Air traffic
- B7.2.1. Smart vehicles

# 1 Team members, visitors, external collaborators

## Research Scientists

- Joseph Frédéric Bonnans [Team leader, Inria, Senior Researcher, HDR]
- Laurent Pfeiffer [Inria, Researcher]

## Post-Doctoral Fellow

- Davin Glen Lunz [Inria]

## PhD Students

- Guillaume Bonnet [Univ. Paris-Saclay]
- Pierre Lavigne [École polytechnique]
- Kang Liu [École polytechnique]

## Administrative Assistant

- Hanadi Dib [Inria]

# 2 Overall objectives

## 2.1 Scientific directions

Commands is a team devoted to dynamic optimization, both for deterministic and stochastic systems. This includes the following approaches: trajectory optimization, deterministic and stochastic optimal control, stochastic programming, dynamic programming and Hamilton-Jacobi-Bellman equation.

Our aim is to derive new and powerful algorithms for solving numerically these problems, with applications in several industrial fields. While the numerical aspects are the core of our approach it happens that the study of convergence of these algorithms and the verification of their well-posedness and accuracy raises interesting and difficult theoretical questions, such as, for trajectory optimization: qualification conditions and second-order optimality condition, well-posedness of the shooting algorithm, estimates for discretization errors; for the Hamilton-Jacobi-Bellman approach: accuracy estimates, strong uniqueness principles when state constraints are present, for stochastic programming problems: sensitivity analysis.

## 2.2 Industrial impact

For many years the team members have been deeply involved in various industrial applications, often in the framework of PhD theses. The Commands team itself has dealt since its foundation in 2009 with several types of applications:

- Space vehicle trajectories, in collaboration with CNES, the French space agency.
- Aeronautics, in collaboration with the startup Safety Line.
- Production, management, storage and trading of energy resources, in collaboration with Edf, ex-Gdf and Total.
- Energy management for hybrid vehicles, in collaboration with Renault and Ifpen.

We give more details in the Bilateral contracts section.

# 3 Research program

## 3.1 Historical aspects

The roots of deterministic optimal control are the “classical” theory of the calculus of variations, illustrated by the work of Newton, Bernoulli, Euler, and Lagrange (whose famous multipliers were introduced in 38), with improvements due to the “Chicago school”, Bliss 30 during the first part of the 20th century, and by the notion of relaxed problem and generalized solution (Young 43).

Trajectory optimization really started with the spectacular achievement done by Pontryagin's group 42 during the fifties, by stating, for general optimal control problems, nonlocal optimality conditions generalizing those of Weierstrass. This motivated the application to many industrial problems (see the classical books by Bryson and Ho 34, Leitmann 41, Lee and Markus 40, Ioffe and Tihomirov 37).

Dynamic programming was introduced and systematically studied by R. Bellman during the fifties. The HJB equation, whose solution is the value function of the (parameterized) optimal control problem, is a variant of the classical Hamilton-Jacobi equation of mechanics for the case of dynamics parameterized by a control variable. It may be viewed as a differential form of the dynamic programming principle. This nonlinear first-order PDE appears to be well-posed in the framework of viscosity solutions introduced by Crandall and Lions 35. The theoretical contributions in this direction did not cease growing, see the books by Barles 29 and Bardi and Capuzzo-Dolcetta 28.

## 3.2 Trajectory optimization

The so-called direct methods consist in an optimization of the trajectory, after having discretized time, by a nonlinear programming solver that possibly takes into account the dynamic structure. So the two main problems are the choice of the discretization and the nonlinear programming algorithm. A third problem is the possibility of refinement of the discretization once after solving on a coarser grid.

In the full discretization approach, general Runge-Kutta schemes with different values of control for each inner step are used. This allows to obtain and control high orders of precision, see Hager 36, Bonnans 31. In the indirect approach, the control is eliminated thanks to Pontryagin's maximum principle. One has then to solve the two-points boundary value problem (with differential variables state and costate) by a single or multiple shooting method. The questions are here the choice of a discretization scheme for the integration of the boundary value problem, of a (possibly globalized) Newton type algorithm for solving the resulting finite dimensional problem in ${\mathbb{R}}^{n}$ ($n$ is the number of state variables), and a methodology for finding an initial point.

## 3.3 Hamilton-Jacobi-Bellman approach

This approach consists in calculating the value function associated with the optimal control problem, and then synthesizing the feedback control and the optimal trajectory using Pontryagin's principle. The method has the great particular advantage of reaching directly the global optimum, which can be very interesting when the problem is not convex.

Optimal stochastic control problems occur when the dynamical system is uncertain. A decision typically has to be taken at each time, while realizations of future events are unknown (but some information is given on their distribution of probabilities). In particular, problems of economic nature deal with large uncertainties (on prices, production and demand). Specific examples are the portfolio selection problems in a market with risky and non-risky assets, super-replication with uncertain volatility, management of power resources (dams, gas). Air traffic control is another example of such problems.

For solving stochastic control problems, we studied the so-called Generalized Finite Differences (GFD), that allow to choose at any node, the stencil approximating the diffusion matrix up to a certain threshold 33. Determining the stencil and the associated coefficients boils down to a quadratic program to be solved at each point of the grid, and for each control. This is definitely expensive, with the exception of special structures where the coefficients can be computed at low cost. For two dimensional systems, we designed a (very) fast algorithm for computing the coefficients of the GFD scheme, based on the Stern-Brocot tree 32.

# 4 Application domains

## 4.1 Electric energy production

Our current research on aggregative optimization and mean-field games has a natural application to the problem of distributed production of electric energy. We are looking forwaard on developments in that direction.

## 4.2 Energy management for hybrid vehicles

In collaboration with Ifpen and in the framework of A. Le Rhun's thesis, we have developped a methodology for the optimal energy management for hybrid vehicles, based on a statistical analysis of the traffic. See 17, 17, 39.

## 4.3 Biological cells culture

In collaboration with the Inbio team (Inst. Pasteur and Inria) we started to study the optimization of protein production based on cell culture.

# 5 Social and environmental responsibility

## 5.1 Impact of research results

Our research on large scale aggregative optimization and mean field games have natural applications in the domain of energy production under environmental constraints. So we hope to develop in the next years numerical tools for the reduction of the impact of energy production.

# 6 Highlights of the year

Significant progress has been achieved in the three following research areas:

- Mean-field games and control.Submission of two research articles. Recruitement of Kang Liu as a PhD student, who works on mean-field control and applications in flexibility management.
- Biology. We continued our collaboration with the Inbio team (Inst. Pasteur and Inria-Paris) on the modelization of mRNA-based protein production.
- Numerical schemes. We submitted several papers on the application of the Selling decomposition (a monotone decomposition of diffusion matrices) for the resolution of variational problems.

# 7 New results

## 7.1 Optimal control of partial differential equations

#### State constraints

In 7 we consider an optimal control problem governed by a semilinear heat equation with bilinear control-state terms and subject to control and state constraints. The state constraints are of integral type, the integral being with respect to the space variable. The control is multidimensional. The cost functional is of a tracking type and contains a linear term in the control variables. We derive second order necessary conditions relying on the concept of alternative costates and quasi-radial critical directions. The appendix provides an example illustrating the applicability of our results.

In 8 we continue the previous work by deriving second order sufficient conditions relying on the Goh transform. The appendix provides an example illustrating the applicability of our results.

#### Receding Horizon control

In the articles 13 and 12, the efficiency of the Receding Horizon method is investigated for two classes of optimal control problems involving PDEs. An explicit rate of convergence is obtained. In the first article, nonlinear stabilization problems are considered. The method takes advantage of computable Taylor expansions of the value function (around some equilibrium point). Linear-quadratic problems are investigated in the second reference. The method takes advantage of the so-called turnpike property, satisfied under a stabilizability and an observability condition. The two works have been written by L. Pfeiffer during his former position at the University of Graz.

## 7.2 Stochastic control and HJB equations

In 27, we consider the framework of high dimensional stochastic control problem, in which the controls are aggregated in the cost function. As first contribution we introduce a modified problem, whose optimal control is under some reasonable assumptions an ε-optimal solution of the original problem. As second contribution, we present a decentralized algorithm whose convergence to the solution of the modified problem is established. Finally, we study the application to a problem of coordination of energy consumption and production of domestic appliances.

In 16, optimality conditions in the form of a variational inequality are proved for a class of constrained optimal control problems of stochastic differential equations. The cost function and the inequality constraints are functions of the probability distribution of the state variable at the final time. The analysis uses in an essential manner a convexity property of the set of reachable probability distributions. An augmented Lagrangian method based on the obtained optimality conditions is proposed and analyzed for solving iteratively the problem. At each iteration of the method, a standard stochastic optimal control problem is solved by dynamic programming. Two academical examples are investigated.

In 26, we consider a continuous time stochastic optimal control problem under both equality and inequality constraints on the expectation of some functionals of the controlled process. Under a qualification condition, we show that the problem is in duality with an optimization problem involving the Lagrange multiplier associated with the constraints. Then by convex analysis techniques, we provide a general existence result and some a priori estimation of the dual optimizers. We further provide a necessary and sufficient optimality condition for the initial constrained control problem. The same results are also obtained for a discrete time constrained control problem. Moreover, under additional regularity conditions, it is proved that the discrete time control problem converges to the continuous time problem, possibly with a convergence rate. This convergence result can be used to obtain numerical algorithms to approximate the continuous time control problem, which we illustrate by two simple numerical examples.

## 7.3 Numerical analysis of variational problems

In 19 we introduce a new strategy for the design of second-order accurate discretiza-tions of non-linear second order operators of Bellman type, which preserves degenerate ellipticity. The approach relies on Selling's formula, a tool from lattice geometry, and is applied to the Pucci and Monge-Ampere equations, discretized on a two dimensional cartesian grid. In the case of the Monge-Ampere equation, our work is related to both the stable formulation and the second order accurate scheme. Numerical experiments illustrate the robustness and the accuracy of the method.

In 20, we design adaptive finite differences discretizations, which are degenerate elliptic and second order consistent, of linear and quasi-linear partial differential operators featuring both a first order term and an anisotropic second order term. Our approach requires the domain to be discretized on a Cartesian grid, and takes advantage of techniques from the field of low-dimensional lattice geometry. We prove that the stencil of our numerical scheme is optimally compact, in dimension two, and that our approach is quasi-optimal in terms of the compatibility condition required of the first and second order operators, in dimension two and three. Numerical experiments illustrate the efficiency of our method in several contexts.

In 18, using an extension of Varadhan's formula to Randers manifolds, we notice that Randers distances may be approximated by a logarithmic transformation of a linear second-order partial differential equation. Following an idea introduced by Crane, Weischedel, and Wardetzky in the case of Riemannian distances, we study a numerical method for approximating Randers distances which involves a discretization of this linear equation. We propose to use Selling's formula, which originates from the theory of low-dimensional lattice geometry, to build a monotone and linear finite-difference scheme. By injecting the logarithmic transformation in this linear scheme, we are able to prove convergence of this numerical method to the Randers distance, as well as consistency to the order two thirds far from the boundary of the considered domain. We explain how this method may be used to approximate optimal transport distances, how has been previously done in the Riemannian case.

## 7.4 Mean-field games

In 21, we propose and investigate a discrete-time mean field game model involving risk-averse agents. The model under study is a coupled system of dynamic programming equations with a Kolmogorov equation. The agents' risk aversion is modeled by composite risk measures. The existence of a solution to the coupled system is obtained with a fixed point approach. The corresponding feedback control allows to construct an approximate Nash equilibrium for a related dynamic game with finitely many players.

In 22, we analyze a system of partial differential equations that model a potential mean field game of controls, briefly MFGC. Such a game describes the interaction of infinitely many negligible players competing to optimize a personal value function that depends in aggregate on the state and, most notably, control choice of all other players. A solution of the system corresponds to a Nash Equilibrium, a group optimal strategy for which no one player can improve by altering only their own action. We investigate the second order, possibly degenerate, case with non-strictly elliptic diffusion operator and local coupling function. The main result exploits potentiality to employ variational techniques to provide a unique weak solution to the system, with additional space and time regularity results under additional assumptions. New analytical subtleties occur in obtaining a priori estimates with the introduction of an additional coupling that depends on the state distribution as well as feedback.

## 7.5 Biology, medecine

In 15 we discuss discrete-state continuous-time Markov processes, an important class of models employed broadly across the sciences. When the system size becomes large, standard approaches can become intractable to exact solution and numerical simulation. Approximations posed on a continuous state space are often more tractable and are presumed to converge in the limit as the system size tends to infinity. For example, an expansion of the master equation truncated at second order yields the Fokker–Planck equation, a widely used continuum approximation equipped with an underlying process of continuous state. Surprisingly, in [Doering et. al. Multiscale Model. Sim. 2005 3:2, p.283–299] it is shown that the Fokker–Planck approximation may exhibit exponentially large errors, even in the infinite system-size limit. Crucially, the source of this inaccuracy has not been addressed. In this paper, we focus on the family of continuous-state approximations obtained by arbitrary-order truncations. We uncover how the exponentially large error stems from the truncation by quantifying the rapid error decay with increasing truncation order. Furthermore, we explain why this discrepancy only comes to light in a subset of problems. The approximations produced by finite truncation beyond second order lack underlying stochastic processes. Nevertheless, they retain valuable information that explains the previously observed discrepancy by bridging the gap between the continuous and discrete processes. The insight conferred by this broader notion of “continuum approximation”, where we do not require an underlying stochastic process, prompts us to revisit previously expressed doubts regarding continuum approximations. In establishing the utility of higher-order truncations, this approach also contributes to the extensive discussion in the literature regarding the second-order truncation: while recognising the appealing features of an associated stochastic process, in certain cases it may be advantageous to dispense of the process in exchange for the increased approximation accuracy guaranteed by higher-order truncations.

In 25, we discuss the chemical master equation and its continuum approximations, are indispensable tools in the modeling of chemical reaction networks. These are routinely used to capture complex nonlinear phenomena such as multimodality as well as transient events such as first-passage times, that accurately characterise a plethora of biological and chemical processes. However, some mechanisms, such as heterogeneous cellular growth or phenotypic selection at the population level, cannot be represented by the master equation and thus have been tackled separately. In this work, we propose a unifying framework that augments the chemical master equation to capture such auxiliary dynamics, and we develop and analyse a numerical solver that accurately simulates the system dynamics. We showcase these contributions by casting a diverse array of examples from the literature within this framework, and apply the solver to both match and extend previous studies. Analytical calculations performed for each example validate our numerical results and benchmark the solver implementation.

#### Optimal Control of an Age-Structured System with State Constraints

In 10 we study an optimal control problem with state constraints where the state is given by an age-structured, abstract parabolic differential equation. We prove the existence and uniqueness of solution for the state equation and provide first and second parabolic estimates. We analyze the differentiability of the cost function and, based on the general theory of Lagrange multipliers, we give a first order optimality condition. We also define and analyze the regularity of the costate. Finally, we present a pregnancy model, where two coupled age-structured equations are involved, and we apply the obtained results to this case.

In 11, we propose a model for the COVID-19 epidemic where the population is partitioned into classes corresponding to ages (that remain constant during the epidemic). The main feature is to take into account the infection age of the infected population. This allows to better simulate the infection propagation that crucially depend on the infection age. We discuss how to compute the coefficients from data available in the future, and introduce a confinement variable as control. The cost function is a compromise between confinement cost, hospitalization peak and the death toll. Our numerical experiments allow to evaluate the interest of confinement varying with age classes.

## 7.6 Other problems

In 9, we prove second-order necessary optimality conditions for the so-called time crisis problem that comes up within the context of viability theory. It consists in minimizing the time spent by solutions of a controlled dynamics outside a given subset $K$ of the state space. One essential feature is the discontinuity of the characteristic function involved in the cost functional. Thanks to a change of time and an augmentation of the dynamics, we relate the time crisis problem to an auxiliary Mayer control problem. This allows us to use the classical tools of optimal control for obtaining optimality conditions. Going back to the original problem, we deduce that way second order optimality conditions for the time crisis problem.

# 8 Partnerships and cooperations

## 8.1 National initiatives

### 8.1.1 IPL

#### Cosy

Inria Project Lab COSY (started in 2017) aims at exploiting the potential of state-of-art biological modelling, control techniques, synthetic biology and experimental equipment to achieve a paradigm shift in control of microbial communities. More precisely, we plan to determine and implement control strategies to make heterogeneous communities diversify and interact in the most profitable manner. Study of yeast cells has started in collaboration with team Lifeware (G. Batt) in the framework of the PhD of V. Andreani, and is pursued in the Postdoc of D. Lunz (started Nov. 2019).

# 9 Dissemination

## 9.1 Promoting scientific activities

### 9.1.1 Journal

#### Member of the editorial boards

F. Bonnans is cofounder and Associate Editor of Math. and Applications, Annals of AOSR (Academy of Science of Romania).

#### Reviewer - reviewing activities

Reviews for major journals in the field.

### 9.1.2 Leadership within the scientific community

F. Bonnans is member of the PGMO (Program Gaspard Monge in Optimization) board.

## 9.2 Teaching - Supervision - Juries

### 9.2.1 Teaching

2nd year of master / 3rd year of engineering school:

- F. Bonnans: Optimal control of partial differential equations, 20h, M2, Optimization Master, IPP and U. Paris-Saclay, France.
- F. Bonnans: Optimal control of ordinary differential equations, 15h, M2, Optimization master (IPP and U. Paris-Saclay) and Ensta-Paris.
- L. Pfeiffer: Optimal control of ordinary differential equations, 18h, M2, Optimization master (U. Paris-Saclay) and Ensta-Paris.
- L. Pfeiffer: Optimisation continue, 17h, Ensta-Paris.
- L. Pfeiffer: Résolution des problèmes d'optimisation discrète ou continue dans le domaine de l'énergie, 17h, Ensta-Paris.

### 9.2.2 Supervision

- PhD in progress: Guillaume Bonnet, Efficient schemes for the Hamilton-Jacobi-Bellman equation. Started Oct. 2018. F. Bonnans and J.-M. Mirebeau, LMO, U. Orsay.
- PhD in progress: Pierre Lavigne, Mathematical study of economic equilibria for renewable energy sources. Started Oct. 2018. F. Bonnans and L. Pfeiffer.
- PhD in progress: Kang Liu, Mean-field optimal control and applications to flexibilities management. Started Oct. 2020. L. Pfeiffer and F. Bonnans.

## 9.3 Popularization

### 9.3.1 Internal or external Inria responsibilities

L. Pfeiffer is correspondent for the hiring mission (mission recrutement).

# 10 Scientific production

## 10.1 Major publications

- 1 article 'Optimal control techniques based on infection age for the study of the COVID-19 epidemic'. Mathematical Modelling of Natural Phenomena 2021
- 2 article'Schauder Estimates for a Class of Potential Mean Field Games of Controls'.Applied Mathematics and OptimizationJuly 2019, 34
- 3 article 'A stochastic data-based traffic model applied to vehicles energy consumption estimation'. IEEE Transactions on Intelligent Transportation Systems 2019
- 4 article'Optimality conditions in variational form for non-linear constrained stochastic control problems'.Mathematical Control & Related Fields1032020, 493-526
- 5 article'Gaussian Mixture Penalty for Trajectory Optimization Problems'.Journal of Guidance, Control, and Dynamics428August 2019, 1857--1862
- 6 inproceedings 'Optimal control of an artificial microbial differentiation system for protein bioproductioń'. ECC 2019 - European Control Conference Naples, Italy June 2019

## 10.2 Publications of the year

### International journals

- 7 article 'State-constrained control-affine parabolic problems I: First and Second order necessary optimality conditions'. Set-Valued and Variational Analysis 2020
- 8 article 'State-constrained control-affine parabolic problems II: second order sufficient optimality conditions'. SIAM Journal on Control and Optimization 2021
- 9 article'Second-order analysis for the time crisis problem'.Journal of Convex Analysis2712020, 139-163
- 10 article'Optimal control of state constrained age-structured problems'.SIAM Journal on Control and Optimization584July 2020, 2206-–2235
- 11 article 'Optimal control techniques based on infection age for the study of the COVID-19 epidemic'. Mathematical Modelling of Natural Phenomena 2021
- 12 article'On the Turnpike Property and the Receding-Horizon Method for Linear-Quadratic Optimal Control Problems'.SIAM Journal on Control and Optimization582January 2020, 26
- 13 article 'The effect of the terminal penalty in receding horizon control for a class of stabilization problems'. ESAIM: Control, Optimisation and Calculus of Variations 58 2020
- 14 article 'To quarantine, or not to quarantine: A theoretical framework for disease control via contact tracing'. Epidemics 34 December 2020
- 15 article 'On continuum approximations of discrete-state Markov processes of large system size'. Multiscale Modeling and Simulation: A SIAM Interdisciplinary Journal 2021
- 16 article'Optimality conditions in variational form for non-linear constrained stochastic control problems'.Mathematical Control & Related Fields1032020, 493-526

### International peer-reviewed conferences

- 17 inproceedings 'An Eco-routing algorithm for HEVs under traffic conditions'. IFAC 2020 - 21st IFAC World Congress Proceedings of the 21st IFAC World Congress Berlin / Virtual, Germany July 2020

### Reports & preprints

- 18 misc 'A linear finite-difference scheme for approximating Randers distances on Cartesian grids'. January 2021
- 19 misc 'Monotone and second order consistent scheme for the two dimensional Pucci equation'. May 2020
- 20 misc 'Second order monotone finite differences discretization of linear anisotropic differential operators'. December 2020
- 21 misc 'Discrete-time mean field games with risk-averse agents'. May 2020
- 22 misc 'Weak solutions for potential mean field games of controls'. December 2020
- 23 misc 'A bi-level energy management strategy for HEVs under probabilistic traffic conditions'. September 2020
- 24 misc 'Approximate Nash equilibria in large nonconvex aggregative games'. November 2020
- 25 misc 'Beyond the chemical master equation: stochastic chemical kinetics coupled with auxiliary processes'. November 2020
- 26 misc 'Duality and approximation of stochastic optimal control problems under expectation constraints'. July 2020
- 27 misc 'Decomposition of High Dimensional Aggregative Stochastic Control Problems'. September 2020

## 10.3 Cited publications

- 28 book 'Optimal control and viscosity solutions of Hamilton-Jacobi-Bellman equations'. Systems and Control: Foundations and Applications Birkhäuser, Boston 1997
- 29 book 'Solutions de viscosité des équations de Hamilton-Jacobi'. 17 Mathématiques et Applications Springer, Paris 1994
- 30 book 'Lectures on the Calculus of Variations'. Chicago, Illinois University of Chicago Press 1946
- 31 article'Computation of order conditions for symplectic partitioned Runge-Kutta schemes with application to optimal control'.Numerische Mathematik10312006, 1--10
- 32 article'Numerical schemes for the two dimensional second-order HJB equation'.ESAIM: M2AN382004, 723-735
- 33 article' Consistency of generalized finite difference schemes for the stochastic HJB equation'.SIAM J. Numerical Analysis412003, 1008-1021
- 34 book 'Applied optimal control'. New-York Hemisphere Publishing 1975
- 35 article'Viscosity solutions of Hamilton Jacobi equations'.Bull. American Mathematical Society2771983, 1--42
- 36 article'Runge-Kutta methods in optimal control and the transformed adjoint system'.Numerische Mathematik8722000, 247--282
- 37 book 'Theory of Extremal Problems'. Russian Edition: Nauka, Moscow, 1974 North-Holland Publishing Company, Amsterdam 1979
- 38 book 'Mécanique analytique'. reprinted by J. Gabay, 1989 Paris 1788
- 39 article'A stochastic data-based traffic model applied to vehicles energy consumption estimation'.IEEE Transactions on Intelligent Transportation Systems2172019, 3025--3034
- 40 book 'Foundations of optimal control theory'. John Wiley, New York 1967
- 41 book 'An introduction to optimal control'. Mc Graw Hill, New York 1966
- 42 book 'The Mathematical Theory of Optimal Processes'. Wiley Interscience, New York 1962
- 43 book 'Lectures on the calculus of variations and optimal control theory'. Philadelphia W. B. Saunders Co. 1969