Section: New Results

Software Engineering for Activity Recognition

Participants : François Brémond, Erwan Demairy, Sabine Moisan, Anh-Tuan Nghiem, Annie Ressouche, Jean-Paul Rigault, Monique Thonnat, Jean-Yves Tigli, Christophe Tornieri.

Introduction

Participants : Monique Thonnat, François Brémond, Erwan Demairy, Christophe Tornieri.

This year Pulsar has developed a new software platform: the SUP platform. It is the backbone of the team experiments to implement the new algorithms proposed by the team in perception, understanding and learning. We study a meta-modeling approach to support the development of video surveillance applications based on SUP. We also introduce the notion of relation in our knowledge description language. We study the development of a scenario recognition module relying on formal methods to support activity recognition in SUP platform. We began to study the definition of multiple services for device adaptive platform for scenario recognition. We continue to develop the Clem toolkit around a synchronous language dedicated to activity recognition applications.

subsubsectionSUP Software Platform

SUP is made as a framework allowing several video surveillance workflows to be implemented. Currently, the workflow is static for a given application. A given workflow is the composition of several plugins, each of them implementing an algorithmic step in the video processing (i.e. the segmentation of images, the classification of objects, etc.).

The design of SUP allows to execute at runtime the selected plugins. Currently 15 plugins are available:

• 6 plugins are wrappers on industrial implementations of algorithms (made available by Keeneo). They allow a quick deployment of a video processing chain encompassing image acquisition, segmentation, short-term and long-term tracking. These algorithms are robust and efficient algorithms, but with the drawback that some algorithms can lack of accuracy.

• 9 are implementations by the team members which cover the following fields:

  • one segmentation removing the shadows;

  • two classifiers, one being based on postures and one on people detection;

  • four frame-to-frame trackers, using as algorithm: (i) a simple tracking by overlapping, (ii) neural networks, (iii) tracking of feature points, (iv) tracking specialized for the tracking of persons in a crowd.

  • two scenario recognizers, one generic allowing expression of probabilities on the recognized events, and the other one focusing on the recognition of events based on postures.

From a software engineering point-of-view, the goal is to obtain a platform being dynamically reconfigurable as described in next section.

Model-Driven Engineering for Video Surveillance

Participants : Sabine Moisan, Jean-Paul Rigault, Guiseppe Santoro.

This year we have explored how model-driven engineering techniques can support the configuration and dynamic adaptation of the video surveillance systems designed with our SUP platform. This work is done in collaboration with the MODALIS team of UNSA/CNRS I3S laboratory.

In the video surveillance community, the focus has moved from individual vision algorithms to integrated and generic software platforms (such as SUP), and now to the security, scalability, evolution, and ease of use of these platforms. The last trends require a modeling effort for video surveillance component platforms as well as for application specification. Our approach is to apply modeling techniques to the application specification (describing the video surveillance task, and its context) as well as to the implementation (assembling the software components). Model-driven engineering uses models to represent partial or complete views of an application or a domain, possibly at different abstraction levels. MDE offers techniques to transform a source model into a target one.

The number of different tasks (such as detection, counting, or tracking), the complexity of contextual information, and the relationships among them induce many possible variants. The first activity of a video surveillance application designer is to sort out these variants to precisely specify the function to realize and its context. In our case study, the underlying software architecture is component-based (SUP). The processing chain consists of components that transform data before passing it to other components. As a result, the designer has to map this specification to software components that implement the needed algorithms.

The challenge is to cope with the many –functional as well as nonfunctional– causes of variability on both sides (specification and implementation). Hence, we first decided to separate these two concerns. We then applied domain engineering to identify the reusable elements on both sides. This led to two models: a generic model of video surveillance applications (for short application model ) and a model of video processing components and chains (component platform configuration model, for short platform model ). Both of them are feature models expressing variability factors. Feature models are a popular formalism used to model software product lines commonalities and variabilities. They compactly define all features in an product line and their valid combinations. A feature model is basically an AND-OR graph with constraints which organizes hierarchically a set of features while making explicit the variability. Our models are also enriched with intra- and inter-models constraints. Inter-models constraints specify how the system should adapt to changes in its environment. It is convenient to use the same kind of models on both sides, leading to a uniform syntax. Feature models are appropriate to describe variants; they are simple enough for video surveillance experts to express their requirements. Yet, they are powerful enough to be liable to static analysis  [58] . In particular, the inter-feature constraints can be analyzed as SAT problem.

The application model describes the relevant concepts and features from the stakeholders' point of view, in a way that is natural in the video surveillance domain: characteristics and position of sensors, context of use (day/night, in/outdoors, target task)... It also includes notions of quality of service (performance, response time, detection robustness, configuration cost...). Such a model provides a user friendly way to specify a problem for SUP. The platform model describes the different software components and their assembly constraints (ordering, alternative algorithms...). Ultimately, we wish to automatically generate a SUP component assembly from an application specification, using model to model transformations. The first model can be transformed into one or several valid component configurations, in order to map the specification onto software components that implement the needed algorithms. Due to the multiple causes of variability, the result of the transformation is usually a set of possible component assemblies fulfilling the task and the context specification; hence the designer has to manually fine tune the assembly. This approach allows designers to define the static initial configuration of a video surveillance system [31] .

Figure 13. Simplified feature model of video surveillance applications

Concretely, we have developed a generic feature diagram editor to manipulate the models, using ECLIPSE meta-modeling facilities (EMF, ECORE, GMF...). At this time, we have a first prototype that allowed us to represent both models. However, the current tool only supports natural language constraints, but we have experimented the KERMETA workbench to implement some model to model transformations. We have also developed an interface with SAT tools to verify the generated configurations.

An additional challenge is to manage the dynamic variability of the context to cope with possible run-time change of implementation triggered by context variations (e.g. lighting conditions, changes in the reference scene, etc.). Video surveillance systems are indeed a good example of dynamic adaptive systems , i.e. software systems which have to dynamically adapt in order to cope with a changing environment. Such systems must be able to sense their environment, to autonomously select an appropriate configuration and to efficiently migrate to this configuration. Handling these issues at the programing level proves to be challenging due to the large number of contexts and of software configurations. The use of models at run-time is an alternative solution. In our approach, the adaptation logic is defined by adaptation rules, corresponding to the dependency constraints between specification elements in one model and software variants in the other [30] . A context change at run-time corresponds to a new configuration in the application model and must lead to a new configuration of the platform model. Adaptation rules use the constraint language of feature models (i.e. propositional logic-based language). These rules address features possibly connected with “and”, ”or”, “not”, their actions correspond to changes in the processing chain. As an example, the following adaptation rule:

Night and HeadLight implies HeadLightDetection

states that if the context changes (from day) to night and if headlights (e.g. of vehicles) must be taken into account, a new component from the platform (HeadLightDetection module) must be integrated in the running processing chain. To ensure its usability, the proposed approach has been built on top of the Domain Specific Modeling Language (DSML  [55] ) for adaptive systems.

Scenario Analyze Module

Participants : Bernard Boulay, Sabine Moisan, Annie Ressouche, Jean Paul Rigault.

This research axis contributes to the study and the development of a module to analyze scenarios.

This year we have studied models of scenarios dealing with both real time (to be realistic and efficient in the analyze phase) and logic time (to benefit from well-known mathematical models allowing re-usability, easy extension and verification). Scenarios are mostly used to specify the way a system may react to sensor inputs. Therefore, models of scenarios must also take into account the uncertainty of sensor results. To address these needs (logic time, real time and uncertainty) we have defined a language to express scenarios as compositions of sub-scenarios. Basic scenarios are composed of events, while general scenarios are expressed as composition of temporal relations (before, during, overlap) between sub-scenarios. Temporal constraints can also be expressed. Moreover, the language supports the definition of external types and functions in order to allow scenarios to handle events of different kinds.

For behavior recognition, as for all automated systems, validation is really a crucial phase and an exhaustive approach of validation is clearly needed. To be trusted behavior analyze must rely on formal methods from the very beginning of its design. Formal methods help to produce a sound code the size and frequency of which can be estimated. Hence, we defined a synchronous model for our scenario language. This modeling supports the representation of scenarios as equation systems which compute two Boolean variables for each scenario: beginning true when the first event(s) or sub scenario(s) of the scenario is recognized and termination true when the scenario is recognized. This theoretical approach leads to the definition of a scenario analyze module (sam ). This module provides users with (1) a simulation tool to test scenario behaviors; (2) a recognition program in C for each scenario which must be completed by the definition (by the user) of external types and functions in a C or C++ environment; (3) an exhaustive verification of safety properties relying on model checking techniques our approach allows. This latter offers also the possibility to define the safety properties we want to prove as “observers”  [57] expressed in the scenario language.

Multiple Services for Device Adaptive Platform for Scenario Recognition

Participants : Annie Ressouche, Jean-Yves Tigli.

Activity recognition and monitoring systems based on multi-sensor and multi-device approaches are more and more popular to enhance events production for scenario analysis. Underlying software and hardware infrastructures can be considered as static (no changes during the overall recognition process quasi-static (no changes during two reconfigurations of the process) or really dynamic (depending on dynamic appearance and disappearance of numerous sensors and devices in the scene, communicating with the system, during recognition process).

In this last case, we need to partially and reactively adapt the application to the evolution of the environment, while preserving invariants required for the validity of the recognition process.

In order to address such a challenge our researches try to federate the inherent constraints of platform devoted to action recognition, like SUP, with a service oriented middleware approach to deal with dynamic evolutions of the system infrastructure. Recent results, using a Service Lightweight Component Architecture (SLCA) [28] to compose services for device and Aspects of Assembly (AA) to adapt them in a reactive way [27] , present interesting prospects to deal with multi-devices and variable systems. They provide a user-friendly separated description for adaptations that shall be applied and composed at runtime as soon as the corresponding required devices are present (in context-sensitive security middleware layer for example [40] ). They also underline performances and response times that allow reactive adaptation on appearance and disappearance of devices. However, although composition between these adaptations can verify proved properties, the use of blackbox components in the composition model of SLCA doesn't allow extracting a model of their behavior. Thus, existing approaches don't really succeed to ensure that the usage contract of these components is not violated during application adaptation. Only a formal analysis of the component behavior models associated with a well sound modeling of composition operation will allow us to secure the respect of the usage contracts.

In this axis, we propose to rely on a synchronous modeling of component behavior and component assembly to allow the usage of model checking techniques to formally validate services composition.

We began to consider this topic in 2008 through a collaborative action (SynComp) between Rainbow team at University of Nice Sophia Antipolis and INRIA Pulsar team. Within the Rainbow team, SLCA/AA experimental platform called WComp is dedicated to the reactive adaptation of applications in the domain of ubiquitous computing. During this collaboration, the management of concurrent access in WComp has been studied as a main source of disturbance for the invariant properties. A modeling of the behavior of components and of their accesses in a synchronous model has been defined in WComp.

This approach allows us to benefit from model checking techniques to ensure that there are no unpredictable states of WComp components on concurrent access. This year, during his training, Vivien Fighiera (already involved in the SynComp action), has completed the theoretical work done in SynComp. He studied how to prove safety properties regarding WComp component models relying on the NuSMV  [52] model checker. This year, the collaboration between Rainbow and Pulsar has been strengthened since Jean-Yves Tigli is a full time researcher at Pulsar team since September, in sabbatical year sponsored by INRIA. Now, we plan to modelize the overall assembly of WComp components with a synchronous approach to allow the usage of model checking techniques to formally validate application design. In order to obtain results based on experimental scenarios to evaluate SynComp improvements for adaptive recognition process, we plan to integrate SUP platform as a software services provider.

The SUP platform gathers a set of modules devoted to design applications in the domain of activity recognition. WComp is aimed at assembling services which evolve in a dynamic and heterogeneous environment. Indeed, the services provided by SUP can be seen as complex high-level services whose functionalities depend on the SUP treatments; this latter dealing with the dynamic change of the environment. Thus, considering SUP services as web services for devices for example, the devices associated with SUP services will be discovered dynamically by WComp and used with other heterogeneous devices.

The Clem Workflow

Participants : Annie Ressouche, Daniel Gaffé.

This research axis concerns the theoretical study of a synchronous language le with modular compilation and the development of a toolkit 14 around the language to design, simulate, verify and generate code for programs.

le language agrees with Model Driven Software Developement philosophy which is now well known as a way to manage complexity, to achieve high re-use level, and to significantly reduce the development effort. Therefore, we benefit from a formal framework well suited to compilation and formal validation. In practice, we defined two semantics for le : a behavioral semantics to define a program by the set of its behaviors, avoiding ambiguities in program interpretations; an equational semantics to allow modular compilation of programs into software and hardware targets (C code, VHDL code, FPGA synthesis, observers...). Our approach fulfills two main requirements of critical realistic applications: modular compilation to deal with large systems and model-based approach to perform formal validation.

Figure 14. The Clem Toolkit

The main originality of this work is to be able to manage both modularity and causality. Indeed, only few approaches consider a modular compilation because there is a deep incompatibility between causality and modularity. Causality means that for each event generated in a reaction, there is a causal chain of events leading to this generation. No causal loop may occur. Program causality is a well-known problem with synchronous languages, and therefore, it needs to be checked carefully. Thus, relying on semantics to compile a language ensures a modular approach but requires to complete the compilation process with a global causality checking. To tackle this problem, we introduced a new way to check causality from already checked sub programs and the modular approach we infer.

This year we focused on the algorithms to check causality. To compile le programs we rely on an equational semantics that translates each program into an equation system. A program is causal (i.e it has no causality cycle) if its associated equation system has no dependency cycle between its variables. Thus, to check causality amounts to find an evaluation order for equation systems. Usually, several total orders are valid when sorting an equation system, and choosing one particular prevents modularity. Indeed, starting with two equation systems totally ordered could result in a false cyclic equation system when merging them. To ensure modularity, we define a sorting algorithm that computes all the valid partial order of an equation system. We complete our approach in also defining an algorithm to merge two previously sorted equation systems. These algorithms are the corner stone of our approach. This year we define the merge algorithm and we prove (1) that the sorting algorithm we have is the greatest fix point of a function defined on the dependency graph of equation systems; (2) that the merge algorithm is correct (i.e we get the same partial orders using the merge algorithm on two previously sorted equation systems or when sorting the union of the two equation systems considered).

On another hand, this year we have extended the language to support data handling. According to le principle, only signals carry data. Data belong to predefined types (usual programming language types) or to some external types. We improve the syntax to allow data definition for signals. Then we also extend the internal exchange format (lec ) we defined to support modularity, to take values into account. We also improve the compiler of the language to deal with signal data and finally, we improve the code generation. Now, the integration of data must be achieved in the simulator and in the code we generate to feed the NuSMV model checker.