- A2.1.5. Constraint programming
- A5.1.1. Engineering of interactive systems
- A5.1.2. Evaluation of interactive systems
- A5.1.7. Multimodal interfaces
- A5.4.4. 3D and spatio-temporal reconstruction
- A5.4.5. Object tracking and motion analysis
- A5.5.1. Geometrical modeling
- A5.10.1. Design
- A5.10.2. Perception
- A5.10.4. Robot control
- A5.10.5. Robot interaction (with the environment, humans, other robots)
- A5.10.8. Cognitive robotics and systems
- A5.11.1. Human activity analysis and recognition
- A6.2.5. Numerical Linear Algebra
- A6.2.6. Optimization
- A6.4.6. Optimal control
- A6.5.1. Solid mechanics
- A8.3. Geometry, Topology
- A9.5. Robotics
- A9.8. Reasoning
- B1.2.2. Cognitive science
- B2.8. Sports, performance, motor skills
- B5.1. Factory of the future
- B5.2. Design and manufacturing
- B5.6. Robotic systems
- B9.6. Humanities
- B9.9. Ethics
1 Team members, visitors, external collaborators
- David Daney [Team leader, Inria, Researcher, HDR]
- Vincent Padois [Inria, Senior Researcher, HDR]
- Sylvain Pion [Inria, Researcher, until Feb 2020]
- Nasser Rezzoug [Univ de Toulon et du Var, Associate Professor, from Sep 2020, HDR]
- Jean-Marc Salotti [Institut National Polytechnique de Bordeaux, Professor, HDR]
- Charles Fage [Institut National Polytechnique de Bordeaux, until Sep 2020]
- Lucas Joseph [Inria]
- Joshua Pickard [Inria, until Feb 2020]
- Nassim Benhabib [Inria]
- Olfa Jemaa [Ecole nationale d'ingénieurs de Sousse - Tunisie]
- Pierre Laguillaumie [Univ de Poitiers]
- Baptiste Prebot [Institut National Polytechnique de Bordeaux, until Jun 2020]
- Nicolas Simonazzi [Orange, CIFRE]
- Antun Skuric [Inria, from Jul 2020]
- Nicolas Torres [Groupe PSA, CIFRE, from Apr 2020]
- Erwann Landais [Inria, Engineer, from Nov 2020]
Interns and Apprentices
- Benjamin Camblor [Institut National Polytechnique de Bordeaux, until Jun 2020]
- Chrystel Plumejeau [Inria]
- Benjamin Camblor [Association pour le développement de l'enseignement et des recherches d'Aquitaine, from Oct 2020]
- Nasser Rezzoug [Univ de Toulon et du Var, until Aug 2020, HDR]
2 Overall objectives
The project of the Auctus team is to design the collaborative robotics cells of the future.
The robotics community still tends to separate the cognitive (HRI) and physical (pHRI) aspects of human/robot interaction. One of the main challenges is to characterize the task as well as biomechanical, physiological and cognitive capacities of humans in the form of physical constraints or objectives for the design of cobotized workstations. This design is understood in a large sense: the choice of the robot's architecture (cobot, exoskeleton, etc.), the dimensional design (human/robot workspace, trajectory calculation, etc.), the coupling mode (comanipulation, teleoperation, etc.) and control. The approach then requires the contributions of the human and social sciences to be considered in the same way as those of exact sciences. The topics considered are broad, ranging from cognitive sciences, ergonomics, human factors, biomechanics and robotics.
The first challenge is to evaluate the hardship at work, the well-being of the operators and, further upstream, their cognitive state which impacts their sensorimotor strategy for performing a task. In industry, the ergonomic analysis of the task is carried out by an ergonomist based on direct but often ad hoc observations. However, the context is changing: the digitization of factories, through the installation of on-site sensors, allows longitudinal observation of machines and humans. The information available can thus allow us to rethink the way in which the evaluation of activities is carried out. Currently, an emerging subdomain named ergonomic robotics adapts the available ergonomic evaluation criteria (RULA, REBA, etc.). However, they are related to the (quasi-static) posture of the operator, which limits the understanding of human motor strategies over a long period of time. Similarly, kinematic or biomechanical analysis may tend to see humans as a high-performance machine to be optimized. This may make sense for a top-level athlete, but repeating actions in the industry over a day, months or years of work means that a temporary change of posture, possibly poorly rated according to usual ergonomic criteria, can in fact be a good long-term strategy. These questions make a direct link between motor and cognitive aspects that can be reflected in particular strategies as the fatigue or the expertise (manual and cognitive). This approach has not been widely explored in robotics to determine the right criteria to adapt the behavior of a cobot.
The second challenge is to define a methodology to link the analysis of the task and the human movements it induces to the robot design. Indeed, as we have been able to verify on several occasions in the context of industrial projects, between the ergonomist, expert in task analysis and psychology, and the robotician, expert in mechanics, control and computer science, there is a significant conceptual distance that makes it very difficult to analyze needs and define the specifications of the technical solution. To fill these methodological gaps, it is necessary, on the basis of case studies, to better define the notion of tasks in the context of a human/robot coupling and to establish a typology of this type of interaction by taking into account, with as much details as possible, the different physical and cognitive constraints and their potential psychological, organizational or ethical impacts.
The third challenge is related to the need to think about the control laws of collaborative robots in terms of human/robot coupling. The effectiveness of this coupling requires an ability to predict future human actions. This prediction should make the interaction more intuitive but also aims at an optimal coupling from the point of view of “slow” phenomena such as fatigue. The major challenge is therefore to move from reactive to predictive control laws, integrating a human prediction model, both in terms of movement strategies and decision strategies. Beyond the great computational complexity of predictive approaches, obtaining prediction models is an ambitious challenge. It is indeed necessary to learn models that are quite complex in terms of the physical realities they can account for and quite simple from a computational point of view.
3 Research program
3.1 Analysis and modelling of human behavior
3.1.1 Scientific Context
The purpose of this axis is to provide metrics to assess human behavior. We place ourselves here from the point of view of the human being and more precisely of the industrial operator. We assume the following working hypotheses: the operator's task and environmental conditions are known and circumscribed; the operator is trained in the task, production tools and safety instructions; the task is repeated with more or less frequent intervals. We focus our proposals on assessing:
- the physical and cognitive fragility of operators in order to meet assistance needs;
- cognitive biases and physical constraints leading to a loss of operator safety;
- ergonomic, performance and acceptance of the production tool.
In the industrial context, the fields that best answer these questions are work ergonomics and cognitive sciences. Scientists typically work on 4 axes: physiological/biomechanical, cognitive, psychological and sociological. More specifically, we focus on biomechanical, cognitive and psychological aspects, as described by the ANACT 18, 19. The aim here is to translate these factors into metrics, optimality criteria or constraints in order to implement them in our methodologies for analysis, design and control of the collaborative robot.
To understand our desired contributions in robotics, we must review the current state of ergonomic workstation evaluation, particularly at the biomechanical level. The ergonomist evaluates the gesture through the observation of workstations and, generally, through questionnaires. This requires long periods of field observation, followed by analyses based on ergonomic grids (e.g. RULA 35, REBA 25, LUBA 30, OWAS 29, ROSA 51,...). Until then, the use of more complex measurement systems was reserved for laboratories, particularly biomechanical laboratories. The appearance of inexpensive sensors such as IMUs (Inertial Measurement Units) or RGB-D cameras makes it possible to consider a digitalized, and therefore objective, observation of the gesture, postures and more generally of human movement. Thanks to these sensors, which are more or less intrusive, it is now possible to permanently install observation systems on production lines. This completely changes paradigms and opens the door to longitudinal observations. It should be noted that this is comparable to the evolution of maintenance, which becomes predictive.
On the strength of this new paradigm, ergonomic robotics has recently taken an interest in this type of evaluation to adapt the robot's movements in order to reduce ergonomic risk scores. This approach complements the more traditional approaches that only consider the performance of the action produced by the human in interaction with the robot. However, we must go further. Indeed, the ergonomic criteria are based on the principle that the comfort positions are distant from the human joints articular limits. In addition, the notation must be compatible with an observation of the human being through the eye of the ergonomist. In practice, evaluations are inaccurate and subjective 54. Moreover, they are made for quasi-static human positions without taking into account the evolution of the person's physical, physiological and psychological state. The repetition of gestures, the solicitation of muscles and joints is one of the questions that must complete these analyses. One of the methods used by ergonomists to limit biomechanical exposures is to increase variations in motor stress by rotating tasks 52. However, this type of extrinsic method is not always possible in the industrial context 33.
One of Auctus' objectives is to show how, through a cobot, the operator's environment can be varied to encourage more appropriate motor strategies. To do so, we must focus on a field of biomechanics that studies the intrinsic variability of the motor system allowed by the joint redundancy of the human body. This motor variability refers to the natural alternation of postures, movements and muscle activity observed in the individual to respond to a requested task 52. This natural variation leads to differences between the motor coordinates used by individuals, which evokes the notion of motor strategy 26.
As shown by the cognitive dimension of ergonomics (see above), we believe that some of these motor strategies are a physically quantifiable reflection of the operator's cognitive state. For example, fatigue 48 and its anticipation or the manual expertise (dexterous and cognitive) of the operator which allows him to anticipate his movements over long periods of time in order to preserve his body, his performance and his pain.
How can we observe, understand and quantify these human motor strategies to better design and control the behavior of the cobotic assistant? When we study the systems of equations considered (kinematic, static, dynamic, musculoskeletal), several problems appear and explain our methodological choices:
- the large dimensions of the problems to be considered, due to joint, muscle and placement redundancy;
- the variabilities of the parameters, for example: physiological (consider not an operator, but a set of operators), geometric (consider a set of possible placements of the operator) and static (consider a set of forces that the operator must produce);
- the uncertainties of measurement and models approximation.
The idea is to start from a description of redundant workspaces (geometric, static, dynamic...). To do this, we use set theory approaches, based on interval analysis 3, 41, which allow us to respond to the uncertainties and variability issues previously mentioned. In addition, one of the advantages of these techniques is that they allow the results to be certified, which is essential to address safety issues. Some members of the team has already achieved success in mechanical design for performance certification and robot design 37. The adaptation of these approaches allows us to obtain a mapping of ergonomic and efficient movements in which we can project the operators' motor strategies and thus define a metric quantifying the sensorimotor commands chosen with regard to the cognitive criteria studied.
It is therefore necessary to:
- propose new indices in the fields of ergonomics, biomechanics and robotics linking different types of performances and taking into account the influence of fatigue, stress, level of expertise;
- divide the gesture into homogeneous phases: this process is complex and depends on the type of index used and the techniques used. We are exploring several ways: inverse optimal control, learning methods, or the use of techniques from signal processing;
- develop interval extensions of the identified indices. These indices are not necessarily the result of a direct model, and algorithms need to be developed or adapted (calculation of manipulability, UCM, etc.);
- Aggregate proposals into a dedicated interval analysis library (use of and contribution to the existing ALIAS-Inria and the open source IBEX library).
The originality and contribution of the methodology is to allow an analysis taking into account in the same model the measurement uncertainties (important for on-site use of analytical equipment), the variability of tasks and trajectories, and the physiological characteristics of the operators.
Other avenues of research are being explored, particularly around the inverse optimal control 42 which allows us to project human movement on the basis of performance indices and thus to offer a possible interpretation in the analysis of behaviors.
We also use automatic classification techniques: 1) to propose cognitive models that will be learned experimentally 2) for segmentation or motion recognition, for example by testing Reservoir Computing 27 approaches.
3.2 Operator / robot coupling
3.2.1 Scientific Context
Thanks to the progress made in recent years in the field of p-HRI (Physical Human-Robot Interaction), robotic systems are beginning to operate in the same workspace as humans, which is profoundly changing industrial issues and allowing a wide variety of human-robot coupling solutions to be considered to perform the same task 20. Different types of interactions exist. They can be classified in different ways: according to the degree of autonomy of the robot and its proximity to the user 24 with particularities for “wearable robots” 22, 23, or for collaborative robotics 53, or according to the role of the human being 49. From a cognitive point of view, classifications are more concerned with autonomy, the complexity of information processing and the type of communication and representation of the human being by the robot 40, 55.
The parameters of the coupling being numerous and complex, the determination of the most appropriate type of coupling for a type of problem is an open problem 43, 2, 39, 34. The traditional approach consists in trying to identify and classify the various possible options and to select the one that seems most relevant with regard to the feasibility, efficiency, budget envelope and acceptability of the operator. One of the main objectives of our research project is to define a typology of cobots or cobotic systems in order to specify the methodology for developing the best solution: what are the criteria for defining the best robotic architecture, what type of coupling, what autonomy of the robot, what role for the operator, what risks for the human, what overall performance? These are the key issues that need to be addressed. To meet this methodological need, we propose an approach guided by experience on use cases obtained thanks to our industrial partners.
Task analysis and human behavior modelling, discussed in the previous sections, should help to characterize the different types of coupling and interaction modalities, their advantages and disadvantages, in order to assist in the decision-making process. One of the ideas we would like to develop is to try to break down the task into a sequence of elementary gestures corresponding to simple motor actions performed in a clearly identified context and to evaluate for each of them the degree of feasibility in automatic mode or in robot assistance mode. The assessment must take into account a large number of parameters that relate to physical interactions, human-robot communication, reliability and human factors, including acceptability and impact on the valuation or devaluation of the operator's work. Concerning the evaluation of human factors, we have already begun to work on the subject within the more general framework of human systems interactions by operating Bayesian networks, drawing inspiration from the work of 21, 46.
The adoption of assessment criteria for a single domain (e. g. robotics or ergonomics) cannot guarantee that the performance of this coupling will be maximized. From design to evaluation, cross-effects must be constantly considered:
- impact of the cobot design on the user's performance: intuitiveness, adaptation to intra- and inter-individual variations, affordance, stress factors (noise, vibrations,...), fatigue factors (control laws, necessary attention,...) and motivation factors (effectiveness, efficiency, aesthetics,...);
- impact of the cobot design on the specific biomechanical constraints imposed on the human body (extra weight supported by the operator, additional joint reaction forces generated at the joints);
- impact of user performance on cobot exploitation: risks of human error (attention error, perseveration, circumvention of procedures, out-of-the-loop syndrome) 21.
In addition to purely physical assistance, some cobotic systems are designed to assist the operator in his decision-making. The issues of trust, acceptance, sharing of representations and co-construction of a shared awareness of the situation are then to be addressed 50.
3.3 Design of cobotic systems
3.3.1 Architectural design
Is it necessary to cobotize, robotize or assist the human being? Which mechanical architecture meets the task challenges (a serial cobot, a specific mechanism, an exoskeleton)? What type of interaction (H/R cohabitation, comanipulation, teleoperation)? These questions are the first requests from our industrial partners. For the moment, we have few comprehensive methodological answers to provide them. Choosing a collaborative robot architecture is a difficult problem 31. It is all the more when the questions are approached from both a cognitive ergonomics, boimechanical and robotics perspectives. There are indeed major methodological and conceptual differences in these areas. It is therefore necessary to bridge these representational gaps and to propose an approach that takes into consideration the expectations of the robotician to model and formalize the general properties of a cobotic system as well as those of the ergonomist to define the expectations in terms of an assistance tool.
To do this, we propose a user-centered design approach, with a particular focus on human-system interactions. From a methodological point of view, this requires first of all the development of a structured experimental approach aimed at characterizing the task to be carried out through a “system” analysis but also at capturing the physical markers of its realization: movements and efforts required, ergonomic stress. This characterization must be done through the prism of the systematic study of the exchange of information (and their nature) by humans in their performance of the considered task. On the basis of these analyses, the main challenge is to define a decision support tool for the choice of the robotic architecture and for the specifications of the role assigned to the robot and the operator as well as their interactions.
The evolution of the chosen methodology is for the moment empirical, based on the user cases regularly treated in the team (see sections on contracts and partnerships).
It can be summarized for the moment as:
- identify difficult jobs on industrial sites. This is done through visits and exchanges with our partners (manager, production manager, ergonomist...);
- select some of them, then observe the human in its ecological environment. Our tools allow us to produce a motion analysis, currently based on ergonomic criteria. In parallel we carry out a physical evaluation of the task in terms of expected performance and an evaluation of the operator by means of questionnaires;
- synthesize these first results to deduce the robotic architectures to be initiated, the key points of human-robot interaction to be developed, the difficulties in terms of human factors to be taken into account.
In addition, the different human and task analyses take advantage of the different expertise available within the team. We would like to gradually introduce the evaluation criteria presented above. Indeed, the team has already worked on the current dominant approach: the use of a virtual human to design the cobotic cell through virtual tools 1. However, the very large dimensions of the problems treated (modelling of the body's dofs and the constraints applied to it) makes it difficult to carry out a certified analysis. We then choose to go through the calculation of the body's workspace, representing its different performances, which is not yet done in this field. The idea here is to apply set theory approaches, using interval analysis already discussed in section 3.1.2. The goal is then to extend to intervals the constraints played in virtual reality during the simulation. This would allow the operator to check his trajectories and scenarios not only for a single case study but also for sets of cases. For example, it can be verified that, regardless of the bounded sets of simulated operator physiologies, the physical constraints of a simulated trajectory are not violated. Thus, the assisted design tools certify cases of use as a whole. Moreover, the intersection between the human and robot workspaces provides the necessary constraints to certify the feasibility of a task. This allows us to better design a cobotic system to integrate physical constraints. In the same way, we will look for ways in which human cognitive markers can be included in this approach.
Thus, we summarize here the contributions of the other research axes, from the analysis of human behavior in its environment for an identified task, to the choice of a mechanical architecture, via an evaluation of torque and interactions. All the previous analyses provide design constraints. This methodological approach is perfectly integrated into an Appropriate Design approach used for the dimensional design of robots, again based on interval analysis. Indeed, to the desired performance of the human-robot couple in relation to a task, it is sufficient to add the constraints limiting the difficulty of the operator's gesture as described above. The challenges are then the change of scale in models that symbiotically consider the human-robot pair, the uncertain, flexible and uncontrollable nature of human behavior and the many evaluation indices needed to describe them.
3.3.2 Control design
The control laws of collaborative robots from the major robot manufacturers differ little or not at all from the existing control laws in the field of conventional industrial robotics. Security is managed a posteriori, as an exception, by a security PLC / PC. It is therefore not an intrinsic property of the controller. This quite strongly restricts the possibilities of physical interaction 1 and collaboration and leads to sub-optimal operation of the robotic system. It is difficult in this context to envision real human-robot collaboration. Collaborative operation requires, in this case, a control calculation that integrates safety and ergonomics as a priori constraints.
The control of truly collaborative robots in an industrial context is, from our point of view, underpinned by two main issues. The first is related to the macroscopic adaptation of the robot's behaviour according to the phases of the production process. The second is related to the fine adaptation of the degree and/or nature of the robot's assistance according to the ergonomic state of the operator. If this second problem is part of a historical dynamic in robotics that consists in placing safety constraints, particularly those related to the presence of a human being, at the heart of the control problem 2436, 28, it is not approached from the more subtle point of view of ergonomics where the objective cannot be translated only in terms of human life or death, but rather in terms of long-term respect for their physical and mental integrity. Thus, the simple and progressive adoption by a human operator of the collaborative robot intended to assist him in his gesture requires a self-adaptation in the time of the command. This self-adaptation is a fairly new subject in the literature 44, 45.
At the macroscopic level, the task plan to be performed for a given industrial operation can be represented by a finite state machine. In order not to increase the human's cognitive load by explicitly asking him to manage transitions for the robot, we propose to develop a decision algorithm to ensure discrete transitions from one task (and the associated assistance mode) to another based on an online estimate of the current state of the human-robot couple. The associated scientific challenge requires establishing a link between the robot's involvement and a given working situation. To do so, we propose an incremental approach to learning this complex relationship. The first stage of this work will consist in identifying the general and relevant control variables to conduct this learning in an efficient and reusable way, regardless of the particular method of calculating the control action. Physically realistic simulations and real word experiments will be used to feed this learning process.
In order to handle mode transitions, we propose to explore the richness of the multi-tasking control formalism under constraints 32 in order to ensure a continuous transition from one control mode to another while guaranteeing compliance with a certain number of control constraints. Some of these constraints are based on ergonomic specifications and are dependent on the state of the robot and of the human operator, which, by nature, is difficult to predict accurately. We propose to exploit the interval analysis paradigm to efficiently formulate ergonomic constraints robust to the various existing uncertainties.
Purely discrete or reactive adaptation of the control law would make no sense given the slow dynamics of certain physiological phenomena such as fatigue. Thus, we propose to formulate the control problem as a predictive problem where the impact of the control decision at a time is anticipated at different time horizons. This requires a prediction of human movement and knowledge of the motor variability strategies it employs. This prediction is possible on the basis of the supervision at all times of the operational objectives (task in progress) in the short term. However, it requires the use of a virtual human model and possibly a dynamic simulation to quantify the impact of these potential movements in terms of performance, including ergonomics. It is impractical to use a predictive command with simulation in the loop with an advanced virtual manikin model. We therefore propose to adapt the prediction horizon and the complexity of the corresponding model in order to guarantee a reasonable computational complexity.
The planned developments require both an approach to modelling human sensorimotor behaviour, particularly in terms of accommodating fatigue via motor variability, and validating related models in an experimental framework based on observation of movement and quantification of ergonomic performance. Experimental developments must also focus on the validation of proposed control approaches in concrete contexts. To begin with, the Woobot project related to gesture assistance for carpenters (Nassim Benhahib's thesis) and a collaboration currently being set up with Safran on assistance to operators in shrink-wrapping tasks (manual knotting) in aeronautics are rich enough background elements to support the research conducted. Collaborative research projects with PSA will also soon provide a larger set of contexts in which the proposed research can be validated.
4 Application domains
4.1 Factory 4.0
The 4th industrial revolution (factory 4.0) is characterized by the integration of digital technologies into the production process, in order to meet the challenge of customizing services and products. This agility requires making manufacturing and maintenance lines flexible and versatile. This capacity for adaptation is the characteristic of the human being, which puts him at the center of the production apparatus. However, this can no longer be done at the expense of their health and well-being. How then can we reconcile the enhancement of our manual and analytical expertise, the ever desired increase in productivity and manufacturing quality, while reducing the hardship at work? Collaborative robotics, which we are seeking to build, is one of the central solutions to meet this societal challenge. By assisting humans in their most dangerous and painful tasks, it complements or replaces them in their phases of physical and cognitive fragility.
More generally, we are interested in workstation cobotization, in the manufacturing and assembly industry but also in the construction and craft industries. The application areas are related to regional needs in aeronautics, including maintenance, water and waste treatment. In most of these cases, it is possible to define the tasks, evaluate the stakes and added value of our work.
5 Highlights of the year
5.1 Start-up creation
In january 2020, the start-up "Touch sensity" was created from researches carried out within the Auctus project team on innovative technologies of sensitive and connected materials (http://
In the framework of the WooBot project which aims at securing carpentry tasks by collaborative robots, Nassim Benhabib, Vincent Padois and David Daney received the 2020 IEEE ICRA Best Paper Award in Automation for their contribution entitled "Securing Industrial Operators with Collaborative Robots: Simulation and Experimental Validation for a Carpentry task" 10.
5.3 High media impact publications
The paper of Jean-Marc Salotti (Salotti, JM. Minimum Number of Settlers for Survival on Another Planet. Sci Rep 10, 9700 (2020). https://
5.4 Learned societies
In october 2020, Nasser Rezzoug was elected at the board of directors of the "Société de Biomécanique". (https://
6 New software and platforms
6.1 New software
- Keywords: Robotics, Dynamic Analysis
- Functional Description: WoobotSim is a simulator that reports the dynamics of the parties involved in an industrial task implying a strong interaction between a machine tool, an operator and a handled object, it also offers the possibility to add a cobot as an actor. Developed on Matlab, this simulator allows to visualize the efforts exchanged by the participants during the task, as well as the dynamics of the object being manipulated. For the specific case of woodworking shaper. It includes a wood cutting model. A model of task control by the craftsman and a model of the robot.
- Contacts: Nassim Benhabib, Vincent Padois
- Keywords: Control, Robot Panda, Synchronization
WoobotCtrl is a set of ROS packages that provide control of a collaborative robot with 7 degrees of freedom (Franka's PANDA) in order to reproduce the cutting forces experienced by an operator during a wood shaping task.
Developed in C++, WoobotCtrl also manages the real-time acquisition and synchronization of data from multiple sensors (Kinect, force sensors, robot sensors). As well as the pre-shrinking of the data.
- Contacts: Nassim Benhabib, Vincent Padois
- Name: rtt_panda
- Keywords: OROCOS component, Robot Operating System (ROS), Robotics, Simulation
A set of tools to control a robot with the OROCOS toolchain software.
Implemented tools: a library using KDL to modelize the robot , a wrapper for qpOASES , a Cartesian trajectory generator using KDL , a cartesian PID controller , a library to publish OROCOS port to ROS topics , a driver for the Panda Franka Emika robot , a Gazebo plugin , a set of tool to launch the scripts for the panda
gitlab. inria. fr/ auctus/ panda/ rtt_panda
- Author: Lucas Joseph
- Contacts: Lucas Joseph, Vincent Padois
- Name: torque_qp
- Keywords: Robot Operating System (ROS), Robot Panda, Simulation, Robotics
- Scientific Description: Formulation of a control problem as a constrained optimization problem at the joint torque level.
- Functional Description: This software is a control architecture for the Franka Emika Panda robot. The control problem is formulated as a constrained optimization at the joint torque level. This software is interfaced with Gazebo for simulaton, and the franka_ros packages for sending the desired velocity to the real robot.
gitlab. inria. fr/ auctus/ panda/ torque_qp
- Author: Lucas Joseph
- Contacts: Lucas Joseph, Vincent Padois
- Name: velocity_qp
- Keywords: Robotics, Robot Operating System (ROS), Robot Panda, Simulation
- Scientific Description: Formulation of a control problem as a constrained optimization problem at the joint velocity level.
- Functional Description: This software is a control architecture for the Franka Emika Panda robot. The control problem is formulated as a constrained optimization at the joint velocity level. This software is interfaced with Gazebo for simulaton, and the franka_ros packages for sending the desired velocity to the real robot.
gitlab. inria. fr/ auctus/ panda/ velocity_qp
- Author: Lucas Joseph
- Contacts: Lucas Joseph, Vincent Padois
- Name: Kinematic Chain Appropriate Design Library
- Keywords: Interval analysis, Uncertainty, Kinematics
- Functional Description: Software for the modelling and analysis of imprecise serial kinematic chains. Chain objects are built by iteratively adding rigid-body segments with associated joint connections (e.g., fixed, revolute, prismatic). Several standard options are provided to model each segment (e.g., Denavit-Hartenberg parameters, transformation matrices, twists). Each option accepts interval and non-interval arguments, allowing to model the uncertainties and variabilities of imprecise serial kinematic chains and also the conventional precise serial kinematic chains. Forward Kinematic (FK) and Inverse Kinematic (IK) solvers are available for Chain objects. The FK solver computes an outer bound of the set of poses associated with a set of joint configurations. The IK solver computes an outer bound of the set of joint configurations associated with a set of poses.
gitlab. inria. fr/ auctus/ kinematic-chain-appropriate-design-library
- Publication: hal-02367664, version 1
- Contacts: Joshua Pickard, David Daney, Nasser Rezzoug, Vincent Padois
- Participants: Joshua Pickard, David Daney, Nasser Rezzoug, Vincent Padois
- Name: On-line force capability evaluation based on efficient polytope vertex search
- Keywords: Robotics, Kinematics, SVD, Real-time application
- Scientific Description: Ellipsoid-based manipulability measures are often used to characterize the force / velocity task-space capabilities of robots. While computationally simple, this approach largely approximate and underestimate the true capabilities. Force / velocity polytopes appear to be a more appropriate representation to characterize robot's task-space capabilities. However, due to the computational complexity of the associated vertex search problem, the polytope approach is mostly restricted to offline use, e.g. as a tool aiding robot mechanical design, robot placement in work-space and offline trajectory planning. In this paper, a novel on-line polytope vertex search algorithm is proposed. It exploits the parallelotop geometry of actuator constraints. The proposed algorithm significantly reduces the complexity and computation time of the vertex search problem in comparison to commonly used algorithms. In order to highlight the on-line capability of the proposed algorithm and its potential for robot control, a challenging experiment with two collaborating Franka Emika Panda robots, carrying a load of 12 kilograms, is proposed. In this experiment, the load distribution is adapted on-line, as a function of the configuration dependant task-space force capability of each robot, in order to avoid, as much as possible, the saturation of their capacity.
- Functional Description: Software implementing new online capable polytope vertex search algorithm developed for force capacity evaluation of serial robotic manipulators, described in the paper: On-line force capability evaluation based on efficient polytope vertex search. The algorithm is capable of finding the vertices of force polytopes for 7DOF robot under 3 ms and 6DOF robot under 2 ms. Software repository consists of Matlab/Octave implementation intended for fast prototyping and benchmarking as well as Python module for easier integration in larger projects.
gitlab. inria. fr/ askuric/ polytope_vertex_search
- Publication: hal-02993408
- Contacts: Antun Skuric, Vincent Padois, David Daney
- Name: keyence_laser_tracker
- Keyword: Driver
- Functional Description: A ROS package interfacing the Keyence SZ-V32N device. It allows the configuration of the device and periodically get the information distance measured by the device through UDP. A ROS message is published with all the laser information.
gitlab. inria. fr/ auctus/ panda/ keyence
- Author: Lucas Joseph
- Contacts: Lucas Joseph, Vincent Padois
- Keywords: 3D, Robotics, Biomechanics
Opensim is a motion modelling and simulation tool, mainly used in biomechanics. It addresses a number of issues related to the manipulation of biological bodies (calculation of muscular forces, precise modelling of musculoskeletal models based on experimental data, management of data from motion capture systems).
A number of robotics projects linking robots and humans could benefit from the functionalities of this tool for modelling the upper body of a human being in real time. This is why the objective of this project is to combine some of the functionalities of Opensim with ROS packages dedicated to the modelling, control and analysis of the movement of polyarticulated robotic systems (torque_qp, Pinocchio).
One of the paths currently being explored is to use OpenSim and ROS packages together for the same model of a human being. As the modeling format of Opensim models (.osim) is not the same as the one classically used by the ROS packages (.urdf), it is first necessary to convert the .osim format to .urdf. A library is being created for this purpose.
Once this library is completed, it will be possible to create the framework allowing the use of OpenSim functionalities to the packages used for the management of polyarticulated systems in real time, in particular for motion capture problems.
- Contacts: Erwann Landais, Vincent Padois, Nasser Rezzoug
6.2 New platforms
In 2020, Auctus has pursued his effort of developping its experimental platform of collaborative Robotics (Arcol).
The Arcol platform provides technical support for the short, medium and long term experimental developments carried out within the framework of Auctus' scientific and dissemination activities. These technological developments are essentially software related in the context of human motion capture and real-time control of collaborative robots. Arcol aims at easing their implementation, deployment, documentation and support.
New developpements in 2020 are mostly related to:
- the real-time capability of the software architecture dedicated to the control of our two 7 degrees of freedom Panda robot (from Franka Emika);
- the ongoing effort related to the generic integration of sensors and robots in the platform through the use of middlewares such as ROS and Orocos;
- the ongoing effort to converge towards an online motion capture system connected to model computation softwares allowing for the online estimation of human capabilities.
7 New results
7.1 HARRY² (INRIA + Region fundings)
Within the framework of the continuation of the HARRY² project funded by the COVR initiative (H2020 program) and by Région Nouvelle Aquitaine, impact testing procedures have been evaluated and a report analysing the obtained results has been submitted as second Milestone of the project. This Milestone was validated by the COVR consortium and a promotional video summarizing the progress of the project has been released2.
The work related to the first milestone has been published in 12. This work is being extended to online trajectory generation in Cartesian space using a Model Predictive Control formulation. This formulation allows the definition of a trajectory taking into account the robot limits in Cartesian space over a time horizon. The objective is to combine this algorithm with interval analysis algorithms allowing to determine the robot capacities over a time horizon given its current configuration. These algorithms will take into account the robot physical capabilities i.e. joint limits, joint velocity limits, etc. and send back the Cartesian acceleration limits over a time horizon as a function of the current configuration. These limits can be directly inserted in the MPC problem. This will allow to generate trajectories online that respect the robot true capabilities. Additional constraints could then be added such as obstacle avoidance.
Related publications: 12
7.2 Control architecture
A robot low level control architecture requires a good handling of the real time communication between each components. The ROS middleware is not really adapted for such usage. Therefore, a new control architecture has been developed around the Orocos toolchain software. This control architecture allows to define trajectories in Cartesian space, compute the robot model, update it online, compute the desired command and send it to a Franka Emika Panda robot. It also interfaces with the Gazebo simulation environment. This architecture is modular and works with components. The communication between each component is strictly enforce and true real time synchronization is achieved.
7.3 On-line force capability evaluation based on efficient polytope vertex search
Ellipsoid-based manipulability measures are often used to characterize the force/velocity task-space capabilities of robots. While computationally simple, this approach largely approximates and underestimates the true capabilities. Force/velocity polytopes appear to be a more appropriate representation to characterize robot's task-space capabilities. However, due to the computational complexity of the associated vertex search problem, the polytope approach is mostly restricted to offline use, e.g. as a tool aiding robot mechanical design, robot placement in work-space and offline trajectory planning.Therefore a new on-line polytope vertex search algorithm is proposed, exploiting the parallelotop geometry of actuator constraints. The proposed algorithm significantly reduces the complexity and computation time of the vertex search problem in comparison to commonly used algorithms. The algorithm is intended to be used with redundant serial robotic manipulators and opens many doors for using this force/velocity capacity polytopes for robot control and human-robot interaction. The algorithm execution time for 7DOF Franka Amika Panda and for 6DOF Universal Robots UR5 robots is under 3 and 2 milliseconds respectively. In order to highlight the on-line capability of the proposed algorithm and its potential for robot control, a challenging experiment with two collaborating Franka Emika Panda robots, carrying a load of 12 kilograms, was developed. In this experiment, the load distribution is adapted on-line, as a function of the configuration dependant task-space force capability of each robot, in order to avoid, as much as possible, the saturation of their capacity.
Related publications: 17
7.4 Synthesis and dynamic analysis of the shared workspace for safety in collaborative robotics
To obtain a safe and effective shared workspace between humans and robots, safety needs to be taken into account both at the design phase and at control algorithm. While concentrating in the control law, collaborative robot solutions ignore its capacity of detecting human presence and interaction such as interrupters that completely turn off the system.
The objective of this project is to work on these principles to avoid the classic restrained static security zones, synthesizing a dynamic representation of the shared workspace to take advantage of state of the art control laws, allowing a fluid collaboration between human-robot. This dynamic synthesis requires knowledge of the robots state (geometric, cinematic and cognitive, such as fatigue, expertise and conscience of the situation), its tasks, capacities, state of humans that surround it and their tasks. Furthermore, it needs to achieve a formal and provable online algorithm that correctly estimates the state of the human and guarantee a safe shared workspace tackling ambitious scientific questions poorly addressed in literature.
An industrial collaboration has been initiated with PSA Group on the synthesis and dynamic analysis of shared workspaces for safety in collaborative robotics. A CIFRE PhD thesis has been approved by ANRT and started on April 2020.
The first stage of the project is focused on the design of a reactive robot control scheme that adapts to its environment to ensure safety of surrounding human operators without sacrificing performance. The controller needs to account for the robot's dynamic braking capabilities and the actuator's constraints. Constructing on the previous work by AUCTUS 12, the current effort is directed towards improving the online estimation of the robot's capabilities to adapt its behaviour as operators get closer/farther to it.
7.5 Design of a analysis methodology of a manual task for the design of a collaborative robotics assistance adaptable to the level of expertise of an industrial operator
In this work, a methodology for analyzing the gesture of an industrial operator is developed in order to design a collaborative robotics assistance adaptable to the needs of the worker. Wood milling is chosen as a pilot task due to its importance in carpentry.
In order to analyze the operator's gesture during wood shaping, a physical simulation of the task was reproduced. Where, a collaborative robot reproduces the cutting forces exerted on the operator.
Force, posture and motion sensors are used to measure the quantities of interest related to performance, safety and comfort criteria. We hypothesize that the evolution of these multiple criteria according to the operator's experience gives a clear indication of the operator's level of expertise.
In our opinion, the needs of an industrial operator vary according to his level of expertise. Identifying this level of expertise would allow us to design more efficient and acceptable robotic assistance.
Two quantifiable assessment criteria have been established for each aspect of task completion (performance, safety and comfort). By observing their evolution at different levels of experience for multiple users . We noticed that a trend was emerging. This helps to analyze the preferences of an operator and to observe which aspects of the task completion are favored as the expertise is acquired.
Related publications: 10
7.6 Study of Motor Variability
In the context of reducing the risk of musculoskeletal disorders (MSDs) in industry, the question of motor variability (VM) of industrial operators has to be raised. Indeed, VM could promote the distribution of effort and fatigue in the body through variations in motor strategies during movement. By optimising this VM, it may be possible to spare the muscles, joints and skeleton of the operators. A project, carried out through the PhD thesis of Raphaël Bousigues in collaboration between INRIA Bordeaux, INRIA Nancy and INRS, aims at verifying a set of hypotheses, such as, first of all, the fundamental existence of VM. Reproducing a repetitive manual task with a same operator allows to observe the kinematics of his arm and to quantify the presence and evolution of motor variability through the various repetitions of the task. The experimental protocol of this experiment is in the process of writing, but a simulation of a human arm performing a repetitive task has already been put into practice. Once the first experimental data will have been acquired, it will be possible to focus on the origin of VM and on the conditions influencing it. An experimental framework, entitled Mover, has been submitted to the COERLE for the experimetal work related to this work.
7.7 Human motion analysis in ecological environment
The estimation of human motion from sensors that can be used in an ecological environment is an important issue being it for home assistance for frail people or for human/robot interaction in industrial contexts. We are continuing our work on data fusion from RGB-D sensors using extended Kalman filters. The original approach uses a biomechanical model of the person to obtain anthropomorphically constrained joint angles to make their estimation physically coherent. In addition, we propose a method for the optimal adjustment of the covariance matrices of the extended Kalman filter. The proposed approach was tested with six healthy subjects performing 4 rehabilitation tasks. The accuracy of the joint estimates was evaluated with a reference stereophotogrammetric system. Our results show that an affordable RGB-D sensor can be used for simple home rehabilitation when using a constrained biomechanical model. This work has led to an accepted article in MDPI Sensors: Physically Consistent Whole-Body Kinematics Assessment Based on an RGB-D Sensor. Application to Simple Rehabilitation Exercises.
In a second step, we compared the joint centre estimates obtained with the new Kinect 3 (Azure Kinect) sensor, the Kinect 2 (Kinect for Windows) and a reference stereophotogrammetric system. Regardless of the system used, we have shown that our algorithm improves the body tracker data. This study also shows the importance of defining good heuristics to merge the data according to the body tracking operation. This study was accepted in International Conference on Activity and Behavior Computing and presented by Jessica Colombel in Kitakyushu, Japan as Markerless 3D Human Pose Tracking in the Wild with fusion of Multiple Depth Cameras: Comparative Experimental Study with Kinect 2 and 3. This article will also be a chapter book in Activity and Behavior Computing by Springer.
Related publications: 14
7.8 Human motion decomposition
The aim of the work is to find ways of representing human movement in order to extract meaningful physical and cognitive information. After the realization of a state of the art on human movement, several methods are compared: principal component analysis (PCA), Fourier series decomposition and inverse optimal control. These methods are used on a signal comprising all the angles of a walking human being. PCA makes it possible to understand the correlations between the different angles during the trajectory. Fourier series decomposition methods are used for a harmonic analysis of the signal. Finally, inverse optimal control sets up a modeling of the engine control to highlight qualitative properties characteristic of the whole motion. These three methods are tested, combined and compared on data from the EWalk database3 in order to test emotion recognition based on these decompositions and simple classifiers. An in-depth study of the inverse optimal control method has been carried out in order to better understand its properties and to develop easier methods of resolution. This study led to a robust resolution of the inverse optimal control based on SVD and orthogonal projection.
7.9 Portage project
In the Portage project, one of the modus operandi of using ashared robotized platform is to allow the remote operation, by a human, of heavy loads. During this year, our task was to analyze the human-system interactions and to provide the technical specifications of the system. More precisely, the objective was to define the uses (including the maintenance task), the interaction modalities for 1 to 3 people and the control interfaces of the platform, paying attention to safety and contingencies. As this task is closely linked to the task of empowerment, particular attention was paid to the control or regaining of control by the operator following the abandonment of a task that could theoretically be performed autonomously. The evaluation of the scenarios makes it possible to predetermine the number of operators assigned to the task.
We have contributed on three main topics :
- System use scenarios (hybrid control of the robot);
- The design, the modeling and the realization of the interface itself;
- The experimental protocol implemented to study the impact of the system on the work of journey persons on the shop floor.
Accident report analysis
We analyzed industrial robotics accidents from the INRS database EPICEA. We adopted an approach based on interactions in a robotic workspace, coupled with Endsley’s model of Situation Awareness (SA) to identify factors that potentially degrade operators’ SA. We analyzed 45 industrial accident reports involving robotic. Our analysis showed that SA demons were mainly associated with the perception level of SA. Additionally, SA demons are generally involved in human-robot interaction and human-environment interaction. We have also been able to describe five patterns of SA demon occurrence. At the end, we could propose recommendations regarding workspace design.
7.10 Interactions with a chatbot
In the context of the CIFRE Orange PhD work by Nicolas Simonazzi under the supervision of Jean-Marc Salotti and with the objective of analyzing and identifying emotions during interactions with a chatbot, a first experiment was conducted. It involved a user, the use of a smartphone, viewing videos and asking questions about the content of the video and the feeling of the user just after the answer. The collected data were numerous: the accuracy of the answers to the questions, the emotional feeling (choice of emoticons by the user) as well as the real-time measurements of the accelerometer of the smartphone. An analysis of the data was carried out with Russell's relatively simple emotional model as an explanatory framework based on two variables, the positive or negative valence of the emotion, and the degree of excitement. The experimental results showed that there was a slight correlation between the valence indicated by the user, the accuracy of the answers to the questions and the accelerations of the smartphone. However, it was hoped that the videos would have an impact on the valence, because their content had an intrinsic valence, but it proved impossible to find a correlation with the valence indicated by the user, probably due to a lack of the user engagement and also because of the focus on the questions that followed and the accuracy of the answers. A new experimental protocol is currently being studied with a priori more impactful videos (likely to produce an emotion with a greater degree of excitement).
Related publications: 15
7.11 Human motor caracterization from hand movement tracing
In collaboration with the Research Groups in Intelligent Machines, (National School of Engineers of Sfax ENIS, University Sfax, Tunisia), we seek to apply modeling trajectory formalisms to identify the state of a subject or operator from hand movement tracing (writing, drawing or hand trajectory). Nowadays, motion capture (kinect, graphical tablets e.g.) associated with appropriate signal processing allow an objective analysis of these trajectories, 1) for documenting the effect of ageing and highlighting the neural, motor and sensory effects, 2) to develop new clinical tests for neurodegenerative disease early detection, 3) for more engineering-oriented applications such as signature automatic recognition, authentication for forensics, 4) to assess the state of an operator during hand movement execution (fatigue). The proposed model (the beta elliptic model) is in a phase of validation for detecting changes in hand movement during ellpse drawing on a database of 99 subjects with age ranging frm 19 to 85 years.
8 Bilateral contracts and grants with industry
8.1 Portage (Akka)
The global objective of this project is to develop a semi-autonomous carrier dedicated to the transport of heavy structures in industrial factories. The Auctus team has been assigned the role of task analysis and human systems interactions analysis in order to determine the best interface, to improve ergonomics, to reduce risks and to account for acceptability. A postdoctoral student, Charles Fage, has been recruited for the first year of the study (10/2019-10/2020). A 2-years contract (2019-2021) has been signed with AKKA Technologies as part of a consortium, which included two other companies, IIDRE and Ez-Wheel, and another research team from IMS laboratory. The total amount of the grant from Akka to Auctus is 110 kEuros.
Project in a nutshell:
- Consortium: AUCTUS@Ims, AKKA, EZ-WHEEL, IIDRE, IMS (productique team)
- Funding: Akka (Auctus is subcontractor)
- Duration: 2020–2023
- Researchers involved: David DANEY, Jean-Marc SALOTTI
In 2019, a contract was signed with Orange for a PhD on emotions detection in chatbot interactions. The PhD student is Nicolas Simonazzi under the supervision of Jean-Marc Salotti. The objective is analyzing and identifying emotions during interactions with a chatbot. For the company, an important issue is indeed to be able to adapt the response of the chatbot on the website of the customer service. The study was further on focused on the use of smartphones and the identification of emotions based on accelerometers and move patterns. The associated grant was 10 kEuros per year during the 3-years period of the PhD.
Project in a nutshell:
- Consortium: AUCTUS@Ims, Orange
- Funding: Orange + ANRT (CIFRE)
- Duration: 2018–2021
- Researchers involved: Jean-Marc Salotti and Nicolas Simonazzi
8.3 LiChIE (Airbus)
The LiChIE project (funded by BPI) aims to design a constellation of mini-satellites for optical Earth observation. Among many other topics, this requires to rethink the way sattelites are being produced in order to ease this highly complex process. There is actually an unprecedented economical and societal demand for robots that can be used both as advanced and easily programmable tools for automatizing complex industrial operations in contexts where human expertise is a key factor to success and as assistive devices for alleviating the physical and cognitive stress induced by such industrial task. Unfortunately, the discrepancy between the expectations related to idealized versions of such systems and the actual abilities of existing so-called collaborative robots is large. Beyond the limitations of existing systems, especially from a safety point of view, there are very few methodological tools that can actually be used to quantify physical and cognitive stress. There is also a lack of formal approaches that can be used to quantify the contribution of collaborative robots to the realization of industrial tasks by expert operators. Of course, in the state-of-the-art, existing works in that domain do consider some aspects of the current state of the operator in order to propose an appropriate robot behaviour. One of their conceptual limitations is to consider an a priori defined human-robot collaboration scenario where the expertise of the human operator is of importance but limited to a single operation. The consideration of larger varieties of tasks is rarely considered and, when it is, only a strict separation of the tasks to be achieved by each member of the human-robot dyad is considered. In this project, we propose to develop a coupled model of human-robot physical abilities that does not make any a priori with respect to the type of assistance. This requires to develop a parameterisable generic model of the potential physical link and implied constraints between the human operator and the robot. This model should allow to describe the task to be achieved by the human alone or using a collaborative robot through different interaction modalities. Online simulation of these scenarios coupled with ergonomic and performance indicators should both allow for the discrete choice of the right assistance mode given the task currently being achieved as well as for the continuous modulation of the robot behaviour.
Project in a nutshell:
- Consortium : AUCTUS@Inria, Airbus
- Funding : BPI
- Duration : 2020 – 2023
- Researchers involved : Antun Skuric (PhD Student), Vincent Padois (thesis advisor) and David Daney (thesis advisor)
The objective of this project is to work on the required principles to avoid the classic restrained static security zones, synthesizing a dynamic representation of the shared workspace to take advantage of state of the art control laws, allowing a fluid collaboration between human-robot. This dynamic synthesis requires knowledge of the robots state (geometric, cinematic and cognitive, such as fatigue, expertise and situation awareness), its tasks, capacities, state of humans that surround it and their tasks. Furthermore, it needs to achieve a formal and provable online algorithm that correctly estimates the state of the human and guarantee a safe shared workspace tackling ambitious scientific questions poorly addressed in literature.
Project in a nutshell:
- Consortium: AUCTUS@Inria, PSA Automobiles
- Funding: PSA Automobiles, ANRT (CIFRE)
- Duration: 2020–2023
- Researchers involved: Nicolas TORRES ALBERTO, Vincent PADOIS, David DANEY
9 Partnerships and cooperations
9.1 International initiatives
9.1.1 Inria international partners
- A collaboration has been carried on with Ephraim Suhir (Portland University) in the domain of robotics risks, and especially on the issue of availability of the robot. Related publication: J.-M Salotti, J Nicolics, E. Suhir. Required Repair Time to Assure the Given/Specified Availability. Universal Journal of Lasers, Optics, Photonics & Sensors, 1, 01-07, 2020.
Joshua Pickard, University of New Brunswick and Eigen Innovations inc and Milan Hladik, Charles University
Research topic of the collaboration: Set-Based Approaches for the Reliable Computation of Robot and Human Capabilities. Publications related to this collaboration: 2 in preparations.
Alessandro Saccon, Assistant Professor in nonlinear control and robotics at the Mechanical Engineering Department, Eindhoven University of Technology (TU/e)
Research topic of the collaboration: Modelling and prediction of robots impacting their environment Publications related to this collaboration: 1 in preparation Student projects: co-advising of 2 M.Sc. students projects.
Clint Hansen, Christian-Albrechts-Universität zu Kiel Department Universitätsklinikum Schleswig - Holstein
Research topic of the collaboration: We are working together on algorithmic assessment of locomotion by means of inertial sensors.
Houcine Boubaker, Thameur Dhieb, Research Groups in Intelligent Machines, National School of Engineers of Sfax ENIS, University Sfax, Tunisia: We are working on modeling of hand-written tracing.
Research topic of the collaboration: The objective is to propose signal processing tools to assess the state of a subject from manuscript production and hands moves (age, pathologies, tiredness, ...).
Olfa Jemaa, Sami Bennour, University of Sousse and Lotfi Romdhane, American University of Sharjah
Research topic of the collaboration: the individual musculo-tendon forces for ergonomic assessment
9.2 European initiatives
9.2.1 Collaborations in European programs, except FP7 and H2020
The objective of the HARRY2 project is to attain more advanced workspace sharing capabilities through fully exploiting the collaborative possibilities defined by ISO TS 15066. This is achieved by:
- Developing PLC software and motion controllers using robot-agnostic industrially-rated components to ease and standardize the development of safe robotic applications with workspace sharing.
- Integrating state-of-the-art energy-based control algorithms using these industrial hardware components, so that safety is no longer treated as an exception but considered as a constraint when computing the control solution in real-time.
- Enabling the use of high-level and intuitive teaching interfaces reducing robot programming time and difficulty.
- Developing a systematic and practical methodology for quantitative safety evaluation.
Project in a nutshell:
- Consortium: AUCTUS@Inria, RoBioSS@PPRIME, FuzzyLogicRobotics
- Funding: H2020 COVR, Région Nouvelle-Aquitaine
- Duration: 2019–2021
- People involved: Lucas Joseph, Vincent Padois
- Dissemination: Video, Harry2@COVR overview (pdf), Twitter
9.3 National initiatives
The general objective of the project is to design a semi-autonomous cobotic system for assistance, able to choose, synchronize and coordinate tasks distributed between humans and robots by adapting to different types of variability in professional gestures, all by anticipating dangerous situations. The orchestration of tasks between man and robot is difficult because it must answer the question of the distribution of roles within the couple according to physical and decision-making skills and constraints as well as the consequences of their interactions. However, we cannot put the two actors at the sae level: the robot has to adapt his actions to the work of an operator and, more precisely, to his motor and cognitive strategies that materialize through the quantifiable variability of professional actions. On the other hand, the very interest of the robot is to assist the operator in his phases of fragility while preserving his physical and mental integrity, in particular considering that human error is inherent in operator action. These considerations are, for Pacbot, the conditions necessary for the joint achievement of efficient work.
Partners of the ANR project:
- Auctus (Leader),
- Auctus@people involved: David Daney (principal investigator), Jean-Marc Salotti, Benjamin Camblor.
- Laboratoire Informatique de Grenoble
- Laboratoire Interuniversitaire de Psychologie (Lyon)
- ANR Funding for Auctus: 246,240 Euros.
- Duration: 3 years.
- Start: January 2021.
9.4 Regional initiatives
MIELS, in partnership with Solvay, Laboratoire du Futur (Pessac)
The main objective of the MIELS project is to develop innovative strategies to characterize and develop neoteric, non-toxic solvents through strategies that will enable to grasp the enormous quantity of required experimental tests all in insuring an absolute safety of the manipulator. For this purpose, we intend to work on two complementary routes, the development of solvent characterization methodologies and the integration of a cobotic approach in solvent handling and evaluation, with the ambition of merging these developments at the end of the project in order to draw as much synergy as possible. This project is built around teams with complementary competencies to achieve these objectives. UMR LOF and Solvay LOF have great experience acquired over several years of research the fields of solvent evaluation and robotics, whereas Auctus INRIA team has a strong expertise in collaborative robotics. By combining our competencies and expertise, the MIELS project aims to merge all these fields in order to expand 1) the fundamental study of solvents and their characterization techniques including theoretical techniques, in particular for green solvents and 2) the development of the use of cobotics, in collaboration with Auctus INRIA team, for increasing the efficacy and safety of laboratory workers in industry, in particular those working on characterization of solvents.
Project in a nutshell:
- Consortium : AUCTUS@Inria, Solvay, LOF
- Funding : Région Nouvelle Aquitaine, Solvay
- Duration : 2020 – 2024
- Researchers involved : Vincent Padois and David Daney
The main goal of the work is to develop a design methodology (control and physical architecture) of collaborative robotic systems dedicated to assisting and securing operator expert gestures in the context of industrial applications involving strong physical interaction between a human, a tool and a manipulated object. The collaborative system should improve ergonomics and safety while maintaining the level of industrial performance and preserving the user’s know-how. The application context is the carpentry, in particular wood shaping.
Future work will include a more complex human model in order to quantify the impact of potential assistance architectures according to relevant criteria of industrial performances, ergonomics/safety and user expertise.
This project is conducted by Nassim Benhabib, Vincent Padois and David Daney. With the collaboration of BTP CFA de la Gironde. This allows to provide realistic information of the task studied and to collect the feedback from the craftsmen.
Project in a nutshell:
- Consortium : AUCTUS@Inria, BTP-CFA-Gironde
- Funding : Région Nouvelle Aquitaine, INRIA
- Duration : 2019 – 2022
- Researchers involved : Nassim Benhabib (PhD Student), Vincent Padois (thesis advisor) and David Daney (thesis advisor)
10.1 Promoting scientific activities
10.1.1 Scientific events: organisation
General chair, scientific chair
- Vincent Padois: Organizer and chair of the National Days of Humanoid Robotics4, June 2020.
Member of the organizing committees
- Vincent Padois: Scientific animation of the humanoid robotics session at the yearly seminar of the GDR Robotique5, November 2020.
10.1.2 Scientific events: selection
- David Daney: Reviewer for ICRA, IROS.
- Jessica Colombel: Reviewer for ICRA.
- Vincent Padois:
- Associate Editor for ICRA and IROS;
- Reviewer for ICRA, IROS and RSS.
- Nasser Rezzoug:
- Scientific commitee of the 45th Congress of the" Société de Biomécanique";
- Reviewer for IEEE EMBS and the 45th Congress of the "Société de Biomécanique".
Member of the editorial boards
- Vincent Padois: Section Editor for the Springer Encyclopedia of Robotics.
Reviewer - reviewing activities
- David Daney: Reviewer for Mechanism and Machine Theory, Journal of Mechanisms and Robotics, IEEE Transactions on Robotics.
- Vincent Padois: Reviewer for Frontiers in Robotics and AI, IEEE-RAS RA-Letters, Elsevier journal of Robotics and Computer-Integrated Manufacturing, IEEE Transactions on Robotics.
- Nasser Rezzoug: Reviewer for Journal of Biomechanics, Sensors, Applied Sciences, Ijerph.
- Jean-Marc Salotti: Reviewer for Acta Astronautica, Human-Intelligent Systems Integration, Journal of Spacecraft and Rockets, Advances in Space Research, Behavioural Brain Research, Foresight.
10.1.4 Leadership within the scientific community
- Vincent Padois: Co-Head of the Working Group (GT7) in Humanoid Robotics of the GDR Robotique with Olivier Stasse.
10.1.5 Scientific expertise
- David Daney
- reviewer for Natural Sciences and Engineering Research Council of Canada
- Vincent Padois
- Reviewer for the 3rd COVR H2020 project call;
- Reviewer for the NWO Talent Programme – Veni;
- Scientific expertise for IRT Jules Verne.
- Nasser Rezzoug
- Reviewer for the ANR generic project call, Technology for health;
- Reviewer for the 2020 Dutch Research Council, Domain Applied and Engineering Sciences project call
10.2 Research administration
- David Daney is the principal investigator of ANR Pacbot
10.3 Teaching - Supervision - Juries
- Master: Jean-Marc Salotti, Intelligence Artificielle, 103,5h éqTD, M1, Ecole Nationale Supérieure de Cognitique / Bordeaux INP, France.
- Master: Jean-Marc Salotti, Facteurs Humains et Ingénierie Cognitique, 15h éqTD, M1, Ecole Nationale Supérieure de Cognitique / Bordeaux INP, France.
- Master: Jean-Marc Salotti, Interactions Hommes Robots, 15h éqTD, M2, Ecole Nationale Supérieure de Cognitique / Bordeaux INP, France. In this course, all students have practical works involving cobotic systems: programming NAOs and UR3 (Universal Robots) and testing an exoskeleton.
- Master: David Daney, Interactions Hommes Robots, 2h éqTD, M2, Ecole Nationale Supérieure de Cognitique / Bordeaux INP, France.
- Master: David Daney, Mathématiques pour la robotique, 30h éqTD, M2, Enseirb/Ensc, Bordeaux INP, France.
- Master: David Daney, Décision, 7h éqTD, M2, Ecole Nationale Supérieure de Cognitique / Bordeaux INP, France.
- Master: Vincent Padois, Literature review - What, Why and How?, 20h éqTD, M2, Enseirb/Ensc, Bordeaux INP, France.
- Master: Vincent Padois, Student projects, 40h éqTD, M2, Enseirb/Ensc, Bordeaux INP, France.
- Master: Vincent Padois, Maintenance du futur - Cours introductif, 2h éqTD, M1, ENSPIMA, Bordeaux INP, France.
- Master: Vincent Padois, Introduction à la Robotique, 2h éqTD, M1, Universié de Bordeaux, France.
- Master: Nassim Benhabib, Interactions Hommes Robots, 15h éqTD, M2, ENSC, Bordeaux INP, France.
- Master: Jessica Colombel, Robot Operating System, 10h éqTD, M2, Enseirb/Ensc, Bordeaux INP, France.
- Master: Jessica Colombel, Traduction des langages (compilation), 18h éqTD, M1, TELECOM Nancy, France.
- Master: Jessica Colombel, Conception et développement Java, 10h éqTD, M1, TELECOM Nancy, France.
- Licence: Jessica Colombel, Mathématiques pour l'informatique, 30h éqTD, L3, TELECOM Nancy, France.
- Licence: Jessica Colombel, Base de données, 17h éqTD, L3, TELECOM Nancy, France.
- Licence: Jessica Colombel, Culture et outils informatique (Latex et Git), 8h éqTD, L3, TELECOM Nancy, France.
- Master: Charles Fage, Cognitive Disabilities and Assistive Technologies, , M2, Master de Sciences cognitives et Ergonomie, Université de Bordeaux, France.
- Master: Charles Fage, Human-Computer Interactions, 15h éqTD, M1, Master de Sciences cognitives et Ergonomie, Université de Bordeaux, France.
- Licence: Charles Fage, Disabilities Management and Ethics, 15h éqTD, L2, Licence de MIASHS, Université de Bordeaux, France.
- Licence: Charles Fage, Alternative and Augmentative Communication, 15h éqTD, 2ème année, Institut de formation en ergothérapie de Bordeaux, France.
- Licence: Charles Fage, Autism and Developmental Disorders, 15h éqTD, 2ème année, Institut de formation en ergothérapie de Bordeaux, France.
- Licence: Charles Fage, Aging and Assistive Technologies, 15h éqTD, 2ème année, Institut de formation en ergothérapie de Bordeaux, France.
- Master: Antun Skuric, Interactions Hommes Robots, 12h éq TD, M2, ENSC, Bordeaux INP, France.
- Master: Antun Skuric, Mathématiques et Informatique (Intermédiaire), 9h éq TD, M1, ENSAM, Bordeaux INP, France.
- Master: Antun Skuric, Mathématiques et Informatique (Avancé), 18h éq TD, M2, ENSAM, Bordeaux INP, France.
- Licence: Nasser Rezzoug, Biomécanique, 124.5héq TD, L1, UFR STAPS, Université de Toulon, France.
- Master: Nasser Rezzoug, Statistiques, 96h éq TD, L2, UFR STAPS, Université de Toulon, France.
- Master: Nasser Rezzoug, Biomécanique, 25h éq TD, M1, UFR Sciences & Tech, Master ISC parcours Robotique et Objets Connectés, université de Toulon, France.
- Master: Nasser Rezzoug, Applications biomédicales des objets connectés, 6h éq TD, M2, UFR Sciences & Tech, Master ISC parcours Robotique et Objets Connectés, Université de Toulon, France.
- PhD in progress: Quoc Bach Hoa (Thèse ED SMAER, Paris), “Biomechanically plausible simulations of healthy and altered human locomotion”, September 2017–, Vincent Padois and Faiz Ben Amar (Sorbonne Université).
- PhD in progress: Nassim Benhabib (Inria / Région NA – Woobot project), “Méthodologie de conception et de commande d'un système robotique collaboratif pour assister et sécuriser les gestes d'un opérateur”, November 2018 – , David Daney and Vincent Padois.
- PhD in progress: Pierre Laguillaumie (Thèse laboratoire PPRIME), “Méthodologie pour la mise en œuvre d'un robot collaboratif de nouvelle génération prenant en compte la sécurité et le confort biomécanique de l'opérateur en situation de travail”, March 2018 – Jean-Pierre Gazeau and Vincent Padois.
- PhD in progress: Nicolas Torres (Thèse ED SPI, Bordeaux / CIFRE PSA), “Synthesis and dynamic analysis of the shared workspace for safety in collaborative robotics”, April 2020 – Vincent Padois and David Daney.
- PhD in progress: Antun Skuric (Thèse ED SPI, Bordeaux / Financement projet Lichie Airbus), “A coupled view of the physical abilities of human-robot dyad for the online quantitative evaluation of assistance needs”, July 2020 – Vincent Padois and David Daney.
- PhD in progress: Raphaël Bousigues (Thèse ED SPI, Bordeaux / INRS-Inria), “Motor variability in human-robot collaboration”, December 2020 – Pauline Maurice, Vincent Padois, David Daney and Jonathan Savin.
- PhD in progress: Jessica Colombel (Inria), “Analyse du mouvement humain pour l'assistance à la personne”, February 2019 – , François Charpillet and David Daney (Inria Nancy Grand-Est).
- PhD in progress: Nicolas Simonazzi (CIFRE Orange), “Analyse comportementale et détection des émotions dans le cadre de l'utilisation de chat-bots en ligne”, May 2018 –, Jean-Marc Salotti.
- PhD passed: Baptiste Prébot (DGA), “Partage des représentations et de la conscience de situation”, December 4th, 2020. Supervisors: Jean-Marc Salotti and Bernard Claverie, ENSC, IMS laboratory.
- PhD in progress: Olfa Jema (Université de Sousse, Tunisie), “Analyse du mouvement humain”, December 2017 –, Lotfi Romdhane, Sami Bennour, David Daney.
- Vincent Padois:
- PhD: Matteo Ciocca, Reviewer, “Balance Preservation and Collision Mitigation for Biped Robots”, Université Grenoble Alpes, 2020/5/27.
- PhD: Maciej Bednarczyk, Reviewer, “Advanced control of interactions in robotics. Application to robot-assisted medical procedures”, Université de Strasbourg, 2020/12/8.
- PhD: Jury member for the Prix de thèse 2020 du GDR Robotique.
- David Daney:
- PhD: Matthieu Furet,, Examinor, “Analyse cinéto-statique de mécanismes de tenségrité : Application à la modélisation de cous d’oiseaux et de manipulateurs bio-inspirés”, 2020/11/13, École Centrale de Nantes.
- PhD: Adrien Malaisé, Reviewer, “Apprentissage du mouvement humain à l’aide de capteurs portés : vers l’automatisation de l’évaluation ergonomique.”, 2020/07/07, ENSTA Bretagne.
- Nasser Rezzoug:
- PhD: Damien Dodelin, Reviewer, “Identifier la pronation podale et son impact lors de la locomotion afin de prévenir les lombalgies en situation professionnelle“, 2020/10/13, Université de Rouen-Normandie.
- PhD:Stan Durand, Reviewer,“Quantification et modélisation biomécanique du poignet : géométrique, cinématique et dynamique”, 2020/01/22, ENSAM ParisTech, The 77 Lab, MIT.
- PhD: President of the jury of the "prix de thèse 2020 de la Société de Biomécanique".
10.4.1 Internal or external Inria responsibilities
- Nasser Rezzoug :
- Creation and development of the first cartrographies of biomechanics french speaking research (https://
www. biomecanique. org/ index. php/ fr/ carte/ laboratoires) and teaching (https:// www. biomecanique. org/ index. php/ fr/ carte/ formations) structures within the Société de Biomécanique (SB).
- Coresponsibility for relations with SB's partners.
- Creation and development of the first cartrographies of biomechanics french speaking research (https://
10.4.2 Articles and contents
- Vincent Padois: participation to the article “[Dossier Cobots] Collaborer sans réduire la cadence” of the Industries and Technologies Magazine6.
- Jean-Marc Salotti's paper published in Scientific Reports "Minimum Number of Settlers for Survival on an Other Planet" received a lot of attention and has been cited in many articles in international newspapers (The National Post, Fox News, new York Post, The Independent, Spoutnik News, Futura-Sciences, Les Echos, Ouest France, etc.). Altmetric score: 459.
- Jean-Marc Salotti supervised a group of students working on an industrial case study during the week of the AI4Industry event (20-24 January 2020), which involved several Masters from different universities in Nouvelle Aquitaine.
11 Scientific production
11.1 Major publications
- 1 article'Human-oriented design of collaborative robots'.International Journal of Industrial Ergonomics572017, 88--102
- 2 incollection'Human factors issues for the Design of a Cobotic System'.Advances in human factors in robots and unmanned systemsSpringer2017, 375--385
- 3 article'Performance analysis and design of parallel kinematic machines using interval analysis'.Mechanism and Machine Theory115September 2017, 218--236
11.2 Publications of the year
International peer-reviewed conferences
Scientific book chapters
Reports & preprints
11.3 Cited publications
- 18 book 'Agir sur la prévention durable des TMS'. Anact 2011
- 19 book 'Agir sur Prévenir les risques psychosociaux'. Anact 2013
- 20 book 'The Robots Are Here'. MIT Technology Review 2004
- 21 book 'Designing for Situation Awareness: An Approach to User-Centered Design'. Taylor & Francis, London 2012
- 22 article'A physiologist's perspective on robotic exoskeletons for human locomotion'.International journal of Humanoid Robotics43September 2007, 507--528URL: http://dx.doi.org/10.1142/S0219843607001138
- 23 article'The exoskeletons are here'.Journal of NeuroEngineering and Rehabilitation61June 2009, URL: http://dx.doi.org/10.1186/1743-0003-6-17
- 24 incollection'Physical Human-Robot Interaction'.Handbook of RoboticsSpringer Verlag2016, 1835--1874
- 25 article'Rapid Entire Body Assessment (REBA)'.Applied Ergonomics312April 2000, 201--205URL: http://www.sciencedirect.com/science/article/pii/S0003687099000393
- 26 article'Biomechanical analysis of upper limb during the use of touch screen: motion strategies identification'.Ergonomics603March 2017, 358--365URL: http://dx.doi.org/10.1080/00140139.2016.1175671
- 27 article'Using Conceptors to Manage Neural Long-Term Memories for Temporal Patterns'.Journal of Machine Learning Research18132017, 1-43URL: http://jmlr.org/papers/v18/15-449.html
- 28 inproceedings'Towards X-ray medical imaging with robots in the open: safety without compromising performances'.Proceedings of the IEEE International Conference on Robotics and AutomationBrisbane, AustraliaMay 2018, 6604--6610URL: https://hal.archives-ouvertes.fr/hal-01614508/en
- 29 article'Correcting working postures in industry: A practical method for analysis'.Applied Ergonomics84December 1977, 199--201URL: http://www.sciencedirect.com/science/article/pii/0003687077901648
- 30 article'LUBA: an assessment technique for postural loading on the upper body based on joint motion discomfort and maximum holding time'.Applied Ergonomics324August 2001, 357--366URL: http://www.sciencedirect.com/science/article/pii/S0003687001000060
- 31 phdthesis 'Conception d'une Interface de Pilotage d'un Cobot'. Université Pierre et Marie Curie - Paris VI March 2011
- 32 article'Generalized hierarchical control'.Autonomous Robots4012016, 17--31
- 33 article'Diversity and variation in biomechanical exposure: What is it, and why would we like to know?'Applied Ergonomics374Special Issue: Meeting Diversity in Ergonomics2006, 419-427URL: http://www.sciencedirect.com/science/article/pii/S0003687006000482
- 34 phdthesis 'Virtual ergonomics for the design of collaborative robots'. Université Pierre et Marie Curie - Paris VI June 2015
- 35 article'RULA: a survey method for the investigation of work-related upper limb disorders'.Applied ergonomics2421993, 91--99
- 36 incollection'Energy-based control for safe Human-robot physical interactions'.Springer Proceedings in Advanced Robotics - The 2016 International Symposium on Experimental RoboticsSpringer International Publishing AG2017, URL: http://hal.archives-ouvertes.fr/hal-01398790/en
- 37 inbook'Appropriate Design of Parallel Manipulators'.Smart Devices and Machines for Advanced ManufacturingL. WangJ. XiLondonSpringer London2008, 1--25URL: https://doi.org/10.1007/978-1-84800-147-3_1
- 38 inproceedings'Human Factors Issues for the Design of a Cobotic System'.Advances in Human Factors in Robots and Unmanned SystemsAdvances in Intelligent Systems and ComputingSpringer International Publishing2017, 375--385
- 39 phdthesis'Conception de systèmes cobotiques industriels : approche cognitique : application à la production pyrotechnique au sein d'Ariane Group'.Université de BordeauxNovember 2017, URL: https://hal.archives-ouvertes.fr/tel-01670146
- 40 incollection'Cognitive Human-Robot Interactions'.Handbook of RoboticsSpringer Verlag2016, 1907--1934
- 41 article'Design Strategy of Serial Manipulators With Certified Constraint Satisfaction'.IEEE Transactions on Robotics251Feb 2009, 1-11URL: http://dx.doi.org/10.1109/TRO.2008.2006867
- 42 book'Inverse optimal control for redundant systems of biological motion'.OrlÃ©ansDecember 2015, URL: http://www.theses.fr/2015ORLE2050
- 43 article'A model for types and levels of human interaction with automation'.IEEE Transactions on Systems, Man, and Cybernetics - Part A: Systems and Humans303May 2000, 286--297URL: http://dx.doi.org/10.1109/3468.844354
- 44 article'Teaching Robots to Cooperate with Humans in Dynamic Manipulation Tasks Based on Multi-modal Human-in-the-loop Approach'.Autonomous Robots361-2January 2014, 123--136URL: http://dx.doi.org/10.1007/s10514-013-9361-0
- 45 article'Robot adaptation to human physical fatigue in human–robot co-manipulation'.Autonomous Robots425June 2018, 1011--1021URL: http://dx.doi.org/10.1007/s10514-017-9678-1
- 46 article 'Bayesian Network for the Prediction of Situation Awareness Errors'. International Journal on Human Factors Modeling and Simulation Special Issue on: Quantifying Human Factors Towards Analytical Human-in-the-Loop January 2018
- 47 article'Classification des Systèmes Cobotiques'.Ingénierie cognitique112018, URL: http://dx.doi.org/10.21494/ISTE.OP.2018.0268
- 48 inproceedings'Movement Variability and Digital Human Models: Development of a Demonstrator Taking the Effects of Muscular Fatigue into Account'.Advances in Applied Digital Human Modeling and SimulationChamSpringer International Publishing2017, 169--179
- 49 inproceedings 'Theory and Evaluation of Human Robot Interactions'. Proceedings of the 36th Annual Hawaii International Conference on System Sciences Washington, DC, USA 2003
- 50 article'Human–Robot Interaction: Status and Challenges'.Human Factors584June 2016, 525--532URL: http://dx.doi.org/10.1177/0018720816644364
- 51 article'Development and evaluation of an office ergonomic risk checklist: ROSA - Rapid Office Strain Assessment'.Applied Ergonomics431January 2012, 98--108URL: http://www.sciencedirect.com/science/article/pii/S0003687011000433
- 52 article'Motor variability in occupational health and performance'.Clinical Biomechanics27102012, 979-993URL: http://www.sciencedirect.com/science/article/pii/S0268003312001817
- 53 inproceedings'Classification of physical human-robot interaction scenarios to identify relevant requirements'.Proceedings of the 41st International Symposium on RoboticsJune 2014, 1--8
- 54 book 'Evaluation of Human Work, Fourth Edition'. Google-Books-ID: uXB3CAAAQBAJ CRC Press April 2015
- 55 inproceedings'Classifying human-robot interaction: an updated taxonomy'.Proceedings of the IEEE International Conference on Systems, Man and Cybernetics3October 2004, 2841--2846URL: http://dx.doi.org/10.1109/ICSMC.2004.1400763