Man-Machine Interaction in a robotized environment: adapting robot behavior to psychological and emotional trends of expected and not expected human interactors

PhD project information

Summary

In this thesis, we propose to develop (i) computational models of interpersonal human-robot interactions able to assess human psychological states (e.g., stress, attention) and to continuously adapt to them, as well as (ii) experimental methodologies of evaluation in real co-working scenarios. We target operational missions as well as not expected situations with non-operational humans (e.g., individual not expected to interact with the collaborative robot). Collaborative robots will be more and more deployed in a real human environment or engaged in face-to-face human interaction when accomplishing their duty. In such context, there is a need for deep and accurate understanding of psychological human co-workers. Mutual and personalized adaptation of behaviours is certainly a key element of long-term safe interactions.
APSYS will use these technological assets to

  • Simulate safety critical situations involving human to robot interactions
  • Process the output of these situations to be able to assess global safety of production process based on man and robot cooperation

Methodological approach
Taking advantage of introduction of robots in a human team, production manager can expect direct contribution from robots to safety process because:

  • Robots can in a real time manner with sensors and perception algorithms understand what happens in the process
  • With specific algorithms they can analyse emotional mood and level of stress of all human actors
  • At every time making the synthesis of (deep understanding of current situation) and (anticipation of potential trajectories and movements of all actors whatever human or robots) they will predict most dreaded behaviours or situations with criticality / probability or level of belief assessment
  • Finally, they will interact with the process to reduce probability or criticality of those situations and send alert / alarm messages for concerned human actors to influence their behaviour to avoid these dreaded situations.

Robots will be:

  • Mobile autonomous: they will move in the production space shared by human actors and other robots
  • Manipulating robots: they work on the production plant and operate in interaction with the system to assemble
  • Interacting robots: they may have interactions and concurrent tasks to share with human actors

The influence of emotional / stress load assessed from all human actors on the behaviour of human and robots, and on the general process will be a central topic of the thesis. Applicable Safety metric to the global process and follow-up of its variations along the process will also be possibly targeted by the thesis.


Facts

  • Industrial partner: APSYS SAS
  • Academic/research partner: Sorbonne University
  • Number of available PhD positions: 1
  • Duration: 3 years
  • This PhD will be funded by EIT Digital and APSYS SAS.

PhD thesis motivation and innovation valorisation

Rationale/challenge

APSYS motivation
Specification of Safety Management Systems for production process is an historical involvement of APSYS : it applies to all processes supported by human teams. What is observed currently in industry is that more and more, manufacturing and productions tasks are automated, and robots can most of the time endorse 100% of the tasks. However, for a few industry domains, among them aeronautic industry, 100% automation of assembling and manufacturing process is not yet in the horizon of what will be feasible. And most probable scenario is that manufacturing teams will be composed of human operators and robots at the same time.

An important question still remains: how Safety of those processes will be assured, combining Human Factors issues and possible Man - Robot interaction?

The idea which is illustrated by the subject of this thesis is that robots being parts of this team will integrate Safety centered analysis and surveillance functions which will directly assure required Safety level of the process, that is to say contribute to avoid undesired events (collisions, damage sensitive unpredicted events etc…)

Most collaborative robots will be dropped in human environment where they will have to accomplish missions. They will be engaged in interaction with human individuals concerned by these missions but potentially all of the individuals sharing the same working space. Optimizing and personalizing robot behaviors in such situations will require having a detailed perception of psychological and emotional features of human co-workers as well as mechanisms of mutual adaptation.
It applies both to:

  • Expected interaction with human co-workers: users whom the cobot will interact with, in coherence with the mission achievement: old person to entertain, child to teach, …
  • Unexpected interactions: situated in the spatial environment of cobot operational perimeter, where those unexpected interactions will interfere with cobot specified mission.

Innovation

Actually, existing collaborative robots in manufacturing situations do not interact with human being by explicitly taking into account co-workers’ personality and psychological traits. In addition, most of missions are specified from an operational point of view without taking into account behavior particularities of expected or unexpected interactions: it is supposed that, assuming qualified workers have the same training, they will process their tasks with same emotional and concentration assets.
When the robot’s mission robot is clearly grounded on cognitive and social interaction, such as socially assistive robotics, automated assessment of and adaptation to human socio-emotional states is obvious and largely addressed in the literature. However, this is less addressed in the case of purely technical missions such as co-working with a robot in an assembly line.

However, in terms of mutual adaptation of behaviors with employees, it seems to be necessary to integrate accurate understanding of psychological and emotional traits of human interaction to be able to adapt adequately their motion and communication modes (non-verbal communication, reactivity, …). The project aims at building a methodology to implement and calibrate such personality recognition algorithmic and include it in an already programmed robot to enable adaptation of the mission completion to the psychological profile of the human co-worker.

Besides, Specification of Safety Management Systems for production process is an historical involvement of APSYS : it applies to all processes supported by human teams.
What is observed currently in industry is that more and more, manufacturing and productions tasks are automated, and robots can most of the time endorse 100% of the tasks.
However, for a few industry domains, among them aeronautic industry, 100% automation of assembling and manufacturing process is not yet in the horizon of what will be feasible. And most probable scenario is that manufacturing teams will be composed of human operators and robots at the same time.
An important question still remains: how Safety of those processes will be assured, combining Human Factors issues and possible Man - Robot interaction?
The idea which is illustrated by the subject of this thesis is that robots being parts of this team will integrate Safety centered analysis and surveillance functions which will directly assure required Safety level of the process, that is to say contribute to avoid undesired events (collisions, damage sensitive unpredicted events etc.).

Expected academic outcomes

Theoretical approach integrating how human assessment from cobots captured from observation and analysis of human co-workers of environment can help robots to improve their behavior in terms of safety, accuracy and operational performance in the achievement of their missions. This will result in a computational model of robot behaviors that will be evaluated in various operational sessions and scenarios. The model should show a significant improvement for both expected and unexpected interactions between the cobot and human.

KPI will focus general Safety and Performance of cobot mission and behavior.

Concrete innovations expected as the outcome of the proposal

Different types of IPs may be concerned by this thesis:

  • Methodological framework how to take into account into a “semi-automated human / robot based manipulation process” (for example a simple assembling process) supported by a robot and a human operator, the fact that the robot will integrate analysis of emotional features and level of stress analysis as well as other psychological insights to adapt its behavior and collaboration process, so to maintain a certain level of safety for the whole process
  • Application of the methodology to a particular “one man with one robot collaborative operation process” and delivery of learning software modules which support this process, while adapting to human character mood and concentration state to adjust speed of interaction as well as accuracy of every manipulation step
  • Script scenarios observed and tracking of every movement and interaction mechanisms from a kinetic and cinematic point of view and corresponding safety analysis with psycho metric tracking of manipulating human actors
  • Comparison of the process when no adaptation of the behavior is made from the robot considering observation of human partner: safety assessment methodology of semi-automated human / robot based process observed.

Part of this technology will be the corner stone of a Safety Management System platform applied to robot and human collaboration safety analysis process.

Expected impact of the PhD outcomes with respect to their business line

APSYS is very much expecting from this thesis’ outcome because it will be a strong step forwards in the mastery of safety for production teams where robots and human agents will be integrated together.
APSYS competitiveness will be fostered in three axes:

  • safety management of mixed human and robotic based processes
  • better human factor knowledge in terms of features which can be measured and analysed by machine and projected into behavioural scenarios
  • specification of robotics assistance systems to make surveillance and produce compensation actions to dangerous or sensitives human features when involved in critical tasks

The outcomes will have an impact in the improvement of the Specification of Safety Management Systems for production process, which has an historical involvement of APSYS.

What is observed currently in industry is that more and more, manufacturing and productions tasks are automated, and robots can most of the time endorse 100% of the tasks. However, for a few industry domains, among them aeronautic industry, 100% automation of assembling and manufacturing process is not yet in the horizon of what will be feasible. And most probable scenario is that manufacturing teams will be composed of human operators and robots at the same time.


PhD thesis time-line and milestones

Deadlines/milestones

M6State of the Art regarding human personality analysis from robot detection and perception
M12 Specification of “algorithms” for personality detection, recognition, and virtual reconstruction to be taken into account through expected or unexpected interaction
M24Specification of adaptative algorithms for cobot missions, integrating psychological analysis of human interactors
M30Application to a dedicated application; cobot for manufacturing
M36Operational validation of adaptative behavior of cobots

International mobility plan

In a first run: Germany, UK, Spain and Portugal, but also China and India in a second range...

BDExp- Will the PhD Student do the Business Development Experience at Industrial Partner premises?

Yes


Apply

Applications, consisting of a CV and a Personal Statement, should be submitted to EIT Digital Doctoral School Office at dsl.office@eitdigital.eu.

© 2010-2020 EIT Digital IVZW. All rights reserved. Legal notice