The aim of the project is to develop key technologies and methods necessary to construct a companion robot control system. The project is focused on the most important research issues related to robot programming by demonstration. The robot learns from examples of tasks performed by humans (look and learn). The way to delegate tasks to the robot must be easy to master by unskilled users. Young children learn to perform activities imitating adults, the robot companion should learn in a similar way. In order to make this a reality, the following problems must be solved: a) perception of the environment and human activities demonstrated to the robot, b) visual tracking of human movements and moving objects, c) synthesis and reproduction of human movements by a device with specific kinematics, d) creating a meta-model of the robot control system enabling automatic generation of the controller code for the task, e) ontology enabling knowledge representation from the robot's point of view, however concerning its environment, and taking into account the necessity to generate action plans and define/learn the tasks to be performed. The problem of teaching a robot by demonstrating human activities is difficult mainly due to: visual data ambiguity, small number of examples and low variability of training data. To solve the ambiguity problem, the approach based on the impact of high-level interpretation of dynamic movements performed by a human, utilizing the principle of feedback from the interpretation of image features, will be studied. Too small training sets will be enhanced with synthetic data. Generative Adversarial Networks or Deep Belief Networks will be used to generate such data. A motion synthesis algorithm for a humanoid robot that reliably reproduces human posture and movement will be elaborated. To perform operations on objects, a computationally effective predictive visual servomechanism with an innovative mechanism for adaptive reconfiguration of the servomechanism will be developed. Based on the metamodel in the form of a hierarchical Petri net, a system model will be built, which will then be transformed automatically into an executable program code of the controller. An important element of the project will be the agent acquiring knowledge from its own observation of the environment as well as through communication with other agents. This knowledge will be encoded in a way understandable to the robot – using a domain ontology. The communication language taking into account the communication acts, as well as interaction protocols and a language of content using the formulated ontology will be defined. An approach to robot task planning combining Hierarchical Task Network with planning under uncertainty in the observation of the environment and uncertainty in the outcome of the action (modeled using the Partially Observable Markov Decision Process) will be used. The structure of the robot control system will contain modules responsible for: knowledge representation and processing, task planning and control of robot effectors and receptors. Knowledge representation module will take into account perception, i.e. the results of processing of current observation of the environment. The methods developed in the project will be experimentally verified on real robots.

1. M. Figat and C. Zieliński
   Parameterised robotic system meta-model expressed by Hierarchical Petri nets
   Robotics and Autonomous Systems, p. 103987, 2022
   [ BIB | DOI | abstract | URL ]

   @article{Figat:2022:RAS-twiki,
     author = {Figat, Maksym and Zieliński, Cezary},
     journal = {Robotics and Autonomous Systems},
     title = {Parameterised robotic system meta-model expressed by Hierarchical Petri nets},
     year = {2022},
     issn = {0921-8890},
     pages = {103987},
     doi = {10.1016/j.robot.2021.103987},
     keywords = {Embodied agent approach, Model Driven Engineering, Robotic System Hierarchical Petri Net, Meta-model parametrisation},
     projects = {Roko},
     twiki = {journal},
     url = {https://www.sciencedirect.com/science/article/pii/S0921889021002487}
   }
   

   The paper presents a model-based approach to developing robotic system controllers. Central to this approach is a parameterised meta-model that describes the generic robotic system from two points of view: structure and activity. By appropriate parametrisation of the meta-model one can obtain a particular model of a robotic system performing desired tasks. The meta-model is expressed using the Robotic System Hierarchical Petri Net (RSHPN), a 6-layer Petri net tailored for robotics. Each layer describes the activity of the robotic system at a completely different level of abstraction. This guarantees the separation of concerns. The required model emerges from the meta-model by appropriate parametrisation of the layers of the RSHPN. Introduction of parametrisation enables the robot designer to focus only on the concepts derived from the domain. It greatly facilitates the robotic system development process as it gives the designer clear guidance on what needs to be defined and what is imposed by the design pattern. The resulting single RSHPN model is used both to verify some properties of the system, e.g. lack of deadlocks, but also to automatically generate controller code. The presented approach is illustrated by examples of the creation of two different robotic systems.

2. T. Winiarski, J. Sikora, D. Seredyński, and W. Dudek
   DAIMM Simulation Platform for Dual-Arm Impedance Controlled Mobile Manipulation
   in 7th International Conference on Automation, Robotics and Applications (ICARA), 2021, pp. 180–184
   [ BIB | DOI | abstract | URL ]

   @inproceedings{winiarski-icara-21-twiki,
     author = {Winiarski, Tomasz and Sikora, Jakub and Seredyński, Dawid and Dudek, Wojciech},
     booktitle = {7th International Conference on Automation, Robotics and Applications (ICARA)},
     title = {{DAIMM Simulation Platform for Dual-Arm Impedance Controlled Mobile Manipulation}},
     year = {2021},
     pages = {180--184},
     publisher = {IEEE},
     doi = {10.1109/ICARA51699.2021.9376462},
     projects = {Roko},
     twiki = {article},
     url = {http://gitlab-stud.elka.pw.edu.pl/robotyka/rpmpg_pubs/-/raw/main/2021/integrated-velma-icara.pdf}
   }
   

   One of the most considerable research problems in robotics is manipulation investigated in robot systems operating in different environments and performing various tasks. There are many advanced robot tasks that require agile manipulation and dual-arm setup to make them feasible (e.g., opening a jar or a box, package wrapping). Moreover, robots can sense torques in joints to be compliant during cooperation with humans and to be mobile to enlarge their workspace. Therefore, the dual-arm, impedance controlled mobile manipulation (DAIMM) problem emerge. In this article we state requirements, which should be considered in the DAIMM problem simulators. Moreover, we propose a model for simulation platforms able to perform DAIMM tasks and verify it by the implementation of an exemplary platform. The robot simulated by the exemplary platform successfully realised two tasks involving agile manipulation and navigation.

3. T. Winiarski, S. Jarocki, and D. Seredyński
   Grasped Object Weight Compensation in Reference to Impedance Controlled Robots
   Energies, vol. 14, no. 20, p. 6693, 2021
   [ BIB | DOI | abstract | URL ]

   @article{en14206693-grav-comp-twiki,
     author = {Winiarski, Tomasz and Jarocki, Szymon and Seredyński, Dawid},
     journal = {Energies},
     title = {Grasped Object Weight Compensation in Reference to Impedance Controlled Robots},
     year = {2021},
     issn = {1996-1073},
     number = {20},
     pages = {6693},
     volume = {14},
     doi = {10.3390/en14206693},
     projects = {Roko,EARL},
     twiki = {journal},
     url = {https://www.mdpi.com/1996-1073/14/20/6693}
   }
   

   This paper addresses the problem of grasped object weight compensation in the one-handed manipulation of impedance controlled robots. In an exemplary identification procedure, the weight of an object and its centre of mass together with gripper kinematic configuration are identified. The procedure is based on the measurements from a 6-axis force/torque sensor mounted near the gripper. The proposed method reduces trajectory tracking errors coming from the model imprecision without compromising the main advantages of impedance control. The whole approach is applied according to the embodied agent paradigm and verified on the two-arm service robot both in simulation and on hardware. Due to the general description that follows system engineering standards, the method can be easily modified or applied to similar systems.