Cut & recombine reuse of robot action components based on simple language instructions

Minija Tamosiunaite*, Mohamad Javad Aein, Jan Matthias Braun, Tomas Kulvicius, Irena Markievicz, Jurgita Kapociute-Dzikiene, Rita Valteryte, Andrei Haidu, Dimitrios Chrysostomou, Barry Ridge, Tomas Krilavicius, Daiva Vitkute-Adzgauskiene, Michael Beetz, Ole Madsen, Ales Ude, Norbert Krüger, Florentin Wörgötter

*Corresponding author for this work

Research output: Contribution to journalJournal articleResearchpeer-review

6 Citations (Scopus)
228 Downloads (Pure)


Human beings can generalize from one action to similar ones. Robots cannot do this and progress concerning information transfer between robotic actions is slow. We have designed a system that performs action generalization for manipulation actions in different scenarios. It relies on an action representation for which we perform code-snippet replacement, combining information from different actions to form new ones. The system interprets human instructions via a parser using simplified language. It uses action and object names to index action data tables (ADTs), where execution-relevant information is stored. We have created an ADT database from three different sources (KUKA LWR, UR5, and simulation) and show how a new ADT is generated by cutting and recombining data from existing ADTs. To achieve this, a small set of action templates is used. After parsing a new instruction, index-based searching finds similar ADTs in the database. Then the action template of the new action is matched against the information in the similar ADTs. Code snippets are extracted and ranked according to matching quality. The new ADT is created by concatenating code snippets from best matches. For execution, only coordinate transforms are needed to account for the poses of the objects in the new scene. The system was evaluated, without additional error correction, using 45 unknown objects in 81 new action executions, with 80% success. We then extended the method including more detailed shape information, which further reduced errors. This demonstrates that cut & recombine is a viable approach for action generalization in service robotic applications.

Original languageEnglish
JournalInternational Journal of Robotics Research
Issue number10-11
Pages (from-to)1179-1207
Number of pages29
Publication statusPublished - 1 Sept 2019


  • Cognitive robotics
  • control architectures and programming
  • manipulation
  • manipulation planning
  • service robotics


Dive into the research topics of 'Cut & recombine reuse of robot action components based on simple language instructions'. Together they form a unique fingerprint.

Cite this