Continuous Semi-autonomous Prosthesis Control using a Depth Sensor on the Hand

Miguel Nobre Castro*, Strahinja Dosen*

*Corresponding author for this work

Research output: Contribution to journalJournal articleResearchpeer-review

12 Citations (Scopus)
54 Downloads (Pure)


Modern myoelectric prostheses can perform multiple functions (e.g., several grasp types and wrist rotation) but their intuitive control by the user is still an open challenge. It has been recently demonstrated that semi-autonomous control can allow the subjects to operate complex prostheses effectively; however, this approach often requires placing sensors on the user. The present study proposes a system for semi-autonomous control of a myoelectric prosthesis that requires a single depth sensor placed on the dorsal side of the hand. The system automatically pre-shapes the hand (grasp type, size, and wrist rotation) and allows the user to grasp objects of different shapes, sizes and orientations, placed individually or within cluttered scenes. The system "reacts" to the side from which the object is approached, and enables the user to target not only the whole object but also an object part. Another unique aspect of the system is that it relies on online interaction between the user and the prosthesis; the system reacts continuously on the targets that are in its focus, while the user interprets the movement of the prosthesis to adjust aiming. Experimental assessment was conducted in ten able-bodied participants to evaluate the feasibility and the impact of training on prosthesis-user interaction. The subjects used the system to grasp a set of objects individually (Phase I) and in cluttered scenarios (Phase II), while the time to accomplish the task (TAT) was used as the performance metric. In both phases, the TAT improved significantly across blocks. Some targets (objects and/or their parts) were more challenging, requiring thus significantly more time to handle, but all objects and scenes were successfully accomplished by all subjects. The assessment therefore demonstrated that the system is indeed robust and effective, and that the subjects could successfully learn how to aim with the system after a brief training. This is an important step toward the development of a self-contained semi-autonomous system convenient for clinical applications.

Original languageEnglish
Article number814973
JournalFrontiers in Neurorobotics
Number of pages17
Publication statusPublished - 25 Mar 2022

Bibliographical note

Copyright © 2022 Castro and Dosen.


  • Computer Vision
  • Grasping
  • Myoelectric Hand Prosthesis
  • Object Segmentation
  • Point Cloud Processing
  • Semi-autonomous control
  • grasping
  • myoelectric hand prosthesis
  • point cloud processing
  • object segmentation
  • semi-autonomous control
  • computer vision


Dive into the research topics of 'Continuous Semi-autonomous Prosthesis Control using a Depth Sensor on the Hand'. Together they form a unique fingerprint.

Cite this