SCIENCE CHINA Technological Sciences, Volume 60 , Issue 3 : 374-384(2017) https://doi.org/10.1007/s11431-016-0434-x

Human stochastic closed-loop behavior for master-slave teleoperation using multi-leap-motion sensor

More info
  • ReceivedJun 11, 2016
  • AcceptedOct 18, 2016
  • PublishedJan 17, 2017


Teleoperation has a wide range of applications that have been under development over the past two decades. Previous researches have focused on the control design of teleoperation machine systems to deal with obstacles such as time-delayed stability and transparency. Recent researches have shown that the inclusion of human closed-loop dynamics in control design can improve the performance of robot telemanipulation. The complexity of human behavior arises from the uncertainty of both human physiology and psychology; hence, the investigation can benefit from empirical studies. This study develops a type of statistical learning method to model and evaluate human stochastic closed-loop behavior, which is considered as a hand motion during the direct incremental control process of master-slave teleoperation. The hand trajectory is empirically considered as having a binary linear regression relationship with the error and error rate between the demanded and simulated teleoperator trajectories, while random movements with zero error and error rate are discovered. Hand movement tracking is achieved using a multi-leap-motion sensor (MLM), which is a markerless and natural infrared vision-based manner for motion capture. The established behavior model and statistical learning results reveal certain human properties of operational activities including visual perception, decision making, and robot telemanipulation. The properties indicate some probable system enhancements for future work.

Funded by

National Natural Science Foundation of China(11402004)


This work was supported by the National Natural Science Foundation of China (Grant No. 11402004).


[1] Iborra A, Pastor J A, Alvarez B, et al. Robots in radioactive environments. IEEE Robot Autom Mag, 2003, 10: 12-22 CrossRef Google Scholar

[2] El-Fakdi A, Cufí X, Hurtós N, et al. Team-based building of a remotely operated underwater robot, an innovative method of teaching engineering. J Intell Robot Syst, 2016, 81: 51-61 CrossRef Google Scholar

[3] Imaida T, Yokokohji Y, Doi T, et al. Ground-space bilateral teleoperation of ETS-VII robot arm by direct bilateral coupling under 7-s time delay condition. IEEE T Robotic Autom, 2004, 20: 499-511 CrossRef Google Scholar

[4] Norris J S, Powell M W, Vona M A, et al. Mars exploration rover operations with the science activity planner. In: Proceedings of the 2005 IEEE International Conference on Robotics and Automation, 2005. Barcelona: IEEE, 2005. 4618–4623. Google Scholar

[5] Ding L, Gao H B, Deng Z Q, et al. Three-layer intelligence of planetary exploration wheeled mobile robots: Robint, virtint, and humint. Sci China Tech Sci, 2015, 58: 1299-1317 CrossRef Google Scholar

[6] Rembold U, Burghart C R. Surgical robotics: An introduction. J Intell Robot Syst, 2001, 30: 1-28 CrossRef Google Scholar

[7] Koizumi N, Warisawa S, Nagoshi M, et al. Construction methodology for a remote ultrasound diagnostic system. IEEE Trans Robot, 2009, 25: 522-538 CrossRef Google Scholar

[8] Enloe C L, Pakula W A, Finney G A, et al. Teleoperation in the undergraduate physics laboratory-teaching an old dog new tricks. IEEE Trans Educ, 1999, 42: 174-179 CrossRef ADS Google Scholar

[9] Baier H, Schmidt G. Transparency and stability of bilateral kinesthetic teleoperation with time-delayed communication. J Intell Robot Syst, 2004, 40: 1-22 CrossRef Google Scholar

[10] Kron A, Schmidt G. Stability and performance analysis of kinesthetic control architectures for bimanual telepresence systems. J Intell Robot Syst, 2006, 46: 1-26 CrossRef Google Scholar

[11] Lii N Y, Chen Z P, Pleintinger B, et al. Toward understanding the effects of visual- and force-feedback on robotic hand grasping performance for space teleoperation. In: IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2010. Taipei: IEEE, 2010. 3745–3752. Google Scholar

[12] Thompson R L, Daniel R W, Murray D W. Experiments on operator matching in visual teleoperation. In: 2000 IEEE International Conference on Systems, Man, and Cybernetics, 2000. Nashville: IEEE, 2000. 931–936. Google Scholar

[13] Cong S, Wang J N. Internet-based and visual feedback networked robot arm teleoperation system. In: International Conference on Networking, Sensing and Control. Chicago: IEEE, 2010. 452–457. Google Scholar

[14] Hu T, Huang X, Tan Q. Active disturbance rejection controller for space teleoperation. In: International Conference on Automatic Control and Artificial Intelligence (ACAI 2000). Wuhan: IEEE, 2012. 334–337. Google Scholar

[15] Anderson R J, Spong M W. Bilateral control of teleoperators with time delay. IEEE T Automat Contr, 1989, 34: 494-501 CrossRef Google Scholar

[16] Salcudean S E, Hashtrudi-Zaad K, Tafazoli S, et al. Bilateral matched impedance teleoperation with application to excavator control. IEEE Contr Syst Mag, 1999, 19: 29-37 CrossRef Google Scholar

[17] Prokopiou P A, Tzafestas S G, Harwin W S. A novel scheme for human-friendly and time-delays robust neuropredictive teleoperation. J Intell Robot Syst, 1999, 25: 311-340 CrossRef Google Scholar

[18] Hirche S, Buss M. Human-oriented control for haptic teleoperation. Proc IEEE, 2012, 100: 623-647 CrossRef Google Scholar

[19] Ou Y, Qian H, Xu Y. Support vector machine based approach for abstracting human control strategy in controlling dynamically stable robots. J Intell Robot Syst, 2009, 55: 39-54 CrossRef Google Scholar

[20] Medina J R, Lorenz T, Hirche S. Synthesizing anticipatory haptic assistance considering human behavior uncertainty. IEEE Trans Robot, 2015, 31: 180-190 CrossRef Google Scholar

[21] Du G, Zhang P. A markerless human-robot interface using particle filter and kalman filter for dual robots. IEEE Trans Ind Electron, 2015, 62: 2257-2264 CrossRef Google Scholar

[22] Lawrence D A. Stability and transparency in bilateral teleoperation. IEEE Trans Rob Aut, 1993, 9: 624-637 CrossRef Google Scholar

[23] Lee D, Spong M W. Passive bilateral teleoperation with constant time delay. IEEE Trans Robot, 2006, 22: 269-281 CrossRef Google Scholar

[24] Chopra N, Spong M W, Hirche S, et al. Bilateral teleoperation over the internet: the time varying delay problem. In: Proceedings of the 2003 American Control Conference. Denver, Colorado, USA. 2003. 155–160. Google Scholar

[25] Guo S P, Li D X, Meng Y H, et al. Task space control of free-floating space robots using constrained adaptive RBF-NTSM. Sci China Tech Sci, 2014, 57: 828-837 CrossRef Google Scholar

[26] Li Z, Xia Y. Adaptive neural network control of bilateral teleoperation with unsymmetrical stochastic delays and unmodeled dynamics. Int J Robust Nonlin, 2014, 24: 1628-1652 CrossRef Google Scholar

[27] Li Z, Xia Y, Wang D, et al. Neural network-based control of networked trilateral teleoperation with geometrically unknown constraints. IEEE Trans Cybern, 2016, 46: 1051-1064 CrossRef PubMed Google Scholar

[28] Ando T, Tsukahara R, Seki M, et al. A haptic interface “force blinker 2” for navigation of the visually impaired. IEEE Trans Ind Electron, 2012, 59: 4112-4119 CrossRef Google Scholar

[29] Du G, Zhang P, Li D. Human-manipulator interface based on multisensory process via Kalman filters. IEEE Trans Ind Electron, 2014, 61: 5411-5418 CrossRef Google Scholar

[30] Bachmann E R, McGhee R B, Yun Z, et al. Inertial and magnetic posture tracking for inserting humans into networked virtual environments. In: Proceedings of the ACM Symposium on Virtual Reality Software and Technology. Banff, Canada, 2001. 9–16. Google Scholar

[31] Rebelo J, Sednaoui T, den Exter E B, et al. Bilateral robot teleoperation: A wearable arm exoskeleton featuring an intuitive user interface. IEEE Robot Autom Mag, 2014, 21: 62-69 CrossRef Google Scholar

[32] Ma Z, Ben-Tzvi P. RML glove—An exoskeleton glove mechanism with haptics feedback. IEEE/ASME Trans Mech, 2015, 20: 641-652 CrossRef Google Scholar

[33] Kofman J, Verma S, Wu X H. Robot-manipulator teleoperation by markerless vision-based hand-arm tracking. Int J Optomechatron, 2007, 1: 331-357 CrossRef Google Scholar

[34] Kofman J, Wu X, Luu T J, et al. Teleoperation of a robot manipulator using a vision-based human-robot interface. IEEE Trans Ind Electron, 2005, 52: 1206-1219 CrossRef Google Scholar

[35] Du G, Zhang P. Markerless human-robot interface for dual robot manipulators using Kinect sensor. Robot Comput Integr Manuf, 2014, 30: 150-159 CrossRef Google Scholar

[36] Weichert F, Bachmann D, Rudak B, et al. Analysis of the accuracy and robustness of the leap motion controller. Sensors, 2013, 13: 6380-6393 CrossRef PubMed Google Scholar

[37] James G, Witten D, Hastie T, et al. An introduction to statistical learning. 2014, http://www.springer.com/series/417. Google Scholar

[38] Xu W, Liang B, Xu Y, et al. A ground experiment system of free-floating robot for capturing space target. J Intell Robot Syst, 2007, 48: 187-208 CrossRef Google Scholar

[39] Matlin M W. Cognitive psychology. 2013, http://www.islib.com/ecommerce/resources/online-read-resource/276403/. Google Scholar

  • Figure 1

    (Color online) Basic representation of a teleoperation system, which consists of a human operator, robotic motion visual feedback, human-teleoperator interface utilizing MLM at local operator site, and teleoperator at a remote robot site.

  • Figure 2

    (Color online) Multi-Leap-Motion sensor and right-handed coordinate definitions including measure coordinate Smi (i=0, 1, 2, 3, 4) and hand coordinate Sh.

  • Figure 3

    The process of the statistical learning of the behavior model.

  • Figure 4

    (Color online) Teleoperation system for the experiment. The haptic interface in the figure is replaced by the movable MLM sensor shown as Figure 2, and the right robot is chosen as the teleoperator.

  • Figure 5

    (Color online) Expected trajectory of remote robot’s end effector plotted as the solid line and simulated trajectory of end effector plotted as the dashed line.

  • Figure 6

    (Color online) Human hand movement consists of 15 continuous trajectories during the tracking task. The breakpoint exists due to the limited workspace of MLM sensor and relatively large range of robotic motion.

  • Figure 7

    (Color online) Samples and BLR model of behavior along (a) yW axis and (b) zW axis.

  • Figure 8

    (Color online) Relearning samples and BLR model of behavior along (a) yW axis and (b) zW axis.

  • Figure 9

    (Color online) Complementary relationship of error rate pair (a) (ey(t), ez(t)) and (b) (e˙y(t), e˙z(t)).

  • Figure 10

    (Color online) Scatters of hand trajectories along (a) yW axis and (b) zW axis during the practical operation and simulated operation.

  • Table 1   Statistical learning results of behavior along yW and zW axes



    Confidence interval



    Confidence interval

    Behavior along yW axis

    Behavior along zW axis



    [2.91, 10.19]



    [–5.15, –2.33]



    [–0.75, –0.32]



    [–0.28, –0.059]



    [–0.75, 0.23]



    [–0.41, 0.080]

    R2=0.193, Fs=12.20, p<2×10–5, RSE2=252.18

    R2=0.087, Fs=4.86, p<0.01, RSE2=42.00

  • Table 2   Relearning results of behavior along yW and zW axes



    Confidence interval



    Confidence interval

    Behavior along yW axis

    Behavior along zW axis



    [12.11, 23.68]



    [–6.42, –1.44]



    [–1.15, –0.63]



    [–0.31, –0.045]



    [–0.89, 0.087]



    [–0.41, 0.075]

    R2=0.472, F=24.56, p<2×10–8, RSE2=236.29

    R2=0.148, F=3.82, p<0.03, RSE2=37.79

Copyright 2020  CHINA SCIENCE PUBLISHING & MEDIA LTD.  中国科技出版传媒股份有限公司  版权所有

京ICP备14028887号-23       京公网安备11010102003388号