SCIENTIA SINICA Informationis, Volume 48 , Issue 4 : 449-465(2018) https://doi.org/10.1360/N112017-00213

Multi-modal human-machine interaction for human intelligence augmentation

More info
  • ReceivedOct 30, 2017
  • AcceptedFeb 7, 2018
  • PublishedApr 13, 2018


Human-machine symbiosis and harmony is an ultimate goal of human-machine interaction (HMI). Recently, the rapid development of artificial intelligence has given rise to concern regarding the domination of machine intelligence over human intelligence. Enhancing human intelligence through multi-modal HMI technology is becoming an important research topic. The studies on neural plasticity indicate that the core capabilities of human cognition including attention control ability and working memory capacity could be enhanced and trained through several approaches including visual-auditory games, haptic interaction tasks, transcranial electromagnetic stimulation, and brain-computer interfaces. In this paper, we propose a systematic paradigm of augmenting cognition through HMI tasks by seamlessly integrating haptic-visual-aural multisensory feedback. This paradigm is constructed on analyzing the inherent features of virtual reality including immersion, interaction, and imagination. Based on the concept of cybernetics, cognitive enhancement methods based on Hebbian learning are proposed to achieve several goals including the realization of controllable cognitive loads, immediate physiological feedback, and bidirectional mind-body interaction. The proposed paradigm may provide new tools for revealing the mechanism of neural plasticity and promote the development of novel human-machine interaction devices; it is promising in the generation of practical values in the domain of personalized education, neural rehabilitation, and cognitive training of specialized personnel.

Funded by




[1] Gruzelier J H. EEG-neurofeedback for optimising performance. I: a review of cognitive and affective outcome in healthy participants. Neurosci Biobehav Rev, 2014, 44: 124--141. Google Scholar

[2] Zheng N, Liu Z, Ren P. Hybrid-augmented intelligence: collaboration and cognition. Front Inf Technol Electron Eng, 2017, 18: 153-179 CrossRef Google Scholar

[3] Lutz A, Greischar L L, Rawlings N B. Long-term meditators self-induce high-amplitude gamma synchrony during mental practice. Proc Natl Acad Sci USA, 2004, 101: 16369-16373 CrossRef PubMed ADS Google Scholar

[4] Gazzaniga M S, Ivry R B, Mangun G R. Cognitive Neuroscience. New York: Norton Company, 2014. Google Scholar

[5] Baddeley A. Working Memory. Science, 1992, 255: 556-559 CrossRef ADS Google Scholar

[6] Klingberg T, Forssberg H, Westerberg H. Increased brain activity in frontal and parietal cortex underlies the development of visuospatial working memory capacity during childhood.. J Cognitive Neuroscience, 2002, 14: 1-10 CrossRef PubMed Google Scholar

[7] Merzenich M M, Kaas J H, Wall J. Topographic reorganization of somatosensory cortical areas 3b and 1 in adult monkeys following restricted deafferentation. Neuroscience, 1983, 8: 33-55 CrossRef Google Scholar

[8] Wall J, Kaas J, Sur M. Functional reorganization in somatosensory cortical areas 3b and 1 of adult monkeys after median nerve repair: possible relationships to sensory recovery in humans. J Neurosci, 1986, 6: 218-233 CrossRef Google Scholar

[9] Maguire E A, Gadian D G, Johnsrude I S. Navigation-related structural change in the hippocampi of taxi drivers.. Proc Natl Acad Sci USA, 2000, 97: 4398-4403 CrossRef PubMed Google Scholar

[10] Elbert T, Pantev C, Wienbruch C. Increased Cortical Representation of the Fingers of the Left Hand in String Players. Science, 1995, 270: 305-307 CrossRef ADS Google Scholar

[11] Hebb D O. The Organization of Behavior, A Neuropsychological Theory. Hoboken: John Wiley, Chapman Hall, 1949. Google Scholar

[12] Song S, Miller K D, Abbott L F. Competitive Hebbian learning through spike-timing-dependent synaptic plasticity.. Nat Neurosci, 2000, 3: 919-926 CrossRef PubMed Google Scholar

[13] Markram H. Regulation of Synaptic Efficacy by Coincidence of Postsynaptic APs and EPSPs. Science, 1997, 275: 213-215 CrossRef Google Scholar

[14] Fan J, McCandliss B D, Sommer T. Testing the efficiency and independence of attentional networks.. J Cognitive Neuroscience, 2002, 14: 340-347 CrossRef PubMed Google Scholar

[15] Raz A, Buhle J. Typologies of attentional networks.. Nat Rev Neurosci, 2006, 7: 367-379 CrossRef PubMed Google Scholar

[16] Tang Y Y, Posner M I. Attention training and attention state training.. Trends Cognitive Sci, 2009, 13: 222-227 CrossRef PubMed Google Scholar

[17] McGonigal J. Reality is Broken: Why Games Make Us Better and How They Can Change the World. New York: Penguin Press, 2011. Google Scholar

[18] Baniqued P L. Cognitive training with casual video games: points to consider. Front Psychol, 2014, 4: 19. Google Scholar

[19] Latham A J, Patston L L M, Tippett L J. The virtual brain: 30 years of video-game play and cognitive abilities. Front Psychol, 2013, 4: 629. Google Scholar

[20] Oei A C, Patterson M D. Enhancing cognition with video games: a multiple game training study. PLoS One, 2013, 8: 16. Google Scholar

[21] Anguera J A, Boccanfuso J, Rintoul J L. Video game training enhances cognitive control in older adults. Nature, 2013, 501: 97-101 CrossRef PubMed ADS Google Scholar

[22] Green C S, Bavelier D. Action video game modifies visual selective attention.. Nature, 2003, 423: 534-537 CrossRef PubMed Google Scholar

[23] Bavelier D, Green C S, Davidson R J, et al. A national science foundation report. In: Proceedings of Workshop on Interactive Media, Attention, and Well-Being, Washington, 2012. Google Scholar

[24] Green C S, Pouget A, Bavelier D. Improved probabilistic inference as a general learning mechanism with action video games.. Curr Biol, 2010, 20: 1573-1579 CrossRef PubMed Google Scholar

[25] Montani V, de Grazia M D F, Zorzi M. A new adaptive videogame for training attention and executive functions: design principles and initial validation. Front Psychol, 2014, 5: 409. Google Scholar

[26] Green C S, Bavelier D. Learning, attentional control, and action video games. Curr Biol, 2012, 22: 197--206. Google Scholar

[27] Jaeggi S M, Buschkuehl M, Jonides J. From the Cover: Improving fluid intelligence with training on working memory. Proc Natl Acad Sci USA, 2008, 105: 6829-6833 CrossRef PubMed ADS Google Scholar

[28] Fukuda K, Vogel E, Mayr U. Quantity, not quality: the relationship between fluid intelligence and working memory capacity.. Psychonomic Bull Rev, 2010, 17: 673-679 CrossRef PubMed Google Scholar

[29] Jolles D D, van Buchem M A, Crone E A. Functional brain connectivity at rest changes after working memory training.. Hum Brain Mapp, 2013, 34: 396-406 CrossRef PubMed Google Scholar

[30] Meyer T, Qi X L, Stanford T R. Stimulus Selectivity in Dorsal and Ventral Prefrontal Cortex after Training in Working Memory Tasks. J Neuroscience, 2011, 31: 6266-6276 CrossRef Google Scholar

[31] Qi X L, Constantinidis C. Correlated discharges in the primate prefrontal cortex before and after working memory training.. Eur J Neurosci, 2012, 36: 3538-3548 CrossRef PubMed Google Scholar

[32] Constantinidis C, Klingberg T. The neuroscience of working memory capacity and training. Nat Rev Neurosci, 2016, 17: 438--449. Google Scholar

[33] Zhang S, Wang D, Afzal N. Rhythmic Haptic Stimuli Improve Short-Term Attention.. IEEE Trans Haptics, 2016, 9: 437-442 CrossRef PubMed Google Scholar

[34] Wang D, Zhang Y, Yang X. Force control tasks with pure haptic feedback promote short-term focused attention.. IEEE Trans Haptics, 2014, 7: 467-476 CrossRef PubMed Google Scholar

[35] Yang X, Wang D, Zhang Y. An adaptive strategy for an immersive visuo-haptic attention training game. In: Proceedings of the 10th International Conference on Haptics: Perception, Devices, Control, and Applications, London, 2016. 9774: 441--451. Google Scholar

[36] Jiang L, Guan C, Zhang H, et al. Brain computer interface based 3D game for attention training and rehabilitation. In: Proceedings of the 6th IEEE Conference on Industrial Electronics and Applications (ICIEA), Beijing, 2011. 124--127. Google Scholar

[37] Bohil C J, Alicea B, Biocca F A. Virtual reality in neuroscience research and therapy.. Nat Rev Neurosci, 2011, 12: 752-762 CrossRef PubMed Google Scholar

[38] Cho B H, Ku J, Jang D P. The effect of virtual reality cognitive training for attention enhancement.. CyberPsychology Behav, 2002, 5: 129-137 CrossRef PubMed Google Scholar

[39] Hasenkamp W, Wilson-Mendenhall C D, Duncan E. Mind wandering and attention during focused meditation: a fine-grained temporal analysis of fluctuating cognitive states.. NeuroImage, 2012, 59: 750-760 CrossRef PubMed Google Scholar

[40] deBettencourt M T, Cohen J D, Lee R F. Closed-loop training of attention with real-time brain imaging.. Nat Neurosci, 2015, 18: 470-475 CrossRef PubMed Google Scholar

[41] Csikszentmihalyi M. Flow and the Psychology of Discovery and Invention. New York: Harper Collins, 1996. Google Scholar

[42] Reiser R A. Introduction to special issue. J Instructional Dev, 1987, 10: 2-2 CrossRef Google Scholar

[43] Caporale N, Dan Y. Spike timing — dependent plasticity: a hebbian learning rule. Annu Rev Neurosci, 208, 31: 25--46. Google Scholar

[44] Petersen S E, Posner M I. The attention system of the human brain: 20 years after.. Annu Rev Neurosci, 2012, 35: 73-89 CrossRef PubMed Google Scholar

[45] Wang D X, Xu M, Zhanq Y R, et al. Preliminary study on haptic-stimulation based brainwave entrainment. In: Proceedings of World Haptics Conference, Daejeon, 2013. Google Scholar

[46] Niksirat K S, Silpasuwanchai C, Ahmed M M H, et al. A framework for interactive mindfulness meditation using attention-regulation process. In: Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems, Denver, 2017. 2672--2684. Google Scholar

Copyright 2020 Science China Press Co., Ltd. 《中国科学》杂志社有限责任公司 版权所有

京ICP备17057255号       京公网安备11010102003388号