SCIENTIA SINICA Informationis, Volume 47 , Issue 9 : 1164-1182(2017) https://doi.org/10.1360/N112017-00075

Multimodal aided neurological disease diagnosis with synergy of cloud and client

More info
  • ReceivedApr 18, 2017
  • AcceptedJul 27, 2017
  • PublishedSep 5, 2017


The dominating physiological characteristics of neurological diseases are reflected in patients' daily behaviors. The aided diagnosis and early warning of neurological diseases will benefit from obtaining and analyzing related physiological information generated during the interaction process. Traditional systems for detecting neurological conditions only analyze a single interaction modal, which may lose important features contained in other modalities. Based on the above, we propose a multi-modal aided neurological disease diagnosis system with synergy of cloud and client. First, we propose an automatic disease diagnosis method based on multi-modal information; users' physiological information collected from multiple modals is then analyzed and the results are integrated to improve accuracy and robustness. Second, a framework of cloud-client synergy is proposed that stores the user's physiological information from different regions and different times in the cloud, thus reducing the geographical restrictions and time constraints of data collection. Third, based on the powerful computing capacity, the system can make real-time and precise automatic diagnosis by analyzing multi-modal physiological information produced by users during natural interaction. Finally, the effectiveness of this multi-modal method for diagnosing aided neurological diseases based on cloud-client synergy is verified by using a hybrid diagnostic system of voice and pen.

Funded by





[1] Wirdefeldt K, Adami H O, Cole P, et al. Epidemiology and etiology of Parkinson's disease: a review of the evidence. Eur J Epidemiol, 2011, 26: 1--58. Google Scholar

[2] Morgado P M. Automated diagnosis of Alzheimer's disease using PET images a study of alternative procedures for feature extraction and selection electrical and computer engineering. Dissertation for M.S. Degree. Lisbon: Technical University of Lisbon, 2012. Google Scholar

[3] Smith E E, O'Donnell M, Dagenais G. Early cerebral small vessel disease and brain volume, cognition, and gait.. Ann Neurol, 2015, 77: 251-261 CrossRef PubMed Google Scholar

[4] Rijk M C D, Launer L J, Berger K, et al. Prevalence of Parkinson's disease in Europe: a collaborative study of population-based cohorts. Neurology, 2000, 54: 21--23. Google Scholar

[5] Prince M, Ali G C, Guerchet M. Recent global trends in the prevalence and incidence of dementia, and survival with dementia.. Alz Res Ther, 2016, 8: 23 CrossRef PubMed Google Scholar

[6] Kim J S. Small vessel disease: not a small problem.. J Stroke, 2015, 17: 1 CrossRef PubMed Google Scholar

[7] Skodda S, Gronheit W, Mancinelli N, et al. Progression of voice and speech impairment in the course of Parkinson's disease: a longitudinal study. Parkinson's Dis, 2013, 2013: 389195. Google Scholar

[8] Khedher L, Ramírez J, Górriz J M. Early diagnosis of Alzheimer?s disease based on partial least squares, principal component analysis and support vector machine using segmented MRI images. Neurocomputing, 2015, 151: 139-150 CrossRef Google Scholar

[9] Jaimes A, Sebe N. Multimodal human?Ccomputer interaction: A survey. Comp Vision Image Understanding, 2007, 108: 116-134 CrossRef Google Scholar

[10] Zhang W, Smith M L, Smith L N. Eye center localization and gaze gesture recognition for human-computer interaction. J Opt Soc Am A, 2016, 33: 314-325 CrossRef ADS Google Scholar

[11] Zhang H, Wang Y, Bai X. Cyclic tensile strain on vocal fold fibroblasts inhibits cigarette smoke-induced inflammation: implications for Reinke edema.. J Voice, 2015, 29: 13-21 CrossRef PubMed Google Scholar

[12] Roy N, Barkmeier-Kraemer J, Eadie T. Evidence-Based Clinical Voice Assessment: A Systematic Review. Am J Speech Lang Pathol, 2013, 22: 212-226 CrossRef Google Scholar

[13] MacCallum J K, Zhang Y, Jiang J J. Vowel selection and its effects on perturbation and nonlinear dynamic measures.. Folia Phoniatr Logop, 2011, 63: 88-97 CrossRef PubMed Google Scholar

[14] Instructions for Contributors. J Voice, 2016, 30: A7-A9 CrossRef Google Scholar

[15] Moers C, M?bius B, Rosanowski F. Vowel- and text-based cepstral analysis of chronic hoarseness.. J Voice, 2012, 26: 416-424 CrossRef PubMed Google Scholar

[16] Alnasheri A, Muhammad G, Alsulaiman M, et al. An investigation of multidimensional voice program parameters in three different databases for voice pathology detection and classification. J Voice, 2016, 31: 9--18. Google Scholar

[17] Bin L, Zheng D, Yu N. An Improved Weighted Centroid Localization Algorithm. IJFGCN, 2013, 6: 45-52 CrossRef Google Scholar

[18] Tsanas A, Gómez-Vilda P. Novel robust decision support tool assisting early diagnosis of pathological voices using acoustic analysis of sustained vowels. In: Proceedings of Multidisciplinary Conference of Users of Voice, Speech and Singing, Las Palmas, 2013. 3--12. Google Scholar

[19] Ünlü A, Brause R, Krakow K. Handwriting analysis for diagnosis and prognosis of parkinson's disease. In: Proceedings of the 7th International Conference on Biological and Medical Data Analysis, Thessaloniki, 2006. 441--450. Google Scholar

[20] Standardized Handwriting to Assess Bradykinesia, Micrographia and Tremor in Parkinson's Disease. PLoS ONE, 2014, 9: e97614 CrossRef PubMed ADS Google Scholar

[21] Kim H, Cho Y S, Do E Y. Context-bounded refinement filter algorithm: improving recognizer accuracy of handwriting in clock drawing test. In: Proceedings of National Conference on Artificial Intelligence, Atlanta, 2010. 53--60. Google Scholar

[22] Davis R, Libon D, Au R. THink: Inferring Cognitive Status from Subtle Behaviors. AIMag, 2015, 36: 49-60 CrossRef Google Scholar

[23] Hagler S, Jimison H B, Pavel M. Assessing executive function using a computer game: computational modeling of cognitive processes.. IEEE J Biomed Health Inform, 2014, 18: 1442-1452 CrossRef PubMed Google Scholar

[24] Drake J M, Griffen B D. Early warning signals of extinction in deteriorating environments. Nature, 2010, 467: 456-459 CrossRef PubMed ADS Google Scholar

[25] Chen D Y, Ding H, Zhou X Y, et al. A system of measuring and analyzing gait of human walking. Chin J Biomed Eng, 1997, 16: 133--141. Google Scholar

[26] Sinha A, Chakravarty K, Bhowmick B, et al. Person identification using skeleton information from kinect. In: Proceedings of International Conference on Advances in Computer-Human Interaction, Nice, 2013. 101--108. Google Scholar

[27] Paul M, Haque S M E, Chakraborty S. Human detection in surveillance videos and its applications - a review. EURASIP J Adv Signal Process, 2013, 2013: 176 CrossRef ADS Google Scholar

[28] Lan K C, Shih W Y. Early Diagnosis of Parkinson's Disease Using a Smartphone. Procedia Comp Sci, 2014, 34: 305-312 CrossRef Google Scholar

[29] Chen Y, Huang M, Hu C, et al. A coarse-to-fine feature selection method for accurate detection of cerebral small vessel disease. In: Proceedings of International Symposium on Neural Networks, Saint Petersburg, 2016. 2609--2616. Google Scholar

[30] Dirican A C, G?ktürk M. Psychophysiological measures of human cognitive states applied in human computer interaction. Procedia Comp Sci, 2011, 3: 1361-1367 CrossRef Google Scholar

[31] Altun K, Barshan B. Human activity recognition using inertial/magnetic sensor units. In: Proceedings of International Conference on Pattern Recognition, Istanbul, 2010. 38--51. Google Scholar

[32] Patel S, Lorincz K, Hughes R. Monitoring motor fluctuations in patients with Parkinson's disease using wearable sensors.. IEEE Trans Inform Technol Biomed, 2009, 13: 864-873 CrossRef PubMed Google Scholar

[33] Robichaud J A, Pfann K D, Leurgans S, et al. Variability of EMG patterns: a potential neurophysiological marker of Parkinson's disease? Clin Neurophysiol, 2009, 120: 390--397. Google Scholar

[34] Jordan K G. Emergency EEG and continuous EEG monitoring in acute ischemic stroke. Clin Neurophysiol, 2004, 21: 341--352. Google Scholar

[35] Armbrust M, Fox A, Griffith R, et al. A view of cloud computing. Commun ACM, 2010, 53: 50--58. Google Scholar

[36] Kamara S, Lauter K. Cryptographic cloud storage. In: Proceedings of International Conference on Financial Cryptography and Data Security, Tenerife, 2010. 136--149. Google Scholar

[37] Subashini S, Kavitha V. A survey on security issues in service delivery models of cloud computing. J Network Comp Appl, 2011, 34: 1-11 CrossRef Google Scholar

[38] Wang H, Yi X, Bertino E. Protecting outsourced data in cloud computing through access management. Concurrency Computat-Pract Exper, 2016, 28: 600-615 CrossRef Google Scholar

[39] Tian F, Dai G Z, Chen Y D, et al. Specification and structure design of 3D interaction tasks. J Softw, 2002, 13: 2099--2105. Google Scholar

[40] Huang J, Chen Y N, Liu J, et al. Multimodal human-computer interaction model for nerve function assessment in mobile environment. J Softw, 2016, 27: 156--171. Google Scholar

[41] Hanchuan Peng , Fuhui Long , Ding C. Feature selection based on mutual information: criteria of max-dependency, max-relevance, and min-redundancy.. IEEE Trans Pattern Anal Machine Intell, 2005, 27: 1226-1238 CrossRef PubMed Google Scholar

[42] Chandrashekar G, Sahin F. A survey on feature selection methods. Comp Electrical Eng, 2014, 40: 16-28 CrossRef Google Scholar

[43] Kumari B, Swarnkar T. Filter versus wrapper feature subset selection in large dimensionality micro array: a review. Int J Comput Sci Inf Technol, 2011, 2: 1048--1053. Google Scholar

[44] Polikar R. Ensemble based systems in decision making. IEEE Circuits Syst Mag, 2006, 6: 21-45 CrossRef Google Scholar

[45] Miao K H, Miao J H, Miao G J. Diagnosing coronary heart disease using ensemble machine learning. Int J Adv Comput Sci Appl, 2016, 7: 30--39. Google Scholar

[46] Yang J J, Li J, Shen R. Exploiting ensemble learning for automatic cataract detection and grading.. Comp Methods Programs Biomed, 2016, 124: 45-57 CrossRef PubMed Google Scholar

[47] Inzamam-Ul-Hossain M, MacKinnon L, Islam M R. Parkinson disease detection using ensemble method in PASW benchmark. In: Proceedings of IEEE International Advance Computing Conference, Banglore, 2015. 666--670. Google Scholar

[48] Lezak M D. Neuropsychological Assessment. New York: Oxford University Press, 1995. 650--680. Google Scholar

[49] Arnett J A, Labovitz S S. Effect of physical layout in performance of the Trail Making Test.. Psychological Assessment, 1995, 7: 220-221 CrossRef Google Scholar

[50] Sofranko J L, Prosek R A. The effect of levels and types of experience on judgment of synthesized voice quality.. J Voice, 2014, 28: 24-35 CrossRef PubMed Google Scholar

[51] Tsanas A, Little M A, McSharry P E. Accurate telemonitoring of Parkinson's disease progression by noninvasive speech tests.. IEEE Trans Biomed Eng, 2010, 57: 884-893 CrossRef PubMed Google Scholar

[52] Yumoto E, Gould W J, Baer T. Harmonics-to-noise ratio as an index of the degree of hoarseness. J Acoust Soc Am, 1982, 71: 1544-1550 CrossRef ADS Google Scholar

[53] Srinivasan B, Spinner T, Rengaswamy R. Control loop performance assessment using detrended fluctuation analysis (DFA). Automatica, 2012, 48: 1359-1363 CrossRef Google Scholar

[54] Godino-Llorente J I, Osma-Ruiz V, Sáenz-Lechón N. The effectiveness of the glottal to noise excitation ratio for the screening of voice disorders.. J Voice, 2010, 24: 47-56 CrossRef PubMed Google Scholar

[55] Arias-Londo?o J D, Godino-Llorente J I, Sáenz-Lechón N. An improved method for voice pathology detection by means of a HMM-based feature space transformation. Pattern Recognition, 2010, 43: 3100-3112 CrossRef Google Scholar

[56] Hall M A. Correlation-based feature subset selection for machine learning. Dissertation for Ph.D. Degree. Hamilton: University of Waikato, 1998. Google Scholar

[57] Sellam V, Jagadeesan J. Classification of normal and pathological voice using SVM and RBFNN. J Signal Inf Process, 2014, 5: 1--7. Google Scholar

[58] Seixas F L, Zadrozny B, Laks J. A Bayesian network decision model for supporting the diagnosis of dementia, Alzheimer?s disease and mild cognitive impairment.. Comp Biol Med, 2014, 51: 140-158 CrossRef PubMed Google Scholar

[59] Kalyani R R, Corriere M, Ferrucci L. Age-related and disease-related muscle loss: the effect of diabetes, obesity, and other diseases. Lancet Diabetes Endocrinology, 2014, 2: 819-829 CrossRef Google Scholar

[60] Ransmayr G, Künig G, Neubauer M. Effect of age and disease duration on parkinsonian motor scores under levodopa therapy. J Neural Transm Gen Sect, 1995, 9: 177-188 CrossRef Google Scholar

  • Figure 1

    Cloud-Client synergy based multi-modal aided neurological diseasediagnosis framework

  • Figure 2

    Neurological disease diagnosis based on multi-modal interaction

  • Figure 3

    Case study of neurological disease diagnosis based on pen and audio interaction

  • Figure 4

    (Color online) (a) Voice data collection; (b) pen based trail making test

  • Figure 5

    (Color online) A diagram of Stacking ensemble_learning classifier

  • Figure 6

    (Color online) Classification effects of five models on different datasets

  • Table 1   Extraction characteristics of voice data and their meanings
    DFA1一段时间内声波信号的包络变化趋势, 噪声的随机自相似性
    发音器官运动MFCC84声波的整体衡量, 频域表示, 短时能量谱, 波形的细微扰动
  • Table 2   Extraction characteristics of pen gesture and their meanings
    位移 Acceleration112笔尖的瞬时加速度, 反映手部运动的力量变化
    UMO40笔尖位置与当前笔迹意图中心的偏离程度, 反映无意识运动程度
    Curvature76当前笔迹在该笔尖位置上的弯曲程度, 反映笔迹的笔直程度
    压力Pressure speed36笔尖压力的变化速度
    Pressure accuracy36笔尖压力的变化加速度, 反映手部运动的纵向力量变化
    笔身姿态 Posture speed148笔身角度的变化速度
    Posture accuracy148笔身角度的变化加速度, 反映手部运动在三维空间中的力量变化
  • Table 3   Classification effects of five different models
    ModelAccuracy (%)PrecisionRecall$F$ value
    DMEM (Ex)89.220.9050.8790.885
  • Table 4   Classification effects of DMEM (Ex) on different ages and genders
    Sample setAccuracy (%)PrecisionRecall$F$ value
    Age 0 $\sim$ 6486.950.8980.8270.846
    Age 65 $\sim$ 7490.400.8460.8520.834
    Age above 75 94.620.8080.8410.819
  • Table 5   Classification effects of DMEM (Ex) on different ages
    Age rangeAccuracy (%)PrecisionRecall$F$ value
    0 $\sim$ 6489.140.9160.8580.873
    65 $\sim$ 7486.990.8450.8240.813
    Age above 75 94.470.8060.8200.807
  • Table 6   Classification effects of DMEM (Ex) on different genders
    ModelAccuracy (%)PrecisionRecall$F$ value

Copyright 2020 Science China Press Co., Ltd. 《中国科学》杂志社有限责任公司 版权所有

京ICP备17057255号       京公网安备11010102003388号