logo

SCIENTIA SINICA Informationis, Volume 50 , Issue 3 : 438-460(2020) https://doi.org/10.1360/N112018-00326

An interpretable gait recognition method based on time series features

More info
  • ReceivedDec 21, 2018
  • AcceptedApr 28, 2019
  • PublishedMar 3, 2020

Abstract

Gait recognition is a type of biometric recognition that can be used as an identification tool in various applications. Deep learning-based methods have recently exhibited promising accuracy in gait recognition tasks; however, in addition to an accurate prediction, these methods are required to explain the recognition results. The black-box nature of deep neural networks makes it very difficult to interpret the basis for their identification. The published studies on the interpretability of gait recognition are also in a blank state. Moreover, deep neural networks require a large amount of data to learn the model parameters and an effective generalization on unseen data is difficult when the problem size is small. Thus, this paper presents a gait recognition method combining accuracy and interpretability. The gait feature is represented as a multi-dimensional time series and a Shapelet-based time series classification method is used for gait recognition. A Shapelet is the most discriminative subsequence in time series that makes the proposed method provide interpretability and accuracy simultaneously. We conducted experiments on the CASIA-B dataset and compared the proposed method with several state-of-the-arts deep learning methods. Experiments show that the proposed method can provide an accuracy close to that of deep neural networks on small-scale data sets. At the same time, the decision-making reason of the model can be explained in detail. Concretely, our method can reveal discriminative gait features and frame numbers for specific subjects.


Funded by

国家自然科学基金(61672086,61702030,61771058)

北京市自然科学基金(4182052)

中央高校基本科研业务费专项资金(2017YJS036)


References

[1] Nixon M S, Tan T N, Chellappa R. Human Identification Based on Gait. Boston: Springer, 2005. Google Scholar

[2] Bouchrika I. A Survey of Using Biometrics for Smart Visual Surveillance: Gait Recognition. In: Advanced Sciences and Technologies for Security Applications. Berlin: Springer, 2018. Google Scholar

[3] Ju Han , Bir Bhanu . Individual recognition using gait energy image.. IEEE Trans Pattern Anal Mach Intell, 2006, 28: 316-322 CrossRef PubMed Google Scholar

[4] Bashir K, Xiang T, Gong S. Gait recognition using gait entropy image. In: Proceedings of the 3rd International Conference on Imaging for Crime Detection and Prevention, London, 2009. 1--6. Google Scholar

[5] Bashir K, Xiang T, Gong S. Gait recognition without subject cooperation. Pattern Recognition Lett, 2010, 31: 2052-2060 CrossRef Google Scholar

[6] Ji S, Xu W, Yang M. 3D convolutional neural networks for human action recognition.. IEEE Trans Pattern Anal Mach Intell, 2013, 35: 221-231 CrossRef PubMed Google Scholar

[7] Taigman Y, Yang M, Ranzato M A, et al. DeepFace: closing the gap to human-level performance in face verification. In: Proceedings of 2014 IEEE Conference on Computer Vision and Pattern Recognition, Columbus, 2014. 1701--1708. Google Scholar

[8] Alotaibi M, Mahmood A. Improved gait recognition based on specialized deep convolutional neural network. Comput Vision Image Understanding, 2017, 164: 103-110 CrossRef Google Scholar

[9] Shiraga K, Makihara Y, Muramatsu D, et al. GEINet: View-invariant gait recognition using a convolutional neural network. In: Proceedings of 2016 International Conference on Biometrics, Halmstad, 2016. 1--8. Google Scholar

[10] Kale A, Sundaresan A, Rajagopalan A N, et al. Identification ofhumans using gait. IEEE Trans Image Process, 2004, 13: 1163--1173. Google Scholar

[11] Shajina T, Sivakumar P B. Human gait recognition and classification using time series shapelets. In: Proceedings of 2012 International Conference on Advances in Computing and Communications, Cochin, 2012. 31--34. Google Scholar

[12] Wang J. Human gait recognition based on one-dimensional motion curve on the side view angle. Dissertation for Master Degree. Beijing: Beijing Jiaotong University, 2017. Google Scholar

[13] Yam C Y, Nixon M S, Carter J N. Automated person recognition by walking and running via model-based approaches. Pattern Recognition, 2004, 37: 1057-1072 CrossRef Google Scholar

[14] Ye L, Keogh E. Time series shapelets: a new primitive for data mining. In: Proceedings of the 15th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, Paris, 2009. 947--956. Google Scholar

[15] Karlsson I, Papapetrou P, Bostr?m H. Generalized random shapelet forests. Data Min Knowl Disc, 2016, 30: 1053-1085 CrossRef Google Scholar

[16] Shi M H, Wang Z H, Yuan J D, et al. Random pairwise shapelets forest. In: Proceedings of the 22nd Pacific-Asia Conference on Knowledge Discovery and Data Mining, Melbourne, 2018. 68--80. Google Scholar

[17] Liu L, Jia W, Zhu Y. Survey of gait recognition. In: Proceedings of 2009 International Conference on Intelligent Computing, Ulsan, 2009. 652--659. Google Scholar

[18] Johansson G. Visual perception of biological motion and a model for its analysis. Perception Psychophysics, 1973, 14: 201-211 CrossRef Google Scholar

[19] Cunado D, Nixon M S, Carter J N. Using gait as a biometric, via phase-weighted magnitude spectra. In: Proceedings of International Conference on Audio and Video-Based Biometric Person Authentication, Crans-Montana, 1997. 93--102. Google Scholar

[20] Fujiyoshi H, Lipton A J, Kanade T. Real-Time Human Motion Analysis by Image Skeletonization. IEICE Trans InfSyst, 2004, 87: 113--120. Google Scholar

[21] Rustagi L, Kumar L, Pallai G N. Human gait recognition based on dynamic and static features using generalized regression neural network. In: Proceedings of the 2nd International Conference on Machine Vision, Dubai, 2009. 64--68. Google Scholar

[22] Iwama H, Okumura M, Makihara Y. The OU-ISIR Gait Database Comprising the Large Population Dataset and Performance Evaluation of Gait Recognition. IEEE TransInformForensic Secur, 2012, 7: 1511-1521 CrossRef Google Scholar

[23] Lishani A O, Boubchir L, Bouridane A. Haralick Features for GEI-based Human Gait Recognition. In: Proceedings of the 26th International Conference on Microelectronics, Doha, 2014. 36--39. Google Scholar

[24] Krizhevsky A, Sutskever I, Hinton G E. ImageNet classification with deep convolutional neural networks. Commun ACM, 2017, 60: 84-90 CrossRef Google Scholar

[25] Szegedy C, Liu W, Jia Y, et al. Going deeper with convolutions. In: Proceedings of 2015 IEEE Conference on Computer Vision and Pattern Recognition, Boston, 2015. 1--9. Google Scholar

[26] Girshick R B, Donahue J, Darrell T, et al. Rich Feature Hierarchies for Accurate Object Detection and Semantic Segmentation. In: Proceedings of 2014 IEEE Conference on Computer Vision and Pattern Recognition, Columbus, 2014. 580--587. Google Scholar

[27] Girshick R B. Fast R-CNN. In: Proceedings of 2015 IEEE International Conference on Computer Vision, Santiago, 2015. 1440--1448. Google Scholar

[28] Ren S, He K, Girshick R B, et al. Faster R-CNN: towards real-time object detection with region proposal networks. In: Proceedings of Annual Conference on Neural Information Processing Systems 2015, Montreal, 2015. 91--99. Google Scholar

[29] He K M, Zhang X Y, Ren S Q. Spatial Pyramid Pooling in Deep Convolutional Networks for Visual Recognition.. IEEE Trans Pattern Anal Mach Intell, 2015, 37: 1904-1916 CrossRef PubMed Google Scholar

[30] Goodfellow I J, Abadie J P, Mirza M, et al. Generative adversarial nets. In: Proceedings of Annual Conference on Neural Information Processing Systems 2014, Montreal, 2014. 2672--2680. Google Scholar

[31] Isola P, Zhu J, Zhou T, et al. Image-to-image translation with conditional adversarial networks. In: Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, 2017. 5967--5976. Google Scholar

[32] Zhu J, Park T, Isola P, et al. Unpaired image-to-image translation using cycle-consistent adversarial networks. In: Proceedings of IEEE International Conference on Computer Vision, Venice, 2017. 2242--2251. Google Scholar

[33] Feng Y, Li Y, Luo J. Learning effective gait features using LSTM. In: Proceedings of the 23rd International Conference on Pattern Recognition, Cancun, 2016. 325--330. Google Scholar

[34] Wu Z, Huang Y, Wang L. A Comprehensive Study on Cross-View Gait Based Human Identification with Deep CNNs.. IEEE Trans Pattern Anal Mach Intell, 2017, 39: 209-226 CrossRef PubMed Google Scholar

[35] Chao H, He Y, Zhang J, et al. GaitSet: regarding gait as a set for cross-view gait recognition. In: Proceedings of the 33rd AAAI Conference on Artificial Intelligence, Honolulu, 2019. Google Scholar

[36] Zhang C, Liu W, Ma H, et al. Siamese neural network based gait recognition for human identification. In: Proceedings of 2016 IEEE International Conference on Acoustics, Speech and Signal Processing, Shanghai, 2016. 2832--2836. Google Scholar

[37] Mclaughlin N, Rincon J M, Miller P C. Recurrent Convolutional Network for Video-Based Person Re-identification. In: Proceedings of 2016 IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, 2016. 1325--1334. Google Scholar

[38] Yu S, Chen H, Reyes E B G, et al. GaitGAN: invariant gait feature extraction using generative adversarial networks. In: Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops, Honolulu, 2017. 532--539. Google Scholar

[39] Hills J, Lines J, Baranauskas E. Classification of time series by shapelet transformation. Data Min Knowl Disc, 2014, 28: 851-881 CrossRef Google Scholar

[40] Yoo J H, Nixon M S, Harris C J. Extracting human gait signatures by body segment properties. In: Proceedings of the 5th IEEE Southwest Symposium on Image Analysis and Interpretation, Santa Fe, 2002. 35--39. Google Scholar

[41] Breiman L. Random forest. Machine Learning, 2001, 45: 5-32 CrossRef Google Scholar

  • Figure 1

    (Color online) Selecting contour boundary to estimate the leg angle. The red line denotes the outline of the thigh, and the green line denotes the outline of the shin

  • Figure 2

    (Color online) Results of human pose estimation

  • Figure 3

    (Color online) Time series of subject (a) No. 1 and (b) No. 105's left shin angle

  • Figure 4

    (Color online) DMDI of time series of subject No. 1 and No. 105's left shin angle

  • Figure 5

    (Color online) Time series of subject (a) No. 1 and (b) No. 105's stride width

  • Figure 6

    (Color online) DMDI of time series of subject No. 1 and No. 105's stride width

  • Figure 7

    (Color online) Time series of subject (a) No. 8, (b) No. 21 and (c) No. 25's left shin angle

  • Figure 8

    (Color online) DMDI of time series of subject No. 8, No. 21 and No. 25's left shin angle

  • Figure 9

    (Color online) Training time comparison of MRPSF and CNN

  • Figure 10

    (Color online) Accuracy with Shapelet length increasing. (a) Normal condition; (b) bagging condition; protect łinebreak (c) clothing condition

  • Figure 11

    (Color online) Accuracy with ensemble size increasing. (a) Normal condition; (b) bagging condition; protect łinebreak (c) clothing condition

  •   

    Algorithm 1 MultiDimensional-RandomPairwiseShapeletsForest

    Input:Multi-dimensional time series dataset $D=\{D_1,\ldots,D_d\}$ ($D_i$ represents the $i$th dimension time series), dimension number $d$, Shapelet minimun length $l$, Shapelet maximum length $u$, tree number $p$, Shapelet candidate number $r$;

    for $i=1$ to $d$

    $R_i$ $\leftarrow$ $\emptyset$;

    end for

    for $i=1$ to $p$

    dimension=SelectRandomDimension();

    $I_i$ $\leftarrow$ Sample($D_{\rm~dimension}$);

    ${\rm~ST}_i$ $\leftarrow$ RandomPairwiseShapeletsTree($I_i$, $l$, $u$, $r$);

    $R_{\rm~dimension}$ $\leftarrow$ $R_{\rm~dimension}$ $\cup$ ${\rm~ST}_i$;

    end for

    return $R$;

    Output Set of decision tree sets trained from different dimensions $R=\{R_1,\ldots,R_d\}$ ($R_i$ represents trees trained by the $i$th dimension).

  • Table 1   Accuracy comparison between MRPSF and CNN in normal walking condition
    Algorithm 5-person experiment (%) 8-person experiment (%) 12-person experiment (%) 15-person experiment (%)
    MRPSF 96.92 97.35 94.55 92.53
    One-on-one Net 99.62 97.12 99.04 98.08
    GEINet 98.85 98.80 99.52 99.23
    GaitSet 100.00 100.00 100.00 100.00
    LBNet 94 92.5 98.33 98.67
  •   

    Algorithm 2 RandomPairwiseShapeletsTree

    Input:Time series dataset $D$, Shapelet minimum length $l$, Shapelet maximum length $u$, Shapelet candidate number $r$;

    if IsTerminal($D$) then

    return MakeLeaf($D$);

    end if

    $S$ $\leftarrow$ $\emptyset$;

    for $i=1$ to $r$

    $S$ $\leftarrow$ $S$ $\cup$ SampleShapeletsPair($D$, $l$, $u$);

    end for

    ($s_1$, $s_2$) $\leftarrow$ BestShapeletsPair($D$, $S$);

    ($D_1$, $D_2$) $\leftarrow$ Split($D$, $s_1$, $s_2$);

    ${\rm~ST}_l$ $\leftarrow$ RandomPairwiseShapeletsTree($D_1$, $l$, $u$, $r$);

    ${\rm~ST}_r$ $\leftarrow$ RandomPairwiseShapeletsTree($D_2$, $l$, $u$, $r$);

    return $s_1$, $s_2$, ${\rm~ST}_l$, ${\rm~ST}_r$;

    Output: A decision tree node consisting of a Shapelets pair $s_1$, $s_2$, left subtree ${\rm~ST}_l$ and right subtree ${\rm~ST}_r$. The node can be considered as a decision tree because it contains pointer to subtrees.

  • Table 2   Accuracy comparison between MRPSF and CNN in clothing walking condition
    Algorithm 5-person experiment (%) 8-person experiment (%) 12-person experiment (%) 15-person experiment (%)
    MRPSF 97.69 93.75 89.42 88.97
    One-on-one Net 99.23 96.63 94.87 97.18
    GEINet 100.00 97.12 98.08 97.69
    GaitSet 86.00 87.50 73.33 88.67
    LBNet 88.00 75.00 86.68 84.00
  • Table 3   Accuracy comparison between MRPSF and CNN in bagging walking condition
    Algorithm 5-person experiment (%) 8-person experiment (%) 12-person experiment (%) 15-person experiment (%)
    MRPSF 96.15 92.79 92.63 89.23
    One-on-one Net 94.62 95.19 91.99 92.82
    GEINet 95.38 95.19 93.91 94.36
    GaitSet 100.00 100.00 97.50 100.00
    LBNet 96.00 67.5 88.33 93.33
  •   

    Algorithm 3 AssessCandidatePair

    Input:A Shapelets pair $s_1$, $s_2$, time series dataset $D$;

    Gain $\leftarrow$ 0, Gap $\leftarrow$ 0, line$_1$ $\leftarrow$ 0, line$_2$ $\leftarrow$ 0;

    for $m=1$ to $|D|$

    $d_1$ $\leftarrow$ subdist($s_1$, $D[m]$);

    $d_2$ $\leftarrow$ subdist($s_2$, $D[m]$);

    if $d_1$ $\leq$ $d_2$ then

    ${\rm~set}_1$ $\leftarrow$ ${\rm~set}_1$ $\cup$ $D[m]$;

    else

    ${\rm~set}_2$ $\leftarrow$ ${\rm~set}_2$ $\cup$ $D[m]$;

    end if

    end for

    Gain $\leftarrow$ InfoGain(${\rm~set}_1$, ${\rm~set}_2$);

    Gap $\leftarrow$ SepGap(${\rm~set}_1$, ${\rm~set}_2$);

    return Gain, Gap;

    Output: Information gain Gain, separation gap Gap.

  • Table 4   Training cost comparison between MRPSF and CNN
    Algorithm Training time Using CUDA
    MRPSF 0 h 3 min 59 s No
    One-on-one Net 0 h 5 min 17 s No
    GaitSet 5 h 17 min 26 s Yes
    LBNet 1 h 26 min 17 s Yes

Copyright 2020 Science China Press Co., Ltd. 《中国科学》杂志社有限责任公司 版权所有

京ICP备17057255号       京公网安备11010102003388号