Get 20M+ Full-Text Papers For Less Than $1.50/day. Start a 14-Day Trial for You or Your Team.

Learn More →

A Temporal Order Modeling Approach to Human Action Recognition from Multimodal Sensor Data

A Temporal Order Modeling Approach to Human Action Recognition from Multimodal Sensor Data From wearable devices to depth cameras, researchers have exploited various multimodal data to recognize human actions for applications, such as video gaming, education, and healthcare. Although there many successful techniques have been presented in the literature, most current approaches have focused on statistical or local spatiotemporal features and do not explicitly explore the temporal dynamics of the sensor data. However, human action data contain rich temporal structure information that can characterize the unique underlying patterns of different action categories. From this perspective, we propose a novel temporal order modeling approach to human action recognition. Specifically, we explore subspace projections to extract the latent temporal patterns from different human action sequences. The temporal order between these patterns are compared, and the index of the pattern that appears first is used to encode the entire sequence. This process is repeated multiple times and produces a compact feature vector representing the temporal dynamics of the sequence. Human action recognition can then be efficiently solved by the nearest neighbor search based on the Hamming distance between these compact feature vectors. We further introduce a sequential optimization algorithm to learn the optimized projections that preserve the pairwise label similarity of the action sequences. Experimental results on two public human action datasets demonstrate the superior performance of the proposed technique in both accuracy and efficiency. http://www.deepdyve.com/assets/images/DeepDyve-Logo-lg.png ACM Transactions on Multimedia Computing, Communications, and Applications (TOMM) Association for Computing Machinery

A Temporal Order Modeling Approach to Human Action Recognition from Multimodal Sensor Data

Download PDF
22 pages

Loading next page...
 
/lp/association-for-computing-machinery/a-temporal-order-modeling-approach-to-human-action-recognition-from-0yZcleeWtT
Publisher
Association for Computing Machinery
Copyright
Copyright © 2017 ACM
ISSN
1551-6857
eISSN
1551-6865
DOI
10.1145/3038917
Publisher site
See Article on Publisher Site

Abstract

From wearable devices to depth cameras, researchers have exploited various multimodal data to recognize human actions for applications, such as video gaming, education, and healthcare. Although there many successful techniques have been presented in the literature, most current approaches have focused on statistical or local spatiotemporal features and do not explicitly explore the temporal dynamics of the sensor data. However, human action data contain rich temporal structure information that can characterize the unique underlying patterns of different action categories. From this perspective, we propose a novel temporal order modeling approach to human action recognition. Specifically, we explore subspace projections to extract the latent temporal patterns from different human action sequences. The temporal order between these patterns are compared, and the index of the pattern that appears first is used to encode the entire sequence. This process is repeated multiple times and produces a compact feature vector representing the temporal dynamics of the sequence. Human action recognition can then be efficiently solved by the nearest neighbor search based on the Hamming distance between these compact feature vectors. We further introduce a sequential optimization algorithm to learn the optimized projections that preserve the pairwise label similarity of the action sequences. Experimental results on two public human action datasets demonstrate the superior performance of the proposed technique in both accuracy and efficiency.

Journal

ACM Transactions on Multimedia Computing, Communications, and Applications (TOMM) – Association for Computing Machinery

Published: Mar 6, 2017

Keywords: Human action recognition

References

  • Comparative study on classifying human activities with miniature inertial and magnetic sensors
    Altun, Kerem; Barshan, Billur; Tunçel, Orkun
  • Recognizing daily and sports activities in two open source machine learning environments using body-worn sensor units
    Barshan, Billur; Yüksek, Murat Cihan
  • The recognition of human movement using temporal templates
    Bobick, Aaron F.; Davis, James W.
  • Lasagne: First Release
    Dieleman, Sander; Schlüter, Jan; Raffel, Colin; Olson, Eben; Sønderby, Søren Kaae; Nouri, Daniel; Maturana, Daniel
  • Behavior recognition via sparse spatio-temporal features
    Dollár, Piotr; Rabaud, Vincent; Cottrell, Garrison; Belongie, Serge
  • Hierarchical recurrent neural network for skeleton based action recognition
    Du, Yong; Wang, Wei; Wang, Liang
  • A short introduction to boosting
    Freund, Yoav; Schapire, Robert; Abe, Naoki
  • Human activities recognition using depth images
    Gupta, Raj; Chia, Alex Yong-Sang; Rajan, Deepu
  • Discriminative human action recognition in the learned hierarchical manifold space
    Han, Lei; Wu, Xinxiao; Liang, Wei; Hou, Guangming; Jia, Yunde
  • Activity recognition from acceleration data based on discrete consine transform and SVM
    He, Zhenyu; Jin, Lianwen
  • Long short-term memory
    Hochreiter, Sepp; Schmidhuber, Jürgen
  • Modeling and assessing quality of information in multisensor multimedia monitoring systems
    Hossain, M. Anwar; Atrey, Pradeep K.; Saddik, Abdulmotaleb El
  • Jointly learning heterogeneous features for RGB-D activity recognition
    Hu, Jian-Fang; Zheng, Wei-Shi; Lai, Jianhuang; Zhang, Jianguo
  • On space-time interest points
    Laptev, Ivan
  • A survey on human activity recognition using wearable sensors
    Lara, Oscar D.; Labrador, Miguel A.
  • Action recognition based on a bag of 3D points
    Li, Wanqing; Zhang, Zhengyou; Liu, Zicheng
  • Recognition and segmentation of 3-D human action using HMM and multi-class AdaBoost
    Lv, Fengjun; Nevatia, Ramakant
  • Motion templates for automatic classification and retrieval of motion capture data
    Müller, Meinard; Röder, Tido
  • Deep convolutional and LSTM recurrent neural networks for multimodal wearable activity recognition
    Ordóñez, Francisco Javier; Roggen, Daniel
  • HON4D: Histogram of oriented 4D normals for activity recognition from depth sequences
    Oreifej, Omar; Liu, Zicheng
  • A comparison of feature extraction methods for the classification of dynamic activities from accelerometer data
    Preece, Stephen J.; Goulermas, John Yannis; Kenney, Laurence P. J.; Howard, David
  • Spatial-geometric approach to physical mobile interaction based on accelerometer and IR sensory data fusion
    Rahman, Abu Saleh Md Mahfujur; Hossain, M. Anwar; Saddik, Abdulmotaleb El
  • Real-time human pose recognition in parts from single depth images
    Shotton, Jamie; Sharp, Toby; Kipman, Alex; Fitzgibbon, Andrew; Finocchio, Mark; Blake, Andrew; Cook, Mat; Moore, Richard
  • Mining actionlet ensemble for action recognition with depth cameras
    Wang, Jiang; Liu, Zicheng; Wu, Ying; Yuan, Junsong
  • View invariant human action recognition using histograms of 3D joints
    Xia, L.; Chen, C.-C.; Aggarwal, J. K.
  • Super normal vector for activity recognition using depth sequences
    Yang, Xiaodong; Tian, Yingli
  • Recognizing actions using depth motion maps-based histograms of oriented gradients
    Yang, Xiaodong; Zhang, Chenyang; Tian, Yingli
  • First-take-all: Temporal order-preserving hashing for 3D action videos
    Ye, Jun; Hu, Hao; Li, Kai; Qi, Guo-Jun; Hua, Kien A.
  • WTA hash-based multimodal feature fusion for 3D human action recognition
    Ye, Jun; Li, Kai; Hua, Kien A.
  • 2015c
    Ye, Jun; Li, Kai; Qi, Guo-Jun; Hua, Kien A.
  • Sensor-based abnormal human-activity detection
    Yin, Jie; Yang, Qiang; Pan, Jeffrey Junfeng
  • Segmentation of discriminative patches in human activity video
    Zhang, Bo; Conci, Nicola; De Natale, Francesco G. B.
  • Microsoft Kinect sensor and its effect
    Zhang, Zhengyou
  • Online human gesture recognition from motion data streams
    Zhao, Xin; Li, Xue; Pang, Chaoyi; Zhu, Xiaofeng; Sheng, Quan Z.
  • Fusing spatiotemporal features and joints for 3D action recognition
    Zhu, Yu; Chen, Wenbin; Guo, Guodong