Human Motion Tracking and Recognition

Contributor: Dongheui Lee 

Human motion capture and analysis is a strongly researched field with many applications in areas such as computer animation, video games, medical therapy, tele-presence, surveillance and human machine interaction. Our group developed approaches for human motion tracking and recognition using machine learning algorithm. We aim at real-time markerless solutions without much environmental settings. 

We developed a method for 3D whole-body motion recovery and motion recognition from a sequence of occluded and monocular camera images based on statistical inference using a 3D motion database. By proposing coordinate transformation of the statistical database, the motion database (compact and statistical representation of motion primitives) in joint space can be compared with 2D image sequences from any view point without the need for depth information. 3D motion recovery and motion recognition are performed simultaneously by coupling them in a single framework where recovery assists action recognition and vise versa. Instead of extracting rich information by expensive computation of image processing, the proposed method realized an inference mechanism from low level image features (e.g. optical flow), inspired by human motion perception shown in the moving light display experiments. The proposed algorithm can recover a reasonable 3D motion from 2D occluded and unlabeled marker data [1]. Also, after the market launch of low cost depth sensors such as Microsoft Kinect and Asus Xtion sensor, we contribute a novel approach, named Multiple Depth Camera Approach (MDCA), to track human motion from multiple Kinect sensors. The approach includes fast data fusion and association, a flexible shape model and an efficient estimation method based on particle filtering.


Related Publications

Journal articles

  1. Dongheui Lee and Yoshihiko Nakamura, Motion Recognition and Recovery from Occluded Monocular Observations. Robotics and Autonomous Systems, 62(6), pp. 818-832, 2014

Peer-reviewed conference papers

  1. Licong Zhang, Juergen Sturm, Daniel Cremers, and Dongheui Lee, Real-time Human Motion Tracking using Multiple Depth Cameras, in Proc. IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 2389-2395, 2012
  2. Ziyuan Liu, Dongheui Lee and Wolfgang Sepp, Particle Filter Based Monocular Human Tracking with a 3D Cardbox Model and a Novel Deterministic Resampling Strategy, in Proc. IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 3626-3631, 2011
  3. Dongheui Lee and Yoshihiko Nakamura, Motion Capturing from Monocular Vision by Statistical Inference Based on Motion Database: Vector Field Approach, in Proc. IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 617-623, San Diego, USA, Oct 29-Nov 2, 2007
  4. Dongheui Lee and Yoshihiko Nakamura, Mimesis Scheme using a Monocular Vision System on a Humanoid, in Proc. IEEE International Conference on Robotics and Automation (ICRA), pp. 2162-2168, Rome, Italy, Apr 10-14, 2007
  5. Dongheui Lee and Yoshihiko Nakamura, Stochastic Theory for Motion Capturing from Onboard Monocular Vision of Humanoid Robots, in Proc. 12th Robotics Symposia, 4B2, pp.424-429, Nagaoka, JAPAN, Mar 15-16, 2007
  6. Dongheui Lee and Yoshihiko Nakamura, 3D Human Motion Capturing from Monocular Images by Statistical Inference, 2nd International Symposium on Information and Robot Technology (ISIRT), pp.22-24. Tokyo, Japan, March 19, 2008
  7. Dongheui Lee and Yoshihiko Nakamura, Human Motion Understanding from 2D Partial Observation of Vector Field based on Particle Filter, Proceedings of the 2007 JSME Conference on Robotics and Mechatronics (ROBOMEC) , 1A2-M07 , Akita, Japan, May 10-12, 2007 (in Japanese)
  8. Dongheui Lee and Yoshihiko Nakamura, Motion Recognition with a Monocular Vision System Based on Mimesis Scheme, Proceedings of the 24th Annual Conference of the Robotics Society of Japan (RSJ), 1M23, Okayama,Japan, September 14-16, 2006 (in Japanese)