Markerless Motion Capture of Interacting Characters Using Multi-view Image Segmentation

Yebin Liu, Carsten Stoll, Juergen Gall, Hans-Peter Seidel, and Christian Theobalt

Abstract

We present a markerless motion capture approach that reconstructs the skeletal motion and detailed time-varying surface geometry of two closely interacting people from multi-view video. Due to ambiguities in feature-to-person assignments and frequent occlusions, it is not feasible to directly apply single-person capture approaches to the multiperson case. We therefore propose a combined image segmentation and tracking approach to overcome these difficulties. A new probabilistic shape and appearance model is employed to segment the input images and to assign each pixel uniquely to one person. Thereafter, a single-person markerless motion and surface capture approach can be applied to each individual, either one-by-one or in parallel, even under strong occlusions. We demonstrate the performance of our approach on several challenging multi-person motions, including dance and martial arts, and also provide a reference dataset for multi-person motion capture with ground truth.

Images/Videos

Video CVPR'11 ~30MB (AVI)

Video TPAMI'13 ~80MB (MP4)

Data

The data used in the paper is available for research purposes. To obtain the data, please contact Carsten Stoll. Some sequences are also available at http://media.au.tsinghua.edu.cn/mtrack.htm. When using the data, please acknowledge the effort that went into data collection by referencing the corresponding paper.

Publications

Liu Y., Gall J., Stoll C., Dai Q., Seidel H.-P., and Theobalt C., Markerless Motion Capture of Multiple Characters Using Multi-view Image Segmentation (PDF), IEEE Transactions on Pattern Analysis and Machine Intelligence, To appear.

Liu Y., Stoll C., Gall J., Seidel H.-P., and Theobalt C., Markerless Motion Capture of Interacting Characters Using Multi-view Image Segmentation (PDF), IEEE Conference on Computer Vision and Pattern Recognition (CVPR'11), 1249-1256, 2011. ©IEEE

Gall J., Stoll C., de Aguiar E., Theobalt C., Rosenhahn B., and Seidel H.-P., Motion Capture Using Joint Skeleton Tracking and Surface Estimation (PDF), IEEE Conference on Computer Vision and Pattern Recognition (CVPR'09), 2009. © IEEE

Slides: Markerless Motion Capture of Interacting Characters Using Multi-view Image Segmentation (PPTX), IEEE Conference on Computer Vision and Pattern Recognition (CVPR'11), Colorado Springs, CO, USA, 2011