2D Action Recognition Serves 3D Human Pose Estimation

Juergen Gall, Angela Yao, and Luc Van Gool

Abstract

3D human pose estimation in multi-view settings benefits from embeddings of human actions in low-dimensional manifolds, but the complexity of the embeddings increases with the number of actions. Creating separate, action-specific manifolds seems to be a more practical solution. Using multiple manifolds for pose estimation, however, requires a joint optimization over the set of manifolds and the human pose embedded in the manifolds. In order to solve this problem, we propose a particle-based optimization algorithm that can efficiently estimate human pose even in challenging in-house scenarios. In addition, the algorithm can directly integrate the results of a 2D action recognition system as prior distribution for optimization. In our experiments, we demonstrate that the optimization handles an 84D search space and provides already competitive results on HumanEva with as few as 25 particles.

Images/Videos

(a) Silhouettes are extracted by background subtraction. (b) Tracks are built over the entire sequence and classified by a 2D action recognition system. (c) Confidences of each action are used to distribute the particles over the action-specific manifolds. (d) Final pose is obtained by optimizing over the manifolds.

Video ~40MB (AVI)

Source Code

By installing, copying, or otherwise using this Software, you agree to be bound by the terms of the license agreement (non-commercial use only). If you do not agree, do not install copy or use the Software. The Software is protected by copyright and other intellectual property laws and is licensed, not sold.

Redistribution and use in source and binary forms, with or without modification, are permitted for any non-commercial purpose provided that the following conditions are met:

1. Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer.
2. Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution.

Some purposes which can be non-commercial are teaching, academic research, public demonstrations and personal experimentation. You may also distribute this Software with books or other teaching materials, or publish the Software on websites, that are intended to teach the use of the Software for academic or other non-commercial purposes.
You may not use or distribute this Software or any derivative works in any form for commercial purposes. Examples of commercial purposes would be running business operations, licensing, leasing, or selling the Software, distributing the Software for use with commercial products, using the Software in the creation or use of commercial products or any other activity which purpose is to procure a commercial gain to you or others.

THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL JUERGEN GALL AND CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

Parts of the source code were contributed by Thomas Brox, Bodo Rosenhahn, and Mohammed Shaheen.

When using this software, please acknowledge the effort that went into development by referencing the corresponding paper.

If you have questions concerning the source code, please contact gall@vision.ee.ethz.

Download

Publications

Yao A., Gall J., and van Gool L., Coupled Action Recognition and Pose Estimation from Multiple Views (PDF), International Journal of Computer Vision, Vol 100(1), 16-37, Springer, 2012. ©Springer

Gall J., Yao A., and van Gool L., 2D Action Recognition Serves 3D Human Pose Estimation (PDF), European Conference on Computer Vision (ECCV'10), 2010. ©Springer

Gall J., Yao A., and van Gool L., Optimizing Over a Set of Manifolds (PDF), Technical Report 271, Computer Vision Laboratory, ETH Zurich, Switzerland, June 2010.