A Deep Learning Framework For Character Motion Synthesis and Editing

16/05/2016

This year at SIGGRAPH I am presenting A Deep Learning Framework For Character Motion Synthesis and Editing. This paper covers a broad range of techniques you can use to apply deep learning to character animation. This includes motion synthesis - generating motion from some parameterization such as the trajectory, as well as motion editing - adjusting the motion in a natural way to achieve the desired goal. It also includes an application of style transfer.

Webpage • Paper • Video • Slides • Code • Data• Talk

Abstract: We present a framework to synthesize character movements based on high level parameters, such that the produced movements respect the manifold of human motion, trained on a large motion capture dataset. The learned motion manifold, which is represented by the hidden units of a convolutional autoencoder, represents motion data in sparse components which can be combined to produce a wide range of complex movements. To map from high level parameters to the motion manifold, we stack a deep feedforward neural network on top of the trained autoencoder. This network is trained to produce realistic motion sequences from parameters such as a curve over the terrain that the character should follow, or a target location for punching and kicking. The feedforward control network and the motion manifold are trained independently, allowing the user to easily switch between feedforward networks according to the desired interface, without re-training the motion manifold. Once motion is generated it can be edited by performing optimization in the space of the motion manifold. This allows for imposing kinematic constraints, or transforming the style of the motion, while ensuring the edited motion remains natural. As a result, the system can produce smooth, high quality motion sequences without any manual pre-processing of the training data.