Papers
arxiv:2406.05338

MotionClone: Training-Free Motion Cloning for Controllable Video Generation

Published on Jun 8
· Submitted by myownskyW7 on Jun 13
Authors:
,
,
,
,
,
,

Abstract

Motion-based controllable text-to-video generation involves motions to control the video generation. Previous methods typically require the training of models to encode motion cues or the fine-tuning of video diffusion models. However, these approaches often result in suboptimal motion generation when applied outside the trained domain. In this work, we propose MotionClone, a training-free framework that enables motion cloning from a reference video to control text-to-video generation. We employ temporal attention in video inversion to represent the motions in the reference video and introduce primary temporal-attention guidance to mitigate the influence of noisy or very subtle motions within the attention weights. Furthermore, to assist the generation model in synthesizing reasonable spatial relationships and enhance its prompt-following capability, we propose a location-aware semantic guidance mechanism that leverages the coarse location of the foreground from the reference video and original classifier-free guidance features to guide the video generation. Extensive experiments demonstrate that MotionClone exhibits proficiency in both global camera motion and local object motion, with notable superiority in terms of motion fidelity, textual alignment, and temporal consistency.

Community

Couldn't agree more with the idea that simulating from an existing video is a clever shortcut. Looking forward to its use in game CG and other areas.

·
Paper author

Thank you for your recognition. We will release our code as soon as possible.

Paper author Paper submitter
edited Jun 14

2024-06-14 15.07.25.gif

Sign up or log in to comment

Models citing this paper 0

No model linking this paper

Cite arxiv.org/abs/2406.05338 in a model README.md to link it from this page.

Datasets citing this paper 0

No dataset linking this paper

Cite arxiv.org/abs/2406.05338 in a dataset README.md to link it from this page.

Spaces citing this paper 0

No Space linking this paper

Cite arxiv.org/abs/2406.05338 in a Space README.md to link it from this page.

Collections including this paper 6