AI companies are making rapid progress in the field of video generation. In recent months, we have seen companies like Stability AI and Pika Labs introduce models capable of creating various types of videos using text and image prompts. Now, Microsoft AI has joined the race with its latest project called DragNUWA, which aims to provide even more control over video production. Unlike its predecessors, DragNUWA incorporates trajectory-based generation, allowing users to manipulate objects and video frames with specific trajectories. This approach enables highly controllable video generation from semantic, spatial, and temporal aspects, resulting in high-quality output. Microsoft has generously open-sourced the model weights and demo for the DragNUWA project, encouraging the community to explore and experiment. However, it’s important to note that this is still an ongoing research effort and not yet perfect.
Improving Fine-Grained Control
Traditionally, AI-driven video generation has relied on inputs from text, images, or trajectories. While these approaches have been successful to some extent, they have struggled to deliver fine-grained control over the desired output. For example, combining text and images alone may fail to capture intricate motion details present in a video. Similarly, using images and trajectories may not adequately represent future objects and trajectories, and language can introduce ambiguity when expressing abstract concepts. To address these limitations, Microsoft’s AI team proposed DragNUWA in August 2023. This open-domain diffusion-based video generation model combines all three factors – text, images, and trajectories – to enable highly controllable video generation from semantic, spatial, and temporal aspects. With DragNUWA, users can define the desired text, image, and trajectory inputs to precisely control camera movements, object motion, and other aspects of the output video.
“This would result in a video of the boat sailing in the marked direction, giving the desired outcome.”
By leveraging the trajectory for motion details, language for future objects, and images for object distinction, DragNUWA can produce videos that accurately reflect the user’s intentions.
A Promising Future
In the initial 1.5 version of DragNUWA released on Hugging Face, Microsoft incorporated Stability AI’s Stable Video Diffusion model to animate images or objects along specific paths. Once further developed, this technology could revolutionize video generation and editing processes. Imagine being able to effortlessly transform backgrounds, animate images, and direct motion paths simply by drawing a line. This exciting development in creative AI has garnered attention from AI enthusiasts, who consider it a significant leap forward. However, it’s essential to assess how the research model will perform in real-world scenarios.
“Firstly, DragNUWA supports complex curved trajectories, enabling the generation of objects moving along specific intricate paths. Secondly, DragNUWA allows for variable trajectory lengths, resulting in different motion amplitudes. Lastly, DragNUWA has the capability to simultaneously control the trajectories of multiple objects.”
The researchers at Microsoft are optimistic about the potential of DragNUWA in advancing controllable video generation and its future applications. This work contributes to the growing body of research in the AI video space. Recently, Pika Labs made headlines by unveiling its text-to-video interface, which functions similar to ChatGPT and produces high-quality short videos with customizable options.
