Video comes to Firefly

Video Comes to Firefly Adobe has announced a significant expansion of its Firefly generative AI capabilities, introducing video generation features that promise to bring the might of Adobe software development to benefit the creative workflow for video professionals. This move marks a crucial step in Adobe's AI strategy, bringing the power of generative AI to video production and editing.

## New Video Features in Firefly Adobe's Firefly video model introduces three primary features that really will transform video creation and editing processes. They are:

At the time of writing, Firefly will create a variety of scenes, from landscapes to abstract animations but restricts videos to 5 seconds in length. It accepts natural language prompts and is ideal for concept visualisation, storyboarding, short social media content or video mock-ups. Other applications might include advertising teams creating concept videos for pitches, filmmakers visualising scenes before shooting or educators creating visual aids for lessons.

Image to Video In contrast to writing prompts, users can also upload still images for conversion into short video clips. This feature could be particularly useful for creating supplementary B-roll footage or filling gaps in production schedules. Firefly will transform static images into dynamic video sequences that maintain the style and content of the original image, while generating realistic motion and transitions. Potential uses of this feature could include adding life to archival photos or stock images or bringing historical or other important photos to life. Commercial applications are numerous: image-to-video would be a great way to create engaging product demonstrations from product photos, for news outlets to create video content from still images or social media managers to enhance photo-based posts.

Generative Extend This feature is similar to Photoshop's Generative Expand for images. With Generative Extend, Firefly will add two seconds to the end of a video clip, maintaining visual consistency with the original footing, while preserving in-camera movement and scene dynamics. It can also generate background audio to match the extended footage.

There are plenty of potential applications for Generative Extend in Firefly. It could be useful for fine-tuning edit timing, smoothing transitions and adjusting clip lengths to match audio. Filmmakers could use it to extend establishing shots or reaction shots and in live production, to extend shots to cover unexpected pauses or transitions. Content creators would find it useful for optimising video lengths for specific platforms.

Together, these features offer video professionals and content creators powerful new tools to enhance their workflows, initiate creative ideas and solve common production challenges.

Integration with Adobe Creative Suite As the technology develops, we can expect to see even more sophisticated applications and integrations within Adobe's Creative Suite. The integration of Firefly's video capabilities into Adobe's existing software ecosystem promises to revolutionise workflows for video professionals and content creators.

Premiere Pro The Generative Extend feature will be embedded as beta into Premiere Pro later this year, allowing editors to seamlessly extend their footage within their familiar editing environment. This integration will enable video editors to extend clips by up to two seconds, maintaining visual consistency and camera movements smooth out transitions between scenes. It will enable editors to adjust timing and pacing without the need for additional footage and generate background audio to match the extended video content. This feature is likely to be particularly valuable for editors working within tight deadlines or with limited footage, as it provides a quick solution for fine-tuning video lengths and transitions.

Standalone Firefly Application Adobe is planning to make the Text to Video and Image to Video features available in beta form as a standalone Firefly application, accessible through the Adobe website. This approach will allow Adobe to gather user feedback and refine the features before full integration. In this period, it's thought that Adobe will provide broader access to these tools, even for users who may not have a full Creative Cloud subscription, so that the company can showcase the capabilities of Firefly's video generation technology to a wider audience.

Remember at this point, users will be able to generate no more than five-second video clips from text prompts or still images. But even at this relatively short duration, this opens up new possibilities for rapid prototyping and content creation.

Future Integrations Adobe has indicated that the Firefly video model will eventually be incorporated into its Creative, Experience and Express platforms, expanding its reach across the Adobe ecosystem. This broader integration could include:

Together, these features could significantly help in the creation of sophisticated visual effects and motion graphics, the time saved allowing visual artists to focus more on the creative direction of their project rather than technical execution.

Character Animator: Revolutionising Animation Creation Potential integrations here would include text-to-animation conversion for rapid prototyping, dynamic background generation based on scene descriptions and intelligent pose suggestions for character movements. Also possible would be AI-generated facial expressions and lip-syncing and the automated rigging of 2D characters. With Firefly-like capabilities, Character Animator could become an even more powerful tool for creating animated content, particularly for social media and short-form video production.

Adobe Express: Democratising Video Creation As the consumer app that lets users create content such as social media posts, images and videos, Firefly-style AI video would simplify the video creation process in Adobe Express. It would allow users to create text-prompted video clips, in the appropriate sizes for different social media platforms. For existing clips, users would be able to trim and edit them, automatically generate subtitles and captions and perform basic "one-click" colour grading.

These features could make Adobe Express an even more accessible platform for marketers, small business owners and content creators who want to produce high-quality video content without extensive technical knowledge.

Adobe Experience Manager: Personalising Digital Marketing Adobe Experience Manager is a software solution that helps businesses manage their digital content across multiple channels, including websites, mobile apps and forms. Possibilities for integrating AI-generated video in AEM would include:

As these integrations roll out, we can expect to see a more seamless workflow between different Adobe applications, allowing creators to take advantage of Firefly's AI-powered video generation across various stages of their projects. This comprehensive ecosystem approach could significantly enhance both productivity and creativity for professionals across multiple industries, from filmmaking and animation to marketing and e-commerce.

Technical Capabilities and Current Limitations While the full extent of Firefly's video capabilities is yet to be revealed, Adobe has shared some insights into the model's performance.

Video Duration Currently, videos created through the Text to Video and Image to Video tools will have a maximum duration of five seconds. However, Adobe has suggested that this limit may increase in the future. This will make Firefly eminently suitable for short-form content and social media clips and for creating quick visual concepts or storyboards, as well a great way to develop and prototype ideas. Limitations for longer-form content creation will remain for some time but over a period of years, we can only expect the duration of generated work to increase.

Quality Early demonstrations suggest that the quality of generated videos is comparable to other leading AI video models, such as OpenAI's Sora. High-resolution output will ensure smooth motion and transitions, accurate representation of textures and lighting and a consistent style throughout the clip. This will make it suitable for professional use in many different industries.

Audio Generation For extended clips, Firefly video can recreate background noise, such as traffic or the ambient sounds of nature. However, it does not generate human voices or music because of licensing constraints. Within these constraints, Firefly will synchronise generated audio with video content with a decent ability to match audio style to the visual scene. In the early stages of this new Ai feature, users should anticipate the need for additional audio editing or sourcing. Workarounds for creators at this stage would include:

Ethical considerations, Commercial Safety and Training Data Ethical considerations are a constant with AI generated content, since the models on which it is trained include a huge variety of sources. Adobe states very clearly that it only trains Adobe Firefly on content where it has permission or rights to do so. All Adobe products have in-built safeguards against generating inappropriate content and Adobe has stated that transparency in AI-generated content for legal and ethical compliance continues to play an important role in the development and refinement of the Firefly Video Model.

A key differentiator for Adobe's Firefly video model is its focus on commercial safety. The model has been trained using Adobe Stock images, openly licensed content and public domain resources. This approach aims to mitigate potential copyright issues, making the generated content safer for commercial use. AI model risks are numerous and include model poisoning (the malicious injection of false and misleading data into the training dataset), bias and hallucination.

The concept of "appropriate" content is always at the forefront of AI-generated imagery. Adobe has implemented safeguards to prevent the generation of inappropriate content and the model blocks the creation of videos containing nudity, drugs and alcohol. Additionally, it is not trained on images of recognisable or public figures or celebrities, further reducing potential ethical concerns.

The introduction of Adobe's Firefly's video capabilities has significant implications for various creative roles:

Video Editors For video editors, the Generative Extend feature could streamline the editing process, allowing for quick adjustments to clip durations without the need for additional footage. Editors can extend existing footage by up to two seconds, giving them the potential to solve common issues such as jump cuts or abrupt endings and fine-tune timing and pace in post-production. Other uses might include using it to maintain visual consistency and camera movements, to extend establishing shots to allow for longer title sequences, to smooth transitions between scenes, adjust clip durations to match audio cues or voiceovers and create pause moments for emphasis without reshooting.

Motion Graphics Designers Motion graphics designers too will find that the Text to Video and Image to Video tools within Firefly AI video can serve well as generators for more complex animations, potentially speeding up the ideation and prototyping phases of their work. Just being able to rapidly generate base footage for further manipulation and inspiration for visual styles and motion concepts provides a worthwhile start. Designers could also use the Firefly AI video features to: