The AI Video Showdown: Runway vs Leonardo vs Pika Labs

Curious Refuge
5 Jan 202425:48

TLDRThis week's AI film news covers exciting advancements in AI filmmaking tools. Mid Journey 6 has made significant improvements, enabling creators to produce high-quality short films with ease. The upcoming release of Mid Journey's own video model is anticipated to be highly promising. Leonardo's new video generation tool challenges Runway and P Labs, offering consistent character animation and dynamic camera movement. P Labs has released P 1.0, allowing anyone to experiment with their animation capabilities. Open voice, a voice cloning tool, and hand refiner, which corrects AI-generated hands, are introduced. The potential of text-based prompt experiences for animation is highlighted by tools like motion GPT and M mask. Dream tuner allows training custom models from a single image, which could greatly benefit character consistency in various projects. Lastly, AI-generated music through tools like Sunno is improving, hinting at a future where AI music may be indistinguishable from real-world recordings. The community is also encouraged to participate in AI filmmaking courses and competitions for exposure and growth in the field.

Takeaways

  • 🎬 **Mid Journey 6**: A significant upgrade from its predecessor, offering impressive results in video generation, including a short film about puppies getting adopted.
  • 🚀 **P Labs & Mid Journey 6**: A visual effects demo showcasing a WWII battle scene, highlighting P Labs' strength in visual effects shots compared to other tools.
  • 🌟 **Leonardo's Video Tool**: A new entry in the video generation space that provides strong competition to Runway and P Labs, with consistent character animation and dynamic camera movement.
  • 📈 **Runway's Ambient Motion**: Introduction of a new feature allowing for the creation of ambient motion in images, enhancing realism with adjustable motion intensity.
  • 📚 **Telescope Magazine**: Runway's upcoming publication featuring generative AI artists, with a unique design that prohibits its use for AI algorithm training.
  • 📹 **Text-to-Video Aspect Ratios**: Runway's new capability to alter the aspect ratio of text-to-video creations, catering to various formats from social media to cinematic projects.
  • 🎓 **AI Filmmaking Course**: An upcoming course starting on January 9th for students to learn alongside visual effects supervisors and talented artists.
  • 🎄 **Pika Labs' Animation**: Official release of P 1.0 allowing public access, with showcased examples of animated characters indicating the potential of AI in animation.
  • 🎙️ **Open Voice Cloning**: A new voice cloning tool that, while currently not perfect, offers the intriguing ability to modify the style of voice delivery, pointing towards future advancements.
  • 🤲 **Hand Refiner Tool**: A solution for common AI-generated image issues with hands, improving their appearance to be more realistic.
  • ✍️ **Text-Based Animation**: Emerging tools like motion GPT and M mask that allow for text prompt-based animation, suggesting a future where animations are generated from textual descriptions.

Q & A

  • What is the latest update in the AI film-making industry mentioned in the transcript?

    -The latest updates include new video generation tools, a music tool that creates music, and a new motion tool inside of Runway. Mid Journey 6 is also highlighted as a notable improvement with amazing results, although the online interface is not yet available.

  • How did Rory utilize Mid Journey 6 to create a short film?

    -Rory used Runway and Mid Journey 6 to create a short film about puppies getting adopted, which is described as really cute with high quality.

  • What is special about the visual effects demo using Mid Journey 6?

    -The visual effects demo using Mid Journey 6 created a World War II battle scene with P Labs and Mid Journey 6, showcasing the potential future of visual effects with high consistency in character and clothing throughout the video.

  • What is the limitation of Mid Journey 6 at the current stage?

    -Mid Journey 6 is slightly limited as it does not currently allow for more control over upscaling or in-painting, and lacks some of the advanced features found in previous versions of Mid Journey.

  • What exciting news was shared by Mid Journey's CEO during their office hours?

    -The CEO announced that they are training their own video model, which is expected to be ready in a few months and promises to be very high-quality, given Mid Journey's reputation for quality.

  • How does Leonardo's new video generation tool compare to Runway and P Labs?

    -Leonardo's new tool is competitive with Runway and P Labs, offering consistent character and clothing throughout videos with dynamic camera movement. However, it currently only animates footage created from an image inside their platform.

  • What is the unique feature of Leonardo's real-time generation tool?

    -Leonardo's real-time generation tool allows users to type in text and generate imagery on the fly, with adjustable sliders to change the overall style, such as making it look like a coloring book or achieving a cinematic result.

  • What is the name of the magazine released by the Runway team?

    -The magazine released by the Runway team is called Telescope, which features writeups and artwork from artists in the generative AI space.

  • How does Runway's new ambient motion feature work?

    -Runway's new ambient motion feature allows users to create ambient motion using their motion brush tool. Users can paint over areas they want to animate and adjust an ambient slider to control the amount of motion in the scene.

  • What is the significance of P Labs releasing P 1.0 to everyone?

    -The release of P 1.0 to everyone means that there are no longer restrictions on who can join the application, allowing more users to experiment with Pika Labs' AI-generated animations.

  • What new tool is available for fixing issues with fingers in AI-generated images?

    -A new tool called Hand Refiner is available to fix issues with fingers in AI-generated images. It integrates into Comfy UI and can make hands appear more regular and less like typical AI-generated hands.

  • How does the voice cloning tool Open Voice work?

    -Open Voice works by uploading a few seconds of reference audio. The tool then uses this audio to clone a voice, which can be adjusted in style, such as changing the delivery to a whisper.

Outlines

00:00

🎬 AI Filmmaking Tools Update: Mid Journey 6 & Leonardo

The video script introduces new advancements in AI filmmaking, highlighting the release of Mid Journey 6, an upgrade from its predecessor with impressive results, though currently limited in advanced features. Rory utilized Mid Journey 6 and Runway to create an adorable short film about puppies getting adopted. The script also discusses a visual effects demo using Mid Journey 6 by martial artwork, emphasizing the tool's strength in visual effect shots. Additionally, fashion design examples using Mid Journey 6 are mentioned. The CEO of Mid Journey announced an upcoming video model, expected to be of high quality. Leonardo, another new video generation tool, is compared to Runway and P Labs in terms of consistency and dynamic camera movement. The script invites viewers to identify which clips were generated by which tool in a quiz-like manner.

05:03

🎨 Real-time Image Styling and Runway's New Ambient Motion Feature

The script discusses Leonardo's real-time generation tool, which allows users to create imagery by simply typing in text, with adjustable sliders for style changes. It also covers Runway's new feature for creating ambient motion using a motion brush tool, which allows for more nuanced control over the direction of movement in generated images. The feature is demonstrated with an example of animating grass and waves. Runway also introduced the ability to change the aspect ratio for text-to-video creations, catering to various social media and cinematic formats. An AI filmmaking course starting on January 9th is mentioned, along with P Labs releasing P 1.0 to the public.

10:05

📹 AI Animation and Voice Cloning Tools: Pika Labs and Open Voice

The script showcases Pika Labs' ability to create animated characters and discusses the potential of AI-generated animation. Open Voice, a new voice cloning tool, is introduced as a free service that clones a voice using a short reference audio. Although the voice quality is not yet professional, the tool's ability to change the style of voice delivery is noted as a significant step towards the future of voice acting. A new tool called Hand Refiner is also mentioned, designed to fix common AI-generated image issues with hands.

15:05

🤖 Advanced AI Tools for Animation and Lip Sync: motion GPT and Dream Tuner

The script highlights a white paper from China about an advanced lip-syncing tool that can adjust a subject's emotion using reference imagery. motion GPT is introduced as a tool that generates motion based on text prompts within a chat environment. M mask is another tool that can produce animations from text prompts, with the ability to apply the results to 3D characters. Dream Tuner, a tool that trains a custom model from a single image, is also discussed, noting its potential for character consistency and implications for advertising.

20:05

📈 AI Job Market and Technological Advancements in AI

The script discusses the lucrative job market for AI researchers and the potential for artists to increase their earnings by mastering AI tools. It mentions Mistral AI's plan to release a competitive model in 2024 and Open AI's prediction of a breakthrough year for AI. The advancements in AI-generated music using a tool called sunno are highlighted, with an example of the generated music provided. A white paper from the University of Texas and Meta is also mentioned, which allows for the creation of footage in different styles.

25:06

🏆 AI Film Festivals and Student Projects: Celebrating Creativity

The script concludes with a showcase of AI films, including 'The End of the World' by Andre, a spec ad for Adidas by Dave Clark, a bank heist film called 'Loose Ends' by a Curious Refuge student and team member, and a project called 'A Lakeside Spell' by JC, which features a poem narrated over curated images. The episode ends with an invitation to subscribe for the latest AI filmmaking news and a thank you note to the community.

Mindmap

Keywords

💡AI film making

AI film making refers to the process of creating films or film-like content using artificial intelligence tools and algorithms. In the context of the video, AI film making is the central theme, showcasing how various AI tools are being used to generate videos, visual effects, and even voice acting. It is exemplified by the creation of short films and visual demos using platforms like Runway, Leonardo, and Pika Labs.

💡Mid Journey 6

Mid Journey 6 is an advanced AI video generation tool mentioned in the video. It is noted for its significant improvements over its predecessor, Mid Journey 5. The tool is used to create high-quality short films and visual effects, as demonstrated by Rory's short film about puppies getting adopted. It represents the progression of AI in enhancing creative storytelling through technology.

💡Leonardo

Leonardo is an AI generation tool that competes with Runway and Pika Labs, as mentioned in the script. It is highlighted for its ability to generate motion videos from still images with a high degree of consistency and dynamic camera movement. Leonardo's real-time generation tool is also mentioned, which allows for quick iterations and adjustments to imagery based on text inputs.

💡Pika Labs

Pika Labs, often abbreviated as P Labs in the script, is an AI video generation platform that specializes in animation and visual effects. It is traditionally recognized for its prowess in creating visual effect shots. The video discusses a comparison between P Labs and other tools, noting the differences in the quality and realism of the generated animations.

💡Voice cloning

Voice cloning is the process of replicating a person's voice using AI technology. The video introduces a tool called Open Voice, which is capable of cloning a voice with just a few seconds of reference audio. This technology is significant for the future of voice acting and could potentially transform the way voices are used in films and other media.

💡Hand refiner

Hand refiner is a tool designed to correct common issues with AI-generated images, particularly with hands that may appear distorted or unrealistic. The tool integrates into a user interface like Comfy UI to adjust and normalize hands, improving the photorealism of AI-generated content. It is an example of how AI tools are evolving to address specific challenges in content creation.

💡Telescope magazine

Telescope is a magazine released by the team at Runway, which features write-ups and artwork from artists within the generative AI space. The magazine is noted for its design, including the edges of the pages creating a unique design. It also humorously includes a note prohibiting the use of the magazine to train AI algorithms, reflecting on the irony of AI's role in creative industries.

💡Text-to-video

Text-to-video is a feature within Runway that allows users to generate videos from textual prompts. The video discusses the ability to change the aspect ratio of the generated footage, catering to various formats from social media to cinematic projects. This feature exemplifies the versatility of AI tools in creating content tailored to different platforms and needs.

💡AI-generated music

AI-generated music refers to the creation of musical compositions using AI algorithms. The video mentions a tool called Sunno, which has been used to create music for projects with impressive results. The tool's ability to generate music quickly from prompts suggests a future where AI-generated music could be as common and high-quality as real-world compositions.

💡Dream Tuner

Dream Tuner is an AI tool introduced in the video that allows for the training of a custom model from a single image. This capability is significant as it enables the creation of consistent characters across multiple images and styles, which is particularly useful for projects like anime and live-action films. It represents a step towards more personalized and efficient AI-assisted content creation.

💡AI film jobs board

The AI film jobs board is a resource mentioned in the video that curates job listings related to AI film making. It serves as a platform for professionals to find opportunities in the growing field of AI and creative industries. The existence of such a board underscores the increasing demand and professionalization of AI skills in the film and entertainment sectors.

Highlights

Mid Journey 6 is a notable improvement from its predecessor, offering impressive results in video generation.

Rory used Runway and Mid Journey 6 to create a short film about puppies getting adopted, showcasing the tool's capabilities.

A visual effects demo using Mid Journey 6 created a World War II battle scene, illustrating the future of visual effects.

Mid Journey 6 is user-friendly, with settings that can be easily accessed and adjusted.

Mid Journey's CEO announced the development of their own video model, expected to be market-ready in a few months.

Leonardo's new video generation tool is a strong contender against Runway and P Labs.

Leonardo's tool allows for the conversion of generated images into video with consistent character and clothing throughout.

A comparison of AI generation tools shows varying levels of realism and movement in animations.

Leonardo's real-time generation tool is in Early Access, offering quick image generation based on text prompts.

Runway's new feature allows for the creation of ambient motion in images, enhancing the realism of generated scenes.

Runway's 'Telescope' magazine features generative AI artwork and is set to hit shelves soon.

P Labs has released P 1.0, making it accessible to everyone and showcasing their advancements in animation.

Open Voice is a new voice cloning tool that uses only a few seconds of reference audio to create a voice clone.

Hand Refiner is a tool designed to fix common issues with AI-generated hands, improving photorealism.

A new AI filmmaking course starting on January 9th will teach alongside visual effects supervisors and talented artists.

Dream Tuner is a tool that allows training a custom model from a single image, useful for character consistency in various styles.

Researchers working on AI projects are reportedly making upwards of $1 million a year, indicating a high demand for AI expertise.

Sunno is an AI-generated music tool that has been updated, producing high-quality results for creative projects.

AI-generated music is expected to become almost indistinguishable from real-world recordings in the near future.

The University of Texas and Meta's white paper presents a tool for creating stylized footage with stability and consistency.