Prompt Engineering - Part1 : Prompt Tricks You Probably Missed for Stable Diffusion

Scott Detweiler
13 Jun 202309:05

TLDRIn this video, Scott Weller discusses prompt engineering for AI models, specifically focusing on the Stable Diffusion model. He introduces various techniques to manipulate prompts during the inference phase, such as switching between different prompts using braces and pipe characters, and adjusting the balance between two elements with the 'from' and 'to' method or decimal values. Weller also demonstrates how to remove or add elements to a prompt after a certain number of steps, which can be useful for creating dynamic scenes. He emphasizes the importance of revisiting these fundamental techniques to enhance creativity and produce high-quality results with AI models.


  • 🚀 **Prompt Switching**: Scott Weller introduces the concept of switching between different prompts during inference to create unique combinations.
  • 🎨 **Artistic Inspiration**: He emphasizes the importance of not solely relying on one artist's style but instead creating a unique blend by mixing artists.
  • 🔄 **Using Braces and Pipes**: Demonstrates the use of braces and pipes to alternate between two different concepts, such as an airship and a train.
  • 🔢 **From and To Method**: Introduces a method to specify the number of steps before switching from one concept to another using a colon and step numbers or percentages.
  • ➗ **Decimal Method**: Explains how to use a decimal to control the blending ratio of two concepts, allowing for a gradual transition from one to the other.
  • 🛠️ **Adding and Removing Elements**: Shows how to add or remove elements from a prompt after a certain number of steps, which can be used for dramatic effects.
  • 🌌 **Creative Combinations**: Highlights the potential for combining various elements like a diesel punk race car with a pirate ship to create a unique scene.
  • 📈 **Controlling the Scene**: Discusses the ability to finely tune the composition by controlling when and how elements are introduced or removed from the prompt.
  • 📚 **Returning to Basics**: Scott encourages viewers to revisit and master the basics of prompt crafting, which are often overlooked in favor of new techniques.
  • 📈 **Mastering Tools**: Stresses the importance of not just briefly handling tools but truly understanding and mastering them for better results.
  • 📅 **Upcoming Content**: Mentions future content including advanced videos on prompt engineering and a potential podcast for sharing AI news and insights.

Q & A

  • What is the main topic of discussion in the video?

    -The main topic of the video is prompt engineering and prompt craft for working with AI models, specifically focusing on techniques to switch between different prompts during the inference process.

  • Why did Scott Weller take a break from making videos?

    -Scott Weller took a break from making videos because he took a job with Stability, where he has been doing quality assurance since November.

  • What is the purpose of using braces and pipe characters in the prompt?

    -The braces and pipe characters are used to alternate between two different concepts during the inference process. The brace groups a concept, and the pipe character followed by a closed brace indicates an alternation or switch between the concepts.

  • How does the 'from and to' method work in the prompt?

    -The 'from and to' method allows the user to specify the number of steps or percentage at which the AI should switch from one concept to another in the prompt. It provides more control over the transition between concepts.

  • What does removing the second part of the prompt after two colons signify?

    -Removing the second part of the prompt after two colons instructs the AI to take one concept, like 'Airship', and replace it with nothing after a certain number of steps, effectively removing that word from the prompt for the remainder of the inference.

  • How can the decimal method be used to control the mix of two concepts in a prompt?

    -The decimal method allows for a percentage-based mix of two concepts. By specifying a decimal, such as 0.5, the AI understands that the first concept should be used for half of the inference steps, and the second concept for the other half.

  • What is the significance of the step count in the 'from and to' method?

    -The step count in the 'from and to' method determines the specific step at which the AI should switch from using the first concept to the second concept in the prompt.

  • How does adding a concept to the prompt after a certain number of steps work?

    -By using a single colon followed by the concept, the AI is instructed to add that concept to the prompt after a specified number of steps, allowing for a gradual introduction of a new element into the inference.

  • What is the advantage of using the decimal method over the step method?

    -The decimal method provides a more intuitive way to control the mix of concepts by representing proportions as percentages, which might be easier for some users to conceptualize compared to specifying exact step numbers.

  • Can the prompt engineering techniques discussed be applied to all AI models?

    -Yes, the prompt engineering techniques discussed, such as switching between prompts and using the 'from and to' method, can be applied to all different models within the AI system.

  • What is Scott Weller's plan for future content after this video?

    -Scott Weller plans to release more advanced videos on prompt engineering very soon and is considering doing a podcast to share more information, particularly AI news that he comes across.

  • How can viewers provide feedback or suggest topics for future videos?

    -Viewers can provide feedback or suggest topics for future videos by leaving comments below the video, which Scott Weller will consider for his upcoming content.



📹 Returning to Video Creation and Exploring Prompt Engineering

Scott Weller announces his return to video creation after a brief hiatus due to a new job in quality assurance. He plans to delve into 'prompt engineering' using the tool Automatic 1111, aiming to revisit and highlight overlooked basic features amidst the introduction of new functionalities. Upcoming content includes advanced videos and a potential podcast focusing on AI news, leveraging his access to specialized information. He introduces a tutorial on dynamically switching between prompts during inference, using an example of blending airship and train images to demonstrate the technique.


🔧 Advanced Prompt Manipulation Techniques

Scott explores more sophisticated prompt manipulation techniques using the example of a race car combined with a pirate ship. He demonstrates how to control the blending of these elements through a decimal and step method, allowing for precise adjustment of how and when different images merge during the rendering process. The goal is to provide viewers with tools to fine-tune the content generation, offering insights into creating more complex and tailored visual outputs. He concludes by encouraging feedback and promising more foundational tips and tricks to help viewers master the available tools.



💡Prompt Engineering

Prompt engineering refers to the strategic crafting of prompts used in AI models to achieve specific outputs or behaviors. In the video, this term is central to exploring different techniques for optimizing interactions with the model Automatic 1111, emphasizing how changes in prompts affect the AI's output. The discussion on how to manipulate and switch between prompts illustrates the importance of this skill in controlling the generative process.

💡Stable Diffusion

Stable Diffusion is a type of generative AI model that is often used to create images from textual descriptions. The speaker in the video discusses applying different prompt techniques in Automatic 1111, which suggests an involvement of similar generative models. The use of advanced prompt tricks aims to enhance the quality and specificity of outputs produced by these models.

💡Automatic 1111

Automatic 1111 appears to be the specific AI model or platform the speaker is using to demonstrate prompt engineering techniques. It serves as a practical example for the audience to understand how different prompt strategies can be implemented in real-time AI applications, influencing the model's response and the final generation of content.

💡Brace notation

In the context of the video, brace notation is used as a prompt trick where the user can alternate between different words or concepts during the generative process. By enclosing words in braces and separating them with a pipe character, the model alternates between these inputs at different inference steps, showcasing a method to blend or switch concepts dynamically.


Steampunk is a genre of science fiction that incorporates technology and aesthetic designs inspired by 19th-century industrial steam-powered machinery. The speaker uses steampunk as a theme to create a unique prompt involving airships, illustrating how specific genres can inspire the creation of distinct visual outputs in AI models.


An airship is typically a large, powered, steerable balloon that is used as a form of aircraft. The video uses an airship as an example to demonstrate how to manipulate AI prompts to generate various iterations by mixing it with other concepts like trains, showcasing the flexibility of AI in creating diverse visual interpretations.


In the video, a train is used as part of a creative prompt that combines it with an airship. This example demonstrates how users can blend different transport modes in AI-generated imagery, leveraging the model's ability to interpret and synthesize complex and disparate elements into a cohesive output.

💡From-to method

The 'from-to method' is a prompt engineering technique discussed in the video, where a user specifies when the AI should switch from one concept to another during the generation process. By using a colon to delineate the transition, this method allows for precise control over the timing and integration of multiple themes within a single output.


In the context of the video, 'cinematic' refers to a style of visual representation that mimics the aesthetics of film, including dramatic lighting and storytelling elements. The speaker mentions using cinematic quality to enhance the visual depth and appeal of AI-generated images, emphasizing the model's ability to produce high-quality, film-like outputs.


The speaker contemplates starting a podcast as a medium to share AI news and insights, which may not necessarily be visual but are valuable for understanding trends and developments in the AI field. This indicates a broader approach to content delivery, aiming to reach the audience through varied formats beyond just video tutorials.


Scott Weller returns to his channel after a break to work with Stability, focusing on quality assurance.

Introduction to prompt craft or prompt engineering with automatic 1111, focusing on overlooked basics.

Exploring the concept of switching between different prompts during inference to create unique combinations.

Using braces and pipe characters to alternate between 'Airship' and 'Train' prompts in the generation process.

The 'from and to' method to control the steps at which a prompt switches from one theme to another.

An example of creating an 'Airship train' by starting with an Airship and switching to a Train after a set number of steps.

Technique to remove a prompt word like 'Airship' after a certain number of steps to evolve the generation.

Adding a prompt word like 'Airship' into the generation process partway through using a single colon.

The ability to adjust the scene dynamically by controlling when and how a prompt word is introduced or removed.

Combining complex concepts like a 'diesel Punk race car' with 'ice clouds' using decimal method for balance.

Using percentages or step numbers to determine the influence of each prompt word in the generation process.

The flexibility to add detailed elements like 'cinematic dramatic lighting' to a prompt and then have them fade away.

Scott's consideration of starting a podcast to share AI news and insights that are better suited to audio format.

The importance of revisiting and mastering the basics of prompt engineering to make fantastic work.

An invitation for viewers to share their thoughts on the video and whether they found the tips and tricks useful.

Scott's commitment to getting back on track with video production and addressing the overwhelming flow of information.

A promise of more advanced videos on prompt engineering coming out soon, with Scott speaking at an event in North Carolina.