Innovations & Integrations (Community of Practice)

Friday 12 January 2024

Automatic1111 and ControlNet

"AUTOMATIC1111/stable-diffusion-webui" is dedicated to a web-based user interface for the Stable Diffusion model. This UI facilitates the generation and manipulation of images using Stable Diffusion, a neural network model specializing in creating images from text descriptions. The UI includes features like text-to-image (txt2img) and image-to-image (img2img) transformations, catering to a broad range of AI-driven image generation and editing tasks. The repository contains code predominantly in Python and JavaScript, and is licensed under AGPL-3.0*.

For more detailed information, you can explore the repository directly on GitHub:

Installation Guide: 

Installation and Running
Alternatively, use online services (like Google Colab):
Installation on Windows 10/11 with NVidia-GPUs using release package
Automatic Installation on Windows
  • Install Python 3.10.6 (Newer version of Python does not support torch), checking "Add Python to PATH".
  • Install git.
  • Download the stable-diffusion-webui repository, for example by running git clone
  • Run webui-user.bat from Windows Explorer as normal, non-administrator, user.
ControlNet is a neural network that enhances text-to-image models like Stable Diffusion by adding precise control over image elements such as shapes, depth, and human poses, making image generation more versatile and accurate. 
This is very useful for quick architectural visualization. For example, based on provided sketch outlines, users can experiment with various moods and contextual settings. They can also transfer specific elements from one image to another, creating multiple iterations according to their prompts.

There’s a standalone Github repository for ControlNet (, maintained by user Illyasviel, and an Extension for popular SD interface Automatic1111 (, maintained by user Mikubill.

ControlNet Installation Guide: 

ControlNet is an extension for AUTOMATIC1111's Stable Diffusion web UI. It allows the Web UI to add ControlNet to the original Stable Diffusion model to generate images. This new addition will be incorporated without extra steps to combine them.

  • Open "Extensions" tab.
  • Open "Install from URL" tab in the tab.
  • Enter to "URL for extension's git repository".
  • Press "Install" button.
  • Wait for 5 seconds, and you will see the message "Installed into stable-diffusion-webui\extensions\sd-webui-controlnet. Use Installed tab to restart".
  • Go to "Installed" tab, click "Check for updates", and then click "Apply and restart UI". (The next time you can also use these buttons to update ControlNet.)
  • Completely restart A1111 webui including your terminal. (If you do not know what is a "terminal", you can reboot your computer to achieve the same effect.)
  • Download models (see below).
  • After you put models in the correct folder, you may need to refresh to see the models. The refresh button is right to your "Model" dropdown.
*AGPL-3.0 stands for the Affero General Public License version 3.0. It's a free, copyleft license for software and other kinds of works, ensuring that all modified versions of the licensed software are also free and open source. Specifically, it requires that the source code be made available to any network user of the AGPL-licensed work, typically used in web applications. This means if you modify an AGPL-licensed program and run it on a server accessible to others, you must also share the modified source code under AGPL.

Related Links and References:
After downloading a ControlNet model, save it in the stable-diffusion-webui\extensions\sd-webui-controlnet\models\controlnet directory for Automatic1111*, or in the ComfyUI\models\controlnet directory for ComfyUI**. Restart the interface afterward to ensure the model is recognized.

*Automatic1111 is user-friendly and straightforward, ideal for casual users or beginners who want to create images with Stable Diffusion without needing deep technical knowledge. 

**ComfyUI, with its node-based interface, offers more detailed control and customization, suited for advanced users or professionals who require precise control over image generation processes. ComfyUI has a steeper learning curve but provides greater flexibility for complex tasks.

Without downloading of models for explorations:

No comments:

Post a Comment