#
Depth-to-Image Generation

##
StableDiffusionDepth2ImgPipeline

The depth-guided stable diffusion model was created by the researchers and engineers from CompVis, Stability AI, and LAION, as part of Stable Diffusion 2.0. It uses MiDas to infer depth based on an image.

StableDiffusionDepth2ImgPipeline lets you pass a text prompt and an initial image to condition the generation of new images as well as a `depth_map` to preserve the images’ structure.

The original codebase can be found here:

- Stable Diffusion v2: Stability-AI/stablediffusion

Available Checkpoints are:

- stable-diffusion-2-depth: stabilityai/stable-diffusion-2-depth

###
class diffusers.StableDiffusionDepth2ImgPipeline

< source >

( vae: AutoencoderKL text_encoder: CLIPTextModel tokenizer: CLIPTokenizer unet: UNet2DConditionModel scheduler: KarrasDiffusionSchedulers depth_estimator: DPTForDepthEstimation feature_extractor: DPTFeatureExtractor )

Parameters

- vae (AutoencoderKL) — Variational Auto-Encoder (VAE) Model to encode and decode images to and from latent representations.
- text_encoder (`CLIPTextModel`) — Frozen text-encoder. Stable Diffusion uses the text portion of CLIP, specifically the clip-vit-large-patch14 variant.
- tokenizer (`CLIPTokenizer`) — Tokenizer of class CLIPTokenizer.
- unet (UNet2DConditionModel) — Conditional U-Net architecture to denoise the encoded image latents.
- scheduler (SchedulerMixin) — A scheduler to be used in combination with `unet` to denoise the encoded image latents. Can be one of DDIMScheduler, LMSDiscreteScheduler, or PNDMScheduler.

Pipeline for text-guided image to image generation using Stable Diffusion.

This model inherits from DiffusionPipeline. Check the superclass documentation for the generic methods the library implements for all the pipelines (such as downloading or saving, running on a particular device, etc.)

In addition the pipeline inherits the following loading methods:

- Textual-Inversion: loaders.TextualInversionLoaderMixin.load_textual_inversion()
- LoRA: loaders.LoraLoaderMixin.load_lora_weights()

as well as the following saving methods:

- LoRA: loaders.LoraLoaderMixin.save_lora_weights()

####
__call__

< source >

( prompt: typing.Union[str, typing.List[str]] = None image: typing.Union[torch.FloatTensor, PIL.Image.Image, numpy.ndarray, typing.List[torch.FloatTensor], typing.List[PIL.Image.Image], typing.List[numpy.ndarray]] = None depth_map: typing.Optional[torch.FloatTensor] = None strength: float = 0.8 num_inference_steps: typing.Optional[int] = 50 guidance_scale: typing.Optional[float] = 7.5 negative_prompt: typing.Union[str, typing.List[str], NoneType] = None num_images_per_prompt: typing.Optional[int] = 1 eta: typing.Optional[float] = 0.0 generator: typing.Union[torch._C.Generator, typing.List[torch._C.Generator], NoneType] = None prompt_embeds: typing.Optional[torch.FloatTensor] = None negative_prompt_embeds: typing.Optional[torch.FloatTensor] = None output_type: typing.Optional[str] = 'pil' return_dict: bool = True callback: typing.Union[typing.Callable[[int, int, torch.FloatTensor], NoneType], NoneType] = None callback_steps: int = 1 cross_attention_kwargs: typing.Union[typing.Dict[str, typing.Any], NoneType] = None ) → StableDiffusionPipelineOutput or `tuple`

Parameters

- prompt (`str` or `List[str]`, optional) — The prompt or prompts to guide the image generation. If not defined, one has to pass `prompt_embeds`. instead.
- image (`torch.FloatTensor`, `PIL.Image.Image`, `np.ndarray`, `List[torch.FloatTensor]`, `List[PIL.Image.Image]`, or `List[np.ndarray]`) — `Image`, or tensor representing an image batch, that will be used as the starting point for the process. Can accept image latents as `image` only if `depth_map` is not `None`.
- depth_map (`torch.FloatTensor`, optional) — depth prediction that will be used as additional conditioning for the image generation process. If not defined, it will automatically predicts the depth via `self.depth_estimator`.
- strength (`float`, optional, defaults to 0.8) — Conceptually, indicates how much to transform the reference `image`. Must be between 0 and 1. `image` will be used as a starting point, adding more noise to it the larger the `strength`. The number of denoising steps depends on the amount of noise initially added. When `strength` is 1, added noise will be maximum and the denoising process will run for the full number of iterations specified in `num_inference_steps`. A value of 1, therefore, essentially ignores `image`.
- num_inference_steps (`int`, optional, defaults to 50) — The number of denoising steps. More denoising steps usually lead to a higher quality image at the expense of slower inference. This parameter will be modulated by `strength`.
- guidance_scale (`float`, optional, defaults to 7.5) — Guidance scale as defined in Classifier-Free Diffusion Guidance. `guidance_scale` is defined as `w` of equation 2. of Imagen Paper. Guidance scale is enabled by setting `guidance_scale > 1`. Higher guidance scale encourages to generate images that are closely linked to the text `prompt`, usually at the expense of lower image quality.
- negative_prompt (`str` or `List[str]`, optional) — The prompt or prompts not to guide the image generation. If not defined, one has to pass `negative_prompt_embeds`. instead. Ignored when not using guidance (i.e., ignored if `guidance_scale` is less than `1`).
- num_images_per_prompt (`int`, optional, defaults to 1) — The number of images to generate per prompt.
- eta (`float`, optional, defaults to 0.0) — Corresponds to parameter eta (η) in the DDIM paper: https://arxiv.org/abs/2010.02502. Only applies to schedulers.DDIMScheduler, will be ignored for others.
- generator (`torch.Generator`, optional) — One or a list of torch generator(s) to make generation deterministic.
- prompt_embeds (`torch.FloatTensor`, optional) — Pre-generated text embeddings. Can be used to easily tweak text inputs, e.g. prompt weighting. If not provided, text embeddings will be generated from `prompt` input argument.
- negative_prompt_embeds (`torch.FloatTensor`, optional) — Pre-generated negative text embeddings. Can be used to easily tweak text inputs, e.g. prompt weighting. If not provided, negative_prompt_embeds will be generated from `negative_prompt` input argument.
- output_type (`str`, optional, defaults to `"pil"`) — The output format of the generate image. Choose between PIL: `PIL.Image.Image` or `np.array`.
- return_dict (`bool`, optional, defaults to `True`) — Whether or not to return a StableDiffusionPipelineOutput instead of a plain tuple.
- callback (`Callable`, optional) — A function that will be called every `callback_steps` steps during inference. The function will be called with the following arguments: `callback(step: int, timestep: int, latents: torch.FloatTensor)`.
- callback_steps (`int`, optional, defaults to 1) — The frequency at which the `callback` function will be called. If not specified, the callback will be called at every step.
- cross_attention_kwargs (`dict`, optional) — A kwargs dictionary that if specified is passed along to the `AttentionProcessor` as defined under `self.processor` in diffusers.cross_attention.

Returns

StableDiffusionPipelineOutput or `tuple`

StableDiffusionPipelineOutput if `return_dict` is True, otherwise a `tuple. When returning a tuple, the first element is a list with the generated images, and the second element is a list of` bool`s denoting whether the corresponding generated image likely represents "not-safe-for-work" (nsfw) content, according to the` safety_checker`.

Function invoked when calling the pipeline for generation.

Examples:

    >>> import torch
    >>> import requests
    >>> from PIL import Image

    >>> from diffusers import StableDiffusionDepth2ImgPipeline

    >>> pipe = StableDiffusionDepth2ImgPipeline.from_pretrained(
    ...     "stabilityai/stable-diffusion-2-depth",
    ...     torch_dtype=torch.float16,
    ... )
    >>> pipe.to("cuda")

    >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg"
    >>> init_image = Image.open(requests.get(url, stream=True).raw)
    >>> prompt = "two tigers"
    >>> n_propmt = "bad, deformed, ugly, bad anotomy"
    >>> image = pipe(prompt=prompt, image=init_image, negative_prompt=n_propmt, strength=0.7).images[0]

####
enable_attention_slicing

< source >

( slice_size: typing.Union[str, int, NoneType] = 'auto' )

Parameters

- slice_size (`str` or `int`, optional, defaults to `"auto"`) — When `"auto"`, halves the input to the attention heads, so attention will be computed in two steps. If `"max"`, maximum amount of memory will be saved by running only one slice at a time. If a number is provided, uses as many slices as `attention_head_dim // slice_size`. In this case, `attention_head_dim` must be a multiple of `slice_size`.

Enable sliced attention computation.

When this option is enabled, the attention module will split the input tensor in slices, to compute attention in several steps. This is useful to save some memory in exchange for a small speed decrease.

####
disable_attention_slicing

< source >

( )

Disable sliced attention computation. If `enable_attention_slicing` was previously invoked, this method will go back to computing attention in one step.

####
enable_xformers_memory_efficient_attention

< source >

( attention_op: typing.Optional[typing.Callable] = None )

Parameters

- attention_op (`Callable`, optional) — Override the default `None` operator for use as `op` argument to the `memory_efficient_attention()` function of xFormers.

Enable memory efficient attention as implemented in xformers.

When this option is enabled, you should observe lower GPU memory usage and a potential speed up at inference time. Speed up at training time is not guaranteed.

Warning: When Memory Efficient Attention and Sliced attention are both enabled, the Memory Efficient Attention is used.

Examples:

    >>> import torch
    >>> from diffusers import DiffusionPipeline
    >>> from xformers.ops import MemoryEfficientAttentionFlashAttentionOp

    >>> pipe = DiffusionPipeline.from_pretrained("stabilityai/stable-diffusion-2-1", torch_dtype=torch.float16)
    >>> pipe = pipe.to("cuda")
    >>> pipe.enable_xformers_memory_efficient_attention(attention_op=MemoryEfficientAttentionFlashAttentionOp)
    >>> # Workaround for not accepting attention shape using VAE for Flash Attention
    >>> pipe.vae.enable_xformers_memory_efficient_attention(attention_op=None)

####
disable_xformers_memory_efficient_attention

< source >

( )

Disable memory efficient attention as implemented in xformers.

####
load_textual_inversion

< source >

( pretrained_model_name_or_path: typing.Union[str, typing.List[str], typing.Dict[str, torch.Tensor], typing.List[typing.Dict[str, torch.Tensor]]] token: typing.Union[str, typing.List[str], NoneType] = None **kwargs )

Parameters

- pretrained_model_name_or_path (`str` or `os.PathLike` or `List[str or os.PathLike]` or `Dict` or `List[Dict]`) — Can be either:
  - A string, the model id of a pretrained model hosted inside a model repo on huggingface.co. Valid model ids should have an organization name, like `"sd-concepts-library/low-poly-hd-logos-icons"`.
  - A path to a directory containing textual inversion weights, e.g. `./my_text_inversion_directory/`.
  - A path to a file containing textual inversion weights, e.g. `./my_text_inversions.pt`.
  - A torch state dict.

  Or a list of those elements.

- token (`str` or `List[str]`, optional) — Override the token to use for the textual inversion weights. If `pretrained_model_name_or_path` is a list, then `token` must also be a list of equal length.
- weight_name (`str`, optional) — Name of a custom weight file. This should be used in two cases:
  - The saved textual inversion file is in `diffusers` format, but was saved under a specific weight name, such as `text_inv.bin`.
  - The saved textual inversion file is in the “Automatic1111” form.
- cache_dir (`Union[str, os.PathLike]`, optional) — Path to a directory in which a downloaded pretrained model configuration should be cached if the standard cache should not be used.
- force_download (`bool`, optional, defaults to `False`) — Whether or not to force the (re-)download of the model weights and configuration files, overriding the cached versions if they exist.
- resume_download (`bool`, optional, defaults to `False`) — Whether or not to delete incompletely received files. Will attempt to resume the download if such a file exists.
- proxies (`Dict[str, str]`, optional) — A dictionary of proxy servers to use by protocol or endpoint, e.g., `{'http': 'foo.bar:3128', 'http://hostname': 'foo.bar:4012'}`. The proxies are used on each request.
- local_files_only(`bool`, optional, defaults to `False`) — Whether or not to only look at local files (i.e., do not try to download the model).
- use_auth_token (`str` or bool, optional) — The token to use as HTTP bearer authorization for remote files. If `True`, will use the token generated when running `diffusers-cli login` (stored in `~/.huggingface`).
- revision (`str`, optional, defaults to `"main"`) — The specific model version to use. It can be a branch name, a tag name, or a commit id, since we use a git-based system for storing models and other artifacts on huggingface.co, so `revision` can be any identifier allowed by git.
- subfolder (`str`, optional, defaults to `""`) — In case the relevant files are located inside a subfolder of the model repo (either remote in huggingface.co or downloaded locally), you can specify the folder name here.
- mirror (`str`, optional) — Mirror source to accelerate downloads in China. If you are from China and have an accessibility problem, you can set this option to resolve it. Note that we do not guarantee the timeliness or safety. Please refer to the mirror site for more information.

Load textual inversion embeddings into the text encoder of stable diffusion pipelines. Both `diffusers` and `Automatic1111` formats are supported (see example below).

This function is experimental and might change in the future.

It is required to be logged in (`huggingface-cli login`) when you want to use private or gated models.

Example:

To load a textual inversion embedding vector in `diffusers` format:

    from diffusers import StableDiffusionPipeline
    import torch

    model_id = "runwayml/stable-diffusion-v1-5"
    pipe = StableDiffusionPipeline.from_pretrained(model_id, torch_dtype=torch.float16).to("cuda")

    pipe.load_textual_inversion("sd-concepts-library/cat-toy")

    prompt = "A <cat-toy> backpack"

    image = pipe(prompt, num_inference_steps=50).images[0]
    image.save("cat-backpack.png")

To load a textual inversion embedding vector in Automatic1111 format, make sure to first download the vector,

e.g. from civitAI and then load the vector locally:

    from diffusers import StableDiffusionPipeline
    import torch

    model_id = "runwayml/stable-diffusion-v1-5"
    pipe = StableDiffusionPipeline.from_pretrained(model_id, torch_dtype=torch.float16).to("cuda")

    pipe.load_textual_inversion("./charturnerv2.pt", token="charturnerv2")

    prompt = "charturnerv2, multiple views of the same character in the same outfit, a character turnaround of a woman wearing a black jacket and red shirt, best quality, intricate details."

    image = pipe(prompt, num_inference_steps=50).images[0]
    image.save("character.png")

####
load_lora_weights

< source >

( pretrained_model_name_or_path_or_dict: typing.Union[str, typing.Dict[str, torch.Tensor]] **kwargs )

Parameters

- pretrained_model_name_or_path_or_dict (`str` or `os.PathLike` or `dict`) — Can be either:
  - A string, the model id of a pretrained model hosted inside a model repo on huggingface.co. Valid model ids should have an organization name, like `google/ddpm-celebahq-256`.
  - A path to a directory containing model weights saved using `~ModelMixin.save_config`, e.g., `./my_model_directory/`.
  - A torch state dict.
- cache_dir (`Union[str, os.PathLike]`, optional) — Path to a directory in which a downloaded pretrained model configuration should be cached if the standard cache should not be used.
- force_download (`bool`, optional, defaults to `False`) — Whether or not to force the (re-)download of the model weights and configuration files, overriding the cached versions if they exist.
- resume_download (`bool`, optional, defaults to `False`) — Whether or not to delete incompletely received files. Will attempt to resume the download if such a file exists.
- proxies (`Dict[str, str]`, optional) — A dictionary of proxy servers to use by protocol or endpoint, e.g., `{'http': 'foo.bar:3128', 'http://hostname': 'foo.bar:4012'}`. The proxies are used on each request.
- local_files_only(`bool`, optional, defaults to `False`) — Whether or not to only look at local files (i.e., do not try to download the model).
- use_auth_token (`str` or bool, optional) — The token to use as HTTP bearer authorization for remote files. If `True`, will use the token generated when running `diffusers-cli login` (stored in `~/.huggingface`).
- revision (`str`, optional, defaults to `"main"`) — The specific model version to use. It can be a branch name, a tag name, or a commit id, since we use a git-based system for storing models and other artifacts on huggingface.co, so `revision` can be any identifier allowed by git.
- subfolder (`str`, optional, defaults to `""`) — In case the relevant files are located inside a subfolder of the model repo (either remote in huggingface.co or downloaded locally), you can specify the folder name here.
- mirror (`str`, optional) — Mirror source to accelerate downloads in China. If you are from China and have an accessibility problem, you can set this option to resolve it. Note that we do not guarantee the timeliness or safety. Please refer to the mirror site for more information.

Load pretrained attention processor layers (such as LoRA) into UNet2DConditionModel and `CLIPTextModel`).

We support loading A1111 formatted LoRA checkpoints in a limited capacity.

This function is experimental and might change in the future.

It is required to be logged in (`huggingface-cli login`) when you want to use private or gated models.

####
save_lora_weights

< source >

( save_directory: typing.Union[str, os.PathLike] unet_lora_layers: typing.Dict[str, typing.Union[torch.nn.modules.module.Module, torch.Tensor]] = None text_encoder_lora_layers: typing.Dict[str, torch.nn.modules.module.Module] = None is_main_process: bool = True weight_name: str = None save_function: typing.Callable = None safe_serialization: bool = False )

Parameters

- save_directory (`str` or `os.PathLike`) — Directory to which to save. Will be created if it doesn’t exist.
- unet_lora_layers (`Dict[str, torch.nn.Module]` or `Dict[str, torch.Tensor]`) — State dict of the LoRA layers corresponding to the UNet. Specifying this helps to make the serialization process easier and cleaner. Values can be both LoRA torch.nn.Modules layers or torch weights.
- text_encoder_lora_layers (`Dict[str, torch.nn.Module] or` Dict[str, torch.Tensor]`) -- State dict of the LoRA layers corresponding to the` text_encoder`. Since the` text_encoder`comes from`transformers`, we cannot rejig it. That is why we have to explicitly pass the text encoder LoRA state dict. Values can be both LoRA torch.nn.Modules layers or torch weights.
- is_main_process (`bool`, optional, defaults to `True`) — Whether the process calling this is the main process or not. Useful when in distributed training like TPUs and need to call this function on all processes. In this case, set `is_main_process=True` only on the main process to avoid race conditions.
- save_function (`Callable`) — The function to use to save the state dictionary. Useful on distributed training like TPUs when one need to replace `torch.save` by another method. Can be configured with the environment variable `DIFFUSERS_SAVE_MODE`.

Save the LoRA parameters corresponding to the UNet and the text encoder.

####
enable_sequential_cpu_offload

< source >

( gpu_id = 0 )

Offloads all models to CPU using accelerate, significantly reducing memory usage. When called, unet, text_encoder, vae and safety checker have their state dicts saved to CPU and then are moved to a `torch.device('meta') and loaded to GPU only when their specific submodule has its` forward` method called.