(tiΝ vddlmZddlmZddlZddlmZddlmZddl m Z m Z ddl m Z dd lmZdd lmZmZmZd d lmZd d lmZmZmZmZd dlmZmZmZmZm Z d dl!m"Z"d dl#m$Z$m%Z%m&Z&d dl'm(Z(ejRe*Z+eGddeZ,GddejZZ.Gdde"ee ee Z/dZ0y)) dataclass)AnyN)nn) functional) ConfigMixinregister_to_config)PeftAdapterMixin)FromOriginalModelMixin) BaseOutputapply_lora_scalelogging)AttentionMixin)ADDED_KV_ATTENTION_PROCESSORSCROSS_ATTENTION_PROCESSORSAttnAddedKVProcessor AttnProcessor)TextImageProjectionTextImageTimeEmbeddingTextTimeEmbeddingTimestepEmbedding Timesteps) ModelMixin)UNetMidBlock2DUNetMidBlock2DCrossAttnget_down_block)UNet2DConditionModelcTeZdZUdZeej ed<ej ed<y)ControlNetOutputa The output of [`ControlNetModel`]. Args: down_block_res_samples (`tuple[torch.Tensor]`): A tuple of downsample activations at different resolutions for each downsampling block. Each tensor should be of shape `(batch_size, channel * resolution, height //resolution, width // resolution)`. Output can be used to condition the original UNet's downsampling activations. mid_down_block_re_sample (`torch.Tensor`): The activation of the middle block (the lowest sample resolution). Each tensor should be of shape `(batch_size, channel * lowest_resolution, height // lowest_resolution, width // lowest_resolution)`. Output can be used to condition the original UNet's middle block activation. down_block_res_samplesmid_block_res_sampleN)__name__ __module__ __qualname____doc__tupletorchTensor__annotations__v/Volumes/fast/ai/experiments/MLX_z-image/.venv/lib/python3.12/site-packages/diffusers/models/controlnets/controlnet.pyr r -s" "%,,//,,&r,r c FeZdZdZ ddededeedfffd ZdZxZS) ControlNetConditioningEmbeddingu Quoting from https://huggingface.co/papers/2302.05543: "Stable Diffusion uses a pre-processing method similar to VQ-GAN [11] to convert the entire dataset of 512 × 512 images into smaller 64 × 64 “latent images” for stabilized training. This requires ControlNets to convert image-based conditions to 64 × 64 feature space to match the convolution size. We use a tiny network E(·) of four convolution layers with 4 × 4 kernels and 2 × 2 strides (activated by ReLU, channels are 16, 32, 64, 128, initialized with Gaussian weights, trained jointly with the full model) to encode image-space conditions ... into feature maps ..." conditioning_embedding_channelsconditioning_channelsblock_out_channels.c t|tj||ddd|_tj g|_tt|dz D]t}||}||dz}|j jtj||dd|j jtj||dddvttj|d|dd|_ y)Nrr kernel_sizepaddingr)r6r7stride) super__init__rConv2dconv_in ModuleListblocksrangelenappend zero_moduleconv_out)selfr0r1r2i channel_in channel_out __class__s r-r;z(ControlNetConditioningEmbedding.__init__Ks yy!68J18M[\fgh mmB' s-.23A+A.J,QU3K KK  ryyZQXYZ [ KK  ryy[aYZcde f 4 $ II(,.M[\fg h  r,c|j|}tj|}|jD]}||}tj|}!|j |}|SN)r=Fsilur?rD)rE conditioning embeddingblocks r-forwardz'ControlNetConditioningEmbedding.forwardasZLL. FF9% [[Ei(Iy)I!MM), r,)r `) r#r$r%r&intr'r;rQ __classcell__rIs@r-r/r/As>&'.?  ),  # "#s(O  , r,r/cB|eZdZdZdZe d:dededededee d fd e dzd eeezd eed fd edede de dedzde dedeeed fzdedzde dzdeeed fzdeeed fzdzdede dzde dzdedzdedzdede d edzd!e d"eed fdzd#ed$ef@fd% Z e d;d&e d!e d"eed fdzd'edef d(Zd)Zd*e ezeezd+dfd,Zed- djC|'tE|D]\}(})|&}*||(}&|(t|dz k(}+tG|)fid"| d#||(d$|*d%|&d&|#d'|+ d(|d)| d*| d+|d,||(d-||(||(n|&d.| d/|d0||(d1|d2|},|j<jC|,tI| D]@}-tj|&|&d!}'tA|'}'|j>jC|'B|+rtj|&|&d!}'tA|'}'|j>jC|'|d3}.tj|.|.d!}'tA|'}'|'|_%|d4k(r#tM|d3|.|#|| | |||d3| ||5 |_'y|d6k(rtQ|d3|#d || | | |d78 |_'ytd9|):NzbMust provide the same number of `block_out_channels` as `down_block_types`. `block_out_channels`: z. `down_block_types`: .zfMust provide the same number of `only_cross_attention` as `down_block_types`. `only_cross_attention`: zdMust provide the same number of `num_attention_heads` as `down_block_types`. `num_attention_heads`: rr4rrr5)re text_proj)rkzMencoder_hid_dim_type defaults to 'text_proj' as `encoder_hid_dim` is defined.zJ`encoder_hid_dim` has to be defined when `encoder_hid_dim_type` is set to text_image_proj)text_embed_dimimage_embed_dimrhzencoder_hid_dim_type: z0 must be None, 'text_proj' or 'text_image_proj'.timestepidentity projectionzX`class_embed_type`: 'projection' requires `projection_class_embeddings_input_dim` be settext) num_heads text_image)rrtime_embed_dim text_timezaddition_embed_type: z& must be None, 'text' or 'text_image'.)r0r2r1)r6 num_layersrir\ out_channels temb_channelsadd_downsample resnet_eps resnet_act_fn resnet_groupsrhrmrlrcrnrarsrtr9r) rir\rrroutput_scale_factorrtrhrmrrnrsrF) r\rrrrrrrt add_attentionzunknown mid_block_type : ))r:r;rA ValueError isinstanceboolrWrr<r=r time_projrtime_embeddingr loggerinfoLinearencoder_hid_projr Embeddingclass_embeddingIdentityr add_embeddingr add_time_projr/controlnet_cond_embeddingr> down_blockscontrolnet_down_blocksrCrB enumeraterr@controlnet_mid_blockr mid_blockr)0rEr\r1r]r^r_r`rar2rbrcrdrerfrgrhrirjrkrlrmrnrorprqrrrsrtrurvrwrxryconv_in_kernelconv_in_paddingrtimestep_input_dimtext_time_embedding_from_dimoutput_channelcontrolnet_blockrFdown_block_type input_channelis_final_block down_block_mid_block_channelrIs0 r-r;zControlNetModel.__init__sR 2G5G ! "c*:&; ;tvHuII_`p_qqrs .5#>R:SWZ[kWl:lxzNyOOefvewwxy -s3>no %)D !  #(8(D#%<<0@.#QD  +#45G#XD  +#%;;~~#ND  -4< n$55Z\j#kD #'D & (*/>,/B,!2,nHe"D !L 0"82DWhv"D !K /!*+BOU_!`D !23XZh!iD  ,45H4IIopq q*I,>q,AB"7* & ==,&(mmB&7# *D 1$8#9C@PL/r299%68IWXY&'78$4! 6 64-I"-M-,#$$:(?$7$7$;-&;!1 DN/ /+.r2,#$$:-(?# DN88HIJ Jr,unetload_weights_from_unetcd|jvr|jjnd}d|jvr|jjnd}d|jvr|jjnd}d|jvr|jjnd} d|jvr|jj nd} |d"id|d|d| d| d|d|jj d |jjd |jjd |jjd |jjd |jjd|jjd|jjd|jjd|jjd|jj d|jj"d|jj$d|jj&d|jj(d|jj*d|jj,d|jj.d|jj0d|jj2d|jj4d|jj6d|d|d |} |r}| j8j;|j8j=| j>j;|j>j=| j@j;|j@j=| jBr3| jBj;|jBj=tE| d!r3| jFj;|jFj=| jHj;|jHj=| jJj;|jJj=| S)#a) Instantiate a [`ControlNetModel`] from [`UNet2DConditionModel`]. Parameters: unet (`UNet2DConditionModel`): The UNet model weights to copy to the [`ControlNetModel`]. All configuration options are also copied where applicable. rir4rjNrkrprqr\r]r^r_rar2rbrcrdrerfrgrhrlrmrnrorrrsrtrur`rvrwr1rr+)&configrirjrkrprqr\r]r^r_rar2rbrcrdrerfrgrhrlrmrnrorrrsrtrur`r=load_state_dict state_dictrrrhasattrrrr) clsrrvrwrr1rirjrkrprq controlnets r- from_unetzControlNetModel.from_unets$9WZ^ZeZe8eDKK 4 4kl %:Kdkk9Y$++55_cCY]a]h]hCht{{??nrAVZ^ZeZeAedkk==ko3LPTP[P[3[DKK / /ae  + !5 !4 %<  *F   //  !KK77 {{-- "[[99 "&!A!A  ${{== "[[99  ${{== $(;;#E#E ;;%% !KK77! "[[))# $!% ? ?% & ${{==' (!% ? ?) *#'++"C"C+ ,"[[99- ."[[99/ 0"[[991 2%)KK$G$G3 437++2c2c5 6 ;;557 83X9 :1T; <#8= B "    . .t||/F/F/H I  0 01J1J1L M  % % 5 5d6I6I6T6T6V W))**::4;O;O;Z;Z;\]z?3((889K9K9V9V9XY  " " 2 243C3C3N3N3P Q  0 01J1J1L Mr,c jtd|jjDr t}nmtd|jjDr t }n8t dt t|jj|j|y)ze Disables custom attention processors and sets the default attention implementation. c3@K|]}|jtvywrK)rIr.0procs r- z=ControlNetModel.set_default_attn_processor.. siKh4t~~!>>Khc3@K|]}|jtvywrK)rIrrs r-rz=ControlNetModel.set_default_attn_processor..shJg$#==JgrzOCannot call `set_default_attn_processor` when attention processors are of type N) allattn_processorsvaluesrrrnextiterset_attn_processor)rE processors r-set_default_attn_processorz*ControlNetModel.set_default_attn_processors i4K_K_KfKfKhi i,.I h$J^J^JeJeJgh h%IabfgklpmAmAmHmHmJhKcLbMN   *r, slice_sizereturnc  g dtjjf fd |jD] }| t }|dk(r Dcgc]}|dz }}n |dk(r|dgz}t |t s||gzn|}t |t k7r=tdt |d|jd t d t d tt |D]&}||} |}|||kDstd |d|d dtjjdt tf fd t t|}|jD] } || y cc}w)a Enable sliced attention computation. When this option is enabled, the attention module splits the input tensor in slices to compute attention in several steps. This is useful for saving some memory in exchange for a small decrease in speed. Args: slice_size (`str` or `int` or `list(int)`, *optional*, defaults to `"auto"`): When `"auto"`, input to the attention heads is halved, so attention is computed in two steps. If `"max"`, maximum amount of memory is saved by running only one slice at a time. If a number is provided, uses as many slices as `attention_head_dim // slice_size`. In this case, `attention_head_dim` must be a multiple of `slice_size`. modulect|drj|j|jD] }| yNset_attention_slice)rrBsliceable_head_dimchildren)rchild$fn_recursive_retrieve_sliceable_dimssliceable_head_dimss r-rzQControlNetModel.set_attention_slice..fn_recursive_retrieve_sliceable_dims(s;v45#**6+D+DE*4U;+r,autormaxr4zYou have provided z, but z has zH different attention layers. Make sure to match `len(slice_size)` to be r{Nzsize z has to be smaller or equal to rct|dr|j|j|jD] }|| yr)rrpopr)rrr fn_recursive_set_attention_slices r-rzMControlNetModel.set_attention_slice..fn_recursive_set_attention_sliceNs>v45**:>>+;<*0 C+r,) r(rModulerrArlistrrr@rWreversed) rErrnum_sliceable_layersdimrFsizereversed_slice_sizerrrs @@@r-rz#ControlNetModel.set_attention_slices! < <mmoF 0 8& ##67  /BB.As#(.AJB 5 -3J@J:W[@\)ZL8bl z?c"56 6$S_$5VDKK=cReNfMghQQTUhQiPjjkm  s:'Aa=D%a(CD3J 5.McURS!TUU ( DUXX__ DRVWZR[ D#8J#78mmoF ,V5H I&=Cs E;cross_attention_kwargssamplerencoder_hidden_statescontrolnet_condconditioning_scale class_labels timestep_condattention_maskadded_cond_kwargs guess_mode return_dictc P |jj} | dk(rn,| dk(rtj|dg}nt d| |2d|j |j z dz}|jd}|}tj|s|jjdk(}|jjd k(}t|tr%|s|rtjntj}n$|s|rtjntj }tj"|g||j }n6t%|j&d k(r|dj |j}|j)|j&d }|j+|}|j |j }|j-||}d}|j.g| t d |jj0dk(r|j+|}|j/|j |j }||z}|jj2"|jj2dk(r|j5|}n|jj2dk(rd| vrt |j6d| j9d}d| vrt |j6d| j9d}|j;|j=}|j?|j&d df}tj@||gd}|j |j }|j5|}|||zn|}|jC|}|jE|}||z}|f}|jFD]=}tI|dr|jJr|||||| \}}n |||\}}||z }?|jLUtI|jLdr-|jLjJr|jM||||| }n|jM||}d}tO||jPD]\}}||}||fz}|}|jS|} | rt|jjTs^tjVdd t%|dz|j}!|!|z}!tO||!D"cgc] \}}"||"z }}}"| |!dz} n|Dcgc]}||z }}| |z} |jjTr=|Dcgc]}tjX|dd}}tjX| dd} | s|| fSt[|| Scc}"}wcc}wcc}w)!a# The [`ControlNetModel`] forward method. Args: sample (`torch.Tensor`): The noisy input tensor. timestep (`torch.Tensor | float | int`): The number of timesteps to denoise an input. encoder_hidden_states (`torch.Tensor`): The encoder hidden states. controlnet_cond (`torch.Tensor`): The conditional input tensor of shape `(batch_size, sequence_length, hidden_size)`. conditioning_scale (`float`, defaults to `1.0`): The scale factor for ControlNet outputs. class_labels (`torch.Tensor`, *optional*, defaults to `None`): Optional class labels for conditioning. Their embeddings will be summed with the timestep embeddings. timestep_cond (`torch.Tensor`, *optional*, defaults to `None`): Additional conditional embeddings for timestep. If provided, the embeddings will be summed with the timestep_embedding passed through the `self.time_embedding` layer to obtain the final timestep embeddings. attention_mask (`torch.Tensor`, *optional*, defaults to `None`): An attention mask of shape `(batch, key_tokens)` is applied to `encoder_hidden_states`. If `1` the mask is kept, otherwise if `0` it is discarded. Mask will be converted into a bias, which adds large negative values to the attention scores corresponding to "discard" tokens. added_cond_kwargs (`dict`): Additional conditions for the Stable Diffusion XL UNet. cross_attention_kwargs (`dict[str]`, *optional*, defaults to `None`): A kwargs dictionary that if specified is passed along to the `AttnProcessor`. guess_mode (`bool`, defaults to `False`): In this mode, the ControlNet encoder tries its best to recognize the input content of the input even if you remove all prompts. A `guidance_scale` between 3.0 and 5.0 is recommended. return_dict (`bool`, defaults to `True`): Whether or not to return a [`~models.controlnets.controlnet.ControlNetOutput`] instead of a plain tuple. Returns: [`~models.controlnets.controlnet.ControlNetOutput`] **or** `tuple`: If `return_dict` is `True`, a [`~models.controlnets.controlnet.ControlNetOutput`] is returned, otherwise a tuple is returned where the first element is the sample tensor. rgbbgrr4)dimsz1unknown `controlnet_conditioning_channel_order`: Ngmpsnpu)dtypedevicer)rz9class_labels should be provided when num_class_embeds > 0rrr text_embedsz has the config param `addition_embed_type` set to 'text_time' which requires the keyword argument `text_embeds` to be passed in `added_cond_kwargs`time_idsz has the config param `addition_embed_type` set to 'text_time' which requires the keyword argument `time_ids` to be passed in `added_cond_kwargs`r9)rhas_cross_attention) hidden_statestembrrr)rr)rrrr+)r)rrT)rkeepdim)r!r").rrvr(fliprtor unsqueeze is_tensorrtyperfloatfloat32float64int32int64tensorrAshapeexpandrrrrorprrIgetrflattenreshapeconcatr=rrrrrziprrrxlogspacemeanr )#rErrrrrrrrrrrr channel_order timestepsis_mpsis_npurt_embembaug_emb class_embrr time_embeds add_embedsr!downsample_block res_samples!controlnet_down_block_res_samplesdown_block_res_samplerr"scalesscales# r-rQzControlNetModel.forwardYsr II E !  e ##jjsCOPQ^P_`a a  %."3"3FLL"AAXMN+55a8N y)]]''50F]]''50F(E**0F (.& u{{ i[fmmTI  !Q &!$**6==9I$$V\\!_5 y) v||,!!%7    +# !\]]{{++z9#~~l; ,,\:==DJJ=OI /C ;; * * 6{{..&8,,-BC00K? (99$>>*+@033MB %66$>>*+|}-00<"001A1A1CD )11;3D3DQ3G2LM "\\; *D"M ']]3995 ,,Z8&2cGmf%88I/)#) $ 0 0 ')>?DTDhDh&6"(*?#1+A '# '7VRU&V# "k 1 "!1 >> %t~~'<=$..BdBd*?#1+A (4-/)7:;QSWSnSn7o 3 !#3$45J$K !0QUjTl0l -8p"C#88@ dkk@@^^B3/E+F+JSYS`S`aF00FJMNdflJm%nJmfunJm "%n#7&*#D Pf%gPfff/A&APf "%g#7:L#L ;; - -Ka&Ka 6vtr#r4NNNFNNNNFdefaultNrrRF@)rrRTr)g?NNNNNFT)r#r$r%r& _supports_gradient_checkpointingr rWrr'strrr; classmethodrrrrrr r(r)dictrr rQrXrYs@r-r[r[ns[BH(,$%& $- &?38.D !"#()&(#'>?&*+/45<@&+'+*..2'+!&'0<@5:FW',-/MCKCK #CK CK  CK  S/ CKd CK#U4[0CK"#s(OCKCK  !CK"!&#CK$%CK&t'CK()CK*!+CK,'*E#s(O&;-CK.t/CK0"Dj1CK2 %S/13CK4!5c?2T95CK6 $7CK8*9CK:!4Z;CK<"%t=CK>*?CK@ACKB"%CCKD03TzECKF03GCKH.338_t-CICKJ!%KCKL(+MCKCKJ6;FW'+%& I"I03I.338_t-C I !% I # IIX + ?JcCi$s).C?J?JB./%(,0-1.2<@8<  I  I ,,&,I  %|| I  I " I llT)I ||d*I  t+I  U\\ 12T9I !%S#X 5I I I  E% c(9":ELL"HI II 0I r,r[cn|jD]!}tjj|#|SrK) parametersrinitzeros_)rps r-rCrC&s*     q! Mr,)1 dataclassesrtypingrr(rtorch.nnrrLconfiguration_utilsrr loadersr loaders.single_file_modelr utilsr r r attentionrattention_processorrrrr embeddingsrrrrrmodeling_utilsrunets.unet_2d_blocksrrrunets.unet_2d_conditionr get_loggerr#rr rr/r[rCr+r,r-r>s" $B'?::& vu' ;   H % 'z' '&*bii*Zu  j.+?UWgu  pr,