(ti_ddlZddlmZddlZddlmZddlmZmZddl m Z ddl m Z e jeZGdd e ZGd d e eZGd d ej$ZGddej$ZGddej$ZGddej$ZGddej$ZGddej$ZGddej$Zy)N)Callable) ConfigMixinregister_to_config)logging) ModelMixinc eZdZdZdedffd Zddejdeedzdeejfd Z dd e e jzd e d ed e de dzf dZede e jzdzfdZxZS) MultiAdaptera MultiAdapter is a wrapper model that contains multiple adapter models and merges their outputs according to user-assigned weighting. This model inherits from [`ModelMixin`]. Check the superclass documentation for common methods such as downloading or saving. Args: adapters (`list[T2IAdapter]`, *optional*, defaults to None): A list of `T2IAdapter` model instances. adapters T2IAdapterc$tt| t||_t j ||_t|dk(r tdt|dk(r td|dj}|dj}tdt|D]]}||j|k7s||j|k7s(td|d|d|d||jd|d ||j ||_ ||_ y) NrzExpecting at least one adapterrzQFor a single adapter, please use the `T2IAdapter` class instead of `MultiAdapter`zjExpecting all adapters to have the same downscaling behavior, but got: adapters[0].total_downscale_factor=z adapters[0].downscale_factor=z adapter[`z`].total_downscale_factor=z`].downscale_factor=) superr __init__len num_adapternn ModuleListr ValueErrortotal_downscale_factordownscale_factorrange)selfr $first_adapter_total_downscale_factorfirst_adapter_downscale_factoridx __class__s g/Volumes/fast/ai/experiments/MLX_z-image/.venv/lib/python3.12/site-packages/diffusers/models/adapter.pyrzMultiAdapter.__init__)s0 lD*,x= h/ x=A => > x=A pq q08{/Q/Q,)1!)E)E&CM*C 448\\C=115SS ::^9_`44R3ST #u$>x}?c?c>de #u$8#9W9W8X Z +'K# >Nxsadapter_weightsreturnc|1tjd|jz g|jz}ntj|}d}t|||jD]c\}}}||}|'|}t t |D] }|||z||<8t t |D]}||xx|||zz cc<e|S)az Args: xs (`torch.Tensor`): A tensor of shape (batch, channel, height, width) representing input images for multiple adapter models, concatenated along dimension 1(channel dimension). The `channel` dimension should be equal to `num_adapter` * number of channel per image. adapter_weights (`list[float]`, *optional*, defaults to None): A list of floats representing the weights which will be multiplied by each adapter's output before summing them together. If `None`, equal weights will be used for all adapters. Nr)torchtensorrzipr rr) rr r! accume_statexwadapterfeaturesis rforwardzMultiAdapter.forwardLs  "#llA0@0@,@+ADDTDT+TUO#ll?;O  _dmmDMAq'qzH#' s<01A&',q/&9LO2s8}-A Oq8A;6O.Ersave_directoryis_main_process save_functionsafe_serializationvariantctd}|}|jD]%}|j||||||dz }|d|z}'y)a6 Save a model and its configuration file to a specified directory, allowing it to be re-loaded with the `[`~models.adapter.MultiAdapter.from_pretrained`]` class method. Args: save_directory (`str` or `os.PathLike`): The directory where the model will be saved. If the directory does not exist, it will be created. is_main_process (`bool`, optional, defaults=True): Indicates whether current process is the main process or not. Useful for distributed training (e.g., TPUs) and need to call this function on all processes. In this case, set `is_main_process=True` only for the main process to avoid race conditions. save_function (`Callable`): Function used to save the state dictionary. Useful for distributed training (e.g., TPUs) to replace `torch.save` with another method. Can also be configured using`DIFFUSERS_SAVE_MODE` environment variable. safe_serialization (`bool`, optional, defaults=True): If `True`, save the model using `safetensors`. If `False`, save the model with `pickle`. variant (`str`, *optional*): If specified, weights are saved in the format `pytorch_model..bin`. r)r/r0r1r2r_N)r save_pretrained) rr.r/r0r1r2rmodel_path_to_saver*s rr5zMultiAdapter.save_pretrainedis[8+}}G  # #" /+#5 $  1HC!3#i!? %rpretrained_model_pathc d}g}|}tjj|rTtj|fi|}|j ||dz }|d|z}tjj|rTt jt|d|dt|dk(r2tdtjj|d|dzd||S) a Instantiate a pretrained `MultiAdapter` model from multiple pre-trained adapter models. The model is set in evaluation mode by default using `model.eval()` (Dropout modules are deactivated). To train the model, set it back to training mode using `model.train()`. Warnings: *Weights from XXX not initialized from pretrained model* means that the weights of XXX are not pretrained with the rest of the model. It is up to you to train those weights with a downstream fine-tuning. *Weights from XXX not used in YYY* means that the layer XXX is not used by YYY, so those weights are discarded. Args: pretrained_model_path (`os.PathLike`): A path to a *directory* containing model weights saved using [`~diffusers.models.adapter.MultiAdapter.save_pretrained`], e.g., `./my_model_directory/adapter`. torch_dtype (`torch.dtype`, *optional*): Override the default `torch.dtype` and load the model under this dtype. output_loading_info(`bool`, *optional*, defaults to `False`): Whether or not to also return a dictionary containing missing keys, unexpected keys and error messages. device_map (`str` or `dict[str, int | str | torch.device]`, *optional*): A map that specifies where each submodule should go. It doesn't need to be refined to each parameter/buffer name, once a given module name is inside, every submodule of it will be sent to the same device. To have Accelerate compute the most optimized `device_map` automatically, set `device_map="auto"`. For more information about each option see [designing a device map](https://hf.co/docs/accelerate/main/en/usage_guides/big_modeling#designing-a-device-map). max_memory (`Dict`, *optional*): A dictionary mapping device identifiers to their maximum memory. Default to the maximum memory available for each GPU and the available CPU RAM if unset. low_cpu_mem_usage (`bool`, *optional*, defaults to `True` if torch version >= 1.9.0 else `False`): Speed up model loading by not initializing the weights and only loading the pre-trained weights. This also tries to not use more than 1x model size in CPU memory (including peak memory) while loading the model. This is only supported when torch version >= 1.9.0. If you are using an older version of torch, setting this argument to `True` will raise an error. variant (`str`, *optional*): If specified, load weights from a `variant` file (*e.g.* pytorch_model..bin). `variant` will be ignored when using `from_flax`. use_safetensors (`bool`, *optional*, defaults to `None`): If `None`, the `safetensors` weights will be downloaded if available **and** if`safetensors` library is installed. If `True`, the model will be forcibly loaded from`safetensors` weights. If `False`, `safetensors` is not used. rrr4z adapters loaded from .zNo T2IAdapters found under z. Expected at least _0) ospathisdirr from_pretrainedappendloggerinforrdirname)clsr7kwargsrr model_path_to_loadr*s rr>zMultiAdapter.from_pretrainedsZ 3ggmm./ 001CNvNG OOG $ 1HC!61SE!B  ggmm./  s8}o%;S.T-UUijCGkGjHHIJ 8}rN)TNTN)__name__ __module__ __qualname____doc__listrr$Tensorfloatr-strr;PathLikeboolrr5 classmethodr> __classcell__rs@rr r s !?l!3!?F%,,et9KW[\a\h\hWi@!%"&#'" (@bkk)(@(@ (@ ! (@ t (@TAC"++4E4LAArr c eZdZdZedgddddfdedeed ed ed ef fd Zd e jdee jfdZ e dZ e dZxZS)r a\ A simple ResNet-like model that accepts images containing control signals such as keyposes and depth. The model generates multiple feature maps that are used as additional conditioning in [`UNet2DConditionModel`]. The model's architecture follows the original implementation of [Adapter](https://github.com/TencentARC/T2I-Adapter/blob/686de4681515662c0ac2ffa07bf5dda83af1038a/ldm/modules/encoders/adapter.py#L97) and [AdapterLight](https://github.com/TencentARC/T2I-Adapter/blob/686de4681515662c0ac2ffa07bf5dda83af1038a/ldm/modules/encoders/adapter.py#L235). This model inherits from [`ModelMixin`]. Check the superclass documentation for the common methods, such as downloading or saving. Args: in_channels (`int`, *optional*, defaults to `3`): The number of channels in the adapter's input (*control image*). Set it to 1 if you're using a gray scale image. channels (`list[int]`, *optional*, defaults to `(320, 640, 1280, 1280)`): The number of channels in each downsample block's output hidden state. The `len(block_out_channels)` determines the number of downsample blocks in the adapter. num_res_blocks (`int`, *optional*, defaults to `2`): Number of ResNet blocks in each downsample block. downscale_factor (`int`, *optional*, defaults to `8`): A factor that determines the total downscale factor of the Adapter. adapter_type (`str`, *optional*, defaults to `full_adapter`): Adapter type (`full_adapter` or `full_adapter_xl` or `light_adapter`) to use. @rYr full_adapter in_channelschannelsnum_res_blocksr adapter_typect||dk(rt|||||_y|dk(rt |||||_y|dk(rt |||||_yt d|d)Nr[full_adapter_xl light_adapterzUnsupported adapter_type: 'zH'. Choose either 'full_adapter' or 'full_adapter_xl' or 'light_adapter'.)rr FullAdapterr* FullAdapterXL LightAdapterr)rr\r]r^rr_rs rrzT2IAdapter.__init__s  > )&{HnN^_DL . .(hP`aDL _ ,' X~O_`DL-l^<88 rr(r"c$|j|S)a This function processes the input tensor `x` through the adapter model and returns a list of feature tensors, each representing information extracted at a different scale from the input. The length of the list is determined by the number of downsample blocks in the Adapter, as specified by the `channels` and `num_res_blocks` parameters during initialization. )r*rr(s rr-zT2IAdapter.forward s||Arc.|jjSrF)r*rrs rrz!T2IAdapter.total_downscale_factors||222rcB|jjjS)zThe downscale factor applied in the T2I-Adapter's initial pixel unshuffle operation. If an input image's dimensions are not evenly divisible by the downscale_factor then an exception will be raised. )r* unshufflerris rrzT2IAdapter.downscale_factors ||%%666r)rGrHrIrJrintrKrNrr$rLr-propertyrrrRrSs@rr r s44 !* s)     ,$u||*<3377rr c eZdZdZdgdddfdedeeded effd Zd ejd eejfd Z xZ S)rc2 See [`T2IAdapter`] for more information. rUrVrrZr\r]r^rc t|||dzz}tj||_tj ||ddd|_tjt|d|d|gtdt|Dcgc]}t||dz |||dc}|_ |dt|dz zz|_ ycc}wNrrrUr kernel_sizepaddingTdown) rrrPixelUnshufflerkConv2dconv_inr AdapterBlockrrbodyrrr\r]r^rr,rs rrzFullAdapter.__init__'s !$4a$77 **+;<yyhqkqRST MMXa[(1+~F #1c(m44!!a%(1+~TXY4   '7s8}q?P9Q&Q#s C r(r"c|j|}|j|}g}|jD]}||}|j||S)a This method processes the input tensor `x` through the FullAdapter model and performs operations including pixel unshuffling, convolution, and a stack of AdapterBlocks. It returns a list of feature tensors, each capturing information at a different stage of processing within the FullAdapter model. The number of feature tensors in the list is determined by the number of downsample blocks specified during initialization. rkryr{r?rr(r+blocks rr-zFullAdapter.forwardAsN NN1  LLOYYEaA OOA r rGrHrIrJrlrKrr$rLr-rRrSs@rrcrc"sj 4 ! RRs)R R  R4$u||*<rrcc eZdZdZdgdddfdedeeded effd Zd ejd eejfd Z xZ S)rdrorUrVrr\r]r^rc tt|||dzz}tj||_tj ||ddd|_g|_tt|D]}|dk(r0|jjt||dz |||8|dk(r2|jjt||dz |||do|jjt|||||tj|j|_|dz|_ yrq)rrrrwrkrxryr{rrr?rzrrr|s rrzFullAdapterXL.__init__Ys  !$4a$77 **+;<yyhqkqRST  s8}%AAv   hq1uox{N![\a   hq1uox{Nae!fg   hqk8A;!WX &MM$)), &6&:#rr(r"c|j|}|j|}g}|jD]}||}|j||S)z This method takes the tensor x as input and processes it through FullAdapterXL model. It consists of operations including unshuffling pixels, applying convolution layer and appending each block into list of feature tensors. r~rs rr-zFullAdapterXL.forwardusN NN1  LLOYYEaA OOA rrrSs@rrdrdTse 4 " ;;s); ;  ;8$u||*<rrdc neZdZdZd dedededeffd Zdejdejfd Z xZ S) rza# An AdapterBlock is a helper model that contains multiple ResNet-like blocks. It is used in the `FullAdapter` and `FullAdapterXL` models. Args: in_channels (`int`): Number of channels of AdapterBlock's input. out_channels (`int`): Number of channels of AdapterBlock's output. num_res_blocks (`int`): Number of ResNet blocks in the AdapterBlock. down (`bool`, *optional*, defaults to `False`): If `True`, perform downsampling on AdapterBlock's input. r\ out_channelsr^rvc8t|d|_|rtjddd|_d|_||k7rtj ||d|_tjt|Dcgc] }t|c}|_ ycc}w)NrTrsstride ceil_moderrs) rr downsampler AvgPool2din_convrx SequentialrAdapterResnetBlockresnets)rr\rr^rvr4rs rrzAdapterBlock.__init__s   llqdSDO , &99[,ANDL}}8=n8M N8M1 .8M N  Ns;Br(r"c|j|j|}|j|j|}|j|}|S)a This method takes tensor x as input and performs operations downsampling and convolutional layers if the self.downsample and self.in_conv properties of AdapterBlock model are specified. Then it applies a series of residual blocks to the input tensor. )rrrrgs rr-zAdapterBlock.forwardsE ?? &"A << # QA LLOrF rGrHrIrJrlrPrr$rLr-rRrSs@rrzrzsE   C  s  C  W[  %,,rrzc`eZdZdZdeffd ZdejdejfdZxZ S)rz An `AdapterResnetBlock` is a helper model that implements a ResNet-like block. Args: channels (`int`): Number of channels of AdapterResnetBlock's input and output. r]ct|tj||dd|_tj |_tj||d|_y)NrUrrrrrrrrxblock1ReLUactblock2rr]rs rrzAdapterResnetBlock.__init__sE ii(1M 779ii(B rr(r"cn|j|j|}|j|}||zS)z This method takes input tensor x and applies a convolutional layer, ReLU activation, and another convolutional layer on the input tensor. It returns addition with the input tensor. rrrrr(hs rr-zAdapterResnetBlock.forward0 HHT[[^ $ KKN1u r rGrHrIrJrlrr$rLr-rRrSs@rrrs1CC  %,, rrc eZdZdZdgdddfdedeeded effd Zd ejd eejfd Z xZ S)rerorU)rWrXrYrZr\r]r^rc t|||dzz}tj||_tj t ||d|gtt|dz Dcgc]}t ||||dz|dc}t |d|d|d|_ |dt|zz|_ ycc}w)NrrrTru) rrrrwrkrLightAdapterBlockrrr{rr|s rrzLightAdapter.__init__s !$4a$77 **+;<MM!+x{NK #3x=1#455&hqk8AE?NY]^5  "(2, nSWX     '7!s8}:L&M#s*C r(r"c|j|}g}|jD]}||}|j||S)z This method takes the input tensor x and performs downscaling and appends it in list of feature tensors. Each feature tensor corresponds to a different level of processing within the LightAdapter. )rkr{r?rs rr-zLightAdapter.forwardsA NN1 YYEaA OOA rrrSs@rreresj . ! NNs)N N  N4  $u||*< rrec neZdZdZd dedededeffd Zdejdejfd Z xZ S) ra< A `LightAdapterBlock` is a helper model that contains multiple `LightAdapterResnetBlocks`. It is used in the `LightAdapter` model. Args: in_channels (`int`): Number of channels of LightAdapterBlock's input. out_channels (`int`): Number of channels of LightAdapterBlock's output. num_res_blocks (`int`): Number of LightAdapterResnetBlocks in the LightAdapterBlock. down (`bool`, *optional*, defaults to `False`): If `True`, perform downsampling on LightAdapterBlock's input. r\rr^rvcdt||dz}d|_|rtjddd|_tj ||d|_tjt|Dcgc] }t|c}|_ tj ||d|_ ycc}w)NrrTrrr) rrrrrrxrrrLightAdapterResnetBlockrout_conv)rr\rr^rv mid_channelsr4rs rrzLightAdapterBlock.__init__s #q(   llqdSDOyylJ }}V[\jVk&lVkQR'>|'LVk&lm  , !L 'ms4B-r(r"c|j|j|}|j|}|j|}|j|}|S)z This method takes tensor x as input and performs downsampling if required. Then it applies in convolution layer, a sequence of residual blocks, and out convolutional layer. )rrrrrgs rr-zLightAdapterBlock.forwardsI ?? &"A LLO LLO MM! rrrrSs@rrrsJ  MC Ms MC MW[ M  %,, rrc`eZdZdZdeffd ZdejdejfdZxZ S)ra A `LightAdapterResnetBlock` is a helper model that implements a ResNet-like block with a slightly different architecture than `AdapterResnetBlock`. Args: channels (`int`): Number of channels of LightAdapterResnetBlock's input and output. r]ct|tj||dd|_tj |_tj||dd|_y)NrUrrrrrs rrz LightAdapterResnetBlock.__init__8sG ii(1M 779ii(1M rr(r"cn|j|j|}|j|}||zS)z This function takes input tensor x and processes it through one convolutional layer, ReLU activation, and another convolutional layer and adds it to input tensor. rrs rr-zLightAdapterResnetBlock.forward>rrrrSs@rrr.s1NN  %,, rr)r;typingrr$torch.nnrconfiguration_utilsrrutilsrmodeling_utilsr get_loggerrGr@r r Modulercrdrzrrerrrrrs  A&   H %y:yxD7[D7T/"))/d/BII/d-299-`<,299,^( (Vbiir