(tiƄDddlmZddlZddlZddlmZddlmZddl m Z ddl m Z ddl mZdd lmZmZmZmZeGd d eZeGd d eZGddej,ZGddej,ZGddej,ZGddej,ZGddej,ZGddej,ZGddeZGddeZGddej,Z Gd d!ej,Z!Gd"d#Z"y)$) dataclassN) BaseOutput) randn_tensor)get_activation) SpatialNorm)AutoencoderTinyBlockUNetMidBlock2Dget_down_block get_up_blockc0eZdZUdZej ed<y) EncoderOutputz Output of encoding method. Args: latent (`torch.Tensor` of shape `(batch_size, num_channels, latent_height, latent_width)`): The encoded latent. latentN)__name__ __module__ __qualname____doc__torchTensor__annotations__p/Volumes/fast/ai/experiments/MLX_z-image/.venv/lib/python3.12/site-packages/diffusers/models/autoencoders/vae.pyrr s LLrrcXeZdZUdZej ed<dZejdzed<y) DecoderOutputz Output of decoding method. Args: sample (`torch.Tensor` of shape `(batch_size, num_channels, height, width)`): The decoded output sample from the last layer of the model. sampleN commit_loss) rrrrrrrr FloatTensorrrrrr-s( LL,0K""T)0rrceZdZdZ ddededeedfdeedfdeded ed effd Zd e jd e jfdZ xZ S)Encodera The `Encoder` layer of a variational autoencoder that encodes its input into a latent representation. Args: in_channels (`int`, *optional*, defaults to 3): The number of input channels. out_channels (`int`, *optional*, defaults to 3): The number of output channels. down_block_types (`tuple[str, ...]`, *optional*, defaults to `("DownEncoderBlock2D",)`): The types of down blocks to use. See `~diffusers.models.unet_2d_blocks.get_down_block` for available options. block_out_channels (`tuple[int, ...]`, *optional*, defaults to `(64,)`): The number of output channels for each block. layers_per_block (`int`, *optional*, defaults to 2): The number of layers per block. norm_num_groups (`int`, *optional*, defaults to 32): The number of groups for normalization. act_fn (`str`, *optional*, defaults to `"silu"`): The activation function to use. See `~diffusers.models.activations.get_activation` for available options. double_z (`bool`, *optional*, defaults to `True`): Whether to double the number of output channels for the last block. in_channels out_channelsdown_block_types.block_out_channelslayers_per_blocknorm_num_groupsact_fndouble_zc t|||_tj||dddd|_tj g|_|d} t|D]Y\} } | } || } | t|dz k(}t| |j| | | dd||| d }|jj|[t|dd|dd|d|d| |_ tj|d|d |_tj |_|rd |zn|}tj|d|dd |_d |_y)Nrr kernel_sizestridepaddingư>) num_layersr"r#add_downsample resnet_epsdownsample_padding resnet_act_fn resnet_groupsattention_head_dim temb_channelsdefault r"r3r5output_scale_factorresnet_time_scale_shiftr7r6r8 add_attention num_channels num_groupsepsrr/F)super__init__r&nnConv2dconv_in ModuleList down_blocks enumeratelenr appendr mid_block GroupNorm conv_norm_outSiLUconv_actconv_outgradient_checkpointing)selfr"r#r$r%r&r'r(r)mid_block_add_attentionoutput_channelidown_block_type input_channelis_final_block down_blockconv_out_channels __class__s rrEzEncoder.__init__Ssr  0yy  q !   ==,,A."+,<"= A*M/2N#&8"9A"==N'00)+#11#$$-#1" J    # #J /%#>*(*2. !$-1"5)1   \\7I"7MZiost  08A ,l "4R"8:KQXYZ &+#rrreturnc|j|}tjrL|jr@|jD]}|j ||}|j |j |}n*|jD] }||} |j |}|j|}|j|}|j|}|S)z*The forward method of the `Encoder` class.) rHris_grad_enabledrTrJ_gradient_checkpointing_funcrNrPrRrS)rUrr\s rforwardzEncoder.forwardsf%  "t'B'B".. :::vN/66t~~vNF#.. #F+/^^F+F##F+v&v& r) rr)DownEncoderBlock2D@r siluTT) rrrrinttuplestrboolrErrrc __classcell__r^s@rr!r!;s2,C.3 !! $C,C,C, S/ C, "#s(O C,  C,C,C,C,Jellu||rr!ceZdZdZ ddededeedfdeedfdeded ed effd Z dd ejdejd zdejfdZ xZ S)Decodera The `Decoder` layer of a variational autoencoder that decodes its latent representation into an output sample. Args: in_channels (`int`, *optional*, defaults to 3): The number of input channels. out_channels (`int`, *optional*, defaults to 3): The number of output channels. up_block_types (`tuple[str, ...]`, *optional*, defaults to `("UpDecoderBlock2D",)`): The types of up blocks to use. See `~diffusers.models.unet_2d_blocks.get_up_block` for available options. block_out_channels (`tuple[int, ...]`, *optional*, defaults to `(64,)`): The number of output channels for each block. layers_per_block (`int`, *optional*, defaults to 2): The number of layers per block. norm_num_groups (`int`, *optional*, defaults to 32): The number of groups for normalization. act_fn (`str`, *optional*, defaults to `"silu"`): The activation function to use. See `~diffusers.models.activations.get_activation` for available options. norm_type (`str`, *optional*, defaults to `"group"`): The normalization type to use. Can be either `"group"` or `"spatial"`. r"r#up_block_types.r%r&r'r( norm_typec t|||_tj||dddd|_tj g|_|dk(r|nd} t|dd|d|dk(rdn||d|| | |_ tt|} | d } t|D]_\} }| }| | } | t|dz k(}t||jdz|| || d||| | | }|jj|| }a|dk(rt!|d | |_n tj$|d |d |_tj&|_tj|d |dd |_d|_y)Nr9rr+r,spatialr0groupr:r;r r1r"r#prev_output_channel add_upsampler3r5r6r7r8r=r?rCF)rDrEr&rFrGrHrI up_blocksr rNlistreversedrKrLr rMr rPrOrQrRrSrT)rUr"r#rqr%r&r'r(rrrVr8reversed_block_out_channelsrWrX up_block_typerwr[up_blockr^s rrEzDecoder.__init__s  0yy  r "   r*'0I'= 4 (*2. !1:g1EI91"5)'1  '+84F+G&H#4Q7 ). 9 A}"0 8;N#&8"9A"==N#0014/+$7!//$-#1+(1 H NN ! !( +"0 +!:0  !!,-?-BM!RD !#;Ma;P]lrv!wD    "4Q"7qRST &+#rNr latent_embedsr_c|j|}tjrN|jrB|j |j ||}|j D]}|j |||}n,|j ||}|j D] }|||} ||j|}n|j||}|j|}|j|}|S)z*The forward method of the `Decoder` class.) rHrrarTrbrNryrPrRrS)rUrrr~s rrczDecoder.forwardsf%  "t'B'B66t~~v}]F!NN::8V][+^^FM:F!NN!&-8+  ''/F'' >Fv&v& r) rrUpDecoderBlock2DrerrgrhruTN rrrrrirjrkrErrrcrmrns@rrprps0*?.3 !! $J,J,J,c3h J, "#s(O J,  J,J,J,J,^.2   ||d*   rrpcheZdZdZdededdffd ZdejdejfdZxZ S) UpSamplea& The `UpSample` layer of a variational autoencoder that upsamples its input. Args: in_channels (`int`, *optional*, defaults to 3): The number of input channels. out_channels (`int`, *optional*, defaults to 3): The number of output channels. r"r#r_Nc|t|||_||_t j ||ddd|_y)Nrr+r,)rDrEr"r#rFConvTranspose2ddeconv)rUr"r#r^s rrEzUpSample.__init__Es= &(((lPQZ[efg rxcRtj|}|j|}|S)z+The forward method of the `UpSample` class.)rrelurrUrs rrczUpSample.forwardOs! JJqM KKNr rrrrrirErrrcrmrns@rrr:sHhhh  h%,,rrc zeZdZdZ d dededededdf fd Zd d ejdejfd ZxZ S) MaskConditionEncoderz) used in AsymmetricAutoencoderKL in_chout_chres_chr.r_Nc 4t |g}|dkDr6|dz}|dz}||kDr|}|dk(r|}|j||f|dz}|dkDr6g}|D]\}} |j| |j|ddg} |}tt |D]f} || } | dk(s| dk(r*| jt j || dddn)| jt j || ddd| }ht j| |_y)Nr+rr9rrr,r) rDrErMrangerLrFrG Sequentiallayers)rUrrrr.channelsin_ch_r#_in_ch_out_chrlout_ch_r^s rrEzMaskConditionEncoder.__init__[s) qjq[FaZF{ OOVV, - aKFqj 'OFG    ( (HRLO,s<()A"1oGAva biiQqZ[\] biiQqZ[\]F *mmV, rrci}tt|jD]O}|j|}||}||tt |j <t j|}Q|S)z7The forward method of the `MaskConditionEncoder` class.)rrLrrkrjshaperr)rUrmaskoutrlayers rrczMaskConditionEncoder.forwards`s4;;'(AKKNEaA'(CE!''N# $ 1 A )  r)irrrns@rrrVsd #-#-#- #-  #-  #-JU\\rrceZdZdZ ddededeedfdeedfdeded ed effd Z dd ejdejd zdejd zdejd zdejf dZ xZ S)MaskConditionDecoderaThe `MaskConditionDecoder` should be used in combination with [`AsymmetricAutoencoderKL`] to enhance the model's decoder with a conditioner on the mask and masked image. Args: in_channels (`int`, *optional*, defaults to 3): The number of input channels. out_channels (`int`, *optional*, defaults to 3): The number of output channels. up_block_types (`tuple[str, ...]`, *optional*, defaults to `("UpDecoderBlock2D",)`): The types of up blocks to use. See `~diffusers.models.unet_2d_blocks.get_up_block` for available options. block_out_channels (`tuple[int, ...]`, *optional*, defaults to `(64,)`): The number of output channels for each block. layers_per_block (`int`, *optional*, defaults to 2): The number of layers per block. norm_num_groups (`int`, *optional*, defaults to 32): The number of groups for normalization. act_fn (`str`, *optional*, defaults to `"silu"`): The activation function to use. See `~diffusers.models.activations.get_activation` for available options. norm_type (`str`, *optional*, defaults to `"group"`): The normalization type to use. Can be either `"group"` or `"spatial"`. r"r#rq.r%r&r'r(rrc 8t|||_tj||dddd|_tj g|_|dk(r|nd} t|dd|d|dk(rdn||d|| |_ tt|} | d } t|D]_\} } | }| | } | t|dz k(}t| |jdz|| d| d||| | | }|jj|| }at!||d |d |_|dk(rt%|d | |_n tj(|d |d |_tj*|_tj|d |dd|_d|_y)Nr9rr+r,rtr0rur:)r"r3r5r<r=r7r6r8rrv)rrrr?rCF)rDrEr&rFrGrHrIryr rNrzr{rKrLr rMrcondition_encoderr rPrOrQrRrSrT)rUr"r#rqr%r&r'r(rrr8r|rWrXr}rwr[r~r^s rrEzMaskConditionDecoder.__init__s  0yy  r "   r*'0I'= 4 (*2. !1:g1EI91"5)'  '+84F+G&H#4Q7 ). 9 A}"0 8;N#&8"9A"==N#0014/+$(!//$-#1+(1 H NN ! !( +"0 +!:0"6%a(%b)"   !!,-?-BM!RD !#;Ma;P]lrv!wD    "4Q"7qRST &+#rNzimagerrr_cj|}|j|}tt|jj j }t jr|jr|j|j||}|j|}|'|%d|z |z}|j|j||}|jD]w} |`|^tt|j} t j"j%||jddd} || z| d| z zz}|j| ||}y||||ztt|jd|z zz}n|j||}|j|}||d|z |z}|j||}|jD]m} |`|^tt|j} t j"j%||jddd} || z| d| z zz}| ||}o|/|-||ztt|jd|z zz}||j'|}n|j'||}|j)|}|j+|}|S)z7The forward method of the `MaskConditionDecoder` class.Nr+nearest)sizemode)rHnextiterry parametersdtyperrarTrbrNtorrkrjrrF functional interpolaterPrRrS) rUrrrrr upscale_dtype masked_imageim_xr~sample_mask_s rrczMaskConditionDecoder.forwardsf%T$..";";"=>?EE  "t'B'B66t~~v}]FYY}-F T%5 !DE1 88** !NN$)9"3uV\\':#;Fv&v& r)rrrrerrgrhru)NNNrrns@rrrs0*?.3 !! O,O,O,c3h O, "#s(O O,  O,O,O,O,h&*$(-1 ? <<?||d"?llT! ? ||d* ?  ?rrc TeZdZdZ ddedededededef fd Zd e jd e jfd Z d e jd e jfd Z d e jd ee je jeffdZde jdeedfd e jfdZxZS)VectorQuantizerz Improved version over VectorQuantizer, can be used as a drop-in replacement. Mostly avoids costly matrix multiplications and allows for post-hoc remapping of indices. n_e vq_embed_dimbeta unknown_indexsane_index_shapelegacyc >t|||_||_||_||_t j|j|j|_|jjjjd|jz d|jz ||_ |j|jdtjt!j"|j||j$j&d|_||_|j*dk(r%|j(|_|j(dz|_t-d|jd|j(d |j*d ||_y||_||_y) Ng?usedrextrar+z Remapping z indices to z indices. Using z for unknown indices.)rDrErrrrrF Embedding embeddingweightdatauniform_remapregister_bufferrtensornploadrrre_embedrprintr) rUrrrrrrrr^s rrEzVectorQuantizer.__init__>sL (  dhh0A0AB ""++D488OS488^L :: !  bggdjj6I)J K  IIOOA.DM!.D !!W,%)]]" $  1 TXXJl4==/B++,,AC !1 DM 0rindsr_c|j}t|dkDsJ|j|dd}|jj |}|dddddf|dk(j }|j d}|jddk}|jdk(rMtjd|j||jj |j||<n|j||<|j|S) Nr+rr9)NN.rrandom)r)device) rrLreshaperrlongargmaxsumrrrandintrr)rUrishapermatchnewunknowns r remap_to_usedzVectorQuantizer.remap_to_usedcs6{Q||F1Ir*yy||D!aDj!T/%::@@Bll2))A,"    ) ==DMMG @R@RSVV^a^h^hViCL--CL{{6""rc|j}t|dkDsJ|j|dd}|jj |}|j |jjdkDrd|||jjdk\<t j|dddf|jddgzddfd|}|j|S)Nr+rr9)rrLrrrrrgather)rUrrrbacks r unmap_to_allzVectorQuantizer.unmap_to_allqs6{Q||F1Ir*yy||D! ==499??1- -/0D++ ,||DqM$**Q-1#*=q*@A1dK||F##rrc|jddddj}|jd|j}t j t j ||jjd}|j|j|j}d}d}|jsa|jt j|j|z dzzt j||jz dzz}n`t j|j|z dz|jt j||jz dzzz}|||z jz}|jddddj}|jB|j|jdd}|j!|}|jdd}|j"r:|j|jd|jd|jd}|||||ffS)Nrrrr+r9dim)permute contiguousviewrrargmincdistrrrrrmeandetachrrrr)rUr z_flattenedmin_encoding_indicesz_q perplexity min_encodingslosss rrczVectorQuantizer.forward{s IIaAq ! , , .ffR!2!23  %||EKK T^^EZEZ,[abcnn1277@  {{99uzz3::g4@?)rr) rrchunkrlogvarclamprexpstdvar zeros_likerr)rUrrs rrEz%DiagonalGaussianDistribution.__init__s$!&Z!B 4;kk$++ud; *99S4;;./99T[[)   "'"2"2 $//"8"8@U@U# DHtx rN generatorr_ct|jj||jj|jj }|j|j |zz}|S)N)rrr)rrrrrrr)rUrrrs rrz#DiagonalGaussianDistribution.samplesR IIOO??))//''   II6) )rotherc|jrtjdgS|Wdtjtj|j d|j zdz |jz gdzSdtjtj|j |j z d|j z |j |j z zdz |jz |jzgdzS)Nrrrr+rrr)rrrrpowrrr)rUr s rklzDiagonalGaussianDistribution.kls   <<& &}UYYIIdii+dhh6r@rCrErrrr8r8s  !rr8)# dataclassesrnumpyrrtorch.nnrFutilsrutils.torch_utilsr activationsrattention_processorr unets.unet_2d_blocksr r r r rrModuler!rprrrrobjectrrr!r,r8rrrrPs" -(-  J     1J 1  1vbiivrCbiiCLryy822992jg299gTwbiiwt565p6?"))?DH"))HV ! !r