(tiddlZddlZddlZddlZddlZddlZddlZddlmZddlm Z ddl m Z m Z ddl ZddlZeeddddlmZmZddlmZddlmZd d lmZd d lmZd d lmZd d lmZm Z m!Z!m"Z"m#Z#m$Z$m%Z%m&Z&e&r,ddl'Z'e'jPjRjUrddl)Z)e"rddl+m,Z,e#rddl-m.Z.e%rddl/m0Z0e$rddl1Z1de2fdZ3dZ4de5fdZ6 dCdededejndejndejndejndejnde8de9ejnejnffdZ:dede;e5ejnffd Ze;e5e fdzd?eOePej|j~dzdefd@ZQGdAdBZRy)GN)contextmanager)partial)AnyIterable distributed) CPUOffloadShardingStrategy)FullyShardedDataParallel)transformer_auto_wrap_policy)UNet2DConditionModel)DiffusionPipeline)SchedulerMixin)convert_state_dict_to_diffusersconvert_state_dict_to_peft deprecateis_accelerate_availableis_peft_availableis_torch_npu_availableis_torchvision_availableis_transformers_available) get_logger)set_peft_model_state_dict) transformsseedc(tj|tjj|tj|t r tj j|ytjj|y)z Helper function for reproducible behavior to set the seed in `random`, `numpy`, `torch`. Args: seed (`int`): The seed to set. Returns: `None` N) randomrnptorch manual_seedrnpumanual_seed_allcuda)rs g/Volumes/fast/ai/experiments/MLX_z-image/.venv/lib/python3.12/site-packages/diffusers/training_utils.pyset_seedr%7sX KKIINN4 d !!$' ""4(c|j}|dz}d|z dz}|j|j|j}t |j t |j kr1|d}t |j t |j kr1|j |j }|j|j|j}t |j t |j kr1|d}t |j t |j kr1|j |j }||z dz}|S)a Computes SNR as per https://github.com/TiankaiHang/Min-SNR-Diffusion-Training/blob/521b624bd70c67cee4bdf49225915f5945a872e3/guided_diffusion/gaussian_diffusion.py#L847-L849 for the given timesteps using the provided noise scheduler. Args: noise_scheduler (`NoiseScheduler`): An object containing the noise schedule parameters, specifically `alphas_cumprod`, which is used to compute the SNR values. timesteps (`torch.Tensor`): A tensor of timesteps for which the SNR is computed. Returns: `torch.Tensor`: A tensor containing the computed SNR values for each timestep. ??device).N)alphas_cumprodtor+floatlenshapeexpand)noise_scheduler timestepsr-sqrt_alphas_cumprodsqrt_one_minus_alphas_cumprodalphasigmasnrs r$ compute_snrr:KsH %33N(#-%(>%9c$A!.00 8H8H0I)TZZ\ !'' (3y+? ?1)< !'' (3y+? ?  & &y 7E$A$D$DIL\L\$D$]^g$h$n$n$p! +11 2S5I I(Ei(P% +11 2S5I I ) 0 0 AE 5=Q C Jr&interpolation_typects td|dk(rtjj}|S|dk(rtjj }|S|dk(rtjj }|S|dk(rtjj}|S|dk(rtjj}|S|dk(rtjj}|S|dk(rtjj}|Std |d ) a Maps a string describing an interpolation function to the corresponding torchvision `InterpolationMode` enum. The full list of supported enums is documented at https://pytorch.org/vision/0.9/transforms.html#torchvision.transforms.functional.InterpolationMode. Args: interpolation_type (`str`): A string describing an interpolation method. Currently, `bilinear`, `bicubic`, `box`, `nearest`, `nearest_exact`, `hamming`, and `lanczos` are supported, corresponding to the supported interpolation modes in torchvision. Returns: `torchvision.transforms.InterpolationMode`: an `InterpolationMode` enum used by torchvision's `resize` transform. zhPlease make sure to install `torchvision` to be able to use the `resolve_interpolation_mode()` function.bilinearbicubicboxnearest nearest_exacthamminglanczoszThe given interpolation mode z is not supported. Currently supported interpolation modes are `bilinear`, `bicubic`, `box`, `nearest`, `nearest_exact`, `hamming`, and `lanczos`.) r ImportErrorrInterpolationModeBILINEARBICUBICBOXNEAREST NEAREST_EXACTHAMMINGLANCZOS ValueError)r;interpolation_modes r$resolve_interpolation_moderOps; $ % v  Z''99BB& % y ('99AA" ! u $'99==  y ('99AA   .'99GG  y ('99AA  y ('99AA  +,>+?@m n  r&unetr3r4noise noisy_latentstargetencoder_hidden_statesdream_detail_preservationreturncX|jj|j|dddf}d|z dz} | |z} d} tj5||||j } dddd\} } |j jdk(rO| }||z j}|j| |j| |z} |j|} | | fS|j jdk(r tdtd|j j#1swYxYw) a Implements "DREAM (Diffusion Rectification and Estimation-Adaptive Models)" from https://huggingface.co/papers/2312.00210. DREAM helps align training with sampling to help training be more efficient and accurate at the cost of an extra forward step without gradients. Args: `unet`: The state unet to use to make a prediction. `noise_scheduler`: The noise scheduler used to add noise for the given timestep. `timesteps`: The timesteps for the noise_scheduler to user. `noise`: A tensor of noise in the shape of noisy_latents. `noisy_latents`: Previously noise latents from the training loop. `target`: The ground-truth tensor to predict after eps is removed. `encoder_hidden_states`: Text embeddings from the text model. `dream_detail_preservation`: A float value that indicates detail preservation level. See reference. Returns: `tuple[torch.Tensor, torch.Tensor]`: Adjusted noisy_latents and target. Nr)r()NNepsilon v_predictionz/DREAM has not been implemented for v-predictionzUnknown prediction type ) r-r.r+rno_gradsampleconfigprediction_typedetachmul_addNotImplementedErrorrM)rPr3r4rQrRrSrTrUr-r6 dream_lambdapred_noisy_latents_targetpredicted_noise delta_noises r$ compute_dream_and_update_latentsrhs7:%3366y7G7GHTXZ^`dIdeN%(>%9c$A!12KKL D M9.CDKK +NG--:.668 &&**+H;+VW**[) 7 ""    / /> A!"STT3O4J4J4Z4Z3[\]] s D  D)ci}|jD]S\}}t|dst|d}|"|j}|j D]\}}|||d|<U|S)zL Returns: A state dict containing just the LoRA parameters. set_lora_layer lora_layerz.lora.) named_moduleshasattrgetattr state_dictitems)rPlora_state_dictnamemodulerkcurrent_lora_layer_sdlora_layer_matrix_name lora_params r$unet_lora_state_dictrws O**, f 6+ , 6J%(2(=(=(?%:O:U:U:W6*JOYOtfF3I2J$KL;X - r&modelct|ts|g}|D]:}|jD]%}|js|j ||_'<y)z Casts the training parameters of the model to the specified data type. Args: model: The PyTorch model whose parameters will be cast. dtype: The data type to which the model parameters will be cast. N) isinstancelist parameters requires_gradr.data)rxdtypemparams r$cast_training_paramsrsG eT " \\^E"""XXe_ $r&rqprefix text_encoderc|jDcic]+\}}|j|s|j|d|-}}}tt |}t ||dycc}}w)aD Sets the `lora_state_dict` into `text_encoder` coming from `transformers`. Args: lora_state_dict: The state dictionary to be set. prefix: String identifier to retrieve the portion of the state dict that belongs to `text_encoder`. text_encoder: Where the `lora_state_dict` is to be set. default) adapter_nameN)rp startswithreplacerrr)rqrrkvtext_encoder_state_dicts r$!_set_state_dict_into_text_encoderrsu3B2G2G2I2I$!QQ\\Z`Ma199VR !A%2I99XYp9qrl,CR[\ s A*A*modules_to_saveci}|jD]+\}}| |jdj||d<-|S)Nr_lora_adapter_metadata)rp peft_configto_dict)r metadatas module_namerss r$_collate_lora_metadatar sTI.446 V  @F@R@RS\@]@e@e@gI %;< = 7 r&weighting_scheme batch_size logit_mean logit_std mode_scaler+ generatorc|dk(rFtj|||f||}tjjj |}|S|dk(rVtj |f||}d|z |tj tj|zdz dzdz |zzz }|Stj |f||}|S)a Compute the density for sampling the timesteps when doing SD3 training. Courtesy: This was contributed by Rafie Walker in https://github.com/huggingface/diffusers/pull/8528. SD3 paper reference: https://huggingface.co/papers/2403.03206v1. logit_normal)meanstdsizer+rmode)rr+rr r,) rnormalnn functionalsigmoidrandcosmathpi)rrrrrr+rus r$%compute_density_for_timestep_samplingrs >) LLjizmTZfo p HH   ' ' * H V # JJZM&I N EJ%))DGGaK!O"<"AA"E"IJ J H JJZM&I N Hr&c|dk(r|dzj}|S|dk(r)dd|zz d|dzzz}dtj|zz }|Stj|}|S)z Computes loss weighting scheme for SD3 training. Courtesy: This was contributed by Rafie Walker in https://github.com/huggingface/diffusers/pull/8528. SD3 paper reference: https://huggingface.co/papers/2403.03206v1. sigma_sqrtgcosmapr r,)r/rrr ones_like)rsigmas weightingbots r$compute_loss_weighting_for_sd3r,sz<'T\((*  X %!f*nq619},3'  OOF+ r&c"tjtjj rtjj ytj jj rtjj ytrtjj yttdr>tjj rtjj yyy)zV Runs garbage collection. Then clears the cache of the available accelerator. xpuN) gccollectrr# is_available empty_cachebackendsmpsr torch_npur!rmrr&r$ free_memoryr>sJJL zz      ( ( *   ! !!#  599#9#9#; $< r&T)offloadmodulesrc'K|r~td|D }|r1|Dcgc]%}t|jj'}}n t |dk(sJ|djg}|D]}|j | d|r&t |D]\}}|j |yycc}w#|r&t |D]\}}|j |wwxYww)a Context manager that, if offload=True, moves each module to `device` on enter, then moves it back to its original device on exit. Args: device (`str` or `torch.Device`): Device to move the `modules` to. offload (`bool`): Flag to enable offloading. c3<K|]}t|tywN)rzr).0rs r$ z!offload_models..YsMW:a):;Wsr rN)anynextr|r+r0r.zip)r+rris_modelroriginal_devicesorig_devs r$offload_modelsrNsMWMMM ELMWQ\\^ 4 ; ;W Mw<1$ $$ ' 1 12 A DDL "7,<= 8X >  N "7,<= 8X > s(C#*B1;C#B6.C#6*C  C#c0|s td|jjd}g}|D]}tjd|}|std|d t |j d}t |j d}|dks|dkr td |d zdk7s|d zdk7rtjd |d |d |j||f|s td|S#t$r}td|d||d}~wwxYw)zGParses a string defining buckets into a list of (height, width) tuples.zBucket string cannot be empty.;z^\s*(\d+)\s*,\s*(\d+)\s*$zInvalid bucket format: 'z'. Expected 'height,width'.r r,rz,Bucket dimensions must be positive integers.zBucket dimension (,z.) not divisible by 8. This might cause issues.z Invalid integer in bucket pair 'z': Nz.No valid buckets found in the provided string.) rMstripsplitrematchintgroupwarningswarnappend) buckets_str bucket_pairsparsed_bucketspair_strrheightwidthes r$parse_buckets_stringrns1 9::$$&,,S1LN 5x@7zA\]^ ^ YQ(F A'E{eqj !OPPzQ%!)q. 26(!E7Bpqr  ! !65/ 2! IJJ   Y?zQCPQWX X YsBC55 D>DDctd}d}t|D]&\}\}}t||z||zz }||ks#|}|}(|S)z6Finds the closes bucket to the given height and width.infN)r/ enumerateabs) hwbucket_options min_metricbest_bucket_idx bucket_idxbucket_hbucket_wmetrics r$find_nearest_bucketrs[uJO,5n,E( (XxQ\AL01 Z J(O -F r&c |jDcic]N\}}|t|tjr,|j j j n|Pc}}Scc}}wr)rprzrTensorr^cpu contiguous) state_dictsrrs r$_to_cpu_contiguousrsW_j_p_p_r s_rW[WXZ[A 1ell0Kqxxz~~**,QR R_r ss ssAA+ci}t|jdd}| td|jj}|tj |d<|S|j xstj |d<|S)zT Extract and convert FSDP config from Accelerator into PyTorch FSDP kwargs. fsdp_pluginNzLAccelerate isn't configured to handle FSDP. Please update your installation.sharding_strategy)rnstaterMrr FULL_SHARDr) acceleratorkwargs fsdp_staters r$ get_fsdp_kwargs_from_acceleratorrs F**M4@Jghh##//K&6&A&A"# M'2&C&C&bGWGbGb"# Mr&use_orig_paramslimit_all_gathers fsdp_kwargstransformer_layer_clscJtt}|Jt|jjj d}|j d|jtt|h}||r t|nd|||d} |r| j|t|fi| } | S)uY Wrap a model with FSDP using common defaults and optional transformer auto-wrapping. Args: model: Model to wrap device: Target device (e.g., accelerator.device) offload: Whether to enable CPU parameter offloading use_orig_params: Whether to use original parameters limit_all_gathers: Whether to limit all gathers fsdp_kwargs: FSDP arguments (sharding_strategy, etc.) — usually from Accelerate config transformer_layer_cls: Classes for auto-wrapping (if not using policy from fsdp_kwargs) Returns: FSDP-wrapped model Nrz8transformer_layer_cls is not provided, auto-inferred as )r)offload_params) device_id cpu_offloadrrauto_wrap_policy) r__name__typerxlanguage_modellayersinforr rupdateFSDP) rxr+rrrrrloggerrr\ fsdp_models r$wrap_with_fsdprs2 !F$ $U[[%?%?%F%Fq%I J NOdOmOmNnop;TiSjk=Dz9$*., F k"e&v&J r&ceZdZdZ ddeej jdedede de dee zd ee zd e d e dzd e e e fdzfd ZedddZdZde defdZej&deej jfdZdeej jddfdZddZd ddZde fdZdeej jddfdZdeej jddfdZde ddfdZy)!EMAModelz6 Exponential Moving Average of models weights Nr|decay min_decayupdate_after_stepuse_ema_warmup inv_gammapowerforeach model_cls model_configc  t|tjjr#d} t dd| d|j }d}| j ddd } t dd| d| d}| j d dd } t d d| d| d }t|}|D cgc] } | jj"c} |_ | j d d&d } t d d| d|j| d d|_ ||_ ||_||_||_||_||_d|_d|_||_| |_| |_ycc} w)ar Args: parameters (Iterable[torch.nn.Parameter]): The parameters to track. decay (float): The decay factor for the exponential moving average. min_decay (float): The minimum decay factor for the exponential moving average. update_after_step (int): The number of steps to wait before starting to update the EMA weights. use_ema_warmup (bool): Whether to use EMA warmup. inv_gamma (float): Inverse multiplicative factor of EMA warmup. Default: 1. Only used if `use_ema_warmup` is True. power (float): Exponential factor of EMA warmup. Default: 2/3. Only used if `use_ema_warmup` is True. foreach (bool): Use torch._foreach functions for updating shadow parameters. Should be faster. device (str | torch.device | None): The device to store the EMA weights on. If None, the EMA weights will be stored on CPU. @crowsonkb's notes on EMA Warmup: If gamma=1 and power=1, implements a simple average. gamma=1, power=2/3 are good values for models you plan to train for a million or more steps (reaches decay factor 0.999 at 31.6K steps, 0.9999 at 1M steps), gamma=1, power=3/4 for models you plan to train for less (reaches decay factor 0.999 at 10K steps, 0.9999 at 215.4k steps). zzPassing a `torch.nn.Module` to `ExponentialMovingAverage` is deprecated. Please pass the parameters of the module instead.z9passing a `torch.nn.Module` to `ExponentialMovingAverage`1.0.0F standard_warnT max_valueNzCThe `max_value` argument is deprecated. Please use `decay` instead. min_valuezGThe `min_value` argument is deprecated. Please use `min_decay` instead.r+z=The `device` argument is deprecated. Please use `to` instead.r*r)rzrrModulerr|getr{cloner^ shadow_paramsr.temp_stored_paramsrrrrrroptimization_stepcur_decay_valuerrr)selfr|rrrrrrrrrrdeprecation_messageps r$__init__zEMAModel.__init__sxF j%((// 2D  K##   $..0J"N ::k4 ( 4"g  k7,?u U;'E ::k4 ( 4"k  k7,?u U{+I*% :DE*Qaggi..0*E ::h % 1"a  h)> !ij j    $lm m**4+<+<=__& -   .:. U%%'( d#r&r)ctd||jz dz }|dkry|jr$dd||jz z|j zz }n d|zd|zz }t ||j }t||j}|S)zN Compute the decay factor for the exponential moving average. rr )maxrrrrminrr)r+r)stepr*s r$ get_decayzEMAModel.get_decayVs1'$*@*@@1DE 19   1tdnn'<#<$**"LLO 4xBI6Ootzz:ot~~>r&c t|tjjr!d}t dd|d|j }t |}|xjdz c_|j|j}||_ d|z }tj}|jrZtrItjj j#r!t j$j'|d}|5|Dcgc]}|j(s|}}t+|j,|Dcgc]\}}|j(s|} }}t/|t/|krgtj0t+|j,|Dcgc]\}}|j(r|c}}|Dcgc]}|j(r|c}d tj2| tj4| || dddyt+|j,|D]\}}trItjj j#r!t j$j'|d}|5|j(r|j7|||z zn|j9|dddycc}wcc}}wcc}}wcc}w#1swYyxYw#1swYxYw) NzPassing a `torch.nn.Module` to `ExponentialMovingAverage.step` is deprecated. Please pass the parameters of the module instead.z>passing a `torch.nn.Module` to `ExponentialMovingAverage.step`rFr r ) modifier_rankT) non_blocking)r7)rzrrr$rr|r{r)rDr* contextlib nullcontextrr transformers integrations deepspeedis_deepspeed_zero3_enabledzeroGatheredParametersr}rr'r0_foreach_copy_ _foreach_sub_ _foreach_subsub_copy_) r+r|r,rone_minus_decaycontext_managerr params_grads_param s_params_grads r$rCz EMAModel.stepisn j%((// 2D  P##   $..0J*%  !#t556$e)$002 <<(*|/H/H/R/R/m/m/o"+.."C"CJ^b"C"c 2<T*@S@Su* T25d6H6H*2U!2UY^YlYlG2U!{#c*o5((7:4;M;Mz7Zv7Z^Webgbubu7Zv,6RJ5e>Q>QJR%) ##!5#5#5m[#QYh!$#&d&8&8*"E,.<3L3L3V3V3q3q3s&/nn&G&G]a&G&bO$** _%%HI e, %_ #F#U! wR!,%_sgKJ:J:K9J?J?AKK )K -K4K K 5K96K:KKK% c t|}|jrytj|Dcgc]}|jc}t |j |Dcgc]*\}}|j|jj,c}}yt |j |D]C\}}|jj|j|jjEycc}wcc}}w)aa Copy current averaged parameters into given collection of parameters. Args: parameters: Iterable of `torch.nn.Parameter`; the parameters to be updated with the stored moving averages. If `None`, the parameters with which this `ExponentialMovingAverage` was initialized will be used. N) r{rrrPr~rr'r.r+rT)r+r|rrXs r$r<zEMAModel.copy_tos*% <<  )344EHI[I[]gEhiEh>7EELL)..Ehi  #&d&8&8*"E   ELL!9!>!>?#F 5is C./C3 ch|jDcgc]}|jc}|_ycc}w)z Move internal buffers of the ExponentialMovingAverage to pinned memory. Useful for non-blocking transfers for offloading EMA params to the host. N)r' pin_memory)r+r-s r$r\zEMAModel.pin_memorys, 7;6H6HI6Halln6HIIs/c|jDcgc]9}|jr|j|||n|j||;c}|_ycc}w)z Move internal buffers of the ExponentialMovingAverage to `device`. Args: device: like `device` argument to `torch.Tensor.to` )r+rrG)r+rGN)r'is_floating_pointr.)r+r+rrGr-s r$r.z EMAModel.tosf''  (""$ DDe,D GV,? @(   s>Ac |j|j|j|j|j|j |j |jdS)z Returns the state of the ExponentialMovingAverage as a dict. This method is used by accelerate during checkpointing to save the ema state dict. rrr)rrrrr'r`)r+s r$rozEMAModel.state_dictsNZZ!%!7!7!%!7!7"11ZZ!//  r&c|Dcgc].}|jjj0c}|_ycc}w)z Saves the current parameters for restoring later. Args: parameters: Iterable of `torch.nn.Parameter`. The parameters to be temporarily stored. N)r^rr&r()r+r|rs r$storezEMAModel.stores8NX"XZE5<<>#5#5#7#=#=#?Z"X"Xs3Ac|j td|jrXtj|Dcgc]}|j c}|jDcgc]}|j c}d|_yt |j|D]*\}}|j j|j ,d|_ycc}wcc}w)aG Restore the parameters stored with the `store` method. Useful to validate the model with EMA parameters without: affecting the original optimization process. Store the parameters before the `copy_to()` method. After validation (or model saving), use this to restore the former parameters. Args: parameters: Iterable of `torch.nn.Parameter`; the parameters to be updated with the stored parameters. If `None`, the parameters with which this `ExponentialMovingAverage` was initialized will be used. NzGThis ExponentialMovingAverage has no `store()`ed weights to `restore()`)r( RuntimeErrorrrrPr~rrT)r+r|rc_params r$restorezEMAModel.restores  " " *hi i <<  )344SWSjSj6kSjw||Sj6k #' #&d&=&=z"J   .#K#'56ks CC roctj|}|jd|j|_|jdks|jdkDr t d|jd|j |_t |j ts t d|jd|j|_t |jts t d|jd |j|_ t |jts t d |jd |j|_ t |jts t d |jd |j|_ t |jttfs t d|jd|j|_t |jttfs t d|jdd}|T||_t |jt s t dt#d|jDs t dyy)a Loads the ExponentialMovingAverage state. This method is used by accelerate during checkpointing to save the ema state dict. Args: state_dict (dict): EMA state. Should be an object returned from a call to :meth:`state_dict`. rr?r)zDecay must be between 0 and 1rzInvalid min_decayr)zInvalid optimization_steprzInvalid update_after_steprzInvalid use_ema_warmuprzInvalid inv_gammarz Invalid powerr'Nzshadow_params must be a listc3PK|]}t|tj ywr)rzrr)rr-s r$rz+EMAModel.load_state_dict../sO@4@&J   D &Y);); <YY'(588+=+=">'4'2.F$.F4.Fr&r)r))NNNrNr)TTTNN)SrHrirrrrrr functoolsrtypingrrnumpyrrrntorch.distributed.fsdprr r rtorch.distributed.fsdp.wrapr modelsr pipelinesr schedulersrutilsrrrrrrrrrJrKrLrMaccelerate.loggingrpeftr torchvisionrrrr%r:rrrOrr/tuplerhrqrwfloat32rr$r{rrrr+ Generatorrrrrkrrrrrsetr rrrr&r$rs %   5-&2CGH((&     **EEG-.&)3)("J)3)h(+3# 3##3#||3# << 3# << 3# LL 3#!<<3# %3# 5<< %&3#l3S%,,=N8O&PU}}-$uxx2G G-"]#u||+,]69]IN]&Dehhoo1E,F4PSUXPX>!&(,        LL3   % 6S$  nrUXX__/@@#PUP\P\J\gk>8 tttT4 ")-?C/ 88??/ %,, // /  / c3h$& /tEHHOO45</ /fLFLFr&