(ti> dZddlZddlZddlZddlmZddlmZddlm Z m Z m Z ddl m Z ejeZe rddlZdZd Zdd edzfd Z dd Zd ZddZdZdZddefdZdeddfdZ ddZdZdZ y)z3 PEFT utilities: Utilities related to peft library N)version)logging)is_peft_availableis_peft_versionis_torch_available)empty_device_cachec ddlm}d}|jD]}t||st |d}n|rlddlm}|jDcgc] \}}d|vs |}}}|D]8} |||\}} } t | dst|| | j:|Sddl m } |jD]\} }tt|j!dkDr t#|d} t|| rt|t$j&j(rt$j&j)|j*|j,|j.du j1|j2j4}|j2|_|j.|j.|_d } nt|| rt|t$j&j6rt$j&j7|j8|j:|j<|j>|j@|jB|jDj1|j2j4}|j2|_|j.|j.|_d } | st|| ~tG |Scc}}w#t$rYswxYw) zd Recursively replace all instances of `LoraLayer` with corresponding new layers in `model`. rBaseTunerLayerF base_layer)_get_submoduleslora) LoraLayerN)biasT)$peft.tuners.tuners_utilsr modules isinstancehasattr peft.utilsr named_modulesAttributeErrorsetattrget_base_layerpeft.tuners.lorarnamed_childrenlenlistchildrenrecurse_remove_peft_layerstorchnnLinear in_features out_featuresrtoweightdeviceConv2d in_channels out_channels kernel_sizestridepaddingdilationgroupsr )modelr has_base_layer_patternmodulerkey_key_listparenttarget target_namernamemodule_replaced new_modules i/Volumes/fast/ai/experiments/MLX_z-image/.venv/lib/python3.12/site-packages/diffusers/utils/peft_utils.pyr r #si8"--/ fn -%,V\%B " " .&+&9&9&;Q&;FCvS?PC&;QC .=eS.I+ v|, V-B-B-DE h LU /!002LD&4)*+a/*62#O&),FEHHOO1T"XX__&&''D0-"V]]))*  %+MM !;;*&,kkJO"&FI.:fehhoo3V"XX__&&''&&MMNNOOMM"V]]))*%+MM !;;*&,kkJO"&tZ0"$O3P LkR"  s K  K - K KKcddlm}|dk(ry|jD] }t||s|j |"y)z Adjust the weightage given to the LoRA layers of the model. Args: model (`torch.nn.Module`): The model to scale. weight (`float`): The weight to be given to the LoRA layers. rr ?N)rr rr scale_layer)r1r'r r3s r=scale_lora_layersrAjs98 }--/ fn -   v &"r'cddlm}||dk(ry|jD]I}t||s|dk7r|j |'|j D]}|j |dKy)a Removes the previously passed weight given to the LoRA layers of the model. Args: model (`torch.nn.Module`): The model to scale. weight (`float`, *optional*): The weight to be given to the LoRA layers. If no scale is passed the scale of the lora layer will be re-initialized to the correct value. If 0.0 is passed, we will re-initialize the scale with the correct value. rr Nr?)rr rr unscale_layeractive_adapters set_scale)r1r'r r3 adapter_names r=unscale_lora_layersrH~sd8 ~3--/ fn -{$$V,$*$:$:L$$\37%; "rBc i}i}t|jdxtt|jdkDrt j |jj ddttfd|j}|jD cic]\}} |jdd| }}} |vt|dkDrgtt|jdkDrt j |jj ddttfd|j}|rd|jD cic]H\}} dj|jddjdjdd | J}}} n~|jD cic];\}} dj|jd djddd | =}}} n't|jjt|jD chc]} | jd dc} } td |D} td|D} ||| | | d}|Scc} }wcc} }wcc} }wcc} w)Nrrc|dk7SNr)xrs r=z!get_peft_kwargs..s QqTQYrBz.lora_B.c|dk7SrKrL)rM lora_alphas r=rOz!get_peft_kwargs..s!A$*2DrB.z.lora_A.z.alphaz.down.z.lorac3$K|]}d|v yw)lora_magnitude_vectorNrL.0ks r= z"get_peft_kwargs..sIA*a/sc3JK|]}d|vxr|jdyw)lora_Bz.biasN)endswithrWs r=rZz"get_peft_kwargs..s&S?aHM9ajj&99?s!#)rNrQ rank_pattern alpha_patterntarget_modulesuse_dora lora_bias)rvaluesrset collectionsCounter most_commondictfilteritemssplitjoinreplacepopkeysany) rank_dictnetwork_alpha_dictpeft_state_dictis_unetmodel_state_dictrGr^r_rYvr:r`rarblora_config_kwargsrQrNs @@r=get_peft_kwargsrxsLM)**,-a00A  3y! "#a'    0 0 2 3 ? ? A! DQ GF#6 8IJK >J>P>P>RS>Rda +A.1>R S%#.@*AA*E s%,,./ 01 4$,,-?-F-F-HIUUWXYZ[\]J!(DFXF^F^F`!abM!. 3 3 5! 51HHQWWZ0399#>?GGRTUWXX 5! `m_r_r_t u_tW[WXZ[!''(*;A*>*D*DS*I#2*N!OQR!R_t u/6689==?Jo>R>R>TU>Td4::g.q1>TUVNIIIHS?SSI $&( ET! !vVs:J>A K*AK ,Kcddlm}|jD](}t||sdt |j cSy)Nrr default_ default_0)rr rrrrN)r1r r3s r=get_adapter_namer|s97--/ fn -c&((m_- -" rBcddlm}|jD]6}t||st |dr|j |/| |_8y)Nrr enable_adapters)enabled)rr rrrr~disable_adapters)r1rr r3s r=set_adapter_layersrsE7--/ fn -v01&&w&7.5+' "rBcVddlm}|jD]7}t||st |dr|j |.t dt|ddrLt |dr?|jj|dt|jdk(r |`d|_ yyyy)Nrr delete_adapterzdThe version of PEFT you are using is not compatible, please use a version that is greater than 0.6.1_hf_peft_config_loadedF peft_config) rr rrrr ValueErrorgetattrrrnrr)r1rGr r3s r=delete_adapter_layersrs7--/ fn -v/0%%l3 z "u.675-;X lD1 u  !Q &!+/E ( ' .get_module_weights,d3% %#5#;#;#= J[($>!!#&q !E!H:\%(<)--c37 rB set_adapter) rr rrrractive_adapterziprF) r1 adapter_namesweightsr rrr3rGr's r=!set_weights_and_activate_adaptersrs7 %224 V fn -v}-""=1(5%),M7(C$ f  /@/UV)D 5rB kwargs_namecfd}|S)a( Decorator to automatically handle LoRA layer scaling/unscaling in forward methods. This decorator extracts the `lora_scale` from the specified kwargs parameter, applies scaling before the forward pass, and ensures unscaling happens after, even if an exception occurs. Args: kwargs_name (`str`, defaults to `"joint_attention_kwargs"`): The name of the keyword argument that contains the LoRA scale. Common values include "joint_attention_kwargs", "attention_kwargs", "cross_attention_kwargs", etc. cFtjfd}|S)NcHddlm}d}|j}|G|j}||<|j dd}|s|dk7rt j dd|r t|| |g|i|}||r t||SS#|r t||wwxYw)Nr)USE_PEFT_BACKENDr?scalezPassing `scale` via `z1` when not using the PEFT backend is ineffective.) rSrrcopyrnloggerwarningrArH) selfargskwargsr lora_scaleattention_kwargsresult forward_fnrs r=wrapperz4apply_lora_scale..decorator..wrapper$s *J%zz+6 +#3#8#8#: &6{#-11'3? 'J#,=NN/ }.decorator#s%  $ : % ::rBrL)rrs` r=apply_lora_scalersB rB min_versionreturncts tdtjtj jdtj|kD}|std|y)z Checks if the version of PEFT is compatible. Args: version (`str`): The version of PEFT to check against. z@PEFT is not installed. Please install it with `pip install peft`peftz_The version of PEFT you are using is not compatible, please use a version that is greater than N)rrrparse importlibmetadata)ris_peft_version_compatibles r=check_peft_versionrGsl  [\\!(y/A/A/I/I&/Q!RU\UbUbcnUo!o % M #   &rBcddlm}||}nt||||||}t|d|vr|drt ddr t dd|vr|drt d d r t d  |d i|S#t $r} t d | d} ~ wwxYw)Nr) LoraConfig)rrrsrtrurGra= 0.9.0. Please upgrade.rbz<=z0.13.2z2lora_bias requires PEFT >= 0.14.0. Please upgrade.z-`LoraConfig` class could not be instantiated.rL)rrrx%_maybe_raise_error_for_ambiguous_keysrr TypeError) state_dictnetwork_alphasrrank_pattern_dictrtrurGrrwes r=_create_lora_configr[s %, -&-%  **<='',>z,J 3 (KL L((-? -L 4 *QR RP/.// PGHaOPs)A11 B : BB c|dj}|d}t|jD]N}|Dcgc] }||k(s | }}|Dcgc]}||vs||k7s|}}|s5|s8tddsEt dycc}wcc}w)Nr^r`rz0.14.1zzThere are ambiguous keys present in this LoRA. To load it, please update your `peft` installation - `pip install -U peft`.)rrrorr)configr^r`r4mod exact_matchessubstring_matchess r=rr}s.)..0L,-NL%%'( )7E#* E,:XNScSjSTWZSNX .sH- Q) FXs BB B B B cRd}|t|dd}|r0|Dcgc]}d|vs||vs |}}|rddj|d}t|dd}|r3|Dcgc]}d|vs||vs |}}|r|ddj|d z }|rtj|yycc}wcc}w) NrSunexpected_keyslora_zSLoading adapter weights from state_dict led to unexpected keys found in the model: z, z. missing_keyszJLoading adapter weights from state_dict led to missing keys in the model: rR)rrlrr)incompatible_keysrGwarn_msgrrYlora_unexpected_keysrlora_missing_keyss r=_maybe_warn_for_unhandled_keysrsH$!"35FM />#e!'Q,S_cdSdA #e# "678< 0.$G ,8 _LqGqL\]^M^L  _  "345Q8 x #$f!`s! BBB B$!B$&B$)N)TNN)T)joint_attention_kwargs)!__doc__rerr packagingrrSr import_utilsrrr torch_utilsr get_logger__name__rr!r rAfloatrHrxr|rrrstrrrrrrrLrBr=rsPP+   H %DN'(8ut|88gk/d 60,W@.#.b C D *ptPD(!rB