(tioddlZddlmcmZddlmZddlmZddlmZm Z erddl Z ejejejejejdZdedej"fd ZGd d ej"ZGd d ej"Z Gddej"ZGddej"ZGddej"ZGddej"Zy)N)nn) deprecate)is_torch_npu_availableis_torch_version)swishsilumishgelureluact_fnreturnc |j}|tvr t|Std|dttj )zHelper function to get activation function from string. Args: act_fn (str): Name of activation function. Returns: nn.Module: Activation function. zactivation function z not found in ACT2FN mapping )lowerACT2CLS ValueErrorlistkeys)r s k/Volumes/fast/ai/experiments/MLX_z-image/.venv/lib/python3.12/site-packages/diffusers/models/activations.pyget_activationr$sO\\^F v  /x7TUYZaZfZfZhUiTjkllcZeZdZdZfdZdej dej fdZxZS)FP32SiLUzH SiLU activation function with input upcasted to torch.float32. c"t|yN)super__init__)self __class__s rrzFP32SiLU.__init__:s rinputsrc~tj|jdj|jS)NF)inplace)Fr floattodtype)rr s rforwardzFP32SiLU.forward=s(vvfllne477 EEr) __name__ __module__ __qualname____doc__rtorchTensorr' __classcell__rs@rrr5s+FellFu||Frrc teZdZdZd dedededeffd Zdejdejfd Z d Z xZ S) GELUa GELU activation function with tanh approximation support with `approximate="tanh"`. Parameters: dim_in (`int`): The number of channels in the input. dim_out (`int`): The number of channels in the output. approximate (`str`, *optional*, defaults to `"none"`): If `"tanh"`, use tanh approximation. bias (`bool`, defaults to True): Whether to use a bias in the linear layer. dim_indim_out approximatebiascjt|tj||||_||_yNr5)rrrLinearprojr4)rr2r3r4r5rs rrz GELU.__init__Ls* IIfgD9 &rgaterc@|jjdk(retddrYtj|j t j|jj |jStj||jS)Nmps<2.0.0r&)r4) devicetyperr#r r%r,float32r4r&rr;s rr z GELU.geluQsp ;;  u $)9#w)G66$'' '6DDTDTUXX_c_i_iXj jvvd(8(899rcJ|j|}|j|}|Sr)r:r r hidden_statess rr'z GELU.forwardWs$ -0  -0 r)noneT) r(r)r*r+intstrboolrr,r-r r'r.r/s@rr1r1AsJ's'S's'SW' ::%,,: rr1cpeZdZdZd dededeffd ZdejdejfdZ d Z xZ S) GEGLUaN A [variant](https://huggingface.co/papers/2002.05202) of the gated linear unit activation function. Parameters: dim_in (`int`): The number of channels in the input. dim_out (`int`): The number of channels in the output. bias (`bool`, defaults to True): Whether to use a bias in the linear layer. r2r3r5cbt|tj||dz||_yNrr8rrrr9r:rr2r3r5rs rrzGEGLU.__init__gs& IIfgk= rr;rc|jjdk(rYtddrMtj|j t jj |jStj|S)Nr=r>r?r@) rArBrr#r r%r,rCr&rDs rr z GEGLU.geluks\ ;;  u $)9#w)G66$'' '67:::L Lvvd|rc t|dkDs|jddd}tdd||j|}t rt j |dddS|jdd \}}||j|zS) NrscalezThe `scale` argument is deprecated and will be ignored. Please remove it, as passing it will raise an error in the future. `scale` should directly be passed while calling the underlying pipeline component i.e., via `cross_attention_kwargs`.z1.0.0)dimr4rrW) lengetrr:r torch_npu npu_gegluchunkr )rrGargskwargsdeprecation_messager;s rr'z GEGLU.forwardqs t9q=FJJw5A#U  gw(; < -0 ! #&&}"!LQO O"/"5"5aR"5"@ M4 499T?2 2rT) r(r)r*r+rIrKrr,r-r r'r.r/s@rrMrM]sB>s>S>>%,, 3rrMc8eZdZdZddededeffd ZdZxZS)SwiGLUa A [variant](https://huggingface.co/papers/2002.05202) of the gated linear unit activation function. It's similar to `GEGLU` but uses SiLU / Swish instead of GeLU. Parameters: dim_in (`int`): The number of channels in the input. dim_out (`int`): The number of channels in the output. bias (`bool`, defaults to True): Whether to use a bias in the linear layer. r2r3r5ct|tj||dz||_tj |_yrO)rrrr9r:SiLU activationrQs rrzSwiGLU.__init__s3 IIfgk= '')rcx|j|}|jdd\}}||j|zS)NrrUrX)r:r]rf)rrGr;s rr'zSwiGLU.forwards> -0 +11!1< ttt444rra) r(r)r*r+rIrKrr'r.r/s@rrcrc~s'$s$S$$ 5rrccjeZdZdZd dededeffd ZdejdejfdZ xZ S) ApproximateGELUa The approximate form of the Gaussian Error Linear Unit (GELU). For more details, see section 2 of this [paper](https://huggingface.co/papers/1606.08415). Parameters: dim_in (`int`): The number of channels in the input. dim_out (`int`): The number of channels in the output. bias (`bool`, defaults to True): Whether to use a bias in the linear layer. r2r3r5c\t|tj||||_yr7rPrQs rrzApproximateGELU.__init__s" IIfgD9 rxrcZ|j|}|tjd|zzS)NgZd;?)r:r,sigmoid)rrks rr'zApproximateGELU.forwards' IIaL5==+++rra) r(r)r*r+rIrKrr,r-r'r.r/s@rriris=:s:S::,,%,,,rric 8eZdZddedededeffd ZdZxZS)LinearActivationr2r3r5rfc|t|tj||||_t ||_yr7)rrrr9r:rrf)rr2r3r5rfrs rrzLinearActivation.__init__s. IIfgD9 (4rcF|j|}|j|Sr)r:rfrFs rr'zLinearActivation.forwards -0 }--r)Tr ) r(r)r*rIrKrJrr'r.r/s@rroros*5s5S55QT5 .rro)r,torch.nn.functionalr functionalr#utilsrutils.import_utilsrrr[reMishr1ReLUrrJModulerrrMrcrirorrrzs  IWW GG GG GG GG  m3m299m" Fryy F29983BII3B5RYY5.,bii,( .ryy .r