+ QV-j4X^RIt^RIt^RIHt^RIt^RIHtHt^RIHt^RI H t ^RI H t ] P!]4t]!R4!RR ]P 44t]t]!R 4!R R ]P 44t]!R 4!RR]P 44t]!R4!RR]P 44t]!R4!RR]P 44t]!R4!RR]P 44t!RR]P 4t!RR]P 4t!RR]P 4t!RR ]P 4t!R!R"]P 4t!R#R$]P 4t!R%R&]P 4t!R'R(]4t!R)R*]P 4t /R+]bR,]R-RIR.^ /3bR/]bR0]bR1]R2R3/3bR4]bR5]R6R3/3bR7]bR8]PBbR9]bR:]PDbR;]bR<]bR=]bR>]PFbR?]bR@]PHbRA]PJRB]RC]RD]PLRE]PNRF]PPRG] /Ct)]!])4t*RHt+]+!R14t,]+!R04t-]+!R+4t.]+!R/4t/]+!R44t0]+!R=4t1]+!RB4t2]+!R<4t3]+!R;4t4R#)JN) OrderedDict)Tensornn)use_kernel_forward_from_hub)logging)is_torchdynamo_compilingGeluTanhcjaa]tRt^toRtR V3RlV3RllltV3RlRltV3RlRltRtVt V;t #) GELUTanha A fast C implementation of the tanh approximation of the GeLU activation function. See https://huggingface.co/papers/1606.08415. This implementation is equivalent to NewGELU and FastGELU but much faster. However, it is not an exact numerical match due to rounding errors. c <V^8dQhRS[/#)use_gelu_tanh_pythonbool)format __classdict__s"i/Volumes/fast/ai/experiments/ui-tars-smoke/.venv/lib/python3.14/site-packages/transformers/activations.py __annotate__GELUTanh.__annotate__(sQQTQc<\SV`4V'dVPVnR#\P !\ PPRR7VnR#)tanh) approximateN) super__init___gelu_tanh_pythonact functoolspartialr functionalgelu)selfr __class__s&&rrGELUTanh.__init__(s<  --DH ((););PDHrc&<V^8dQhRS[RS[/#r inputreturnr)rrs"rrr/swwvw&wrc VR,R\P!\P!R\P, 4VR\P !VR4,,,4,,#??@Hm?g@torchrmathsqrtpipowr"r's&&rrGELUTanh._gelu_tanh_python/sPs{cEJJtyytww/G5S[^c^g^ghmor^sSsKs/t$uuvvrc&<V^8dQhRS[RS[/#r&r))rrs"rrr2Vrc$VPV4#Nrr6s&&rforwardGELUTanh.forward2xxrr<F) __name__ __module__ __qualname____firstlineno____doc__rrr=__static_attributes____classdictcell__ __classcell__r#rs@@rr r s1QQwwrr NewGELUc6a]tRt^:toRtV3RlRltRtVtR#)NewGELUActivationz Implementation of the GELU activation function currently in Google BERT repo (identical to OpenAI GPT). Also see the Gaussian Error Linear Units paper: https://huggingface.co/papers/1606.08415 c&<V^8dQhRS[RS[/#r&r))rrs"rrNewGELUActivation.__annotate__AswwVwwrc RV,R\P!\P!R\P, 4VR\P !VR4,,,4,,#r+r0r6s&&rr=NewGELUActivation.forwardAsPU{cEJJtyytww/G5S[^c^g^ghmor^sSsKs/t$uuvvrNrArBrCrDrEr=rFrGrs@rrLrL:s wwrrLGeLUcjaa]tRt^EtoRtR V3RlV3RllltV3RlRltV3RlRltRtVt V;t #) GELUActivationa Original Implementation of the GELU activation function in Google BERT repo when initially created. For information: OpenAI GPT's GELU is slightly different (and gives slightly different results): 0.5 * x * (1 + torch.tanh(math.sqrt(2 / math.pi) * (x + 0.044715 * torch.pow(x, 3)))) This is now written in C in nn.functional Also see the Gaussian Error Linear Units paper: https://huggingface.co/papers/1606.08415 c <V^8dQhRS[/#)r use_gelu_pythonr)rrs"rrGELUActivation.__annotate__Ns***rc<\SV`4V'dVPVnR#\P P VnR#r;)rr _gelu_pythonrrr r!)r"rXr#s&&rrGELUActivation.__init__Ns/  ((DH}}))DHrc&<V^8dQhRS[RS[/#r&r))rrs"rrrYUsGG&GVGrcVR,R\P!V\P!R4, 4,,#)r,r-r.)r1erfr2r3r6s&&rr[GELUActivation._gelu_pythonUs,s{cEIIediin.D$EEFFrc&<V^8dQhRS[RS[/#r&r))rrs"rrrYXr9rc$VPV4#r;r<r6s&&rr=GELUActivation.forwardXr?rr<r@) rArBrCrDrErr[r=rFrGrHrIs@@rrVrVEs/**GGrrVSiLUc6a]tRt^\toRtV3RlRltRtVtR#)SiLUActivationa See Gaussian Error Linear Units (Hendrycks et al., https://arxiv.org/abs/1606.08415) where the SiLU (Sigmoid Linear Unit) was originally introduced and coined, and see Sigmoid-Weighted Linear Units for Neural Network Function Approximation in Reinforcement Learning (Elfwing et al., https://arxiv.org/abs/1702.03118) and Swish: a Self-Gated Activation Function (Ramachandran et al., https://arxiv.org/abs/1710.05941v1) where the SiLU was experimented with later. c&<V^8dQhRS[RS[/#r&r))rrs"rrSiLUActivation.__annotate__fs))V))rc@\PPV4#r;)rr silur6s&&rr=SiLUActivation.forwardfs}}!!%((rrQNrRrSs@rrfrf\s))rrfFastGELUc6a]tRt^jtoRtV3RlRltRtVtR#)FastGELUActivationzu Applies GELU approximation that is slower than QuickGELU but more accurate. See: https://github.com/hendrycks/GELUs c&<V^8dQhRS[RS[/#r&r))rrs"rrFastGELUActivation.__annotate__psiiViircRV,R\P!VR,RRV,V,,,4,,#)r,r-g3E?r/)r1rr6s&&rr=FastGELUActivation.forwardps:U{cEJJu|/CsXX]M]`eMeGe/f$gghhrrQNrRrSs@rrnrnjsiirrn QuickGELUc6a]tRt^ttoRtV3RlRltRtVtR#)QuickGELUActivationzj Applies GELU approximation that is fast but somewhat inaccurate. See: https://github.com/hendrycks/GELUs c&<V^8dQhRS[RS[/#r&r))rrs"rr QuickGELUActivation.__annotate__zs44V44rcJV\P!RV,4,#)gZd;?)r1sigmoidr6s&&rr=QuickGELUActivation.forwardzsu}}UU]333rrQNrRrSs@rruruts44rrucTaa]tRt^~toRtV3RlV3RlltV3RlRltRtVtV;t #)ClippedGELUActivationar Clip the range of possible GeLU outputs between [min, max]. This is especially useful for quantization purpose, as it allows mapping negatives values in the GeLU spectrum. For more information on this trick, please refer to https://huggingface.co/papers/2004.09602. Gaussian Error Linear Unit. Original Implementation of the gelu activation function in Google Bert repo when initially created. For information: OpenAI GPT's gelu is slightly different (and gives slightly different results): 0.5 * x * (1 + torch.tanh(math.sqrt(2 / math.pi) * (x + 0.044715 * torch.pow(x, 3)))). See https://huggingface.co/papers/1606.08415 c&<V^8dQhRS[RS[/#)r minmax)float)rrs"rr"ClippedGELUActivation.__annotate__sErcl<W8d\RV RV R24h\SV` 4WnW nR#)zmin should be < max (got min: z, max: )N) ValueErrorrrr~r)r"r~rr#s&&&rrClippedGELUActivation.__init__s8 9=cU'#aPQ Q rc&<V^8dQhRS[RS[/#r xr(r))rrs"rrrs777F7rcl\P!\V4VPVP4#r;)r1clipr!r~r)r"rs&&rr=ClippedGELUActivation.forwards!zz$q'488TXX66r)rr~ rArBrCrDrErr=rFrGrHrIs@@rr|r|~s# 777rr|cHaa]tRt^toRtV3RltV3RlRltRtVtV;t #)AccurateGELUActivationz Applies GELU approximation that is faster than default and more accurate than QuickGELU. See: https://github.com/hendrycks/GELUs Implemented along with MEGA (Moving Average Equipped Gated Attention) c<\SV`4\P!^\P, 4VnR#)r N)rrr2r3r4precomputed_constantr"r#s&rrAccurateGELUActivation.__init__s' $(IIa$''k$:!rc&<V^8dQhRS[RS[/#r&r))rrs"rr#AccurateGELUActivation.__annotate__sttVttrc RV,^\P!VPVR\P!V^4,,,4,,#)r,r/)r1rrr5r6s&&rr=AccurateGELUActivation.forwardsEU{a%**T-F-F%RZ]b]f]fglno]pRpJp-q"rrssr)rrrIs@@rrrs!;tttrrcZaa]tRt^toRtV3RltV3RlRltV3RlRltRtVt V;t #)MishActivationz See Mish: A Self-Regularized Non-Monotonic Activation Function (Misra., https://huggingface.co/papers/1908.08681). Also visit the official repository for the paper: https://github.com/digantamisra98/Mish cb<\SV`4\PPVnR#r;)rrrr mishrrs&rrMishActivation.__init__s ==%%rc&<V^8dQhRS[RS[/#r&r))rrs"rrMishActivation.__annotate__sAA&AVArcvV\P!\PP V44,#r;)r1rrr softplusr6s&&r _mish_pythonMishActivation._mish_pythons%uzz"--"8"8"?@@@rc&<V^8dQhRS[RS[/#r&r))rrs"rrrr9rc$VPV4#r;r<r6s&&rr=MishActivation.forwardr?rr<) rArBrCrDrErrr=rFrGrHrIs@@rrrs* &AArrc6a]tRt^toRtV3RlRltRtVtR#)LinearActivationzS Applies the linear activation function, i.e. forwarding input directly to output. c&<V^8dQhRS[RS[/#r&r))rrs"rrLinearActivation.__annotate__sVrcV#r;rQr6s&&rr=LinearActivation.forwards rrQNrRrSs@rrrsrrc.a]tRt^toRtRRltRtVtR#)LaplaceActivationz Applies elementwise activation based on Laplace function, introduced in MEGA as an attention activation. See https://huggingface.co/papers/2209.10655 Inspired by squared relu, but with bounded range and gradient for better stability cW, PV\P!R4,4pRR\P!V4,,#)r.r,r-)divr2r3r1r_)r"r'musigmas&&&&rr=LaplaceActivation.forwards:  3!78cEIIe,,--rrQN)g۞?g ^/ ?rRrSs@rrrs..rrc*a]tRt^toRtRtRtVtR#)ReLUSquaredActivationzV Applies the relu^2 activation introduced in https://huggingface.co/papers/2109.08668 cp\PPV4p\P!V4pV#r;)rr relur1square)r"r' relu_appliedsquareds&& rr=ReLUSquaredActivation.forwards)}}))%0 ,,|,rrQNrRrSs@rrrsrrc*a]tRt^toRtRtRtVtR#)SqrtSoftplusActivationuFsqrt(softplus(x)) — the router scoring function used by DeepSeek V4.c\\PPV4P4#r;)rr rr3r6s&&rr=SqrtSoftplusActivation.forwards }}%%e,1133rrQNrRrSs@rrrsP44rrc2aa]tRt^toV3RltRtVtV;t#)ClassInstantiercn<\SV`V4p\V\4'dTMV/3wr4V!R/VB#)NrQ)r __getitem__ isinstancetuple)r"keycontentclskwargsr#s&& rrClassInstantier.__getitem__s7'%c*!+GU!;!;g'2 }V}rrQ)rArBrCrDrrFrGrHrIs@@rrrsrrcaa]tRt^toRtRRRR ]P R3V3RlltV3RlRltV3RlR lt V3R lR lt R t Vt V;t #)XIELUActivationz Applies the xIELU activation function introduced in https://arxiv.org/abs/2411.13010 If the user has installed the nickjbrowning/XIELU wheel, we import xIELU CUDA Otherwise, we emit a single warning and use xIELU Python g?r,Fc t<\S V`4\P!\P !\P !\P!WR744P^44Vn \P!\P !\P !\P!W#, VR744P^44Vn VPR\P!W5R74VPR\P!WER74W`n \V4Vn\V4VnRVn^RIp\P$P&P)4VnRp^RIHp V !VP.4VnVR, p\4P7V4R# \2d)p TRT R2, pTP.TnRp ? LERp ? ii;i \2d%p \4P7R T R 24Rp ? R#Rp ? ii;i) )dtypebetaepsNzUsing experimental xIELU CUDA.)allow_in_graphz& Enabled torch._dynamo for xIELU CUDA.z+ Could not enable torch._dynamo for xIELU (z*) - this may result in slower performance.z CUDA-fused xIELU not available (u) – falling back to a Python version. For CUDA xIELU (experimental), `pip install git+https://github.com/nickjbrowning/XIELU`)rrr Parameterr1logexpm1tensor unsqueezealpha_palpha_nregister_bufferwith_vector_loadsr _beta_scalar _eps_scalar_xielu_cuda_obj xielu.opsclassesxieluXIELUtorch.compilerr _xielu_cuda_xielu_cuda_fn Exceptionlogger warning_once) r" alpha_p_init alpha_n_initrrrrrmsgrerrr#s &&&&&&& rrXIELUActivation.__init__s ||EIIekk%,,|:a.b$c$m$mno$pq || IIekk%,,|/B%"PQ R \ \]^ _   VU\\$%DE UELL$BC!2!$K :#  #(==#6#6#<#<#>D 2C 79&4T5E5E&F#??    $ 7DSEIstt&*&6&6## 7    23%8jj   sB!3H&G;H HH;HHH H7H22H7c&<V^8dQhRS[RS[/#rr))rrs"rrXIELUActivation.__annotate__s  v & rc \PPVP4pVP\PPVP 4,p\ P!V^8W!,V,VPV,,\ P!\ P!WP44V, V,VPV,,4#)r) rr rrrrr1whererr~r)r"rrrs&& r _xielu_pythonXIELUActivation._xielu_pythons--((6))bmm44T\\BB{{ E K!Odii!m + [[1hh/ 01 4 ?$))a- O  rc&<V^8dQhRS[RS[/#rr))rrs"rrr"s++V++rc dVPpVP4^8dVP^4pK(VP4^8d#VPR^VP R44pW!P8wd"\ P RVVP4VPPVVPPVP4VPPVP4VPVPVP4pVPV4#)zDFirewall function to prevent torch.compile from seeing .item() callsz_Warning: xIELU input tensor expects 3 dimensions but got (shape: %s). Reshaping to (shape: %s).)shapedimrviewsizerrrr=rtorrrrr)r"roriginal_shaperesults&& rrXIELUActivation._xielu_cuda"seegk AA 557Q;r1affRj)A WW $   q  %%-- LLOOAGG $ LLOOAGG $        " " {{>**rc&<V^8dQhRS[RS[/#r&r))rrs"rrr;s))V))rcVPeIVP'd7\4'gVPV4#\P R4VP V4#)Nz:torch._dynamo is compiling, using Python version of xIELU.)ris_cudarrrrrr6s&&rr=XIELUActivation.forward;sP    + +--**511##$`a!!%((r)rrrrrrrgư)rArBrCrDrEr1bfloat16rrrr=rFrGrHrIs@@rrrsK  nn(T  ++2)))rrr!gelu_10r~r gelu_fastgelu_new gelu_pythonrXTgelu_pytorch_tanhgelu_python_tanhr gelu_accurate hardswishlaplace leaky_relulinearr quick_gelurrelu2relu6ryrj sqrtsoftplusswishrprelurc V\9d\V,#\RV R\\P44 24h)z function z not found in ACT2FN mapping )ACT2FNKeyErrorlistkeys)activation_strings&rget_activationrasBF"'((#4"55RSWX^XcXcXeSfRghiiri)5rr2 collectionsrr1rrintegrations.hub_kernelsrutilsrutils.import_utilsr get_loggerrArModuler PytorchGELUTanhrLrVrfrnrur|rrrrrrrr Hardswish LeakyReLUReLUReLU6SigmoidrdTanhPReLUACT2CLSrrrrr!rrr rjr linear_actrQrrr)s] # A8   H %Z(ryy)0Y'w w(wV$RYY%,V$ )RYY )% )Z(ii)i[)4"))4*47BII72 tRYY t RYY"ryy . .BII4RYY4kZ)biiZ)z  N  %sE2'>? # !   N%6$=>     $:D#AB +    ",,    N %  BGG   "! " RXX# $rzz N* RWW BGG RXX _1 4  !j]+ * %f ; ' "#67 L ) ff H % r