+ QV-j aR t0t^RIt^RIt^RIHt^RIHt^RIHtH t H t ^RI H t H t ] P!]4t] !4'd^RIt]'d^RIHtRtR!RR lltR"R R lltR!R R lltR#RRlltR#RRlltR#RRlltR]R]R]R]R]R]/t]^k!RR] 4t!RR4tR$RRlltR#)%N)Callablewraps) TYPE_CHECKINGOptional TypedDict)is_torch_availablelogging)PreTrainedConfigcPaaaRRloRRlo\S4RVVV3Rll4pV#)aD Decorator function to update the RoPE parameters in the forward pass, if the model is using a dynamic RoPE (i.e. a RoPE implementation that may recompute its frequencies in the forward pass). Args: rope_forward (Callable): The forward pass of the RoPE implementation. Returns: The decorated forward pass. c\P!V4^,pVf9VPpVPpRpVPP R,pMJVPV,p\ W R24pV R2pVPP V,R,pWH8di\W R24'g-\V,p V !VPVV^,VR7wrVPV R2X R R 7\W R 2V 4R#VPV4pVPV R2VR R 7\W R 2V4R#) zbLongrope uses long factor if sequence is larger than original pretraining length, short otherwise.N original_max_position_embeddings_original_inv_freq__long_inv_freqseq_len layer_typeinv_freqF persistent long_inv_freqoriginal_inv_freq) torchmax rope_typerconfigrope_parametersgetattrhasattrROPE_INIT_FUNCTIONSregister_buffersetattrto) self position_idsdevicerrrrprefixr rope_init_fnrrs &&&& q/Volumes/fast/ai/experiments/ui-tars-smoke/.venv/lib/python3.14/site-packages/transformers/modeling_rope_utils.pylongrope_frequency_update6dynamic_rope_update..longrope_frequency_update/sO))L)A-  I $ 6 6 F/3{{/J/JKm/n ,z2I ' ??29= #/KK.dynamic_frequency_updateRsi ))L)A-  I!%!8!8  $ 6 6 Fz2I!(=P/QSWSjSj!k  ' W>W X 4b .r.c<Vf VPMVPV,pVeRV/M/pRV9dS!W3RVP/VBMVR8XdS!W3RVP/VBS!WV3/VB#)Nrdynamicr(longrope)rr() r&xr'rrkwargsr4r, rope_forwards &&&& r+wrapper$dynamic_rope_update..wrapperxs{&0&8DNNdnnZ>X /9/E, +2  ! $T S SF S * $ %d T TV TD\5 ! 3&r.c VP4VeVPV,M VPpVR,pVR,pVPRR4p\VRR4;'gVPVP ,p\ W,4p Rp RV\P!^V ^\PR7PV\PR7V , ,, p W,p W3#) a Computes the inverse frequencies with linear scaling. Credits to the Reddit user /u/kaiokendev Args: config ([`~transformers."PreTrainedConfig"`]): The model configuration. This function assumes that the config will provide at least the following properties: * rope_theta (`float`, *optional*): The base wavelength from which the inverse frequencies will be derived. Defaults to `config.default_theta` if omitted. * hidden_size (`int`): The numerator when deriving a head_dim, if not provided directly. * num_attention_heads (`int`): The denominator when deriving a head_dim, if not provided directly. Additionally, this function will make use of the following properties if they are found in the config: * head_dim (`int`, *optional*): The size of the key-value heads in the model. If None, this value will be derived as hidden_size // num_attention_heads. * partial_rotary_factor (`float`, *optional*): If less than 1.0, inverse frequencies will be returned for the first fraction of the head_dim. Defaults to 1.0. device (`torch.device`): The device to use for initialization of the inverse frequencies. seq_len (`int`, *optional*): The current sequence length. Unused for this type of RoPE. Returns: Tuple of (`torch.Tensor`, `float`), containing the inverse frequencies for the RoPE embeddings and the post-processing scaling factor applied to the computed cos/sin (unused in this type of RoPE). Nfactor rope_thetapartial_rotary_factor?head_dimdtyper(rT) standardize_rope_paramsrgetr hidden_sizenum_attention_headsrGrarangeint64r%rJ) rr(rrrope_parameters_dictrNbaserPrRdimattention_factorrs &&&& r+'_compute_linear_scaling_rope_parametersr`sB ""$AKAW611*=]c]s]s !( +F -D0445LcRvz40ddF4F4F&JdJd4dH h. /Cdu||AsAU[[ILLTZbgbmbmLnqttuvH  H  %%r.cV^8dQhR\R,R\R,R\R,R\R,R\R \R \3,/#) rBrr r(rCrNr head_dim_keyrDrErF)rKs"r+rLrLsiC&C& ' (C& ^ $C&4ZC&d C&  C&  >5 ! C&r.c VP4VeVPV,M VPp\WR4;'gVPVP,pVR,pVP RR4pVP RR4p Rp \ W,^,4p RV\P!^^V ,^\PR7PV\PR7V, ,, p V^,V , p V ^8dA\P!V \P!V \PVR73^R 7pMT pW,pW3#) a Computes the inverse frequencies with proportional RoPE. Args: config ([`~transformers.PretrainedConfig`]): The model configuration. This function assumes that the config will provide at least the following properties: * rope_theta (`float`, *optional*): The base wavelength from which the inverse frequencies will be derived. Defaults to `config.default_theta` if omitted. * hidden_size (`int`): The numerator when deriving a head_dim, if not provided directly. * num_attention_heads (`int`): The denominator when deriving a head_dim, if not provided directly. Additionally, this function will make use of the following properties if they are found in the config: * head_dim (`int`, *optional*): The size of the key-value heads in the model. If None, this value will be derived as hidden_size // num_attention_heads. * partial_rotary_factor (`float`, *optional*, defaults to 1.0): The proportion of the embedding dimension to apply rotary positional encoding, e.g., [0.0, 0.25, 0.5, 0.75, 1.0]. Unlike other RoPE functions that use this parameter, proportional RoPE will always return an encoding that is the size of `head_dim`. device (`torch.device`): The device to use for initialization of the inverse frequencies. seq_len (`int`, *optional*): The current sequence length. Unused for this type of RoPE. Returns: Tuple of (`torch.Tensor`, `float`), containing the inverse frequencies for the RoPE embeddings and the post-processing scaling factor applied to the computed cos/sin (unused in this type of RoPE). NrOrNrQrPrSrUrTr()r^)rVrr rXrYrWrGrrZr[r%rJcatzerosfloat32)rr(rrrbr\rRr]rNrope_proportionr_ rope_anglesinv_freq_rotated nope_anglesrs&&&&& r+%_compute_proportional_rope_parametersrlsEJ ""$AKAW611*=]c]s]svT2fff6H6HFLfLf6fH  -D ! % %h 4F*../FLOo0A56K LLA OQekk B E EV[`[f[f E gjr r t a-+-KQ99  Ku}}VL   $ H  %%r.c V^8dQhR\R,R\R,R\R,R\R,R\R \3,/#rArF)rKs"r+rLrLs^C&C& ' (C& ^ $C&4ZC&d C&  >5 ! C&r.c VP4VeVPV,M VPpVR,pVPRR4p\VRVPVP ,4p\ Wv,4pVR,p Rp VfVPpM\V\P4'dN\P!V\P!VPVPVPR74pM\W P4pWYV,VP, V ^, , W^, , ,,pRV\P !^V^\P"R7P%V\P&R7V, ,, p W3#) a Computes the inverse frequencies with NTK scaling. Credits to the Reddit users /u/bloc97 and /u/emozilla Args: config ([`~transformers."PreTrainedConfig"`]): The model configuration. This function assumes that the config will provide at least the following properties: * rope_theta (`float`, *optional*): The base wavelength from which the inverse frequencies will be derived. Defaults to `config.default_theta` if omitted. * hidden_size (`int`): The numerator when deriving a head_dim, if not provided directly. * num_attention_heads (`int`): The denominator when deriving a head_dim, if not provided directly. * max_position_embeddings (`int`): The default sequence length used to update the dynamic RoPE at inference time * rope_parameters (`dict[str, float]`): The standard RoPE scaling parameters, from which `factor` will be accessed. The value of `factor` is used to determine the new base frequency, along with the current sequence length (seq_len), the maximum positional embeddings (max_position_embeddings), and the computed dimensionality (dim) of the rotary embeddings. If seq_len <= max_position_embeddings, this factor has no effect. If seq_len <= max_position_embeddings, this factor effectively stretches the context window using an exponent derived from `dim`. Additionally, this function will make use of the following properties if they are found in the config: * head_dim (`int`, *optional*): The size of the key-value heads in the model. If None, this value will be derived as hidden_size // num_attention_heads. * partial_rotary_factor (`float`, *optional*): If less than 1.0, inverse frequencies will be returned for the first fraction of the head_dim. Defaults to 1.0. device (`torch.device`): The device to use for initialization of the inverse frequencies. seq_len (`int`, *optional*): The current sequence length, used to update the dynamic RoPE at inference time. If `None` or shorter than max_position_embeddings, this value will be overridden by max_position_embeddings. Returns: Tuple of (`torch.Tensor`, `float`), containing the inverse frequencies for the RoPE embeddings and the post-processing scaling factor applied to the computed cos/sin (unused in this type of RoPE). rOrPrQrRrNrdrSrU)rVrrWr rXrYrGmax_position_embeddings isinstancerTensormaximumtensorrTr(rrZr[r%rJ) rr(rrr\r]rPrRr^rNr_rs &&&& r+_compute_dynamic_ntk_parametersrtsmV ""$AKAW611*=]c]s]s  -D0445LcRvz6+=+=A[A[+[\H h. /C !( +F00 GU\\ * *--  LL77w}}U\UcUc d  g==> W$v'E'EE&ST*U[^hibi[jk kDdu||AsAU[[ILLTZbgbmbmLnqttuvH  %%r.c V^8dQhRRR\R,R\R,R\R,R\R \3,/#rArF)rKs"r+rLrLGsXD&D& D& ^ $D&4ZD&d D&  >5 ! D&r.caVP4VeVPV,M VPpVR,pVPRR4p\VRVPVP ,4p\ Wv,4pVR,p VPR4p VPR4p VPR4p VR ,p V fVPV , p RR lpV f8V 'd(V 'd \V!W4V!W4, 4p MV!V 4p VPR 4;'g^ pVPR 4;'g^pR oV3RlpRpV\P!^V^4PV\PR7V, ,pRV, pRV V,, pVPPRR4pV!VVWV V4wpp^V!VVV^,4PV\PR7, pV^V, ,VV,,pVV 3#)a Computes the inverse frequencies with NTK scaling. Please refer to the [original paper](https://huggingface.co/papers/2309.00071) Args: config ([`~transformers."PreTrainedConfig"`]): The model configuration. This function assumes that the config will provide at least the following properties: * rope_theta (`float`, *optional*): The base wavelength from which the inverse frequencies will be derived. Defaults to `config.default_theta` if omitted. * hidden_size (`int`): The numerator when deriving a head_dim, if not provided directly. * num_attention_heads (`int`): The denominator when deriving a head_dim, if not provided directly. * max_position_embeddings (`int`): The maximum length of the positional embeddings. * rope_parameters (`dict[str, float | int]`): The standard RoPE scaling parameters, from which the following keys will be accessed: * `attention_factor` (`float`, *optional*): The scaling factor to be applied to the computed cos/sin. If None, the value is inferred from `factor`, `mscale`, and `mscale_all_dim` as available. * `beta_fast` (`float`, *optional*, defaults to 32): Parameter to set the boundary for extrapolation (only) in the linear ramp function. * `beta_slow` (`float`, *optional*, defaults to 1): Parameter to set the boundary for interpolation (only) in the linear ramp function. * `factor` (`float`, *optional*): The scaling factor applied when interpolating the position IDs to extend the possible context length. Additionally, if `attention_factor` is None, the log of this value is used to compute a value for `attention_factor`, possibly in conjunciton with `mscale` and `mscale_all_dim`, if provided. * `mscale` (`float`, *optional*): If `attention_factor` is None and both `mscale` and `mscale_all_dim` are provided, `mscale` acts scalar augmenting `log(factor)` when computing the numerator for the inferred value of `attention_factor`. If not provided, `attention_factor` will be calculated based on `factor` only. * `mscale_all_dim` (`float`, *optional*): If `attention_factor` is None and both `mscale` and `mscale_all_dim` are provided, `mscale_all_dim` acts scalar augmenting `log(factor)` when computing the denominator for the inferred value of `attention_factor`. If not provided, `attention_factor` will be calculated based on `factor` only. * `original_max_position_embeddings` (`int`): The original max position embeddings used during pretraining. * `truncate` (`bool`, *optional*): Whether to truncate the correction range. Additionally, this function will make use of the following properties if they are found in the config: * head_dim (`int`, *optional*): The size of the key-value heads in the model. If None, this value will be derived as hidden_size // num_attention_heads. * partial_rotary_factor (`float`, *optional*, defaults to 1.0): If less than 1.0, inverse frequencies will be returned for the first fraction of the head_dim. device (`torch.device`): The device to use for initialization of the inverse frequencies. seq_len (`int`, *optional*): The current sequence length. Unused for this type of RoPE. Returns: Tuple of (`torch.Tensor`, `float`), containing the inverse frequencies for the RoPE embeddings and the post-processing scaling factor applied to the computed cos/sin. rOrPrQrRrNr_mscalemscale_all_dimrcjV^8:dR#RV,\P!V4,R,#)rQg?)mathlog)scalerws&&r+ get_mscale,_compute_yarn_parameters..get_mscales( A:V|dhhuo-33r. beta_fast beta_slowcV\P!W0^,\P,, 4,^\P!V4,, #)zPInverse dimension formula to find the dimension based on the number of rotations)r{r|pi) num_rotationsr^r]ros&&&&r+find_correction_dim5_compute_yarn_parameters..find_correction_dims@dhh6!:Kdgg:UVWW\]`d`h`him`n\noor.c<S!WW44pS!WW44pV'd-\P!V4p\P!V4p\V^4\ Wr^, 43#)z.Find dimension range bounds based on rotations)r{floorceilrmin) low_rothigh_rotr^r]rotruncatelowhighrs &&&&&& r+find_correction_range7_compute_yarn_parameters..find_correction_rangesR!'N"8$P **S/C99T?D3{CAg...r.cW8Xd VR, p\P!V\PR7V, W, , p\P!V^^4pV#)gMbP?rS)rrZrgclamp)rrr^ linear_func ramp_funcs&&& r+linear_ramp_factor4_compute_yarn_parameters..linear_ramp_factorsH : 5LC||Cu}}=C R KK Q2 r.rUrT)rz) rVrrWr rXrYrGrorJrrZr%)rr(rrr\r]rPrRr^rNr_rwrxrr~rrrr pos_freqsinv_freq_extrapolationinv_freq_interpolationrrrinv_freq_extrapolation_factorrrs&&&& @r+_compute_yarn_parametersrGs5t ""$AKAW611*=]c]s]s  -D0445LcRvz6+=+=A[A[+[\H h. /C !( +F+//0BC ! % %h /F)--.>?N';<^'_$ ~//2RR4  n$Z%?*VBd%de )&1 %((5;;I$((5::Ip/aa03363UX[[\I 9_ FY$67%%))*d;H%iCGgiqrIC%&(:3cQh(O(R(RZ`hmhshs(R(t$t!!&C"CD #@ @ A  % %%r.c V^8dQhRRR\R,R\R,R\R,R\R \3,/#rArF)rKs"r+rLrLsXU&U& U& ^ $U&4ZU&d U&  >5 ! U&r.cVP4VeVPV,M VPpVR,pVPRR4p\VRVPVP ,4p\ Wv,4pVR,p VR,p VPR4p VPR4p VR ,p V fVPV , p V fWV R8:dRp ML\P!^\P!V 4\P!V 4, ,4p V'd/W-8d)\P!V \PVR 7pM'\P!V \PVR 7p\P!^V^\PVR 7P!4V, pRWV,,, pVV 3#) ay Computes the inverse frequencies with LongRoPE scaling. Please refer to the [original implementation](https://github.com/microsoft/LongRoPE) Args: config ([`~transformers."PreTrainedConfig"`]): The model configuration. This function assumes that the config will provide at least the following properties: * rope_theta (`float`, *optional*): The base wavelength from which the inverse frequencies will be derived. Defaults to `config.default_theta` if omitted. * hidden_size (`int`): The numerator when deriving a head_dim, if not provided directly. * num_attention_heads (`int`): The denominator when deriving a head_dim, if not provided directly. * max_position_embeddings (`int`): The maximum length of the positional embeddings. * original_max_position_embeddings (`int`, *optional*): The original max position embeddings used during pretraining. If not provided, defaults to `max_position_embeddings`. * rope_parameters (`dict[str, float]`): The standard RoPE scaling parameters, from which the following keys will be accessed: * `attention_factor` (`float`, *optional*): The scaling factor to be applied on the attention computation. If unspecified, it defaults to value recommended by the implementation, inferred from the value of `factor`. * `factor` (`float`, *optional*): The scaling factor to apply to the RoPE embeddings. If both `max_position_embeddings` and `original_max_position_embeddings` are provided, this value will be overridden s the ratio between those values. * `long_factor` (`float`, *optional*): The scale factor applied when computing the inverse frequencies if `seq_len` is provided and greater than `original_max_position_embeddings`. * `short_factor` (`float`, *optional*): The scale factor applied when computing the inverse frequencies if `seq_len` is None or less-than-or-equal-to `original_max_position_embeddings`. Additionally, this function will make use of the following properties if they are found in the config: * head_dim (`int`, *optional*): The size of the key-value heads in the model. If None, this value will be derived as hidden_size // num_attention_heads. * partial_rotary_factor (`float`, *optional*, defaults to 1.0): If less than 1.0, inverse frequencies will be returned for the first fraction of the head_dim. device (`torch.device`): The device to use for initialization of the inverse frequencies. seq_len (`int`, *optional*): The current sequence length. Returns: Tuple of (`torch.Tensor`, `float`), containing the inverse frequencies for the RoPE embeddings and the post-processing scaling factor applied to the computed cos/sin. rOrPrQrR long_factor short_factorrNr_rrd)rVrrWr rXrYrGror{sqrtr|rrsrgrZr[rJ)rr(rrr\r]rPrRr^rrrNr_r ext_factorsinv_freq_shapers&&&& r+_compute_longrope_parametersrsd ""$AKAW611*=]c]s]s  -D0445LcRvz6+=+=A[A[+[\H h. /C&}5K'7L ! % %h /F+//0BC';<^'_$ ~//2RR S=" #yyTXXf-=Ii@j-j)jk 7=ll;emmFS ll5 ! L,r.c VP4VeVPV,M VPpVR,pVPRR4p\VRR4;'gVPVP ,p\ Wv,4pRp RV\P!^V^\PR7PV\PR7V, ,, p VR,p VR ,p VR ,p VR ,pW, pW, p^\P,V , p\P!VV8W, V 4pVV, V , W, , p^V, V,V , VV,,pVV8(VV8(,p\P!VVV4pVV 3#) a, Computes the inverse frequencies for llama 3.1. Args: config ([`~transformers."PreTrainedConfig"`]): The model configuration. This function assumes that the config will provide at least the following properties: * rope_theta (`float`, *optional*): The base wavelength from which the inverse frequencies will be derived. Defaults to `config.default_theta` if omitted. * hidden_size (`int`): The numerator when deriving a head_dim, if not provided directly. * num_attention_heads (`int`): The denominator when deriving a head_dim, if not provided directly. * rope_parameters (`dict[str, float | int]`): The standard RoPE scaling parameters, from which the following keys will be accessed: * `factor` (`float`, *optional*): The scaling factor applied to the inverse frequencies when 1) the wavelength is greater than `low_freq_wavelen` prior to smoothing, and 2) to all inverse frequencies during smoothing. * `high_freq_factor` (`float`): The scale factor used to compute `high_freq_wavelen` and the value for the denominator of the smoothing factor prior to the `low_freq_factor` shift. * `low_freq_factor` (`float`): The scale factor used to compute `low_freq_wavelen` and the shift applied to the numerator and denominator of the smoothing factor. frequencies if `seq_len` is None or less-than-or-equal-to `original_max_position_embeddings`. * `original_max_position_embeddings` (`int`): The original max position embeddings used during pretraining. If not provided, the function falls back to `max_position_embeddings`. Additionally, this function will make use of the following properties if they are found in the config: * head_dim (`int`, *optional*): The size of the key-value heads in the model. If None, this value will be derived as hidden_size // num_attention_heads. * partial_rotary_factor (`float`, *optional*): If less than 1.0, inverse frequencies will be returned for the first fraction of the head_dim. Defaults to 1.0. device (`torch.device`): The device to use for initialization of the inverse frequencies. seq_len (`int`, *optional*): The current sequence length. Unused for this type of RoPE. Returns: Tuple of (`torch.Tensor`, `float`), containing the inverse frequencies for the RoPE embeddings and the post-processing scaling factor applied to the computed cos/sin. NrOrPrQrRrSrUrNlow_freq_factorhigh_freq_factorr)rVrrWr rXrYrGrrZr[r%rJr{rwhere)rr(rrr\r]rPrRr^r_rrNrrold_context_lenlow_freq_wavelenhigh_freq_wavelenwaveleninv_freq_llama smooth_factorsmoothed_inv_freqis_medium_freqs&&&& r+_compute_llama3_parametersr&sZ ""$AKAW611*=]c]s]s -D0445LcRvz40ddF4F4F&JdJd4dH h. /Cdu||AsAU[[ILLTZbgbmbmLnqttuvH !( +F*+<=O+,>?*+MNO&8':$''kH$G[[+;!;X=NPXYN$w.@EUEghM]*n