+ QV-j aREt0t^RIt^RIt^RIt^RIt^RIt^RIt^RIt^RIt^RI t ^RI H t ^RIH t ^RI HtHt^RIHt^RIHtHt^RIHtHt^RIHt^R IHt^R IHtHtHtHtH t ^R I!H"t"^RI#t#^R I$H%t%H&t&^R I'H(t(^RI)H*t*^RI+H,t-^RI+H.t/^RI#H0t0H1t1^RI2H3t3^RI4H5t5^RI6H7t8^RI9H:t:^RI;Ht>H?t?H@t@HAtA^RIBHCtC^RIDHEtE^RIFHGtGHHtH^RIIHJtJHKtKHLtLHMtMHNtN^RIOHPtPHQtQHRtRHStSHTtTHUtUHVtV^RIWHXtX^RIYHZtZ^RI[H\t\^R I]H^t^^R!I_H`t`^R"IaHbtb^R#IcHdtdHete^R$IfHgtg^R%IhHiti^R&IjHktk^R'IlHmtm^R(InHotoHptpHqtqHrtrHstsHtttHutu^R)IvHwtw^R*IxHytyHztzH{t{H|t|^R+I}H~t~^R,IHtHt^R-IHt^R.IHt^R/IHt^R0IHt^R1IHt^R2IHtHtHtHtHtHtHtHtHtHtHtHtHtHtHtHtHtHtHtHtHt^R3IHtHtHt^R4IHtHtHtHt^R5IHtHtHtHtHt^R6IHtHt^R7IHtHt^R8IHt]!4'd ^R9IHt^R:IHt]'d^R;IHt]#PEP4t]!4'd8^RIHuH#t^Rt]EP!]4t]EPEPR?R@4EP4t]EPEPRAR@4EP4t]!RBRCRD7tR>sR>s]!RERF7!RGRH44tRIRJltRKRLltRMt]RN4t]RO4t]RRPRQll4tRRtRStRT]#EPRU]#EPRV]#EPRW]#EPRX]#EPRY]#EPRZ]#EPR[]#EPR\]#EPR]]#EPR^]#EPR_]#EPR`]#EPRa]#EPRb]#EP/tRcRdltRReRflltRgRhltRiRjltRkRlltRmRnltRoRpltRqRrltRRsRtlltRRuRvlltRRwRxllt!RyRz4t!R{R|4t!R}RC]1EP]]]]J4t]!]EP4]n]EPEPe<]EPEPEPR~RRR7]EPn] RRRll4t] RRRll4tRRRlltRRlEtRRRllEtRRlEt!RR]4EtE]!4Et]^k!RR]4EtR#)N)abstractmethod) defaultdict)CallableIterator)contextmanager) dataclassfield)partialwraps)cycle)Thread) TYPE_CHECKINGAnyTypeVarget_type_hintsoverload) is_zipfile)is_offline_mode"split_torch_state_dict_into_shards)version) safe_open)load) save_file)Tensornn) constraints) checkpoint)initializationPreTrainedConfig)get_model_conversion_mapping)WeightConverterWeightRenaming$convert_and_load_state_dict_in_modelrevert_weight_conversion)DistributedConfig)custom_object_save) CompileConfigGenerationConfig)PeftAdapterMixindeepspeed_config hub_kernelsis_deepspeed_zero3_enabledis_fsdp_enabled)_get_device_mapaccelerate_disk_offloadaccelerate_dispatchcheck_and_set_device_mapexpand_device_map get_deviceload_offloaded_parameter)!_load_state_dict_into_zero3_model)eager_paged_attention_forward)ALL_FP8_EXPERTS_FUNCTIONS)flash_attention_forward)paged_attention_forward)flex_attention_forward)allow_all_hub_kernels is_kernel)ALL_EXPERTS_FUNCTIONS)maybe_load_adapters)sdpa_attention_forward)sdpa_attention_paged_forward)ALL_PARALLEL_STYLES_get_parameter_tp_plandistribute_modelgather_state_dict_for_saveinitialize_tensor_parallelismshard_and_distribute_moduleverify_tp_plan) LOSS_MAPPING)$FLASH_ATTENTION_COMPATIBILITY_MATRIXFLASH_ATTN_KERNEL_FALLBACKlazy_import_flash_attention!lazy_import_paged_flash_attention)ROPE_INIT_FUNCTIONS) apply_patchespatch_output_recorders)id_tensor_storage) HfQuantizer)get_hf_quantizer)get_module_from_name)auto_conversion)ADAPTER_SAFE_WEIGHTS_NAME DUMMY_INPUTSSAFE_WEIGHTS_INDEX_NAMESAFE_WEIGHTS_NAMEWEIGHTS_INDEX_NAME WEIGHTS_NAMEContextManagers KernelConfigPushToHubMixin cached_filecheck_torch_load_is_safe copy_funchas_fileis_accelerate_availableis_bitsandbytes_availableis_env_variable_trueis_kernels_availableis_torch_flex_attn_availableis_torch_npu_availableis_torch_xpu_availablelogging)GeneralInterfaceis_flash_attention_requestedsplit_attention_implementation)DownloadKwargscreate_and_tag_model_cardget_checkpoint_shard_fileshf_api)is_flash_attn_greater_or_equal#is_huggingface_hub_greater_or_equalis_sagemaker_mp_enabledis_torch_cuda_available is_tracing)LoadStateDictInfolog_state_dict_report)_CAN_RECORD_REGISTRYOutputRecorder)QuantizationMethod)add_hook_to_module)extract_model_from_parallel)DeviceMeshLike) __version__z1.10F XLA_USE_BF160XLA_DOWNCAST_BF16SpecificPreTrainedModelTypePreTrainedModel)boundT)frozenca]tRt^toRtRt]!]R7tRt Rt Rt Rt Rt RtRt]!]R7tRtRtRtRtRt]V3RlRl4tV3RltR tVtR#) LoadStateDictConfigzY Config for loading weights. This allows bundling arguments that are just passed around. N)default_factoryFTc <V^8dQhRS[/#returnbool)format __classdict__s"l/Volumes/fast/ai/experiments/ui-tars-smoke/.venv/lib/python3.14/site-packages/transformers/modeling_utils.py __annotate__ LoadStateDictConfig.__annotate__s--d-cVPRJ#N) hf_quantizerselfs&r is_quantized LoadStateDictConfig.is_quantizeds  ,,rc<V^8dQh/S[R,;R&S[R,;R&S[R,;R&S[;R&S[R,;R&S[R,;R&S[R,;R&S[;R &S[P R,;R &S[;R &S[R,;R &R ;R&S[;R&S[S[S[ ,,R,;R&S[R,;R&#)rNpretrained_model_name_or_pathdownload_kwargsuse_safetensorsignore_mismatched_sizessharded_metadata device_mapdisk_offload_folderoffload_buffersdtype dtype_planrDeviceMeshLike | None device_mesh weights_onlyweight_mapping disable_mmap) strrnrdicttorchrrRlistr"r#)rrs"rrrs$':4$d*RD['")Tk(t "t*! ;; $ 2!"$+#$)/%&'(>9:TAH)*+$+r)__name__ __module__ __qualname____firstlineno____doc__rr rnrrrrrrrrrrrrrrrpropertyr__annotate_func____static_attributes____classdictcell__rs@rrrs~ 15!-2>-RO#'O$)$("J&*!O $ET2J'+L+/KLDHN $L --1rrc$V^8dQhR\/#rr)rs"rrrs4rc\;'dA\\PR4;'d\PP 4#)is_initialized)_torch_distributed_availablehasattrr distributedrrrr!_is_torch_distributed_initializedrsA$ / / E%%'7 8 / /    , , .rc$V^8dQhR\/#rint)rs"rrrs..3.rc\4'd!\\PR4'g^#\PP 4#)get_world_size)rrrrrrrr!_get_torch_distributed_world_sizers6 , . .ge>O>OQa6b6b    + + --rc~\4;'d-\\PP RR44^8H#) LOCAL_RANKz-1)rrosenvirongetrrris_local_dist_rank_0rs. , . _ _3rzz~~lTX7Y3Z^_3__rc#."RsRxRsR# Rsi;i5iTNF) _is_quantizedrrrset_quantized_statersM   c#."RsRxRsR# Rsi;i5ir)_is_ds_init_calledrrrset_zero3_staters!# "UrcRV^8dQhR\PR\R,/#)rrmodel_class_nameN)rrr)rs"rrrs"00U[[0C$J0rc#0"VP'gVe V RV R2pMRV R2p\V4h\P!4p\P!V4Rx\P!V4R# \P!T4i;i5i)z Locally change the torch default dtype to `dtype`, and restore the old one upon exiting the context. If `model_class_name` is provided, it's used to provide a more helpful error message if `dtype` is not valid. Nz% cannot be instantiated under `dtype=z$` as it's not a floating-point dtypez Cannot set `z7` as torch's default as it's not a floating-point dtype)is_floating_point ValueErrorrget_default_dtypeset_default_dtype)rr error_messageoriginal_dtypes&& rlocal_torch_dtypers  " " "  '#$$I%Ptu +5'1hiM'',,.N0 &  //sAB A;#B;BBc\P!.4Pp\P!4pW8Xd V\P!R48wdV#R#V#)z Test if a device context manager is currently in use, or if it is not the case, check if the default device is not "cpu". This is used to infer the correct device to load the model on, in case `device_map` is not provided. cpuN)rtensordeviceget_default_device)device_in_contextdefault_devices r*get_torch_context_manager_or_global_devicersM  R(//--/N* U\\%0 0! ! rcVP4F_pVP4'gKR\VP49gK7R\VP49gKSVPu# \ V4^8Xd\ P #\\VP444P#)zl Returns the first found floating dtype in `state_dict` if there is one, otherwise returns the first dtype. float8_float4_) valuesrrrlenrfloat32nextiter) state_dictts& rget_state_dict_dtypers      Yc!''l%ByX[\]\c\cXdGd77N!  :!}} Z&&() * 0 00rBOOLU8I8I16U16F16BF16I32U32F32F64I64U64F8_E4M3F8_E5M2c(V^8dQhRRR\/#)rpathzstr | os.PathLikerr)rs"rrr4s-$rc\PPR4'gR#\PP \P !V44p\RRR7;_uu_4p\RRV44RR R 7pR R R 4XF?wrEW8Xg0VPVPR 4R ,4'gK:VR 8Hu# R# +'giLW;i \\3dR#i;i)aTrue if `path` lives on an hf-mount FUSE filesystem (device string 'hf-mount'). hf-mount's mmap + readahead interaction deadlocks under parallel page-faults, so callers should load the file into memory instead. Linux-only; returns False on other platforms. linuxFz /proc/mountsutf-8encodingc3j"TF)p\V4^8gKV^,V^,3xK+ R#5i)rNr).0ps& r "_is_on_hf_mount..As*N'>!#a&A+!A$!'>s33c3@"TFqP4xK R#5ir)split)r ls& rrrAs'>2a 2sc&\V^,4#)r )es&r!_is_on_hf_mount..Bs c!A$irT)keyreverseN/zhf-mount) sysplatform startswithrrrealpathfspathopensortedrstripOSErrorr)rrealfhentriesdevmps& r_is_on_hf_mountr)4s << " "7 + +  ww $0 .7 3 3rN'>2'>N'G4 GCzT__RYYs^c-ABBj(( 4 3 Z    s6AC11C?C1C1C1 C. )C11DDc V^8dQhR\\P,R\\P,R\ R\ R,R\ \\P3,/#)rcheckpoint_file map_locationrrNr)rrPathLikerrrrr)rs"rrrMse/k/k2;;&/k $/k/k+ /k  #u||  /krc \P!V4pVf \V4pVPR4'EdXV'dyVR8wdr\ VR4;_uu_4p\ VP 44pRRR4VR8wd3XP4UUu/uFwrxWxPV4bK pppX#\VRR7;_uu_4p /pV P4FpVR8XdtV PV4p V P4p V \9d\V ,p M\RV 24h\P !V P#4V RR 7Wg&K}V P%V4PV4Wg&K VuuRRR4#V'd \'4/p VR8wd\)V4'dR R /p \P*!V3R VR V/V B# +'giELn;iuuppi +'giLu;i)aC Reads a `safetensor` or a `.bin` checkpoint file. We load the checkpoint on "cpu" by default. When `disable_mmap` is True, safetensors files are read fully into memory instead of being memory-mapped. When `disable_mmap` is None (default), it is auto-detected to True on hf-mount FUSE filesystems (see `_is_on_hf_mount`). N .safetensorsmetarbrpt) frameworkz)Cannot load safetensors of unknown dtype )sizerrmmapTr,r)rrr)endswithr _safe_load_bytesreaditemstorkeys get_slice get_dtypestr_to_torch_dtyperrempty get_shape get_tensorr`rr)r+r,rrcheckpoint_path_fhrkvf_slicek_dtyper extra_argss&&&& rload_state_dictrJMsii0O&7 // LF2ot,,-chhj9 -u$@J@P@P@RS@Ral!33@R S  $ 7 71JVVX6)[[^F$..0G"44 27 ;(+TU\T])^__$)KKV5E5E5Gu]c$dJM$%LLO$6$6|$DJM8 7  "Jv*_"="=d^ ::o jL j| j_i jj9-,,T 7 7s!G G4B7G: G1 : H cDV^8dQhR\PR\/#)rrr)rrr)rs"rrrsU\\crcVP4'd>VPR4R,P4VP4,pV#VP4pV#)r)nelementviewdata_ptr element_size)rstops& r_end_ptrrSsR {{2r"++-0C0C0EE K  KrcZV^8dQhR\PR\\,/#)rmoduler)rModulerr)rs"rrrs""))S rc .pVP4FYwr#\VR/4;'g/pTPVP4Uu.uFqR'dV RV 2MTNK up4K[ V#uupi)_tied_weights_keys.) named_modulesgetattrextendr;)rUtied_weight_keysname submoduletiedrDs& r_get_tied_weight_keysrasv"$!//1y"6;AArtyy{ S{!$D61#A!={ ST2 !Ts A2 c V^8dQhR\\\,,R\\\P 3,R\ \\\,,\\,3,/#rtensorsrrrsetrrrrtuple)rs"rrrsV,,DSN,S%,,=N8O,TYZ^_bcf_gZhjnorjsZsTt,rcf.pVFp\V4^8dVPV4K&.pVF6pW,pVPVP4\V4V34K8 VP 4V^,wrxp VPV 04VR,F9wrpW8dVPV04MVR,P V4T pK; K .p .p VFEp\V4^8Xd"V PVP 44K4V PV4KG W3#)r:rNNrM)rappendrPrSsortaddpop)rdrfiltered_tensorssharedareasr^r_ last_stop last_namestartrRdisjoint_tensorsshared_tensorss&& r_find_disjointrvs  v;?  # #F + D%F LL&//+Xf-=tD E  "'(i ,!&r E! ''/ $((.I "+&N# w<1   # #GKKM 2  ! !' * $  ++rc V^8dQhR\\\,,R\\\P 3,R\ \\\,,\\\,,3,/#rcre)rs"rrrsT%% #c(^%)-c5<<.?)@% 4C>4C> )*%rc~.p.pVFp\V4^8dK\P!\4pVFHpW,pVPVP 4\ V43pWX,PV4KJ \V4^8XdVPV4KVPV4K W#3#)r) r collectionsrrfrrPrSrkri) rdrru identicalrnror^rareas && r_find_identicalr|sN "I v;? '',D%FMM6??#4hv6FGD KOOD ! u:?   V $  ! !& )  $$rcV^8dQhR\\\P3,RRR\\\P3,/#)rrmodelrrrrrr)rs"rrrsEMMS%,,&'M0AM #u|| Mrca\P!\4pVP4Fwop\ V\ P 4'g$V\V4,PS4KIVPPR8Xd5VPS4pV\V4,PS4KV\V4,PS4K VP4UUu/uFwrE\V4^8gKWEbK ppp\\V44p.p\4p VeVP!4Fp^p \#V4F|o\$;QJdV3RlV4F 'gK RM RM!V3RlV44p V 'gKGSV9gKPV ^, p V \V48gKkV P'S4K~ K \)VP!4V4wrV FoVS,P+4VS&K \-W4wrVFTpVP/V 4pVFoVSK VP1V 4p\V4^8gKCVPV4KV V 'dVP3V 4\V4^8d\5RV RV R24hV#uuppi)aH Remove all tied weights from the given `state_dict`, making sure to keep only the main weight that `model` will expect when reloading (even if we now tie weights symmetrically, it's better to keep the intended one). This is because `safetensors` does not allow tensor aliasing - so we're going to remove aliases before saving. r0c3R<"TFp\P!VS4xK R#5irresearch)r patr^s& rr6remove_tied_weights_from_state_dict..s!%fFesbiiT&:&:Fes$'TFz8The weights trying to be saved contained shared tensors z\ which are not properly defined. We found all the potential target tied weights keys to be: zo. This can also just mean that the module's tied weight keys are wrong vs the actual tied weights in the model.)ryrrr9 isinstancerridrirtype get_parameterrQrrfrarr!anyrkrvcloner| intersection differencer\ RuntimeError)rr~ptrsrptrnames shared_ptrsall_potential_tied_weights_keys error_namesto_delete_namesfoundmatches_pattern shared_namesdisjoint_namesidentical_namesinamesknownunknownr^s&& @r#remove_tied_weights_from_state_dictrsv  " "4 (D"((* f&%,,// F  # #D ) ]]  6 )((.F F  # #D ) "6* + 2 24 8+"15 O *#E Q:3: KO'**?*F&G#KeO'2 '')EEu "%#%fFe%f###%fFe%f"f"?tz'9QJEs5z)'++D1 &*$2+2D2D2F #S L%d+113 4%4L$M!L!##O4D4 ##O4 wVRJd\ P P\ P PW\\V444'd1\ P PW\\V44pEMVRJd\ P P\ P PW\\V444'd3\ P PW\\V44pRpEM.V'g\ P P\ P PW\\V444'd1\ P PW\\V44pEMV'g\ P P\ P PW\\V444'd3\ P PW\\V44pRpEMV'd!\R\\V4 RV R24h\R\\V4 R\\V4 RV R24h\ P P\ P PW44'dTpRpEMyVeTpVPR4pM'VRJd\\V4pM\\V4pRVRV RV RV RV /pRV RVR VRRRRRVRV/VCp\!4'*;'d-\#R4'*;'dV'*;'dVR 8Hp\%VV3/VBpVfV\\V48Xd\%V\\V43/VBpVeRpM~V'dZVR8XdV'd\'V3/VBwpppVVR&Vf1\V R\\V4 R\\V4 R24hM\\V4p\%VV3/VBpVf7V\\V48Xd"\%V\\V43/VBpVeRpVehV'd\M\pV\\39dB\)VV3/VB'g/V'd'\+\&V3R R/VCR!R"7P-4MmVe:\)V\3/VB'd#\V R\\V4 R#V R$24h\V R\\V4 R\\V4 R24hV'd\0P3R(X 24TpMp\0P3R(X R)X 24MTV'dM\ P PV4'dTpM$RV RV RV RV RV RVRVR VRRRRRV/ p\%W3/VBpR pV'd\5VXV V V V V VVVVVR*7 wppVV3#VeX.MR pVV3# \dh\.d*p\R%T R&T R'\\T4 R24ThR p?ii;i)+zGet all the checkpoint filenames based on `pretrained_model_name_or_path`, and optional metadata if the checkpoints are sharded. This function will download the data if necessary. cache_dirforce_downloadFproxieslocal_files_onlytokenrevisionmain subfolder commit_hashNr/z.safetensors.index.jsonzadapter_model.binzThe transformers file in the config seems to be incorrect: it is neither a safetensors file (*.safetensors) nor a safetensors index file (*.safetensors.index.json): TzError no file named z found in directory rYz, or z, found in directory r _raise_exceptions_for_gated_repo%_raise_exceptions_for_missing_entries _commit_hashrDISABLE_SAFETENSORS_CONVERSIONz& does not appear to have a file named z or zX and thus cannot be loaded with `safetensors`. Please do not set `use_safetensors=True`.ignore_errors_during_conversionzThread-auto_conversion)targetargskwargsr^z) but there is a file without the variant z;. Use `variant=None` to load this model from those weights.zCan't load the model for 'z'. If you were trying to load it from 'https://huggingface.co/models', make sure you don't have a local directory with the same name. Otherwise, make sure 'z=' is the correct path to a directory containing a file named zloading weights file z from cache at ) rrrrrrrrrr)rnrr6rrrrisdirjoinisfilerrYrXr[rZr#rrer_rUrbr rs Exceptionloggerinforp)rrrrrrrrrrrrrrrrr is_shardedis_local archive_filefilenamehas_file_kwargscached_file_kwargscan_auto_convertresolved_archive_filesafe_weights_namerrcheckpoint_filess&&&&&&&&& r_get_resolved_checkpoint_filesr%s>&99)9O##K0I$(()95AN!!),G&**+=uE    (E"":.88&H##K4I!%%m4K%1-66~FFOmOvOv %P P .1DD `568 J$0Y5F(+,I(J%77==!>? 8-9!ww||,IVtu ;DDE^_  -"''.. :|TegnGop33 "ww||1lK\^e>f  !-"''.. :|TkmtGuv33 "ww||1lKbdk>l  " $ :|T`biGjk** "ww||1la  % :|TfhoGpq** "ww||1lK]_f>g  "  *<8I7+S*TU56a9 *<8I7+S*TTYZfgsu|Z}Y~++H*ILWW^^BGGLLR S S8LH.99;DDE^_  -'(97C' g>H7Y"$4 O!.jY2E7 j "" " $%%$$,-MNN$$'&$$O Y )44QS[(r_q(r%)0XN_ahAi5i,75$% ,\7 CDE 'y(ce &<=> ,\7 CDDVgipIqHrrsu$  KK/ ~> ?$0 ! KK/zI^H_` a  77>>) $ $$- ! Y .7"$4jHY2E7  " %00M$o\n$o !-G ) !)-!$! . **" - --7T6_12ei - --}  01N0OP99V8WX::F|U\:]9^^_a   s? A]-]-,B,]-7]-&]-8A-]--^,^,$^''^,cJV^8dQhR\\P,\,R,R\\,R,R\ R\R,R\R,R\ R\R,R \\ \P3,/#) rrNrconfigrrrrr) rrrrrr rrRrg)rs"rrr1sPP  t #d *P3i$&P PTk P t P  P$P U[[ ()PrcVRJpVEe\V\4'Ed?VR8Xd\VR4'd5VPe'VPp\P RV R24MV'dRV9d VR,pMcVe \ V4pMSVe0V^,PR4'd\PpM \V^,RVR7p\ V4p\P R V R 24M2\\V4'd\\V4pM \R 4h\V\4'd\\V4MTpMJ\V\\P34'g\R V 24hM\P!4pVeVPV4p\V\4'dhVP!R \P!44p\V\4'd\\V4MTp\P#RV R24MTpWnVP$Fp \W)4;p fKWnK W(3#)aFind the correct `dtype` to use based on provided arguments. Also update the `config` based on the inferred dtype. We do the following: 1. If dtype is "auto", we try to read the config, else auto-detect dtype from the loaded state_dict, by checking its first weights entry that is of a floating type - we assume all floating dtype weights are of the same dtype 2. Else, use the dtype provided as a dict or str NautorzWill use dtype=z$ as defined in model's config objectz.ggufr0)r,rzTSince the `dtype` attribute can't be found in model's config object, will use dtype=z as derived from model's weightsze`dtype` provided as a `str` can only be `'auto'`, or a string representation of a valid `torch.dtype`z`dtype` can be one of: `torch.dtype`, `'auto'`, a string of a valid `torch.dtype` or a `dict` with valid `dtype` for each sub-config in composite configs, but received rzUsing different dtypes per module is deprecated and will be removed in future versions Setting different dtypes per backbone model might cause device errors downstream, therefore setting the dtype=z for all modules.)rrrrrrrr6rrrJr[rrr update_dtyper warning_once sub_configs) rrrrrrrr main_dtypesub_config_key sub_configs &&&&&&& r _get_dtyper1s2"-J  eS ! !67++ 0H"LLEKK/%8\ ]^!g1A&A 0 9#/ 4Z @)5:J1:M:V:VW^:_:_ % %4,Q/fS_& !5Z @KK**/0PR&&u- { .8s-C-CGE5)EED%++#677JJOR 8'')))%0%YYr5#:#:#<= 3=j#3N3NWUJ/T^  !!+ ,= ?  L ,,!&9 9J F) -  rca]tRtRtoRt]V3RlRl4t]V3RlRl4tV3RlRlt] R 4t RV3R lR llt RV3R lRllt Rt VtR #)ModuleUtilsMixiniz@ A few utilities for `torch.nn.Modules`, to be used as a mixin. c8<V^8dQhRRRS[P/#rrrr)rr)rrs"rrModuleUtilsMixin.__annotate__s$AA&A5<<Arc B\RVP444#)zu `torch.device`: The device on which the module is (assuming that all the module parameters are on the same device). c38"TFqPxK R#5irrr params& rr*ModuleUtilsMixin.device..s@.?ULL.?sr parametersrs&rrModuleUtilsMixin.devices @doo.?@@@rc8<V^8dQhRRRS[P/#r)rr)rrs"rrrs$]]%]%++]rc B\RVP444#)zg `torch.dtype`: The dtype of the module (assuming that all the module parameters have the same dtype). c3h"TF(qP4'gKVPxK* R#5ir)rrrs& rr)ModuleUtilsMixin.dtype..s!\->EBYBYB[KEKK->s22rrs&rrModuleUtilsMixin.dtypes \T__->\\\rc*<V^8dQhRRRS[RS[/#)rrrencoder_attention_maskr)r)rrs"rrrs$//$5/v/Z`/rc P\PR4VP4^8Xd VR,pVP4^8Xd VR,pXPVPR7pRV, \ P !VP4P,pV#)z Invert an attention mask (e.g., switches 0. and 1.). Args: encoder_attention_mask (`torch.Tensor`): An attention mask. Returns: `torch.Tensor`: The inverted attention mask. zDetected the usage of `invert_attention_mask`: This function is deprecated and will be removed in v5.12.0. Please use the new API in `transformers.masking_utils`r?NNNNrrrNNr)rrdimr:rrfinfomin)rrencoder_extended_attention_masks&& rinvert_attention_mask&ModuleUtilsMixin.invert_attention_masks  E " % % '1 ,.D].S + ! % % '1 ,.DEU.V ++J*L*LSWS]S]*L*^'+.1P+PTYT_T_`d`j`jTkToTo*o'..rcH\PR4VPpVwr4\P!WBR7pVR,P W4^4VR,8*pVP VP4pVP^,VP^,8dhVP^,VP^,, p\P!\P!W4V3W&PR7V.RR7pVR,VR,,pV#) zDetected the usage of `create_extended_attention_mask_for_decoder`: This function is deprecated and will be removed in v5.12.0. Please use the new API in `transformers.masking_utils`rrraxis)NNr)NrNrMrr) rrrrarangerepeatr:rshapecatones) input_shapeattention_maskr batch_size seq_lengthseq_ids causal_maskprefix_seq_lenextended_attention_masks && r*create_extended_attention_mask_for_decoder;ModuleUtilsMixin.create_extended_attention_mask_for_decoders E  &&!, ,,z9m,33JANRYZgRhh !nn^%9%9:   Q ."6"6q"9 9+11!4{7H7H7KKN))JJ GPV^o^op K#.m"<~N^?_"_&&rNc l<V^8dQhRRRS[RS[S[R3,RS[PR,RS[/#) rrrrr.rNr)rrgrrr)rrs"rrrsL4'4'4'4'38_4'{{T! 4'  4'rc \PR4Vf VPpVP4^8Xd VR ,pMnVP4^8Xd>\ VP RR4'd\ PW!4pM&VR ,pM\RV RVP R24hVPVR7pRV, \P!V4P,pV#) a Makes broadcastable attention and causal masks so that future and masked tokens are ignored. Arguments: attention_mask (`torch.Tensor`): Mask with ones indicating tokens to attend to, zeros for tokens to ignore. input_shape (`tuple[int]`): The shape of the input to the model. Returns: `torch.Tensor` The extended attention mask, with a the same dtype as `attention_mask.dtype`. zDetected the usage of `get_extended_attention_mask`: This function is deprecated and will be removed in v5.12.0. Please use the new API in `transformers.masking_utils`N is_decoderz!Wrong shape for input_ids (shape z) or attention_mask (shape )rrrr)rrrrr[rrrrr r:rrr)rrrrrs&&&& rget_extended_attention_mask,ModuleUtilsMixin.get_extended_attention_masks$  E =JJE    1 $&4]&C #    !Q &t{{L$77*:*e*e+'+99I*J'3K=@[\j\p\p[qqrs #:"<"<5"<"I#&)@#@EKKPUDVDZDZ"Z&&rc0<V^8dQhRRRS[RS[RS[/#)rrronly_trainableexclude_embeddingsr)rr)rrs"rrrs,**.**bf*sv*rc V'dIVP4UUu.uF,wr4\V\P4'gK'V R2NK. ppp\ VRR4pV'd^RIp^pVP 4Fwr9V'd VX9dKV P'g V'dK2V'd\V XPP4'du\V R4'dV P4p M+\V R4'dV PPp M^p WP4^,V ,, pKWP4, pK V#uuppi)a Get number of (optionally, trainable or non-embeddings) parameters in the module. Args: only_trainable (`bool`, *optional*, defaults to `False`): Whether or not to return only the number of trainable parameters exclude_embeddings (`bool`, *optional*, defaults to `False`): Whether or not to return only the number of non-embeddings parameters Returns: `int`: The number of parameters. z.weightis_loaded_in_4bitFNrQ quant_storage)rZrr Embeddingr[ bitsandbytesnamed_parametersr Params4bitrrQr%itemsizenumel) rr!r"r^ module_typeembedding_param_namesr$bnb total_paramsr num_bytess &&& rnum_parametersModuleUtilsMixin.num_parameterss( :>:L:L:N%:N%6TR\]hjljvjvRw 4& :N "%$D*=uE  & 002KD!d.C&C"""..%E366;L;L)M)Mun55$)$6$6$8  88$)$7$7$@$@ $%  KKMA$5 $AAL KKM1L3"5%s $E, E,rrFF)rrrrrrrrr staticmethodrrr1rrrs@rrrsnAA]] //6''44'4'l***rrcXa]tRtRtoRtRtV3RlRltV3RlRltRtR t R t Vt R #) EmbeddingAccessMixini1z Base utilities to regroup getters and setters for embeddings. Introduces the `input_layer_embed` attribute, which indicates where the input embeddings come from and where they should be set. embed_tokensc4<V^8dQhRS[P/#rrrV)rrs"rr!EmbeddingAccessMixin.__annotate__;s% % bii% rc \VRR4p\WR4;peV#\VRR4pVe\W14'd \W14#\VRR4pVe\WA4'd \WA4#\VRR4pVe(WPJd#\VR4'dVP4#\VRR4pVe(W`Jd#\VR4'dVP4#\R VPP R 24h) zv Returns the model's input embeddings. Returns: `nn.Module`: A torch module mapping vocabulary to hidden states. _input_embed_layerr7N embeddingsr~language_modelget_input_embeddings base_modelu.`get_input_embeddings` not auto‑handled for "; please override in the subclass.)r[rr?NotImplementedError __class__r)rr^default_embeddingr=r~r>r@s& rr?)EmbeddingAccessMixin.get_input_embeddings;st1>B")T!: :  G$ $T<6  !gj&?&?:, ,gt,  !5!55' ' '7>  &*(>??!668 8T<6  !j&<UkAlAl224 4!` (direct attribute) 2. `self.embeddings.<_input_embed_layer>` (nested embeddings for vision/audio models) 3. `self.model.<_input_embed_layer>` (encoder/decoder models) 4. delegate to the *base model* if one exists 5. otherwise raise `NotImplementedError` so subclasses still can (and should) override for exotic layouts. r<r7r=Nr~r>set_input_embeddingsr@u.`set_input_embeddings` not auto‑handled for rA)r[rrrIrBrCr)rrGr^r=r~r>r@s&& rrI)EmbeddingAccessMixin.set_input_embeddingsbst1>B 4   D &#D,==j JwWaOhOh Je ,tWd33e @WUEYEY E ' 't-=tD D^ Q*(>??  / / 6#4t< RV3R7lR8llt?V3R9lR:lt@R;tAR<tBRV3R=lR>lltCRV3R?lR@lltDRAtERBtF]"P!4RC4tHRV3RDlRElltI]"P!4]JP!4RF44tLRV3RGlRHlltMRV3RJlRKlltNRLtORV3RMlRNlltPRROltQRV3RPlRQlltRRV3RRlRSlltSRTtTRV3RUlRVlltURWtVRXtWV3RYlRZltXV3R[lR\ltYR]tZRR^lt[RI]\3V3R_lR`llt]Rat^]!V3RblRcl4t_RV3RdlRellt`]a!]bP4V3Rfl4tcRRgltd]a!]"PPP4V3Rhl4tg]a!]"PPP4V3Ril4thV3RjltiV3Rkltj]0V3RllRml4tkV3RnlRoltlRV3RplRqlltm]0RrRRsRRtRRuRRvRRwRRxRyRzRR{RIR|RR}R/ V3R~lRll4tn]oRV3RlRll4tp]oV3RlRl4tqRRltr]0RRl4tsRtt]!R4tu]!R4tv]!R4tw]!R4tx]xPXR4txRRlty]!V3RlRl4tz]zPXV3RlRl4tzV3RlRlt{V3RlRlt|]0R4t}V3RlRlt~V3RlRltV3RlRltRtV3RlRltRV3RlRlltRV3RlV3RllltRt]0V3RlRl4tV3RltRtVtV;t#)riam Base class for all models. [`PreTrainedModel`] takes care of storing the configuration of the models and handles methods for loading, downloading and saving models as well as a few methods common to all models to: - resize the input embeddings Class attributes (overridden by derived classes): - **config_class** ([`PreTrainedConfig`]) -- A subclass of [`PreTrainedConfig`] to use as configuration class for this model architecture. - **base_model_prefix** (`str`) -- A string indicating the attribute associated to the base model in derived classes of the same architecture adding modules on top of the base model. - **main_input_name** (`str`) -- The name of the principal input to the model (often `input_ids` for NLP models, `pixel_values` for vision models and `input_values` for speech models). - **can_record_outputs** (dict): NrF input_idstextc6<V^8dQhRS[S[S[3,/#r)rrrz)rrs"rrPreTrainedModel.__annotate__s'.'.Dn)<$='.rc .VP;'g/#)a Maps output names (e.g., "attentions", "hidden_states") to either: - A module class (e.g., `LlamaDecoderLayer`), using default index conventions: * index=0 for "hidden_states" * index=1 for "attentions" - Or an `OutputRecorder(...)` with `target_class`, optional `index`, and `layer_name`. Examples: These two are equivalent: ```python _can_record_outputs = { "attentions": LlamaAttention, "hidden_states": LlamaDecoderLayer } _can_record_outputs = { "attentions": OutputRecorder(LlamaAttention, index=1), "hidden_states": OutputRecorder(LlamaDecoderLayer, index=0) } ``` This means you can record outputs from the same class, by specifying a layer name. Before collecting outputs, we check that they come from this layer. If you have cross attention that come from `LlamaAttention` and self attention that also come from `LlamaAttention` but from `self_attn` you can do this: ```python class LlamaModel(PreTrainedModel): _can_record_outputs = { "attentions": OutputRecorder(LlamaAttention, index=1, layer-name="self_attn"), "cross_attentions": OutputRecorder(LlamaAttention, index=1, layer_name="cross_attn") } ``` )_can_record_outputsrs&rcan_record_outputs"PreTrainedModel.can_record_outputssR''--2-rcJ<V^8dQhRS[S[S[P3,/#rr)rrs"rrrWs"99d3 #459rc :R\P!\4/#)zN `dict[str, torch.Tensor]`: Dummy inputs to do a forward pass in the network. rT)rrrWrs&r dummy_inputsPreTrainedModel.dummy_inputss U\\,788rc Z<\SV`!R/VB\P!V4P RR4pVP P RR4p\ V4P RR4pVPpVe W0nR#Ve W nR#Ve WPnR#Ve W@nR#R#)rN config_classr)super__init_subclass__inspectget_annotationsr__dict__rra)clsrchild_annotationchild_attributefull_annotationfull_attributerCs&, rrc!PreTrainedModel.__init_subclass__#s !+F+ #2237;;HdK,,**>4@)-11(DA))  &.   )/   '-   (. )rc <V^8dQhRS[/#)rrr)rrs"rrrW;s-M-M/-Mrc<\SV`4\V\4'g;\ RVP P RVP P R24hWnVPVnRVn VPVPPR\PR7VPnVPVPP 4VPnVP%4'd"VP&P)V4VnVP P pV\.9d`RRP1\.4 R2p\2P4!WPP P 4p\7V4^8d V^,pMRpW@nVP:\<\?VP 4&R# \,dTP'4TnLi;i) zParameter config in `zt(config)` should be an instance of class `PreTrainedConfig`. To create a model from a pretrained model use `model = z(.from_pretrained(PRETRAINED_MODEL_NAME)`NT is_init_checkallow_all_kernels(|r) rb__init__rr TypeErrorrCrr name_or_path kernel_config%_check_and_adjust_attn_implementation_attn_implementationr,ALLOW_ALL_KERNELS_attn_implementation_internal(_check_and_adjust_experts_implementation_experts_implementation _experts_implementation_internal can_generategeneration_config_classfrom_model_configgeneration_configrBrIrrfindallr loss_typerYryr)rrinputsrr loss_groupsrCs&&*, rrtPreTrainedModel.__init__;s &"233'(?(?'@A NN3344\^   "//"594^4^ KK , ,);; 5_5  18<7d7d KK / /8  4      H)-)E)E)W)WX^)_& NN++ L (chh|45Q7K ;0G0GHI9~!%aL  "484L4LS01' H)-)E)E)G& Hs G G?>G?c  ///uVnVnVnVPVJdVPP e%VPP P 4M/VnVPPe%VPPP 4M/VnVPPe%VPPP 4M/VnVPRR7Vn \VP;'g.4Vn \VP;'g.4Vn \VP;'g.4Vn\VP;'g.4Vn\VP ;'g.4Vn\VP";'g.4Vn\VP$;'g.4VnVP'4EF wr\)VRR4;p'dRVPP+VP 4P-4UUu/uF wrEV RV 2VbK upp4\)VRR4;p'dRVPP+VP 4P-4UUu/uF wrEV RV 2VbK upp4\)VRR4;p'dRVPP+VP 4P-4UUu/uF wrEV RV 2VbK upp4\)VRR4;p'dWVPP+VP 4P-4UUu/uFwrEV RV 2V RV 2bK upp4\)VR R4;p'dVPP+V4\)VR R4;p'dVPP+V4\)VR R4;p 'dVPP+V 4\)VR R4;p 'dVPP+V 4\)VR R4;p 'dVP P+V 4\)VRR4;p 'dVP"P+V 4\)VRR4;p 'gEKVP$P+V Uu0uF qA RV 2kK up4EK VP/4VP14R#uuppiuuppiuuppiuuppiuupi)a A method executed at the end of each Transformer model initialization, to execute code that needs the model's modules properly initialized (such as weight initialization). It is also used to obtain all correct static properties (parallelism plans, tied_weights_keys, _keep_in_fp32_modules, etc) correctly in the case of composite models (that is, the top level model should know about those properties from its children). NF all_submodels_ep_planrY_tp_plan_pp_planall_tied_weights_keys_keep_in_fp32_modules_keep_in_fp32_modules_strict_no_split_modules_skip_keys_device_placement"_keys_to_ignore_on_load_unexpected_keys_to_ignore_on_load_missing_keys_to_ignore_on_save)rrrr@rbase_model_pp_plancopybase_model_tp_planbase_model_ep_planget_expanded_tied_weights_keysrrfrrrrrrrnamed_childrenr[updater9 init_weights._backward_compatibility_gradient_checkpointing)rr^rUplanrDrE tied_keys keep_fp32keep_fp32_strictno_split skip_keysignore_unexpectedignore_missing ignore_saves& r post_initPreTrainedModel.post_initjs}79"b3 t}dm ??d "EI[[EcEcEoDKK::??AuwDMEI[[EcEcEoDKK::??AuwDMEI[[EcEcEoDKK::??AuwDM%)%H%HW\%H%]"%()C)C)I)Ir%J",/0Q0Q0W0WUW,X)!$T%;%;%A%Ar!B+.t/O/O/U/USU+V(25d6]6]6c6cac2d//243W3W3]3][]/^,'*4+G+G+M+M2'N$!//1LDvz488t8 $$499;CTCTCV%WCV41asmQ&6CV%WXvz488t8 $$499;CTCTCV%WCV41asmQ&6CV%WXvz488t8 $$499;CTCTCV%WCV41asmQ&6CV%WX#F,CTJJyJ**11\e\j\j\l\r\r\t2u\tTXTUdV1QC=TF!A3-3O\t2uv#F,CTJJyJ**11)<#*63QSW#XXX11889IJ"6+>EExE&&--h7#F,I4PPyP0077 B%,F4XZ^$__ _77>>?PQ!(1RTX!YY~Y44;;NK%f.GNN{NN,,33K4XKqvQqc]K4XY;2@  ;;==&X%W%W3v&5Ys$U U 0U U U c6<V^8dQhRS[S[S[3,/#rrr)rrs"rrrWsc3hrc \VPR4'd3VPPP'd VP#VP #)z* The full tp plan for the model's modules distributed_config)rrrenable_expert_parallelrrrs&rtp_planPreTrainedModel.tp_plans? 4;; 4 5 5$++:X:X:o:o:o== }}rcL<V^8dQhRS[S[S[S[S[3,3,/#rrrrg)rrs"rrrWs&c5c?23rcVP#r)rrs&rpp_planPreTrainedModel.pp_plans }}rcD<V^8dQhRS[S[S[3,R,/#rrNr)rrs"rrrWs!!!DcNT1!rc XVf /VnR#\V\4'g \R4hVP 4F@wr#V\ 9gK\RV RV R\ \ P!44 24h VP4UUu.uFwrEVNK pppVP4FdpVPRR4pRpVF#p \P!Wy4'gK!RpM V'dKJ\P!R V R 24Kf WnR#uuppi) Nz&Can only set a dictionary as `tp_plan`z#Unsupported tensor parallel style 'z ' for layer 'z'. Supported styles are *z\d+FTzLayer pattern 'z' does not match any parameters in the model. This rule may not be applied during tensor parallelization, or may lead to dimension mismatches)rrrrr9rBrr;r(replacermatchwarningswarn) rr layer_patternparallel_styler^rpmodel_param_names regex_patternpattern_matchedrs && rrrs% <DM $%%EF F.2ZZ\ )M%88 9.9IWdVef,,01D1I1I1K,L+MO.:261F1F1HI1HgdT1HI!YY[M)11#v>M#O/ 88M66&*O0#? %m_5dd) !Js D&cZ<V^8dQhRS[S[S[S[S[3,3,R,/#rr)rrs"rrrWs+DeCHo!56=rcnVf /VnR#\V\4'g \R4hWnR#)Nz&Can only set a dictionary as `pp_plan`)rrrr)rrs&&rrrs/ <DM $%%EF F rc ^\VRR4pVf \R4hVPWR7#)zv Potentially dequantize the model in case it has been quantized by a quantization method that support dequantization. rNz?You need to first quantize your model in order to dequantize itr)r[r dequantize)rrrs&& rrPreTrainedModel.dequantizes8 t^T:  ^_ _&&t&99rcVP'dH\VPRR4'd)VP4\ VPR4R#R#R#)gradient_checkpointingFN)supports_gradient_checkpointingr[rgradient_checkpointing_enabledelattrrs&rr>PreTrainedModel._backward_compatibility_gradient_checkpointingsF  / / /GDKKIach4i4i  . . 0 DKK!9 :5j /rcD<V^8dQhRS[S[,S[,RR/#)rtagsrN)rr)rrs"rrrWs#,,49s?,t,rc \V\4'dV.pVPf.VnVF0pW P9gKVPPV4K2 R#)a Add custom tags into the model that gets pushed to the Hugging Face Hub. Will not overwrite existing tags in the model. Args: tags (`Union[list[str], str]`): The desired tags to inject in the model Examples: ```python from transformers import AutoModel model = AutoModel.from_pretrained("google-bert/bert-base-cased") model.add_model_tags(["custom", "custom-bert"]) # Push the model to your namespace with the name "my-custom-bert". model.push_to_hub("my-custom-bert") ``` N)rr model_tagsri)rrtags&& radd_model_tagsPreTrainedModel.add_model_tagssO, dC 6D ?? " DOC//)&&s+rc VPRVP4pVPRR4;pe*\PR4W1P8wdTMTp\ V\ 4'd\ \V4pW1nVPFp\ W4;pfKW6nK RV9dVPR4Vn RV9dVPR4Vn VPRR4p\4.pVe%VP\W0P44V'dVP\!44\#4;'d\$'*;'d \&'*p V 'dk\P)R 4^RIp VP-\.P0!4V P2P5\74R 7\94.4\;V4;_uu_4V!V3/VBp \=V 4RRR4V 'd^R IH p V !X 4V PC4X # +'giL8;i) z All context managers that the model should be initialized under go here. Args: dtype (`torch.dtype`, *optional*): Override the default `dtype` and load the model under this dtype. r torch_dtypeNz1`torch_dtype` is deprecated! Use `dtype` instead!attn_implementationexperts_implementationrqF@Detected DeepSpeed ZeRO-3: activating zero.init() for this modelconfig_dict_or_path)initialize_weights_zero3)"rlrrrrrr[rrryr}rrOrirrr<r-rrr deepspeedr\initno_init_weightszeroInitr+rr\rPintegrations.deepspeedr tie_weights) rgrrrrrrrq init_contextsneeds_zero3_initrr~rs &&, r _from_configPreTrainedModel._from_configs 7FLL1!::mT: :K G    S T"ll2E E eS ! !E5)E  $00N%f== J#( 1 !F **0**5J*KF ' $v --3ZZ8P-QF *#JJ':EB&)    !25,,!G H   !6!8 957hh .ssXr;T466XrTFzT is not available on CPU. Please make sure you are on any of the supported devices: rYz requires compute capability >= z , but found z with compute capability z.x) rr ImportErrorrrziprrurcudaget_device_capability)rrrrrrrr info_messageprefacedevice_availability_checks device_namesmajorrps&&&&&&& r_flash_attn_import_error(PreTrainedModel._flash_attn_import_errorsF8P 3 %(** L)8P *++&'9&:;BCG*++!i>?Q>RRpq$q(1OPW1X1X!WI_=O>**++MN`MabWW[WbWbWpWpVqrnn  Y .9:L9MMefg g %%a(LM_(`a  !t{{$788T[[=Z=Z]^=^##@AS@TU~~  !! =   <=Obcu>v'??:-122'+ ++HI[H\]FFQ]RVX ?$$DEWDXYVV/"Os H3c&<V^8dQhRS[RS[/#rrprr)rrs"rrrWsrc VP'g#\VPP R24h\P P e\PP4^8d|\ P!\P4\ P!R48d?\PR4\PPPR4R#)a Check the availability of SDPA for a given model. Args: is_init_check (`bool`, *optional*): Whether this check is performed early, i.e. at __init__ time, or later when the model and its weights are fully instantiated. This is needed as we also check the devices of the weights, which are only available later after __init__. This allows to raise proper exceptions early before instantiating the full models if we know that the model does not support the requested attention. a does not support an attention implementation through torch.nn.functional.scaled_dot_product_attention yet. Please request the support for this architecture: https://github.com/huggingface/transformers/issues/28005. If you believe this error is a bug, please open an issue in Transformers GitHub repository and load your model with the argument `attn_implementation="eager"` meanwhile. Example: `model = AutoModel.from_pretrained("openai/whisper-tiny", attn_implementation="eager")`z2.4.1zUsing the `SDPA` attention implementation on multi-gpu setup with ROCM may lead to performance issues due to the FA backend. Disabling it to use alternative backends.FT)_supports_sdparrCrrrhipr device_countparserrrbackendsenable_flash_sdprrps&&r_sdpa_can_dispatch"PreTrainedModel._sdpa_can_dispatchs""">>**+,OO  MM   ) '')A- e//07==3II   y  NN   0 0 7rc <V^8dQhRS[/#rr)rrs"rrrW;s  $ rc vVP4'g#\VPP R24hR#)z9 Check the availability of Grouped MM for a given model. z1 does not support setting experts implementation.T)_can_set_experts_implementationrrCrrs&r_grouped_mm_can_dispatch(PreTrainedModel._grouped_mm_can_dispatch;s6 3355 7 788ijk krc&<V^8dQhRS[RS[/#rr)rrs"rrrWFsTdrc VP'g#\VPP R24h\ 4'g \ R4hR#)a  Check the availability of Flex Attention for a given model. Args: is_init_check (`bool`, *optional*): Whether this check is performed early, i.e. at __init__ time, or later when the model and its weights are fully instantiated. This is needed as we also check the devices of the weights, which are only available later after __init__. This allows to raise proper exceptions early before instantiating the full models if we know that the model does not support the requested attention. a does not support an attention implementation through torch's flex_attention. Please request the support for this architecture: https://github.com/huggingface/transformers/issues/34809. If you believe this error is a bug, please open an issue in Transformers GitHub repository and load your model with the argument `attn_implementation="eager"` meanwhile. Example: `model = AutoModel.from_pretrained("openai/whisper-tiny", attn_implementation="eager")`z]PyTorch Flex Attention requirements in Transformers are not met. Please install torch>=2.5.0.T)_supports_flex_attnrrCrrgrr#s&&r_flex_attn_can_dispatch'PreTrainedModel._flex_attn_can_dispatchFsY'''>>**+,tt ,--o  rc@<V^8dQhRS[R,RS[RS[RS[/#)rrNrprqrrr)rrs"rrrWas7q.q.#&:q.>Bq._cq. q.rc  r\V4wrET\VRR4;'g.9dRpVeh\VRR4p\VR7'dIVeEWV9d?V'd RV^, 2MV^,p\P RV RV RV R 24Tp\V4wrETpR p \VR7'dL\ P !4F2p VR V 28XgK\ V ,R ,!4'dK0Rp M VP'd\V 'dT\4'dD\4'g4\V,p\4'd VR 8XdR p V'dRV 2p\V4'dFV'd\WR7M \WR7V 'd\P RV R24V#VP%W4p\VR7'd \V4V# \d4p T 'd%\!TR,4p TP#YR7T hRp ? ii;i)a@ Check that the `attn_implementation` exists and is supported by the models, and try to get the kernel from hub if it matches hf kernels pattern. Args: attn_implementation (`str` or `None`): The attention implementation to check for existence/validity. is_init_check (`bool`, *optional*): Whether this check is performed early, i.e. at __init__ time, or later when the model and its weights are fully instantiated. This is needed as we also check the devices of the weights, which are only available later after __init__. This allows to raise proper exceptions early before instantiating the full models if we know that the model does not support the requested attention. allow_all_kernels (`bool`, optional): Whether to load kernels from unverified hub repos, if `attn_implementation` is a custom kernel outside of the `kernels-community` hub repository. Returns: `str`: The final attention implementation to use, including potential fallbacks from sdpa to eager, or from None to sdpa (to potentially eager). !_compatible_flash_implementationsNT"requested_attention_implementationzpaged|zNThis model is compatible with the following flash attention implementations: `z"`. Automatically falling back to `z` instead of `z`.Fflash_attention_rflash_attention_2)rqz/You do not have `flash_attn` installed, using `z%` from the `kernels` library instead!rrprM)rmr[rlrrrJr;rrfrhrKrir=rMrLrrrget_correct_attn_implementation) rrrprqis_pagedbase_implementation compatible_flash_implementationsdefault_flash_implementationapplicable_attn_implementationrequested_original_flash_attn fa_versionrs &&&& rrx5PreTrainedModel._check_and_adjust_attn_implementationasb.)GGZ([% 741TVZ#[#a#a_a b $   */6t=`bf/g ,,Pcdd4@'OGOf=a@ABTtuvTw-##dfFeGG66R5SSabuavvxz'C#(FGZ([%)<&(-% 'K^ _ _BGGI (-=j\+JJ@LMijll481J  % % %-$&&*,,-GH[-\ *%'',?CV,V16-39:X9Y1Z. 3 4 4 5600Nt1''IJhIij>>*.-.2-Q-Q.. * ,Omnn+,JK--# 0!$%8%FFc4<V^8dQhRS[R,RS[/#)rrequested_expertsNrr)rrs"rrrWs ""C$J"SV"rcVfRMTpR.\\\P!44\\P!44,4,pVUu.uF pRV R2NK ppRVR ,,VR &RP V4pW#9dRV RV R 2p\ V4hVR8XdVP4V#V#uupi \ \3dpTR8XdThRpRp?T#Rp?ii;i) N grouped_mmrKz`experts_implementation="z"`zand rNz#Specified `experts_implementation="z5"` is not supported. The only possible arguments are rYrM) rrfr>r;r8rrr)r) rrYapplicable_expertsbase_experts_fnsfnvalid_experts_str_listvalid_experts_strrUrs && rrD2PreTrainedModel.get_correct_experts_implementations->-F\L]#9tC0E0J0J0L,MPSTmTrTrTtPu,u'vvO_!`O_$=bT"DO_!`%+.DR.H%Hr" II&<=  556H5II~$%Q( W% %  - ---/ "!!!'"a , -$ 4G%,"!!  -sC 6CC6 C11C6c <V^8dQhRS[/#rr)rrs"rrrW sTrc \PPVP4pVe\ VR4'gR#VP p\ VRRR7;_uu_4pVP4pRRR4\P!RX4'dRV9;'dR V9#R # +'giLA;i) zDetect whether the class supports setting its attention implementation dynamically. It is an ugly check based on opening the file, but avoids maintaining yet another property flag. N__file__Frrrzclass \w+Attention\(nn.Module\)eager_attention_forwardz&ALL_ATTENTION_FUNCTIONS.get_interface(T) rmodulesrrrrdr r8rrrg class_module class_filerFcodes& r_can_set_attn_implementation,PreTrainedModel._can_set_attn_implementations {{s~~6  w|Z'H'H!** *cG 4 4668D5 997 > >,4ii9aei9i i5 4s $B.. B> c <V^8dQhRS[/#rr)rrs"rrrW3s 5 5 5rc 2\PPVP4pVe\ VR4'gR#VP p\ VRRR7;_uu_4pVP4pRRR4RV9# +'giRX9#;i)zDetect whether the class supports setting its experts implementation dynamically. It is an ugly check based on opening the file, but avoids maintaining yet another property flag. NrdFrerrz@use_experts_implementation)rrgrrrrdr r8rhs& rr(/PreTrainedModel._can_set_experts_implementation2s~ {{s~~6  w|Z'H'H!** *cG 4 4668D5-445 4-44s $B B c6<V^8dQhRS[S[,RS[/#)rrrq)rrr)rrs"rrrWAs$d8d83:d8Z^d8rc  \V\4'gTM%VPRVPP4pW0PP8wdhVP 4'g.\ PVPP R24M$VPVRVR7pW0Pn VP4EFgpW@JgK \V\4'gK#VPPVPP8wgKT\VPR4'dKrVP 4'g.\ PVPP R24MTp\V\4'diVPPFNp\!VPV4VPJgK)VPWdPP4pM VP#V4pWTPn RVPnEKj VPPFp\!VPV4;pfK\V\4'gTMVPWgP4p\VR4'gWWP8wduVR.\&P)4,9d0\+R V R V R \-\&P)44 24hWWn \ PR V R V R24K\VR4'gKV=EK R#)aS Set the requested `attn_implementation` for this model. Args: attn_implementation (`str` or `dict`): The attention implementation to set for this model. It can be either a `str`, in which case it will be dispatched to all submodels if relevant, or a `dict` where keys are the sub_configs name, in which case each submodel will dispatch the corresponding value. allow_all_kernels (`bool`, optional): Whether to load kernels from unverified hub repos, if `attn_implementation` is a custom kernel outside of the `kernels-community` hub repository. rz does not support setting its attention implementation dynamically, because it does not follow the functional approach based on AttentionInterface (see https://huggingface.co/docs/transformers/en/attention_interface)Fro_attn_was_changedTNrKrLz"` is not supported for zd. The only possible arguments are "eager" (manual attention implementation)or one of the following: z8We set the attention implementation for the sub-config `z` to `z` without finding the associated sub-model. For this reason we could not check if the model supports it. You may encounter undefined behavior.)rrrrryrlrrrCrrxr{rgrrrr[r9rsrQrRrr)rrrqrequested_implementationr_sub_implementation subconfig_key subconfigs&&& rset_attn_implementation'PreTrainedModel.set_attn_implementationAs1488 $((T[[-M-MN ! ${{'G'G G4466~~../0\\ ,0+U+U,EUf,V,(=U 9I%y/::$$..$++2G2GG 0 02EFF!==??NN$..7789``*B&!"5t<<-1[[-D-DM&t{{MBiFVFVV5H5L5L$13C3C3X3X6" 2!& .E*3)R)RSe)f&EW$$B6:   2E(J"[[44M$T[[-@@ M&&94@@-,00@^@^_# +>??*.L.LL)'=T=_=_=a1aa(>?Q>RRjkxjyz88<=T=_=_=a8b7ce ?Q;NNRS`Raaghzg{|@@y*=>>%795rc0<V^8dQhRS[S[,/#)rr)rr)rrs"rrrWs6W6Wt6Wrc x\V\4'gTM%VPRVPP4pVPPpRW239dW28wd\ RV: RV: R24hW PP8wd"VP V4pW PnVP4FpW@JgK \V\4'gK"VPPVPP8wgKSTp\V\4'diVPPFNp\VPV4VPJgK)VPWdPP4pM VPV4pWTPnK R#)a Set the requested `experts_implementation` for this model. Args: experts_implementation (`str` or `dict`): The experts implementation to set for this model. It can be either a `str`, in which case it will be dispatched to all submodels if relevant, or a `dict` where keys are the sub_configs name, in which case each submodel will dispatch the corresponding value. rdeepgemm_megamoez*Cannot switch experts implementation from  to z at runtime: `deepgemm_megamoe` is a load-time choice. Reload via `from_pretrained(..., experts_implementation=...)` to switch.N)rrrrr}rr|r~rgrrCrr[rD)rrrtcurrentr_rurvs&& rset_experts_implementation*PreTrainedModel.set_experts_implementations|4d;; #'++B 0S0ST !++55 '!D DIl"4d;;)-)@)@ "4;; >)BRBRR1G1K1K -/?/?/W/W2." *A&/%Q%QRd%e"DV  A)(rc RRp.p\4pRpVP4Fp\V\4'd\ VR4'gK-VP 4pTe\ TR4'gKV\T4pYs9dKiTPT4TPTPT44RpK W n V'dV^,Vn V'g/\PVPP R24R#R# \ dEKi;i)z Enables the gradients for the input embeddings. This is useful for fine-tuning adapter weights while keeping the model weights fixed. c(VPR4R#)TN)requires_grad_)rUinputoutputs&&&rmake_inputs_require_gradsMPreTrainedModel.enable_input_require_grads..make_inputs_require_gradss  ! !$ 'rFr?Nregister_forward_hookTa does not expose input embeddings. Gradients cannot flow back to the token embeddings when using adapters or gradient checkpointing. Override `get_input_embeddings` to fully support those features, or set `_input_embed_layer` to the attribute name that holds the embeddings.)rfrgrrrr?rBrrkrir_require_grads_hooks_require_grads_hookrrrCr)rrhooks seen_modulesfound_embeddingsrUinput_embeddings embedding_ids& renable_input_require_grads*PreTrainedModel.enable_input_require_gradss  (u  llnFv77GFLb#>#@  'w7GI`/a/a./L+   \ * LL)??@YZ [# %%(%*! ',QxD $   >>**+,ww  %'  sD D&%D&c \VRR4pV'gR#VFpVP4K .Vn\VR4'dV=R#R#)z$ Removes the `_require_grads_hook`. rNr)r[removerrr)rrhooks& rdisable_input_require_grads+PreTrainedModel.disable_input_require_grads sO4d; D KKM%'! 4. / /( 0rc.<V^8dQhRS[R,/#rmodalityNr)rrs"rrrW sC$Jrc fVR9d.ROpM#VR8Xd.R OpMVfRR.pM\RV 24hVF!p\W4'gK\W4u# VPVJdK\VPR4'd/VPP VR7pW@P8wdV#V#) aA Best-effort lookup of the *encoder* module. If provided with `modality` argument, it looks for a modality-specific encoder in multimodal models (e.g. "image_encoder") By default the function returns model's text encoder if any, and otherwise returns `self`. Possible `modality` values are "image", "video" and "audio". audio text_encoderencoderHUnnrecognized modality, has to be "image", "video" or "audio" but found get_encoderrimagevideo vision_towervisual vision_modelvision_encoder image_tower) audio_tower audio_encoderspeech_encoder)rrr[r@r)rrpossible_module_namesr^ base_encoders&& rrPreTrainedModel.get_encoder s ) )$o !  $V !  %3Y$? !ghpgqrs s)Dt""t*** ??$ &74??M+R+R??666IL.## rc.<V^8dQhRS[R,/#rr)rrs"rrrW: s%%S4Z%rc ZVR 9d.R OpM#VR8XdRR.pMVfRR.pM\RV 24hVF#p\W4'gK\WV4R# VPVJdC\VPR4'dVPP WR 7R#WnR#R#) zC Symmetric setter. Mirrors the lookup logic used in `get_encoder`. rrrNrrr set_encoderrrr)rrrr@rr~)rrrrr^s&&& rrPreTrainedModel.set_encoder: s ) )$o !  %2O$D !  %3Y$? !ghpgqrs s)Dt""G,* ??$ &t 66++G+G$ 'rc .ROpVF!p\W4'gK\W4u# VPVJd7\VPR4'dVPP4#V#)ad Best-effort lookup of the *decoder* module. Order of attempts (covers ~85 % of current usages): 1. `self.decoder/self.language_model/self.text_model` 2. `self.base_model` (many wrappers store the decoder here) 3. `self.base_model.get_decoder()` (nested wrappers) 4. fallback: raise for the few exotic models that need a bespoke rule get_decoder)r> text_modeldecoder text_decoder)rr[r@r)rrr^s& rrPreTrainedModel.get_decoderT sc!\)Dt""t*** ??$ &74??M+R+R??..0 0 rc .ROpVF#p\W4'gK\WV4R# VPVJdB\VPR4'dVPPV4R#WnR#R#)zC Symmetric setter. Mirrors the lookup logic used in `get_decoder`. N set_decoder)r>rr)rrr@rr~)rrrr^s&& rrPreTrainedModel.set_decoderk sh !L)Dt""G,* ??$ &t 66++G4$ 'rc  \VPR4'd"VPP;'gRpM\VPR4'dVPPpMX\VPR4'dVPPpM%\ VPP 4RR4p\V\P\P\P\P\P\P34'de\ VRR4e$\P !VP"RVR7VP$e#\P&!VP$4R#R#\V\P(4'd[VP+4FDwr4RV9d\P,!V4K$R V9gK-\P.!VR4KF R#\V\P04'd\P !VP"RVR7VP2eS\ VP"R R 4'g4\P&!VP"VP2,4R#R#R#\V\P44'dVP74R#\V\P8\P:\P<\P>34'g7R VP@PB9gR VP@PB9d\ VRR4e!\PD!VP"4\ VR R4e!\P&!VP$4\ VRR4ec\P&!VPF4\PD!VPH4\P&!VPJ4R#R#RVP@PB9d\VR4'dVPLR8wd\NVPL,M VPPpV!VP4wrg\PR!VPTV4\PR!VPVV4R#R#R#)aD Initialize the weights. This is quite general on purpose, in the spirit of what we usually do. For more complex initialization scheme, it should be overridden by the derived `PreTrainedModel` class. In case a model adds an explicit `nn.Parameter`, this method should also be overridden in order to initialize it correctly. initializer_rangeg{Gz?init_stdinitializer_factorweightNgmeanstdbias_is_hf_initializedF LayerNormRMSNorm running_meanRotaryEmbeddingoriginal_inv_freqdefault),rrrrrr[get_text_configrrLinearConv1dConv2dConv3dConvTranspose1dConvTranspose2drnormal_rrzeros_LSTMr(xavier_uniform_ constant_r& padding_idxMultiheadAttention_reset_parameters GroupNorm BatchNorm1d BatchNorm2d BatchNorm3drCrones_r running_varnum_batches_tracked rope_typerNcompute_default_rope_parameterscopy_inv_freqr)rrUrr^rrope_fn buffer_valuerps&& r _init_weightsPreTrainedModel._init_weights| sN 4;; 3 4 4++//774C T[[* - -++&&C T[["6 7 7++00C$++5579LdSC fryy"))RYY 2K]K]_a_q_qr s svx.: V]]#>{{& FKK('  ( (%668 t#((/t^NN5#. 9  - - LLSc :!!-gfmmMach6i6i FMM&*<*<=>7j-  5 5 6 6  $ $ & v bnnbnnbnn] ^ ^f..777F,,555vx.: 6==)vvt,8 FKK(v~t4@ F//0 6--. F667A &"2"2";"; ;Pc@d@d##y0$F$4$45;;  &fmm4OL JJv 5 JJv// >Ae ;rc <V^8dQhRS[/#)rrr)rrs"rrrW s))$)rc \VRR4'dR#V'd\;QJd,RVPRR74F 'dK RM! RM!RVPRR744'dd\;QJd,RVPRR74F 'dK RM! RM!RVPRR744'd RVnR#VP V4RVnR#)z= Initialize the weights if they are not already initialized. rFNc3<"TFp\VRR4xK R#5i)rFNr[rs& rr6PreTrainedModel._initialize_weights.. snMmEGE#7??Mms)recurseTc3H"TFpVfK \VRR4xK R#5i)NrFr)r buffers& rrr s);F= 4e<<;s"")r[allrbuffersrr)rrUrs&&&r_initialize_weights#PreTrainedModel._initialize_weights s 6/ 7 7  nVM^M^glM^MmnnVM^M^glM^Mmnnn$nnUn;$nnUn; )-F %  6"$(!rc a\\PPR4'g/RV3Rllo\ \PPRS4\ VR4pV!VP VP44R#)a This is equivalent to calling `self.apply(self._initialize_weights)`, but correctly handles composite models. This function dynamically dispatches the correct `init_weights` function to the modules as we advance in the module graph along the recursion. It can handle an arbitrary number of sub-models. Without it, every composite model would have to recurse a second time on all sub-models explicitly in the outer-most `_init_weights`, which is extremely error prone and inefficient. smart_applycV^8dQhR\PR\\P\.R3,R\/#)rrUr^Nr)rrVrr)rs"rr8PreTrainedModel.initialize_weights..__annotate__ s9  BII 8RYY.smart_apply sJ#__.E!%99#E+D+DnU#E~> / 6* rN)rrrrVrr[rr)rsmart_apply_fnrs& @rinitialize_weights"PreTrainedModel.initialize_weights sauxx 66   EHHOO]K @!}5t//1D1D1FGrc&<V^8dQhRS[RS[/#)rrr)rr)rrs"rrrW sp%p%Dp%Tp%rc  aaaV'd/pVPRR7Fuwr4\V\4'gKVPRR7pVR8wd/VP 4UUu/uFwrgV RV 2V RV 2bK pppVP V4Kw V#VP p\VPRR4p V 'g/#Vf/#\P!R4o\;QJdBV3RlVP4VP4,4F 'dK RM7 R M3!V3RlVP4VP4,44'dVP4#/pVPRR7UU u0uFwrjVkK up pVP!RR7UU u0uFwrjVkK up p,p VP 4FwooR S,oR S,o\#\%V3R lV 44p \#\%V3R lV 44p \'V 4^8d1\'V 4^8d!\'V 4\'V 4,^8wd\)R S RS RV RV 24h\+V \-V 44F)wrWP49d W/,W.&K%WV&K+ K V#uuppiuup piuup pi)a Return the expanded tied weight keys (in case they contain modules or regex patterns) for only the current model, or recursively for all submodels if `all_submodels=True` (i.e. it will re-check the config values for all submodels). For almost all models, we only require to tie the embeddings, so the model has an internal property `_tied_weights_keys = {"lm_head.weight": "model.embed_tokens.weight"}`. In this case, the mapping is already "expanded", i.e. it already contains full parameters, and this function will simply return a copy of the property. For more complex patterns, e.g. for `DFineForObjectDetection`, we have the following attribute ``` _tied_weights_keys = { r"bbox_embed.(?![0])\d+": "bbox_embed.0", r"class_embed.(?![0])\d+": "class_embed.0", "model.decoder.class_embed": "class_embed", "model.decoder.bbox_embed": "bbox_embed", } ``` In this case, the function looks up all the model's parameters and buffers, and matches all the params, returning the following: ``` { 'bbox_embed.1.layers.0.bias': 'bbox_embed.0.layers.0.bias', 'bbox_embed.1.layers.0.weight': 'bbox_embed.0.layers.0.weight', 'bbox_embed.1.layers.1.bias': 'bbox_embed.0.layers.1.bias', 'bbox_embed.1.layers.1.weight': 'bbox_embed.0.layers.1.weight', 'bbox_embed.1.layers.2.bias': 'bbox_embed.0.layers.2.bias', 'bbox_embed.1.layers.2.weight': 'bbox_embed.0.layers.2.weight', 'bbox_embed.2.layers.0.bias': 'bbox_embed.0.layers.0.bias', 'bbox_embed.2.layers.0.weight': 'bbox_embed.0.layers.0.weight', ... 'class_embed.1.bias': 'class_embed.0.bias', 'class_embed.1.weight': 'class_embed.0.weight', 'class_embed.2.bias': 'class_embed.0.bias', 'class_embed.2.weight': 'class_embed.0.weight', ... 'model.decoder.class_embed.0.bias': 'class_embed.0.bias', 'model.decoder.class_embed.0.weight': 'class_embed.0.weight', 'model.decoder.class_embed.1.bias': 'class_embed.0.bias', 'model.decoder.class_embed.1.weight': 'class_embed.0.weight', ... 'model.decoder.bbox_embed.0.layers.0.bias': 'bbox_embed.0.layers.0.bias', 'model.decoder.bbox_embed.0.layers.0.weight': 'bbox_embed.0.layers.0.weight', 'model.decoder.bbox_embed.0.layers.1.bias': 'bbox_embed.0.layers.1.bias', 'model.decoder.bbox_embed.0.layers.1.weight': 'bbox_embed.0.layers.1.weight', ... } ``` i.e. all the parameters matching the regex and modules patterns in `_tied_weights_keys` F)remove_duplicaterrrYtie_word_embeddingsz ^[A-Za-z0-9_\.]+(weight)|(bias)$c3F<"TFpSPV4xK R#5ir)r)r rDcommon_case_regexs& rrAPreTrainedModel.get_expanded_tied_weights_keys..@ s"_3^a &&q))3^s!T^c2<\P!SV4#rr)x source_names&rr@PreTrainedModel.get_expanded_tied_weights_keys..L BIIk14Mrc2<\P!SV4#rr)r  target_names&rrr M rrzAThere is an issue with your definition of `tie_weights_keys` for :z . We found z to tie into )rZrrrr9rrXr[rrcompilerr;rrr( named_buffersr!filterrrrr )rrexpanded_tied_weightsprefixr_submodel_tied_weightsrDrE tied_mappingrrpall_param_names source_params target_paramstarget_nsource_nrr rs&& @@@rr.PreTrainedModel.get_expanded_tied_weights_keys sd $& !%)%7%7%7%O!i99,5,T,Tch,T,i)|I^IdIdIf1IfvhasOxq_)>PU)>)VW)V1)VW,,e,D[ D$!AD[  )5(:(:(< $K +K +K"6*M#_`M"6*M#_`M &*=)A-}%M(::a? WXcWddefqers -mM?L '*-}9M&N"99;;6K6U)37?(3'O!)=6%$o12X[ s)K K 6 KTcD<V^8dQhRS[S[,R,RS[/#)r missing_keysNrecompute_mapping)rfrr)rrs"rrrWe s(Q8Q8C4Q8SWQ8rc V'gVPpMVPRR7p\VP44p\ V4EFwpwrVVeRpWa9pWQ9p V'd~V 'dv\ P !VPV4VPV44'g:\PRV RV R24VPPV4KMmV'g V 'dYereMZV'gSV 'gKW4^,RFwrW8XgK W9p V 'gKT pM Rp\PRV RV R 24VPV4p R V9d'VPR ^4wrVPV4pMTpTp\VW4VPVV 4VfEKpX'gEK{VP!V4EK R#) a Tie the model weights. If `recompute_mapping=False` (default when called internally), it will rely on the `model.all_tied_weights_keys` attribute, containing the `{target: source}` mapping for the tied params. If `recompute_mapping=True`, it will re-check all internal submodels and their config to determine the params that need to be tied. This is the default when `model.tie_weights()` is called on its own, outside of `__init__`, and `from_pretrained`, in case the config values were changed somewhere. Note that during `from_pretrained`, tying is *symmetric*: if the mapping says "tie target -> source" but `source` is missing in the checkpoint while `target` exists, we *swap* source and target so we can still tie everything to the parameter that actually exists. TrNzDThe tied weights mapping and config for this model specifies to tie r}z, but both are present in the checkpoints with different values, so we will NOT tie them. You should update the config with `tie_word_embeddings=False` to silence this warning.FzYThis checkpoint seem corrupted. The tied weights mapping for this model specifies to tie zk, but both are absent from the checkpoint, and we could not find another related tied weight for those keysrY)rrrr9 enumeraterequalrrrrlget_parameter_or_bufferr get_submoduler _adjust_biasdiscard)rr r!ritarget_param_namesource_param_nameremove_from_missingsource_is_theretarget_is_there target_backup source_backuptarget_backup_is_there source_param parent_namer^rs&&& rrPreTrainedModel.tie_weightse s!22I;;$;OI*+ 9B99M 5A5!'&*#"3"G"3"G#!;;t'9'9:K'LdN`N`arNsttbctbuuy012 22667HI u)_;L'8(8Aa%'8J4 )=5B5V2 654A 1 %9K /4+w016G5HI__ 778IJL''$5$<$= 7.5` (Volta), or on TPUs which benefit from having sequence lengths be a multiple of 128. For more details about this, or help on choosing the correct value for resizing, refer to this guide: https://docs.nvidia.com/deeplearning/performance/dl-performance-matrix-multiplication/index.html#requirements-tc mean_resizing (`bool`): Whether to initialize the added embeddings from a multivariate normal distribution that has old embeddings' mean and covariance or to initialize them with a normal distribution that has a mean of zero and std equals `config.initializer_range`. Setting `mean_resizing` to `True` is useful when increasing the size of the embeddings of causal language models, where the generated tokens' probabilities won't be affected by the added embeddings because initializing the new embeddings with the old embeddings' mean will reduce the kl-divergence between the next token probability before and after adding the new embeddings. Refer to this article for more information: https://nlp.stanford.edu/~johnhew/vocab-expansion.html Return: `torch.nn.Embedding`: Pointer to the input tokens Embeddings Module of the model. Nr modifier_rank) _resize_token_embeddingsrrr-rrGatheredParametersrr rr vocab_sizer)rr@rArB model_embedsrrrHs&&&& rresize_token_embeddings'PreTrainedModel.resize_token_embeddings sH44^Yfg  !&8&@ t^4VV9J9JRV9V % ' ' 22<3F3FVZ2[[)0066q9 \[&,,2215J4> ##%0$ \[s D D c`VP4pVPWAW#4p\VR4'dVPp\ WV4VP P pVPV4VPV4\VR4;'dVPRJpVe\4'dcV'g[^RI p V PPVP RR7;_uu_4VP P^,pRRR4MVP P^,pVP4eVP4p \!V \"P$P&4'dVPWVR7p MVP)WVR7p \V R4'dV Pp\ W4V P P p V PV 4VP+V 4VP4# +'giL;i)_hf_hookrNrD)rB)r?_get_resized_embeddingsrrMr|rrrrIrr-rrrGr rMrrrr&_get_resized_lm_headrQ) rr@rArBold_embeddingsrProld_embeddings_requires_gradrr old_lm_head new_lm_headold_lm_head_requires_grads &&&& rrF(PreTrainedModel._resize_token_embeddings s22455 ,>  >: . .!**D ~ 4'5'<'<'J'J$%%&BC !!.1t^4VV9J9JRV9V   ))++L ^^66~7L7L\`6aa%3%:%:%@%@%CNba"0!6!6!= 7.5` (Volta), or on TPUs which benefit from having sequence lengths be a multiple of 128. For more details about this, or help on choosing the correct value for resizing, refer to this guide: https://docs.nvidia.com/deeplearning/performance/dl-performance-matrix-multiplication/index.html#requirements-tc mean_resizing (`bool`): Whether to initialize the added embeddings from a multivariate normal distribution that has old embeddings' mean and covariance or to initialize them with a normal distribution that has a mean of zero and std equals `config.initializer_range`. Setting `mean_resizing` to `True` is useful when increasing the size of the embeddings of causal language models, where the generated tokens' probabilities will not be affected by the added embeddings because initializing the new embeddings with the old embeddings' mean will reduce the kl-divergence between the next token probability before and after adding the new embeddings. Refer to this article for more information: https://nlp.stanford.edu/~johnhew/vocab-expansion.html Return: `torch.nn.Embedding`: Pointer to the resized Embedding Module or the old Embedding Module if `new_num_tokens` is `None` Nz5Asking to pad the embedding matrix to a multiple of `z@`, which is not and integer. Please make sure to pass an integerzYou are resizing the embedding layer without providing a `pad_to_multiple_of` parameter. This means that the new embedding dimension will be a.. This might induce some performance reduction as *Tensor Cores* will not be available. For more details about this, or help on choosing the correct value for resizing, refer to this guide: https://docs.nvidia.com/deeplearning/performance/dl-performance-matrix-multiplication/index.html#requirements-tcrrDzOld embeddings are of type , which is not an instance of zj. You should either use a different resize function or make sure that `old_embeddings` are an instance of rYrzThe new embeddings will be initialized from a multivariate normal distribution that has old embeddings' mean and covariance. As described in this article: https://nlp.stanford.edu/~johnhew/vocab-expansion.html. To disable this, use `mean_resizing=False`r)rrrrr rrrrr-rrrGr4r8rr&rurrrrr(_init_added_embeddings_weights_with_meanrr;r) rrPr@rArBrrold_num_tokensold_embedding_dimrPadded_num_tokensnparamss &&&&& rrN'PreTrainedModel._get_resized_embeddings& s~V  )0#66 KL^K_``a%!/!6!6!3H3HX\2]]4B4I4I4N4N4P1^]1?0E0E0J0J0L -N ^ +4N4P4P,: )! !.",,77-d>.B-CCabdbnbnaopLL>$   !((// ''--    *=   ~ .  ,   =  .> )++L ^^668M8M7N^b6ccAA&dc =="N  / % ' ' $++^-B-BCF2262KK4B4I4I4N4NrPQrSTu4U%%**2A2q51LK1?0E0E0J0J2A2q50QN ! ! & &rr1u - & ' ' $++^-B-BCF2262KK(6(=(=%0>0E0E0J0J0P0PQR0S-"--9~PQ?QUcUoUo>o15N.L *8)>)>)C)CN ! ! &,:,A,A,F,F,L,LQ,ON )))5>A;MQ_QkQk:k-1*w^]]^dcc$LKKLKs15S+ S?=T6A+T'+ S< ? T  T$ ' T8 c n<V^8dQhRS[PRS[R,RS[RS[RS[P/#)rrRr@N transposedrBr)rrrr)rrs"rrrW sPAAYYAd A A  A  Arc  VfV#\VR4;'dVPRJp\4'dV'g^RIpVPP VP RR7;_uu_4V'gVP P4M'VP P4P4wrxRRR4MLV'gVP P4M'VP P4P4wrxXV8Xd\4'g W!n V#\V\P4'g;\R\V4 R\P R\P R24hV'gXV3MVX3p VPRJp \P!V RV R VP P R VP P"/p W'8dV'gVP%V 4MW'8dV'd\&P)R 4W', p \4'dV'g^RIpVP .p V 'dWP., p VPP V RR7;_uu_4VP+WWW4V 'dVP-WV 4RRR4M-VP+WWW4V 'dVP-WV 4\/Wr4p\4'd}V'gu^RIpVP VPV P V P.p VPP V ^R7;_uu_4VP1WWV 4RRR4MVP1WWV 4\3V R R 4V # +'giEL;i +'giL;i +'giLF;i)a Build a resized Linear Module from a provided old Linear Module. Increasing the size will add newly initialized vectors at the end. Reducing the size will remove vectors from the end Args: old_lm_head (`torch.nn.Linear`): Old lm head liner layer to be resized. new_num_tokens (`int`, *optional*): New number of tokens in the linear matrix. Increasing the size will add newly initialized vectors at the end. Reducing the size will remove vectors from the end. If not provided or `None`, just returns a pointer to the input tokens `torch.nn.Linear` module of the model without doing anything. transposed (`bool`, *optional*, defaults to `False`): Whether `old_lm_head` is transposed or not. If True `old_lm_head.size()` is `lm_head_dim, vocab_size` else `vocab_size, lm_head_dim`. mean_resizing (`bool`): Whether to initialize the added embeddings from a multivariate normal distribution that has old embeddings' mean and covariance or to initialize them with a normal distribution that has a mean of zero and std equals `config.initializer_range`. Setting `mean_resizing` to `True` is useful when increasing the size of the embeddings of causal language models, where the generated tokens' probabilities will not be affected by the added embeddings because initializing the new embeddings with the old embeddings' mean will reduce the kl-divergence between the next token probability before and after adding the new embeddings. Refer to this article for more information: https://nlp.stanford.edu/~johnhew/vocab-expansion.html Return: `torch.nn.Linear`: Pointer to the resized Linear Module or the old Linear Module if `new_num_tokens` is `None` NrrDz#Old language model head is of type rXzg. You should either use a different resize function or make sure that `old_lm_head` are an instance of rYrrraThe new lm_head weights will be initialized from a multivariate normal distribution that has old embeddings' mean and covariance. As described in this article: https://nlp.stanford.edu/~johnhew/vocab-expansion.html. To disable this, use `mean_resizing=False`rT)rrr-rrrGrr4rr7rrrrurrrrrrr%_init_added_lm_head_weights_with_mean"_init_added_lm_head_bias_with_meanr!_copy_lm_head_original_to_resizedr)rrRr@rarBrrrZold_lm_head_dimnew_lm_head_shapehas_new_lm_head_biasrSr\r^num_tokens_to_copys&&&&& rrO$PreTrainedModel._get_resized_lm_head snH  ! t^4VV9J9JRV9V % ' ' 22;3E3EUY2ZZ5?K&&++-[EWEWEYEYE[E`E`Eb0[Z 2< ""'')ASASAUAUAWA\A\A^ ,N ^ +4N4P4P'5 $ +ryy115d;6G5HHfgigpgpfqrII;a! FP_n=VdfuUv*//t; ii  % %%,, $$**    *=   { +  ,   =  .> )++L %,,-'//00F^^66vT6RR>>#/Sc,?? Zjk SR::oO_(;;KVfg @ % ' ' !((+*:*:K)covariance_matrix) sample_shapeNrrMNr)rr;r:rrrTrpositive_definitecheckr distributionsmultivariate_normalMultivariateNormalsamplerr ) rrPrPrZr\old_embeddings_weightmean_embeddingsold_centered_embeddings covarianceepsilonis_covariance_psd distributions &&&&& rrY8PreTrainedModel._init_added_embeddings_weights_with_meanH sY!/ 5 5 : : = =emm L**%:C"7"I,..1HH>Y '99??@TUYY[  ..BBUUZ3GVLFREXEX.0FYFb&&,,-  ! ! & &r,<'<'>'A B  (//0@!DGGH]H]HcHcd  ! ! & &r,<'<'>'A Brc <V^8dQhRS[/#)rrar)rrs"rrrWa s@@@rcV'd_VPPPVPnVPPPVPnVPWWE4V'daVPPPVPnVPPPVPnR#R#r)rr;rprY)rrRrSrfrZr\ras&&&&&&&rrc5PreTrainedModel._init_added_lm_head_weights_with_meana s &1&8&8&=&=&?&?K   #&1&8&8&=&=&?&?K   # 55kP^q &1&8&8&=&=&?&?K   #&1&8&8&=&=&?&?K   # rc~\P!VPP^\PR7p\P !VPP^R7P \P4pVPPRV,RPVRV,R7R#)r)r rr NrlrrM)rrrr;rrr:r)rrRrSr\ bias_meanbias_stds&&&& rrd2PreTrainedModel._init_added_lm_head_bias_with_meanw sJJ{//441EMMR 99[--22;>>u}}Mb#3356>>ISWZbSb>crc|V'g>VPPRV1R3,VPPRV1R3&MI>P>P>U>UViWiViklVl>mK   # #$7%7$7$: ;>I>P>P>U>UVWYlZlYlVl>mK   # #A':(:':$: ; 9D9I9I9N9NObPb9cK   ! !"5#5 6 rc <V^8dQhRS[/#)rnew_num_position_embeddingsr)rrs"rrrW s  c rc |\RVP RVP RVPP R24h)z4`resize_position_embeddings` is not implemented for B`. To implement it, you should overwrite this method in the class  in `modeling_.py`rBrCr)rrs&&rresize_position_embeddings*PreTrainedModel.resize_position_embeddings sH!B4>>BRS226..1APTP^P^PiPiOjjn p  rch<V^8dQhRS[PS[S[P,,/#r)rr&rg)rrs"rrrW s&  bll8K)K rc |\RVP RVP RVPP R24h)z1`get_position_embeddings` is not implemented for rrrrrs&rget_position_embeddings'PreTrainedModel.get_position_embeddings sH!??OP226..1APTP^P^PiPiOjjn p  rc \4\P!R48wdVP4VP RR7R#)z Initialize and tie the weights if needed. If using a custom `PreTrainedModel`, you need to implement any initialization logic in `_init_weights`. r0F)r!N)rrrrrrs&rrPreTrainedModel.init_weights s5 6 75<<;O O  # # % 51rc <VP'g#\VPP R24hVfRR/p\P !\ 3/VBpR\P!VP4P9pV'gVPRVR7M;VP\ VPRR74\PR 4VPR 8HpT;'g\VR R4pV'dVP!4R#R#) a Activates gradient checkpointing for the current model. We pass the `__call__` method of the modules instead of `forward` because `__call__` attaches all the hooks of the module. https://discuss.pytorch.org/t/any-different-between-model-input-and-model-forward-input/3690/2 Args: gradient_checkpointing_kwargs (dict, *optional*): Additional keyword arguments passed along to the `torch.utils.checkpoint.checkpoint` function. z) does not support gradient checkpointing.N use_reentrantFrGT)enablegradient_checkpointing_funcrGVYou are using an old version of the checkpointing format that is deprecated (We will also silently ignore `gradient_checkpointing_kwargs` in case you passed it).Please update to the new format on your modeling file. To use the new format, you need to completely remove the definition of the method `_set_gradient_checkpointing` in your model.rT_hf_peft_config_loaded)rrrCr functoolsr rrd signature_set_gradient_checkpointingrapplyrrmain_input_namer[r)rgradient_checkpointing_kwargsr_is_using_old_formatneeds_embedding_gradsenable_input_gradss&& rr-PreTrainedModel.gradient_checkpointing_enable s333 7 788abc c ( 0-@[\ 8&A04- % ->>**+,]] -rc zVP'dR\P!VP4P9pV'gVPRR7M;\ P R4VP\VPRR74\VRR4'dVP4R#R#)z; Deactivates gradient checkpointing for the current model. rGF)rrrrN) rrdrrrrrrr r[r)rrs& rgradient_checkpointing_disable.PreTrainedModel.gradient_checkpointing_disable s  / / /$+g.?.?@`@`.a.l.l#l '000>L 74#C#C5QR 415 9 9  , , . :rc <V^8dQhRS[/#rr)rrs"rrrW snn4nrc \;QJd*RVP44F 'gK R# R#!RVP444#)zD Whether gradient checkpointing is activated for this model or not. c3b"TF%p\VR4;'d VPxK' R#5i)rN)rr)r ms& rr. s,m^lYZ7167TTA If a single weight of the model is bigger than `max_shard_size`, it will be in its own checkpoint shard which will be bigger than `max_shard_size`. variant (`str`, *optional*): If specified, weights are saved in the format model..safetensors. token (`str` or `bool`, *optional*): The token to use as HTTP bearer authorization for remote files. If `True`, or not specified, will use the token generated when running `hf auth login` (stored in `~/.huggingface`). save_peft_format (`bool`, *optional*, defaults to `True`): For backward compatibility with PEFT library, in case adapter weights are attached to the model, all keys of the state dict of adapters needs to be prepended with `base_model.model`. Advanced users can disable this behaviours by setting `save_peft_format` to `False`. save_original_format (`bool`, *optional*, defaults to `True`): For backward compatibility with the previous versions of `transformers` you can save the checkpoint with its reverse mapping. The reverse mapping needs to exists even if the model was loaded from a None legacy checkpoint. kwargs (`dict[str, Any]`, *optional*): Additional key word arguments passed along to the [`~utils.PushToHubMixin.push_to_hub`] method. NrrFrzThe model is quantized with z and is not serializable - check out the warnings from the logger on the traceback to understand the reason why the quantized model is not serializable.z0.31.4z[Saving a model with tensor parallelism requires `huggingface_hub` version 0.31.4 or higher.zProvided path (z#) should be a directory, not a fileT)exist_okcommit_messagerepo_id create_prrr2rrYFSDP)rzhDetected adapters on the model, saving the model in the PEFT format, only adapter weights will be saved.)rzTo match the expected format of the PEFT library, all keys of the state dict of adapters will be prepended with `base_model.model`.zbase_model.model.zMultiple active adapters detected, saving multiple active adapters is not supported yet. You can save adapters separately one by one by iteratively calling `model.set_adapter(adapter_name)` then `model.save_pretrained(...)` hf_device_maprdiskz}Attempting to save a model with offloaded modules. Ensure that unallocated cpu memory exceeds the `shard_size` (50GB default)z.binz {suffix}.binr/z{suffix}.safetensors)filename_patternrmetadatatotal_parameters weight_maprz(.*?)-\d{5}-of-\d{5}zWriting model shards)descr0)rzModel weights saved in wrr)indent sort_keys z:The model is bigger than the maximum size per checkpoint (z) and is going to be split in z^ checkpoint shards. You can find where each parameters has been saved in the index located at )rz README.md)rrrrM)_r[rrRis_serializablerquantization_config quant_method_tp_sizersrrrrrerrormakedirsrrlrseprq create_repor_get_files_timestampsget_state_dict_and_metadata unwrap_modelrrrrCr removeprefix architectures _auto_classr'save_pretrainedrrrget_adapter_state_dictr9active_adaptersr peft_configrrrfrrrrIS_SAGEMAKER_MP_POST_1_10smpstatemodule_managertranslate_functionsrrEr _device_meshrr%rYrrVrrrr1rtensor_to_filenamelistdirrrrrfilename_to_tensors fullmatchrrjtqdmrrr5 contiguoussafe_save_filerXr jsondumpswriterorsave_upload_modified_files)2rrrrrrrrrrrrrquantization_serializablesave_directory_pathrrrfiles_timestampsr model_to_saverpeft_state_dictrrGactive_adaptercurrent_peft_config is_offloaded smp_to_hfrp ignore_keyrrstate_dict_splitindexr full_filenameweights_no_suffixfilename_no_suffixreg shard_file tensor_namesshard_state_dict tensor_namerpath_to_weightssave_index_filerFcontent model_cards2&&&&&&&&&&, rrPreTrainedModel.save_pretrained sv  #7O!(/G!Ot^T:  $ q qL+)N q qS_SoSoSq "  #,BKd.|/O/O/\/\.]^uu  == $-PQY-Z-Zm  77>>. ) ) LL?>*::]^ _  NT2 ii7 #ZZ(8$?Njj,?,E,Ebggkk,RSU,VWG ;6Ih**7LTLVLTTG#99.I   ##/#K#KD#Q J!%T* ##%(Z%5%5c%:1%= "/<.E.E.N.N.[.[\b.c-d *    ' tDKK H ?)$$44^D  ""//??O% ~+AAZAX #KK^')O&0&6&6&8 UEJ*;C5(AB'9!0J!%!5!5!7~&*$u"0!2&*&6&6~&F##33NC  &113J D/ * *C**113459$,,33554CUCUC\C\C^9^L MM:  % $ # 8 8 L L 1&z2 !M  ' ' 3D1-LJ&,L' g>L4L'//GOOP^`vw= )9.   & & &/1D1D1FdJZJcJcd.AAE  >2HGGLLBM!- 4 4VR @ H HY[ \ "*!1!1&"!=!E!EnVX!Y **45C##$566GGNN=11$4$H$HH#OMM"45A -(#3()0  0 0 6 6 8?U) $J ww||NJ?H! + # 4  FMM$6$6&$@5m[QF170A0A0C - , +X I /) 2 = ggll><HO KK1/1BC D5O ggll><Y`;abOosW==**U1EL > KKL^L\] 0 D DEFG$$3#4A7  27DOOSXYJ OOBGGLLE F  ' ' -# (  >=s 2d d, c<VPe VPM.pVPR.4p\V\4'dV.pVFpWS9gK VP V4K V'dW2R&\ SV`!V/VB#)Nr)rrrrrirbr)rrrr tags_kwargsrrCs&*, rrPreTrainedModel.push_to_hubsw"&//"=t2jj, k3 ' '&-KC C  !6Nw"D3F33rc \RVP444pV'd)\RVP444pW#,pV#)a Get the memory footprint of a model. This will return the memory footprint of the current model in bytes. Useful to benchmark the memory footprint of the current model and design some tests. Solution inspired from the PyTorch discussions: https://discuss.pytorch.org/t/gpu-memory-that-model-uses/56822/2 Arguments: return_buffers (`bool`, *optional*, defaults to `True`): Whether to return the size of the buffer tensors in the computation of the memory footprint. Buffers are tensors that do not require gradients and not registered as parameters. E.g. mean and std in batch norm layers. Please see: https://discuss.pytorch.org/t/what-pytorch-means-by-buffers/120266/2 c3j"TF)qP4VP4,xK+ R#5irrNrQrs& rr7PreTrainedModel.get_memory_footprint..6s(YGXe.."U%7%7%999GX13c3j"TF)qP4VP4,xK+ R#5irr)r bufs& rrr8s%Y.3<<>C,<,<,>>>.r)sumrr)rreturn_buffersmemmem_bufss&& rget_memory_footprint$PreTrainedModel.get_memory_footprint*s@YtGXYY Y$,,.YYH.C rc<\VRR4\P8Xd^RIHp\ SV`!V/VBVP4FSp\WC4'gK\V4^8d V^,pMVPRR4pVP V4KU V#\VRR4\P8Xd\VRR4'd \R4h\ SV`!V/VB#)quantization_methodN HQQLinearrris_loaded_in_8bitFzCalling `cuda()` is not supported for `8-bit` quantized models. Please use the model as it is, since the model has already been set to the correct devices.) r[r{HQQhqq.core.quantizerrbrrgrrrBITS_AND_BYTESr)rrrrrUrrCs&*, rrPreTrainedModel.cuda<s 4. 59K9O9O O 3 GL$ )& ),,.f004y1}!%a!'Hf!=KK' )K 4. 59K9Z9Z Zt0%88 sw|T,V,,rc<RV9pV'g.VF'p\V\P4'gK%RpM \VRR4\P 8Xd^RIHp\S V`$!V/VBVP4Fip\We4'gKRV9d VR,pM V^,pRV9d VR,pM V'dXpMRpVeWn VPV4Kk V#V'd,\VRR4\P8Xd \R4h\VRR4\P8XdDV'd \R4h\VRR 4'd\!R 4'g \R 4hM3\VRR4\P"8XdV'd \R 4h\S V`$!V/VB#) rTrNrrzBCasting a Quark quantized model to a new `dtype` is not supported.zYou cannot cast a bitsandbytes model in a new `dtype`. Make sure to load the model using `from_pretrained` using the desired `dtype` by passing the correct `dtype` argument.r Fz0.48zsYou need to install `pip install bitsandbytes>=0.48.0` if you want to move a 8-bit model across devices using to().zYou cannot cast a GPTQ model in a new `dtype`. Make sure to load the model using `from_pretrained` using the desired `dtype` by passing the correct `dtype` argument.)rrrr[r{r!r"rrbr:rg compute_dtyperQUARKrr#rdGPTQ) rrrdtype_present_in_argsargrrUrrrCs &*, rr:PreTrainedModel.toVs!(6 1$c5;;//,0) 4. 59K9O9O O 3 GJ ' ',,.f006)!'!1!%a&( &w. # $(/4,KK'#)$K WT3H$%OSeSkSk%kab b 4. 59K9Z9Z Z$ P t0%88AZ[aAbAb JT0$ 7;M;R;R R$ Hwz4*6**rc\<\VRR4'd \R4h\SV` !V!#)rFz`.half()` is not supported for quantized model. Please use the model as it is, since the model has already been casted to the correct `dtype`.)r[rrbhalfrrrCs&*rr-PreTrainedModel.halfs6 4 / /I  7<& &rc\<\VRR4'd \R4h\SV` !V!#)rFz`.float()` is not supported for quantized model. Please use the model as it is, since the model has already been casted to the correct `dtype`.)r[rrbfloatr.s&*rr1PreTrainedModel.floats6 4 / /I  7=$' 'rc T<V^8dQhRS[PRS[RS[RS[R,/#)rrrrrqN)rrr)rrs"rrrWs7KK/3IMbfimbmrc\WP4\P!4\ 4.pV'dVP \ 44\4'd^RIpV'gqV'gi\PR4VP\P!4VPP\4R7\!4.4V#V'd0VP\"P$!R4\'4.4V#VP\"P$!R4\P(!4.4V#)rNrrr0)rrrno_tie_weightsrOrir<r-rrrr\rrrr+rrrrmeta_device_safe_creation_ops)rgrrrrqrrs&&&&& rget_init_context PreTrainedModel.get_init_contexts +5,,?ATATAVXeXgh   !6!8 9 % ' '  (: ^_$$,,.!++@P@R+S')$$ell6&:VP\P VP\P 44VPecV\P\P39d>VP\P VP\P 44V#)z\Create the dtype_plan describing modules/parameters that should use the `keep_in_fp32` flag.) rrrrrfromkeysrrr)rrrs&& r_get_dtype_planPreTrainedModel._get_dtype_plans  % % 1eu}}6L   dmmD,F,F V W  , , 8Uu}}V[VdVdFe=e   dmmD,M,Mu}}] ^rc.<V^8dQhRS[R,/#)rrwN)r])rrs"rrrWs"&"&,:M"&rc 8V'd\4'g \R4h^RIHpV!4VeH\ V\ 4'd2W nVPV4VPV4RVn R#RVn RVnR#RVn RVnR#)aw Set whether or not to use the `kernels` library to kernelize some layers of the model. Args: use_kernels (`bool`): Whether or not to use the `kernels` library to kernelize some layers of the model. kernel_config (`KernelConfig`, *optional*): The kernel configuration to use to kernelize the model. If `None`, the default kernel mapping will be used. zk`use_kernels=True` requires kernels>=0.9.0. Please install the latest version with `pip install -U kernels`)$register_kernel_mapping_transformersNTF) rfrintegrations.hub_kernelsr@rr]rwsanitize_kernel_mappingcreate_compatible_mapping use_kernels)rrDrwr@s&&& rset_use_kernelsPreTrainedModel.set_use_kernelss ')) B W 0 2(Z |-T-T%2"55d;77=#' $( %)"$D !%D rrrrrrrrrrr fusion_configrc<V^8dQhRS[S[,RS[S[P,R,RS[S[,S[P,R,RS[S[P,R,RS[RS[RS[R S[S[,R,R S[R S[R,R S[R S[S[S[S[S[S[3,,3,R,RS[R,RS[/#)rrgrNrrrrrrrrrrGrr) rrrrr-r rrr)rrs"rrrWsSS - .S'*R[['84'?S!3&4t; S $t+ S "& SSSTzD SSSSCS#X!667$>STkS %!Src  *VPRR4pVPRR4pVPRR4pVPRR4pVPRR4pVPRR4pVPR R4pVPR R4pVPR R4pVPR R4pVPR R4pVPRR4pVPRR4pVPRR4pVPRR4pVPRR4pVPR/4;'g/P4pVPRR4p VPRR4p!VPRR4p"VPRR4p#VPRR4p$VPRR4p%VPRR4p&VPRR4p'VPRR4p(VPRR4p)VPR R4p*VPR!R4p+V%eV#fR"p#RKFp,VPV,R4p-K Ve VeTMTpVfR"p\4'd V'gR#pR$VR%VRVR&VR'VR(VRV/p./V.CR)V/Cp/VeVfV"e \R*4hVR"8XdE\ \ P PR+R,44'd\PR-4V#fV$e\V#V$V&VR.7wpp&p$V"e\4'g \R/4hVf/p\VV/3/VBwp0pp\V4pR0R1R2R3R4V/p1VeVV1R5&\V\4'g|VeTMTp2VP p3V3f\VP" R624hV3P$!V23R7R#RV"RVRV/V.BVBwpp4RV49dV4PR4V4PRV4pM%\P&!V4pTp4\)VRV4pVV/R)&R8V9dVPR84VnR9V9dVPR94Vn\/VVVV V14wp5ppV"'dQV5e \R:4hVe>\V\04'dR;VP349gR;V9d \5R<4hV*e V)'g\P7R=4R#p)\9VVV"V V/V1VP;4\)VR>R4VR?7 wp6p7V5RJp8\=VV6VV7WV54wppV"'dP^R@IH p9\BPD!RA4;_uu_4V!V4p:RRR4V9!V6^,R#X:VRB7RC,pWn#V e\P&!V 4Vn$\)VRDR4p V e^REI%H&p;V;!WV 4V*eV)'d^RFI'H(p<V\YV>4V5eV5P[V>VVV6V)RG7RRR4X>P]V4p?\_V>V+V54p@\`'dV&e\cV>V#V%V&V$4p>Ve\eV>VVV54p\gVVV7VVVVV?V5V&V X@V V.V RH7pAVPiV>VV6VA4wpBpCVPkV>VAVB4pBV>Pm4V>PoV)V*4V>Pq4'd7\sV>RI4'd%V"'gV>Pt!V!VVV3/V.BRV'/BVBVe8\w\yVP3444^8d\{V>V5VVXCV4V5eV5V>n>V5PV>4V0eVeVVR'&V>PV0V XAVRJ7pBV'dV>XBP43#V># +'giEL;i +'giEL;i)La: Instantiate a pretrained pytorch model from a pre-trained model configuration. The model is set in evaluation mode by default using `model.eval()` (Dropout modules are deactivated). To train the model, you should first set it back in training mode with `model.train()`. The warning *Weights from XXX not initialized from pretrained model* means that the weights of XXX do not come pretrained with the rest of the model. It is up to you to train those weights with a downstream fine-tuning task. The warning *Weights from XXX not used in YYY* means that the layer XXX is not used by YYY, therefore those weights are discarded. Parameters: pretrained_model_name_or_path (`str` or `os.PathLike`, *optional*): Can be either: - A string, the *model id* of a pretrained model hosted inside a model repo on huggingface.co. - A path to a *directory* containing model weights saved using [`~PreTrainedModel.save_pretrained`], e.g., `./my_model_directory/`. - `None` if you are both providing the configuration and state dictionary (resp. with keyword arguments `config` and `state_dict`). model_args (sequence of positional arguments, *optional*): All remaining positional arguments will be passed to the underlying model's `__init__` method. config (`Union[PreTrainedConfig, str, os.PathLike]`, *optional*): Can be either: - an instance of a class derived from [`PreTrainedConfig`], - a string or path valid as input to [`~PreTrainedConfig.from_pretrained`]. Configuration for the model to use instead of an automatically loaded configuration. Configuration can be automatically loaded when: - The model is a model provided by the library (loaded with the *model id* string of a pretrained model). - The model was saved using [`~PreTrainedModel.save_pretrained`] and is reloaded by supplying the save directory. - The model is loaded by supplying a local directory as `pretrained_model_name_or_path` and a configuration JSON file named *config.json* is found in the directory. state_dict (`dict[str, torch.Tensor]`, *optional*): A state dictionary to use instead of a state dictionary loaded from saved weights file. This option can be used if you want to create a model from a pretrained configuration but load your own weights. In this case though, you should check if using [`~PreTrainedModel.save_pretrained`] and [`~PreTrainedModel.from_pretrained`] is not a simpler option. cache_dir (`Union[str, os.PathLike]`, *optional*): Path to a directory in which a downloaded pretrained model configuration should be cached if the standard cache should not be used. ignore_mismatched_sizes (`bool`, *optional*, defaults to `False`): Whether or not to raise an error if some of the weights from the checkpoint do not have the same size as the weights of the model (if for instance, you are instantiating a model with 10 labels from a checkpoint with 3 labels). force_download (`bool`, *optional*, defaults to `False`): Whether or not to force the (re-)download of the model weights and configuration files, overriding the cached versions if they exist. proxies (`dict[str, str]`, *optional*): A dictionary of proxy servers to use by protocol or endpoint, e.g., `{'http': 'foo.bar:3128', 'http://hostname': 'foo.bar:4012'}`. The proxies are used on each request. output_loading_info(`bool`, *optional*, defaults to `False`): Whether or not to also return a dictionary containing missing keys, unexpected keys and error messages. local_files_only(`bool`, *optional*, defaults to `False`): Whether or not to only look at local files (i.e., do not try to download the model). token (`str` or `bool`, *optional*): The token to use as HTTP bearer authorization for remote files. If `True`, or not specified, will use the token generated when running `hf auth login` (stored in `~/.huggingface`). revision (`str`, *optional*, defaults to `"main"`): The specific model version to use. It can be a branch name, a tag name, or a commit id, since we use a git-based system for storing models and other artifacts on huggingface.co, so `revision` can be any identifier allowed by git. To test a pull request you made on the Hub, you can pass `revision="refs/pr/"`. attn_implementation (`str`, *optional*): The attention implementation to use in the model (if relevant). Can be any of - `"eager"` (manual implementation of the attention) - `"sdpa"` (using [`F.scaled_dot_product_attention`](https://pytorch.org/docs/master/generated/torch.nn.functional.scaled_dot_product_attention.html)) - `"flash_attention_2"` (using [Dao-AILab/flash-attention](https://github.com/Dao-AILab/flash-attention)) - `"flash_attention_3"` (using [Dao-AILab/flash-attention/hopper](https://github.com/Dao-AILab/flash-attention/tree/main/hopper)) - `"flash_attention_4"` (using [Dao-AILab/flash-attention/flash_attn/cute](https://github.com/Dao-AILab/flash-attention/tree/main/flash_attn/cute)). By default, if available, SDPA will be used. The default is otherwise the manual `"eager"` implementation. Accept HF kernel references in the form: /[@][:] - and are any non-"/" and non-":" sequences. - "@" is optional (branch, tag, or commit-ish), e.g. "@main", "@v1.2.0", "@abc123". - ":" is optional and selects a function inside the kernel repo. - Both options can appear together and in this order only: @revision first, then :kernel_name. - We intentionally allow a leading "|" prefix (e.g., "flash|...") because the code strips it before loading; '|' is not excluded in the character classes here. Examples that match: "org/model" "org/model@main" "org/model:custom_kernel" "org/model@v1.2.3:custom_kernel" experts_implementation (`str`, *optional*): The experts implementation to use in the model (if relevant). Can be any of: - `"eager"` (sequential implementation of the experts matrix multiplications). - `"batched_mm"` (using [`torch.bmm`](https://pytorch.org/docs/stable/generated/torch.bmm.html)). - `"grouped_mm"` (using [`torch.nn.functional.grouped_mm`](https://docs.pytorch.org/docs/main/generated/torch.nn.functional.grouped_mm.html)). By default, if the model supports it, `"grouped_mm"` will be used. The default is otherwise the manual `"eager"` implementation. > Parameters for big model inference dtype (`str` or `torch.dtype`, *optional*, defaults to `"auto"`): Override the default `torch_dtype` and load the model under a specific `dtype`. The different options are: 1. `torch.float16` or `torch.bfloat16` or `torch.float`: load in a specified `dtype`, ignoring the model's `config.dtype` if one exists. If not specified - the model will get loaded in `torch.float` (fp32). 2. `"auto"` - A `dtype` or `torch_dtype` entry in the `config.json` file of the model will be attempted to be used. If this entry isn't found then next check the `dtype` of the first weight in the checkpoint that's of a floating point type and use that as `dtype`. This will load the model using the `dtype` it was saved in at the end of the training. It can't be used as an indicator of how the model was trained. Since it could be trained in one of half precision dtypes, but saved in fp32. 3. A string that is a valid `torch.dtype`. E.g. "float32" loads the model in `torch.float32`, "float16" loads in `torch.float16` etc. For some models the `dtype` they were trained in is unknown - you may try to check the model's paper or reach out to the authors and ask them to add this information to the model's card and to insert the `dtype` or `torch_dtype` entry in `config.json` on the hub. device_map (`str` or `dict[str, Union[int, str, torch.device]]` or `int` or `torch.device`, *optional*): A map that specifies where each submodule should go. It doesn't need to be refined to each parameter/buffer name, once a given module name is inside, every submodule of it will be sent to the same device. If we only pass the device (*e.g.*, `"cpu"`, `"cuda:1"`, `"mps"`, or a GPU ordinal rank like `1`) on which the model will be allocated, the device map will map the entire model to this device. Passing `device_map = 0` means put the whole model on GPU 0. To have Accelerate compute the most optimized `device_map` automatically, set `device_map="auto"`. For more information about each option see [designing a device map](https://hf.co/docs/accelerate/main/en/usage_guides/big_modeling#designing-a-device-map). max_memory (`Dict`, *optional*): A dictionary device identifier to maximum memory if using `device_map`. Will default to the maximum memory available for each GPU and the available CPU RAM if unset. tp_plan (`Optional[Union[dict, str]]`, *optional*): A torch tensor parallel plan, see [here](https://pytorch.org/tutorials/intermediate/TP_tutorial.html). Use `tp_plan="auto"` to use the predefined plan based on the model. If it's a dict, then it should match between module names and desired layout. Note that if you use it, you should launch your script accordingly with `torchrun [args] script.py`. This will be much faster than using a `device_map`, but has limitations. tp_size (`str`, *optional*): A torch tensor parallel degree. If not provided would default to world size. device_mesh (`torch.distributed.DeviceMesh`, *optional*): A torch device mesh. If not provided would default to world size. Used only for tensor parallel for now. If provided, it has to contain dimension named `"tp"` in case it's > 1 dimensional, this dimension will be used for tensor parallelism offload_folder (`str` or `os.PathLike`, *optional*): If the `device_map` contains any value `"disk"`, the folder where we will offload weights. offload_buffers (`bool`, *optional*): Whether or not to offload the buffers with the model parameters. quantization_config (`Union[QuantizationConfigMixin,Dict]`, *optional*): A dictionary of configuration parameters or a QuantizationConfigMixin object for quantization (e.g bitsandbytes, gptq). subfolder (`str`, *optional*, defaults to `""`): In case the relevant files are located inside a subfolder of the model repo on huggingface.co, you can specify the folder name here. variant (`str`, *optional*): If specified load weights from `variant` filename, *e.g.* pytorch_model..bin. use_safetensors (`bool`, *optional*, defaults to `None`): Whether or not to use `safetensors` checkpoints. Defaults to `None`. If not specified and `safetensors` is not installed, it will be set to `False`. weights_only (`bool`, *optional*, defaults to `True`): Indicates whether unpickler should be restricted to loading only tensors, primitive types, dictionaries and any types added via torch.serialization.add_safe_globals(). When set to False, we can load wrapper tensor subclass weights. disable_mmap (`bool`, *optional*): Whether to disable memory mapping when loading safetensors checkpoints. When `None` (default), it is auto-detected to `True` when the checkpoint lives on an `hf-mount` FUSE filesystem (used by HF Spaces/Endpoints), where mmap + parallel page-faults can deadlock. When `True`, files are read fully into memory and parsed with `safetensors.torch.load`. When `False`, the default memory-mapped loader is always used. fusion_config (`dict[str, bool | dict[str, Any]]`, *optional*): Optional fusion configuration applied before model instantiation. Each key enables a fusion family and its value can either be `True` to enable that fusion with default options or a dictionary of family-specific options. For example, `{"patch_embeddings": True}` enables patch embedding fusion. This should only be used as an inference optimization, as it can slightly change outputs. If omitted, `from_pretrained()` falls back to `config.fusion_config` when available. Refer to the fusion mapping guide in `docs/source/en/fusion_mapping.md` for more details. key_mapping (`dict[str, str], *optional*): A potential mapping of the weight names if using a model on the Hub which is compatible to a Transformers architecture, but was not converted accordingly. kwargs (remaining dictionary of keyword arguments, *optional*): Can be used to update the configuration object (after it being loaded) and initiate the model (e.g., `output_attentions=True`). Behaves differently depending on whether a `config` is provided or automatically loaded: - If a configuration is provided with `config`, `**kwargs` will be directly passed to the underlying model's `__init__` method (we assume all relevant updates to the configuration have already been done) - If a configuration is not provided, `kwargs` will be first passed to the configuration class initialization function ([`~PreTrainedConfig.from_pretrained`]). Each key of `kwargs` that corresponds to a configuration attribute will be used to override said attribute with the supplied `kwargs` value. Remaining keys that do not correspond to any configuration attribute will be passed to the underlying model's `__init__` function. Activate the special ["offline-mode"](https://huggingface.co/transformers/installation.html#offline-mode) to use this method in a firewalled environment. Examples: ```python >>> from transformers import BertConfig, BertModel >>> # Download model and configuration from huggingface.co and cache. >>> model = BertModel.from_pretrained("google-bert/bert-base-uncased") >>> # Model was saved using *save_pretrained('./test/saved_model/')* (for example purposes, not runnable). >>> model = BertModel.from_pretrained("./test/saved_model/") >>> # Update configuration during loading. >>> model = BertModel.from_pretrained("google-bert/bert-base-uncased", output_attentions=True) >>> assert model.config.output_attentions == True ``` rNrroutput_loading_infoF_from_pipeline _from_autorrr max_memoryoffload_folderrrrrrradapter_kwargs adapter_namerrrrtp_sizerrtrust_remote_coderqrDrw key_mappingrTrrrrrrzq`state_dict` cannot be passed together with a model name or a `gguf_file`. Use one of the two loading strategies. WORLD_SIZEraYou've set device_map=`auto` while triggering a distributed run with torchrun. This might lead to unexpected behavior. If your plan is to load the model on each device, you should set device_map={: PartialState().process_index} where PartialState comes from accelerate library)rQrrzIaccelerate is required when loading a GGUF file `pip install accelerate`. file_typer~r3pytorchfrom_auto_classusing_pipelinezN does not define `config_class`; pass an explicit config to `from_pretrained`.return_unused_kwargsrrzYou cannot combine Quantization and loading a model from a GGUF file, try again by making sure you did not passed a `quantization_config` or that you did not load a quantized model from the Hub.rzxOne or more modules is configured to be mapped to disk. Disk offload is not supported for models loaded from GGUF files.zA kernel_config was provided but use_kernels is False; setting use_kernels=True automatically. To suppress this warning, explicitly set use_kernels to True.transformers_weights) rrrrrrrrr)load_gguf_checkpointr0)return_tensors model_to_loadrrdrG)register_fusion_patches)(register_kernel_replacements_and_fusions)r~rrrrD)rrrrrrrrrrrrrrradjust_generation_fn)rP load_configrO)mirror _fast_initlow_cpu_mem_usagefrom_tf from_flaxoffload_state_dict)BrlrrrrrrrrrrFrcr?r2rr rarfrom_pretraineddeepcopyr[ryr}rSrrrrrrrmodeling_gguf_pytorch_utilsr[rrrvrGfusion_mappingr^rAr_r7rr\rPpreprocess_modelr<r!rrDr/r_load_pretrained_model_finalize_model_loadingevalrErrr`rrfr1rpostprocess_model load_adapterto_dict)DrgrrrrrrrrrrrGr model_argsrrrrrJ from_pipelinerWrrrrMrNrrrrrrOrPrrrrQrrrRrqrDrwrSr^rprdownload_kwargs_with_commit_adapter_model_pathr config_pathra model_kwargsrrrrr[ dummy_modelr^r_model_init_contextr~rweight_conversionsra loading_infodisk_offload_indexsD&&$$$$$$$$$$$*, rrhPreTrainedModel.from_pretrainedsjZZ d3 **Y-ZZ d3 $jj)>F #3T:  **\59 7D)jj5 ZZ d3 ZZ d3 $4d; **%6>$jj)>EJJ{B/ jj6 **Y- **%5r:@@bFFHzz.)< "JJ':DAJJ{D1 **Y-**Y-06 ;OQU0Vjj5 "JJ':DA"JJ':EBjj6  ?D9 jj5  )goGpD 4&Ap  "".EKE =E   %5#   n w  0 U   'V&U-&U#  !'D'PT]TiD   C |S(I$J$J KKc   '"5/Lkj0 ,J W  )@)B)Bhi i  !NM` ) 'N N J:N .j9 !7KDUWfg  $+8J' (&"233$*$6&;GK]]6*F!L!&.+FK5@#M2 !F **0**5J*KF ' #v --3ZZ8P-QF *+; '\:, ( fj ' Y%J--&J  $ ? #C ?  $ Z 4S- P 11%GY[lmv& / 0 0<<|  ` c<V^8dQhRRRS[R,RS[S[,R,RS[RS[S[,R,RS[S[S[3,/#) rr~rrNrra expected_keysr)rrrrrgrw)rrs"rrrWsnc0c0 c04Kc0s)d*c0) c0 Cy4' c0  $& ' c0rc  VPpVPpVRJ;'d8VPP\P \P 09pVf(\VP4P44MTp\P\P8d\V\VRR44RpVP ecRVP P#49dD\%VVP&VVP VP(VP*VP,4pVP e5V'g-\/VP V4p \1W VP4.p \34'd}V'guVf@/p VF5p V P5\7V RVP8VP:R74K7 T p\=WV4wr\?V V \A4\A4/R7pW3#\A4pVeTp EM&VeV^,PCR4'dVf/p VFpVP:'g\EV4'dH\GVR4;_uu_4pV P5\IVPK444RRR4Km\MVR RR 7pVPOV4VP4FpVPQV4V V&K K M@Ve2/p VF(p V P5\7WP:R 74K* M \SR 4h\UVV VVPVVR 7wrVFpVPYRRR4K W3# +'giL;i)zzPerform the actual loading of some checkpoints into a `model`, by reading them from disk and dispatching them accordingly.Nrrr)r,rr)r  error_msgsunexpected_keysmismatched_keysconversion_errorsr/r1r2)r3r)rz5Neither a state dict nor checkpoint files were found.)r~rrarr})-rrrrr{r!r'rrr;rlevelrjWARNINGrHr[rrr0rrrrr3caching_allocator_warmupr-rrJrrr6rwrfr6r)r r7r8rrkr<rr$r__exit__)r~rrrarrris_hqq_or_quarkr}expanded_device_maprmerged_state_dict ckpt_filer r| all_pointerfilerC file_pointerrDs&&&&& rrm&PreTrainedModel._load_pretrained_models3#// "// &d2  |7W7W7d7d  " "  $ $i 8  ?VZVlVl"!((01! )G cs,EE E 8'E c \V\4'g VPp^RIHuHp\ W!4'g\V R24hWnR#)a% Register this class with a given auto class. This should only be used for custom models as the ones in the library are already mapped with an auto class. Args: auto_class (`str` or `type`, *optional*, defaults to `"AutoModel"`): The auto class to register this new model with. Nz is not a valid auto class.) rrrtransformers.models.automodelsrrrr)rg auto_class auto_modules&& rregister_for_auto_class'PreTrainedModel.register_for_auto_classsE*c**#,,J66{// |+FGH H$rc  \V4'dR#VfVPPfR#VPPVRR ^.3,9Ed/Rp\VPRR4pVPPe0VPPVPP8XgfVPP e0VPP VPP8XgVemW@PP8XdSVRVPP RVPP RVPP RV R 2 , p\ PV4R#R#) zf Shows a one-time warning if the input_ids appear to contain padding and no attention mask was given. NrzWe strongly recommend passing in an `attention_mask` since your input_ids may be padded. See https://huggingface.co/docs/transformers/troubleshooting#incorrect-output-when-padding-tokens-arent-masked. sep_token_idz5 You may ignore this warning if your `pad_token_id` (z&) is identical to the `bos_token_id` (z), `eos_token_id` (z), or the `sep_token_id` (z ), and your input is not padded.rM)rvr pad_token_idr[ bos_token_id eos_token_idrr)rrTr warn_stringrs&&& r%warn_if_padding_and_no_attention_mask5PreTrainedModel.warn_if_padding_and_no_attention_masksK i   &DKK,D,D,L  ;; # #yRG'< <F #4;;EL))5$++:R:RVZVaVaVnVn:nKK,,8T[[=U=UY]YdYdYqYq=q ,AYAY1YLT[[MeMeLfg..2kk.F.F-GGZ[_[f[f[s[sZtu..:^;[]    ,- =rc VP'dR#VPP'dR#VPP'dR#R#)z: Returns whether the model has a tensor parallelism plan. TF)rr@rrrs&rsupports_tp_plan PreTrainedModel.supports_tp_plans9 === ?? # # # ;; ) ) )rc VP#)z0 Returns the model's tensor parallelism degree. )rrs&rrQPreTrainedModel.tp_sizes }}rcVP'dR#VPP'dR#VPP'dR#R#)TF)rr@rrrs&rsupports_pp_plan PreTrainedModel.supports_pp_plans9 === ?? # # # ;; ) ) )rc\VR4'd VP#\VRR4pVe V\9d\P RV R24Rp\V,#)_loss_functionrNz `loss_type=zZ` was set in the config but it is unrecognized. Using the default loss: `ForCausalLMLoss`. ForCausalLM)rrr[rIrr)rrs& r loss_functionPreTrainedModel.loss_functionsh 4) * *&& &D+t4    =   i[)=> &II&&rcWnR#r)rrrGs&&rrr$s#rc  \4'g \R4h^RIHpHpHpRpRpVP V4VP'g VPMVf VPMTpVPew^RIH pVPP'*pV!VPPVR7;_uu_4V!W!VPPR7VR 7RRR4M%V!W!VPPR7VR 7R VnVP V4R# +'giL*;i TP T4i;i) zYTemporarily register hidden kernel wrappers so `kernelize` can discover and replace them.z`Kernels are not available. To use kernels, please install kernels using `pip install -U kernels`)DeviceMode kernelizec\VR/4P4FewrV\VP449gK%\ V\ P 4'g\RV R24hVPW4Kg R#)_hidden_kernelsz#Attempted to register a kernel for z, but it was not a `torch.nn.Module`. This means the underlying function needs to be decorated with `@use_kernel_func_from_hub`. Please submit and issue to the transformers repo: `https://github.com/huggingface/transformers/issues`.N) r[r9rrrrrVrregister_module)rUr^r^s& rattach_hidden_kernels8PreTrainedModel.kernelize..attach_hidden_kernels0sz#F,=rBHHJtF$9$9$;<<%b"))44(A$HFF **44Krcj\VR/4F!p\W4'gK\W4K# R#)rN)r[rr)rUr^s& rdetach_hidden_kernels8PreTrainedModel.kernelize..detach_hidden_kernels;s+(92>6((F) ?rN)use_kernel_mapping)inherit_mapping)r)rmodeT)rfrkernelsrrrrtraining INFERENCETRAININGrwruse_local_kernelkernel_mappingrr _use_kernels) rrrrrrrrrs && rrPreTrainedModel.kernelize(s #%%r  43 5 * . JJ, -)-4>>T\DMM_cD!!-6&*&8&8&I&I"I'(:(:(I(I[jkkd6t{{7G7G+HtTlk$v4;;3C3C'D4P $D  JJ, -lk JJ, -s*"EA3E&D1)5E1 E <EEc <V^8dQhRS[/#rr)rrs"rrrWTs44T4rc\VRR4#)rFrrs&rrDPreTrainedModel.use_kernelsSst^U33rc$<V^8dQhRS[RR/#)rrGrNr)rrs"rrrWXs & & &$ &rc\V4'd\VRR4'dR#V'dVP4R#\VRR4'd\P R4RVnR#)rFNzmDisabling kernels at runtime is a no-op as there is no 'unkernelize' routine; keeping current kernels active.)rr[rrrrrs&&rrDrWsX ;;74??   NN t^U33##D!&D rc <V^8dQhRS[/#r)r()rrs"rrrWfs   rc hVPPR8Xd\RRRR7#\4#)a5Build the default `CompileConfig` for `get_compiled_call`. Inductor + `reduce-overhead` (the `CompileConfig` defaults) target CUDA. torch_tpu registers its own TorchDynamo backend named `"tpu"`; route `device.type == "tpu"` through it with static shapes to match the common StaticCache + fixed-prefill usage.tpuFr)backenddynamicr)rrr(rs&r_default_compile_config'PreTrainedModel._default_compile_configfs- ;;  u $ IN Nrc4<V^8dQhRS[R,RS[/#)rcompile_configNr)r(r)rrs"rrrWqs ## 0D##rc RVPP9d VP#T;'gVP4p\ VP RR4;'gVP4p\ VR4'd\ VRV4V8wd;Wn\P!VP3/VP4BVn VP#)atReturn a `torch.compile`'d version of `self.__call__`. This is useful to dynamically choose between non-compiled/compiled `forward` during inference, especially to switch between prefill (where we don't want to use compiled version to avoid recomputing the graph with new shapes) and iterative decoding (where we want the speed-ups of compiled version with static shapes).llama4rN_compiled_call_last_compile_config) r model_type__call__rr[rrrrrrrr)rrdefault_configs&& rget_compiled_call!PreTrainedModel.get_compiled_callqs t{{-- -== 'II4+G+G+I !7!79I4PrrTXTpTpTr.//t3^DV(6 %"'-- "ZAWAWAY"ZD """rcVP#r)_supports_attention_backendrgs&ris_backend_compatible%PreTrainedModel.is_backend_compatibles...rc `<V^8dQhRS[S[,RS[R,RRRS[R,RR/#)rr rNrrrr)rrrrR)rrs"rrrWsJ/9/93i/94K/9- /9 "D( /9  /9rc  4VRJp\4'd V'gR#\4'd\4'gV'g}VP4F)wrg\P !VRR7p\ WV4K+ VP4F)wri\P !V RR7p\ WV4K+ R#WPP4, FhpVPV4p\W&RR7p \P!WzR7pVe!\WWvRRVP4V4K\\ WV4Kj VP4F5wri\W&RR7p \P!WR7p\ WV4K7 R#)aMove the missing keys (keys that are part of the model parameters, but were NOT found in the loaded state dicts) back from meta device to their device according to the `device_map` if any, else cpu. Takes care of sharding those missing parameters if `device_mesh` is provided, i.e. we are using TP. All non-persistent buffers are also moved back to the correct device (they are not part of the state_dict, but are not missing either). NrrT)valid_torch_deviceF)r-r.rr(r zeros_likerrrr;r%r4 empty_likerGget_local_ranknamed_non_persistent_buffers) rr rrrrrrrGr param_device buffer_devices &&&&& rr6PreTrainedModel._move_missing_keys_from_meta_to_devicesW$4/ % ' '    %9%;%;L"335 ((u=*4e<6 $113 ((>*4e< 4   "<"<"A"A"CCCC005E%j$OL$$U@E&+T5+:T:T:VXc +4e<D <<>KC&z4PM$$VBE &t% 8?rc$<V^8dQhRS[RR/#)rrrNr)rrs"rrrWs"&"&T"&d"&rc  ~\4'dH\4'g8VP4FpVPV4pRVnK RVn\ 4'dV'g^RIp\VPRR7P4Uu0uFp\VRR4'dKVkK up4pVPPV^R7;_uu_4VP4RRR4R#VP4R# \ dKi;iuupi +'giR#;i)a Initialize the missing keys (keys that are part of the model parameters, but were NOT found in the loaded state dicts), according to `_initialize_weights`. Indeed, since the corresponding weights are missing from the state dict, they will not be replaced and need to be initialized correctly (i.e. weight initialization distribution). Also marks non-missing params/buffers with `_is_hf_initialized` and propagates this flag to modules, so that `_initialize_weights` can skip fully-initialized modules entirely. TN) keep_varsrFrD)r.rrr%rAttributeErrorr-rrrr[rrGr)rrrparam_or_bufferrrEnot_initialized_parameterss&& rr(PreTrainedModel._initialize_missing_keyss    %9%;%; (&*&B&B3&GO9=O6) '+D # & ' ' *. OOdO;BBDtDqGTUWkmrLsDt* &223M]^2__'')`_  # # %&u___s)DD&5D&'D+ D#"D#+ D< c$<V^8dQhRS[RR/#)rr|rN)rw)rrs"rrrWs  @Q VZ rc \;QJd*RVP44F 'gK RM RM!RVP444pV'dR0M \4p\;QJd*RVP44F 'gK RM RM!RVP444pV'dVPR4VP;'g \4pVP ;'g \4V,pRRr\ V4^8d-\P!RPR V444p\ V4^8d-\P!RPR V444pVe6VPU u0uFqPV 4eKV kK up Vn Ve8VPU u0uFqPV 4eKV kK up Vn R#R#uup iuup i) zAdjust the `missing_keys` and `unexpected_keys` based on current model's exception rules, to avoid raising unneeded warnings/errors. This is performed in-place. c3H"TFwrVPR4xK R#5i)zrotary_emb.inv_freqNr6r rrps& rrFPreTrainedModel._adjust_missing_and_unexpected_keys..s!"p[oif6??3H#I#I[o "TFzrotary_emb\.inv_freqc3H"TFwrVPR4xK R#5i) position_idsNr r s& rrr s &mXl96v~'F'FXlr z(^|\.)position_ids$Nrsc3."TF pRV R2xK R#5irrrNrr patterns& rrr s6gVf7!G9AVfrc3."TF pRV R2xK R#5irrrs& rrr s9mYlgQwiq/Ylr) rrrfrkrrrrrrr rr) rr|has_inv_freq_buffersadditional_unexpected_patternshas_position_ids_buffersmissing_patternsunexpected_patternsignore_missing_regexignore_unexpected_regexrs && rr3PreTrainedModel._adjust_missing_and_unexpected_keyss #s"p[_[m[m[o"psss"p[_[m[m[o"ppFZ*A)B`c`e&#&3&mX\XjXjXl&m333&mX\XjXjXl&m#m # * . ./E F??HH35#FFOO#%Sqq8wB38n#CN3c3h'(/OT&*1UV!(W^"&-_b+cr)rrrrrrrrrrrrrrrrrrrrrrrwrr~rrvrDrHr)rNr1r3)NT)NNT)NFT)TNF50GBNNTT)T) AutoModel)TT)rrrrrrar)rrrr7rrr8rrrrrXrrrrrr-r3rrrrr9rrYrrcompilerallow_in_graphrZr^rcrtrrrsetterrrr classmethodrr@rr rr$r)r.rxr|r9rDrlr(rxrrrrrrrno_gradrrrguard_torch_init_functionsrrrr'rJrFrNrOrYrcrdrerrrrrrrrrr r^rrrrVrr:r-r1r7r<rErhr4rmrnrrrrrQrrrrDrrrrrrrr%rr-rorrrr __classcell__rCrs@@rrrsY(37L6FKL#'J'O)/6:?C :>@D *.CG#FJ&;?!N!& %:>% $HH,0H-2##().'+  ^^""'.#'.R99 /0-M-M^?>B ^^!!F ^^ :; ,,@GGR;; $$L@@DIIV<  6q.q.f 1 1$$$$L""0$ 5 5d8d8L6W6Wp*X )@%%4.%" ]]_=?=?~))2 ]]_ $$&H'H8p%p%dQ8Q8f M99v%+N]]~AAF2@@,d d    2(.T:>gq,/(nn \\| > % %& 4' 4$ 588??   -!-2 588??  :+:+x'(> "&"&HS?C S /3 S ). S %S"'S$(SS(,S"SBFS%)SSSjc0c0c0J$$L!.%%*#-J     ' '$$).V44 & &  ##$///9/9b"&"&H  D8]]0 # # !++Wurr~r;z model file)object object_class object_filesc<V^8dQhR\R\R\/#rr~ recursiver)rr)rs"rrrUsYYYDY_YrcR#rrr~rIs&&rrrTsVYrcdV^8dQhR\PR\R\P/#rHrrVr)rs"rrrYsMM MdMryyMrcR#rrrKs&&rrrXsJMrcdV^8dQhR\PR\R\P/#rHrM)rs"rrr\s) dryyrc\4'd/pV'dWR&\V3/VB#\VR4'd\VP4#V#)a Recursively unwraps a model from potential containers (as used in distributed training). Args: model (`torch.nn.Module`): The model to unwrap. recursive (`bool`, *optional*, defaults to `False`): Whether to recursively extract all cases of `module.module` from `model` as well as unwrap child sublayers recursively, not just the top-level distributed containers. rIrU)rcr}rrrU)r~rIrs&& rrr\sP   "+; *5;F;; 5( # # - -LrcpV^8dQhR\\,\P,R\/#)rrr)rrrrr)rs"rrrus+@@#)ell":@t@rcZVR8XdR#\P!V4PR9#)zCheck if the device is an accelerator. We need to function, as device_map can be "disk" as well, which is not a proper `torch.device`. rF)r0r)rrrrs&ris_accelerator_devicerSus)||F#((??rcJV^8dQhR\R\R\R,/#)rr~accelerator_device_maprNrrrR)rs"rrrs* 48HSVZHZrc\R4pVPP4p\4'd VPM.pVP 4FwrgWd9dK VP V4pVeVPWV4p MVP4p VP4V ,p \V4^8d*\WeRR7RJp Y'd \4M^,p W7;;,V , uu&K V#)z This utility function calculates the total bytes count needed to load the model on each device. This is useful for caching_allocator_warmup as we want to know how much cache we need to pre-allocate. c^#)rrrrrr&get_total_byte_count..s1rNT) is_weight) rrr;rrr9r%param_element_sizerQr+rrCr) r~rUrtotal_byte_counttied_param_namesrrrr dtype_sizeparam_byte_countis_part_of_plans &&& rget_total_byte_countras#9-22779@BBemmG4::<  ) --j9  #%88ERJ++-J ;;=:5 w 1) as cudaMalloc is not a bottleneck at all anymore - Loading speed bottleneck is now almost only tensor copy (i.e. changing the dtype) and moving the tensors to the devices. However, we cannot really improve on those aspects obviously, as the data needs to be moved/copied in the end. NmpsF)rrr)rxpugAg333333A)r9rSrrrarr[rcurrent_device mem_get_infomemory_reservedmemory_allocatedrr?rr)r~rrrrrUr\ byte_countaccelerator_modulerfree_device_memorytotal_device_memory unused_memoryrps&&& rrrsx*:M9R9R9T9T XmntXu#u||F##9T "+E<X/446 ;;/ )!( !< $*LL$>uEHZHkHklqHrrM)M9'7 +m;!1$'99!"J"/!2J Z})LMJ [[E !  KKJ!O,EMM&`e fS7s F,F,cnaa]tRtRtoRtR]R]R]R]R]R]R ]R ]R ] R ] / t V3R lV3Rllt Rt VtV;t#)AttentionInterfaceiaO Dict-like object keeping track of allowed attention functions. You can easily add a new attention function with a call to `register()`. If a model needs to locally overwrite an existing attention function, say `sdpa`, it needs to declare a new instance of this class inside the `modeling_.py`, and declare it on that instance. flash_attention_4flash_attention_3r7rOrJzpaged|flash_attention_4zpaged|flash_attention_3zpaged|flash_attention_2z paged|sdpaz paged|eagerc,<V^8dQhRS[RS[RS[/#)rrrr)rr)rrs"rrAttentionInterface.__annotate__s" 9 9 9x 9H 9rc <Vf\PR4MVR8wdW9d\RV R24h\SV`W4#)zcReturn the requested `attn_implementation`. Also strictly check its validity, and raise if invalid.a You tried to access the `AttentionInterface` with a `config._attn_implementation` set to `None`. This is expected if you use an Attention Module as a standalone Module. If this is not the case, something went wrong with the dispatch of `config._attn_implementation`rKr#zP` is not a valid attention implementation registered in the `AttentionInterface`)rrKeyErrorrbr)rrrrCs&&&r get_interface AttentionInterface.get_interfaces[  &   K  !G +0C0O'((xy w{.88rr)rrrrrr9r;r@r:rAr7_global_mappingrwrrrBrCs@@rrprps^ 4440&!#:!#:!#:24 O 9 9 9rrpc\a]tRtRtoRt]V3RlRl4t]V3RlRl4tRtVt R#) PreTrainedAudioTokenizerBaseia Class that additionally defines the behavior of any `audio_tokenizer` to be added. Characteristic for any of them: 1. Encode raw audio into discrete audio codebooks (with x channels) 2. Decode from discrete audio codebooks back to raw audio It is possible that they can decode in different ways given a different representation but they are forced to support 2. nonetheless, e.g. see `DAC`. c4<V^8dQhRS[P/#)r input_valuesrr)rrs"rr)PreTrainedAudioTokenizerBase.__annotate__s  5<< rc R#)zq Encode raw audio retrieved from a respective `FeatureExtractor` into discrete audio codebooks (with x channels) Nr)rr}rrs&&*,rencode#PreTrainedAudioTokenizerBase.encoderc4<V^8dQhRS[P/#)r audio_codesr~)rrs"rrr$sEE%,,Erc R#)z6Decode from discrete audio codebooks back to raw audioNr)rrrrs&&*,rdecode#PreTrainedAudioTokenizerBase.decode#rrrN) rrrrrrrrrrrs@rr{r{s4  EErr{c@V^8dQh/^\9d \;R&#)rrQ)__conditional_annotations__rp)rs"rrrs B`CB+BC`rr)rTN)NNNr1(rryrrrdrrrrrabcrrcollections.abcrr contextlibr dataclassesrr r r itertoolsr threadingr typingrrrrrzipfilerrhuggingface_hubrr packagingr safetensorsrsafetensors.torchrr7rrrrtorch.distributionsrtorch.utils.checkpointrrrrconfiguration_utilsr conversion_mappingr!core_model_loadingr"r#r$r%rr&dynamic_module_utilsr' generationr(r) integrationsr*r+r,r-r.integrations.accelerater/r0r1r2r3r4r5rr6integrations.eager_pagedr7integrations.finegrained_fp8r8integrations.flash_attentionr9integrations.flash_pagedr:integrations.flex_attentionr;rAr<r=integrations.moer>integrations.peftr?integrations.sdpa_attentionr@integrations.sdpa_pagedrAintegrations.tensor_parallelrBrCrDrErFrGrHloss.loss_utilsrImodeling_flash_attention_utilsrJrKrLrMmodeling_rope_utilsrNmonkey_patchingrOrP pytorch_utilsrQ quantizersrRquantizers.autorSquantizers.quantizers_utilsrTsafetensors_conversionrUutilsrVrWrXrYrZr[r\r]r^r_r`rarbrcrdrerfrgrhrirj utils.genericrkrlrm utils.hubrnrorprqutils.import_utilsrrrsrtrurvutils.loading_reportrwrxutils.output_capturingryrzutils.quantization_configr{accelerate.hooksr|accelerate.utilsr}_typingr~ is_availabler!smdistributed.modelparallel.torch modelparallelrsmdistributed.modelparallelr SMP_VERSIONr r get_loggerrrrrupperrrrrrrrrrrrrrrruint8int8int16uint16rrint32uint32rfloat64int64uint64 float8_e4m3fn float8_e5m2r>r)rJrSrarvr|rrrrrrr6rVrrrrrrSrarrprQr{r)rs@rrs?  #.%($HH O!69+-$1< +47vvFCCA=?F32?A* 5B,#-=3.jiddKH93<' %00==?33F ' k :gmmF>S S %   H %zz~~nc288: JJNN#6<BBD%&CK\]  $---6. `##00.  1  EJJ%++%** 5;; 5<< 5== ENN 5;; 5<< 5== 5== 5;; 5<< u"" u  &2/kd,>%*M`(I.XPfjjZj*j*Zl:+bii!57GYil:+^u((C(CD&&2*9*E*E*M*M*T*T[|+U+O'  Y Y M M2@DFgR"9)"9L/A.BBE?Er