+ QV-jRt^RIt^RIt^RIt^RIt^RIHt^RIH t ^RI H t H t H t ] !4'd ^RIt^RIHt] P !]4tRt] !4'd]!4'd^RIHtM^RIHt!R R ]4t!R R ]4tRsR tRtRtRtRtRt RRlt!Rt"RRlt#RRlt$RRlt%Rt&R#)z Integration with Deepspeed N) partialmethod)dep_version_check)is_accelerate_availableis_torch_availablelogging)nnc\PPR4RJpV'd#\PPR4pR#R# \PPdR#i;i) deepspeedNTF) importlibutil find_specmetadataPackageNotFoundError)package_exists_s t/Volumes/fast/ai/experiments/ui-tars-smoke/.venv/lib/python3.14/site-packages/transformers/integrations/deepspeed.pyis_deepspeed_availabler$sc^^--k:$FN ""++K8A!!66  sAA10A1)HfDeepSpeedConfig)objectc6aa]tRt^9toRtV3RltRtVtV;t#)ra" This object contains a DeepSpeed configuration dictionary and can be quickly queried for things like zero stage. A `weakref` of this object is stored in the module's globals to be able to access the config from areas where things like the Trainer object is not available (e.g. `from_pretrained` and `_get_resized_embeddings`). Therefore it's important that this object remains alive while the program is still running. [`Trainer`] uses the `HfTrainerDeepSpeedConfig` subclass instead. That subclass has logic to sync the configuration with values of [`TrainingArguments`] by replacing special placeholder values: `"auto"`. Without this special logic the DeepSpeed configuration is not modified in any way. Args: config_file_or_dict (`Union[str, Dict]`): path to DeepSpeed config file or dict. ch<\V4\R4\R4\SV` V4R#) accelerater N)set_hf_deepspeed_configrsuper__init__selfconfig_file_or_dict __class__s&&rrHfDeepSpeedConfig.__init__Js)%,'+& ,-) __name__ __module__ __qualname____firstlineno____doc__r__static_attributes____classdictcell__ __classcell__r __classdict__s@@rrr9s ..r rcpaa]tRt^RtoRtV3RltRtRtR Rlt] !]RR7t R Rlt R t R t VtV;t#) HfTrainerDeepSpeedConfigz The `HfTrainerDeepSpeedConfig` object is meant to be created during `TrainingArguments` object creation and has the same lifespan as the latter. cB<\SV`V4RVn.VnR#N)rr_dtype mismatchesrs&&rr!HfTrainerDeepSpeedConfig.__init__Xs ,- r cLVPf \R4hVP#)Nz8trainer_config_process() wasn't called yet to tell dtype)r0 ValueError)rs&rdtypeHfTrainerDeepSpeedConfig.dtype]s" ;; WX X{{r c:VPV4pVfR#VR8H#)NFauto) get_value)r ds_key_longvals&& ris_auto HfTrainerDeepSpeedConfig.is_autobs"nn[) ;&= r c  VPV4wrVVfR#VPV4R8XdW%V&R#V'gR#VPV4pVe2Wr8wd*VPPRV RV RV RV 24R#R#R#)a A utility method that massages the config file and can optionally verify that the values match. 1. Replace "auto" values with `TrainingArguments` value. 2. If it wasn't "auto" and `must_match` is true, then check that DS config matches Trainer config values and if mismatched add the entry to `self.mismatched` - will assert during `trainer_config_finalize` for one or more mismatches. Nr8z- ds =z vs hf )find_config_nodegetr1append)rr:hf_valhf_key must_matchconfigds_keyds_vals&&&&& r fill_match#HfTrainerDeepSpeedConfig.fill_matchis..{; >  ::f  '#6N  F#  &"2 OO " "U;-qxqQWPX#Y Z#3 r F)rEc VPVP,VP,pVPRVPRV'*4VPRVPR4VPRVRV'*4VPRVPR4VPRVP R 4VPR VP VP.R 4VPR VPR 4VPRVPR4VPR^4VPRVP R 4VP'dJVPPR/4VPR&VPVPR,R&TPRVP;'g VPR4TPRVP ;'g VP"R4VP%R4'd\&P(VnR#VP%R4'd\&P,VnR#\&P.VnR#)zr Adjust the config with `TrainingArguments` values. This stage is run during `TrainingArguments` object creation. train_micro_batch_size_per_gpuper_device_train_batch_sizegradient_accumulation_stepstrain_batch_sizeztrain_batch_size (calculated)gradient_clipping max_grad_normzoptimizer.params.lr learning_ratezoptimizer.params.betaszadam_beta1+adam_beta2zoptimizer.params.eps adam_epsilonzoptimizer.params.weight_decay weight_decayzscheduler.params.warmup_min_lrzscheduler.params.warmup_max_lr checkpointuse_node_local_storagez fp16.enabledzfp16|fp16_full_evalz bf16.enabledzbf16|bf16_full_evalN) world_sizerMrNrIrQrR adam_beta1 adam_beta2rSrT fill_onlysave_on_each_noderFrAfp16fp16_full_evalbf16bf16_full_evalis_truetorchbfloat16r0float16float32)rargsauto_find_batch_sizerOs&&& rtrainer_config_process/HfTrainerDeepSpeedConfig.trainer_config_processs ??T-M-MMPTPpPpp  ,  , , )$ $   )  , , )    +$ $  +T-?-?Q -t/A/A?S  $ __doo . # .0A0A>R 79J9JN[ 7; 8$:L:Lo^  ! ! !(,  b(IDKK %BFBXBXDKK %&> ? )I)Id6I6ILab )I)Id6I6ILab << ' '..DK \\. ) )--DK--DKr c *.ROpVUu.uFqPPV4'gKVNK pp\V4^8EdRp\VR4'Ed?\VPR4'dVPPpEM \VPR4'd!\ VPP 4pM\VPR4'dH\VPPR4'd"VPPPpMk\VPR4'dP\VPPR4'd*\ VPPP 4pVf\R V R 24hVPRWw,4VP4'dCVPR\R V,V,44VPR^ V,4VPR VR 4VPRVPV4R4\VP4^8d+RPVP4p\RV R24hR#uupi)zx This stage is run after we have the model and know num_training_steps. Now we can complete the configuration process. $zero_optimization.reduce_bucket_size-zero_optimization.stage3_prefetch_bucket_size4zero_optimization.stage3_param_persistence_thresholdNrF hidden_size hidden_sizes text_configzThe model's config file has neither `hidden_size` nor `hidden_sizes` entry, therefore it's not possible to automatically fill out the following `auto` entries in the DeepSpeed config file: zb. You can fix that by replacing `auto` values for these keys with an integer value of your choice.g?z scheduler.params.total_num_stepsznum_training_steps (calculated)z!scheduler.params.warmup_num_steps warmup_steps z]Please correct the following DeepSpeed config values that mismatch TrainingArguments values: zF The easiest method is to set these DeepSpeed config values to 'auto'.)rjrkrl)r<lenhasattrrFrmmaxrnror4rZis_zero3intrIget_warmup_stepsr1join) rremodelnum_training_stepshidden_size_based_keysxhidden_size_auto_keysrmr1s &&&& rtrainer_config_finalize0HfTrainerDeepSpeedConfig.trainer_config_finalizes""  -C V,BqllSTo,B V $ % )Kuh''5<<77"',,":":KU\\>::"%ell&?&?"@KU\\=99gellF^F^`m>n>n"',,":":"F"FKU\\=99gellF^F^`n>o>o"%ell&>&>&K&K"LK" 55J4KLYY NNA;C\ ]}}Ck)K78J$  .  -  /  ! !"4 5  t ! #4??3J'L(oq  $a!Ws JJ)r0r1)NTF)r"r#r$r%r&rr5r<rIrrZrgr~r'r(r)r*s@@rr-r-Rs?   ![4jU;I8(tCCr r-c2\P!V4sR#r/)weakrefref_hf_deepspeed_config_weak_ref)hf_deepspeed_config_objs&rrr s %,KK0G$H!r c RsR#r/)rr!r runset_hf_deepspeed_configrs %)!r c^\e%\4e\4P4#R#)NF)rrur!r ris_deepspeed_zero3_enabledrs&$05R5T5`,.7799r cV\e!\4e\4P#R#r/)rrFr!r rdeepspeed_configrs#$05R5T5`,.555r c Naaaa^RIo^RIp^RIHp^RIHoVP 4oVVVV3RloVP!4;_uu_4V!4;_uu_4S!WP4RRR4RRR4R# +'giL;i +'giR#;i)a- DeepSpeed ZeRO-3 variant of `PreTrainedModel.initialize_weights`. Mirrors the `smart_apply` dispatch logic but gathers each module's partitioned parameters before calling `_initialize_weights`, so initialization operates on full tensors instead of empty shards. Only rank 0 performs the actual init. N)guard_torch_init_functions)PreTrainedModelc<VP4F1p\VS4'dS!W"P4K)S!W!4K3 \VP RR74pV'dYSP P V^R7;_uu_4SPP4^8Xd V!VS4RRR4R#V!VS4R# +'giR#;i)F)recurse modifier_rankN) children isinstance_initialize_weightslist parameterszeroGatheredParameterscommget_rank)model_or_modulefnchildparamsr _apply_zero3r is_remote_codes&& rr.initialize_weights_zero3.._apply_zero34s$--/E%11U$=$=>U' 0 o000?@ 2262KK>>**,17LK  / LKKs )C  C ) r rainitializationrmodeling_utilsrrno_gradr)ryrarrrr rs& @@@@rinitialize_weights_zero3r%sj;0))+N 0 0  ' ) )  9 9 :*  ) ) s$ BB .B B B B$ c a!\4pVeVPR/4PR^4pVPR/4p\V\4'd,\ WEPR/4PR^44pV^8d \ R4h^RIHpHpH o!H p\VRR4p VPp /p VP4P4F4wr\P !V P"V P$R R 7W&K6 VUu.uFp\W4'gKVNK ppVUu.uFp\W4'gKVNK pp\'V4^8XdG/pVP4F#wppV!VV.WR 7wppVV 9gKVVV&K% V eV VnV#VUUu/uFpVP*FpVVbK K ppp/p/p\-VP/4V!3R lR 7pVFpVP1V4pV!VVVWR 7wppVV 9gK,Ve[VV,pV!VP*VP2VP4R7pVP7VV4pVP9VVVV4KVVV&K VP4FewppVP;VVVP<R7pVP4F,wpp \V \>4'd V ^,MT p V VV&K. Kg V eV VnV#uupiuupiuuppi \@dp \CRT RT 24T hRp ? ii;i)z Apply weight conversions (renaming and merging/splitting operations) to a state dict. This is a simplified version that handles the conversion without loading into the model. Ntensor_parallel autotp_size inferencetp_sizezWeight conversions (e.g., MoE expert fusion) with DeepSpeed Tensor Parallelism are not yet implemented but support is coming soon. Please disable tensor_parallel in your DeepSpeed config or convert your checkpoint to the expected format first.)WeightConverterWeightRenamingdot_natural_keyrename_source_key _metadatameta)r5device)base_model_prefixmeta_state_dictc<S!V4#r/r!)krs&r9_apply_weight_conversions_to_state_dict..s /!:Lr )key)source_patternstarget_patterns operations)ryrFz'Failed to apply weight conversion for 'zb'. This likely means the checkpoint format is incompatible with the current model version. Error: )"rrArdictrtNotImplementedErrorcore_model_loadingrrrrgetattrr state_dictitemsraemptyshaper5rrrrsortedkeyspoprr setdefault add_tensorconvertrFr Exception RuntimeError)"ryrweight_mapping ds_configrinference_configrrrr rmodel_state_dictrparamentry renamings convertersnew_state_dict original_keytensor renamed_keyr converterrpattern_to_converterconversion_mapping sorted_keyssource_pattern new_convertermappingrealized_value target_nameers"&&& @r'_apply_weight_conversions_to_state_dictrHst!"I-- 126::=!L$==b9 & - -'#7#78I2#N#R#RS\^_#`aG Q;%d  ihz;5H//&&(..0  % EKKu{{SY Z1%3XN5j6WNIX%3Z^Ez%7Y%%^JZ :!$.$4$4$6 L&.i?PNK...4{+ %7  '/N $;Eh*YiNgNgAyLNgA*h N*0LMK#  -&7 )ZCT' # ^ * *)1@ /$-$=$=$-$=$=(33! -77 ]S""; nfU/5{+/$4!3 8 8 : W $__||-N '5&:&:&<" U$.ud$;$;a.3{+'=!;$#+ SYZ"iX 9+G   s7L0L<LL7L!(AL'' M 2MM c aa a a \VRR4o VP4pS eS VnRpVe\VRR4pVe#\V4^8d\ WV4pW0n.oVP 4p\VP44o \VRR4pVP4UUu/uF'wrgVPV RV 24eV RV 2MTVbK) pppR RVV V V 3Rlllo S !WRR 7SS 3#uuppi) a Loads state dict into a model specifically for Zero3, since DeepSpeed does not support the `transformers` tensor parallelism API. Nearly identical code to PyTorch's `_load_from_state_dict` Args: model_to_load: The model to load weights into state_dict: The state dict containing the weights load_config: Optional LoadStateDictConfig containing weight_mapping and other loading options rNrr.Fc8V^8dQhR\P/#)module)rModule)formats"r __annotate__7_load_state_dict_into_zero3_model..__annotate__s)W)WRYY)Wr c*<Sf/MSPVRR/4pW4R&WVR..S3p\4'Edt^RIp\VP VRRRR74p.pVF<p W9gK Wy,p RV nVP V 4SPV 4K> \V4^8dbVPPV^R7;_uu_4\PP4^8XdVP!V!RRR4\VPVRRRR74p V P!4FhwrW9gK V fKSPV 4\P"!4;_uu_4V P%W,4RRR4RV nKj VP&P!4F"wrVfK S!WW-,R,V4K$ R# +'giL;i +'giLp;i)Nassign_to_params_buffersTF)prefixrrr)rArr rnamed_parameters_is_hf_initializedrBdiscardrrrrra distributedr_load_from_state_dict named_buffersrrcopy__modules)rrrrlocal_metadatarer rparams_to_gatherrrrbufnamer error_msgsloadr missing_keyss&&&& rr/_load_state_dict_into_zero3_model..loads'/X\\&"+r5R5M12ND"b*M & ' '  $F$;$;6#2;X]$;$^_ ! %?,/E/3E,$++E2 ((+ &#$q(^^667GWX6YY((113q844d;Z !!5!5VCR[RW!5!XYM'--/?s ((+ *-0)-1C* 0"??002KD U (;=UV3ZY)s4G/H/ G?  H )r)F) rcopyrrrr_weight_conversionsrsetrrrA) model_to_loadr load_configrmeta_model_state_dict prefix_modelrvrrr rs &&& @@@@r!_load_state_dict_into_zero3_modelr s-z;5H"J' N .>E!c.&9A&=<]Xfg ,:)J)446,1134L=*=tDL$$&&DA#8";";|nAaS1# dehi i&)W)WV UC | ##is1-C=cDaa^RIHpHpVPpRpRV9d V!VR7pM@VP 4'd\ P R4SP4pRVR&Rp RV9d V!V4p W3#\W4'dVV3R lp V!WR 7p W3#) zQ A convenience wrapper that deals with optimizer and lr scheduler configuration. ) DummyOptimDummySchedulerN optimizer)rzDetected ZeRO Offload and non-DeepSpeed optimizers: This combination should work as long as the custom optimizer has both CPU and GPU implementation (except LAMB)Tzero_allow_untested_optimizer schedulerch<\P!S4pRVnVPSVR7pV#)N)rzr)r lr_schedulercreate_scheduler)r trainer_copyrrztrainers& r_lr_scheduler_callable5deepspeed_optim_sched.._lr_scheduler_callable7s=#yy1 -1 )+<<'9Y =  $#r )lr_scheduler_callable) accelerate.utilsr r rF is_offloadloggerinfocreate_optimizerr) rhf_deepspeed_configrerzmodel_parametersr r rFrrrs f&&f& rdeepspeed_optim_schedr s< ' 'FIf&67  ) ) + + KKV ,,. 26./Lf%i0 "  "" i , , $*)bL  ""r c^RIHpVPpVPpVPP P PpVPWTV4VPVP44V'dLVP4'g \R4hVPR4VPR4RRrRp Wx3#RVnVPP!R/4P!R^4p V ^8d2^RIp V P%VV VP'4VPR7p\)\+R VP-444p \/WWQV 4wrxWx3#) a Init DeepSpeed, after updating the DeepSpeed configuration with any relevant Trainer's args. If `resume_from_checkpoint` was passed then an attempt to resume from a previously saved checkpoint will be made. Args: trainer: Trainer object num_training_steps: per single gpu resume_from_checkpoint: path to a checkpoint if to resume from after normal DeepSpeedEngine load inference: launch in inference mode (no optimizer and no lr scheduler) auto_find_batch_size: whether to ignore the `train_micro_batch_size_per_gpu` argument as it's being set automatically by the auto batch size finder Returns: optimizer, lr_scheduler We may use `deepspeed_init` more than once during the life of Trainer, when we do - it's a temp hack based on: https://github.com/deepspeedai/DeepSpeed/issues/1394#issuecomment-937405374 until Deepspeed fixes a bug where it can't resume from a checkpoint after it did some stepping https://github.com/deepspeedai/DeepSpeed/issues/1612 )rzMZeRO inference only makes sense with ZeRO Stage 3 - please adjust your configrrNrr)ryrr5rFcVP#r/) requires_grad)ps&rr deepspeed_init..sr )deepspeed.utilsrryre acceleratorstatedeepspeed_plugin hf_ds_configr~setLevelget_process_log_levelrur4del_config_sub_treerrFrAr tp_model_initr5rfilterrr ) rrzr ds_loggerryrerrrrdeepspeed_tp_sizer s &&& rdeepspeed_initr2Gsc*4 MME <s  &tyyO3DM1R'S T $%) 00ABC'77 1"&%) 8  L_L]^_ _ <_!>I 11$Mr ca aRV9dRV9d VR,VR&V!R /VBpVPpVPR8Xd)VP^8d^RIHpVP 4pM;VP e#VP R,P4pM \R4hVPp \PPPPWhR7oVR,R 8gPR 4P4p \PPPPWR7o \V V3Rl\!V 444p \S 4p V \#V ^4, pV'dWe3#T#) aA Computes the loss under sequence parallelism with `sp_backend="deepspeed"` and `sp_size > 1`. Performs weighted loss aggregation across SP ranks, accounting for varying numbers of valid tokens per rank (e.g., when some ranks receive only padding or prompt tokens that are masked with -100). Args: accelerator (`Accelerator`): The accelerator instance with `torch_device_mesh` support. model (`torch.nn.Module`): The model to compute the loss for. inputs (`dict[str, torch.Tensor | Any]`): The input data for the model. Must include `"shift_labels"` key. return_outputs (`bool`): Whether to return the model outputs along with the loss. pc (`accelerate.parallelism_config.ParallelismConfig`): The parallelism configuration. Returns: The loss, or a tuple of `(loss, outputs)` if `return_outputs` is `True`. labels shift_labelsr )groupsspzSequence parallelism is enabled but no SP process group is available. Ensure torch_device_mesh is initialized or sp_backend='deepspeed' with sp_size > 1.)groupc3r<"TF,pSV,^8gKSV,SV,,xK. R#5i)r4Nr!).0rankgood_tokens_per_ranklosses_per_ranks& r ,deepspeed_sp_compute_loss..s:(D  % ) ; 4T :::(s77r!ir)loss sp_backendsp_sizer&rE_get_sequence_parallel_grouptorch_device_mesh get_groupr4rarr functional all_gatherviewsumrangert)r'ryinputsreturn_outputspcoutputsrOrEsp_group sp_world_size good_tokens total_losstotal_good_tokensrKrLs&&&&& @@rdeepspeed_sp_compute_lossrcs\,v.F":!.1xofoG <rArcr!r rrms #9HH   H % !7!9!9O3..2p0ph!%I)  ;Fl^W$t3#l@#FP0N"77r