+ QV-jsPRt^RIt^RIt^RIt^RIt^RIt^RIt^RIt^RIt^RI t ^RI t ^RI t ^RI t ^RI t ^RIHtHtHt^RIHt^RIHt^RIHtHt^RIHt^RIt^RIt^RIt^RIH t!^RI"H#t#H$t$^RI%H&t&^R IH't'^R I(H)t)H*t*H+t+H,t,H-t-^R I.H/t/^R I0H1t1^R I2H3t3H4t4H5t5^RI6H7t7H8t8^RI9H:t:^RI;Ht>H?t?^RI@HAtA^RIBHCtCHDtDHEtEHFtFHGtG^RIHHItIHJtJ^RIKHLtL^RIMHNtNHOtO^RIPHQtQ^RIRHStSHTtTHUtU^RIVHWtWHXtX^RIYHZtZ^RI[H\t\H]t]^RI^H_t_H`t`^RIaHbtbHctc^RIdHete^RIfHgtg^R IhHitiHjtjHktkHltlHmtmHntnHotoHptp^R!IqHrtrHstsHtttHutu^R"IvHwtwHxtxHytyHztzH{t{H|t|H}t}H~t~HtHtHtHtHtHtHt^R#IHtHtHtHtHtHtHtHtHtHtHtHtHtHtHtHtHtHtHtHtHtHtHtHtHtHtHtHtHtHt^R$IHtHtHt^R%IHtHtHtHtHtHtHtHtHtHtHtHtHtHtHtHtHtHtHtHtHtHtHtHtHtHtHt^R&IHt^R'IHt]j.t]mt]!4'd ^R(IHt]t]!4'd^RIt]!4'dX^RIHuHt^RIHuHt^RIHt^R IH/t]&EP!]4]&EP!]48t]'d ^RIH uHtMR)t]!4'd^RIHuHt^R*IvHtHtHt]!4'd^R+IHt]!4'd>^R,IHtHt^R-IHt^R.IHtHtHtHtHtHtHtHtHt^R/IHt]F!4'd^R0IHt]'d^RIt]EP!]4tR1tR2tR3EtR4EtR5EtR6EtR7Et]!R;R87!R9R:44EtR#)llt,V3R?lR@lt-V3RAlRBlt.V3RClRDlt/V3RElRFlt0V3RGlRHlt1V3RIlRJlt2V3RKlRLlt3V3RMlRNlt4V3ROlRPlt5V3RQlRRlt6RV3RSlRTllt7RUt8RVt9RWt:RRXlt;RV3RYlRZlltRV3R_lR`llt?RV3RalRbllt@V3RclRdltAV3RelRfltBV3RglRhltCV3RilRjltDV3RklRlltEV3RmlRnltFRV3RolRplltGV3RqlRrltHV3RslRtltIV3RulRvltJV3RwlRxltKV3RylRzltLV3R{lR|ltMRV3R}lR~lltNRV3RlRlltORV3RlRlltPV3RlRltQV3RlRltRRV3RlRlltSRV3RlRlltTRV3RlRlltUV3RlRltVV3RlRltWRV3RlRlltXRV3RlRlltYV3RlRltZV3RlRlt[V3RlRlt\V3RlRlt]V3RlRlt^V3RlRlt_V3RlRlt`V3RlRltaV3RlRltbRtcVtdR#)Traineru Trainer is a simple but feature-complete training and eval loop for PyTorch, optimized for 🤗 Transformers. Args: model ([`PreTrainedModel`] or `torch.nn.Module`, *optional*): The model to train, evaluate or use for predictions. If not provided, a `model_init` must be passed. [`Trainer`] is optimized to work with the [`PreTrainedModel`] provided by the library. You can still use your own models defined as `torch.nn.Module` as long as they work the same way as the 🤗 Transformers models. args ([`TrainingArguments`], *optional*): The arguments to tweak for training. Will default to a basic instance of [`TrainingArguments`] with the `output_dir` set to a directory named *tmp_trainer* in the current directory if not provided. data_collator (`DataCollator`, *optional*): The function to use to form a batch from a list of elements of `train_dataset` or `eval_dataset`. Will default to [`default_data_collator`] if no `processing_class` is provided, an instance of [`DataCollatorWithPadding`] otherwise if the processing_class is a feature extractor or tokenizer. train_dataset (`torch.utils.data.Dataset` | `torch.utils.data.IterableDataset` | `datasets.Dataset`, *optional*): The dataset to use for training. If it is a [`~datasets.Dataset`], columns not accepted by the `model.forward()` method are automatically removed. Note that if it's a `torch.utils.data.IterableDataset` with some randomization and you are training in a distributed fashion, your iterable dataset should either use a internal attribute `generator` that is a `torch.Generator` for the randomization that must be identical on all processes (and the Trainer will manually set the seed of this `generator` at each epoch) or have a `set_epoch()` method that internally sets the seed of the RNGs used. eval_dataset (`torch.utils.data.Dataset` | dict[str, `torch.utils.data.Dataset`] | `datasets.Dataset`, *optional*): The dataset to use for evaluation. If it is a [`~datasets.Dataset`], columns not accepted by the `model.forward()` method are automatically removed. If it is a dictionary, it will evaluate on each dataset prepending the dictionary key to the metric name. processing_class (`PreTrainedTokenizerBase` or `BaseImageProcessor` or `FeatureExtractionMixin` or `ProcessorMixin`, *optional*): Processing class used to process the data. If provided, will be used to automatically process the inputs for the model, and it will be saved along the model to make it easier to rerun an interrupted training or reuse the fine-tuned model. model_init (`Callable[[], PreTrainedModel]`, *optional*): A function that instantiates the model to be used. If provided, each call to [`~Trainer.train`] will start from a new instance of the model as given by this function. The function may have zero argument, or a single one containing the optuna/Ray Tune trial object, to be able to choose different architectures according to hyperparameters (such as layer count, sizes of inner layers, dropout probabilities etc). compute_loss_func (`Callable`, *optional*): A function that accepts the raw model outputs, labels, and the number of items in the entire accumulated batch (batch_size * gradient_accumulation_steps) and returns the loss. For example, see the default [loss function](https://github.com/huggingface/transformers/blob/052e652d6d53c2b26ffde87e039b723949a53493/src/transformers/trainer.py#L3618) used by [`Trainer`]. compute_metrics (`Callable[[EvalPrediction], Dict]`, *optional*): The function that will be used to compute metrics at evaluation. Must take a [`EvalPrediction`] and return a dictionary string to metric values. *Note* When passing TrainingArgs with `batch_eval_metrics` set to `True`, your compute_metrics function must take a boolean `compute_result` argument. This will be triggered after the last eval batch to signal that the function needs to calculate and return the global summary statistics rather than accumulating the batch-level statistics callbacks (List of [`TrainerCallback`], *optional*): A list of callbacks to customize the training loop. Will add those to the list of default callbacks detailed in [here](callback). If you want to remove one of the default callbacks used, use the [`Trainer.remove_callback`] method. optimizers (`tuple[torch.optim.Optimizer, torch.optim.lr_scheduler.LambdaLR]`, *optional*, defaults to `(None, None)`): A tuple containing the optimizer and the scheduler to use. Will default to an instance of [`AdamW`] on your model and a scheduler given by [`get_linear_schedule_with_warmup`] controlled by `args`. optimizer_cls_and_kwargs (`tuple[Type[torch.optim.Optimizer], dict[str, Any]]`, *optional*): A tuple containing the optimizer class and keyword arguments to use. Overrides `optim` and `optim_args` in `args`. Incompatible with the `optimizers` argument. Unlike `optimizers`, this argument avoids the need to place model parameters on the correct devices before initializing the Trainer. preprocess_logits_for_metrics (`Callable[[torch.Tensor, torch.Tensor], torch.Tensor]`, *optional*): A function that preprocess the logits right before caching them at each evaluation step. Must take two tensors, the logits and the labels, and return the logits once processed as desired. The modifications made by this function will be reflected in the predictions received by `compute_metrics`. Note that the labels (second parameter) will be `None` if the dataset does not have them. Important attributes: - **model** -- Always points to the core model. If using a transformers model, it will be a [`PreTrainedModel`] subclass. - **model_wrapped** -- Always points to the most external model in case one or more other modules wrap the original model. This is the model that should be used for the forward pass. For example, under `DeepSpeed`, the inner model is wrapped in `DeepSpeed` and then again in `torch.nn.DistributedDataParallel`. If the inner model hasn't been wrapped, then `self.model_wrapped` is the same as `self.model`. - **is_model_parallel** -- Whether or not a model has been switched to a model parallel mode (different from data parallelism, this means some of the model layers are split on different GPUs). - **place_model_on_device** -- Whether or not to automatically place the model on the device. Defaults to `True` unless model parallel, DeepSpeed, FSDP, full fp16/bf16 eval, or SageMaker MP is active. Can be overridden by subclassing `TrainingArguments` and overriding the `place_model_on_device` property. - **is_in_train** -- Whether or not a model is currently running `train` (e.g. when `evaluate` is called while in `train`) )get_learning_ratesget_num_trainable_parametersget_optimizer_group log_metricsmetrics_format save_metrics save_stateNc<V^8dQhRS[S[P,R,RS[R,RS[R,RRRRR S[S[,S[,S[,R,R S[ R S[3,R,R S[ R,R S[ S[ .S[ 3,R,RS[ S[ ,R,RS[S[P P"R,S[P P$P&R,3,RS[S[S[P P",S[ S[S[3,3,R,RS[ S[P.S[P..S[P.3,R,/ #)modelNargs data_collator train_datasetz3Dataset | IterableDataset | datasets.Dataset | None eval_datasetz6Dataset | dict[str, Dataset] | datasets.Dataset | Noneprocessing_class model_init.compute_loss_funccompute_metrics callbacks optimizersoptimizer_cls_and_kwargspreprocess_logits_for_metrics)r1rModulertrr8rrr7rrWdictlistr>tupletorchoptim Optimizer lr_schedulerLambdaLRtypestrr Tensor)format __classdict__s"e/Volumes/fast/ai/experiments/ui-tars-smoke/.venv/lib/python3.14/site-packages/transformers/trainer.py __annotate__Trainer.__annotate__ls~7~7*T1~7 $&~7$d* ~7 M ~7 O ~72  !   ~7S/12T9~7$d?~7">"2D"89D@~7(4/~7 %++//$6 8P8P8Y8Y\`8``a!~7"#(U[[-B-B(CT#s(^(S"TW["[#~7$(0u||0Lell0Z'[^b'b%~7c Vf(Rp\PRV R24\VR7pW nVPP'd \ VPP 4M\VPP 4RVnWn VP4\VPP4Vn VPP4VP4p\ P"!V4VP$Vf'VeWpnVP)4pM \+R4hVe \-R4hWpnVP.P0\29d$\-RVP.P0 R24h\5V4R Vn\9VR R4e\;VP<P?44Uu.uFpVR 9gK VNK pp\AV4^8d R VnMI\AV4^8Xd:VPPB\DPB!V^,48gVnVPF;'dVPHPKR R 4Vn&VPF'dZVPN'd \-R 4hVPL'g+VPP\RPT8wd \-R4hVPVeVPVVn+MVP6'gkVPN'gYVPX'gGVPZ'g5VPL'g#VP\'g\_4'd R Vn+MR Vn+VPV'd<\9VRR4\`Pb8wdVPeWPB4VP6'd^VPn3Wn4Wn \kVPlPoV44p\qVR4'dVPrVn:M\vPx!VPz4P|p\~;QJd*RVP?44F 'gK R M R M!RVP?444Vn:\9VPlRR4pVe+VPR8XdVP'dR Vn:\kVP4p\VP.4pVPPfTMVPPVnC\VP.4VnD\PJ!\9VRR44\JVnGVPP^8wdp\9VPPRR4R8Xd$\P!R\4RVnMM-\VPPR7VnMMRVnMVe(\V\\34'd \V4M\pVeTMTVnTW@nUWPnVW`nWVPVnXWnYWnZWn[V wVn\Vn]Wn^VP4\\VPP4,pVPP'd)^RIdHepV!4pVV.,pVPV4V fTMVV ,p \WPVPVPVP4VnhTPVPP'd\M\4RVnmVPP'dVP4VPP'd-\P!VPPR R7\4Vnu\VP4VP4VPPVP.,Uu.uFp\V\4'gKVNK upR7Vn{R Vn|RVn}RVn~^VnR VnRVnVEPVnR VnVPEP VPVPVP4Vnu\9VPRR4e1VPEP VPPnVPF;'dVPHPKRR 4VnVEP'dE\'g \-R4hE\EP!4pE\EP!E\EP!E\EP!E\!V44V^3R!R74VPL;'dVEP'*VnVPEP%4R#uupiuupi)"N tmp_trainerz1No `TrainingArguments` passed, using `output_dir=z`.) output_dirz<`Trainer` requires either a `model` or `model_init` argumentzK`Trainer` requires either a `model` or `model_init` argument, but not both.zThe model you have picked (a) cannot be used as is for training: it only computes hidden states and does not accept any labels. You should choose a model with a head suitable for your task like any of the `AutoModelForXxx` listed at https://huggingface.co/docs/transformers/model_doc/autoF hf_device_mapTxlazZUsing --fsdp xxx together with --deepspeed is not possible, deactivate one of those flags.z.Using fsdp only works in distributed training.quantization_methodaccepts_loss_kwargsc3n"TF+qP\PP8HxK- R#5iN)kindinspect Parameter VAR_KEYWORD).0ks& r #Trainer.__init__..s'1AXA'++777AXs35parallelism_config deepspeed loss_type problem_typemulti_label_classificationzsLabel smoothing is not compatible with multi-label classification. Disabling label smoothing for this training run.)epsilon)JITCheckpointCallbackexist_ok)is_local_process_zerois_world_process_zerostateful_callbacksconfig xla_fsdp_v2z*FSDPv2 requires `torch_xla` 2.2 or higher.) axis_names)cpudisk)fsdptensor)loggerinfortrfull_determinismrdseedrlrr"create_accelerator_and_postprocessr]skip_memory_metrics_memory_trackerstartget_process_log_levelr set_verbosity_setup_devicesrcall_model_init RuntimeError ValueError __class____name__r4rqis_model_parallelgetattrsetrvalueslendevicerr fsdp_configgetis_fsdp_xla_enabledis_deepspeed_enabled parallel_moders DISTRIBUTEDplace_model_on_devicefp16_full_evalbf16_full_evalis_fsdp_enabledrrBITS_AND_BYTES_move_model_to_device_n_gpu model_wrappedrp acceleratorr2hasattrrmodel_accepts_loss_kwargsr signatureforward parametersany sp_backend sp_enabledr label_namesrr.r/_loss_shifts_labelslabel_smoothing_factorrwarningswarn UserWarninglabel_smootherrG isinstancer8rrrrrrrneftune_noise_alpharrr optimizerrr_validate_argsDEFAULT_CALLBACKSr report_toenable_jit_checkpointtrainer_jit_checkpointr set_trainerr9callback_handler add_callback disable_tqdmr<DEFAULT_PROGRESS_CALLBACK hub_model_id push_to_hub init_hf_repo should_saveosmakedirsrr?controlr@rrrr;state is_in_trainhp_namehp_search_backend current_flos_loggers_initialized_signature_columnstrain_batch_size_train_batch_size_created_lr_scheduler on_init_end use_cacheis_fsdp_xla_v2_enabledIS_XLA_FSDPV2_POST_2_2xrglobal_runtime_device_countxsset_global_meshMeshnparrayrangeis_fsdp_xla_v1_enabledstop_and_update_metrics)selfrrrrrrrrrrrrrr log_levelr devicesunwrapped_modelforward_paramspcmodel_to_inspectdefault_label_namesdefault_collatordefault_callbacksr jit_callbackcb num_devicess&&&&&&&&&&&&&& r__init__Trainer.__init__lssD <&J KKKJbmm{:r}}}-2D *,TZZ8)*:*D*DE26))2G2G2O.UYU^U^UjUj./?/I/IJ$0#3#3G"&"<"<">!22<< ~M MrQ[\^`oQpM   ! !%$)!"&!%!6!6%*",,88DJJPTP\P\]  4::x . :*.))*=*=DJJ   '&*ii&^&^D4D4D4H4HX]4^#  & & &)) !MNN88:K   rwwrxxk0B'CkSTEUbtu v&*&>&>&b&btGbGbCb# 446YpX s k .k >k k c<V^8dQhRR/#rreturnN)rrs"rrrlsPPPrc VPp\4'dVP'd \R4hVP\ P PP8wd\PR\ P PP RVP R\ P PP 24\ P PPVnVP'dIVPe;R\P!VP4P9d \R4hVPe9VPR8wd(VP f\R VP R 24hVP"\$P&8XgVP('dVP*f \R 4hVP,eVP.e \1R 4hVP2e(VP.fVP4e \1R 4h\74'dVP.eVP8P4FpVP:pM VP.P<F6p\?VR,4^8gKVR,^,P:pM XX8wd \R4hVP@'gVPB'd(VP.fVP4e \1R4h\EVPF4'g2\E\IVPFRR44'd \KR4hVPL^8d'VPN^8d\PQR4VPRe8\UVPR4'gVPL^8:d \R4hVPRef\WVPR\XPZP\P^4'd&\PQRVP` R24R#R#R#)zJValidate constructor arguments and fail fast on incompatible combinations.zOSageMaker Model Parallelism does not support BF16 yet. Please use FP16 instead z(FP16 provided in SM_HP_MP_PARAMETERS is z+, but FP16 provided in trainer argument is z , setting to Ncompute_resultzWhen using `batch_eval_metrics`, your `compute_metrics` function must take a `compute_result` boolean argument which will be triggered after the last batch of the eval set to signal that the summary statistics should be returned by the function.noz%You have set `args.eval_strategy` to zx but you didn't pass an `eval_dataset` to `Trainer`. Either set `args.eval_strategy` to `no` or pass an `eval_dataset`. z`args.metric_for_best_model` must be provided when using 'best' save_strategy or if `args.load_best_model_at_end` is set to `True`.zSPassing both `optimizers` and `optimizer_cls_and_kwargs` arguments is incompatible.zPassing a `model_init` is incompatible with providing the `optimizers` argument. You should subclass `Trainer` and override the `create_optimizer_and_scheduler` method.paramsa[The model and the optimizer parameters are not on the same device, which probably means you created an optimizer around your model **before** putting on the device and passing it to the `Trainer`. Make sure the lines `import torch_xla.core.xla_model as xm` and `model.to(xm.xla_device())` is performed before the optimizer creation in your script.zPassing `optimizers` is not allowed if PyTorch FSDP is enabled. You should subclass `Trainer` and override the `create_optimizer_and_scheduler` method. collate_batchzRThe `data_collator` should be a simple callable (function, class with `__call__`).zHmax_steps is given, it will override any value given in num_train_epochszThe train_dataset does not implement __len__, max_steps has to be specified. The number of steps needs to be known in advance for the learning rate scheduler.zThe `train_sampling_strategy='z'` option is ignored when using an `IterableDataset`. Samplers cannot be used with IterableDataset as they require indexed access to the dataset.)1rrbf16rfp16smpr=cfgrwarningbatch_eval_metricsrrrr eval_strategyr save_strategyr\BESTload_best_model_at_endmetric_for_best_modelrr+rrrrrr  param_groupsr r rcallablerr TypeError max_stepsnum_train_epochsrrrgr)rutilsdatartrain_sampling_strategy)rUrparam model_device param_groupoptimizer_devices& rr,Trainer._validate_argslsyy # $ $yyy !rssyyCIIMM...>syy}}?Q?Q>RS@@D {K""%))--"4"4!57  IIMM..   " " "t';';'Gw'8'89M9M'N'Y'YY N    )d.@.@D.HTM^M^Mf78J8J7KLDE    !2!2 2d6Q6Q6Q))1 Z  ( ( 49Stu u ?? &DNN,F$J[J[Jgj  " # #(B..0$|| 1 $~~:: {8,-1'28'>A $"7"7!"; KKb c    )*T=O=O2P2PUYUcUcghUhd     )j9K9KU[[M]M]MmMm.n.n KK01M1M0NOnn /o )rc6<V^8dQhRS[S[S[3,/#rrf)rrr )rrs"rrrs994S>9rc RVPPRVPP/pVPV4VPPeVPPpM\\ VP \4'd;VP P;'gVPP'*pMRpRV/pVPPeVPPVR&VPPeVPPVR&VPPeVPPVR&\R/VB.VR &VPPe<R p\V4'g\!R V R 24hVPPVR &\#VP RR4eVP P$^8dVPPf\\R 4'd@VPPf'^RIHpV!VP P$R7VR &MY\+R4hVR ,P$VP P$8wd#VP P$VR ,n\R4'd=^RIHpV!VPP0VPP2R7pWR&V#)z>Helper method to build accelerator-specific keyword arguments.mixed_precisiondeepspeed_pluginNTfind_unused_parameters bucket_cap_mbbroadcast_buffers static_graphkwargs_handlersz1.12.0z'ParallelismConfig requires accelerate>=z1). Please upgrade accelerate to use this feature.rtp_size)ParallelismConfig)rz5Requires accelerate>1.12.0 to use Tensor Parallelism.z1.2.0)TorchDynamoPlugin)backendmode dynamo_pluginrg)rrrupdateddp_find_unused_parametersr)rr1is_gradient_checkpointinggradient_checkpointingddp_bucket_cap_mbddp_broadcast_buffersddp_static_graphrrr ImportErrorrr acceleraterraccelerate.utilsrtorch_compile_backendtorch_compile_mode) rUkwargsr find_unused ddp_kwargsmin_accelerate_versionrrrs &, r_build_accelerator_argsTrainer._build_accelerator_argssm tyy88  : :  F 99 / / ;))>>K  O 4 4 $zzCCggtyyGgGghKK. < 99 & & 2*.))*E*EJ ' 99 * * 6.2ii.M.MJ* + 99 % % 1)-)C)CJ~ &#@#N:#N"O  99 ' ' 3%- "*+ABB!=>T=UVGH*.)E)ED% & 4::y$ / ; @R@RUV@Vyy++3*844yy33;@5FtzzOaOa5b12$%\]]*+33tzz7I7II59ZZ5G5G)*2 "7 + + :- 77dii>Z>ZM%2 ! rc<V^8dQhRR/#rerg)rrs"rrrsjxjxDjxrc  j /pVPPPe!VPPPpRV9d?VPP^8d \ R4hVR,VPn^VR&\ R/VBpVPPP 4p.ROp\R/VUu/uFqUVPV4bK upBpVPPVn VPR4pV'd2VPP'g\PR4Wvn VPR4RpVPPe#^RIHp V !R/VPPBpVP#VVVR7p \%R/V BVnVP&P(VnR \,P.!VP*4P09d<\2P4!VP*VPP6R 7Vn\9VP&P:R R4RJVn\9VP&P:R R4RJVnVP>'dXVP&P:P@PB'd(VPPD'd \ R 4hVP<'d:\9VPRR4f!\GVP&VP4VPPH'deVP<'gVP>'dAVPPJ'd%VP<'dRMRp \ V R24hVP<'dWVP&P:PLPN^8Xd(VPPP'd \ R4hVPPH'dZVP>'dFR\SVP&P:P@PT49d \ R4hR#R#R#uupi)zOCreate the accelerator and perform post-creation setup (FSDP, DeepSpeed, etc.).N num_stepszThe `AcceleratorConfig`'s `num_steps` is set but `gradient_accumulation_steps` is greater than 1 in the passed `TrainingArguments`If using the passed `AcceleratorConfig` is desired, do not set the `TrainingArguments` `gradient_accumulation_steps`. non_blockingzx`non_blocking` is enabled but `dataloader_pin_memory` is not. For the best performance, it's recommended to enable both.gradient_accumulation_kwargs)FullyShardedDataParallelPlugin)dataloader_config fsdp_plugingradient_accumulation_pluginuse_gather_object)rrrzThe activation_checkpointing in FSDP config and the gradient_checkpointing in training arg can't be set to True simultaneously. Please use FSDP's activation_checkpointing logic when using FSDP.hf_deepspeed_config DeepSpeedFSDPzJ can't be used with `save_only_model` along with `load_best_model_at_end`.zo`auto_find_batch_size` isn't supported yet with DeepSpeed Zero-3. Please consider using Zero-2, Zero-1, or FSDPSHARDED_STATE_DICTzWsave_only_model option is not compatible with FSDP state dict type 'SHARDED_STATE_DICT'rg) split_batchesdispatch_batches even_batchesuse_seedable_sampler)+raccelerator_configrgradient_accumulation_stepsrrto_dictrpop data_seeddataloader_pin_memoryrrqrfsdp_plugin_argsrrrrrgather_for_metricsgather_functionrrr functoolsreval_use_gather_objectrr=rrractivation_checkpointingrr$save_only_modelrvr zero_stageauto_find_batch_sizerstate_dict_type) rUgrad_acc_kwargsrrdataloader_paramsrrrrrrwrappers & rr*Trainer.create_accelerator_and_postprocesss 99 ' ' D D P"ii::WWO / )yy44q8 L 9H 8T 5() $ (B'TO'T$!YY99AACj3 ARSAR(,,U33ARS '+ii&9&9#)--n=  ? ? ? NNK *6&=> 99 % % 1 G8V499;U;UVK++/#)E, '..#//BB '"3"3D4H4H"I"T"T T#,#4#4$$ 8X8X$D %,D,<,<,B,BDVX\$]ei$i!&t'7'7'='=}dS[__    %%11JJJtyyOoOoOo '  $ $ $%>%>kFGy(rst t  % % %  &&77BBaG ...B  II % % %$$$$D,<,<,B,B,N,N,^,^(__vw w`% &OTs R0 c <V^8dQhRS[/#r)r)rrs"rrrgs  j rc VPf \R4hVPVPRVPVPRR7#)a Returns the training [`~torch.utils.data.DataLoader`]. Will use no sampler if `train_dataset` does not implement `__len__`, a random sampler (adapted to distributed training if necessary) otherwise. Subclass and override this method if you want to inject some custom behavior. z+Trainer: training requires a train_dataset.TrainingT)dataset description batch_size sampler_fn is_training)rr_get_dataloaderrE_get_train_samplerrUs&rget_train_dataloaderTrainer.get_train_dataloadergsU    %JK K##&&"--.. $  rcD<V^8dQhRS[S[,R,RS[/#)rrNrf)rrr)rrs"rrr{s%% % g 0D% PZ% rc VfVPf \R4h\V\4'dTMRp\ VR4'd@W P 9d0VP P'dVP V,#\V\4'dVPV,MVeTM VPpVPVRVP PVPVR7#)a Returns the evaluation [`~torch.utils.data.DataLoader`]. Subclass and override this method if you want to inject some custom behavior. Args: eval_dataset (`str` or `torch.utils.data.Dataset`, *optional*): If a `str`, will use `self.eval_dataset[eval_dataset]` as the evaluation dataset. If a `Dataset`, will override `self.eval_dataset` and must implement `__len__`. If it is a [`~datasets.Dataset`], columns not accepted by the `model.forward()` method are automatically removed. z-Trainer: evaluation requires an eval_dataset.eval_eval_dataloaders Evaluation)rrrrdataloader_key) rrr)rrrrdataloader_persistent_workersreval_batch_size_get_eval_sampler)rUrrs&& rget_eval_dataloaderTrainer.get_eval_dataloader{s  D$5$5$=LM M*4L#)F)FF D- . ."8"88 777)).9 9,,,   l +'"" ## $yy00--) $  rc&<V^8dQhRS[RS[/#)r test_datasetrf)rr)rrs"rrrs   J rc hVPVRVPPVPR7#)a} Returns the test [`~torch.utils.data.DataLoader`]. Subclass and override this method if you want to inject some custom behavior. Args: test_dataset (`torch.utils.data.Dataset`, *optional*): The test dataset to use. If it is a [`~datasets.Dataset`], columns not accepted by the `model.forward()` method are automatically removed. It must implement `__len__`. test)rrrr)rrrr)rUrs&&rget_test_dataloaderTrainer.get_test_dataloaders8## yy00-- $  rc&<V^8dQhRS[RS[/#)r dataloaderrf)rint)rrs"rrrs K Kz Kc Krc 4VPp\V\4'd \VPP4#\VP4# \\ \ 3d*\T4TPP,u#i;i)z Helper to get number of samples in a [`~torch.utils.data.DataLoader`] by accessing its dataset. When dataloader.dataset does not exist or has no length, estimates as best it can ) rr)rFr  NameErrorAttributeErrorrzrper_device_train_batch_size)rUrrs&& r num_examplesTrainer.num_examplessz  K ((G'#788:--5566z))* *>95 Kz?TYY%J%JJ J KsAAA\)\*VPPVPP,R 7V R&VP.P1\3V3/V B4p VeGVPP'd+\5VR4'dWP6V&V #Wj/VnV #)zACreate a [`~torch.utils.data.DataLoader`] from the given dataset.)rr collate_fn num_workers pin_memorypersistent_workersmultiprocessing_contextforkNsampler drop_lastprefetch_factor)rrankworker_init_fnr)rrr)datasetsr_remove_unused_columns"_get_collator_with_removed_columnsrrmps is_availablerdataloader_num_workersrrr}r~rdataloader_drop_lastdataloader_prefetch_factorrrk process_indexrpreparerrr) rUrrrrrrr should_forkrrs &&&&&&& rrTrainer._get_dataloaders** " "z'8;K;K'L'L11'1SG CCDDVDVdoCpMnn((557``DII<\<\_`<`  * - 499;; $))99 $))"I"I %v$  '5;;#3#3#C#CDD%/9'/B!),-1YY-K-K k *37993W3W / 06=TYY-M-MTXT]T]TkTk7!"23%%--j.VDU.VW   %$))*Q*Q*Qt0119C&&~6+9)E&rc~<V^8dQhRS[R,RS[PPPR,/#)rrNrfrrr}r~r)rrs"rrrs500$0%++JZJZJbJbeiJi0rc Vf VPpVe\V4'gR#VPPR8Xd\ 4'de\ V\ P4'dEVPPVP9dWPP,MRpMRpVPeVPP^,MRp\VPPVPP,VVVR7#VPPR8Xd \V4#\!V4#)z?Return the training sampler based on `train_sampling_strategy`.Ngroup_by_lengthrlengthsmodel_input_name sequential)rrgrrrr)rrlength_column_name column_namesrmodel_input_namesrHrDrrr)rUrrrs&& rrTrainer._get_train_samplers   ..M  =(A(A 99 , ,0A A$&&:mXEUEU+V+Vyy33}7Q7QQ"))">">? >B>S>S>_%%77:ei ( **TYY-R-RR%!1   YY . ., >$]3 3 / /rcp<V^8dQhRS[RS[PPPR,/#)rrrfNr)rrs"rrrs1g%++:J:J:R:RUY:Yrc BVe\V4'gR#VPPR8Xd\4'de\ V\ P 4'dEVPPVP9dWPP,MRpMRpVPeVPP^,MRp\VPPVVVR7#VPP^8:d \V4#R#)zNReturn the evaluation sampler, using sequential ordering when not distributed.Nrr)rgrrrr)rrrrrrrHr world_sizer)rUrrrs&& rrTrainer._get_eval_samplers  z,'?'? 99 , ,0A A$&&:lHDTDT+U+Uyy33|7P7PP!!=!=> >B>S>S>_%%77:ei ( ))$!1   99  1 $$\2 2rc<V^8dQhRR/#rerg)rrs"rrr0s\\$\rc 0VPEfVPp\VP4'dX\VPR4'dVPP 4pM VPP Pp\ P!VP4p\VPP44VnV;P\\RR.VP,44, unR#R#)zTPopulate `_signature_columns` from the model's forward signature if not already set.Nget_base_modellabel label_ids)rCrr_rr! base_modelrrrrrkeysrr")rUr[rs& r _set_signature_columns_if_needed(Trainer._set_signature_columns_if_needed0s  " " *#zz djj))4::'788'+zz'@'@'B$(,zz'<'<'B'B$))*:*B*BCI&*9+?+?+D+D+F&GD #  # #tC+0FIYIY0Y,Z'[ [ # +rc6<V^8dQhRRRS[R,RR/#)rrzdatasets.DatasetrNrfr)rrs"rrr@s)!;!;)!;8;d !; !;rc  VPP'gV#VP4VPp\ \ VP 4\ V4, 4p\V4^8dVfRMRV R2p\PRV RVPPP RRPV4 RRPV4 R VPPP R 2 4VUu.uFqfVP 9gKVNK pp\V4^8Xd1\R RPV4 R RPV4 R 24h\P !\"P$4\P !R48d8VP'VP(R,WqP(R,R7V#VP+V4#uupi)zBRemove dataset columns not accepted by the model's forward method.zin the z setzThe following columns z) don't have a corresponding argument in `z!.forward` and have been ignored: , z. If z are not expected by `z/.forward`, you can safely ignore this message.zGNo columns in the dataset match the model's forward method signature: (z-). The following columns have been ignored: [zp]. Please check the dataset and model. You may need to set `remove_unused_columns=False` in `TrainingArguments`.z1.4.0r format_kwargs)rcolumnsr-)rremove_unused_columnsr&rCrrrr rrrrrjoinrr parserr set_formatrremove_columns)rUrrsignature_columnsignored_columnsdset_descriptionrr.s&&& rrTrainer._remove_unused_columns@syy...N --/ 33s7#7#783?P;QQR  ! #%0%8r }TX>Y  KK()9(:;JJ((1122STXT]T]^mTnSopyy122HI]I]IfIfHgh77 0M/8L8L3L11/M w<1 YZ^ZcZcduZvYwx==AYY=WN)create_optimizercreate_scheduler)rUr>s&&rcreate_optimizer_and_scheduler&Trainer.create_optimizer_and_schedulerus"  1CDrcH<V^8dQhRS[PP/#r)rrr)rrs"rrrs!JJekk.C.CJrc  \Vf VPMTpVPEfVPV4pRVP4UUu.uF#wrEWC9gK VP'gK!VNK% uppRVP P /RVP4UUu.uF#wrEWC9gK VP'gK!VNK% uppRR/.pVPeVPwrxMVPVP V4wrx\V4'dV!4!V3/VBVnMVRV9dVPR4pRV9dVPR4pRV9dVPR4pV!V3/VBVnR\V49EdGVPRR4^8XEd/^RI p V PPP!4p ^p VP#4Fp \%V \&P(4'gK%T \+V P-4Uu/uF!qUP/4VP14bK# upP344, p \4P7R V R V R, R 24V P9V R R^ /4\4P;R V R24K \4P7RV R, R 24\=4'd&\>P@!VP4VnVP#uuppiuuppiuupi)a Setup the optimizer. We provide a reasonable default that works well. If you want to use something else, you can pass a tuple in the Trainer's init through `optimizers`, or subclass and override this method in a subclass. Returns: `torch.optim.Optimizer`: The optimizer instance. Nrk weight_decayroptimizer_dict bitsandbytes optim_bitszskipped z: zM paramsweightzbitsandbytes: will optimize z in fp32z skipped: i)!rr+get_decay_parameter_namesnamed_parameters requires_gradrrHrget_optimizer_cls_and_kwargsrDrrr rKrGlobalOptimManager get_instancemodulesr)r Embeddingsumrdata_ptrnumelrrrregister_module_overridedebugrroDistributedOptimizer) rUr opt_modeldecay_parametersnpoptimizer_grouped_parameters optimizer_clsoptimizer_kwargsrKmanagerskippedmodules && rrBTrainer.create_optimizers#(-DJJU >> !#==iH &/&@&@&B&BdaqG\abapap&B#DII$:$: &/&@&@&B&BdaqG`efetet&B#C  , (,,8262O2O/ /262S2STXT]T]_h2i/ $M22!.!O>N!O //3C3G3G3Q0..3C3G3G3P0$'773C3G3GHX3Y0!./K!`O_!`]!338H8L8L\[_8`de8e#&,,??LLN'//1F!&",,773IZIZI\']I\A aggi(?I\']'d'd'f#gg hvhb58I$RS88LZ\K]^ 'CF88%TU 2  i%'8AB " $ $ 55dnnEDN~~u V(^s* LL*L L# -L# L# %'L) c<V^8dQhRS[RS[PPR,RS[PPP /#)rr>r+Nrf)rrrrr LRScheduler)rrs"rrrsD!!"%!27++2G2G$2N!  ! ! - -!rc VPfVf^\4'dB\PPP 'dVP P pM VP p\VPPVVPPV4VVPPR7VnRVn VP#)a, Setup the scheduler. The optimizer of the trainer must have been set up either before this method is called or passed as an argument. Args: num_training_steps (int): The number of training steps to do. Returns: `torch.optim.lr_scheduler.LRScheduler`: The learning rate scheduler instance. )r+num_warmup_stepsr>scheduler_specific_kwargsT) rrror=rprnr+r6rlr_scheduler_typeget_warmup_stepslr_scheduler_kwargsrF)rUr>r+s&&&rrCTrainer.create_schedulers    $ *,,1C1C1C $ 8 8I $I - ++#!%!;!;++779!< ??7 # #llnG" &#a&0NN#vwG  s!C.. D+9'D&%D&&D+cl<V^8dQhRS[S[,R,RRRS[S[,R,RS[/#)rresume_from_checkpointNtrial$optuna.Trial | dict[str, Any] | Noneignore_keys_for_evalrf)rrrr^)rrs"rrr3sKkk #d T 1k6k#3i$. k  krc  VRJdRpVPP4VPpRVnVPeVP 'd\ VP4M\VP4VPV4Vn RRuVn Vn VP'd'VPVPVP4VPVnVPP"'d+\%VPVPP&4VP('gVP*'dGVP,'g5VPf'VPVPVP4VP.'d'VPP1VP2R7\5VP6\8\:34'd=\=VPR4'd!\?VPVP64VP@e1\CVPVP@VPD4Vn#VPIV4\JPLVPN9d2VPP^8d \SR4h\UVP4\5V\V4'd;V'd3\YVPZ4pVf\SRVPZ R24hVe\]4'g6VP^'g$VP`'gVPcV4\dPf!\hPjPmV\n44pVPpe$VPr'dVPpVn:\wVPxVPtVPr4pVPz'd \}4M\~P!4pV;_uu_4V!VVVVR 7uuRRR4# +'giR#;i) a Main training entry point. Args: resume_from_checkpoint (`str` or `bool`, *optional*): If a `str`, local path to a saved checkpoint as saved by a previous instance of [`Trainer`]. If a `bool` and equals `True`, load the last checkpoint in *args.output_dir* as saved by a previous instance of [`Trainer`]. If present, training will resume from the model/optimizer/scheduler states loaded here. trial (`optuna.Trial` or `dict[str, Any]`, *optional*): The trial run or the hyperparameter dictionary for hyperparameter search. ignore_keys_for_eval (`list[str]`, *optional*) A list of keys in the output of your model (if it is a dictionary) that should be ignored when gathering predictions for evaluation during the training. Returns: [`~trainer_utils.TrainOutput`]: Object containing the global step count, training loss, and metrics. FNT)gradient_checkpointing_kwargsrz\Currently --debug underflow_overflow is not supported under DP. Please use DDP with torchrunz/No valid checkpoint found in output directory ())rrrr)Arrrr>rrrdrrlrrr+rrrr ruse_liger_kernelr'liger_kernel_configrrrrgradient_checkpointing_enablerr)rr8r7rr`r*r(rneftune_hook_handle_hp_search_setuprUNDERFLOW_OVERFLOWrZn_gpurrrrfrrrr_load_from_checkpointr@load_from_jsonr:pathr0TRAINER_STATE_NAMErDrrEre_inner_training_loopr7ro contextlib nullcontext)rUrrrrr=inner_training_loopr{s&&&& rtrain Trainer.train3sJ. "U *%) " ""$yy ?? &262G2G2G #DII .XVZV_V_M`--e4DJ04d -DND-)))**4::t{{C!%D  99 % % % tzz499+H+H I    4#6#6#6@V@V@V[_[j[j[r  & &tzz4;; ?  & & & JJ 4 4SWSuSu 4 v d++.E~-V W W\c JJ] ]  !T-B-B C  # # /'7 DD\D\^b^n^n'oD $ e$  ) )TZZ 7zzA~!r'tzz2 ,d 3 38N%8%I "%- #RSWSbSbRccd!eff ! -+,,T5N5N5NW[WkWkWk**+AB // =SUg0hiE%%1d6O6O6O).)?)?&8  % %t'='=t?X?X +/*:*:*:$& @V@V@X S&'=%9 SSSs ? Q Q' c <V^8dQhRS[R,RS[R,RS[R,RRRS[S[,R,RS[/#) rrNrrrrrrf)rrtrrr^)rrs"rrrsm[M[M$J[M $&[M!$d [M 6 [M #3i$. [M [Mrc VPP4VP'dVPV4VP 4pVP 'd \ V4pVPW&4wppp p p p p VPWWsV4wrVPWV4wpp\PR4\PRV R 24\PRVR 24\PRVR 24\PRVPPR 24VPPVP8wd$\PRVPR 24\PRV R 24\PR VP 24\PR V R 24\PR \!VR R 7R 24Veh\PRV RVP"P$ 24VPP&'g\PRV RV R24\(P(!4pVP"P*Vn\.P0!RVP2R7VnRVnVP"P$VnVP;4VP<P?W P"VP@4Vn VPB'dVPEWER R7\GW4F}pVP<PIVPVP"VP@4Vn VPKVVVV VVVVVVVR7 VP@PL'gK}M VPOWJV4#)z\Run the actual training loop: forward, backward, optimizer step, logging, and checkpointing.z***** Running training ***** Num examples = ,z Num Epochs = z Num update steps per epoch = z( Instantaneous batch size per device = zA Training with DataParallel so batch size has been adjusted to: zE Total train batch size (w. parallel, distributed & accumulation) = z Gradient Accumulation steps = z Total optimization steps = z# Number of trainable parameters = T)trainable_onlyz/ Resuming training from checkpoint with epoch z and global step z& Fast-forwarding the dataloader past z epochs and z1 batches to resume from the exact training state.rIr )skip_scheduler) repochtrain_dataloadersteps_in_epochnum_update_steps_per_epochrr start_timerepochs_trainedsteps_trained_in_current_epoch)(r free_memoryr_update_auto_batch_sizerrIr,set_initial_training_values_init_training_state_prepare_for_trainingrrrrrErrKr= global_stepignore_data_skiptimenum_input_tokens_seen_initial_num_input_tokens_seenrrr _tr_loss_total_loss_scalar_globalstep_last_logged zero_gradr2on_train_beginr< eval_on_start _evaluaterRon_epoch_begin _run_epochshould_training_stop_finalize_training)rUrrrrrrr|rrnum_train_samplestotal_train_batch_sizerr{rrrrrs&&&&&& rrTrainer._inner_training_loopsq $$&  $ $ $  ( ( 4446  & & &23CD   , ,T D  &   "  :>9R9R 3C]b: 6#'"<"tyy?d?def>ghi 99 0 0D4J4J J KK[\`\r\rst[uv w [\rst[uvw 6t7W7W6XYZ 3Ia=AB 9:OPUfj:klm9nop ! - KKA.AQQbcgcmcmcycybz{ 99--- <^> sCx>rc  \VPPVP.,Uu.uFp\ V\ 4'gKVNK upR7VnVRJVP nVPVP n VP PVPV4^p^pVEe0\PP\PPV\ 44'd\P"!\PPV\ 44Vn\%VPVP 4VP'4\)VP P*V,4pVPP,'g:VP P*V,pWPP.,pVP P1WW54Wx3#uupi)zxInitialize TrainerState, optionally restoring from checkpoint. Returns (epochs_trained, steps_trained_in_current_epoch).)rN)r@r2rr<r)r;r=is_hyper_param_searchrErD compute_stepsrr:risfiler0rrra_load_callback_staterrrrinit_training_references) rUr{rr|rrr`rrs &&&&&& rrTrainer._init_training_states|"!22<< ~M MrQ[\^`oQpM   ,1+< (&*&<&< #   I6)*& ! -"''.. GGLL/1C D3 3 &44RWW\\BXZl5mnDJ / 4:: F  % % ' !7!7;U!UVN99---151G1GJd1d..))2W2WW. ++D=MU==1 s H  H c  T \4;'g!VP;'g VPpVP;'d/\VPP P R^4^8HpV'dRpVP'dRVnRVnVP'd\WR7wVn VnV'gVP4VPVP4pW`PJpV'EdnV'dVP'd<\!VP4'd!\#VPVP4VPP%VP4pVPV4VPP%VP4Vn MVP'dn\'VP4P(R8XdJVPP%VPVPVP4wq`n VnMbVPP%VPVP4wq`n M*VPP%VP4Vn VP+VR7W`nVP'gVP'dV;VnVnVP'dVPVnVP'dG\/VPR4'd+\0P2P5VPR4\VPRR4pVe?VP6R8Xd.VP8'dVPP;W&4pVeVP'd2\=VPV\!VP4'*R 7M=\4'gVP'dVP?W0P4VPAV4VPCV4R F#p \EVPFV \W 44K% W PFn$Wb3#) zkWrap model, create optimizer and scheduler, and run accelerator.prepare. Returns (model, train_dataloader). fsdp_versionFNrADummySchedulergeneraterrload_module_strict)rr+r)%rr rrrr=rrFrrr r+rB _wrap_modelrrr_r&r rrrCrrdistrregister_fsdp_forward_methodr r!deepspeed_ulysses_dl_adapterr!r_load_optimizer_and_scheduler _load_scalersetattrr2r) rUr{rrdelay_optimizer_creationis_fsdp2ruse_accelerator_preparerZattrs &&&& rrTrainer._prepare_for_trainings#:#<#p#p@X@X#p#p\`\p\p ''qqWT5E5E5K5K5W5WYgij-kop-p ', $  % % % $D ).D &  $ $ $0>t0b -DND-'  ! ! #  !3!34#(::"5 # "''''N4::,F,F+DJJ8H8HI((00<%%e,!%!1!1!9!9$..!I***tD4E4E/F/O/OSc/c;?;K;K;S;SJJ0A0A<8~t'8)-(8(8(@(@T^^(\%~!--55dnnEDN ;#    4#;#;#;/4 3DJ+  $ $ $!//DN    tzz:.. 66tzz:NT%%';T B >bmm{:r}}}#//LLM]e  " -((()&&(>WefjfpfpWqSq)**d.B.B.B**+ACUCUV  . ./E F   4 5;D D))41D E;1A.&&rc  Rp Rp RpRp^pW*8XdsV eoV ^8dPVPP'g4\W;4pV ^, p WPP,pRpMV ^8XdVP V 4\ VR4'dVP V4\V4pW@PP,pV^8XdVPPp\VV4EFpVV^, 8wdVPPMTpVPVVVPP4wpp\V4Vn V'dVP V 4Rp\V4EFwppV ^, p V ^,VPP,^8H;'g V ^,V8HpVPPP!V4WPP,^8XdAVP"P%VPVP&VP(4VnVPPP*P-RR4'gAVPP.\0P28XgV\V4^, 8Xd\4P6pM,\8P:!VPP<VR7pV!4;_uu_4VP?VVV4pRRR4VPP@'d\C4'g\DPF!X4'g\DPH!V4'dXV;PJVPJ^VP&PL,VPN, , , un%MmVPJPXP8wd0\QRVPJP RVP 24hV;PJV, un%V;PR\UVPWV44, un)VPYV4V'Ed!Rp VPPZ^8dVP]V4p VP_WR 7p VP"PaVPVP&VP(4VnVPbPe4VP"PgVPVP&VP(4VnVPi4pVPPj'g_\mVPn\DPpPnPr\t34'gVPnPe4VPw4VP&;PL^, un&W,^,V, ,VP&n<VP"P{VPVP&VP(4VnVP}VPJV VVVVVVR 7M@VP"PVPVP&VP(4VnVP(P'g VP(P'gEKM VP(P'g VP(P'gEKM \C4'd\P!4V ^8dV\PR VP&PL R VP&P R 24RVP(nAVP"PVPVP&VP(4VnVP}VPJV VVVVVVR 7R# +'giELH;i)z&Run one full pass over the dataloader.NFT set_epochsync_each_batch)rz0Calculated loss must be on the original device: z but device in use is ) grad_norm)rwzXThere seems not to be a single sample in your epoch_iterator, stopping training at step zI! This is expected if you're using an IterableDataset and set num_steps (z.) higher than the number of available samples.)Hrrrr_load_rng_staterriterrRget_batch_samplesr r #current_gradient_accumulation_steps enumeratergradient_state_set_sync_gradientsr2 on_step_beginr=r< plugin_kwargsr distributed_typer DEEPSPEEDrrrrno_sync training_steplogging_nan_inf_filterrrisnanisinfrrrrrArfloating_point_ops_track_num_input_tokens max_grad_norm_clip_grad_norm_get_grad_normon_pre_optimizer_stepr+stepon_optimizer_steproptimizer_step_was_skippedr)rrrr5rr on_step_end_maybe_log_save_evaluateon_substep_endshould_epoch_stoprxm mark_steprrqr{ on_epoch_end)rUrrrrrrrrrrrrrrw rng_to_syncnum_update_steps_trainedepoch_iterator remainder update_step num_batches batch_samplesnum_items_in_batchiinputs do_sync_step sync_context tr_loss_steps&&&&&&&&&&&& rrTrainer._run_epoch~s    $%  "'='I-1$)):T:T:T#56F#g 59+IYYMrMr+r(" /14$$%;< #[ 1 1  & &u -./#YY%J%JJ > ==I !!9;UVK9DIcfgIg9h 55nw 150F0F~Wbdhdmdmdtdt0u -M-8;=7ID 4$$%;<# '}5 6  $qDII,Q,QQUVVvv[_bc[chvZv   //CCLQ))???1D#'#8#8#F#FtyyRVR\R\^b^j^j#kDL$$33AAEEFWY^__''88OV>V>h>hjr=stt --224OO%JJ**a/*',qN/J'JDJJ$#'#8#8#D#DTYYPTPZPZ\`\h\h#iDL11 !,"&32 $(#8#8#G#G SWS]S]_c_k_k#lDL<<111T\\5V5V5VU6V||---1R1R1R}WD " # # LLN !8 NNJJ**+,#zz3344bd  15DLL -,,99$))TZZQUQ]Q]^  %% MM     ' & U$^^s a& & a7 c \PR4V;PVPP 4, un\ VP PR4pVPV, p\RVVVP PR7pVP4VP PVR&WVR&VPPV4VPV4VPP 'd)VP P"eVP%4\'VP)V4VP P"R7pVPP*'dVP P"eVPP,^8XdvVFop\.P0P3WP P"4'dK>\PRV R 24\4P6!VR R 7Kq VP8P;VPVP VP<4VnVP?4VP@e,\CVPDVPFVPH4R Vn%\MVP PWV4#) zMFinalize training: metrics, best-model loading, cleanup. Returns TrainOutput.zU Training completed. Do not forget to share your model on huggingface.co/models =) gMbP?r num_samplesr total_flos train_loss)rbest_model_checkpointzDeleting older checkpoint [z] due to args.save_total_limitT) ignore_errorsF)'rrrrrmaxr=rrnr{ store_flosr!rrTlogrrvr#_load_best_modelrm_get_output_dirr9save_total_limitr:rsamefileshutilrmtreer2 on_train_endr<_finish_current_pushr*r)rrrr>r^) rUrrreffective_global_stepr"metricscheckpoints_sorted checkpoints &&&& rrTrainer._finalize_training#s op 4==#5#5#77 #DJJ$:$:E B,,/DD   )jj**    $ 5 5  *  44W=  99 + + + 0P0P0\  ! ! #-++E2$**JjJj  99 TZZ%E%E%QVZV_V_VpVptuVu0 ww'' JJ4T4TUUKK"=j\Ig hiMM*DA1 ,,99$))TZZQUQ]Q]^  !!#  # # / tzz4+C+CTEUEU V 4::11:GGrc<V^8dQhRS[PRS[S[S[P S[,3,RS[P S[,R,RS[P /#)rrrrNrf)rrrrrrr r)rrs"rrrTsbE!E!yyE!S%,,,,-E!"LL3.5 E!  E!rc  VPW4wrBV!4;_uu_4VP4\VPR4'd@\ VPP4'dVPP4VP V4p\ 4'dl\WVPP4pVP4P4PVPP4uuRRR4#VP4;_uu_4VPWVR7pRRR4?VPP eAVP"P$VPP ,^8Xd \'4/pVPP(\*P,\*P.39dVP14VR&VPP2^8dXP54pVP6'dVf"VP8fXVP:, pVP<P>\@PB8XdRVR&VP<PD!X3/VBVP4uuRRR4# +'giEL;i +'giR#;i)a  Perform a training step on a batch of inputs. Subclass and override to inject custom behavior. Args: model (`nn.Module`): The model to train. inputs (`dict[str, torch.Tensor | Any]`): The inputs and targets of the model. The dictionary will be unpacked before being fed to the model. Most models expect the targets under the argument `labels`. Check your model's documentation for all accepted arguments. Return: `torch.Tensor`: The tensor with training loss on this batch. rNrrwF scale_wrt_gas)# _prepare_context_parallel_inputsrrr+ry_prepare_inputsrrrr reduce_meandetachtor compute_loss_context_manager compute_losstorch_empty_cache_stepsr=rrrrrLOMOADALOMOrrmeanrrrrrrrbackward)rUrrr cp_contextloss_mblossrs&&&& rrTrainer.training_stepTs2"BB5Q \\ KKMt~~w//HT^^=Q=Q4R4R$$&))&1F&((.udii>c>cd**,335889I9IJ\2244((K](^5 11=JJ**TYY-N-NNRSS"$Fyy>#6#68N8N"OO*.*A*A*C'yy"yy{2226H6PVZVlVlVtdFFF00O4M4MM*/'    % %d 5f 5;;=S\544\\s2C-KK3J= C+K3A?K= K  K K" c <V^8dQhRS[PRS[S[S[P S[,3,RS[RS[P S[,R,RS[P S[ S[P S[3,,/#)rrrreturn_outputsrNrf) rrrrrrr rrr)rrs"rrrsW;W;yyW;S%,,,,-W; W; "LL3.5 W; ellC/0 0 W;rc  $\VPRR4pVeWVPR8XdFVP'd4VPP 'd\ VPWW54#VPfVPeRV9dVPR4pMRpVP'd/pVeWGR&/VCVCpV!R/VBpVPe0Vf\PR4VPVVVR7p EM5VeVPPV4p \V 4'd%V PPP!4MV P!4p V \"P$!49dVPWRR 7p MVPW4p M\'V\(4'dTR V9dM\+R R P-VP/44 R R P-VP/44 R24h\'V\(4'd VR ,MV^,p VP0P2'dVP'gVP'dVe}VPP4p \VPRR4;peWP6,p YP0P8^8:dT MVP0P8,p V'dW3#T #)a How the loss is computed by Trainer. By default, all models return the loss in the first element. Args: model (`nn.Module`): The model to compute the loss for. inputs (`dict[str, torch.Tensor | Any]`): The input data for the model. return_outputs (`bool`, *optional*, defaults to `False`): Whether to return the model outputs along with the loss. num_items_in_batch (Optional[torch.Tensor], *optional*): The number of items in the batch. If not passed, the loss is computed using the default batch size reduction logic. Returns: The loss of the model along with its output if return_outputs was set to True Subclass and override for custom behavior. If you are not using `num_items_in_batch` when computing your loss, make sure to overwrite `self.model_accepts_loss_kwargs` to `False`. Otherwise, the loss calculation might be slightly inaccurate when performing gradient accumulation. rNrlabelsrz|Trainer: `compute_loss_func` is defined but `labels=None`. Your custom loss function will still be called with labels=None. r7T) shift_labelsrGzJThe model did not return a loss from the inputs, only the following keys: rz,. For reference, the inputs it received are .rg)rrr r!rtrainingr"r(rrrrrqr2r_r$ _get_namer3rr)rrr0r%raverage_tokens_across_devices num_processesrr) rUrrrJrrZrLroutputsrGrXr: loss_scales &&&&& rr?Trainer.compute_losss6T%%';T B >bmm{:r}}}QUQ[Q[QdQdQd,T-=-=una a    +t/E/E/QW_ciWiZZ)FF  ) ) )F!-/A+,))&)F/&/  ! ! -~X))#5*D  "..;;EBO"/22 **00::<$..0  >EEGG**7*N**7;'4((V7-B `xx /00\]`]e]eflfqfqfs]t\uuvx '1$&?&?76?WQZD II 3 3 3///43I3I3I".))77Jd..0DdKKXzz) ))//Q"6JDIIOO KD"0:d:rc4<V^8dQhRS[P/#r)r ExitStack)rrs"rrrs  j.B.B rc \P!4pVP4p\V\P4'gVP V4V#)z6 A helper wrapper to group together context managers. )rrWautocast_smart_context_managerr)r enter_context)rU ctx_stack autocast_ctxs& rr>$Trainer.compute_loss_context_managersG((* ::< , (>(>??  # #L 1rcH<V^8dQhRS[R,RS[P/#)r cache_enabledNrf)rrAbstractContextManager)rrs"rrrs&((D4K(S]StSt(rc ,\P!4#)z A helper wrapper that creates an appropriate context manager for `autocast` while feeding it the desired arguments, depending on the situation. We rely on accelerate for autocast, hence we do nothing here. )rr)rUr_s&&rrY&Trainer.autocast_smart_context_managers %%''rc<V^8dQhRS[PRS[PS[,R,RS[PRRRS[RS[S[,R,R S[R S[R,R R/ #) rtr_lossrNrrrrrrrwrf)rrrrrrr)rrs"rrrs0^0^0^<<%'$.0^yy 0^ 6 0^  0^#3i$.0^0^t|0^ 0^rc  VPP'EdkVPPVP8EdE\ 4'd\ P!4/p \WPP4P4P4p W,pWPPVP, , V R&Ve6\V\P4'dVP4MTV R&VeWR&MVP!4V R&V;P"V , unVPPVnVP%4VP'W4Rp VPP('d]VP+WF4p VP-WR7p VPP.\0P28XdWPnVPP4'dTVP7W44VP8P;VPVPVP4VnR#R#)z\Log metrics, run evaluation, and save checkpoints if the current training state requires it.rGNrrw)r1r)r< should_logr=rrrr rrOrrrCrr)rrrrr&r'should_evaluater_determine_best_metricrtr\rur9_save_checkpointr2on_save) rUrdrrrrrrrwlogstr_loss_scalarr1is_new_best_metrics &&&&&&&&& rr  Trainer._maybe_log_save_evaluates << " " "tzz'='=@\@\'\%'' %'D+7II4K4KLQQSXXZN  G)ZZ-C-CdFbFb-bcDL$8B9ell8[8[INN$4aj[!((5_%(,(?(?(A_%  # #~ 5 #+/::+A+AD ( OO  HHT & << ' ' 'nnUAG!%!pRV9d VR,M*VP'dVR,R,MVR,NK@ pp\ RV44pVEekVPP'dUVPP^8d9VPPVPV44P 4pM7VPP^8dW0PP,p\P!V4'dVPV4pVPP^8dKVP!4^8Xd6VP#^4P%VPPR4p\'VPRR4;p'dW7P(,pV#uupi \ \ 3dELi;i)aN Counts the number of items in the batches to properly scale the loss. Args: batch_samples (`list`): List of batches device (`torch.device`): The device on which the number of items in the batch should be. Returns: None if the number of items in the batch doesn't need to be computed else the number of items in the batch NrLrMc3^"TF#qPR4P4xK% R#5i)dN)nerV)rrLs& rr2Trainer._get_num_items_in_batch..ps$(^M]64)<)<)>)>M]s+-r).NNr)r rrr#rVrzrrrQrrgatherr=rrrdim unsqueezeexpandrnon_data_parallel_size)rUrr rcount_num_items_in_batchbatchlabels_for_countrZs&&& rrtTrainer._get_num_items_in_batchKs"   "  M!,,   ..66))5 ! $ "/ $ "/ &..)///x1x ) "/ !$&)(^M](^%^"  )yy66699''!+)-)9)9)@)@ASAVAVW]A^)_)c)c)e&1$&899??%J"122%7%:%:6%B"99??Q&+=+A+A+Cq+H);)E)Ea)H)O)OPTPYPYP_P_ac)d& !1!13GNN2N);?X?X)X&!!?$~.  s*H*&H%H% H*%H**H?>H?cn<V^8dQhRS[PS[,RS[PS[,/#)rr~rf)rrr )rrs"rrrs-5<<##5%,,:Lrc  a\V\4'dD\V4!VP4UUu/uFwr#VSP V4bK upp4#\V\ \ 34'd\V4!V3RlV44#\V\P4'dRSPP/pSP'd\P!V4'g\P!V4'dJVPRSPP P"P$P'4/4VP(!R/VB#V#uuppi)zl Prepares one `data` before feeding it to the model, be it a tensor or a nested list/dictionary of tensors. c3F<"TFpSPV4xK R#5ir)_prepare_input)rvrUs& rr)Trainer._prepare_input..sCdd11!44d!r dtyperg)r)rritemsrrrrrrr ris_floating_point is_complexrrr=r hf_ds_configrr=)rUr~rrrsf& rrTrainer._prepare_inputs dG $ $:TZZ\R\TQq$"5"5a"88\RS S udm , ,:CdCC C ell + + 0 01F(((e.E.Ed.K.KuO_O_`dOeOe w(8(8(>(>(O(O(\(\(b(b(def77$V$ $ SsE1 c<V^8dQhRS[S[S[PS[,3,RS[S[S[PS[,3,/#rrrf)rrrrr )rrs"rrrsE  d3 s0B+B&C SRWR^R^adRdMdHe rc VPV4p\V4^8Xd)\RRPVP4 R24hV#)z Prepare `inputs` before feeding them to the model, converting them to tensors if they are not already and handling potential state. zThe batch received was empty, your model won't be able to train on it. Double-check that your training dataset contains keys expected by the model: rrN)rr rr0rC)rUrs&&rr:Trainer._prepare_inputssY $$V, v;! IILRVRiRiIjHkkln   rc <V^8dQhRS[PRS[S[S[P S[,3,RS[S[S[S[S[P S[,3,3,/#)rrrrf) rrrrrrr rr)rrs"rrrs^\.\.YY\.(,S%,,2D-D(E\. xc5<<##5566 7\.rc \VPRR4Ee VPPP'EdVPPPR8Xd\ VR4'd?VP PR8wd$\RVP P R24hRV9db\PR 4R V9dFRp\PPVR ,RVR 7pVR,P4VR&R V9d~\PR 4\P !VR,P#^4VR,P$R7P'VR,P#^4R4VR &.p.pRV9d*VP)VR,4VP)^4R V9d*VP)VR ,4VP)^4RV9d*VP)VR,4VP)^4RV9d\VRR4'g-VR,p\+V4'g \R4hRVnVP,'dCVR,pVP/4^8Xd$VP)V4VP)^4MR V9d5VR ,e*VP)VR ,4VP)^4\1VPP2VV\5V4R7V3#\6P8V3#)a3 Prepare inputs for context parallelism by setting up buffers and validation. Args: model: The model being trained inputs: Input tensors to prepare Returns: tuple: (context_manager, prepared_inputs) where context_manager is either the context parallelism wrapper or a no-op context rNrrsdpazIContext parallelism is supported only with SDPA attention, you are using rNrMz7Shift labels not found in the inputs, shifting manuallyrL)value position_idsz9Position IDs not found in the inputs, generating manually input_idsrattention_mask_attn_mask_causal_checkedFzContext parallelism only supports causal attention masks. The provided attention_mask is not causal. Please ensure your data uses causal masking (lower triangular) or remove the attention_mask to use the model's default causal masking.T)buffersbuffer_seq_dimsno_restore_buffersr{)r):NNNr~r)rrr cp_enabled cp_backendrr_attn_implementationrr warning_oncer functionalpad contiguousrarangesizer rrqrMrrrmaybe_context_parallelrrr)rUrr _ignore_indexrLrrrs&&& rr9(Trainer._prepare_context_parallel_inputss D$$&:D A M  33>>>22==H5(++||88FB(ghmhthtiJiJhKKLM"/''(ab6)(, !#!2!26(3CVS`!2!a171I1I1K~.V+##$_`).;',,Q/{8K8R8R*& ,11!4b9~&G Of$vk23&&q)6!vh/0&&q)'vn56&&q)6)t%@%HH&,,<%=N3NCC(f 6:D2111%+,<%=N%))+q0~6'..q1'F>,B,Nvn56&&q)  77 /#&w<    %%v--rc d<V^8dQhRS[RS[RS[S[S[S[S[S[S[R,S[3,/#)rrrrfN)rtrrr)rrs"rrr sBD D %D 3=D sCc3d C7 8D rc VPpV^8p\V4'd \V4MRpVPV4pVP 4pV^8d Ve WW,pVeo\ WQP ,\WQP ,^84,^4pV'd(\P!VPV,4pV'dVPV4p VP^8d,VX,\W8,^84,p W6,p M\P!VP4p VPV4VP,p MaVP^8d9\Pp TpWaP,p VPV,p M\RVP 24hVeTMW1P ,p V XV V VV V3#)z Calculates and returns the following values: - `num_train_epochs` - `num_update_steps_per_epoch` - `num_examples` - `num_train_samples` - `total_train_batch_size` - `steps_in_epoch` (total batches per epoch) - `max_steps` NzYargs.max_steps must be set to a positive value if dataloader does not have a length, was )r{rgr get_total_train_batch_size get_sp_sizer%rrmathceilr|rsysmaxsizer) rUrrr{ epoch_basedlen_dataloaderrsp_sizerrr|rrs &&& rr#Trainer.set_initial_training_values sNN !m ,6z,B,BZ!%!@!@!F""$ Q;>5+5N  %),"B"BBn'G'GG!KLM* &  IId&;&;>X&XY ,,Z8L~~!#,0J#JS:Q>N$ %.$F!#'99T-B-B#C $($5$5j$ADDYDY$Y! ^^a "{{ )2 &1NNBL $1G G NN#% ,:+E9WwWwKw  &   "    rc&<V^8dQhRS[RS[/#)rrrf)rtr)rrs"rrrM s YY/@YSYrc VPVP4,VP4,VP4,pVPVP ,V,#)aCalculates total batch size (micro_batch * grad_accum * dp_world_size). Accounts for all parallelism dimensions: TP, CP, and SP. Formula: dp_world_size = world_size // (tp_size * cp_size * sp_size) Where: - TP (Tensor Parallelism): Model layers split across GPUs - CP (Context Parallelism): Sequences split using Ring Attention (FSDP2) - SP (Sequence Parallelism): Sequences split using ALST/Ulysses (DeepSpeed) All dimensions are separate and multiplicative: world_size = dp_size * tp_size * cp_size * sp_size )r get_tp_size get_cp_sizerrEr)rUr dp_world_sizes&& rr"Trainer.get_total_train_batch_sizeM sT4+;+;+==AQAQASSW[WgWgWii %%(H(HH=XXrc <V^8dQhRS[/#rr?)rrs"rrr_ Src |\VPRR4f^#VPPpVP#)zGet the sequence parallel sizerN)rrrrrUrZs& rrTrainer.get_sp_size_ 6 4##%94 @ H!!44B:: rc <V^8dQhRS[/#rr?)rrs"rrrg rrc |\VPRR4f^#VPPpVP#)zGet the context parallel sizerN)rrrcp_sizers& rrTrainer.get_cp_sizeg rrc <V^8dQhRS[/#rr?)rrs"rrro s  S rc \VPRR4;peV#VP'dL\VPRR4;p'd-VPP R/4P R^4#^#)zGGet the tensor parallel size from either the model or DeepSpeed config._tp_sizeNrtensor_parallel autotp_size)rrrrrr )rUmodel_tpdeepspeed_configs& rrTrainer.get_tp_sizeo sv  J= =H JO  $ $ $gdiiQfhl>m*m*:*m#**../@"EII-YZ[ [rch<V^8dQhRS[PRS[RS[R,RS[P/#)rrrOrNrf)rrrr)rrs"rrr} s:   d z\`O` lnlulu rc VPPVRR7VJdV#\4'dX\V\P P 4'dV#\P !WPPR7#VPP^8d=\VRR4'g*\VRR4'g\P!V4pV'gV#VP'd#\WPVP4pV#\!4'd@\P"P%V\'\(P*!R44.R7pV#)zMWrap `model` for distributed training if needed (DDP, FSDP, SageMaker, etc.).Fkeep_torch_compile)backward_passes_per_stepis_loaded_in_8bitis_loaded_in_4bitSMDATAPARALLEL_LOCAL_RANK) device_ids)rr2rr)rorDistributedModelrrrrr DataParallelr r-rIrparallelDistributedDataParallelrr:getenv)rUrrOrs&&&&rrTrainer._wrap_model} s    ( (5 ( IQV VL " $ $%!;!;<< '' HmHmn n IIOOa E#6>>E#6>>OOE*EL  # # #'yy$:U:UVE  % & &KK773ryy1L'M#N"O8E rc Wn\VP4VP'd\ VP RR7VnVP VnVP 'dVPPpVP\^VPP4,VPn\VPVPRR7W PnR#R#)zwFree memory, reset model wrapping, and update DeepSpeed config for the new batch size when using `auto_find_batch_size`T) recursive)rN) rErrrr2rrrrr%rr$r)rUr original_bss&& rrTrainer._update_auto_batch_size s",t))*    %djjDADJ!ZZ  $ $ $))??K484J4JcRSUYU^U^UdUdNe4eDII 1 '(8(8$))Z^ _4?II 1 %rc VPPR8XdR#\VPRR4pW!9d\P R4R#VPPR8XdRV9dVR,P 4pMVPec\VPR4'dGVPPe/W,VPP8gP 4pMB\P R 4W,P4pMW,P4p\P!W0PP\PR 7pVP;P VP"P%V4P 4P'4, unR#) z>Count input tokens seen (all or non-padding) and update state.rjNmain_input_namerzTried to track the number of tokens seen, however the current model is not configured properly to know what item is the input. To fix this, add a `main_input_name` attribute to the model class you are using. non_paddingr pad_token_idz\Could not determine method to count non-padding tokens, falling back to counting all tokens.)r r)rinclude_num_input_tokens_seenrrrrqrVrrrrXr as_tensorr int64r=rrrr)rUrr input_tokenss&& rrTrainer._track_num_input_tokens sT 99 2 2d : !$**.?M  ( NNR   99 2 2m C6)%&67;;= %%1D11>BB))66B & 74;P;P;];] ]bbd r &6<<> !288:L|II>33DII4K4KL L//0@0@0BDIID[D[\\rc Vf4VPPVP4\R44pVPP\ P 8Xd#\VR4'dVP4pV#)z+Return the gradient norm as a Python float.infr) rrrrrrrrr)rUrrs&&&rrTrainer._get_grad_norm sg  ((889I9I9KUSX\ZI    , ,0I0I Iy&))%NN, rc <V^8dQhRS[S[S[S[3,,R,RS[S[,R,RS[RS[S[S[3,/#)rrN ignore_keysmetric_key_prefixrf)rrrrr)rrs"rrr s^bbS'\ 22T9b#Y%b b c5j  brc  LVRJpV'dTM VPp\V\4'dR/pVP4F9wrgTP V'dTMTVV RV 2R7pVP V4K; V#VP P4VPV4p VP'd \V 4p \P!4p TPT RVPfRMRVVR7p VPPVPP ,p V R2V P"9dWP"V R2,, p V P"P \%VV V P&\(P*!V P&V , 4R74VP-V P"4\.P0VPP29d*\4P6!\8P:!44VP<P?VPVP@VPBV P"4Vn!VP PEV P"4V P"#) a Run evaluation and returns metrics. The calling script will be responsible for providing a method to compute metrics, as they are task-dependent (pass it to the init `compute_metrics` argument). You can also subclass and override this method to inject custom behavior. Args: eval_dataset (`Dataset` | dict[str, `Dataset`], *optional*): Pass a dataset if you wish to override `self.eval_dataset`. If it is a [`~datasets.Dataset`], columns not accepted by the `model.forward()` method are automatically removed. If it is a dictionary, it will evaluate on each dataset, prepending the dictionary key to the metric name. Datasets must implement the `__len__` method. If you pass a dictionary with names of datasets as keys and datasets as values, evaluate will run separate evaluations on each dataset. This can be useful to monitor how training affects other datasets or simply to get a more fine-grained evaluation. When used with `load_best_model_at_end`, make sure `metric_for_best_model` references exactly one of the datasets. If you, for example, pass in `{"data1": data1, "data2": data2}` for two datasets `data1` and `data2`, you could specify `metric_for_best_model="eval_data1_loss"` for using the loss on `data1` and `metric_for_best_model="eval_data2_loss"` for the loss on `data2`. ignore_keys (`list[str]`, *optional*): A list of keys in the output of your model (if it is a dictionary) that should be ignored when gathering predictions. metric_key_prefix (`str`, *optional*, defaults to `"eval"`): An optional prefix to be used as the metrics key prefix. For example the metrics "bleu" will be named "eval_bleu" if the prefix is "eval" (default) Returns: A dictionary containing the evaluation loss and the potential metrics computed from the predictions. The dictionary also contains the epoch number which comes from the training state. Nru)rrrrT)rprediction_loss_onlyrr_model_preparation_timer)#rr)rrevaluaterrrrrIr,revaluation_looprrrrr1rnr rrr'rTPU_METRICS_DEBUGrZr  master_printmetmetrics_reportr2 on_evaluater=r<rT) rUrrroverrider1eval_dataset_name _eval_datasetdataset_metricseval_dataloaderroutputtotal_batch_sizes &&&& rrTrainer.evaluate s)Z t+'/|T5F5F lD ) )G4@4F4F4H0!"&--2:@Q +):(;1=N>U)VW WJ !"..))F$6$69I$IJ      ( (DIIOO ; OOC..0 1,,88DJJPTP\P\^d^l^lm  44V^^D~~rc j<V^8dQhRS[RS[RS[R,RS[S[,R,RS[RS[/#)rrrrNrrrf)rrrrrV)rrs"rrrL sdMuMuMuMu#Tk Mu #Y% Mu  Mu Murc  VPpVeTM VPpVP'd VPf\ V^RR7wpVP VP RR7p\VPP4^8XEdWP JEd \P!4p VP'g/VP'd8VPP'gVPPV4MVPPVRR7p\\P!4V , ^4VnVP'dWnWP JdWnVP'dVP"VnVP$'g|VP&'d-VP)\*P,VP.R7pM=VP0'd,VP)\*P2VP.R7pVPP4p \6P9RV R 24\;V4'd)\6P9R VP=V4 24M\6P9R 4\6P9R V 24\?VR 4'd,\AVPB4'dVPC4\?VPDR 4'd@\AVPDPB4'dVPDPC4WPFn$\KVRR4p \MVPPNR!R7p \MVPPNR!R7p \MVPPNR!R7p\MVPPNR!R7pRp/p^p\QV4EFwpp\SV4pVeVV, pV fTp VPUVVW4R7wppp\KVP RR4pRVPV9dVPYVV,4MRp\[4'd\\P^!4Ve2VPaVPcV 44pV PeV4VedVPPgV^R!R7pVPaV4pVPPh'dVR8XdVPeV4VeVPPgV^R!R7pVeVPPgV^R!R7pVPjeVPkVV4pVPaV4pVPPh'dVR8XdV PeV4VeFVPaV4pVPPh'dVR8XdVPeV4VPFPmW`PnVPp4Vn8VPPh'dVPre{VewVesVPPtPvp/pRVPV9dTMRVR&RVPV9dTMRVR&VPs\yR"RVRV/VBVR7p????\*PzP}4EK0VP~fEKAV^,VP~,^8XgEKcV P4V P4VP4VP4????\*PzP}4EK VPPVn0V P4p V P4p VP4pVP4p\;V 4'd \V 4pMZ\V \4'd \KV R^4^8dV P<pM%\;V4'dVP=V4pMTpV^8Xd V^8dTpVPretV epVelVPPh'gPRVPV9dT MRVR&RVPV9dTMRVR&VPs\yR"RV RV/VB4pMVf/p\V4p\V \4'dBV 'd:\P!V 4P4P4VV R2&MD\V \P4'd%V P4P4VV R2&\?VR4'dVP VV R2&\VP44F8pVPV R24'dKVPV4VV RV 2&K: \WVVR 7#)#zy Prediction/evaluation loop, shared by `Trainer.evaluate()` and `Trainer.predict()`. Works both with or without labels. NT)r> inferenceF)rO)evaluation_mode)rr z ***** Running z *****rz Num examples: Unknownz Batch size = rr) padding_indexrrrr)r pad_index PredictionrGlosses predictionsr#)rir_lossmodel_preparation_timerru)rr#r1r r{rg)Prrrrr rrr r_modelsrr torch_compiler  prepare_modelroundrrr>rr=rfloat16r rbfloat16rrrrgrrryrr+r2r rrEeval_do_concat_batchesrrJprediction_stepinclude_for_metricsrrr rrrepeataddpad_across_processesrrron_prediction_stepr=r<rrend_of_dataloaderrWcuda empty_cacheeval_accumulation_stepsto_cpu_and_numpyr get_arraysr)rFrcrrP concatenaterCrndarrayr% startswithrrV)rUrrrrrrrurrrr all_losses all_preds all_labels all_inputsr1eval_set_kwargsobserved_num_examplesrrobserved_batch_sizerlogitsrLr inputs_decode is_last_step batch_kwargsr keys&&&&&& rrTrainer.evaluation_loopL syy7K7W3]a]v]v  $ $ $)?!$1MDAq  e < t'' (A -%::2EJ,,,1E1E1EdiiNeNeNe  ((/%%33E43P  +0 j0H!*LD '###" JJ&%*"(((!%!3!3"""u}}T[[I$$$u~~dkkJYY..  &{m6:; j ! ! KK+D,=,=j,I+JK L KK1 2 oj\23 5& ! !huzz&:&: JJL 4>>6 * *x8K8K/L/L NN   !0:-z9d; 'tyy'G'GW[\ %dii&F&FVZ[ &tyy'G'GW[\ &tyy'G'GW[\ !"&j1LD&"1&"9 ".%)<<%%!4J&*%9%9%I]%9%w "FFF%djj2C[QO@HDLdLd@d##F?$;>v1X\>]!))>>v1X\>]55A!??OF--f5yy333{l7RMM&)!--f5yy333{l7RNN6*00CCD**VZVbVbcDLyy+++''38JvOa#'#3#3#B#B#T#TL#%L7=AYAY7YV_cL*7?4C[C[7[VaeL*"22&\6\V\|\'33G FFF &&(--9tax4KgKg>gkl>l++-**,++-++-FFF &&(G2L $//BB **, ((* **, **,  l # #l+K &: ; ; Vdfg@hkl@l&33K*%%"// ; 3 !  5 9/K  ,%&II0006<@X@X6X ^bOH %6>$BZBZ6Z `dOH %**^9^ ^o^G_G(0 j$ ' 'J35>>*3M3R3R3T3Y3Y3[G()/ 0  BJJ / /3=??3D3I3I3KG()/ 0 41 2 2EIE`E`G())@A B 'C>>%6$7q"9::8? C8H,-Qse45()SZhsttrcP<V^8dQhRS[RS[S[,R,RS[RS[/#)rrrNrrf)rrrrZ)rrs"rrr s<;t;t#;t26s)d2B;t^a;t ;trc  RVPP4VPV4p\P!4pVP VRW#R7pVP P VP P,pV R2VP9dWVPV R2,, pVPP\VVVP\P!VPV, 4R74VPPVP VP VP"VP4VnVPP%VP4\'VP(VP*VPR7#)a Run prediction and returns predictions and potential metrics. Depending on the dataset and your use case, your test dataset may contain labels. In that case, this method will also return metrics, like in `evaluate()`. Args: test_dataset (`Dataset`): Dataset to run the predictions on. If it is an `datasets.Dataset`, columns not accepted by the `model.forward()` method are automatically removed. Has to implement the method `__len__` ignore_keys (`list[str]`, *optional*): A list of keys in the output of your model (if it is a dictionary) that should be ignored when gathering predictions. metric_key_prefix (`str`, *optional*, defaults to `"test"`): An optional prefix to be used as the metrics key prefix. For example the metrics "bleu" will be named "test_bleu" if the prefix is "test" (default) If your predictions or labels have different sequence length (for instance because you're doing dynamic padding in a token classification task) the predictions will be padded (on the right) to allow for concatenation into one array. The padding index is -100. Returns: *NamedTuple* A namedtuple with the following keys: - predictions (`np.ndarray`): The predictions on `test_dataset`. - label_ids (`np.ndarray`, *optional*): The labels (if the dataset contained some). - metrics (`dict[str, float]`, *optional*): The potential dictionary of metrics (if the dataset contained labels). r)rrrrr)rr#r1)rrrrrrrrr1rrnr rrr2 on_predictr=r<rTrZrr#)rUrrrtest_dataloaderrr r s&&&& rpredictTrainer.predict sMH ""$22<@YY[ %% ;&  9944tyy7K7KK 7 8FNN J ..,=+>>U)VW WJ !"..))F$6$69I$IJ   ,,77 4::t||]c]k]kl  44V^^DF,>,>&JZJZdjdrdrssrc$<V^8dQhRS[PRS[S[S[P S[,3,RS[RS[S[,R,RS[ S[P R,S[P R,S[P R,3,/#)rrrrrNrf) rrrrrrr rrr)rrs"rrrX sg&g&yyg&S%,,,,-g&# g& #Y% g& u||d"ELL4$79LL M g&rc  aa\VP4^8XdRMM\;QJd)V3RlVP4F 'dK RM RM!V3RlVP44pSPR4pVf VPp\VP4^8H;'dTpVP S4oSfB\ VPR4'd$\VPPRR.4oM.oV'g V'dj\\;QJd!.V3R lVP4FNK 5M!V3R lVP444p\V4^8Xd V^,pMRp\P!4;_uu_4\4'EdL\VS4p V'g V'd\!V \"4'dYV R ,p \;QJd%.V3R lV P%44FNK 5M!V3R lV P%444p MV ^,p V R ,p V P'4P)4P+4p \-V 4p EMRp \!V \"4'dP\;QJd%.V3R lV P%44FNK 5M!V3R lV P%444p MT p \-V 4p EMV'g V'dVP/4;_uu_4VP1S.VP2P44pVP7VSRVR7wrRRR4X P)4P94p \!X\"4'dP\;QJd%.V3RlVP%44FNK 5M!V3RlVP%444p MVR ,p MRp VP/4;_uu_4V!R/SBpRRR4\!X\"4'dP\;QJd%.V3RlVP%44FNK 5M!V3RlVP%444p MTp RRR4V'dX RR3#\X 4p \V 4^8Xd V ^,p X W3# +'giELm;i +'giL;i +'giLl;i)ap Perform an evaluation step on `model` using `inputs`. Subclass and override to inject custom behavior. Args: model (`nn.Module`): The model to evaluate. inputs (`dict[str, torch.Tensor | Any]`): The inputs and targets of the model. The dictionary will be unpacked before being fed to the model. Most models expect the targets under the argument `labels`. Check your model's documentation for all accepted arguments. prediction_loss_only (`bool`): Whether or not to return the loss only. ignore_keys (`list[str]`, *optional*): A list of keys in the output of your model (if it is a dictionary) that should be ignored when gathering predictions. Return: tuple[Optional[torch.Tensor], Optional[torch.Tensor], Optional[torch.Tensor]]: A tuple with the loss, logits and labels (each being optional). Fc3J<"TFpSPV4RJxK R#5irr )rrrs& rr*Trainer.prediction_step..v s"Aveu`a&**Q-W[B[eus #T return_lossNrkeys_to_ignore_at_inferencepast_key_valuesc3F<"TFpSPV4xK R#5irrF)rnamers& rrrG s(WFVdD)9)9FVrrGc3N<"TFwrVSR.,9gKVxK R#5irGNrgrrrrs& rrrG s()o8KqXcgmfnXnOn!!8K% %r~c3><"TFwrVS9gKVxK R#5irrgrOs& rrrG s)d8KqXcOc!!8K  )rJrc3N<"TFwrVSR.,9gKVxK R#5irNrgrOs& rrrG s&&h_TQQ\`f_gQgHgqq_rPc3><"TFwrVS9gKVxK R#5irrgrOs& rrrG s&]_TQQ\H\qq_rRrg)r r"allr rr:rrrrrNrrno_gradrrr)rrr;r<rrr>rtrr r?rC)rUrrrr has_labelsrHloss_without_labelsrL raw_outputsrF logits_mbrGr7rrSs&&f&f rr!Trainer.prediction_stepX s<"$"2"23q8UccAveieueuAvcccAveieueuAv>v jj/  ..K!$"2"23q8HH[%%f-  tzz8,,%djj&7&79VYjXkl   ,"55(WdFVFV(W55(WdFVFV(W#WXF6{aF ]]__&((.uf= !4!+t44"-f"5$)E)o 8I8I8K)oEE)o 8I8I8K)o$o "-a.$/O "..0779==?D.y9FD!+t44$)E)d 8I8I8K)dEE)d 8I8I8K)d$d $/ .y9F!4::<<-1-I-I6(TXT]T]TdTd-e*(,(9(9!6$Se):) =  ;;=--/D!'400!&&hW]]_&h&hW]]_&h!h!(D::<<"'/&/=!'400!&&]W]]_&]&]W]]_&]!]!(QT $% %v& v;! AYFf%%7=<<=+>>--/<<>''?+@+@@.2ll.Ft||E*dSYRZOHggll499#7#7BGii**Grc<<V^8dQhRS[PRRRR/#)rrrrrfNrr)rrs"rrr s-CCbiiC8^CcgCrc \ RVPP 2pVPfVfVP 4VP VR7p\ PPWC4pVPVRR7VPP\P\P\P39EdVPP 'd\#4'd\$P&!R4MdVPP(\*P,8Xd\.P0!4M%\34'd\4P0!4\ RVPP 2p\ PPWF4p\ PP7V4'dWpPnVPP:'g4VP=V4VP?V4VPAV4VPPB'Ed2VPDPFVPH.,Uu.uFp\KV\L4'gKVNK upFpVPNPPp VP4p \KVPPRV ,\T4'd/VPPRV ,PWV 4KWPPRV &K VPPY\ PPV\Z44VPP\'dVP_V4VPPB'd:\aVVPPbVPP8RR7R#R#uupi)zSSave model checkpoint, optimizer, scheduler, scaler, RNG states, and trainer state.-N)rT_internal_callrv)rr*r# use_mtime)2rTr=rr@r&r)r:rr0 save_modelrrtr\STEPSEPOCHrubest_global_steprr  rendezvousrrsrrbarrierrroexistsr#r_save_optimizer_and_scheduler _save_scaler_save_rng_stater9r2rr<r)r;rrrrrq save_to_jsonrr7_push_from_checkpointrjr*) rUrrcheckpoint_folderrwrbest_checkpoint_folderbest_checkpoint_dirr`cb_namecb_states &&& rriTrainer._save_checkpoint s 55Qtzz7M7M6NO  ! ! )em OO &&U&3WW\\'=   48 II # # (:(:L$"Gm$"rv$"rc  RpVPPEerVPPpVPR4'gRV 2pW,pTPP 'd\PM\PpTPPfBTPP 'd \R4M \R4TPn T!YPPP4'dYPPn TPP\P\P \P"39d&TPP$TPnR pV# \d0p\RT R\ TP 44 R24ThRp?ii;i) z Determine if the model should be saved based on the evaluation metrics. Returns: bool: True if a new best metric was found, else False FNr^r_r`zJ. Consider changing the `metric_for_best_model` via the TrainingArguments.z-infrT)rrwr/rbrr%greater_is_betterrPgreaterlessr= best_metricrrtr\rrrurr)rUr1rrmrc metric_valuerdoperators&&& rrhTrainer._determine_best_metric9 sj# 99 * * 6"ii==O"--g66$)/):"; &7 &*YY%@%@%@rzzbggHzz%%-:>)):U:U:Uv[`af[g & jj&<&<==)5 &99**|/A/AP@!VPBPD4PFPI449pV'd;\KVP 4'd VPBPEVR R7EM1VPBPEV4EMVPL'd\OVPPP4PRVPPVP V3/\U4B\WVPPP4PRVPPVPVP V4MmVPPX'dR\ZP!VPP 4\PPV\44VP:;'d \]VP$\^4'*pVPPX'dVP:'d V'd\4'g\ P"!R R 7;_uu_4p\ZP!VP$P 4\PPV\&44R R R 4\)X4R #R #R #R # +'giEL;i +'giL7;i)zBSave optimizer and learning rate scheduler states to `output_dir`.saving_optimizer_statesr+shard_metadatar-of-r|F) master_onlyTrecordN)gather_if_shard)rv3exclude_frozen_parametersr)0rr rrSr+ state_dictrget_shard_metadatarr:rr0rr rOPTIMIZER_NAMEr%catch_warningsrSCHEDULER_NAMErPrlocal_state_dictrorrdp_rankr=rpshard_optimizer_staterrrrrsave_checkpointrr%r_rrrrr%rr9rr)r)rUroptmcaught_warningsopt_state_dict accept_exclude_frozen_parametersis_deepspeed_custom_schedulers&& rr%Trainer._save_optimizer_and_scheduler s ! # # MM3 4***!:!:!<$djj&C&C&EGGLL"d499+B+B*C4 H\H\G]]^_m^n$o!& 113RWW\\*n5]^((55))446 ZQ_8`a#O465% & &!^^<  ! ! !   &&22D4D4DdjjR\ `t`v    &&22D4D4DdnnVZV`V`bl YY " " " JJt~~002BGGLL^4\ ])-(A(A) ) *   8K G % II ! ! !...2O*,,((55 4,,779277<< Tb;cd6  0 -3P "Q655X65s AU1AV1 V  V c$<V^8dQhRS[RR/#rr))rrs"rrr s11s1t1rc  VPPpTfR#\4'd\P !R4\ P!RR7;_uu_4p\P!TPPP4\PPT\44\T4RRR4TPP 'd\4'g\ P!RR7;_uu_4p\"P!TPPP4\PPT\44RRR4\X4R#R#R# \dR#i;i +'giL;i +'giLF;i)z-Save the gradient scaler state if one exists.Nsaving_scaler_stateTr)rscalerrrr rr%rrrr:rr0 SCALER_NAMErPrr9r)rUrrrs&& rrTrainer._save_scaler s( %%,,F >  ! # # MM/ 0((55((//::cd#O46 99 )?)A)A((55 4++22==?jZeAfg6  0*B     65 65s+F#A'F$AF7 F! F!$ F4 7 G cL<V^8dQhRS[RS[PR,RR/#)rrrNrf)rrr)rrs"rrr s2==C= TXHX=dh=rc  aVf VPp\PPS\4p\PPS\ 4p\PPS\ 4p\PPS\4p\PPS\4p\PPS\4p\PPS\4p \PPS4;'d\;QJd3V3Rl\P!S44F 'gK RM( RM$!V3Rl\P!S444;'gE\PP\PPS\ R244p \PPS4'd\P!S4U u.uFp \PP\PPSV 44'gKH\PP\PPSV \ 44'gK\PP\PPSV \ 44'gKV NK up M.p V 'd"VP 'g\#RS R24h\;QJd"RVVVV VV34F 'gK RM RM!RVVVV VV344'gV 'gV 'g\#R S 24h\$P'R S R 24\PPV4'dR\(P*!V4p V P,pVe,V\.8wd!\$P1R V R \. R24\PPV4'g/\PPV4'g V 'Ed\34'd!\4P6!S\RRR7R#VP 'dB\9VP:P<P>VP:VS3/\A4BR#\PPV4'd#\BPDPGVRR7pM#\I4\DPJ!VRRR7pVPMVR4p?VPOV4R#\QV4'Ed4\SVR4'Ed\SVR4'd\PPUS4'dVPVp\YV4^8d\$P1R4V^,pV 'dTV F:p\PPSV4pVP[VVVV8HR7K< VP]V4R#VP[SVRR7R#\$P1R\ R24R#\$P1R\^ R24R#\aVS\34R7p\34'gVPOV4R#R#uup i)z/Load model weights from a checkpoint directory.Nc3<"TFSp\PP\PPSV44'gKH\V9xKU R#5ir)r:risdirr0FSDP_MODEL_NAME)r folder_namers& rr0Trainer._load_from_checkpoint.. s?#EK77==.Dk!RS/;.#Es AA ATFz.binzCheckpoint found at z* is only supported when using PyTorch FSDPc3`"TF$p\PPV4xK& R#5ir)r:rr)rfs& rrr s+ Aq!!s,.z!Can't find a valid checkpoint at zLoading model from rNz9You are resuming training from a checkpoint trained with z- of Transformers but your current version is zJ. This is not recommended and could yield to errors or unwanted behaviors.rtagrload_optimizerrr map_location weights_onlyactive_adapters load_adapterzFMultiple active adapters detected will only consider the first adapter) is_trainablejThe intermediate checkpoints of PEFT may not be saved correctly, consider using a custom callback to save i in corresponding saving folders. Check some examples here: https://github.com/huggingface/peft/issues/968Could not load adapter model, make sure to have PEFT >= installedstrict)1rr:rr0rxrwrvr}r|r{rzrrlistdirrrrrrrrfrom_json_filetransformers_versionrrqrrorrrr=rr% safetensorsr load_filerloadload_state_dict_issue_warnings_after_loadr_rrrr r set_adapterr*rh)rUrr config_fileadapter_weights_fileadapter_safe_weights_file weights_fileweights_index_filesafe_weights_filesafe_weights_index_file is_fsdp_ckptradapter_subdirsrcheckpoint_versionr load_resultractive_adapter subdir_namepeft_ids&f& rrTrainer._load_from_checkpoint sl =JJEggll#9;G !ww||,BDXY$&GGLL1GIb$c!ww||$:LI WW\\*@BTUGGLL)?ARS"$'',,/EG^"_ww}}%;<   C#%::.D#ECCC#%::.D#E  ^ ^ ww~~bggll+AoEVVZC[\] *ww}}344$&::.D#E #EK77==.Dk!RSGGNN277<<0F Ui#jkkww~~bggll3I;Xq&rs  #E    4 4 434J3KKuvw w C !%&+(-  CCC !%&+(-    @AW@XYZ Z )*@)ACD 77>>+ & &%44[AF!'!>, ' '277>>:K+L+LP\P\&((**/\5af%%%$$**66$$*  +, 77>>"344!,!2!2!>"899&+&;&;O?+a/'op%4Q%7N&+:K&(ggll3I;&WG!..w S^bpSp.r,;)).9**+A>`d*eNNDDXCYZbb !YZjYkkuvw2%9OXoXqrK*,,// <-M s A\ *A\ 2A\ <\ c<V^8dQhRR/#rerg)rrs"rrrl shh$hrc \PRVPP RVPP R24\ P PVPP\4p\ P PVPP\4p\ P PVPP\4p\ P PVPP\4p\4'd VPM VPpVP'dG\!VPVPP\#VP4'*R7R #VP$'dV\'VP(PP*VP(VVPP3/\-4BpR #\ P P/V4'gq\ P P/V4'gL\ P P/V4'g'\ P P/V4'Ed+Rp\4'd5\0P2!VPP\RRR7R #\#V4'Ed(\5VR4'd\5VR 4'dVP6^,p\9VP64^8d\P;R 4\ P P/V4'g&\ P P/V4'd8VP=VPPV4^RI$H%p T !..4pM\P;R\ R24RpM\P;R\L R24RpM|\ P POV4'd#\PPRPUVRR7p M#\W4\RPX!VRRR7p VP[V R4p\4'gV'dVP]X4R #R #R #\ P P/\ P PVPP\^44'g\\ P P/\ P PVPP\`44'dO\cWPPP\4R7p\4'gVP]V4R #R #\P;RV R24R # \>dap TP@T,PB'd8R TP@T,PDPF R 2p \?T 4T hhR p ? ii;i)zFLoad the best model found during training based on the tracked metric.zLoading best model from z (score: z).rTFrrrzCDetected multiple active adapters, will only consider the first onez0When using prompt learning PEFT methods such as z, setting load_best_model_at_end=True can lead to errors, it is recommended to set this to False and to load the model manually from the checkpoint directory using PeftModel.from_pretrained(base_model, ) after training has finished.N)_IncompatibleKeysrrrrrrrrz#Could not locate the best model at zi, if you are running a distributed training on multiple nodes, you should activate `--save_on_each_node`.)2rrr=r#rr:rr0r}r{rwrvrrrrr!r_rrrrr%rrorrrr rqrr peft_configis_prompt_learning peft_typertorch.nn.modules.modulerr*rrrrrrrrrzr|rh) rUbest_model_pathbest_safe_model_pathbest_adapter_model_pathbest_safe_adapter_model_pathrrhas_been_loadedrrdmsgrrs & rr(Trainer._load_best_modell s .tzz/O/O.PPYZ^ZdZdZpZpYqqstu'',,tzz'G'GV!ww||DJJ,L,LN_`"$'',,tzz/O/OQe"f')ww||DJJ4T4TVo'p$&=&?&?""TZZ  $ $ $ %"" 00'5djj'A#A   ! ! !)  &&22   00  '( K GGNN? + +ww~~233ww~~566ww~~:;;"O&((**99$!#( "%((u&788WUN=[=[).)>)>q)Au4459"NN+pq77>>*ABBbggnnUqFrFr* % 2 24::3S3SUc d R*;B*CK"NN!LL`Kabj!j /4OVWgVhhrs+0ww~~&:;;%0%6%6%@%@AU^c%@%d 02%*ZZebf%g #("7"7 E"JK.00_33K@6E0 WW^^BGGLL)I)IKbc d dhjhohohvhv GGLL99;M Ni i 2zz77@W@YK+,,// <- NN5o5FGPP k$0 *#(#4#4^#D#W#W#W)[+0+<+<^+L+V+V+\+\*]^8)8%(+7s*;$D$) *s*&W** Y5AYYc2<V^8dQhRS[R,RR/#rr3Nrfr))rrs"rrr s#+_+_#*+_+_rc VfR#VPP^8d|VPPp\PP VRV R24p\PP V4'g\PRV R24R#M\\PP VR4p\PP V4'g\PR4R#\4;_uu_4\4\P!VRR 7pRRR4\P!XR ,4\PP!VR ,4\PP#VR ,4\%4'd\&P"!VR ,4VPP(\*P,8Hp\P.P14'd\3R\P.WE4\54'd\3R\P6WE4\94'd\3R\P:WE4\=4'd\3R\P>WE4\A4'd\3R\PBWE4R#R# +'giEL;i)z9Restore random number generator states from a checkpoint.Nrrz$Didn't find an RNG file for process zr, if you are resuming a training that wasn't launched in a distributed fashion, reproducibility is not guaranteed.rzDidn't find an RNG file, if you are resuming a training that was launched in a distributed fashion, reproducibility is not guaranteed.TrrrrrCUDANPUHPUMLUMUSA)"rrr r:rr0rrrrRrrrrsetstaterP set_state set_rng_staterr rrsrr(rrSrrrrrrrr)rUr3r rng_filecheckpoint_rng_stateis_distributeds&& rrTrainer._load_rng_state s    99  ! # II33Mww||J*]O40PQH77>>(++ :=/Jcc ,ww||J@H77>>(++ B ^^ $ &#(::hT#J  ,X67 09: ""#7#>? ! # #   1%8 900L4L4LL :: " " $ $ $VUZZ9M ^ ! # # $UEII7K \ ! # # $UEII7K \ ! # # $UEII7K \ " $ $ $VUZZ9M ^ %%^^s #K"" K3 c2<V^8dQhRS[R,RR/#rr))rrs"rrrs#f5f5d f5tf5rc  aSfR#VP'd\VP\4'g\P !RR7;_uu_4p\ 4VPP\P!\PPS\4RR74RRR4\X4R#\4'dA\ P !\PPS\"4R,4EM\PP%\PPS\"44;'g\PP%\PPS\&44;'g\PP)S4;'dc\*;QJd3V3Rl\P,!S44F 'gK RM( RM$!V3Rl\P,!S444pVP.'dS\ P !\PPSRVP0P2 R \" 244MTpV'Ed\PP%\PPS\44'Ed\54'EdVP.'d\ 4\P!\PPSR VP0P6 R VP0P2 R \" 24R RR 7pVR,pME\ 4\P!\PPS\"4R RR 7p\P !RR7;_uu_4p\ 4\P!\PPS\4R RR 7pRRR4\X4\8P:!W@P0P<4\8P:!XVP0P<4VP>PV4VPPV4R#\4'd#V3RlpVP@PCV4MVP0P2^8dVP0P<MR pVPD'dV\GVPHPJPLVPHVP>VPNS3/\Q4BM^\ 4VP>P\P!\PPS\"4VRR 74\P !RR7;_uu_4p\ 4VPP\P!\PPS\4RR74RRR4\X4R#R#R# +'giEL;i +'giEL};i +'giLI;i)z3If optimizer and scheduler states exist, load them.NTrr_*c3<"TFip\PP\PPSV44'gKH\P R4^,V9xKk R#5i)rNN)r:rrr0OPTIMIZER_NAME_BINsplit)rrr3s& rr8Trainer._load_optimizer_and_scheduler..sP+AK77==j+)NOH*005a8KG+As AA4 'A4Fz rank*-of-r|rrrrr+c<VP\P!\PP S\ 4RR74R#)TrN)rrorr:rr0r)modoptr3s&&r opt_load_hook.opt_load_hookKs,++CHHRWW\\*n5]gk,lmr))rr)rrr%rrrrrr:rr0rrPrglobrrr"rrrrSrrrr r send_cpu_data_to_devicer r+rregister_post_step_hookrrrr=rrr%)rUr3rcheckpoint_file_existsoptimizer_statelr_scheduler_stater(rs&f rr%Trainer._load_optimizer_and_schedulers     $ $ $d//1JKK,,D99_,.%%55 277<< N#KZ^_: $O4 '(( IIbggll:~>E Frww||JGH  77>>"'',,z;M"NO  GGMM*-+-::j+A+-::j+A $*** IIbggll:499;O;O:PPQR`Qa/bc d'  " !bggnnRWW\\*n5]&^&^%''...,.&+jj &$tyy/F/F.GtDIIL`L`Kaabcqbr(s&+%) 'O'6k&BO,.&+jj Z@ucg'O,,D99_,.). Z@ucg*&: $O4**?II>}M 8> ,, NN JJ&  34 1266!JJ " Z HWcrv ,,D99_,.%%55 277<< N#KZ^_: $O4C'_ !;:99b:99P:9s'AZ5AZ/$A[ Z, / [  [ c2<V^8dQhRS[R,RR/#rr))rrs"rrrks55sTz5d5rc  VfR#\PP\PPV\44pV'Edw\ 4'd\ P!RR7;_uu_4p\4\P!\PPV\4RRR7pRRR4\X4\P!XVPP4VP P"P%V4R#\ P!RR7;_uu_4p\4VP P"P%\P!\PPV\4RR74RRR4\X4R#R# +'giEL ;i +'giL3;i)z If scaler state exists, load it.NTrrrr)r:rrr0rrr%rrrrrPr r+rr rrr)rUr3r-r scaler_states&& rrTrainer._load_scalerks9   !# Z0U!V ! !&'',,D99_,.#(:: Z=E`d$L: $O4**<9I9IJ  ''77 E,,D99_,.$$++;; 277<< K#HW[\: $O4% ":99:9sAG  A(G G  G- c<V^8dQhRR/#rerg)rrs"rrrs%9%9d%9rc a VPP'gR#.p.pVPPVP.,pVP P P4EFwwo p\V\4'gV.p\;QJdV 3RlV4F 'gK RM RM!V 3RlV44'EdVUu.uF!qUPPS 8XgKVNK# pp\Wd4FwrWVPR/4pVPR/4p \V4!R /VBp V P4Fwr\!WV 4K \V\"4'dWnMVP%V 4VPP'\V 44K \(P+R4EKfVP%S 4EKz \-V4^8d)\(P/RR P1V4 R 24VFpVPP3V4K R#uupi) zLIf callback states exist and were passed in, restore their states if enabledNc3T<"TFqPPS8HxK R#5ir)rr)rcallbackstored_callbacks& rr/Trainer._load_callback_state..s"eRdh%%../ARds%(TFr attributeszPContinuing training from checkpoint, restoring any callbacks that were passed inzPCheckpoint included callbacks not included in current configuration. Ignoring. (r,rrg)r'restore_callback_states_from_checkpointr2rr<r=rrr)rrrrzipr rrr?rqremove_callbackrrr rqr0r3)rU not_found new_callbacksoriginal_callbacksr~r8 duplicates callback_datarr; new_callback attributerr9s& @rrTrainer._load_callback_statesyy@@@   !22<< ~M%)ZZ%B%B%H%H%J !OTdD))vseRdessseRdeee.@-?CUCUC^C^bqCqHH-?03:/D+H(,,VR8D!.!2!2<!DJ#'>#9D#9L,6,<,<,>(  ?-?!(N;;'3 %,,\:))99$|:LM0E no  1/&K0 y>A  NNbcgclclmvcwbxxyz &H  ! ! . .x 8&/s I 6I c$<V^8dQhRS[RR/#)rrrfN)r )rrs"rrrscdrc \VP4^8wdVPPew\VPP4^8dS\ VP4\ VPP48XdVPP 4M#\ PRVP R24\VP4^8wd&\ PRVP R24R#R#)zGLog warnings for missing or unexpected keys after loading a checkpoint.Nz8There were missing keys in the checkpoint model loaded: rNz;There were unexpected keys in the checkpoint model loaded: ) r  missing_keysr_keys_to_ignore_on_saver tie_weightsrrqunexpected_keys)rUrs&&rr"Trainer._issue_warnings_after_loads {'' (A - 22> ::;a? 001S9[9[5\\ &&(!YZeZrZrYsstuv {** +q 0 NNMkNiNiMjjkl  1rc8<V^8dQhRS[R,RS[RR/#)rrNr~rf)rr)rrs"rrrs,8[8[S4Z8[8[Y]8[rc VfVPPp\4'dE\VPVPVP VP VPV4EM\4'd\P!VRR7VPP4pVPP'dVPWR7\\P P#VR44P%4EMVP&'dR\)VP P*P,P.49dTVP P1VP4pVPP'dVPWR7EMdVP2'Ed%R\5\6P8!VPP:4P<P?449pVP@PBPER/4PER R4^8HpV'dA\GVP4'd&V'dVP@PIRR 7pM%VP P1VP@4pVPP'dVPWR7M-VPP'dVPV4VPPV'd4V'g*VPWR VPPXR 7R#R#R# \Jd\LPOR 4TPP'dTPT/R7\QTPPT\R\T.4TPP;T4Li;i)zm Will save the model, so you can reload it using `from_pretrained()`. Will only save from the main process. NTrrzuser_content.ptFULL_STATE_DICTrzero_optimizationstagerz| stage3_gather_16bit_weights_on_model_save=false. Saving the full checkpoint instead, use zero_to_fp32.py to recover weightsz Model save)commit_messagerevision)-rrrr+rrrrSrr:r;rrr9_saverrr0touchrrr=rrget_state_dictrrrrrrr%rrr r_$_zero3_consolidated_16bit_state_dictrrrqrQr}r{r7 hub_revision)rUrr~rrzero3_shardings&&& rrTrainer.save_models  --J ! # #  DIIt'7'79N9NPTPkPkmw % & & KK T 2++668Jyy$$$ : = j*;< = C C E  ! ! ! C(8(8(>(>(J(J(Z(Z$[[!--< ?s' B1M,=AM,M,,;P(APPcF<V^8dQhRS[R,RS[R,RR/#)rrNrrf)rr)rrs"rrrs0%L%Ld %Ltd{%LVZ%Lrc VeTMVPPp\P!VRR7\P RV 24\ 4'g\3M \\3p\VPV4'gVfVPP4p\VPPVPRR7V4'd9VPPVPRR7PWR7Mx\P R4\P P#V\P$P'V\(4R R /R 7MVPPWR7VP*eVP*PV4M|VP,eo\/VP,R 4'dSVP,P0e;\P R 4VP,P0PV4\ P2!VP\P$P'V\444R#)zHSave model weights, configuration, and processing class to `output_dir`.NTrzSaving model checkpoint to FrrPzETrainer.model is not a `PreTrainedModel`, only saving its state dict.rpt)metadata tokenizerzWSaving Trainer.data_collator.tokenizer by default as Trainer.processing_class is `None`)rrr:r;rrrr1rr)rrrr2save_pretrainedrr save_filerr0r{rrrrarTRAINING_ARGS_NAME)rUrrsupported_classess&&& rrV Trainer._saves$.#9Ztyy?S?S  J. 1*>?6G6I6I_.P_ajOk$**&788!!ZZ224 $**77 W\7]_pqq  --djjU-Sccd cd!!++ Z9J KW_aeVf, JJ & &z & I  ,  ! ! 1 1* =    ***K88"",,8 KKq r    ( ( 8 8 D  499bggll:7IJKrcN<V^8dQhRS[S[S[3,RS[R,RR/#)rrkrNrf)rrr)rrs"rrr"s4__S%Z(_edl_d_rc bVPPeVPPVR&VPPR8wdaVPPVR&VeDVPPVP , pVP \RW#R74/VCRVPP/CpVPPPV4VPPVPVPVPV4Vn R#)z Log `logs` on the various objects watching training. Subclass and override this method to inject custom behavior. Args: logs (`dict[str, float]`): The values to log. start_time (`Optional[float]`): The start of training. Nrrjrr) num_tokensr)r=rrrrrrrnr log_historyrqr2on_logr<)rUrkrcurrent_session_num_tokensr s&&& rr' Trainer.log"s ::   ' JJ,,DM 99 2 2d :,0JJ,L,LD( )%-1ZZ-M-MPTPsPs-s* M':ef9D9&$**"8"89 %%f-,,33DIItzz4<&> > JJ ! !-t/@/@.A$))JZJZ[__affh  !!"D  JJ ! !T%6%6 6 ! !D rc`<V^8dQhRS[S[S[PS[,3,RS[/#r)rrrrr r)rrs"rrrEs.c5<<#3E.E)F3rc \VPRR4;q!9d[\VPR4'd?^W,P4,VPP RR7,#^#)a For models that inherit from [`PreTrainedModel`], uses that method to compute the number of floating point operations for every backward + forward pass. If using another model, either implement such a method in the model or subclass and override this method. Args: inputs (`dict[str, torch.Tensor | Any]`): The inputs and targets of the model. Returns: `int`: The number of floating-point operations. rrnum_parametersT)exclude_embeddings)rrrrXrs)rUr main_inputs&& rrTrainer.floating_point_opsEsj"$**.?M MJ X]d JJ(^ ^ v)//11DJJ4M4Mae4M4ff frc2<V^8dQhRS[R,RR/#)rtokenNrfr))rrs"rrrZs%%#*%%rc VP4'gR#VPPf9\VPP4P 4P pMVPPpVeTMVPPp\4PW!VPPRR7pVPVnRVn R#)z5 Initializes a git repo in `self.args.hub_model_id`. NT)rxprivater) rrr6rrabsoluterL hub_tokenr create_repohub_private_reporepo_idpush_in_progress)rUrx repo_namerepo_urls&& rr8Trainer.init_hf_repoZs ))++  99 ! ! )TYY112;;=BBI ..I* 0C0C8''  HbHbmq'r$,, $rcr<V^8dQhRS[R,RS[R,RS[S[S[,,R,RS[R,RS[R,RS[S[S[,,R,RS[S[S[,,R,R S[S[S[,,R,R S[S[S[,,R,R R/ #) rlanguageNlicensetagsr:finetuned_fromtasks dataset_tagsr dataset_argsrf)rr)rrs"rrrlsHhHh*HhtHhDIo$ Hh $J Hh d HhT#Y%HhDIo,HhtCy4'HhDIo,Hh Hhrc  VP4'gR#\PPVPP R4p Rp \PP V 4'd\P!V 4PPR4p V R8Hp \P!V 4PPp Ve@V e<\V\4'dV.pV FpW9gK VPV4K \P !VVVVVVVVVV R7 pVP#4p\%V R4;_uu_4pVP'V4RRR4V 'dKVP(P+VP,4P/VPP 4R#R# +'giLd;i)aM Creates a draft of a model card using the information available to the `Trainer`. Args: language (`str`, *optional*): The language of the model (if applicable) license (`str`, *optional*): The license of the model. Will default to the license of the pretrained model used, if the original model given to the `Trainer` comes from a repo on the Hub. tags (`str` or `list[str]`, *optional*): Some tags to be included in the metadata of the model card. model_name (`str`, *optional*): The name of the model. finetuned_from (`str`, *optional*): The name of the model used to fine-tune this one (if applicable). Will default to the name of the repo of the original model given to the `Trainer` (if it comes from the Hub). tasks (`str` or `list[str]`, *optional*): One or several task identifiers, to be included in the metadata of the model card. dataset_tags (`str` or `list[str]`, *optional*): One or several dataset tags, to be included in the metadata of the model card. dataset (`str` or `list[str]`, *optional*): One or several dataset identifiers, to be included in the metadata of the model card. dataset_args (`str` or `list[str]`, *optional*): One or several dataset arguments, to be included in the metadata of the model card. Nz README.mdF library_namepeft) rrrr:rrrrrw)rr:rr0rrrr rr~r rr)rrqr0 from_trainer to_model_cardopenwriterr2rcreate_or_update_model_card)rUrrrr:rrrrrmodel_card_filepathis_peft_libraryr existing_tagsrtraining_summary model_cardrs&&&&&&&&&& rcreate_model_cardTrainer.create_model_cardlsuJ))++  ggll499+?+?M 77>>- . .$>>*=>CCGGWL*f4O&NN+>?DDIIMM$=dC(( 6D(C C()+77 !)%%  &335 %s + +q GGJ ,     ) )$** 5 Q QRVR[R[RfRf g , +s G G c b<V^8dQhRS[R,RS[RS[R,RS[R,RS[/#)rrTNblockingrxrUrf)rrr )rrs"rrrsPP P d P P Tz P * P  P rc VPPVPVPVP4VP RR4pVfVPP 'doVPPf+\VPP4PpM,VPPPR4R ,pVeTMVPPpVPfVPVR7VPRR7VP4'gR#\!VP"RR4e{RV9d.VR&\%VR,\&4'dVR,.VR&VP"P(F*pWuR,9gKVR,P+V4K, VP,!R RV/VBVfVPP.pVP14\34P5VPVPPVVV'*R \6 R 2.VR 7#)u Upload `self.model` and `self.processing_class` to the 🤗 model hub on the repo `self.args.hub_model_id`. Parameters: commit_message (`str`, *optional*, defaults to `"End of training"`): Message to commit while pushing. blocking (`bool`, *optional*, defaults to `True`): Whether the function should return only when the `git push` has finished. token (`str`, *optional*, defaults to `None`): Token with write permission to overwrite Trainer's original args. revision (`str`, *optional*): The git revision to commit from. Defaults to the head of the "main" branch. kwargs (`dict[str, Any]`, *optional*): Additional keyword arguments passed along to [`~Trainer.create_model_card`]. Returns: The URL of the repository where the model was pushed if `blocking=False`, or a `Future` object tracking the progress of the commit if `blocking=True`. r:N/)rxTr} model_tagsrr -*r folder_pathrTrx run_as_futureignore_patternsrUrrg)r2 on_push_beginrr=r<rr9r6rrrLr#r|r8rrrrr)rrrqrrZr/r upload_folderrT)rUrTrrxrUrr: model_tags&&&&&, rr7Trainer.push_to_hubs6 ++DIItzz4<<PZZ d3  $))"7"7"7yy%%-!$))"6"67<< !YY3399#>rB * 0C0C    $   E  * t,))++  4::|T 2 >V#!#v&.#.."(.!1v!ZZ22 6N26N)))43 ?*??  yy--H !!#x%%%% ,,)&,!&;%Y>Y%Z Z%)YY%;%;{?U?U%U![_`q[r[w[w %h44))-)->ii))"//5O   _ -  (D,A,A,I,I,K,K$29$=D !  ! ! & & - -i 8e&%%s .%T  T1 c<V^8dQhRR/#rerg)rrs"rrrMs44d4rc \VR4'gR#VPeTVPP4'g2\P R4VPP 4R#R#R#)z5Wait for any in-progress push to the Hub to complete.rNz\Waiting for the current checkpoint push to be finished, this might take a couple of minutes.)rrrrrwait_until_doners&rr/Trainer._finish_current_pushMs[t/00   ,T5J5J5R5R5T5T KKv w  ! ! 1 1 36U ,rc<<V^8dQhRS[R.S[S[S[3,3,R,RS[S[S[S[3,.S[3,R,RS[RS[S[S[,,RS[S[,R,RS[R.S[3,R,R S[S[S[,,/#) rhp_spacez optuna.TrialNcompute_objectiven_trials directionrr?rf)rrrrrrrXrU)rrs"rrrWsOON+T#u*-==>EO$T#u*%5$6$=>EO O c? O &- O>*C/047O 4= Orc LVf \4p\V4p\V,!4pVP4WPnVP f \ R4hVf VPMTVnW`n Vf\MTVn VP!WV3/VBp RVnV #)a Launch a hyperparameter search using `optuna` or `Ray Tune`. The optimized quantity is determined by `compute_objective`, which defaults to a function returning the evaluation loss when no metric is provided, the sum of all metrics otherwise. To use this method, you need to have provided a `model_init` when initializing your [`Trainer`]: we need to reinitialize the model at each new run. This is incompatible with the `optimizers` argument, so you need to subclass [`Trainer`] and override the method [`~Trainer.create_optimizer_and_scheduler`] for custom optimizer/scheduler. Args: hp_space (`Callable[["optuna.Trial"], dict[str, float]]`, *optional*): A function that defines the hyperparameter search space. Will default to [`~trainer_utils.default_hp_space_optuna`] or [`~trainer_utils.default_hp_space_ray`] depending on your backend. compute_objective (`Callable[[dict[str, float]], float]`, *optional*): A function computing the objective to minimize or maximize from the metrics returned by the `evaluate` method. Will default to [`~trainer_utils.default_compute_objective`]. n_trials (`int`, *optional*, defaults to 100): The number of trial runs to test. direction (`str` or `list[str]`, *optional*, defaults to `"minimize"`): If it's single objective optimization, direction is `str`, can be `"minimize"` or `"maximize"`, you should pick `"minimize"` when optimizing the validation loss, `"maximize"` when optimizing one or several metrics. If it's multi objectives optimization, direction is `list[str]`, can be List of `"minimize"` and `"maximize"`, you should pick `"minimize"` when optimizing the validation loss, `"maximize"` when optimizing one or several metrics. backend (`str` or [`~training_utils.HPSearchBackend`], *optional*): The backend to use for hyperparameter search. Will default to optuna or Ray Tune, depending on which one is installed. If all are installed, will default to optuna. hp_name (`Callable[["optuna.Trial"], str]]`, *optional*): A function that defines the trial/run name. Will default to None. kwargs (`dict[str, Any]`, *optional*): Additional keyword arguments for each backend: - `optuna`: parameters from [optuna.study.create_study](https://optuna.readthedocs.io/en/stable/reference/generated/optuna.study.create_study.html) and also the parameters `timeout`, `n_jobs` and `gc_after_trial` from [optuna.study.Study.optimize](https://optuna.readthedocs.io/en/stable/reference/generated/optuna.study.Study.html#optuna.study.Study.optimize) - `ray`: parameters from [tune.run](https://docs.ray.io/en/latest/tune/api_docs/execution.html#tune-run). If `resources_per_trial` is not set in the `kwargs`, it defaults to 1 CPU core and 1 GPU (if available). If `progress_reporter` is not set in the `kwargs`, [ray.tune.CLIReporter](https://docs.ray.io/en/latest/tune/api/doc/ray.tune.CLIReporter.html) is used. Returns: [`trainer_utils.BestRun` or `list[trainer_utils.BestRun]`]: All the information about the best run or best runs for multi-objective optimization. Experiment summary can be found in `run_summary` attribute for Ray backend. NzXTo use hyperparameter search, you need to pass your model through a model_init function.) rrXrensure_availabler@rrdefault_hp_spacerr?rbrrr) rUrrrrrr?r backend_objbest_runs &&&&&&&, rhyperparameter_searchTrainer.hyperparameter_searchWsz ?/1G!'*8AC $$&!( ?? "j 9A8H 44h  >O>W!:]n??49GG!%rc8<V^8dQhRRRS[P/#rgrz)rrs"rrrs#  %K WYW`W` rc \VP4pV^8XdVP4pM$V^8XdVPV4pM \R4hVf \R4hV#)zdInvoke `model_init` to get a fresh model instance, optionally conditioned on a hyperparameter trial.z'model_init should have 0 or 1 argument.z"model_init should not return None.)rirr)rUrmodel_init_argcountrs&& rrTrainer.call_model_inits]1$//B ! #OO%E A %OOE*EHI I =CD D rc"<V^8dQhRRRR/#)rrrrfNrg)rrs"rrrs4242&L42QU42rc WnVPeVfR#VP\P8XdVP V4pMUVP\P 8XdTpVP RR4M!VP\P8XdTpXP4F~wr4\VPV4'g\PRV R24K<\VPVR4pVe\V4!V4p\VPW44K VP\P8Xd#\P!RVP" 24VP\P8Xd\P!RV 24VP$'dVPP&f \)R4hTP*P-4^RIHp^R IHpV!VPP&4VPnVPP6P9VP4V!VPP6R 7VPn\=4P?4VPP@Vn!VPE4R#) zRSet up training arguments and accelerator state for a hyperparameter search trial.NrqzTrying to set zY in the hyperparameter search but there is no corresponding field in `TrainingArguments`.zTrial: zW&B Sweep parameters: z7For sweeps with deepspeed, `args.deepspeed` must be set)DeepSpeedPlugin)HfTrainerDeepSpeedConfig)r)#_trialr@rXrirrkrrprrrrrqrrrrrkrrrrrrr#transformers.integrations.deepspeedrrtrainer_config_processrr _reset_staterDrEr)rUrrkr;rold_attrrrs&& rrTrainer._hp_search_setups  ! ! )U]   ! !_%;%; ;]]5)F  # #':': :F JJw %  # #'<'< <F ,,.JC499c**$SE*,,tyy#t4H#Xu- DIIs *)  ! !_%;%; ; KK'%,,0 1  ! !_%:%: : KK08 9  $ $ $yy""* !Z[[    ( ( * 9 T,DTYYEXEX,YDII ) II ) ) @ @ K)8diiFcFc)dDII &   + + -"&!;!; //1rcD<V^8dQhRRRS[RS[S[S[3,RR/#)rrrrr1rfN)rrrr)rrs"rrrs?@@;@CF@QUVY[`V`Qa@ @rc VPeVfR#VP4pVPV4VnVP\P 8Xd^RIp\VR4'dVPP4'gVPVPV4VP4'dMVPPVPVPVP 4VP"!4hR#R#R#VP\P$8Xd^RIp\(P*!4;_uu_4pRpVP P,'d8VP/VR7VP0P2P5V4pVPVR&VP0PW7R7RRR4R#R# +'giR#;i)zHReport intermediate metrics to the active hyperparameter search backend.Nstudy)checkpoint_dir objective)r3)r@rrrrXrioptunarr_is_multi_objectivereport should_pruner2r.rr=r< TrialPrunedrkrltempfileTemporaryDirectoryr9_tune_save_checkpointrm Checkpointfrom_directory)rUrrr1rrutemp_checkpoint_dirr3s&&&& rraTrainer._report_to_hp_searchsr  ! ! )U] ,,.//8  ! !_%;%; ; ug&&u{{/N/N/P/P T^^T2%%''))66tyy$**dll[ ,,..(0Q&  # #':': : ,,..2E! <<+++..>Q.R!$!4!4!C!CDW!XJ'+~~ $? /.;/..s BG!! G2 c$<V^8dQhRS[RR/#)rrrfNr))rrs"rrrs a aC aD arc \PPV\ RVPP 24pVP VRR7VPP'EdVPP 4VPPR&VPP\PPV\44\P!VPP!4\PPV\"44\P!VP$P!4\PPV\&44R#R#)z@Save a checkpoint during a Ray Tune hyperparameter search trial.r|Tr}r?N)r:rr0rTr=rrrr9r<rrrrrr+rrrr)rUrrs&& rrTrainer._tune_save_checkpointsWW\\.5J4K1TZZMcMcLd2ef   48 99 >Bll>P>P>RDJJ ) )*: ; JJ # #BGGLL=O$P Q JJt~~002BGGLL^4\ ] JJt((335rww||JP^7_ ` !rcD<V^8dQhRS[S[,S[,RR/#rr8rfNrr>)rrs"rrrs& 5 5T/%:_%L 5QU 5rc <VPPV4R#)a7 Add a callback to the current list of [`~transformers.TrainerCallback`]. Args: callback (`type` or [`~transformers.TrainerCallback]`): A [`~transformers.TrainerCallback`] class or an instance of a [`~transformers.TrainerCallback`]. In the first case, will instantiate a member of that class. N)r2r3rUr8s&&rr3Trainer.add_callbacks **84rcT<V^8dQhRS[S[,S[,RS[R,/#rr)rrs"rrr s,<rs&&rr>Trainer.remove_callback0s --h7rc <V^8dQhRS[/#rr)rrs"rrr=s22t2rc 4VPP^8H#)z Whether or not this process is the local (e.g., on one machine if training in a distributed fashion on several machines) main process. )rlocal_process_indexrs&rrTrainer.is_local_process_zero=s yy,,11rc <V^8dQhRS[/#rr)rrs"rrrDs , ,t ,rc \4'd\P!4^8H#VPP^8H#)z Whether or not this process is the global main process (when training in a distributed fashion on several machines, this is only going to be `True` for one process). )rrorrr rs&rrTrainer.is_world_process_zeroDs2 # $ $88:? "yy&&!++rcR<V^8dQhRS[PRS[PRR/#)rrr rfN)rrrr )rrs"rrrOs*   299  ell  t  rc \VRR4e\PR4R#VPV4pVPP \ P8Xd'\VR4'dVP4R#R#R#)zJMove the model to the specified device, re-tying weights on XLA if needed.rNzZThe model is already on multiple devices. Skipping the move to device specified in `args`.rK) rrrqr=rrrsTPUrrK)rUrr s&&&rrTrainer._move_model_to_deviceOsl 5/4 0 < NNl    99 " "l&6&6 675-;X;X    rr"r(rrrrrrrr*rr+rrrrrr=rNN) NNNNNNNNNNrNNr)NFN)NNN)NNNNN)FN)T)TN)NNr)Nr)F)NF) NNNNNNNNN)zEnd of trainingTNN)NNminimizeNN)er __module__ __qualname____firstlineno____doc__trainer_pt_utilsrrrrrrrrbr,rrrrrrrrrr&rrrDrBrC staticmethodrQrNrrrrrrrrr?r>rYr rrtrr:r9rrrrrrrrrrrrrAr!rr)rirhrrrrr(rrrrrrrVr'r&rr8rr7rr/rrrrarr3rr>rrr__static_attributes____classdictcell__)rs@rrrs[|~7~7@PPd99vjxjx\  (% % N  $ K K00d00>:\\ !;!;F''$ E EJJX!!<<   >>`'Dc J/HbE!E!NW;W;r  ((0^0^f11"<"<"|$  \.\.|D D LYY$    D@*_B] bbHMuMu^;t;tzg&g&R@&CCJ$"$"L1i1if@1@1D11.==BhhT+_+_Zf5f5P556%9%9N$8[8[t%L%LR__0 " "*%%$HhHhTP P dC9C9J44OOb  4242l@@6 a a 5 5<< 8 822 , ,   rr)rr(rrrr*rrrr:rr,rrrr%collections.abcrrrrpathlibrtypingrr integrationsr rrPsafetensors.torchrrtorch.distributed distributedrhuggingface_hubr r packagingr rtorch.utils.datarrrrrr+rconfiguration_utilsrdata.data_collatorrrr debug_utilsrr!feature_extraction_sequence_utilsrfeature_extraction_utilsrrrrimage_processing_utilsrintegrations.deepspeedr r!r"r#r$integrations.fsdpr%r&integrations.ligerr'integrations.neftuner(r)integrations.peftr*integrations.tpur+r,r-loss.loss_utilsr.r/ modelcardr0modeling_utilsr1r2models.auto.modeling_autor3r4 optimizationr5r6processing_utilsr7tokenization_utils_baser8trainer_callbackr9r:r;r<r=r>r?r@trainer_optimizerrArBrCrDrrErFrGrHrIrJrKrLrMrNrOrPrQrRrS trainer_utilsrTrUrVrWrXrYrZr[r\r]r^r_r`rarbrcrdrerfrgrhrirjrkrlrmrnrorprq training_argsrrrsrtr}rurvrwrxryrzr{r|r}r~rrrrrrrrrrrrrrrrrutils.import_utilsrutils.quantization_configrr-r5utils.notebookrrtorch_xla.core.xla_modelcore xla_modelr torch_xla.debug.metricsrZr1rtorch_xla.runtimeruntimerK torch_xla XLA_VERSIONr1rJtorch_xla.distributed.spmdspmdrM!smdistributed.modelparallel.torch modelparallelrorrrrrrrraccelerate.staterrrrrrrrrrraccelerate.utils.memoryrrr get_loggerrrrdrrrr"rrrrgrrrMs~   77%   1cc1\\<G<`6M2F/[[:&92,<    "@KJ:)9)),8 8))))"4$]];77==I_;``//"33[[:1   ;>   H %)) $&  XA XA  XA r