+ QV-jRt^RIt^RIt^RIt^RIt^RIt^RIt^RIt^RIt^RI t ^RI t ^RI H t H t ^RIHt^RIHtHt^RIHt^RIHt^RIHt^RIt^RIt^RIHt^RIHt^R IH t ^R I!H"t"H#t#H$t$H%t%^R I&H't'^R I(H)t)^R I*H+t+^RI,H-t-H.t.H/t/H0t0Ht]0!4'd#]Pb!]!] Pd44]/!4'd^RI3H4t5].!4'd^RI6H7t7]Pp!]94t:Rt;RRltRTRlt?Rt@RtARtBRtCRURRlltDRURltERtFR]P!R43RR lltHR!tI]R"R#l4tJ!R$R%]'4tK!R&R'4tLR(R)ltMRTR*ltNRTR+ltOR,tP]!R-R.44tQRVR/ltR!R0R1]%4tS!R2R3]'4tT!R4R5]%4tU!R6R7]#4tVR8tWR9R:ltXR;tYRWR<ltZR=t[R>R?lt\R@RAlt]RURBRCllt^RXRDlt_RURElt`RFtaRGtb]-!4'd@^RIcHduHte]eP!4RYRHl4tg]eP!4RI4thRJtiRKtj]!RLRM44tk!RNRO]PP4tn!RPRQ]74toRRtpRStqR#)Zz( Torch utilities for the Trainer class. N)IteratorMapping)contextmanager) dataclassfield)chain) StreamHandler)Any)version)nn)DatasetIterableDataset RandomSamplerSampler)DistributedSampler)is_deepspeed_zero3_enabled) BatchEncoding)is_sagemaker_mp_enabledis_torch_availableis_torch_xla_availableis_training_run_on_sagemakerlogging) LRSchedulerc\VR4'd$VPe\VP4#\VR4'd VP#R#) batch_samplerNsampler)hasattrrget_dataloader_samplerr) dataloaders&n/Volumes/fast/ai/experiments/ui-tars-smoke/.venv/lib/python3.14/site-packages/transformers/trainer_pt_utils.pyrrCsIz?++ 0H0H0T%j&>&>?? Y ' '!!! (cbV^8dQhR\P\P,/#)tensor_or_array)torchTensornpndarray)formats"r __annotate__r)Js  rzz 9r c\V\P4'd\P!V4pV#\P!V4pV#N) isinstancer$r% atleast_1dr&)r#s&rr-r-JsA/5<<00**?; --8 r c\V4p\V4p\VP4^8Xg*VP^,VP^,8Xd\P!W3^R7#VP^,VP^,,\ VP^,VP^,43VPR,,pVP W24pWRVP^,1RVP^,13&WVP^,R1RVP^,13&V#)z`Concatenates `tensor1` and `tensor2` on first axis, applying padding on the second if necessary.dimr"NNN)r-lenshaper$catmaxnew_full)tensor1tensor2 padding_index new_shaperesults&&& rtorch_pad_and_concatenater<Rs !G!G 7==Q'--"2gmmA6F"Fyy'+33q!GMM!$44c'--:JGMMZ[L\6]^ahananoqarrI  i 7F5< W]]1  1q!11 125<7==  1q!11 12 Mr c\V4p\V4p\VP4^8Xg*VP^,VP^,8Xd\P!W3^R7#VP^,VP^,,\ VP^,VP^,43VPR,,p\P !WVR7pWRVP^,1RVP^,13&WVP^,R1RVP^,13&V#)z^Concatenates `array1` and `array2` on first axis, applying padding on the second if necessary.)axisr1r3N)r-r2r3r& concatenater5 full_like)array1array2r9r:r;s&&& rnumpy_pad_and_concatenaterDds  F  F 6<<AaFLLO!C~~v.Q77a6<<?2C QVW4YZ]c]i]ijl]mmI\\&y AF39 V\\!_ / Q/ /0396<<? / Q/ /0 Mr c a\V\P4'd!\V\P4'g9\V4\V4Jg!QR\V4 R\V4 R24h\V\\ 34'd%\V4!V3Rl\ W444#\V\P4'd\WSR7#\V\4'dG\V4!VP4UUu/uFwr4V\WAV,SR7bK upp4#\V\P4'd\WSR7#\R\V4 24huuppi)z Concat the `new_tensors` to `tensors` on the first dim and pad them on the second if needed. Works for tensors or nested list/tuples/dict of tensors. zEExpected `tensors` and `new_tensors` to have the same type but found z and .c3B<"TFwr\WSR7xK R#5i)r9N) nested_concat).0tnr9s& r nested_concat..s!tZsRVRS]1}MMZssrHz(Unsupported type for concatenation: got )r,r$r%typelisttuplezipr<ritemsrIr&r'rD TypeError)tensors new_tensorsr9krKs&&f rrIrIvsA w - -*[%,,2W2WG}[ 11 STXY`TaSbbghlmxhygzz{ | 1'D%=))G}tZ]^eZsttt GU\\ * *(][[ GW % %G}ZaZgZgZi jZiRVRSQ aQ}M MZi j   GRZZ ( ((][[B4=/RSS ks F c\V\\34'dVFp\V4pVfKVu# R#\V\4'd-VP 4Fp\V4pVfKVu# R#\V\ P\P34'd0\VP4^8dVP^,#R#R#)zN Find the first dimension of a tensor in a nested list/tuple/dict of tensors. N) r,rPrQfind_batch_sizervaluesr$r%r&r'r2r3)rUrKr;values& rrYrYs'D%=))A$Q'F!  GW % %^^%E$U+F! & GellBJJ7 8 8#&w}}#5#:w}}QDD 9r c \V\\34'd\V4!RV44#\V\4'd>\V4!VP 4UUu/uFwrV\ V4bK upp4#VP4pVP\P8Xd VP\P4pVP4#uuppi)zENumpify `tensors` (even if it's a nested list/tuple/dict of tensors).c38"TFp\V4xK R#5ir+)nested_numpifyrJrKs& rrM!nested_numpify..s@1^A..)r,rPrQrOrrSr^cpudtyper$bfloat16tofloat32numpyrUrWrKs& rr^r^s'D%=))G}@@@@'7##G}w}}Otqa!22OPP Aww%..  DD  779Ps)C& c \V\\34'd\V4!RV44#\V\4'd>\V4!VP 4UUu/uFwrV\ V4bK upp4#\V\P4'dVP4#T#uuppi)zDDetach `tensors` (even if it's a nested list/tuple/dict of tensors).c38"TFp\V4xK R#5ir+) nested_detachr_s& rrM nested_detach..s?w!]1--wra) r,rPrQrOrrSrkr$r%detachrhs& rrkrks'D%=))G}?w??? GW % %G}gmmoNodaaq!11oNOO)'5<<@@7>> MgMOs)B: ca\4'd^RIHuHp\ V\ \ 34'd%\V4!V3Rl\V444#\ V\4'dR\V4!\VP44UUUu/uFwpwrEV\VS RV 24bK uppp4#\V4pVPSV\P4#\!R4huupppi)Nc3J<"TFwr\VS RV 24xK R#5i)_N)nested_xla_mesh_reduce)rJirKnames& rrM)nested_xla_mesh_reduce..s* iVhda!7dV1QC=!I!IVhs #rqz;Torch xla must be installed to use `nested_xla_mesh_reduce`)rtorch_xla.core.xla_modelcore xla_modelr,rPrQrO enumeraterrSrrr- mesh_reducer$r4 ImportError)rUrtxmrsrWrKs&f rrrrrs-- ge} - -= iV_`gVh ii i gw ' '=NWX_XeXeXgNhiNhFQ*1asm<.sZSYa 216H I ISYr/N(Not currently using distributed training)r,rQrPrOrrSrr- contiguousrangedistget_world_sizeclone all_gatherr$r4AssertionError)r~rrWrKrqoutput_tensorsconcats&f rrrsI fudm , ,<ZSYZZ Z fg & &q1  )//0F !jO IGHHIs9D%D%4D%.D D%9D%D :D% D%%D<c^RIHpVfR#\4'dVfRp\W4pV#\ 4'd\ V4pV#WP 8Xd \V4pV#)zV Gather value of `tensors` (tensor or list/tuple of nested tensors) across processes. ) ParallelModeN nested_gather) training_argsrrrrr smp_gather DISTRIBUTEDr)rU parallel_modertrs&&& rrrsl, <"D(7 N ! " "W% N 22 2$W- Nr cVfR#VP4^8XdR#VP4R9dVPR,pV^8:dR#VP4^8Xd VR,pM V^,p\P!V^R7p\P!VR8*4;'g\P!VR84p\ V\P 4'dVP4#T#R#) aj Check if an attention mask is causal (compatible with causal attention). Context parallelism only supports causal attention patterns. This function checks if the provided attention mask is compatible. Args: attention_mask (`torch.Tensor`): The attention mask to check. Returns: `bool`: True if the mask is causal or compatible with causal attention. T)diagonalgư>F))rorog)r0r3r$triuallr,r%item)attention_maskseq_lenmaskupper_triangular is_causals& ris_attention_mask_causalrsq      '!&&r* a<    1 $!$'D!!$D!::dQ7II.$67]]599EUX\E\;] #-i#F#Fy~~UIU r cudacV^8dQhR\\\,,R\R,R\PR,R\P /#)r"scalarsrNdevicer)rPrfloatr$rr%)r(s"rr)r)sPII #+ Id I LL4 I \\ Ir cV\P!WR7p\\P!44Uu.uFqCP 4NK pp\P !WS4\P!V^R7pVeVRVpV#uupi \d \R4hi;i))rr/Nr) r$r~rrrrrr4r)rrrtensorized_scalarrqrrs&&& rdistributed_broadcast_scalarsrs I!LL@=B4CVCVCX=YZ=Y113=YZ :>q1  )//0F [ IGHHIs7BB :B BB(c\V4^8dMVFDpVP\JgK\P!VP VP4KF R#R#)N)r2category UserWarningwarningswarnmessage)caught_warningsws& rreissue_pt_warningsr/s@ ?a Azz, aii4! r c$V^8dQhR\/#)r" local_rankr)r(s"rr)r)8s  S r c#"VR9d\P!4RxV^8Xd\P!4R#R#5i)z Decorator to make all processes in distributed training wait for each local_master to do something. Args: local_rank (`int`): The rank of the local process. N)rro)rbarrier)rs&rtorch_distributed_zero_firstr7s0   Q sAAcBaa]tRtRtoRtV3RltV3RltRtVtV;t #)DistributedSamplerWithLoopiFa Like a torch.utils.data.distributed.DistributedSampler` but loops at the end back to the beginning of the shuffled samples to make each process have a round multiple of batch_size samples. Args: dataset (`torch.utils.data.Dataset`): Dataset used for sampling. batch_size (`int`): The batch size used with this sampler kwargs (`dict[str, Any]`, *optional*): All other keyword arguments passed to `DistributedSampler`. c 6<\SV`!V3/VBW nR#r+)super__init__ batch_size)selfdatasetrkwargs __class__s&&&,rr#DistributedSamplerWithLoop.__init__Ts +F+$r c|<\\SV` 44p\V4VP,^8Xd^M,VP\V4VP,, pVP \VP 4VP,8d^M^pWW3V,, p\V4#)ro) rPr__iter__r2rrankr num_replicasiter)rindices remainderstart_remainderrs& rr#DistributedSamplerWithLoop.__iter__Xsuw')*W71.sLVOA{;;VrrNNr?)r,rPrQrOr&rAr3)rrr9s&f&rrrsL&4-((F|LVLLL <<k5UFLLQSDT5U VVr c\P!WVP^,V3VPR,,R7pWRRVP^,13&V#)zmExpand the `arrays` so that the second dimension grows to `new_seq_length`. Uses `padding_index` for padding.r1r?NNNN)r&rAr3)rnew_seq_lengthr9r;s&&& r expand_liker sQ \\& Q7X[a[g[ghj[k7k lF#)1 Q  Mr c 0a\V\\34'd\V4!V3RlV44#\V\4'd?\V4!VP 4UUu/uFwr#V\ VS4bK upp4#VRS#uuppi)zQTruncate `tensors` at `limit` (even if it's a nested list/tuple/dict of tensors).c3<<"TFp\VS4xK R#5ir+)nested_truncate)rJrKlimits& rrM"nested_truncate..sH1_Q66rN)r,rPrQrOrrSr )rUrrWrKs&f rr r sz'D%=))G}HHHH'7##G}w}}WtqaE!::WXX 6E?Xs-B cBa]tRtRtoRtRtRtR RltV3RltRt Vt R#) LabelSmootheria$ Adds label-smoothing on a pre-computed output from a Transformers model. Args: epsilon (`float`, *optional*, defaults to 0.1): The label smoothing factor. ignore_index (`int`, *optional*, defaults to -100): The index in the labels to ignore when computing the loss. g?c\V\4'd VR,MV^,pV'd4VRRR 1R3,P4pVR ,P4p\PP VR R7)pVP 4VP 4^, 8XdVPR 4pVPVP4p\P!V^R7pVPR VR7pVPR R\PR7pVPVR 4VPVR 4VP!4VP#4P4, p VP4V , pVP4WP$R ,,, p^VP&, V,VP&V,,#) logits.Nrr/)min)r0indexT)r0keepdimrcgr).r)r,dictrr functional log_softmaxr0 unsqueezeeq ignore_indexr$clampgathersumrf masked_fill_numellongr3epsilon) r model_outputlabels shift_labelsr log_probs padding_masknll_loss smoothed_lossnum_active_elementss &&&& r__call__LabelSmoother.__call__s+5lD+I+Ih'|\] C"aK(335FG_//1F]]..v2.>> ::<9==?Q. .%%b)Fyy!2!23 V+##&#9! "d%-- P lC0""<5+002\5F5F5H5L5L5NN<<>$77%))+/B__UWEX/XY DLL H,t||m/KKKr c2<V^8dQh/S[;R&S[;R&#)r"r#r)rr)r(rs"rr)LabelSmoother.__annotate__sr rNrF) rrrrrr#rr,__annotate_func__rrrs@rrrs#GLLr rc aVf-\\S4V^,,^24pV^8Xd^p\P!\S4VR7pW!,p\ ^\S4V4Uu.uFqdWfV,P 4NK ppVUu.uFp\ VV3RlRR7NK ppVUu.uFpSV^,,NK p p\P!\P!V 44P4p Wz,^,V^,^,uV^,^&Wz,^&VUUu.uF qFqfNK K upp#uupiuupiuupiuuppi)a Return a list of indices so that each slice of `batch_size` consecutive indices correspond to elements of similar lengths. To do this, the indices are: - randomly permuted - grouped in mega-batches of size `mega_batch_mult * batch_size` - sorted by length in each mega-batch The result is the concatenation of all mega-batches, with the batch of `batch_size` containing the element of maximum length placed first, so that an OOM happens sooner rather than later.  generatorc<SV,#r+r)rslengthss&r,get_length_grouped_indices..s 71:r T)keyreverse) rr2r$randpermrtolistsortedargmaxr~r) r6rmega_batch_multr4rmegabatch_sizers megabatches megabatchmegabatch_maximumsmax_idxs f&&& rget_length_grouped_indicesrEsMc'lzA~>C a OnnS\Y?G$1NEJ1cRYl\jEklEk1>1299;EkKl^ij^iQZ6))=tL^iKjBMMI')A,//Mll5<<(:;<AACG1<1Ea1H+VW.YZJ[.KN1{+A.* >{)IqAIA{ >>mjN ?s3!EE!;E&E+cFa]tRtRtoRtR V3RlRlltRtRtRtVt R#) LengthGroupedSampleriz Sampler that samples indices in a way that groups together features of the dataset of roughly the same length while keeping a bit of randomness. Nc l<V^8dQhRS[RS[R,RS[S[,R,RS[R,/#)r"rrNr6model_input_name)rr rPstr)r(rs"rr)!LengthGroupedSampler.__annotate__sD###4#cT! # * #r cVfVf \R4hWnVfiVeTMRp\V^,\\34'dWB^,9d\RV R24hVUu.uFp\ Wd,4NK ppME\V\ P4'd&\PR4VP4pW0n WPn R#uupi)N,One of dataset and lengths must be provided. input_idsXCan only automatically infer lengths for datasets whose items are dictionaries with an '' key.zcIf lengths is a torch.Tensor, LengthGroupedSampler will be slow. Converting lengths to list[int]...) ValueErrorrr,rrr2r$r%loggerinfor<r6r4)rrrr6rIr4features&&&&&& rrLengthGroupedSampler.__init__s ?wKL L$ ?3C3O/U` gaj4*?@@DTde\fDf ()1FMMW's745WGMG  . . KKu nn&G "Ns)Cc,\VP4#r+)r2r6rs&r__len__LengthGroupedSampler.__len__%s4<<  r cp\VPVPVPR7p\ V4#)r3)rEr6rr4r)rrs& rrLengthGroupedSampler.__iter__(s),T\\4??VZVdVdeG}r )rr4r6)NNNN) rrrrrrrWrrrrs@rrGrGs# ##:!r rGcLa]tRtRtoRtR V3RlRlltV3RlRltRtVtR#) DistributedLengthGroupedSampleri-z Distributed Sampler that samples indices in a way that groups together features of the dataset of roughly the same length while keeping a bit of randomness. Nc<V^8dQhRS[RS[R,RS[R,RS[R,RS[RS[RS[S[,R,R S[R,/#) r"rrNrrseed drop_lastr6rI)rr rrPrJ)r(rs"rr),DistributedLengthGroupedSampler.__annotate__4st77747Dj 7 Dj 7  77cT!7*7r c VfVf \R4hVf<\P!4'g \R4h\P!4pVf<\P!4'g \R4h\P !4pWnW0nW@n^Vn W`n VfiVeTMRp\V^,\\34'dW^,9d\RV R24hVU u.uFp \W,4NK pp ME\V\P 4'd&\"P%R4VP'4pWpnVP'd}\VP(4VP,^8wdR\*P,!\VP(4VP, VP, 4VnM?\*P,!\VP(4VP, 4VnVP.VP,VnWPnR#uup i)NrMz,Requires distributed package to be availablerNrOrPznIf lengths is a torch.Tensor, DistributedLengthGroupedSampler will be slow. Converting lengths to list[int]...)rQr is_available RuntimeErrorrget_rankrrrepochr_r,rrr2r$r%rRrSr<r6mathceilr total_sizer^) rrrrrr^r_r6rIrTs &&&&&&&&& rr(DistributedLengthGroupedSampler.__init__4s ?wKL L  $$&&"#QRR..0L <$$&&"#QRR==?D$(  " ?3C3O/U` gaj4*?@@DTde\fDf ()1FMMW's745WGMG  . . KK  nn&G  >>>c$,,/$2C2CCqH $yy#dll*;d>O>O*OSWSdSd)deD #yyT\\):T=N=N)NOD **T->->> )NsIc <V^8dQhRS[/#r"r)r)r(rs"rr)r`ms(r c.\P!4pVPVPVP,4\ VP VPVR7pVP'g'W"RVP\V4, , pMVRVPp\V4VP8XgQhW PVPVP1,p\V4VP8XgQh\V4#)r3N)r$ Generator manual_seedr^rerEr6rr_rhr2rrrr)rgrs& rr(DistributedLengthGroupedSampler.__iter__ms OO  dii$**,-,T\\4??VWX~~~ A$//CL"@B BG/0G7|t...))doo8I8IIJ7|t/////G}r ) rr_rer6rrrr^rh)NNNroFNN) rrrrrrrrrrs@rr\r\-s 77rr r\cFa]tRtRtoRtR V3RlRlltRtRtRtVt R#) ShardSampleria Sampler that shards batches between several processes. Dispatches indices batch by batch: on 2 processes with batch size 4, the first two batches are `[0, 1, 2, 3, 4, 5, 6, 7]` and `[8, 9, 10, 11, 12, 13, 14, 15]`, which shard into `[0, 1, 2, 3]` and `[8, 9, 10, 11]` for GPU-0 and `[4, 5, 6, 7]` and `[12, 13, 14, 15]` for GPU-1. The sampler thus yields `[0, 1, 2, 3, 8, 9, 10, 11]` on GPU-0 and `[4, 5, 6, 7, 12, 13, 14, 15]` on GPU-1. c 8<V^8dQhRS[RS[RS[RS[RS[/#)r"rrr_ num_processes process_index)r rr)r(rs"rr)ShardSampler.__annotate__sD@@@@ @  @  @r cWnW nW0nW@nWPnW$,;VnpV'd\ V4V,M%\P!\ V4V, 4pWv,Vn R#r+) rrr_rtrutotal_batch_sizer2rfrgtotal_num_samples)rrrr_rtrurx num_batchess&&&&&& rrShardSampler.__init__sd $"**3=3MM 0:Cc'l&66SVW^S_brSrIs !,!?r c\\\VP444p\V4VP8d(WRVP\V4, , pKA.p\VP VP ,VPVP4FpW!W3VP ,, pK \V4#r+) rPrr2rryrrurxr)rrr; batch_starts& rrShardSampler.__iter__suS./0'lT333 H$"8"83w<"GI IG 43E3E!EtG]G]_c_t_tuK k$//,IJ JFvF|r c<VPVP,#r+)ryrtrs&rrWShardSampler.__len__s%%););;;r )rrr_rtrurxryN)rFrro) rrrrrrrrWrrrs@rrrrrs%@@& < If your IterableDataset implements some randomization that needs to be applied the same way on all processes (for instance, a shuffling), you should use a `torch.Generator` in a `generator` attribute of the `dataset` to generate your random numbers and call the [`~trainer_pt_utils.IterableDatasetShard.set_epoch`] method of this object. It will set the seed of this `generator` to `seed + epoch` on all processes before starting the iteration. Alternatively, you can also implement a `set_epoch()` method in your iterable dataset to deal with this. Args: dataset (`torch.utils.data.IterableDataset`): The batch sampler to split in several shards. batch_size (`int`, *optional*, defaults to 1): The size of the batches per shard. drop_last (`bool`, *optional*, defaults to `False`): Whether or not to drop the last incomplete batch or complete the last batches by using the samples from the beginning. num_processes (`int`, *optional*, defaults to 1): The number of processes running concurrently. process_index (`int`, *optional*, defaults to 0): The index of the current process. seed (`int`, *optional*, defaults to 0): A random seed that will be used for the random number generation in [`~trainer_pt_utils.IterableDatasetShard.set_epoch`]. c ><V^8dQhRS[RS[RS[RS[RS[RS[/#)r"rrr_rtrur^)r rr)r(rs"rr)!IterableDatasetShard.__annotate__sG      r cjWnW nW0nW@nWPnW`n^Vn^VnR#)roN)rrr_rtrur^re num_examples)rrrr_rtrur^s&&&&&&&rrIterableDatasetShard.__init__s3 $"**  r cWn\VPR4'dVPPV4R#R#) set_epochN)rerrr)rres&&rrIterableDatasetShard.set_epochs/ 4<< - - LL " "5 ) .r c#"^Vn\VPR4'g\VPR4'du\VPP\ P 4'dAVPPPVPVP,4VPVP,p\VPVP,VP^,VP,4pRp.pVPFhpV;P^, unVPV4\V4V8XgK?VF pWF,xK VfVP!4p.pKj VP"'gV\V4^8dDVfVP!4p\V4V8d WC, pKVF pWF,xK R#R#R#5i)rorr4N)rrrr,r4r$rmrnr^rerrtrrurr2copyr_)rreal_batch_size process_slice first_batch current_batchelementrss& rrIterableDatasetShard.__iter__s k22 k224<<115??CC LL " " . .tyy4::/E F//D,>,>>d004??BTEWEWZ[E[_c_n_nDno   ||G    "    )=!_4&A'**'&"/"4"4"6K " $~~~#m"4q"8"+002 m$6, "#&&# #9~sE$H+>H*AHcnVP'dI\VP4VPVP,,VP,#\ P !\VP4VPVP,, 4VP,#r+)r_r2rrrtrfrgrs&rrWIterableDatasetShard.__len__sr >>> %$//DL""$ a.s9'8!#c!f++'8!#c3J"TFp\\V44xK R#5ir+rrs& rrMrsB'A!#c!f++'Arz z )is_world_process_zeroprintrr5rZr=keys)rsplitrmetrics_formattedk_widthv_widthr9s&&& r log_metricsr>s^  % % ' ' F5' ()&w/9'899GB'8'?'?'ABBG',,./ 3r'l#3'8'=ay[&IJK0r c BVP4'gR#\PPVPP V R24p\ VR4;_uu_4p\P!W%^RR7RRR4V'd\PPVPP R4p\PPV4'd4\ V4;_uu_4p\P!V4pRRR4M/pXPV4\ VR4;_uu_4p\P!We^RR7RRR4R#R# +'giL;i +'giLn;i +'giR#;i)a Save metrics into a json file for that split, e.g. `train_results.json`. Under distributed environment this is done only for a process with rank 0. Args: split (`str`): Mode/split name: one of `train`, `eval`, `test`, `all` metrics (`dict[str, float]`): The metrics returned from train/evaluate/predict combined (`bool`, *optional*, defaults to `True`): Creates combined metrics by updating `all_results.json` with metrics of this call To understand the metrics please read the docstring of [`~Trainer.log_metrics`]. The only difference is that raw unformatted numbers are saved in the current method. Nz _results.jsonrT)indent sort_keyszall_results.json) rospathjoinargs output_diropenjsondumpexistsloadupdate)rrrcombinedrf all_metricss&&&& r save_metricsrs $  % % ' ' 77<< ,,}.E FD dCA 'Q$7 ww||DII002DE 77>>$  dq"iil K7# $__ IIkQ$ ?_   __s$$E':E:F ' E7 : F F cVP4'gR#\PPVPP R4pVP PV4R#)z Saves the Trainer state, since Trainer.save_model saves only the tokenizer with the model. Under distributed environment this is done only for a process with rank 0. Nztrainer_state.json)rrrrrrstate save_to_json)rrs& r save_statersG  % % ' ' 77<< ,,.B CDJJD!r c$V^8dQhR\/#rkr)r(s"rr)r)sNN#Nr cV\RVPP444#)z) Get the number of trainable parameters. c3h"TF(qP'gKVP4xK* R#5ir+) requires_gradr!)rJps& rrM/get_num_trainable_parameters..s M"9Q__yqwwyy"9s22)rmodel parametersrs&rget_num_trainable_parametersrs" M$**"7"7"9M MMr c:V^8dQhR\\,/#rk)rPr)r(s"rr)r)sBBU Br cVPf \R4hVPPUu.uF qR,NK up#uupi)zB Returns the learning rate of each parameter from self.optimizer. PTrainer optimizer is None, please make sure you have setup the optimizer before.lr optimizerrQ param_groups)rgroups& rget_learning_ratesrsB ~~kll%)^^%@%@ A%@E$KK%@ AA AsAcV^8dQhR\\PPP,R,/#)r"paramN)rJr$r parameter Parameter)r(s"rr)r)s0FFS588+=+=+G+G%G$%NFr cVPf \R4hVe/VPPFpWR,9gKVu# VPPUu.uF q"R,NK up#uupi)z Returns optimizer group for a parameter if given, else returns all optimizer groups for params. Args: param (`str` or `torch.nn.parameter.Parameter`, *optional*): The parameter for which optimizer group needs to be returned. rparamsr)rrrs&& rget_optimizer_grouprsm ~~kll ^^00Eh' 1*.)D)D E)D(OO)D EE Es$A8c|aa\4'dRoMRo\VV3RlVP444#)zg Calculate model's total param count. If trainable_only is True then count only those requiring grads. c^\VR4'd VP#VP4#)ds_numel)rrr!rs&rr!$get_model_param_count..numels#!(J!7!71:: FQWWY Fr c"VP4#r+)r!rs&rr!rs779 r c3l<"TF)pS'dVP'gKS!V4xK+ R#5ir+)r)rJrr!trainable_onlys& rrM(get_model_param_count..s#]!3A>Q__xuQxx!3s 444)rrr)rrr!s&f@rget_model_param_countrs5"## G   ]!1!1!3] ]]r c aa a Ve'VUu.uFp\P!V4NK upM.p.pVP4Fwo p\WaV4pTVUau.uFko\ V\ V44'dK \ ;QJd VV 3RlV4F 'gK RM RM!VV 3RlV44'dKdS RS 2NKm up, pK TVPU a u.uFGo \ ;QJdV 3RlV4F 'gK RM RM!V 3RlV44'dKES NKI up , pV#uupiuupiuup i)zR Returns the names of the model parameters that are not inside a forbidden layer. c3j<"TF(qPS RS 2P44xK* R#5i)rFNsearchlower)rJpatternrLrts& rrM&get_parameter_names..s0fMe'$q}':':'<==Mes03TFrFc3`<"TF#qPSP44xK% R#5ir+r)rJrrWs& rrMrs#/rYqgqwwy0I0IYqs+.)recompilenamed_childrenget_parameter_namesr,rQany _parameters) rforbidden_layer_typesforbidden_layer_namesrforbidden_layer_patternsr;child child_paramsrLrWrts &&& ``@rrrs' G\Fg,AB,AG ,ABmoF++- e*5I^_ ! !eU+@%AB CfMefCCCfMeff tfAaSM!  . $$$aCC/rYq/rCCC/rYq/r,r$F M! C  s;EE< EEE E(E 8E E /E c\VP44pVPPV8Xd VP#\ V4^8XdR#VFp\ W14pVfKVu# R#)z Gets a class from a module by its name. Args: module (`torch.nn.Module`): The module to get the class from. name (`str`): The name of the class. N)rPchildrenrrr2get_module_class_from_name)modulertmodules_children child_module module_classs&& rrrsgFOO-.   D(  ! #,L5lIL'##-r cV'dhVF_p\PPW4p\PPV4'gKI\P!V4Ka R#R#r+)rrrisfileremove)is_main_processr filenamesfilenamefiles&&& rremove_dummy_checkpointr.sD!H77<< 5Dww~~d## $"r cV!R/VBp\V\4'd VR,MV^,pWB,pVPV4V#)lossr)r,rbackward)rinputsgradient_accumulation_stepsoutputsrs&&& rsmp_forward_backwardr9sA/&/",Wd";";wv + t r cV!R/VB#)Nrr)rrs&&rsmp_forward_onlyrAsvr c \V\\34'd\V4!RV44#\V\4'd>\V4!VP 4UUu/uFwrV\ V4bK upp4#\V\P4'g\R\V4 R24h\P!V\PP4pVUu.uFp\V4NK pp\P!VUu.uFqDP!4NK up^R7#uuppiuupiuupi)c38"TFp\V4xK R#5ir+)rr_s& rrMsmp_gather..Gs>v! 1 vraz Can't gather the values of type z-, only of nested list/tuple/dicts of tensors.r/)r,rPrQrOrrSrr$r%rTsmp allgather CommGroupDP_GROUPr-r4rb)r~rWr all_tensorsrKs& rrrEs ftUm , ,<>v>> >  % % 3EE c t\V\\34'd\V4!RV44#\V\4'd>\V4!VP 4UUu/uFwrV\ V4bK upp4#VP4P4P4#uuppi)c38"TFp\V4xK R#5ir+)smp_nested_concatr_s& rrM$smp_nested_concat..TsEf 1! 4 4fra) r,rPrQrOrrSr(rmrrb)r~rWrs& rr(r(Rs ftUm , ,<EfEE E  % %.s^;CSE]E]:]CC;s$ $zThe config file at z had unknown keys (zu), please try upgrading your `transformers` version or fix (and potentially remove these keys) from your config file.r) rrriorrrr=r2rQ)r4 json_file open_filer config_dict extra_keyssf& rfrom_json_file AcceleratorConfig.from_json_files "ww~~i88BGGd y# 8 8A))A,K9^;^^ z?Q %i[0CJ<P]] ![!!9 8s B.. B> cB\P!VP4#r+)rdeepcopy__dict__rs&rto_dictAcceleratorConfig.to_dicts}}T]]++r c8VPPW4#r+)r?pop)rr9r-s&&&rrCAcceleratorConfig.pops}}  ..r c<V^8dQh/S[;R&S[R,;R&S[;R&S[;R&S[;R&S[R,;R&S[;R&#) r" split_batchesNdispatch_batches even_batchesuse_seedable_sampler non_blockinggradient_accumulation_kwargsuse_configured_state)rr)r(rs"rr)AcceleratorConfig.__annotate__\sZ[lTkm|}LM`at#'+ uJKr rr+)rrrrrrrFrGrHrIrJrKrL classmethodr;r@rCr1rrrs@rr+r+\s(V  E M%* O %  L"' u "  L16 Z  1 "' K " " ",/r r+cfaa]tRtRtoRtR V3RlltR V3RlRlltR V3RlRlltRtVt V;t #) LayerWiseDummyOptimizeria For Layer-wise optimizers such as GaLoRE optimizer, the optimization step is already done through the post gradient hooks. Therefore the trick is to create a dummy optimizer that can take arbitrary args and kwargs and return a no-op during training. Initial idea from @hiyouga in LLaMA-Factory: https://github.com/hiyouga/LLaMA-Factory/commit/8664262cde3919e10eaecbd66e8c5d356856362e#diff-ebe08ab14496dfb9e06075f0fdd36799ef6d1535cc4dd4715b74c4e3e06fe3ba c <\P!^^4pWn\SV`V.RVP RR4/4R#)rrgMbP?N)r$randnoptimizer_dictrrget)rrSr dummy_tensorrs&&, rr LayerWiseDummyOptimizer.__init__s:{{1a( , ,$ 40G)HIr c$<V^8dQhRS[RR/#)r" set_to_nonerN)r)r(rs"rr)$LayerWiseDummyOptimizer.__annotate__s  T T r cR#r+r)rrXs&&r zero_grad!LayerWiseDummyOptimizer.zero_grad r c.<V^8dQhRS[R,/#r)r)r(rs"rr)rYs  EDL r cR#r+r)rclosures&&rstepLayerWiseDummyOptimizer.stepr]r )rSr+T) rrrrrrr[rarrrrs@@rrPrPs)J      r rPcBaa]tRtRtoRtV3RltRtRtRtVt V;t #)LayerWiseDummyScheduleria  For Layer-wise optimizers such as GaLoRE optimizer, the optimization and scheduling step are already done through the post gradient hooks. Therefore the trick is to create a dummy scheduler that can take arbitrary args and kwargs and return a no-op during training. c\<VR,Vn\R/VBpRp\SV` W44R#)rNrr) default_lrrPrr)rrrr last_epochrs&*, rr LayerWiseDummyScheduler.__init__s- ,+5f5   /r c VP.pVPegVPPP4UUu.uF$q"PUu.uF q3R,NK upNK& ppp\ \ V!4pV#uupiuuppi)Nr)rgrrSrZrrPr)rlrsoptimrparam_wise_lrss& rget_lrLayerWiseDummyScheduler.get_lrs >> %KO>>KhKhKoKoKqKq%*<*<=*<t*<=Kq un-.C >sB B &B B cVP#r+)base_lrsrs&r_get_closed_form_lr+LayerWiseDummyScheduler._get_closed_form_lrs }}r )rg) rrrrrrrnrrrrrrs@@rreres0 r rec@VP4pRpV'd$VPPW$,4R#VPPW$,4R# \d1p\ P TPYR74Rp?R#Rp?ii;i)zIHelper to set RNG state for a specific device type (CUDA, NPU, MLU, MUSA)zDidn't manage to set back the RNG states of the {backend} because of the following error: {exception} This won't yield the same results as if the training had not been interrupted.)backend exceptionN)rrandomset_rng_state_all set_rng_state ExceptionrRerrorr() device_name device_modulecheckpoint_rng_stateis_distributeddevice_state_key err_templatees&&&& rset_rng_state_for_devicers"((*MLL   2 23G3Y Z  . ./C/U V L \(((JKKLs!A"!A"!A"" B-%BBc\P!\P4P\P!R4P8d\ P !4#\P!\P4\P!R48d\PM\PpVPP\P\P.pV\\P!\P44., p\P P#V4#)a Context manager to allowlist numpy objects for torch.load with weights_only=True. Starting from version 2.4 PyTorch introduces a check for the objects loaded with torch.load(weights_only=True). Starting from 2.6 weights_only=True becomes a default and requires allowlisting of objects being loaded. See: https://github.com/pytorch/pytorch/pull/137602 See: https://pytorch.org/docs/stable/notes/serialization.html#torch.serialization.add_safe_globals See: https://github.com/huggingface/accelerate/pull/3036 z2.6z2.0.0)r parser$ __version__release contextlib nullcontextr&_corerw multiarray _reconstructr'rcrOuint32 serialization safe_globals)np_core allowlists rrr#s}}U&&'//'--2F2N2NN%%''!--77==;QQbhhWYW^W^G##00"**bhhGI$rxx *+,,I    + +I 66r )rr+)NNrcr0)r)rrrrrr6rrfrrsysrcollections.abcrrr dataclassesrr itertoolsrrrtypingr rgr&r$torch.distributed distributedr packagingr r torch.utils.datar r rrtorch.utils.data.distributedrintegrations.deepspeedrtokenization_utils_baserrrrrr add_handlerstdouttorch_xla.runtimeruntimertorch.optim.lr_schedulerr get_loggerrrRrr-r<rDrIrYr^rkrrrrrrrrrrrrrr r rrErGr\rrrrrrrrrrrrrrr!smdistributed.modelparallel.torch modelparallelr!rarrrr(r+rl OptimizerrPrerrrr rrs  -%(! MM;>2 !!  cjj12"4   H %"$$T.E$  NY"I&&*^&*"',,v"6I&5  !3811h_ W 'L'L 'LT ?F(7(VR&8Rj,<7,<^fk?fkRA6WLv#@N "NBF"^"0$( 33XXZ XXZ @. // /D ekk33 .k> L7r