+ QV-j~Rt^RIHt^RIt^RIt^RIHt^RIHt^RI t ^RI H t ^RI H t Ht^RIHtHt^RIHt^R IHt]P,!]4tRBR ltRCR R lltR RltRRltRCRRlltRRltRCRltRRlt RDRRllt!RRlt"RERRllt#RRlt$RFR lt%R!R/R"R#llt&RGR$R%llt'R&R'/R(R)llt RHR*R+llt(R&R'R,R/R-R.llt)RIR/R0llt*R1R2lt+RJR3R4llt,!R5R64t-!R7R84t.R9R:lt/]P`]]Pb]!]Pd]#]Pf]%]Ph]]Pj]]Pl]']Pn]]Pp](]Pr]*]Pt],]Pv]// t] 4t>!R?R@] 4t?RLRAlt@R#)Mz$PyTorch optimization for BERT model.) annotationsN)partial)Any) Optimizer)LambdaLRReduceLROnPlateau)LayerWiseDummyOptimizerLayerWiseDummyScheduler) SchedulerType)loggingc^#)_s&j/Volumes/fast/ai/experiments/ui-tars-smoke/.venv/lib/python3.14/site-packages/transformers/optimization.py_get_constant_lambdar#s c V^8dQhRRRR/#) optimizerr last_epochintr)formats"r __annotate__r'sLLYLCLrc&\V\VR7#)a Create a schedule with a constant learning rate, using the learning rate set in optimizer. Args: optimizer ([`~torch.optim.Optimizer`]): The optimizer for which to schedule the learning rate. last_epoch (`int`, *optional*, defaults to -1): The index of the last epoch when resuming training. Return: `torch.optim.lr_scheduler.LambdaLR` with the appropriate schedule. r)rr)rrs&&rget_constant_scheduler's I3 KKrcV^8dQhRR/#rrrr)rs"rrr8s22i2rc \V3/VB#)a Create a schedule with a constant learning rate that decreases when a metric has stopped improving. Args: optimizer ([`~torch.optim.Optimizer`]): The optimizer for which to schedule the learning rate. kwargs (`dict`, *optional*): Extra parameters to be passed to the scheduler. See `torch.optim.lr_scheduler.ReduceLROnPlateau` for possible parameters. Return: `torch.optim.lr_scheduler.ReduceLROnPlateau` with the appropriate schedule. )rrkwargss&,rget_reduce_on_plateau_scheduler#8s Y 1& 11rc V^8dQhRRRR/#)r current_steprnum_warmup_stepsr)rs"rrrJssY\rc\W8d&\V4\\RV44, #R#)?floatmax)r%r&s&$r,_get_constant_schedule_with_warmup_lr_lambdar,Js(&\"U3s4D+E%FFF rc$V^8dQhRRRRRR/#)rrrr&rrr)rs"rrrPs'AAAcA_bArc>\\VR7p\WVR7#)a4 Create a schedule with a constant learning rate preceded by a warmup period during which the learning rate increases linearly between 0 and the initial lr set in the optimizer. Args: optimizer ([`~torch.optim.Optimizer`]): The optimizer for which to schedule the learning rate. num_warmup_steps (`int`): The number of steps for the warmup phase. last_epoch (`int`, *optional*, defaults to -1): The index of the last epoch when resuming training. Return: `torch.optim.lr_scheduler.LambdaLR` with the appropriate schedule. r&r)rr,r)rr&r lr_lambdas&&& r!get_constant_schedule_with_warmupr1Ps"DWghI IZ @@rc$V^8dQhRRRRRR/#)rr%rr&num_training_stepsr)rs"rrres(uuSuWZupsurc W8d&\V4\\^V44, #\R\W , 4\\^W!, 44, 4#)rr))r%r&r3s&$$r*_get_linear_schedule_with_warmup_lr_lambdar6esP&\"U3q2B+C%DDD sE,;\\VVR7p\WV4#)a Create a schedule with a learning rate that decreases linearly from the initial lr set in the optimizer to 0, after a warmup period during which it increases linearly from 0 to the initial lr set in the optimizer. Args: optimizer ([`~torch.optim.Optimizer`]): The optimizer for which to schedule the learning rate. num_warmup_steps (`int`): The number of steps for the warmup phase. num_training_steps (`int`): The total number of training steps. last_epoch (`int`, *optional*, defaults to -1): The index of the last epoch when resuming training. Return: `torch.optim.lr_scheduler.LambdaLR` with the appropriate schedule. )r&r3)rr6r)rr&r3rr0s&&&& rget_linear_schedule_with_warmupr8ks&&2)-I I* 55rc(V^8dQhRRRRRRRR/#)rr%rr&r3 num_cyclesr*r)rs"rrrs2ZZZ,/ZEHZV[Zrc nW8d&\V4\\^V44, #\W, 4\\^W!, 44, p\RRR\P!\P\V4,R,V,4,,4#)rr5?r(@r*r+mathcospir%r&r3r:progresss&$$$ r*_get_cosine_schedule_with_warmup_lr_lambdarDs&\"U3q2B+C%DDD\45c!EWEj>k8llH sC3$''E*4E*E*Kh*V!WWX YYrc ,V^8dQhRRRRRRRRRR/#) rrrr&rr3r:r*rr)rs"rrrs4666,/6EH6V[6or6rc@\\VVVR7p\WV4#)a Create a schedule with a learning rate that decreases following the values of the cosine function between the initial lr set in the optimizer to 0, after a warmup period during which it increases linearly between 0 and the initial lr set in the optimizer. Args: optimizer ([`~torch.optim.Optimizer`]): The optimizer for which to schedule the learning rate. num_warmup_steps (`int`): The number of steps for the warmup phase. num_training_steps (`int`): The total number of training steps. num_cycles (`float`, *optional*, defaults to 0.5): The number of waves in the cosine schedule (the defaults is to just decrease from the max value to 0 following a half-cosine). last_epoch (`int`, *optional*, defaults to -1): The index of the last epoch when resuming training. Return: `torch.optim.lr_scheduler.LambdaLR` with the appropriate schedule. r&r3r:)rrDrrr&r3r:rr0s&&&&& rget_cosine_schedule_with_warmuprIs)22)- I I* 55rc(V^8dQhRRRRRRRR/#)rr%rr&r3r:r)rs"rrrs2^^^,/^EH^VY^rc W8d&\V4\\^V44, #\W, 4\\^W!, 44, pVR8dR#\RRR\P!\P\V4V,R,,4,,4#)rr(r5r<r>rBs&$$$ r=_get_cosine_with_hard_restarts_schedule_with_warmup_lr_lambdarLs&\"U3q2B+C%DDD\45c!EWEj>k8llH3 sC3$''eJ6G(6RVY5Y*Z![[\ ]]rc ,V^8dQhRRRRRRRRRR/#)rrrr&rr3r:rr)rs"rrrs4666,/6EH6VY6kn6rc@\\VVVR7p\WV4#)aJ Create a schedule with a learning rate that decreases following the values of the cosine function between the initial lr set in the optimizer to 0, with several hard restarts, after a warmup period during which it increases linearly between 0 and the initial lr set in the optimizer. Args: optimizer ([`~torch.optim.Optimizer`]): The optimizer for which to schedule the learning rate. num_warmup_steps (`int`): The number of steps for the warmup phase. num_training_steps (`int`): The total number of training steps. num_cycles (`int`, *optional*, defaults to 1): The number of hard restarts to use. last_epoch (`int`, *optional*, defaults to -1): The index of the last epoch when resuming training. Return: `torch.optim.lr_scheduler.LambdaLR` with the appropriate schedule. rG)rrLrrHs&&&&& r2get_cosine_with_hard_restarts_schedule_with_warmuprOs)0E)- I I* 55rc 0V^8dQhRRRRRRRRRRRR/#) rr%rr&r3lr_endr*powerlr_initr)rs"rrrsF     rcW8d&\V4\\^V44, #W8d W5, #WS, pW!, p^W, V, , pWhV,,V,p W, #r r)) r%r&r3rQrRrSlr_range decay_steps pct_remainingdecays &$$$$$ r4_get_polynomial_decay_schedule_with_warmup_lr_lambdarYsr&\"U3q2B+C%DDD  *#(; \< KK %//&8rc VPR,pWc8g\RV RV R24h\\VVVVVR7p\ WV4#)aE Create a schedule with a learning rate that decreases as a polynomial decay from the initial lr set in the optimizer to end lr defined by *lr_end*, after a warmup period during which it increases linearly from 0 to the initial lr set in the optimizer. Args: optimizer ([`~torch.optim.Optimizer`]): The optimizer for which to schedule the learning rate. num_warmup_steps (`int`): The number of steps for the warmup phase. num_training_steps (`int`): The total number of training steps. lr_end (`float`, *optional*, defaults to 1e-7): The end LR. power (`float`, *optional*, defaults to 1.0): Power factor. last_epoch (`int`, *optional*, defaults to -1): The index of the last epoch when resuming training. Note: *power* defaults to 1.0 as in the fairseq implementation, which in turn is based on the original BERT implementation at https://github.com/google-research/bert/blob/f39e881b169b9d53bea03d2d341b31707a6c052b/optimization.py#L37 Return: `torch.optim.lr_scheduler.LambdaLR` with the appropriate schedule. lrzlr_end (z#) must be smaller than initial lr ())r&r3rQrRrS)defaults ValueErrorrrYr)rr&r3rQrRrrSr0s&&&&&& r)get_polynomial_decay_schedule_with_warmupr_sa>  &G  8F8+NwiWXYZZ<)- I I* 55r timescalec$V^8dQhRRRRRR/#)rr%rr&r` int | Noner)rs"rrr s#sQTakrcW8d&\V4\\^V44, #W!, pR\P!W,V, 4, pV#)rr()r*r+r?sqrt)r%r&r`shiftrXs&$$ r$_get_inverse_sqrt_schedule_lr_lambdarf sM&\"U3q2B+C%DDD  (E $))\1Y>? ?E Lrc(V^8dQhRRRRRRRR/#)rrrr&rr`rbrr)rs"rrr(s2AAA,/A)r%r&r3r:rjrCfactors&$$$$ rrDrDFs&\"U3q2B+C%DDD\45c!EWEj>k8llH C$((477U:->#>#Dx#OPP QF q; '+ 5F q&>rc4V^8dQhRRRRRRRRRRR R R R /#) rrrr&rr3r:r*rmin_lr float | Nonerjr)rs"rrrQsN1616161616 16  16  1616rcVeVe \R4hVeWPPR,, pMVf \R4h\\VVVVR7p\ WV4#)a Create a schedule with a learning rate that decreases following the values of the cosine function between the initial lr set in the optimizer to min_lr, after a warmup period during which it increases linearly between 0 and the initial lr set in the optimizer. Args: optimizer ([`~torch.optim.Optimizer`]): The optimizer for which to schedule the learning rate. num_warmup_steps (`int`): The number of steps for the warmup phase. num_training_steps (`int`): The total number of training steps. num_cycles (`float`, *optional*, defaults to 0.5): The number of waves in the cosine schedule (the defaults is to just decrease from the max value to 0 following a half-cosine). last_epoch (`int`, *optional*, defaults to -1): The index of the last epoch when resuming training. min_lr (`float`, *optional*): The minimum learning rate to reach after the cosine schedule. min_lr_rate (`float`, *optional*): The minimum learning rate as a ratio of the initial learning rate. If set, `min_lr` should not be set. Return: `torch.optim.lr_scheduler.LambdaLR` with the appropriate schedule. /Only one of min_lr or min_lr_rate should be setr[LOne of min_lr or min_lr_rate should be set through the `lr_scheduler_kwargs`)r&r3r:rj)r^r]rrDr)rr&r3r:rrorjr0s&&&&&&& r+get_cosine_with_min_lr_schedule_with_warmuprtQsoFk5JKK  11$77  ghh2)- I I* 55rwarmup_lr_ratec 0V^8dQhRRRRRRRRRRRR /#) rr%rr&r3r:r*rjrurpr)rs"rrrsF     !rc \V4p\V4p\V4pW8dYVfVR,\RV4, #\V4pVRV, V,\^V^, 4, ,#W, R,\RW!, 4, pRR\P!\PV,R,V,4,,pV^V, ,V,p\^V4#)Nr(r<r=r>)r%r&r3r:rjrurCrms&$$$$$ r;_get_cosine_with_min_lr_schedule_with_warmup_lr_rate_lambdarxs&L-.12&  ! 3&#c3C*DD D">2N!S>%9l$KsSTVfijVjOk$ll l/#5#cCUCh:ijH C$((477Z#7##=#HII JF q; '+ 5F q&>rc8V^8dQhRRRRRRRRRRR R R R R R /#) rrrr&rr3r:r*rrorprjrur)rs"rrrsX5656565656 56  56  5656!56rc VeVe \R4hVeWPPR,, pMVf \R4h\\VVVVVR7p\ WV4#)aV Create a schedule with a learning rate that decreases following the values of the cosine function between the initial lr set in the optimizer to min_lr, after a warmup period during which it increases linearly between 0 and the initial lr set in the optimizer. Args: optimizer ([`~torch.optim.Optimizer`]): The optimizer for which to schedule the learning rate. num_warmup_steps (`int`): The number of steps for the warmup phase. num_training_steps (`int`): The total number of training steps. num_cycles (`float`, *optional*, defaults to 0.5): The number of waves in the cosine schedule (the defaults is to just decrease from the max value to 0 following a half-cosine). last_epoch (`int`, *optional*, defaults to -1): The index of the last epoch when resuming training. min_lr (`float`, *optional*): The minimum learning rate to reach after the cosine schedule. min_lr_rate (`float`, *optional*): The minimum learning rate as a ratio of the initial learning rate. If set, `min_lr` should not be set. warmup_lr_rate (`float`, *optional*): The minimum learning rate as a ratio of the start learning rate. If not set, `warmup_lr_rate` will be treated as float(1/num_warmup_steps). Return: `torch.optim.lr_scheduler.LambdaLR` with the appropriate schedule. rrr[rs)r&r3r:rjru)r^r]rrxr) rr&r3r:rrorjrur0s &&&&&&&& r3get_cosine_with_min_lr_schedule_with_warmup_lr_rater{srLk5JKK  11$77  ghhC)-% I I* 55rc8V^8dQhRRRRRRRRRRRRR R R R /#) rr%rr&num_stable_stepsnum_decay_steps warmup_typestr decay_type min_lr_ratior*r:r)rs"rrrsZ#### #  #  ####rcW8d\V4\\^V44, pVR8XdTp MlVR8Xd;RR\P!\PV,4, ,p M+VR8Xd%R\P !RV, 4, p X RV, ,V,p \RV 4#WV,8dR#WV,V,8d\W, V, 4\\^V44, pVR8Xd RV, p M|VR8XdRRR\P!\P\V4,R,V,4,,p M$VR8XdR\P !V4, p X RV, ,V,p \RV 4#V#)rlinearcosiner<r(1-sqrtr5r=)r*r+r?r@rArd) r%r&r}r~rrrr:rCrms &$$$$$$$ r_get_wsd_scheduler_lambdarsn&&s16F/G)HH ( "F H $C$((477X+=">>?F H $499S8^44F3-.=3)999)99OKK8;KKLuUXYZ\kUlOmm  !8^F 8 #C$((477U:5F+F+Lx+W"XXYF 8 #499X..F3-.=3 rc@V^8dQhRRRRRRRRRRR R R R R R RR RR/ #)rrrr&rr~r3rbr}rrrrr*r:rr)rs"rrrsxF6F6F6F6F6# F6 ! F6  F6F6F6F6F6rc (VfVf \R4hVeVe\P!R4VR9d\RV R24hVR9d\RV R24hVfW1, V, p\\VVVVVVVR7p \ W V 4#)a Create a schedule with a learning rate that has three stages: 1. warmup: increase from min_lr_ratio times the initial learning rate to the initial learning rate following a warmup_type. 2. stable: constant learning rate. 3. decay: decrease from the initial learning rate to min_lr_ratio times the initial learning rate following a decay_type. Args: optimizer ([`~torch.optim.Optimizer`]): The optimizer for which to schedule the learning rate. num_warmup_steps (`int`): The number of steps for the warmup phase. num_decay_steps (`int`): The number of steps for the decay phase. num_training_steps (`int`, *optional*): The total number of training steps. This is the sum of the warmup, stable and decay steps. If `num_stable_steps` is not provided, the stable phase will be `num_training_steps - num_warmup_steps - num_decay_steps`. num_stable_steps (`int`, *optional*): The number of steps for the stable phase. Please ensure that `num_warmup_steps + num_stable_steps + num_decay_steps` equals `num_training_steps`, otherwise the other steps will default to the minimum learning rate. warmup_type (`str`, *optional*, defaults to "linear"): The type of warmup to use. Can be 'linear', 'cosine' or '1-sqrt'. decay_type (`str`, *optional*, defaults to "cosine"): The type of decay to use. Can be 'linear', 'cosine' or '1-sqrt'. min_lr_ratio (`float`, *optional*, defaults to 0): The minimum learning rate as a ratio of the initial learning rate. num_cycles (`float`, *optional*, defaults to 0.5): The number of waves in the cosine schedule (the defaults is to just decrease from the max value to 0 following a half-cosine). last_epoch (`int`, *optional*, defaults to -1): The index of the last epoch when resuming training. Return: `torch.optim.lr_scheduler.LambdaLR` with the appropriate schedule. z@Either num_training_steps or num_stable_steps must be specified.zZBoth num_training_steps and num_stable_steps are specified. num_stable_steps will be used.zUnknown warmup type: z), expected 'linear', 'cosine' or '1-sqrt'zUnknown decay type: )r&r}r~rrrr:)rrr)r^warningswarnrrr) rr&r~r3r}rrrr:rr0s &&&&&&&&&& rget_wsd_schedulersZ!&6&>[\\%*:*F rs880 =fghh77/ |;deff-@?R!))'! I I* 55rcJ]tRtRtRtRRltRRltRRltR R ltR t R #) StreamingAverageiEa*Rolling window average for smoothing metric values. Maintains a sliding window of values and computes their average, useful for smoothing noisy metric values before making learning rate decisions. Args: window_size (`int`): The maximum number of values to keep in the rolling window. c V^8dQhRRRR/#)r window_sizerreturnNoner)rs"rrStreamingAverage.__annotate__PsCDrc .Wn.VnRVnR#)r5Nrvaluessum)selfrs&&r__init__StreamingAverage.__init__Ps +#% rc V^8dQhRRRR/#)rvaluer*rr)rs"rrrUs + +u + +rc fVPPV4V;PV, un\VP4VP8d5VPP ^4pV;PV,unVP\VP4, #)z3Add a value and return the current rolling average.)rappendrlenrpop)rrremoveds&& r streamavgStreamingAverage.streamavgUsq 5! E t{{ d.. .kkooa(G HH Hxx#dkk***rcV^8dQhRR/#rrdict[str, Any]r)rs"rrr`s  N rc jRVPRVPP4RVP/#r)rrcopyrrs&r state_dictStreamingAverage.state_dict`s2 4++ dkk&&( 488  rc V^8dQhRRRR/#rrrrrr)rs"rrrgs....T.rc VPRVP4VnVPR.4P4VnVPRR4VnR#)rrrr5N)getrrrr)rrs&&rload_state_dict StreamingAverage.load_state_dictgsH%>>-9I9IJ nnXr2779 >>%-r)rrrN) __name__ __module__ __qualname____firstlineno____doc__rrrr__static_attributes__rrrrrEs  + ..rrc]tRtRtRtRRRlltRRRlltRR ltR R ltR R lt RRlt RRlt RRlt RRlt RtR#)GreedyLRima Adaptive learning rate scheduler that responds to training metrics. GreedyLR dynamically adjusts the learning rate based on training performance: - Increases LR when metrics improve consistently (divides by factor) - Decreases LR when metrics plateau (multiplies by factor) This differs from traditional schedulers like cosine annealing by responding to actual training dynamics rather than following a predetermined schedule. Reference: `GreedyLR: A Novel Adaptive Learning Rate Scheduler `_ Args: optimizer ([`~torch.optim.Optimizer`]): The optimizer for which to schedule the learning rate. mode (`str`, *optional*, defaults to `"min"`): One of 'min' or 'max'. In 'min' mode, LR will be reduced when the metric has stopped decreasing; in 'max' mode when it has stopped increasing. factor (`float`, *optional*, defaults to 0.95): Factor by which the learning rate will be adjusted. LR is multiplied by factor on plateau and divided by factor on improvement. Must be < 1.0. patience (`int`, *optional*, defaults to 10): Number of epochs with no improvement after which learning rate will be adjusted. threshold (`float`, *optional*, defaults to 1e-06): Threshold for measuring the new optimum. threshold_mode (`str`, *optional*, defaults to `"abs"`): One of 'rel' or 'abs'. cooldown (`int`, *optional*, defaults to 0): Number of epochs to wait before resuming normal operation after LR has been reduced. warmup (`int`, *optional*, defaults to 0): Number of epochs to wait before resuming normal operation after LR has been increased. min_lr (`float` or `list[float]`, *optional*, defaults to 0.001): A lower bound on the learning rate. max_lr (`float` or `list[float]`, *optional*, defaults to 1.0): An upper bound on the learning rate. eps (`float`, *optional*, defaults to 1e-08): Minimal decay applied to lr. verbose (`bool`, *optional*, defaults to `False`): If True, prints a message to stdout for each update. smooth (`bool`, *optional*, defaults to `False`): If True, applies streaming average smoothing to metrics. window_size (`int`, *optional*, defaults to 50): The window size for the streaming average when smooth=True. reset_start (`int`, *optional*, defaults to 500): Number of steps to wait at min_lr before resetting to initial state. Example: ```python >>> optimizer = torch.optim.SGD(model.parameters(), lr=0.1) >>> scheduler = GreedyLR(optimizer, mode="min", patience=10) >>> for epoch in range(100): ... train(...) ... val_loss = validate(...) ... scheduler.step(val_loss) ``` c XV^8dQhRRRRRRRRR RR RR RR RR RRRRRRRRRRRRRRR/#)rrrmoderrmr*patiencer thresholdthreshold_modecooldownwarmuprozfloat | list[float]max_lrepsverboseboolsmoothr reset_startrrr)rs"rrGreedyLR.__annotate__sE@E@E@E@ E@  E@  E@E@E@E@$E@$E@E@E@E@E@ !E@" #E@rc xVR8d \R4h\V\4'g"\\ V4P R24hWnW0nW@nWn Wpn Wn ^Vn ^Vn W nWPnW`nWnWnWnWnWn^Vn\V \.\034'db\3V 4\3VP448wd.\R\3VP44 R\3V 4 24h\/V 4VnM"V .\3VP44,Vn\V \.\034'db\3V 4\3VP448wd.\R\3VP44 R\3V 4 24h\/V 4VnM"V .\3VP44,VnVP4Uu.uF pVR,NK upVnVP:P=4VnVR8Xd \AR4M \AR 4Vn!^Vn"^Vn#VR9d\R V R 24hVR9d\R V R 24hR Vn$V 'd\KV4Vn$R #R #uupi)r(zFactor should be < 1.0.z is not an Optimizerz expected z min_lrs, got z max_lrs, got r[mininf-infzmode z is unknown!zthreshold mode N)rr+)relabs)&r^ isinstancer TypeErrortyperrrmrrrrcooldown_counterwarmup_counterrrrrrrrreset_start_originalrlisttupler param_groupsmin_lrsmax_lrs _init_lrsr_last_lrr*bestnum_bad_epochsnum_good_epochs_streaming_avgr)rrrrmrrrrrrorrrrrrgroups&&&&&&&&&&&&&&&& rrGreedyLR.__init__sN$ S=67 7)Y//tI7788LMN N"     ! ", &&$/! ftUm , ,6{c)"8"899 9S1G1G-H,IX[\bXcWd!efflKL L7; "2;"?D  Ks?J7Nc$V^8dQhRRRRRR/#)rmetricsr*epochrbrrr)rs"rrrs&,O,OE,O*,O,Orc \V4pVP'd*VPeVPPV4pVfVP^,pW nVP ^8d*V;P ^,un^Vn^VnEM*VP^8d)V;P^,un^Vn^VnMVPW0P4'd(W0n ^VnV;P^, unM V;P ^, un^VnVPVP8d+VPV4VPVn^VnMDVP VP8d*VPV4VPVn^VnVP P"Uu.uF qDR,NK upVnR#uupi)zPerform a scheduler step based on the given metrics. Args: metrics (`float`): The metric value to use for LR adjustment decisions. epoch (`int`, *optional*): The current epoch number. If None, uses internal counter. Nr[)r*rrrrrrrr is_betterrr _increase_lrr _reduce_lrrrrr)rrrcurrentrs&&& rstep GreedyLR.steps. ;;;4..:))33G>>!666""e+t~~)=!>>>!666rc V^8dQhRRRR/#rrrrrr)rs"rrr'src  0Rp\VPP4Fwr4\VR,4p\ WPP ,VP V,4pWV, VP8d-WdR&VP'd\RV RV RVR R24VR,VP V,8gKRpK V'd?V;P^,un VP^8:dVP4R#R#R#) Tr[Epoch z": reducing learning rate of group  to .4e.FN) enumeraterrr*r+rmrrrprintr_reset)rr all_at_mini param_groupold_lrnew_lrs&& rrGreedyLR._reduce_lr's '(C(CDNA;t,-F++-t||A?F)$*D!<<<F5')KA3dSYZ]R^^_`a4 4<<?2" E     ! 1$ % rc V^8dQhRRRR/#rr)rs"rrr:s 5 5# 5$ 5rc  \VPP4Fwr#\VR,4p\ W@P , VP V,4pWT, VP8gK]WSR&VP'gKu\RV RV RVR R24K VPVn R#)r[rz$: increasing learning rate of group rrrN) rrrr*rrmrrrrrr)rrrrrrs&& rrGreedyLR._increase_lr:s'(C(CDNA;t,-F++-t||A?F)$*D!<<<F5')MaSPTU[\_T``abcE 44rcV^8dQhRR/#)rrrr)rs"rrrFs777rc \VPP4FwrVPV,VR&K VPR8Xd \ R4M \ R4Vn^Vn^Vn^Vn ^Vn VPVn VP'd)VPe\VP 4VnVP"'d\%R4R#R#)r[rrrNz!Scheduler reset to initial state.)rrrrrr*rrrrrrrrrrrrr)rrrs& rrGreedyLR._resetFs'(C(CDNA $q 1K E%)II$6E%LE&M   !44 ;;;4..:"243C3C"DD  <<< 5 6 rcV^8dQhRR/#)rrz list[float]r)rs"rrrWs[rc VP#)z8Return last computed learning rate by current scheduler.)rrs&r get_last_lrGreedyLR.get_last_lrWs }}rcV^8dQhRR/#rr)rs"rrr[sNrc  /RVPbRVPbRVPbRVPbRVPbRVP bRVP bRVPbR VPbR VPbR VPbR VPbR VPbRVPbRVPbRVPbRVP bRVP"RVP$RVP&RVP(RVP*RVP,/CpVP"'d,VP.eVP.P14VR&V#)z2Return the state of the scheduler as a dictionary.rmrrrrrrrrrrrrrrrrrrrrrrr)rmrrrrrrrrrrrrrrrrrrrrrrrr)rstates& rrGreedyLR.state_dict[s dkk t||  t||    t||     dkk   5 5  d11  DII    d11  DII  d11  t33 488! " $//# $ dkk 4++ 4++ "D$=$=  / 4 ;;;4..:&*&9&9&D&D&FE" # rc V^8dQhRRRR/#rr)rs"rrr|s!'!'.!'T!'rc VPRVP4VnVPRVP4VnVPRVP4VnVPRVP4VnVPRVP 4VnVPRVP 4VnVPRVP4VnVPRVP4VnVPR VP4Vn VPR VP4Vn VPR VP4Vn VPR VP4Vn VPR VP4Vn VPRVP4VnVPRVP4VnVPRVP 4VnVPRVP"4VnVPRVP$4VnVPRVP&4VnVPRVP(4VnVPRVP*4VnVPRVP,4VnVPRVP.4VnRV9dKVP0f\3VP&4VnVP0P5VR,4RV9d:\7VP8P:VP,4F wr#W2R&K R#R#)zLoad state from a dictionary.rmrrrrrrrrrrrrrrrrrrrrrrrNr[)rrmrrrrrrrrrrrrrrrrrrrrrrrrrziprr)rrrr[s&& rrGreedyLR.load_state_dict|s nnXt{{; !~~i> !~~i> "z4==A !~~i> "z4==A  nnXt{{; */A4CXCX Y(nn-=t?R?RSNN64995 # T^^D(nn-=t?R?RSNN64995 (nn-=t?R?RS)~~.?AUAUV>>%2$..tG nnXt{{; %>>-9I9IJ%>>-9I9IJ$.NN3I4KdKd$e!"z4==A # T^^D z )""*&6t7G7G&H#    / / ;K0L M  ##&t~~'B'BDMM#R $&D!$S $r)rrrrrrrrmrrrrrrrrrrrrrrrrr)rgffffff? ư>rrMbP?r(g:0yE>FF2iN)rrrrrrrrrrrrrrrrrrrrms@6pE@N,O\ 7& 57"B!'!'rrcV^8dQhRR/#rr)rs"rrrs ) )9 )rc \V3/VB#)a Create an adaptive learning rate scheduler that adjusts LR based on training metrics. Args: optimizer ([`~torch.optim.Optimizer`]): The optimizer for which to schedule the learning rate. kwargs (`dict`, *optional*): Extra parameters passed to the scheduler. See [`GreedyLR`] for possible parameters. Return: [`GreedyLR`] with the appropriate schedule. )rr!s&,rget_greedy_schedulers I ( ((rc ,V^8dQhRRRRRRRRRR /#) rnamezstr | SchedulerTyperrr&rbr3scheduler_specific_kwargsz dict | Noner)rs"rrrsA^^ ^^!^# ^ + ^rc Ja \V4p\V,pVe\V\4'dVPp/o VFp\ VWg,VVVR7S V&K V 3RlpVF(pVP 'gKVPV4K* \WaPR,R7#V\P8Xd V!V4#Vf/pV\P8Xd V!V3/VB#V\P8Xd V!V3/VB#Vf\V R24hV\P8Xd V!WR7#V\P8Xd V!V3RV/VB#V\P 8XdV!V3RVRV/VB#Vf\V R 24hV!V3RVRV/VB#) a Unified API to get any scheduler from its name. Args: name (`str` or `SchedulerType`): The name of the scheduler to use. optimizer (`torch.optim.Optimizer`): The optimizer that will be used during training. num_warmup_steps (`int`, *optional*): The number of warmup steps to do. This is not required by all schedulers (hence the argument being optional), the function will raise an error if it's unset and the scheduler type requires it. num_training_steps (`int``, *optional*): The number of training steps to do. This is not required by all schedulers (hence the argument being optional), the function will raise an error if it's unset and the scheduler type requires it. scheduler_specific_kwargs (`dict`, *optional*): Extra parameters for schedulers such as cosine with restarts. Mismatched scheduler types and scheduler parameters will cause the scheduler function to raise a TypeError. )rr&r3rc6<SV,P4R#r)r)paramscheduler_dicts&rscheduler_hook%get_scheduler..scheduler_hooks 5 ! & & (rr[)optimizer_dictr[z; requires `num_warmup_steps`, please provide that argument.r/r&r3z= requires `num_training_steps`, please provide that argument.)r TYPE_TO_SCHEDULER_FUNCTIONrrr" get_scheduler requires_grad"register_post_accumulate_grad_hookr r]CONSTANTREDUCE_ON_PLATEAUGREEDYr^CONSTANT_WITH_WARMUP INVERSE_SQRTWARMUP_STABLE_DECAY) rrr&r3r schedule_funcr"rr rs &&&&& @rr$r$s2  D.t4MI7N!O!O"11#E$1(/!1#5*C %N5 !$ ) $E"""88H$'nI[I[\`Iabb }%%%Y'' ($&! }...YD*CDD }###YD*CDDD6!\]^^ }111YJJ })))Yg9IgMfgg }000  -  2 (   !D6!^_``  ) .  $  rca]tRtRtRtR V3Rllt]R4t]R4t]R4t ]R4t ] P!4R Rl4t R tV;t#) Adafactori!a- AdaFactor pytorch implementation can be used as a drop in replacement for Adam original fairseq code: https://github.com/pytorch/fairseq/blob/master/fairseq/optim/adafactor.py Paper: *Adafactor: Adaptive Learning Rates with Sublinear Memory Cost* https://huggingface.co/papers/1804.04235 Note that this optimizer internally adjusts the learning rate depending on the `scale_parameter`, `relative_step` and `warmup_init` options. To use a manual (external) learning rate schedule you should set `scale_parameter=False` and `relative_step=False`. Arguments: params (`Iterable[nn.parameter.Parameter]`): Iterable of parameters to optimize or dictionaries defining parameter groups. lr (`float`, *optional*): The external learning rate. eps (`tuple[float, float]`, *optional*, defaults to `(1e-30, 0.001)`): Regularization constants for square gradient and parameter scale respectively clip_threshold (`float`, *optional*, defaults to 1.0): Threshold of root mean square of final gradient update decay_rate (`float`, *optional*, defaults to -0.8): Coefficient used to compute running averages of square beta1 (`float`, *optional*): Coefficient used for computing running averages of gradient weight_decay (`float`, *optional*, defaults to 0.0): Weight decay (L2 penalty) scale_parameter (`bool`, *optional*, defaults to `True`): If True, learning rate is scaled by root mean square relative_step (`bool`, *optional*, defaults to `True`): If True, time-dependent learning rate is computed instead of external learning rate warmup_init (`bool`, *optional*, defaults to `False`): Time-dependent learning rate computation depends on whether warm-up initialization is being used This implementation handles low-precision (FP16, bfloat) values, but we have not thoroughly tested. Recommended T5 finetuning settings (https://discuss.huggingface.co/t/t5-finetuning-tips/684/3): - Training without LR warmup or clip_threshold is not recommended. - use scheduled LR warm-up to fixed LR - use clip_threshold=1.0 (https://huggingface.co/papers/1804.04235) - Disable relative updates - Use scale_parameter=False - Additional optimizer operations like gradient clipping should not be used alongside Adafactor Example: ```python Adafactor(model.parameters(), scale_parameter=False, relative_step=False, warmup_init=False, lr=1e-3) ``` Others reported the following combination to work well: ```python Adafactor(model.parameters(), scale_parameter=True, relative_step=True, warmup_init=True, lr=None) ``` When using `lr=None` with [`Trainer`] you will most likely need to use [`~optimization.AdafactorSchedule`] scheduler as following: ```python from transformers.optimization import Adafactor, AdafactorSchedule optimizer = Adafactor(model.parameters(), scale_parameter=True, relative_step=True, warmup_init=True, lr=None) lr_scheduler = AdafactorSchedule(optimizer) trainer = Trainer(..., optimizers=(optimizer, lr_scheduler)) ``` Usage: ```python # replace AdamW with Adafactor optimizer = Adafactor( model.parameters(), lr=1e-3, eps=(1e-30, 1e-3), clip_threshold=1.0, decay_rate=-0.8, beta1=None, weight_decay=0.0, relative_step=False, scale_parameter=False, warmup_init=False, ) ```c  <VeV 'd \R4hV 'dV 'g \R4hRVRVRVRVRVRVR VR V R V / p \S V` W4R#) Nz;Cannot combine manual `lr` and `relative_step=True` optionsz0`warmup_init=True` requires `relative_step=True`r[rclip_threshold decay_ratebeta1 weight_decayscale_parameter relative_step warmup_init)r^superr) rparamsr[rr1r2r3r4r5r6r7r] __class__s &&&&&&&&&&& rrAdafactor.__init__vsn >mZ[ [ }OP P " 3 n * U L  ] ;   *rc DVR,pVR,'dPVR,'dRVR,,MRp\VR\P!VR,4, 4pRpVR,'d"\VR,^,VR ,4pWB,#) r[r6r7rrg{Gz?r(r5rRMS)rr?rdr+)r param_state rel_step_szmin_step param_scales&& r_get_lrAdafactor._get_lrs!$'  ' '5@5O5Otk&11UYHhdii F8K.L(LMK ( ) )k%03[5GHK((rc :\V4^8pVR,RJpW#3#)rr3N)r)r param_shapefactoreduse_first_moments&& r _get_optionsAdafactor._get_optionss*{#q(&w/t;))rc \VP^4VP4R,, #)rr<)normnumel)tensors&r_rmsAdafactor._rmss {{1~3!677rc WPRRR7, P4PR4pVPR4P4p\P !W#4#)rT)dimkeepdim)meanrsqrt_ unsqueezersqrttorchmul)exp_avg_sq_rowexp_avg_sq_colr_factorc_factors&& r_approx_sq_gradAdafactor._approx_sq_gradsY#%8%8R%8%NNVVXbbcef!++B/557yy,,rc RpVeV!4pVPEFpVR,EFpVPfKVPpVP\P\P 09dVP 4pVP'd \R4hVPV,pVPpVPW74wr\V4^8Xd^VR&V 'd\P!V4VR&V'db\P!VRR4PV4VR&\P!VRRVRR,4PV4VR&M\P!V4VR&^VR &M}V 'dVR,PV4VR&V'd8VR,PV4VR&VR,PV4VR&MVR,PV4VR&Tp VP\P\P 09dV P 4p VR;;,^, uu&VP!V 4VR &VP#W64p R \$P&!VR,VR ,4, p V^,VR ,^,,p V'dVR,pVR,pVP)V 4P+V P-RR 7R V , R7VP)V 4P+V P-RR 7R V , R7VP/W4p V P)V4MQVR,pVP)V 4P+V R V , R7VP14P)V4p V P3VP!V 4VR,, P5R R74V P)V 4V 'dCVR,pVP)VR,4P+V ^VR,, R7Tp VR,^8wd"V P+WR,)V ,R7V P+V )4VP\P\P 09gEKVP7V 4EK EK V#)z Performs a single optimization step Arguments: closure (callable, optional): A closure that reevaluates the model and returns the loss. Nr9z,Adafactor does not support sparse gradients.rexp_avgr[r\ exp_avg_sqr=r(r2r)rQ)alphar1)rr3r4rSrT)rgraddtyperYfloat16bfloat16r* is_sparse RuntimeErrorr shaperHr zeros_likezerostorNrBr?powmul_add_rUr_rXdiv_clamp_copy_)rclosurelossrprer  grad_shaperFrG p_data_fp32r[beta2tupdater[r\rcrbs&& rrAdafactor.steps  9D&&E8__66>vv::%--!@@::>>&'UVV 1 !ZZ -1->->u-Q*u:?$%E&M'+0+;+;D+Ai(27++j"o2N2Q2QRV2W./27++j"oPZ[][^P_>_2`2c2cdh2i./.3.>.>t.Dl+#$E%L'+0+;+>+>t+Di(278H2I2L2LT2R./278H2I2L2LT2R./.3L.A.D.DT.Jl+ 77u}}enn=="-"3"3"5Kf " #yy5e \\%/txxf u\7JKK'U5\!_4%*+;%))** 88-- ]]_[[rr/c6a]tRtRtRtRV3RlltRtRtV;t#)AdafactorScheduleia( Since [`~optimization.Adafactor`] performs its own scheduling, if the training loop relies on a scheduler (e.g., for logging), this class creates a proxy object that retrieves the current lr values from the optimizer. It returns `initial_lr` during startup and the actual `lr` during stepping. c <aV3RlpVPFpSVR&K \SV` W4VPFpVRK R#)c<S#rr)r initial_lrs&rr0-AdafactorSchedule.__init__..lr_lambdas  rrN)rr8r)rrrr0rr:s&&f rrAdafactorSchedule.__init__sI ++E",E, , .++El#,rc  .VPpVPUu.uFQpVR,^,PfK!VPW!PVR,^,,4NKS pp\ V4^8Xd VP pV#uupi)r9)rrrerBr rbase_lrs)roptrlrss& rget_lrAdafactorSchedule.get_lr#snn)) )Xq!&& >CKKyyx);< =)  s8q=--C  s B4Brr5) rrrrrrrrrrs@rrrs$  rrc\W4#)a4 Get a proxy schedule for [`~optimization.Adafactor`] Args: optimizer ([`~torch.optim.Optimizer`]): The optimizer for which to schedule the learning rate. initial_lr (`float`, *optional*, defaults to 0.0): Initial lr Return: [`~optimization.Adafactor`] proxy schedule object. )r)rrs&&rget_adafactor_scheduler/s Y 33rr)rS)r<rS)rrS)gHz>r(rS)NrS)r<rSNN)r<rSNNN)NNrrrr<rS)NNNr)Ar __future__rr?r functoolsrtypingrrY torch.optimrtorch.optim.lr_schedulerrrtrainer_pt_utilsrr trainer_utilsr utilsr get_loggerrloggerrrr#r,r1r6r8rDrIrLrOrYr_rfrirtrxr{rrrrrLINEARCOSINECOSINE_WITH_RESTARTS POLYNOMIALr'r*r+r(COSINE_WITH_MIN_LRCOSINE_WARMUP_WITH_MIN_LRr,r)r#r$r/rrrrrrs+"  !@N(   H % L"2$ A*u 66Z6D^6B*+6\nrA<ru16h  $(256p#LF6R%.%.Pp'p'f )"99&&(ZG1&&(I 9##%C$$&Q++-`%%'7-  ^Bm m`<4r