iq/ ddlmZmZmZmZddlZddlmZddl m Z ddl m Z m Z erddlmZGddZ dd eee d fd ee d ee d edddf dZddZddZy)) TYPE_CHECKINGLiteralOptionalUnionN)Callback)MisconfigurationException)EVAL_DATALOADERSTRAIN_DATALOADERS) _LRFinderc.eZdZdZddZ ddddeeedfd eed eed edd e d de de de de de de de dee fdZ d dddeeedfd eed eed edd e d de de de de dee dede dedfdZy)!TunerzTuner class to tune your model.returnNc||_y)N)_trainer)selftrainers s/Volumes/fast/ai/experiments/voice-extract-mac/.venv/lib/python3.12/site-packages/pytorch_lightning/tuner/tuning.py__init__zTuner.__init__s  modelzpl.LightningModuletrain_dataloaderspl.LightningDataModuleval_dataloaders dataloaders datamodulemethodfitvalidatetestpredictmodesteps_per_trialinit_val max_trialsbatch_arg_namemarginmax_valc t||||t|jd| cxkrdks nJd| ddlm}|||| | | | | }d|_|g|jj z|j_|dk(r|jj||||nk|d k(r|jj||| nG|d k(r|jj||| n#|d k(r|jj||| |jj Dcgc] }||us| c}|j_|jScc}w) ad Iteratively try to find the largest batch size for a given model that does not give an out of memory (OOM) error. Args: model: Model to tune. train_dataloaders: A collection of :class:`torch.utils.data.DataLoader` or a :class:`~pytorch_lightning.core.datamodule.LightningDataModule` specifying training samples. In the case of multiple dataloaders, please see this :ref:`section `. val_dataloaders: A :class:`torch.utils.data.DataLoader` or a sequence of them specifying validation samples. dataloaders: A :class:`torch.utils.data.DataLoader` or a sequence of them specifying val/test/predict samples used for running tuner on validation/testing/prediction. datamodule: An instance of :class:`~pytorch_lightning.core.datamodule.LightningDataModule`. method: Method to run tuner on. It can be any of ``("fit", "validate", "test", "predict")``. mode: Search strategy to update the batch size: - ``'power'``: Keep multiplying the batch size by 2, until we get an OOM error. - ``'binsearch'``: Initially keep multiplying by 2 and after encountering an OOM error do a binary search between the last successful batch size and the batch size that failed. steps_per_trial: number of steps to run with a given batch size. Ideally 1 should be enough to test if an OOM error occurs, however in practise a few are needed init_val: initial batch size to start the search with max_trials: max number of increases in batch size done before algorithm is terminated batch_arg_name: name of the attribute that stores the batch size. It is expected that the user has provided a model or datamodule that has a hyperparameter with that name. We will look for this attribute name in the following places - ``model`` - ``model.hparams`` - ``trainer.datamodule`` (the datamodule passed to the tune method) margin: Margin to reduce the found batch size by to provide a safety buffer. Only applied when using 'binsearch' mode. Should be a float between 0 and 1. Defaults to 0.05 (5% reduction). max_val: Maximum batch size limit, defaults to 8192. Helps prevent testing unrealistically large or inefficient batch sizes (e.g., 2**25) when running on CPU or when automatic OOM detection is not available. gg?z1`margin` should be between 0 and 1. Found margin=rBatchSizeFinder)r"r#r$r%r&r'r(Trr)rr r!) _check_tuner_configuration%_check_scale_batch_size_configurationr-pytorch_lightning.callbacks.batch_size_finderr+ _early_exit callbacksrrr r!optimal_batch_size)rrrrrrrr"r#r$r%r&r'r(r+batch_size_findercbs rscale_batch_sizezTuner.scale_batch_sizesVp ##4o{TZ[-dmm<f"s"Z&XQWPY$ZZ" R&5+!)' )-%#4"5 8O8O"O  U? MM  e%6 T z ! MM " "5+* " M v  MM  ukj  I y MM ! !% ! L04 0G0G"g0G"2UfKf20G"g  333#hs ' E1Emin_lrmax_lr num_trainingearly_stop_threshold update_attr attr_namer c |dk7r tdt||||t|jddlm}|||| | | | | }d|_|g|jjz|j_|jj|||||jjDcgc] }||us| c}|j_|jScc}w)aXEnables the user to do a range test of good initial learning rates, to reduce the amount of guesswork in picking a good starting learning rate. Args: model: Model to tune. train_dataloaders: A collection of :class:`torch.utils.data.DataLoader` or a :class:`~pytorch_lightning.core.datamodule.LightningDataModule` specifying training samples. In the case of multiple dataloaders, please see this :ref:`section `. val_dataloaders: A :class:`torch.utils.data.DataLoader` or a sequence of them specifying validation samples. dataloaders: A :class:`torch.utils.data.DataLoader` or a sequence of them specifying val/test/predict samples used for running tuner on validation/testing/prediction. datamodule: An instance of :class:`~pytorch_lightning.core.datamodule.LightningDataModule`. method: Method to run tuner on. It can be any of ``("fit", "validate", "test", "predict")``. min_lr: minimum learning rate to investigate max_lr: maximum learning rate to investigate num_training: number of learning rates to test mode: Search strategy to update learning rate after each batch: - ``'exponential'``: Increases the learning rate exponentially. - ``'linear'``: Increases the learning rate linearly. early_stop_threshold: Threshold for stopping the search. If the loss at any point is larger than early_stop_threshold*best_loss then the search is stopped. To disable, set to None. update_attr: Whether to update the learning rate attribute or not. attr_name: Name of the attribute which stores the learning rate. The names 'learning_rate' or 'lr' get automatically detected. Otherwise, set the name here. Raises: MisconfigurationException: If learning rate/lr in ``model`` or ``model.hparams`` isn't overridden, or if you are using more than one optimizer. rz>method='fit' is the only valid configuration to run lr finder.rLearningRateFinder)r5r6num_training_stepsr"r8r9r:T) rr,_check_lr_find_configurationr%pytorch_lightning.callbacks.lr_finderr=r/r0r optimal_lr)rrrrrrrr5r6r7r"r8r9r:r=lr_finder_callbackr3s rlr_findz Tuner.lr_findwsd U?+,lm m"#4o{TZ[$T]]3 M'9+!5#( *."69P9P"P  %!2OZP04 0G0G"h0G"2UgKg20G"h !,,,#is / C9Crz pl.TrainerrN) NNNNrpower batch_sizeg?i ) NNNNrg:0yE>d exponentialg@T)__name__ __module__ __qualname____doc__rrrr r rstrintfloatr4boolrCrrr r s) [_6:269=@E *V4#V4$E*;=U*U$VWV4""23 V4 ./ V4 56 V4<=V4V4V4V4V4V4V4V4 #V4v[_6:269=@E!03 L-#L-$E*;=U*U$VWL-""23 L- ./ L- 56 L-<=L-L-L-L-L-'uoL-L-L- + L-rr rrrrrrrcd}||vrtd|d|d|dk(r|td|dy||td|d y) Nrzmethod z is invalid. Should be one of .rzIn tuner with method=zs, `dataloaders` argument should be None, please consider setting `train_dataloaders` and `val_dataloaders` instead.zIn tuner with `method`=zt, `train_dataloaders` and `val_dataloaders` arguments should be None, please consider setting `dataloaders` instead.) ValueErrorr)rrrrsupported_methodss rr,r,s ? &&76*,JK\J]]^_``   "+'z2^^  #  (O,G+)&4\\ -Hrcddlm}|jDcgc]}t||s|}}|r t dycc}w)Nrr<zqTrainer is already configured with a `LearningRateFinder` callback.Please remove it if you want to use the Tuner.)r@r=r0 isinstancerY)rr=r3configured_callbackss rr?r?sLH)0):):a):2jM_>`B):a =  bs;;c|jjr tdddlm}|j Dcgc]}t ||s|}}|r tdycc}w)NzMTuning the batch size is currently not supported with distributed strategies.rr*znTrainer is already configured with a `BatchSizeFinder` callback.Please remove it if you want to use the Tuner.)_accelerator_connectoris_distributedrYr.r+r0r\)rr+r3r]s rr-r-si%%44hiiN)0):):^):2j_>]B):^ =  _s AA)NNNrrD)typingrrrrpytorch_lightningpl$pytorch_lightning.callbacks.callbackr&pytorch_lightning.utilities.exceptionsr!pytorch_lightning.utilities.typesr r !pytorch_lightning.tuner.lr_finderr r r,r?r-rVrrrhs;:9LQ;j-j-\W[26.2