i dddlZddlmZmZddlmZmZddlmZddl m Z m Z m Z m Z mZddlZddlmZddlmZddlmZddlmZdd lmZdd lmZdd lmZdd lmZdd l m!Z!ddl"m#Z#m$Z$ddl%m&Z&ddl'm(Z(m)Z)ddl*m+Z+ddl,m-Z-ddl.m/Z/ddl0m1Z1ddl2m3Z3ddl4m5Z5m6Z6e dZ7e dZ8ejre:Z;GddeZ<GddZ=y)N)ABCabstractmethod) GeneratorMapping)contextmanager)AnyCallableOptionalTypeVarUnion)Tensor)Module) Optimizer) CheckpointIO)_StrategyRegistry)move_data_to_device)ReduceOp) _EmptyInit)_optimizer_to_device_optimizers_to_device)_PATH)LightningOptimizer"_init_optimizers_and_lr_schedulers)TorchCheckpointIO)_WrappingCheckpointIO) Precision) _Launcher) TrainerFn) STEP_OUTPUTLRSchedulerConfig TBroadcastTReducec eZdZdZ dYdeddeedeeddfdZedee fd Z ededfd Z e jdod Z edefd Z e jd eddfdZ edefdZejdeeddfdZedeefdZejdeeddfdZdpdZdqdZdqdZdrdZdrdZdqdZdedeeeffdZdedeeded edef d!Z dsded"egefdeede fd edef d#Z!de deede"e eeffd$Z#de de fd%Z$dedefd&Z%dtd'ed(ee&jNd)e(defd*Z)ee*de&jNfd+Z+e*dqd,Z,ee*de-fd-Z.e* dud.eeefd/eed0eee/efdeeeffd1Z0e*dsd2eeddfd3Z1e*dvd4e2d5e(de2fd6Z3e*dwd.ed/eed7e-defd8Z4dxd9e-d:e-de-fd;Z5deddfd<Z6deddfd=Z7edee fd>Z8e8jd?ee ddfd@Z8ededfdAZ9dsdBe:dCee-deeeffdDZ;dxdEeded ede?fdIZ@dqdJZAded ede?fdKZBded ede?fdLZCded edefdMZDdNeEdeEfdOZFede-fdPZGede-fdQZHede-fdRZIdeeeffdSZJ dsdEeeefdTe:dUeeddfdVZKdTe:ddfdWZLeMdsdXee-deNdYfdZZOeMdeNdYfd[ZPdqd\ZQeRd]eSddfd^ZTdqd_ZUdqd`ZVdqdaZWdqdbZXdqdcZYdqddZZdqdeZ[dqdfZ\d'edge(ddfdhZ]die^ddfdjZ_dqdkZ`defdlZadmeddfdnZby)yStrategyzcBase class for all strategies that change the behaviour of the training, validation and test- loop.N acceleratorpl.accelerators.Accelerator checkpoint_ioprecision_pluginreturnc||_||_d|_||_d|_d|_d|_t|_g|_ g|_ g|_ yN) _accelerator_checkpoint_io_precision_pluginr(_lightning_module_model _launcher_ForwardRedirection_forward_redirection _optimizers_lightning_optimizerslr_scheduler_configs)selfr%r'r(s z/Volumes/fast/ai/experiments/voice-extract-mac/.venv/lib/python3.12/site-packages/lightning/pytorch/strategies/strategy.py__init__zStrategy.__init__2s_ DO6C6: 0?C(, .29L9N!,.?A"=?!c|jSr+)r1r7s r8launcherzStrategy.launcherEs ~~r:c|jSr+r,r<s r8r%zStrategy.acceleratorIs   r:c||_yr+r?)r7r%s r8r%zStrategy.acceleratorMs 'r:c|jt|_|jSt|jtrt|j_|jSr+)r-r isinstancerr'r<s r8r'zStrategy.checkpoint_ioQsW    &"3"5D """++-B C0A0CD   -"""r:ioc||_yr+)r-)r7rCs r8r'zStrategy.checkpoint_ioZs  r:cF|j |jStSr+)r.rr<s r8r(zStrategy.precision_plugin^s")-)?)?)Kt%%\QZQ\\r:c||_yr+)r.)r7r(s r8r(zStrategy.precision_pluginbs !1r:c|jSr+)r4r<s r8 optimizerszStrategy.optimizersfsr:rHcn||_|Dcgc]}tj||c}|_ycc}wr+)r4r_to_lightning_optimizerr5)r7rHopts r8rHzStrategy.optimizersjs7%gq%rgq`c&8&P&PQTVZ&[gq%r"%rs2modelpl.LightningModulec ||_||_y)z=Called by the Trainer to connect the strategy with the model.N)r/rLr7rLs r8connectzStrategy.connectos"' r:cy)z&Attach the launcher based on Strategy.Nr<s r8_configure_launcherzStrategy._configure_launchervr:cj|jJ|jj|jy)zSetup any processes or distributed connections. This is called before the LightningModule/DataModule setup hook which allows the user to access the accelerator environment before setup is complete. N)r% setup_device root_devicer<s r8setup_environmentzStrategy.setup_environmentys0+++ %%d&6&67r:cd|jJt|j\|_|_y)zCreates optimizers and schedulers. Args: trainer: the Trainer, these optimizers should be connected to N)lightning_modulerrHr6r7trainers r8setup_optimizerszStrategy.setup_optimizerss1$$0005WX\XmXm5n22r:cF|jJ|jj||jJ|jj |j|_|j |j |j|_|jjtjk(r|j||j|jjtjk(r!t|j|jyy)zSets up the accelerator, plugins and initializes the optimizers (if needed). Args: trainer: the trainer instance N)r%setuprLr(convert_modulemodel_to_device _setup_modelstatefnrFITTINGr]setup_precision_pluginrrHrWr[s r8r_zStrategy.setups+++ w'zz%%%**99$**E  &&tzz2 ==  y00 0  ! !' * ##% ==  y00 0 !$//43C3C D 1r:c|jJ|jj|j|j|j\}}}||_||_||_y)z.Attaches the precision plugin to the strategy.N)rLr(rPrHr6)r7rLrHr6s r8rfzStrategy.setup_precision_plugins]zz%%%262G2G2O2O JJ)B)B3 /z/ $$8!r: optimizerct|tr |j}t|dr.|j |j r|j SiS|j S)zReturns state of an optimizer. Allows for syncing/collating optimizer state from processes in custom strategies. consolidate_state_dict)rBr _optimizerhasattrrjis_global_zero state_dictr7rhs r8optimizer_statezStrategy.optimizer_states` i!3 4!,,I 96 7  , , .-1-@-@9'') Hb H##%%r: closure_lossargskwargscV|j||jJ|jj||j}|jj||j|g|i||jj ||j}|j ||S)a'Forwards backward-calls to the precision plugin. Args: closure_loss: a tensor holding the loss value to backpropagate optimizer: An optional optimizer that gets passed down to the precision plugin's backward \*args: Positional arguments that get passed down to the precision plugin's backward, intended as arguments for the actual function that performs the backward, like :meth:`~torch.Tensor.backward`. \**kwargs: Keyword arguments for the same purpose as ``*args``. ) pre_backwardrZr(backward post_backward)r7rqrhrrrss r8rvzStrategy.backwards" ,'$$000,,99,H]H]^ &&&|T5J5JIgX\g`fg,,::<I^I^_  <(r:closurec |xs |j}t|tjsJ|jj |f||d|S)aHPerforms the actual optimizer step. Args: optimizer: the optimizer performing the step closure: closure calculating the loss value model: reference to the model, optionally defining optimizer step related hooks \**kwargs: Keyword arguments to ``optimizer.step`` )rLrx)rZrBplLightningModuler(optimizer_step)r7rhrxrLrss r8r|zStrategy.optimizer_stepsQ ...%!3!34443t$$33IfUT[f_effr:cr|j|}|Dcgc]}|j|}}||fScc}w)zSetup a model and multiple optimizers together. The returned objects are expected to be in the same order they were passed in. The default implementation will call :meth:`_setup_model` and :meth:`_setup_optimizer` on the inputs. )rb_setup_optimizer)r7rLrHrhs r8_setup_model_and_optimizersz$Strategy._setup_model_and_optimizerssD!!%(HRS 9d++I6 Sj  Ts4c|S)zDPerforms setup for the model, e.g., by wrapping it by another class.rRrOs r8rbzStrategy._setup_models  r:c|S)zHPerforms setup for the optimizer, e.g., by wrapping it by another class.rRros r8r~zStrategy._setup_optimizers r:batchdevicedataloader_idxc~|j}|xs |j}||j|||St||S)azMoves the batch to the correct device. The returned batch is of the same type as the input batch, just having all tensors on the correct device. Args: batch: The batch of samples to move to the correct device device: The target device dataloader_idx: The index of the dataloader to which the batch belongs. )rr)rZrW_apply_batch_transfer_handlerr)r7rrrrLs r8batch_to_devicezStrategy.batch_to_devicesK%%+4++  66uV\j6k k"5&11r:cy)zReturns the root device.NrRr<s r8rWzStrategy.root_devicerTr:cy)z&Moves the model to the correct device.NrRr<s r8razStrategy.model_to_devicerTr:cy)zcWhether the current process is the rank zero process not only on the local node, but for all nodes.NrRr<s r8rmzStrategy.is_global_zero"rTr:tensorgroup reduce_opcy)a,Reduces the given tensor (e.g. across GPUs/processes). Args: tensor: the tensor to sync and reduce group: the process group to reduce reduce_op: the reduction operation. Defaults to 'mean'. Can also be a string 'sum' or ReduceOp. NrR)r7rrrs r8reducezStrategy.reduce'rTr:namecy)zSynchronizes all processes which blocks processes until the whole group enters this function. Args: name: an optional name to pass into barrier. NrR)r7rs r8barrierzStrategy.barrier8rTr:objsrccy)zBroadcasts an object to all processes. Args: obj: the object to broadcast src: source rank NrR)r7rrs r8 broadcastzStrategy.broadcastArTr: sync_gradscy)aPerform an all_gather on all processes. Args: tensor: the tensor to all_gather group: the process group to gather results from sync_grads: flag that allows users to synchronize gradients for all_gather op NrR)r7rrrs r8 all_gatherzStrategy.all_gatherKrTr:decisionallc|S)z/Reduce a boolean decision across all processes.rR)r7rrs r8reduce_boolean_decisionz Strategy.reduce_boolean_decisionVsr:cy)z.Run before precision plugin executes backward.NrRr7rqs r8ruzStrategy.pre_backwardZrTr:cy)z-Run after precision plugin executes backward.NrRrs r8rwzStrategy.post_backward]rTr:cJ|j |jS|jS)z0Returns the potentially wrapped LightningModule.)r0r/r<s r8rLzStrategy.model`s"#kk5t{{Q4;Q;QQr: new_modelc||_yr+)r0)r7rs r8rLzStrategy.modeles  r:c|jS)zWhether the strategy handles gradient accumulation internally.FrRr<s r8handles_gradient_accumulationz&Strategy.handles_gradient_accumulationsr:cR|jJ|jjS)zReturns model state.)rZrnr<s r8lightning_module_state_dictz$Strategy.lightning_module_state_dicts)$$000$$//11r:filepathstorage_optionscZ|jr|jj|||yy)a?Save model/training states as a checkpoint file through state-dump and file-write. Args: checkpoint: dict containing model and trainer state filepath: write-target file's path storage_options: parameter for how to save to storage, passed to ``CheckpointIO`` plugin )rN)rmr'save_checkpoint)r7rrrs r8rzStrategy.save_checkpoints.       . .z8Ud . e r:cT|jr|jj|yy)zqRemove checkpoint filepath from the filesystem. Args: filepath: Path to checkpoint N)rmr'remove_checkpoint)r7rs r8rzStrategy.remove_checkpoints&       0 0 : r: empty_init)NNNc# Ktt|}|5|j5|jj 5ddddddddddy#1swYxYw#1swYxYw#1swYyxYww)aControls how tensors get created (device, dtype). Args: empty_init: Whether to initialize the model with empty weights (uninitialized memory). If ``None``, the strategy will decide. Some strategies may not support all options. )enabledN)rboolrWr(tensor_init_context)r7rempty_init_contexts r8rzStrategy.tensor_init_contextsZ(Z0@A !1!143H3H3\3\3^ 4_!1  3^3^!1!1  sKB A8A,A A,A8 B A) %A,,A5 1A88B=Bc#Kdyw)aProvide hook to create modules in a distributed aware context. This is useful for when we'd like to shard the model instantly, which is useful for extremely large models which can save memory and initialization time. Returns: Model parallel context. NrRr<s r8model_sharded_contextzStrategy.model_sharded_contexts  sct|jtjd|jFt j |jjd|jj|jj|jJ|jj|jjy)zThis method is called to teardown the training process. It is the right place to release memory and free other resources. cpuNz: moving model to CPU)rrHrrrZlogdebug __class____name__rr(teardownr%r'r<s r8rzStrategy.teardowns doou||E/BC  , II0011FG H  ! ! % % ' &&(+++ !!# ##%r:strategy_registrycyr+rR)clsrs r8register_strategieszStrategy.register_strategies r:cy)zCalled when train begins.NrRr<s r8on_train_startzStrategy.on_train_start"rr:cy)zCalled when validation begins.NrRr<s r8on_validation_startzStrategy.on_validation_start&rr:cy)zCalled when test begins.NrRr<s r8 on_test_startzStrategy.on_test_start*rr:cy)zCalled when predict begins.NrRr<s r8on_predict_startzStrategy.on_predict_start.rr:cy)zCalled when train ends.NrRr<s r8 on_train_endzStrategy.on_train_end2rr:cy)zCalled when validation ends.NrRr<s r8on_validation_endzStrategy.on_validation_end6rr:cy)zCalled when test end.NrRr<s r8 on_test_endzStrategy.on_test_end:rr:cy)zCalled when predict ends.NrRr<s r8on_predict_endzStrategy.on_predict_end>rr: batch_idxcy)zCCalled in the training loop before anything happens for that batch.NrR)r7rrs r8on_train_batch_startzStrategy.on_train_batch_startBrr: exceptioncy)zACalled when the trainer execution is interrupted by an exception.NrR)r7rs r8 on_exceptionzStrategy.on_exceptionFrr:c.g|_g|_g|_yr+)r4r5r6r<s r8 _reset_optimizers_and_schedulersz)Strategy._reset_optimizers_and_schedulersJs%'"$&!r:c8tt|}g|d<|S)Nr5)dictvarsr7rcs r8 __getstate__zStrategy.__getstate__Os T$Z )+%& r:rcc4||_|j|_yr+)__dict__rHrs r8 __setstate__zStrategy.__setstate__Us //r:)r%r&r)N)rLrMr)N)r)N)r\z pl.Trainerr)Nr+)Nr)Nmean)r)NF)T)cr __module__ __qualname____doc__r rrr9propertyrr=r%setterr'r(listrrHrPrSrXr]r_rfrstrr rprrvr r rr|tuplerrbr~rrintrrrWrarrmrrrr!rrrrurwrLrZrrrrrrrrrrrobjectrrrrrrrrrrrr classmethodrrrrrrrrrrr BaseExceptionrrrr rRr:r8r$r$/sm@D0404 @;<@ -@#9- @  @&(9-!X&CD!!((#|##! !!!])]]2)1D222 DO  sT)_sss58oE.9&&tCK7H&$I&     @@D gg"c'"g2F:;< g  g g* ! !T)_ !Y^_egklugv_vYw !&V )  2S2(5<<2H2ad2mp2$'U\\''55rrr $4:  fck" } E(C-01  vs{     HSM T   Z c *      RV ci  44==D=<&GCH4E>$> H3 H# H+ H  JS JC JK J Ds Dc Dk D G# G G GVTt2T#s(^2 ]a fsCx. f49 fLTUXM f  f;%;D; htn  RbHc  y1A'B&  4E $           # # $  m  ' d *$*4*r:r$c XeZdZdZdedddedededef d Zdedddd fd Z dedddd fd Z y ) r2zImplements the `forward-redirection`. A method call to a wrapped module gets rerouted through the wrapper's `forward` method instead. wrapper_moduleoriginal_modulerM method_namerrrsr)cdk7sJjdtdtdtffd }|_|i|}j|S)aReroutes a method call through the `wrapper_module`'s `forward` method. Args: wrapper_module: The module that has `original_module` wrapped. original_module: The module that was wrapped inside `wrapper_module`. method_name: The name of the method that should be called on the `original_module` after inputs get redirected through the `wrapper_module`'s `forward` method. *args: The positional arguments to the method `method_name`. They will get passed to a patched `forward` method instead. **kwargs: The keyword arguments to the method `method_name`. They will get passed to a patched `forward` method instead. forward_args_kwargsr)cb_t}||i|}j|Sr+)rgetattron_after_inner_forward) rrmethodoutroriginal_forwardrr7rs r8wrapped_forwardz5_ForwardRedirection.__call__..wrapped_forwardts='7O #_k:F%+7+C  ' ' HJr:)rron_after_outer_forward) r7rrrrrrsr&wrapper_outputr%s ```` @r8__call__z_ForwardRedirection.__call__asm i'''*22 C C C  #2'88 ##NODr:Ncyr+rRr7rrs r8r"z*_ForwardRedirection.on_after_inner_forward r:cyr+rRr+s r8r'z*_ForwardRedirection.on_after_outer_forwardr,r:) rr r rrrrrr)r"r'rRr:r8r2r2Zsu "$"7K"Z]"fi"ux" "H V Nb gk  V Nb gk r:r2)>loggingabcrrcollections.abcrr contextlibrtypingrr r r r rr torch.nnr torch.optimrlightning.pytorchpytorchrzlightning.fabric.pluginsrlightning.fabric.strategiesrlightning.fabric.utilitiesr&lightning.fabric.utilities.distributedrlightning.fabric.utilities.initr$lightning.fabric.utilities.optimizerrr lightning.fabric.utilities.typesr lightning.pytorch.core.optimizerrrlightning.pytorch.pluginsr$lightning.pytorch.plugins.io.wrapperr#lightning.pytorch.plugins.precisionr/lightning.pytorch.strategies.launchers.launcherr lightning.pytorch.trainer.statesr!lightning.pytorch.utilities.typesrr r!r" getLoggerrrr$r2rRr:r8rFs#.%:: !19:;6\2c7F9E6L \ " ) g!h*sh*V/ / r: