iJ ddlZddlmZmZddlmZddlmZmZddl m Z m Z m Z m Z mZmZddlZddlmZddlmZddlmZdd lmZdd lmZdd lmZdd lmZdd lmZddl m!Z!ddl"m#Z#ddl$m%Z%ddl&m'Z'ddl(m)Z)m*Z*m+Z+m,Z,e rddl-m.Z.edZ/edZ0ejbe2Z3GddeZ4GddeZ5GddeZ6dee7dee7de8ddfd Z9d!e7d"e:e7e e7e ge8ffd#e;d$e:e7e fddf d%Zeddfd?Z:dede6e.e'ffd@Z; dVd7e5d8ee,eee6e.e,eee%fffd>edAeede6e.e%ff dBZr1cy)zcWhether the current process is the rank zero process not only on the local node, but for all nodes.Nr3r4s r/is_global_zerozStrategy.is_global_zeroCr6r1c|jSr()r,r4s r/launcherzStrategy.launcherHs ~~r1c|jSr(r)r4s r/r#zStrategy.acceleratorLs   r1c||_yr(r<)r.r#s r/r#zStrategy.acceleratorPs 'r1cP|jt|_|jSr()r*rr4s r/r$zStrategy.checkpoint_ioTs%    &"3"5D """r1ioc||_yr()r*)r.r?s r/r$zStrategy.checkpoint_ioZs  r1cF|j |jStSr()r+rr4s r/r%zStrategy.precision^s"&//"=tN9;Nr1c||_yr()r+)r.r%s r/r%zStrategy.precisionbs #r1cy)z&Attach the launcher based on Strategy.Nr3r4s r/_configure_launcherzStrategy._configure_launcherfr6r1cj|jJ|jj|jy)zSetup any processes or distributed connections. This must be called by the framework at the beginning of every process, before any distributed communication takes place. N)r# setup_devicer5r4s r/setup_environmentzStrategy.setup_environmentis0+++ %%d&6&67r1 dataloaderc|S)zWraps the dataloader if necessary. Args: dataloader: iterable. Ideally of type: :class:`torch.utils.data.DataLoader` r3)r.rHs r/process_dataloaderzStrategy.process_dataloaderss r1c|jj}t}|j|j|j||S)z1Controls how tensors get created (device, dtype).)r%tensor_init_contextr enter_contextr5)r.precision_init_ctxstacks r/rLzStrategy.tensor_init_context|sD!^^??A  D,,- ./ r1 empty_initc|jj}t}|j|j|jt t ||j||S)aA context manager wrapping the model instantiation. Here, the strategy can control how the parameters of the model get created (device, dtype) and or apply other patches to the model. Args: empty_init: Whether to initialize the model with empty weights (uninitialized memory). If ``None``, the strategy will decide. Some strategies may not support all options. )enabled)r%module_init_contextrrMr5rbool)r.rPprecision_module_ctxrOs r/rSzStrategy.module_init_contexts^ $~~AAC  D,,- JtJ/?@A 01 r1module optimizers schedulerrct|j|}|Dcgc]}|j|}}|||fScc}w)zSet up a model and multiple optimizers together. The returned objects are expected to be in the same order they were passed in. The default implementation will call :meth:`setup_module` and :meth:`setup_optimizer` on the inputs. ) setup_modulesetup_optimizer)r.rVrWrX optimizers r/setup_module_and_optimizersz$Strategy.setup_module_and_optimizerssF""6*GQRz)d**95z Rz9,,Ss5c|S)zDPerforms setup for the model, e.g., by wrapping it by another class.r3r.rVs r/rZzStrategy.setup_modules r1r\c|S)zHPerforms setup for the optimizer, e.g., by wrapping it by another class.r3r.r\s r/r[zStrategy.setup_optimizersr1cy)z&Moves the model to the correct device.Nr3r_s r/module_to_devicezStrategy.module_to_devicer6r1batchdevicec:|xs |j}t||S)a&Moves the batch to the correct device. The returned batch is of the same type as the input batch, just having all tensors on the correct device. Args: batch: The batch of samples to move to the correct device device: The target device )r5r)r.rdres r/batch_to_devicezStrategy.batch_to_devices!+4++"5&11r1tensorargskwargsc|jj|||jj||g|i||jj||y)z0Forwards backward-calls to the precision plugin.N)r% pre_backwardbackward post_backward)r.rhrVrirjs r/rmzStrategy.backwardsJ ##FF3@@@ $$VV4r1c <|jj|fi|S)zPerforms the actual optimizer step. Args: optimizer: the optimizer performing the step **kwargs: Any extra arguments to ``optimizer.step`` )r%optimizer_step)r.r\rjs r/rpzStrategy.optimizer_steps -t~~,,YA&AAr1group sync_gradscy)aPerform an all_gather on all processes. Args: tensor: the tensor to all_gather group: the process group to gather results from sync_grads: flag that allows users to synchronize gradients for all_gather op Nr3)r.rhrqrrs r/ all_gatherzStrategy.all_gatherr6r1 reduce_opcy)a,Reduces the given tensor (e.g. across GPUs/processes). Args: tensor: the tensor to sync and reduce group: the process group to reduce reduce_op: the reduction operation. Defaults to 'mean'. Can also be a string 'sum' or ReduceOp. Nr3)r.rhrqrus r/ all_reducezStrategy.all_reducer6r1namecy)zSynchronizes all processes which blocks processes until the whole group enters this function. Args: name: an optional name to pass into barrier. Nr3)r.rxs r/barrierzStrategy.barrierr6r1objsrccy)zBroadcasts an object to all processes. Args: obj: the object to broadcast src: source rank Nr3)r.r{r|s r/ broadcastzStrategy.broadcastr6r1decisionallc|S)z/Reduce a boolean decision across all processes.r3)r.rrs r/reduce_boolean_decisionz Strategy.reduce_boolean_decisionsr1pathstatestorage_optionsfilterc|j||xsi}|jr|jj|||yy)aSave model, optimizer, and other state as a checkpoint file. Args: path: A path to where the file(s) should be saved state: A dictionary with contents to be saved. If the dict contains modules or optimizers, their state-dict will be retrieved and converted automatically. storage_options: Additional options for the ``CheckpointIO`` plugin filter: An optional dictionary containing filter callables that return a boolean indicating whether the given item should be saved (``True``) or filtered out (``False``). Each filter key should match a state key, where its filter will be applied to the ``state_dict`` generated. )r) checkpointrrN)"_convert_stateful_objects_in_stater8r$save_checkpoint)r.rrrrs r/rzStrategy.save_checkpointsH&77v|QS7U       . .%d\k . l r1c"|jS)zReturns model state.) state_dictr_s r/get_module_state_dictzStrategy.get_module_state_dicts  ""r1rstrictc*|j||y)z%Loads the given state into the model.rN)load_state_dict)r.rVrrs r/load_module_state_dictzStrategy.load_module_state_dicts z&9r1ct|dr.|j|jr|jSiS|jS)zReturns state of an optimizer. Allows for syncing/collating optimizer state from processes in custom plugins. consolidate_state_dict)hasattrrr8rras r/get_optimizer_statezStrategy.get_optimizer_state%sJ 96 7  , , .-1-@-@9'') Hb H##%%r1 weights_onlyctjj|jj ||}|s|St |t r|j|||iSt |tr|j|iSt|j|j||jjD]\}}||vr t |trUt |t r$|j||j||O|j|j|p|j|||<|S)aLoad the contents from a checkpoint and restore the state of the given objects. Args: path: A path to where the file is located state: Can be one of: - A dictionary of objects whose state will be restored in-place from the checkpoint path. - ``None`` or the empty dict: The loaded checkpoint will be returned in full. - A :class:`~torch.nn.Module` instance, if the checkpoint file contains a raw module state dict. - A :class:`~torch.optim.Optimizer` instance, if the checkpoint file contains a raw optimizer state. strict: Whether to enforce that the keys in `state` match the keys in the checkpoint. weights_only: Defaults to ``None``. If ``True``, restricts loading to ``state_dicts`` of plain ``torch.Tensor`` and other primitive types. If loading a checkpoint from a trusted source that contains an ``nn.Module``, use ``weights_only=False``. If loading checkpoint from an untrusted source, we recommend using ``weights_only=True``. For more information, please refer to the `PyTorch Developer Notes on Serialization Semantics `_. Returns: The remaining items that were not restored into the given state dictionary. If no state dictionary is given, the full checkpoint will be returned. )r)rVrrr)torchcuda empty_cacher$load_checkpoint isinstancerrrr!_validate_keys_for_strict_loadingkeyscopyitemsrpop)r.rrrrrrxr{s r/rzStrategy.load_checkpoint4s!<   ''77<7X   eV $  ' 'uTZ ' [I eY '  ! !* -I)%**, 8IRXY++-ID#:%#y)c6*//sz~~VZG[dj/k'' t(<=(nnT2d .r1c|jj|jJ|jj|jjy)zThis method is called to teardown the training process. It is the right place to release memory and free other resources. N)r%teardownr#r$r4s r/rzStrategy.teardownlsJ !+++ !!# ##%r1max_norm norm_typeerror_if_nonfinitec|jj||jj|}tjj j ||||S)zClip gradients by norm.)rrr)r%unscale_gradients main_paramsrnnutilsclip_grad_norm_)r.rVr\rrr parameterss r/clip_gradients_normzStrategy.clip_gradients_normwsT ((3^^// : xx~~-- YSe.  r1clip_valc|jj||jj|}tjj j ||S)zClip gradients by value.) clip_value)r%rrrrrclip_grad_value_)r.rVr\rrs r/clip_gradients_valuezStrategy.clip_gradients_valuesF ((3^^// : xx~~..zh.OOr1strategy_registrycyr(r3)clsrs r/register_strategieszStrategy.register_strategiess r1c4dt|jdS)NzThe `z` does not support setting up the module and optimizer(s) independently. Please call `setup_module_and_optimizers(model, [optimizer, ...])` to jointly set them up.)type__name__r4s r/_err_msg_joint_setup_requiredz&Strategy._err_msg_joint_setup_requireds%DJ''()j j r1c(i}|jD]|\}}t|tr|j|}nFt|tr|j |}n#t|t r|j}n|}t||||~|S)N)rV)r\) rrrrrrrr _apply_filter)r.rrconverted_statekeyr{ converteds r/rz+Strategy._convert_stateful_objects_in_states+- HC#v& 66c6B C+ 44s4C C+NN,  #vy/ B&r1)NNN)r&Nr()NF)Nmean)r)T)NN)NTN)g@T)Gr __module__ __qualname____doc__r rrrr0propertyrrrer5rTr8rr:r#setterr$r%rDrGrrJrrLrSrlistrtupler]rZr[rcr rgrrmrrprtr rstrrwrzrintr~rrdictr rrrrrrrfloatrr classmethodrrrrr3r1r/r"r"-sm.204)- Kk* K - KI& K  K'U\\''rrr(9-!Xk2!!({(t((#|## ! !!!O9OO$8I#6$4$$58ZJ%;htnH^&bf - -*.y/ -FN~F^ - vtI(@@ A -6fy5v5$55 2S 2(5<<2H 2TW 25v5x/?55WZ5_c5 B B B B    RV ci   $4:  fck" } E(C-01  vs{     HSM T   Z c *  44*.BF mmCvy#5667m"# m c8S#J,<#==>? m  m.#F#tCsF{AS  r1rrequested_keyscheckpoint_keysrr&cd|Dcgc] }||vs| }}|r|rtd|ddyycc}w)Nz$The requested state contains a key 'rz^' that does not exist in the loaded checkpoint. To disable strict loading, set `strict=False`.)KeyError)rrrk invalid_keyss r/rrsU .J~!/1IA~LJ ,2<?2CD> ?  vKs --rr source_dict target_dictc||vrRt|trB||}|jD])\}}|||s|j|i||||<+y|||<yr()rrr setdefault)rrrr filter_fnrvs r/rrsh f}K63K %%'DAqA&&sB/&' C # (' Cr1)=loggingabcrrcollections.abcr contextlibrrtypingrr r r r r rrtorch.nnr torch.optimrtorch.utils.datarlightning_fabric.acceleratorsr)lightning_fabric.plugins.io.checkpoint_ior$lightning_fabric.plugins.io.torch_ior"lightning_fabric.plugins.precisionr.lightning_fabric.strategies.launchers.launcherr$lightning_fabric.strategies.registryr%lightning_fabric.utilities.apply_funcrlightning_fabric.utilities.initr lightning_fabric.utilities.typesrrrrtorch.optim.lr_schedulerrrr getLoggerrlogr"rrrrTrrobjectrr3r1r/rs#$8II !'5BB8DBE6TT5 \ " ) g!xsxv  3 $  s   SM 4