i PddlZddlZddlmZddlmZmZmZddlm Z ddl m Z ddl m Z ddlmZmZmZmZmZmZddlZddlmZdd lmZdd lmZdd lmZdd lmZdd l m!Z!m"Z"ddl#m$Z$ddl%m&Z&m'Z'm(Z(ddl)m*Z*ddl+m,Z,ddl-m.Z.ddl/m0Z0ddl1m2Z2ddl3m4Z4m5Z5m6Z6m7Z7m8Z8ddl9m:Z:m;Z;mZ>ddl9m?Z@ddlAmBZBmCZCddlDmEZEddlFmGZGmHZHmIZImJZJddlKmLZLmZmMZMddlNmOZOddlPmQZQmRZRerZod1ed,ed?fd@Zp dNd1edAe`dBe_d,ed?fdCZqdDe d,e_fdEZrdDe d,e_fdFZsd1etd,e!efdGZud1ejj:dHejd,dfdIZxdJeheyefdDe d,dfdKZzdLeheyefdDe d,dfdMZ{y)ON) Generator)AbstractContextManager ExitStack nullcontext) timedelta)partial)Path) TYPE_CHECKINGAnyCallableLiteralOptionalUnion)RequirementCache)rank_zero_only)Tensor)Module) Optimizer) TypeGuardoverride) Accelerator) CheckpointIOClusterEnvironment Precision)default_pg_timeout) FSDPPrecision)_SubprocessScriptLauncher)ParallelStrategy)_StrategyRegistry) TBroadcast _apply_filter_BackwardSyncControl_Sharded!_validate_keys_for_strict_loading)ReduceOp_distributed_is_initialized-_get_default_process_group_backend_for_device_init_dist_connection_sync_ddp_if_availablegroup)_TORCH_GREATER_EQUAL_2_2_TORCH_GREATER_EQUAL_2_3)&_has_meta_device_parameters_or_buffers)_METADATA_FILENAME _lazy_load_materialize_tensors_move_state_into)rank_zero_deprecationrrank_zero_warn) reset_seed)_PATH _Stateful) DeviceMesh) CPUOffloadMixedPrecisionShardingStrategyModuleWrapPolicy) _LRScheduler) FULL_SHARD SHARD_GRAD_OPNO_SHARD HYBRID_SHARD)fsdpfsdp_cpu_offloadignorez.*FSDP.state_dict_type.*)categorymessagec!eZdZdZdddddeddddddddfdeedeeejdee dee d ee d ee d eed dfd eddeddeeeeeeefdeddddeddeeeedfdeddf fd ZeedefdZej6ededdfdZeedejfdZedefd Zej6d!eddfd"Zedefd#Zeedee effd$Z edee fd%Z!ededfd&Z"eede#fd'Z$e$j6edee ddfd(Z$edQd)Z%edQfd* Z&e dRd+ed,ee'd-ed.deeee'ed.ffd/Z(ed+edefd0Z)ed1e'de'ffd2 Z*ed+eddfd3Z+edRd4eede,fd5Z-ede,fd6Z.e dSd7e/d8eed9eee0e fde/fd:Z1ed;ededdfd<Z2edTd=e3d>ede3fd?Z4e dUd+ed1e'd@ee5efdAee5efdBede/f dCZ6e dVdDe7dEee eee'effdFeedGeee e8e egeffddf dHZ9e dWdDe7dEeeee'ee eee'efffdIedJeedee eff dKZ:e;edLede fdOZ?dQdPZ@xZAS)X FSDPStrategyae Strategy for Fully Sharded Data Parallel provided by torch.distributed. Fully Sharded Training shards the entire model across all available GPUs, allowing you to scale model size, whilst using efficient communication to reduce overhead. In practice, this means we can remain at parity with PyTorch DDP, whilst scaling our model sizes dramatically. The technique is similar to ZeRO-Stage 3. For more information check out `this blogpost `__. Defaults have been set and options have been exposed, but may require configuration based on your level of memory/speed efficiency. We suggest having a look at `this tutorial `__ for more information. Arguments: cpu_offload: See ``cpu_offload`` parameter in :class:`torch.distributed.fsdp.FullyShardedDataParallel`. mixed_precision: See ``mixed_precision`` parameter in :class:`torch.distributed.fsdp.FullyShardedDataParallel`. auto_wrap_policy: Same as ``auto_wrap_policy`` parameter in :class:`torch.distributed.fsdp.FullyShardedDataParallel`. For convenience, this also accepts a set of the layer classes to wrap. activation_checkpointing: Deprecated. Use ``activation_checkpointing_policy``. activation_checkpointing_policy: Same as ``auto_wrap_policy`` parameter in :class:`torch.distributed.fsdp.FullyShardedDataParallel` but used when selecting the modules for which you want to enable activation checkpointing. Enabling this can free up a significant amount of memory at the cost of speed since activations in these layers need to be recomputed during backpropagation. For convenience, this also accepts a set of the layer classes to wrap. sharding_strategy: Select whether to shard model parameters, gradients, optimizer states, or a combination of them. Available values are: - ``"FULL_SHARD"``: Shards model parameters, gradients, and optimizer states (default). - ``"SHARD_GRAD_OP"``: Shards gradients and optimizer states only. Model parameters get replicated. - ``"NO_SHARD"``: No sharding (identical to regular DDP). - ``"HYBRID_SHARD"``: Shards model parameters, gradients, and optimizer states within a single machine, but replicates across machines. See also the `device_mesh` parameter below. Also accepts a :class:`torch.distributed.fsdp.ShardingStrategy` enum value. device_mesh: A tuple `(replication size, sharding size)` that defines over how many devices to shard and replicate the model. The product of the two numbers must equal the world size. Only valid in combination with the `HYBRID_SHARD` sharding strategy. state_dict_type: The format in which the state of the model and optimizers gets saved into the checkpoint. - ``"full"``: The full weights and optimizer states get assembled on rank 0 and saved to a single file. - ``"sharded"``: Each rank saves its shard of weights and optimizer states to a file. The checkpoint is a folder with as many files as the world size. \**kwargs: See available parameters in :class:`torch.distributed.fsdp.FullyShardedDataParallel`. Nr?sharded acceleratorparallel_devicescluster_environment precisionprocess_group_backendtimeout cpu_offloadr9mixed_precisionr:auto_wrap_policy_POLICYactivation_checkpointingactivation_checkpointing_policysharding_strategy_SHARDING_STRATEGYstate_dict_type)fullrJ device_meshr8kwargsreturnc t|||||d|_||_||_t |_t| ||_|jjdd| ts td||jd<t| | |_ | |_ t| |j|_t!||_||_y)N)rKrLrMrNuse_orig_paramsTz=The `device_mesh` argument is only supported in torch >= 2.2.r[)super__init__ _num_nodes_process_group_backend_timeout_FSDPBackwardSyncControl_backward_sync_control_auto_wrap_policy_kwargs _fsdp_kwargs setdefaultr, ValueError _activation_checkpointing_kwargs_state_dict_type_init_sharding_strategyrW_init_cpu_offloadrQrR)selfrKrLrMrNrOrPrQrRrSrUrVrWrYr[r\ __class__s u/Volumes/fast/ai/experiments/voice-extract-mac/.venv/lib/python3.12/site-packages/lightning_fabric/strategies/fsdp.pyrbzFSDPStrategy.__init__s$ #- 3  5J#-4 &>&@#45EvN $$%6=  "+ !`aa/:D  m ,0P $&E1 -!0!89JDL]L]!^,[9.cFtdt|jd)NThe `z3` does not use the `CheckpointIO` plugin interface.NotImplementedErrortype__name__rps rr checkpoint_iozFSDPStrategy.checkpoint_io$"E$t**=*=)>>q"rssrsiocFtdt|jd)Nruz3` does not support setting a `CheckpointIO` plugin.rv)rpr}s rrr{zFSDPStrategy.checkpoint_ior|rscP|jJ|j|jSN)rL local_rankrzs rr root_devicezFSDPStrategy.root_devices+$$000$$T__55rsc|jSrrcrzs rr num_nodeszFSDPStrategy.num_nodess rsrc||_yrr)rprs rrrzFSDPStrategy.num_nodess #rscH|jt|jSdS)Nr)rLlenrzs rr num_processeszFSDPStrategy.num_processess$-1-B-B-Ns4(()UTUUrscN|j|jz|jdS)N) num_replicasrank)rr global_rankrzs rrdistributed_sampler_kwargsz'FSDPStrategy.distributed_sampler_kwargss$"&$2D2D!DtO_O_``rsc|jSr)rdrzs rrrOz"FSDPStrategy.process_group_backends***rsc|jr |jS|j}t|tr |jSyr)rRrN isinstancermixed_precision_configrpplugins rrrz#FSDPStrategy.mixed_precision_configs9   '' ' fm ,00 0rsc\|j}|t|tsJ|StdS)Nz32-true) _precisionrrrs rrrNzFSDPStrategy.precisions3  fm4 44MY''rscR|t|tstd|||_y)NzGThe FSDP strategy can only work with the `FSDPPrecision` plugin, found )rr TypeErrorr)rprNs rrrNzFSDPStrategy.precisions/  I})Mefoepqr r#rsc|jJ|jjs1t|j|j|j|_yyr)rMcreates_processes_externallyrrr _launcherrzs rr_configure_launcherz FSDPStrategy._configure_launchersM''333''DD6t7O7OQUQcQceiesestDNErsct||jt|jj dt r*ddlm}|d|jd|jd<yy)Nr[r)init_device_meshcuda) rasetup_environment_setup_distributedrrigettupletorch.distributed.device_meshr)rprrqs rrrzFSDPStrategy.setup_environmentsc !# ! d''++M:E B F/?HYHYZgHh/iD  m , Crsmodule optimizers schedulerr>c|jjd}|dur"tdt|jd|j |}|||fS)zWraps the model into a :class:`~torch.distributed.fsdp.fully_sharded_data_parallel.FullyShardedDataParallel` module and sets `use_orig_params=True` to keep the reference to the original parameters in the optimizer.r`Fz You set `z(use_orig_params=False)` but this is not supported when setting the model and optimizer up jointly. Either set it to `True` or set the objects up in this order: Create the model, call `setup_module`, create the optimizer, call `setup_optimizer`.)rirrkrxry setup_module)rprrrr`s rrsetup_module_and_optimizersz(FSDPStrategy.setup_module_and_optimizerssk ++//0AB e #DJ//01++  ""6*z9,,rsc ddlmtfd|jDr=t |r t dd|j vrct d|j d=nJd||j|j|j|jjd|j }t||jt||j|S) z|Wraps the model into a :class:`~torch.distributed.fsdp.fully_sharded_data_parallel.FullyShardedDataParallel` module.rFullyShardedDataParallelc36K|]}t|ywrr).0modrs rr z,FSDPStrategy.setup_module..sUDTSz#78DTzYThe model is already wrapped in `FSDP` but there are still parameters on the meta device.rSz_A FSDP `auto_wrap_policy` is set, but the model is already wrapped. The policy will be ignored.)rrQrRrW device_id)torch.distributed.fsdpranymodulesr.r4rirQrrWrindex_move_torchmetrics_to_device_setup_activation_checkpointingrl)rprrs @rrrzFSDPStrategy.setup_modules D UFNNDTU U5f=o"T%6%66u%%&89- ,, $ ; ;"&"8"8**00  ## F %VT-=-=> (0U0UV rs optimizerc|jjdrt| |St |s t d|S)aISet up an optimizer for a model wrapped with FSDP. This setup method doesn't modify the optimizer or wrap the optimizer. The only thing it currently does is verify that the optimizer was created after the model was wrapped with :meth:`setup_module` with a reference to the flattened parameters. r`zThe optimizer does not seem to reference any FSDP parameters. HINT: Make sure to create the optimizer after setting up the model.)rirrasetup_optimizer_optimizer_has_flat_paramsrk)rprrqs rrrzFSDPStrategy.setup_optimizer:sL    !2 37*95 5))4/ rscyrr)rprs rrmodule_to_devicezFSDPStrategy.module_to_deviceMs rs empty_initc|jj}|j}t}|r$|j t j d|j ||j ||S)Nmeta)rNmodule_init_contextmodule_sharded_contextr enter_contexttorchdevice)rprprecision_init_ctxmodule_sharded_ctxstacks rrrz FSDPStrategy.module_init_contextQsj!^^??A!88:      V 4 5 ./ ./ rsc ddlm}ddlm}|d||j|j |j |jjd|jS)Nrr) enable_wrap) wrapper_clsrQrRrWrr) 2torch.distributed.fsdp.fully_sharded_data_parallelrtorch.distributed.fsdp.wraprrQrrWrrri)rprrs rrrz#FSDPStrategy.module_sharded_context_sW_; 0(( 77"44&&,,     rstensorr+ reduce_opcBt|trt|||S|S)N)r)rrr))rprr+rs rr all_reducezFSDPStrategy.all_reducems" ff %)&%9M M rsargsctsytjjdk(r6tjj |j j gytjj y)Nnccl) device_ids)r&r distributed get_backendbarrierrr)rprr\s rrrzFSDPStrategy.barrierusZ*,     ( ( *f 4    % %$2B2B2H2H1I % J    % % 'rsobjsrccts|S|g}tjj||tj |dS)Nr*r)r&rrbroadcast_object_list_groupWORLD)rprrs rr broadcastzFSDPStrategy.broadcast~s<*,Je //S /M1v rsmax_norm norm_typeerror_if_nonfinitecddlm}t||s9tdt |j d|j j d|jj||j||S)zClip gradients by norm.rrzFGradient clipping with FSDP is only possible if the module passed to `zE.clip_gradients_norm` is wrapped in `FullyShardedDataParallel`. Got: .)rr) rrrrrxryrqrNunscale_gradientsclip_grad_norm_)rprrrrrrs rrclip_gradients_normz FSDPStrategy.clip_gradients_norms} `&":;$Z(()*))22316  ((3%%x9%MMrspathstatestorage_optionsfilterc| td||jdk(r tdt|j |}|j r(|jdk(rt |std|ddlm }|jDcgc]}t|s|}}t|dk(r td t|d kDr td |d}|jdk(r|jr|j|j!d d t#|}i} i} |5|j%D]\} } t'| t(r| j+} | }nIt'| t,r|j/|| } | }n$t'| t0r| j+n| } | }t3| |xsi| | dddt5| ||j6dk(rt9j:| |t<z yy|jdk(rt |rt?j@|tC||jD}i}|5|j%D]}\} } t'| t(r| j+} nEt'| t,r|j/|| } n"t'| t0r| j+n| } t3| |xsi| | ddd|j6dk(rt9j:||yytd|jcc}w#1swYlxYw#1swYZxYw)aSave model, optimizer, and other state to a checkpoint on disk. If the state-dict-type is ``'full'``, the checkpoint will be written to a single file containing the weights, optimizer state and other metadata. If the state-dict-type is ``'sharded'``, the checkpoint gets saved as a directory containing one file per process, with model- and optimizer shards stored per file. Additionally, it creates a metadata file `meta.pt` with the rest of the user's state (only saved from rank 0). Nz`FSDPStrategy.save_checkpoint(..., storage_options=...)` is not supported because `FSDPStrategy` does not use the `CheckpointIO`.rJzVFSDP doesn't support loading sharded filtered checkpoints, so saving them is disabled.rZz/The checkpoint path exists and is a directory: rraCould not find a FSDP model in the provided checkpoint state. Please provide the model as part of the state like so: `save_checkpoint(..., state={'model': model, ...})`. Make sure you set up the model (and optimizers if any) through the strategy before saving the checkpoint.r_zFound multiple FSDP models in the given state. Saving checkpoints with FSDP is currently limited to a single model per checkpoint. To save multiple models, call the save method for each model separately with a different path.T)parentsexist_ok) world_sizezUnknown state_dict_type: )#rrmrwr ris_dir_is_sharded_checkpointIsADirectoryErrorrrvalues_has_fsdp_modulesrrkis_fileunlinkmkdir_get_sharded_state_dict_contextitemsrr state_dictroptim_state_dictr7r!_distributed_checkpoint_saverrsaver/shutilrmtree_get_full_state_dict_contextr)rprrrrFSDPrrstate_dict_ctxconverted_statemetadatakeyr converted target_dict full_states rrsave_checkpointzFSDPStrategy.save_checkpoints  &C   $"7"79"D%h  DNN4() ;;=T22f%>?% " "f ,%d+ d#9&T__]N)+J % HC!#v.$'NN$4 #C3$($9$9&#$F 8B3 8RCNN$4X[ !#v|Y K!. 1$ :t,%89N9N8OPQ QAU4 2 s&L3%L36BL8BM8MMstrict weights_onlyc|std|dt|j|}t|trddlm}||||j|iSt|tr tdddl m }ddl m }|jD cic]\}} t| s|| } }} t!| dk(r td |jD cic]\}} t| ts|| } }} t!| d kDr td t#| jd\} } t%|r7t'| }|5| | j)i}t+||| j-|| | | r]dd lm}||}| jD];\}} ||| ||}|j3||| | }| j-|=dddt5j6|t8z |}|j;| j;z | j;z }t=||j;| |D]}||vr|j?|||<|StA|r)tC|}ddlm"}m#}||j?| | |j|t|triStI|}| jD]c\}} tK| |jd5||j?|| }|j3|| | }| j-|ddde|j;| j;z | j;z }t=||j;| tM||||StdtO|dcc} }wcc} }w#1swYxYw#1swYxYw)zOLoad the contents from a checkpoint and restore the state of the given objects.z,Got FSDPStrategy.load_checkpoint(..., state=z) but a state with at least a model instance to reload is required. Pass it in like so: FSDPStrategy.load_checkpoint(..., state={'model': model, ...})r) _load_raw_module_state_from_path)rrrz`Loading a single optimizer object from a checkpoint is not supported yet with the FSDP strategy.)!load_sharded_optimizer_state_dictraCould not find a FSDP model in the provided checkpoint state. Please provide the model as part of the state like so: `load_checkpoint(..., state={'model': model, ...})`. Make sure you set up the model (and optimizers if any) through the strategy before loading the checkpoint.r_zFound multiple FSDP models in the given state. Loading checkpoints with FSDP is currently limited to a single model per checkpoint. To load multiple models, call the load method for each model separately with a different path.)rFileSystemReaderr)model_state_dict optimizer_keystorage_reader)rmodeloptimN)r)_load_raw_module_state _rekey_optimizer_state_if_neededF)r rank0_only)source destinationkeysz The path z does not point to a valid checkpoint. Make sure the path points to either a directory with FSDP checkpoint shards, or a single file with a full checkpoint.)(rkr rrr*lightning_fabric.strategies.model_parallelrrrrw&torch.distributed.checkpoint.optimizerrrrrrrlistrrr_distributed_checkpoint_loadload_state_dicttorch.distributed.checkpointroptim_state_dict_to_loadrloadr/r r$pop_is_full_checkpointr0rrr1rr2str)rprrrrrrrr rrrr module_keyr module_staterreader optim_key optim_state flattened_osdrrequested_metadata_keys checkpointrrtemp_state_dictrs rrload_checkpointzFSDPStrategy.load_checkpoints>uiHRR  DNN4() eV $ c ,T%DOOdj kI eY '%r  ]K27++-]-;3CTU[C\3;-] w<1 t  49;;=a=ZS%JuV_D`c5j= a w( 5&G-9*-E*3+1' )-(E(E-8-C"("')F) --m<-? 4zz$);";,WH&+jjlW\\^&CjooFW&W # -.Ex}}_e f.h&%\\#.c / O t $#D)J  #:>>*#=fY]YhYhqw x%( .j9J%/$4$4$6 51&T__afg&Fz~~V_G`bh&iO'+'D'D)8$#(E($ ))*:;hg%7',jjlW\\^&CjooFW&W # -.EzGXag h JEH_ ` D }%_ _  }^b rhgs1O(OO 5O BO>OOO' strategy_registryctjjsy|jd|d|jd|ddy)NrCz+Fully Sharded Data Parallel (FSDP) training) descriptionrDzQFully Sharded Data Parallel (FSDP) training with Full Sharding and CPU OffloadingT)r8rQ)rr is_availableregister)clsr6s rrregister_strategiesz FSDPStrategy.register_strategiessV  --/ ""  E # ""  k # rsc<t|j|j|_|jJd|j i}t r*|jjdk7r |jnd|d<t|j|jfi|y)NrPcpur) r5_set_world_ranks_get_process_group_backendrdrMrer-rrxr()rpr\s rrrzFSDPStrategy._setup_distributeds  &*&E&E&G#''333"+T]]!; #6:6F6F6K6Ku6T$"2"2Z^F; d668S8S^W]^rscH|jxst|jSr)rdr'rrzs rrr@z'FSDPStrategy._get_process_group_backends **m.[\`\l\l.mmrsc>|jq|jj|j|jz|jz|jj |j |jz|jxt_ t_ yr) rMset_global_rank node_rankrrset_world_sizerrrrutils_rank_zero_onlyrzs rrr?zFSDPStrategy._set_world_rankssy  # # /  $ $ 4 4T^^dFXFX5X[_[j[j5j k  $ $ 3 3DNNTEWEW4W X;?:J:JJ27rs)r]Nr)Nmean)r)g@T)NN)NTN)Bry __module__ __qualname____doc__rrrr#rrrrr+rrboolrxrr rintr rbpropertyrrr{setterrrrdictrrOrrrNrrrrrrrrrrrr%rrr rfloatrr6r r r5 classmethodrr<rr@r? __classcell__)rqs@rrrIrIVs1j.29=<@)-/3'97;6:04VZ?C2>6?AE,/k*,/#4 #56,/&&89 ,/ I& ,/ (} ,/)$,/4t34,/""23,/#9-,/#+5ftDL?Q1Q+R"S,/*2))<,/0,/!!23,/eE#J $<=>,/ !,/" #,/\ t|tt t ttt 6U\\663$3$4$$VsVV aDcNaa+x}++1A(B (=(( $8I#6$4$$ uu jjae--*.y/-FN~F^- vtI(@@ A-- 6fBy$ v $   htn H^    (>    gm%-c]FNuU]_bUbOcFd (S(C(D((Zc* (+#' NNNs # N $ N ! N NN* *.BF aRaRCvy#5667aR"# aR c8S#J,<#==>? aR  aRaRF_c'+ @ @ fic5TWAW;X6X1YYZ[@  @ tn @ c3h @ @ D  4E $  _nCnKrsrIrUrVrTr]c||iS| | td|At|tr t|n|ft d|dt ddfdiSt|t r t |iSd|iS)NzeYou cannot set both `activation_checkpointing` and `activation_checkpointing_policy`. Use the latter.z'`FSDPStrategy(activation_checkpointing=zD)` is deprecated, use `FSDPStrategy(activation_checkpointing_policy=z )` instead.check_fnct|Srr) submoduleclassess rrz2_activation_checkpointing_kwargs..s jG.LrsrS)rkrr#rr3setrh)rUrVrWs @rrrlrls ',K,S +0O0[ s   + . 545G/1G56N5OP==@\N+ W LMM137'(GLL  ? @@rspolicyr\cT||St|trddlm}||}||d<|S)Nrr<rS)rrYrr=)rZr\r=s rrrhrhs3 ~ &#@!&)!'F  Mrsractivation_checkpointing_kwargsc|syddlmtfd|jDr t dyddlm}m}m}tst||j}||fd|i|y)Nr)CheckpointWrapperc36K|]}t|ywrr)rrr^s rrrz2_setup_activation_checkpointing..s J9I#:c, -9IrztFSDP checkpointing is configured, but the model already contains checkpointed layers. Checkpointing will be ignored.)CheckpointImplapply_activation_checkpointingcheckpoint_wrapper)checkpoint_implcheckpoint_wrapper_fn) ;torch.distributed.algorithms._checkpoint.checkpoint_wrapperr^rrr4r`rarbr,r NO_REENTRANT)rr\r`rarbr^s @rrrrsj *] J9I JJ .   $$%7IdIde"6wASwWvwrsc*eZdZedededefdZy)rfrenabledr]c|s tSddlm}t||s9t dt |j d|jj d|jS)zuBlocks gradient synchronization inside the :class:`~torch.distributed.fsdp.FullyShardedDataParallel` wrapper.rrzABlocking backward sync is only possible if the module passed to `zB.no_backward_sync` is wrapped in `FullyShardedDataParallel`. Got: r) rrrrrrxryrqno_sync)rprrhrs rrno_backward_syncz)_FSDPBackwardSyncControl.no_backward_syncsl= _&":;$Z(()*))22316  ~~rsN)ryrHrIrrrKrrkrrsrrrfrfs*  v  AW  rsrfrQr9cNddlm}t||r|S|t|S)Nr)r9)offload_params)rr9rrK)rQr9s rrroros%1$[*=;o:]abm]nCoorsrWrXr;cNddlm}|jd|jd tdt |t r||j n|}d|jvr>|jd-|jd|jd td|S) Nr)r; process_groupr[znThe arguments `FSDPStrategy(process_group=..., device_mesh=...)` are mutually exclusive.Pass only one of them.HYBRIDrSzThe hybrid sharding strategy requires you to pass at least one of the parameters: `auto_wrap_policy`, `process_group` tuple, or `device_mesh`.) rr;rrkrr+uppername RuntimeError)rWr\r;strategys rrrnrns7 zz/".6::m3L3X %  ?IIZ\_>` 1 7 7 9:fwHHMM! JJ) * 2 JJ ' / JJ} % - 8   Orsrc:td|jDS)Nc3JK|]}|dD]}t|ddyw)params_fsdp_flattenedFN)getattr)rr+params rrrz-_optimizer_has_flat_params..s1>TUbghpbqY^(%0bq0>Ts!#)r param_groups)rs rrrrs" >G>T>T rs)NNNcddlm}ddlm}m}m}|d}|d}|j ||j||}|S)Nrr)ShardedOptimStateDictConfigShardedStateDictConfig StateDictTypeT)offload_to_cpurrYstate_dict_configoptim_state_dict_config)rrtorch.distributed.fsdp.apir}r~rrYSHARDED_STATE_DICT)rrr}r~rrrstate_dict_type_contexts rrrrsQGmm.dC9N"22%88+ 7 3 #"rsrrcddlm}m}ddlm}ddlm}|d|}|d|}|j ||j||} | S)Nr)FullStateDictConfigrr)FullOptimStateDictConfigT)rrr)rrrrrrrYFULL_STATE_DICT) rrrrrrrrrrs rrrr.sYJGC+4JW6dWab"22%55+ 7 3 #"rsrcT|jxr|tz jS)z]A heuristic check to determine whether the path points to a directory with checkpoint shards.)rr/rrs rrrrAs" ;;= Bd%77@@BBrsc"|jSr)rrs rrr*r*Fs <<>rscxddlmt|txr"t fd|j DS)Nrrc36K|]}t|ywrr)rmrs rrrz$_has_fsdp_modules..Ms-p_oZ[j.XsH.*Q2G1.s)rrrrto)rrmetricrs @rrrrPs6 N +#Hfnn.H &Irsrctrddlm}|||yddlm}trddlm}nddlm}||d}|||y)Nr)r checkpoint_id)FileSystemWriter)save_state_dictT)rsingle_file_per_rank)r-r&rrr,r)rrrrwriters rrrr\s65 _D1A # 9 L!t$G _f%rsr-ctrddlm}|||yddlm}trddlm}nddlm}||}|||y)Nr)r(rr)r%r)r-r&r(rr,r%)r-rr(rr.s rrr$r$os45 \.A # 9 L!t, \6"rs)T)|rwarningscollections.abcr contextlibrrrdatetimer functoolsrpathlibr typingr r r r rrr lightning_utilities.core.importsr"lightning_utilities.core.rank_zerorrFrtorch.nnr torch.optimrtyping_extensionsrrlightning_fabric.acceleratorsrlightning_fabric.pluginsrrr5lightning_fabric.plugins.collectives.torch_collectiver'lightning_fabric.plugins.precision.fsdpr7lightning_fabric.strategies.launchers.subprocess_scriptr$lightning_fabric.strategies.parallelr$lightning_fabric.strategies.registryr$lightning_fabric.strategies.strategyr r!r"r#r$&lightning_fabric.utilities.distributedr%r&r'r(r)r+r"lightning_fabric.utilities.importsr,r-lightning_fabric.utilities.initr.lightning_fabric.utilities.loadr/r0r1r2$lightning_fabric.utilities.rank_zeror3r4lightning_fabric.utilities.seedr5 lightning_fabric.utilities.typesr6r7rr8rr9r:r;rr=torch.optim.lr_schedulerr>rYrxrKrLrTrX _FSDP_ALIASESfilterwarnings FutureWarningrIr#rOrlrhrrfrornrrrrr*objectrnnrrr+rr$rrsrrrs%EE =U!15PPTA]ABCSrrff6=8oo<5CV %xs0CT0I'JL\\]G/9r1sst- =B\]S K#XS KlA&uT&\4V ;M-M'NOA%-i%8A A2 Xi%8 $ 4 xFxUYx^bx0 3 &p8E$ 2D,E#Fp<p /CTVh.) #F #yAQ7R # 9=# # ##15# #&CC$C dtqfq6):q  %,, SW &$sCx.&&QU&&#tCH~#T#d#rs