iddlmZmZddlmZddlmZddlmZm Z ddl Z ddl m Z ddl m Z ddlmZddlmZmZdd lmZmZdd lmZdd lmZdd lmZGd deeZy))ABCabstractmethod) Generator)contextmanager)AnyOptionalN)Tensor)override) CheckpointIOClusterEnvironment)ReduceOp_all_gather_ddp_if_available) LayerSync) Precision)Strategyc LeZdZdZ ddeddeeejdeedee dee f fd Z e e ed ejfd Ze d efd Ze d efd Ze d efdZe d efdZe ed efdZe d eeejfdZej0deeejd dfdZe d eeeffdZeddedeeded efdZeddeded efdZe d e!fdZ"ed fd Z#xZ$S)!ParallelStrategyz:Strategy for training with multiple processes in parallel.N acceleratorzpl.accelerators.Acceleratorparallel_devicescluster_environment checkpoint_ioprecision_plugincTt||||||_||_d|_y)N)rrr)super__init__rr _layer_sync)selfrrrrr __class__s z/Volumes/fast/ai/experiments/voice-extract-mac/.venv/lib/python3.12/site-packages/lightning/pytorch/strategies/parallel.pyrzParallelStrategy.__init__"s2 [ `pq 0AT 04returncy)zReturn the root device.Nrs r root_devicezParallelStrategy.root_device/sr cR|j|jjSdSNr)r global_rankr$s rr(zParallelStrategy.global_rank5s(9=9Q9Q9]t''335dcddr cR|j|jjSdSr')r local_rankr$s rr*zParallelStrategy.local_rank9(8<8P8P8\t''224cbccr cR|j|jjSdSr')r node_rankr$s rr-zParallelStrategy.node_rank=s(7;7O7O7[t''113babbr cR|j|jjSdS)N)r world_sizer$s rr0zParallelStrategy.world_sizeAr+r c |jdk(Sr')r(r$s ris_global_zerozParallelStrategy.is_global_zeroEs1$$r c|jSN_parallel_devicesr$s rrz!ParallelStrategy.parallel_devicesJs%%%r c||_yr4r5)rrs rrz!ParallelStrategy.parallel_devicesNs !1r cb|jt|jnd|jdS)Nr) num_replicasrank)rlenr(r$s rdistributed_sampler_kwargsz+ParallelStrategy.distributed_sampler_kwargsRs3;?:O:O:[C 5 56ab$$  r tensorgroup sync_gradsct|||S)z&Perform a all_gather on all processes.)r>r?)r)rr=r>r?s r all_gatherzParallelStrategy.all_gatherYs,F%JWWr decisionallctjt||j}|j |t j }|rt||jk(}|St|}|S)aReduces a boolean decision over distributed processes. By default is analogous to ``all`` from the standard library, returning ``True`` only if all input decisions evaluate to ``True``. If ``all`` is set to ``False``, it behaves like ``any`` instead. Args: decision: A single input decision. all: Whether to logically emulate ``all`` or ``any``. Defaults to True. Returns: bool: The reduced boolean decision. )device) reduce_op) torchr=intr%reducer SUMboolr0)rrBrCs rreduce_boolean_decisionz(ParallelStrategy.reduce_boolean_decision^si<<H d6F6FG;; ll 9<4DOO34BFhr c#Kt|jtjjj r(|jj 5ddddydy#1swYyxYww)zBlocks ddp sync gradients behaviour on backwards pass. This is useful for skipping sync when accumulating gradients, reducing communication overhead Returns: context manager with sync behaviour off N) isinstancemodelpl utilitiestypesDistributedDataParallelno_syncr$s rblock_backward_syncz$ParallelStrategy.block_backward_synctsR djj",,"4"4"L"L M##% &%J&%sAA4A(A4(A1-A4cr|jJ|jjt|yr4)rteardownr)rrs rrWzParallelStrategy.teardowns2''333   ))+ r )NNNNN)NF)T)r!N)%__name__ __module__ __qualname____doc__rlistrGrEr r rrpropertyrr r%rHr(r*r-r0rKr2rsetterdictstrrr<r rArLrrrUrW __classcell__)rs@rrrsSD@D9=<@0404 5;< 5#4 #56 5&&89 5  - 5 #9- 5 &U\\&&eSeedCddc3ccdCdd %%%&(4 +=">&&2$u||:L1M2RV22 DcN  XX XRVXciXX44* Y  r r)abcrrcollections.abcr contextlibrtypingrrrGr typing_extensionsr lightning.pytorchpytorchrPlightning.fabric.pluginsr r &lightning.fabric.utilities.distributedr rlightning.pytorch.pluginsr#lightning.pytorch.plugins.precisionr%lightning.pytorch.strategies.strategyrrr#r rrns>$%% &EY/9:gxgr