i,ddlZddlmZmZmZmZmZddlZddlmZddl m Z ddl m Z ddl mZddlmZddlmZdd lmZdd lmZmZmZdd lmZdd lmZdd lmZmZddl m!Z!ddl"m#Z#ddl$m%Z%ddl&m'Z'm(Z(erddl)m*Z*GddeZ+y)N) TYPE_CHECKINGAnyCallableOptionalUnion)Tensor)Module) Optimizer) DataLoader)override) Accelerator)_XLA_GREATER_EQUAL_2_1) CheckpointIO Precision XLAPrecision)XLAEnvironment)XLACheckpointIO)ParallelStrategy_StrategyRegistry) _XLALauncher) TBroadcast)rank_zero_only)_PATHReduceOpMpDeviceLoaderceZdZdZ d2deedeeejdee dee de ddf fd Z e edejfd Ze defd Ze ede fd Zej&ed eeddfdZe ede fdZej&edeeddfdZe edeffd Ze edeffd Ze edeffd Ze edeffd Zed3dZed3fd ZededefdZededdfdZede ddfdZ!ed4de"dee#de de"fd Z$e d5d!e%e"e#fdee#d"ee%e&e'fde"fd#Z(ed6d$ee'd%e#d&e#ddfd'Z)ed7d(e*d)ede*fd*Z+e d5d+e,d,e-e'e%ee.e#ffd-ee#d.ee-e'e/e'e#ge ffddf fd/ Z0e1ed0e2ddfd1Z3xZ4S)8 XLAStrategyzxStrategy for training multiple TPU devices using the :func:`torch_xla.distributed.xla_multiprocessing.spawn` method.N acceleratorparallel_devices checkpoint_io precisionsync_module_statesreturncht|||t||d|_d|_||_y)N)rr cluster_environmentr!r"F)super__init__r_backward_sync_control _launched_sync_module_states)selfrr r!r"r# __class__s t/Volumes/fast/ai/experiments/voice-extract-mac/.venv/lib/python3.12/site-packages/lightning_fabric/strategies/xla.pyr(zXLAStrategy.__init__+sB #- . 0'  '+##5 cb|js tdddlmcm}|j S)NzFAccessing the XLA device before processes have spawned is not allowed.r)r* RuntimeErrortorch_xla.core.xla_modelcore xla_model xla_device)r,xms r. root_devicezXLAStrategy.root_device>s)~~gh h--}}r/cH|jt|jSdSNr)r lenr,s r. num_processeszXLAStrategy.num_processesGs$-1-B-B-Ns4(()UTUUr/cZ|j}|t|tsJ|StSN)_checkpoint_io isinstancerr,plugins r.r!zXLAStrategy.checkpoint_ioKs3$$  fo6 66M  r/iocR|t|tstd|||_y)NzHThe XLA strategy can only work with the `XLACheckpointIO` plugin, found )r@r TypeErrorr?)r,rCs r.r!zXLAStrategy.checkpoint_ioTs/ >*R"Afgifjkl l r/c\|j}|t|tsJ|StdS)Nz32-true) _precisionr@rrAs r.r"zXLAStrategy.precision[s3  fl3 33MI&&r/cR|t|tstd|||_y)NzEThe XLA strategy can only work with the `XLAPrecision` plugin, found )r@rrErG)r,r"s r.r"zXLAStrategy.precisionds/  I|)Lcdmcnop p#r/c4|jr t|SdSr9)r*r' global_rankr,r-s r.rJzXLAStrategy.global_rankks'+nnuw";!;r/c4|jr t|SdSr9)r*r' local_rankrKs r.rMzXLAStrategy.local_rankp&*^^uw!::r/c4|jr t|SdSr9)r*r' node_rankrKs r.rPzXLAStrategy.node_rankus%)NNuw 99r/c4|jr t|SdS)N)r*r' world_sizerKs r.rSzXLAStrategy.world_sizezrNr/c$t||_yr>)r _launcherr;s r._configure_launcherzXLAStrategy._configure_launchers%d+r/c|jJt|jdk(r"tdt|jdd|_|j t_t|)y)NrRzThe z does not support running on a single device with the PjRT runtime. Try using all devices or the `SingleDeviceXLAStrategy` strategyT) r r:NotImplementedErrortype__name__r*rJrrankr'setup_environmentrKs r.r\zXLAStrategy.setup_environmentsw$$000 t$$ % *&tDz**+,SS  ".. !#r/modulecT|jrtrddlm}nddlm}|||S)Nr)broadcast_master_param)r+rr2r_torch_xla.experimental.pjrt)r,r]r_s r. setup_modulezXLAStrategy.setup_modules!  # #%KN "6 * r/c:|j|jyr>)tor7)r,r]s r.module_to_devicezXLAStrategy.module_to_devices $""#r/ dataloaderrcddlm}t||r|S|||j}|jj |_t |jdd|_|S)Nrr batch_sampler)%torch_xla.distributed.parallel_loaderrr@r7_loaderdatasetgetattrrg)r,rers r.process_dataloaderzXLAStrategy.process_dataloadersYH j. 1 #J0@0@A '//77 #*:+=+=PT#U  r/tensorgroup sync_gradsc|js|St|ts$tdt |j d||j dk(r|jd}|j}|j|j}ddl m cm }ddlm cm}|r|j!|n|j!|}|j|}|S)aCFunction to gather a tensor from several distributed processes. Args: tensor: tensor to all-gather. group: unused. sync_grads: flag that allows users to synchronize gradients for the all-gather operation. Return: A tensor of shape (world_size, ...) `z4.all_gather` is only implemented for tensors. Given rN)r*r@rrXrYrZdim unsqueezedevicercr7torch_xla.core.functionsr3 functionsr2r4 all_gather)r,rmrnrooriginal_devicexfr6s r.rwzXLAStrategy.all_gathers~~M&&)%DJ''((\]c\de  ::<1 %%a(F --4++,----*4v&"--:O?+ r/output reduce_opct|ts!tj||j}t|t xr|t j k7}t|txr|jdv}|s|rtd|ddl m cm }|jd|t}t|tr!|jdvr||jz }|S)N)rt)summeanavgz]Currently, the XLAStrategy only supports `sum`, `mean`, `avg` for the reduce operation, got: rreduce)rr~)r@rtorchrmr7rSUMstrlower ValueErrorr2r3r4 mesh_reducer}rS)r,rzrnr{invalid_reduce_opinvalid_reduce_op_strr6s r. all_reducezXLAStrategy.all_reduces&&)\\&1A1ABF&y(;Y X\\@Y *9c : ny?PXn?n  5;   .-&#6 i %)//*;*Ndoo-F r/nameargskwargscZ|jsyddlmcm}|d}|j |y)Nr)r*r2r3r4 rendezvous)r,rrrr6s r.barrierzXLAStrategy.barriers(~~ -- <D dr/objsrcc|js|Sddlmcm}t |t }|rL|j dk(r|jd}|j}|j|j}nqtj}tj||tjt!|j#|jtj$}|g}|j'|||d}|sVtj|j)j+j-}tj.|}|S|j}|S)Nr)rtdtype) root_ordinal)r*r2r3r4r@rrrrsrtrcr7rCBytesIOrsaverm bytearray getbufferfloatcollective_broadcastcpubytenumpyload)r,rrr6 is_tensorrxbuffers r. broadcastzXLAStrategy.broadcasts~~J--sF+ wwyA~mmA&!jjO&&))*CZZ\F JJsF #,,&**,-d6F6FekkCe #6!fZZ  0 6 6 89F**V$C &&)C r/pathstatestorage_optionsfilterc^ddlmcm}|jt|||||y)aSave model, optimizer, and other state as a checkpoint file. Args: path: A path to where the file(s) should be saved state: A dictionary with contents to be saved. If the dict contains modules or optimizers, their state-dict will be retrieved and converted automatically. storage_options: Additional options for the ``CheckpointIO`` plugin filter: An optional dictionary of the same format as ``state`` mapping keys to callables that return a boolean indicating whether the given parameter should be saved (``True``) or filtered out (``False``). rN)rr)r2r3r4 mark_stepr'save_checkpoint)r,rrrrr6r-s r.rzXLAStrategy.save_checkpoints-& .-  e_U[\r/strategy_registryc@|jd||jy)Nxla) description)registerrZ)clsrs r.register_strategieszXLAStrategy.register_strategies-s ""5#3<<"Hr/)NNNNT)r$N)NF)NNr>)r)5rZ __module__ __qualname____doc__rr listrrtrrboolr(propertyr r7intr<r!setterrr"rrJrMrPrSrVr\r rardr rlrrrwrrrrrrrrdictr rr classmethodrr __classcell__)r-s@r.rr's) .29=37,0#' 6k*6#4 #566 0 6 L) 6 ! 6 6& U\\VsVV !!! ! 6!4!!  '<'' $8I#6$4$$  ? ]  ]]2 I4EI$IIr/r),rCtypingrrrrrrrtorch.nnr torch.optimr torch.utils.datar typing_extensionsr lightning_fabric.acceleratorsr !lightning_fabric.accelerators.xlarlightning_fabric.pluginsrrr%lightning_fabric.plugins.environmentsrlightning_fabric.plugins.io.xlarlightning_fabric.strategiesrr)lightning_fabric.strategies.launchers.xlar$lightning_fabric.strategies.strategyr$lightning_fabric.utilities.rank_zeror lightning_fabric.utilities.typesrrrhrrr/r.rsZ @@ !'&5DJJ@;KB;?<DII"IIr/