#tiRddlZddlZddlmZddlZgdZdZdZdZ dZ dZ dejeejzd dfd Z deddfd Zddeddfdd deejd ejeejzdzdededeej j"dzdeejdzd dfdZ ddeejded dfdZ ddeejdeejd dfdZeddfdeejdeejded dfdZy)N)Sequence) all_reducereduce broadcast all_gatherreduce_scatterc4ttjdstjddyt }|D]W}|j ry|jsy|jsy|j}||vry|j|Yy)N_nccl_all_reducez)PyTorch is not compiled with NCCL support stacklevelFT) hasattrtorch_Cwarningswarnset is_sparse is_contiguousis_cuda get_deviceadd)tensorsdevicestensordevices ^/Volumes/fast/ai/experiments/MLX_z-image/.venv/lib/python3.12/site-packages/torch/cuda/nccl.py is_availablers 588/ 0 AaPeG   ##%~~""$ W  F ctjj}|dz }|dz dz}|dz}tjjj d}|dk(r|||fS||||fS)a Returns the version of the NCCL. This function returns a tuple containing the major, minor, and patch version numbers of the NCCL. The suffix is also included in the tuple if a version suffix exists. Returns: tuple: The version information of the NCCL. izutf-8)rr _nccl_version_nccl_version_suffixdecode)vermajorminorpatchsuffixs rversionr,#sv (( "C 2IE BY% E %KE XX * * , 3 3G tjjSN)rr_nccl_unique_idrr unique_idr18s 88 # # %%rcDtjj|||Sr.)rr_nccl_init_rank) num_ranksuidranks r init_rankr7<s 88 # #IsD 99rinputsreturnct|tjjrt|tj r t dy)Nz(Inputs should be a collection of tensors) isinstance collectionsabc ContainerrTensor TypeError)r8s r_check_sequence_typerA@s9 fkoo77 8J =BCC=rc~t|||}t|tjj|||||yr.)rArrr r8outputsopstreamscommss rrrGs6 ! HHfgr7EBr)rDoutputrootrErFrDct||/| tdtjdtd||}nit |t jsFt |tjjr"tjdtd||}n |||n|}t jj||||||y)Nz'output' and 'outputs' can not be both specified. 'outputs' is deprecated in favor of 'output', taking in a single output tensor. The signature of reduce is: reduce(inputs, output=None, root=0, op=SUM, streams=None, comms=None).z`nccl.reduce` with an output tensor list is deprecated. Please specify a single output tensor with argument 'output' instead instead.r r z\nccl.reduce with an output tensor list is deprecated. Please specify a single output tensor.) rA ValueErrorrr FutureWarningr;rr?r<r=rr _nccl_reduce)r8rHrIrErFrGrD_outputs rrrQs   Y  MM`   dmG  -* ((3   5   ,"(.&,f HH&'4WeDrc^t|tjj||||yr.)rArr_nccl_broadcast)r8rIrFrGs rrr|s$  HHVT7E:rctt|t|tjj||||yr.)rArr_nccl_all_gather)r8rDrFrGs rrrs,  ! HHfgw>rcvt|t|tjj|||||yr.)rArr_nccl_reduce_scatterrCs rrrs. ! HH!!&'2wFr)rNN)NN)r<rcollections.abcr torch.cudar__all__SUMrr,r1r7r?rArintcudaStreamrrrrr0rrr\s$ P*-*&:D0F!FD4D $TC<@26  (E.2(E U\\ "(E LL8ELL1 1D 8(E (E (E ejj'' (4 / (Eell #d *(E (EXHL; U\\ ";*-; ;   ? U\\ "? ell #?  ?   G U\\ " G ell # G  G  Gr