i#n UddlmZddlmZmZmZmZmZmZddl Z ddl m Z ddl m Z mZer ddlmZddlmZeeeeeffZGdd ZGd d eZ dd e j4j6d ege j8fdeee j8ge j8fdefdZide j<de j>ddde j@de jBde jDdide j<de j>ddde j@de jBde jDdide j<de j>ddd e j@d!e jBd!e jDd"id#e j<d$e j>d%dd&e j@d e jBd e jDd'id(e j<d)e j>d*dd+e j@d,e jBd,e jDd-id.e j>d/dd/e j@d/e jBd/e jDd0d1d2id3e j>d4dd4e j@d4e jBd4e jDd5d1d6id7e j>d8dd8e j@d8e jBd8e jDd9d1d:id;e j>dd1d?id@e j>dAddAe j@dBe jBdBe jDdCd1dDidEe j<dFe j>dGddHe j@dIe jBdIe jDdJidKe j>dLddMe j@dLe jBdLe jDdNd1dOidPe j>dQddRe j@dQe jBdQe jDdSd1dTidUe j>dVddWe j@dXe jBdXe jDdYd1dZid[e j>d\dd\e j@d\e jBd\e jDd]d1d^id_e j>d`dd`e j@d`e jBd`e jDdad1dbidce j>ddddde j@dde jBdde jDded1dfie j>dgddge j@dge jBdge jDdhd1diie j>djddje j@dje jBdje jDdkd1dlie j>dme jBdne jDdod1dpie j>dqe jBdrie j>dqe jBdse jDdtd1duie j>dve jBdwe jDdxd1dyie j>dze jBd{e jDd|d1d}ie j>d~e jBde jDdd1die j>de jBde jDdd1die j<de j>de jBdXie j<de j>de jBdie j<de j>de jBdoid Z#eeeeee jHfeffe%d<dddddZ&de jNdee jHefdeefdZ(ddde jHfdZ)edeZ*Gdde+e*Z,y))deque) TYPE_CHECKINGAnyCallableOptionalTypeVarUnionN)override)rank_zero_onlyrank_zero_warn)Fabric) Precisionc ~eZdZdZ ddeededededdf dZddd d ed ed ed eedeeddf dZ de fdZ ddZ y) ThroughputaComputes throughput. +------------------------+-------------------------------------------------------------------------------------+ | Key | Value | +========================+=====================================================================================+ | batches_per_sec | Rolling average (over ``window_size`` most recent updates) of the number of batches | | | processed per second | +--------------------------+-----------------------------------------------------------------------------------+ | samples_per_sec | Rolling average (over ``window_size`` most recent updates) of the number of samples | | | processed per second | +--------------------------+-----------------------------------------------------------------------------------+ | items_per_sec | Rolling average (over ``window_size`` most recent updates) of the number of items | | | processed per second | +--------------------------+-----------------------------------------------------------------------------------+ | flpps_per_sec | Rolling average (over ``window_size`` most recent updates) of the number of flops | | | processed per second | +--------------------------+-----------------------------------------------------------------------------------+ | device/batches_per_sec | batches_per_sec divided by world size | +--------------------------+-----------------------------------------------------------------------------------+ | device/samples_per_sec | samples_per_sec divided by world size | +--------------------------+-----------------------------------------------------------------------------------+ | device/items_per_sec | items_per_sec divided by world size. This may include padding depending on the data | +--------------------------+-----------------------------------------------------------------------------------+ | device/flops_per_sec | flops_per_sec divided by world size. | +--------------------------+-----------------------------------------------------------------------------------+ | device/mfu | device/flops_per_sec divided by world size. | +--------------------------+-----------------------------------------------------------------------------------+ | time | Total elapsed time | +--------------------------+-----------------------------------------------------------------------------------+ | batches | Total batches seen | +--------------------------+-----------------------------------------------------------------------------------+ | samples | Total samples seen | +--------------------------+-----------------------------------------------------------------------------------+ | lengths | Total items seen | +--------------------------+-----------------------------------------------------------------------------------+ Example:: throughput = Throughput() t0 = time() for i in range(1000): do_work() if torch.cuda.is_available(): torch.cuda.synchronize() # required or else time() won't be correct throughput.update(time=time() - t0, samples=i) if i % 10 == 0: print(throughput.compute()) Notes: - The implementation assumes that devices FLOPs are all the same as it normalizes by the world size and only takes a single ``available_flops`` value. - items_per_sec, flops_per_sec and MFU do not account for padding if present. We suggest using samples_per_sec or batches_per_sec to measure throughput under this circumstance. Args: available_flops: Number of theoretical flops available for a single device. world_size: Number of devices available across hosts. Global metrics are not included if the world size is 1. window_size: Number of batches to use for a rolling average. separator: Key separator to use when creating per-device and global metrics. Navailable_flops world_size window_size separatorreturnc||_||_|dkDsJ||_|dkDsJt||_t||_t||_t||_t||_ y)Nr)maxlen) rrr_MonotonicWindow_time_batches_samples_lengthsr_flops)selfrrrrs z/Volumes/fast/ai/experiments/voice-extract-mac/.venv/lib/python3.12/site-packages/lightning_fabric/utilities/throughput.py__init__zThroughput.__init__^sr /"A~~$Q/?k.R /?{/S /?{/S /?{/S "'{"; )lengthsflopstimebatchessamplesr#r$cX|jj|||krtd|d|d|jj||jj||||krtd|d|d|j j|t |jt |j k7r8tdt |j dt |jd|)|jj||jzyy) adUpdate throughput metrics. Args: time: Total elapsed time in seconds. It should monotonically increase by the iteration time with each call. batches: Total batches seen per device. It should monotonically increase with each call. samples: Total samples seen per device. It should monotonically increase by the batch size with each call. lengths: Total length of the samples seen. It should monotonically increase by the lengths of a batch with each call. flops: Flops elapased per device since last ``update()`` call. You can easily compute this by using :func:`measure_flops` and multiplying it by the number of batches that have been processed. The value might be different in each device if the batch size is not the same. zExpected samples (z') to be greater or equal than batches ()NzExpected lengths (z') to be greater or equal than samples (zIf lengths are passed (z1), there needs to be the same number of samples () rappend ValueErrorrrrlen RuntimeErrorrr)rr%r&r'r#r$s r updatezThroughput.updateqs. $ W 1':abiajjklm m W% W%    #5gY>efmenno!pqq MM  )4==!S%77"-c$--.@-ABT]]+,A/   KK  ut6 7 r"c4|jd|jd|jdd}|jr|jd|d<|jdkD}t |j|jj k(rF|jd|jdz }|jd|jdz }|jd|jdz }||z }||z }|jd|jd||z d|jd|i|r0||jz}|j|||jzd t |j|jj k(rM|jd|jdz } | |z } | |d|jd <|r| |jz} | |d <t |j|jj k(rt|j|jdz } |jd|jdz }| |z } | |jz }|r| |d <||d|jd <|jr ||jz |d|jd <|S) zCompute throughput metrics.)r%r&r'r#rrdevicebatches_per_secsamples_per_sec)r2r3 items_per_sec flops_per_secmfu) rrrrrr,rr.rrsumr)rmetricsadd_global_metrics elapsed_timeelapsed_batcheselapsed_samplesdev_samples_per_secdev_batches_per_secr3elapsed_lengthsdev_items_per_secr4 elapsed_flopsr5dev_flops_per_secs r computezThroughput.computesJJrN}}R(}}R(  ==!%r!2GI !__q0 tzz?djj// /::b>DJJqM9L"mmB/$--2BBO"mmB/$--2BBO"1L"@ "1L"@  NN(8/L:X(8:M ""5"G'6':T__'L  4==!T]]%9%99"&--"3dmmA6F"F$3l$B!BS& 0 >?%$5$GM/DJJqM9L)L8M - ? !+8(>OGfT^^,M: ;##8IDL`L`8`& 045r"c|jj|jj|jj|jj|j jyN)rclearrrrrrs r resetzThroughput.resetsR     r")Nrd/)rN) __name__ __module__ __qualname____doc__rfloatintstrr!r._THROUGHPUT_METRICSrCrHr"r rr s;|vy<'r"rE) rKrLrMrNrr!rrPrRrb __classcell__rds@r rUrUsA0 0x 03 04 0 HSM C L_ r"rUmodel forward_fnloss_fnrcddlm}|d}|5||n||jddd|jS#1swY|jSxYw)a-Utility to compute the total number of FLOPs used by a module during training or during inference. It's recommended to create a meta-device model for this: Example:: with torch.device("meta"): model = MyModel() x = torch.randn(2, 32) model_fwd = lambda: model(x) fwd_flops = measure_flops(model, model_fwd) model_loss = lambda y: y.sum() fwd_and_bwd_flops = measure_flops(model, model_fwd, model_loss) Args: model: The model whose FLOPs should be measured. forward_fn: A function that runs ``forward`` on the model and returns the result. loss_fn: A function that computes the loss given the ``forward_fn`` output. If provided, the loss and `backward` FLOPs will be included in the result. r)FlopCounterModeF)displayN)torch.utils.flop_counterrmbackwardget_total_flops)rirjrkrm flop_counters r measure_flopsrs s^89"51L  ? L JL ! * * ,   ' ' ))   ' ' ))s &AA( h200 sxm1g =BgwBtfloat32g2# Cg4&kCg4&k,C h200 nvl1gWHBgWHBgCgy`(Cg?r'Ch100 nvlg$^/lBgS CgSCg />2,Ch100 sxmgwBg$^/lBgSBg />2C h100 pciegvHBgvHBg|Bg|Cg@.CCzrtx 4090g$BgbCint4gbCzrtx 4080gm%Bg)Bg)Czrtx 4080 superg:Bg:Bg:Cl4gĎBg$ Bg$ Bg$ Bl40g˓Bg˓Bg˓Bg˓Ca100gꤡBg2Bg2Bg2Bg2Ca6000g\EBg\EBggBg@$Ca40gxBgxBgbcBgCa10ggP`Bg4&kBg4&kBg4&kBg4&kBz rtx 3090 tig@0Bg@0Bg@0Czrtx 3090gPb0Bgq$Bgq$Cz rtx 3080 tigcBgcBgYBgjZBg.Bg.BgIvvBggH|BggH|Bg}wBgBgBgBg h_BgHBgHBgHBgHBg`cԩBgH`Bg3Bg3Bg xHBg xHBg xHBgFBgBg~"4" t^~!$!47d?" T\#tO5D d "ZZ_Q'FE$ &%)D _D t^D d]D t^D T\D $ &$D D D 4 D D D T !D 1+A B { " 1+ 4(S T$T* EMM ! N"$)K)K)MQZ)Z"  & k_,>ugF G>%()) {{eL ! / 2//#kk&)VW5G-H-N-Ns-STU-V   "+s+++ z ! 5k_F5'R S:d#$$!r"pluginrc ddlm}m}m}m}m}m}m}m}m } t||std|t||r |jSt|||fr |jSt||rtjSt|| |fr |j St||rtj"St||r(|j$j&xstj(St||rtj(St+|)Nr) BitsandbytesPrecisionDeepSpeedPrecisionDoublePrecision FSDPPrecision HalfPrecisionMixedPrecisionrTransformerEnginePrecision XLAPrecisionz!Expected a precision plugin, got )lightning_fabric.pluginsrrrrrrrrrrr-rc_desired_input_dtyperdouble_desired_dtypeint8mixed_precision_config reduce_dtyperNotImplementedError) rrrrrrrrrrs r r[r[}s    fi (>vhGHH&/0||&=.9:***&/*||&<);<=$$$&45zz&-(,,99JU]]J&)$}} f %%r"T)boundceZdZdZdeddffd ZedeefdZ e deddfdZ e d e d e ddfd Z xZS) rzjCustom fixed size list that only supports right-append and ensures that all values increase monotonically.rrNc0t|||_yrE)r_r!r)rrrds r r!z_MonotonicWindow.__init__s  r"c*t|dkDr|dSy)Nrr0)r,rGs r lastz_MonotonicWindow.lasts t9q=8Or"xc|j}|||k\rtd|d|tj||t ||j kDr|d=yy)Nz&Expected the value to increase, last: z , current: r)rr+listr*r,r)rrrs r r*z_MonotonicWindow.appends\yy   EdV;WXVYZ[ [ D! t9t{{ "Q #r"keyvaluectd)Nz__setitem__ is not supported)r)rrrs r __setitem__z_MonotonicWindow.__setitem__s""@AAr")rKrLrMrNrPr!propertyrrrr r*rrrgrhs@r rrststhqk dBsB3B4BBr"rrE)- collectionsrtypingrrrrrr rtyping_extensionsr $lightning_fabric.utilities.rank_zeror r lightning_fabricr rrdictrQrPrOrRrrUnnModuleTensorrsfloat64rbfloat16float16rrrc__annotations__rr1r^r[rrrrSr"r rsII &O'23c5j 112 ssl1 1nAE$* 88??$*U\\)*$*h ~u||;< =$* $*N`@ v vF  v F `@ v vF  v F `@( u xH   y I )`@8 w wH  x I 9`@H w wF  v G I`@\ wG  w H ]`@l wG  w H m`@| wG  w H }`@L  wE  v F  M`@\  wG  v F  ]`@r  v wF  v F  s`@B  wG  w H C`@R  wG  w H  S`@d  wG  v F  e`@t uE  u F u`@D wG  w F E`@T wG  w H U`@f  wG  w F   wG  w H   v u F    w w  w w H   w w H   w w H   v w F   w w F   v w v  t u v  v w vw`@ T#tE#u{{"23U:;; <`N     X% X%U5;;;K5LX%QYZ]Q^X%v&[&U[[&B CuBtAwBr"