iddlmZddlmZmZmZddlZddlmZddlm Z ddl m Z ddl m Z mZddlmZdd lmZdd lmZdd lmZdd lmZGd deZdedefdZy))AbstractContextManager)AnyLiteralOptionalN)apply_to_collection)Tensor)Module)LBFGS Optimizer)override) Precision)_convert_fp_tensor)_TORCH_GREATER_EQUAL_2_4) Optimizablec :eZdZdZ ddeddededddfd Zede fd Z ed e de fd Z ed e de fd Z ededeede de ddf fd Zedede de ffd Zedeee ffdZedeee fddfdZededdfdZxZS)MixedPrecisionaFPlugin for Automatic Mixed Precision (AMP) training with ``torch.autocast``. Args: precision: Whether to use ``torch.float16`` (``'16-mixed'``) or ``torch.bfloat16`` (``'bf16-mixed'``). device: The device for ``torch.autocast``. scaler: An optional :class:`torch.cuda.amp.GradScaler` to use. N precision16-mixed bf16-mixeddevicescalerztorch.amp.GradScalerreturnc|dvr%tdt|jd|d||_|]|jdk(rNtr t j j|n't jj j}||jdk(rtd|d ||_ ||_ |jdk(rt j|_ yt j|_ y) NrzPassed `z (precision=z1)`. Precision must be '16-mixed' or 'bf16-mixed'.r)rrz6`precision='bf16-mixed'` does not use a scaler, found .) ValueErrortype__name__rrtorchamp GradScalercudarrbfloat16float16_desired_input_dtype)selfrrrs {/Volumes/fast/ai/experiments/voice-extract-mac/.venv/lib/python3.12/site-packages/lightning_fabric/plugins/precision/amp.py__init__zMixedPrecision.__init__(s 6 64:../{9-HAA  # >dnn :s~~dkk1J1JKKr)datacDt|tt|jSN)functionr+dst_type)rrrr%r&r/s r' convert_inputzMixedPrecision.convert_inputBs"42DF]a]v]vwwr)cTt|tttjSr1)rrrrget_default_dtyper4s r'convert_outputzMixedPrecision.convert_outputFs"42DF]b]t]t]vwwr)tensormodelargskwargsc||j|jj|}t| ||g|i|yN)rscalesuperbackward)r&r9r:r;r< __class__s r'rAzMixedPrecision.backwardJs: ;; "[[&&v.F 888r) optimizerc |jt||fi|St|tr t d|jj |fi|}|jj|S)Nz/AMP and the LBFGS optimizer are not compatible.)rr@optimizer_step isinstancer TypeErrorstepupdate)r&rCr< step_outputrBs r'rEzMixedPrecision.optimizer_stepPsk ;; 7))>v> > i 'MN N&dkk&&y;F;  r)cR|j|jjSiSr>)r state_dictr-s r'rLzMixedPrecision.state_dict`s$ ;; ";;))+ + r)rLcT|j|jj|yyr>)rload_state_dict)r&rLs r'rNzMixedPrecision.load_state_dictfs# ;; " KK ' ' 3 #r)cp|j}|(t|r td|j|yy)NzKGradient clipping is not implemented for optimizers handling the unscaling.)r_optimizer_handles_unscalingNotImplementedErrorunscale_)r&rCrs r'unscale_gradientsz MixedPrecision.unscale_gradientsks6  +I6)*wxx OOI & r)r>)r __module__ __qualname____doc__rstrrr(r rr.rr5r8rr rArrEdictrLrNr rS __classcell__)rBs@r'rrs48 h34hh/0 h  h,L!7LLx#x#xxx3x3xx9v9hv.>9s9VY9^b99      DcN 4$sCx.4T44'9'''r)rrCrct|ddS)aTDetermines whether a PyTorch optimizer handles unscaling gradients in the step method rather than through the :class:`torch.cuda.amp.GradScaler`. Since, the current implementation of this function checks a PyTorch internal variable on the optimizer, the return value will only be reliable for built-in PyTorch optimizers. _step_supports_amp_scalingF)getattr)rCs r'rPrPts 9:E BBr)) contextlibrtypingrrrr#lightning_utilities.core.apply_funcrrtorch.nnr torch.optimr r typing_extensionsr ,lightning_fabric.plugins.precision.precisionr (lightning_fabric.plugins.precision.utilsr"lightning_fabric.utilities.importsr lightning_fabric.utilities.typesrrboolrPr)r'risS.)) C(&BGG8S'YS'lCCCDCr)