iddlmZmZddlmZmZmZmZddlm Z ddl m Z ddl m Z ddlmZmZedZdd d d d Zed Zed ZeeeefZGddZy))AbstractContextManager nullcontext)AnyLiteralOptionalUnion)Tensor)Module) Optimizer) _PARAMETERS Optimizable)@ 64-true32-true16-mixed bf16-mixed)643216bf16)ztransformer-engineztransformer-engine-float16z16-truerz bf16-truerrrc 6eZdZUdZdZeed<dedefdZde fdZ de fdZ de fd Z d e de fd Zd e de fd Zd edeede fdZd edeede de ddf dZd edeede fdZdede de fdZdedefdZdeddfdZdeee ffdZdeee fddfdZddZy) PrecisionzBase class for all plugins handling the precision-specific parts of the training. The class attribute precision must be overwritten in child classes. The default value reflects fp32 training. r precisionmodulereturnc|S)zConvert the module parameters to the precision type this plugin handles. This is optional and depends on the precision limitations during optimization. )selfrs /Volumes/fast/ai/experiments/voice-extract-mac/.venv/lib/python3.12/site-packages/lightning/fabric/plugins/precision/precision.pyconvert_modulezPrecision.convert_module0s  ctS)z1Controls how tensors get created (device, dtype).rr s r!tensor_init_contextzPrecision.tensor_init_context8 }r#ctS)zInstantiate module parameters or tensors in the precision type this plugin handles. This is optional and depends on the precision limitations during optimization. r%r&s r!module_init_contextzPrecision.module_init_context<s }r#ctS)zWA contextmanager for managing model forward/training_step/evaluation_step/predict_step.r%r&s r!forward_contextzPrecision.forward_contextDr(r#datac|S)zConvert model inputs (forward) to the floating point precision type of this plugin. This is a no-op in the base precision plugin, since we assume the data already has the desired type (default is torch.float32). rr r-s r! convert_inputzPrecision.convert_inputH  r#c|S)zConvert outputs to the floating point precision type expected after model's forward. This is a no-op in the base precision plugin, since we assume the data already has the desired type (default is torch.float32). rr/s r!convert_outputzPrecision.convert_outputQr1r#tensorcy)zRuns before precision plugin executes backward. Args: tensor: The tensor that will be used for backpropagation module: The module that was involved in producing the tensor and whose parameters need the gradients Nrr r4rs r! pre_backwardzPrecision.pre_backwardZr#modelargskwargsNc(|j|i|y)zPerforms the actual backpropagation. Args: tensor: The tensor that will be used for backpropagation model: The module that was involved in producing the tensor and whose parameters need the gradients N)backward)r r4r9r:r;s r!r=zPrecision.backwardcs ((r#cy)zRuns after precision plugin executes backward. Args: tensor: The tensor that will be used for backpropagation module: The module that was involved in producing the tensor and whose parameters need the gradients Nrr6s r! post_backwardzPrecision.post_backwardmr8r# optimizerc &|jdi|S)zHook to run the optimizer step.r)step)r r@r;s r!optimizer_stepzPrecision.optimizer_stepvs y~~'''r#c#LK|jD]}|dEd{y7w)zThe main params of the model. Returns the plain model params here. Maybe different in other precision plugins. paramsN) param_groups)r r@groups r! main_paramszPrecision.main_params~s( ++EX & &, &s $"$cy)Nr)r r@s r!unscale_gradientszPrecision.unscale_gradientssr#ciS)zCalled when saving a checkpoint, implement to generate precision plugin state_dict. Returns: A dictionary containing precision plugin state. rr&s r! state_dictzPrecision.state_dicts  r#rLcy)zCalled when loading a checkpoint, implement to reload precision plugin state given precision plugin state_dict. Args: state_dict: the precision plugin state returned by ``state_dict``. Nr)r rLs r!load_state_dictzPrecision.load_state_dicts r#cy)zThis method is called to teardown the training process. It is the right place to release memory and free other resources. Nrr&s r!teardownzPrecision.teardownr8r#)rN) __name__ __module__ __qualname____doc__r_PRECISION_INPUT_STR__annotations__r r"rr'r*r,rr0r3r rr7r=r?r rCr r rHrJdictstrrLrNrPrr#r!rr's\ '0I#/V%;%;!7##33 6 8F3C  )v)hv.>)s)VY)^b) F HV4D  ((( ('Y';'9DcN $sCx. T  r#rN) contextlibrrtypingrrrrtorchr torch.nnr torch.optimr lightning.fabric.utilities.typesr r _PRECISION_INPUT_INT%_PRECISION_INPUT_STR_ALIAS_CONVERSION_PRECISION_INPUT_STR_ALIASrU_PRECISION_INPUTrrr#r!rcsw;00!Ez*/8 Q[eq(r%$%=> -/CE__`{ { r#