i ddlmZmZmZmZmZddlmZddlm Z ddl m Z ddl m Z ddlmZddlmZmZesdgZGd d e Zy ) )AnyListOptionalSequenceUnion)Tensor)lip_vertex_error)Metric) dim_zero_cat)_MATPLOTLIB_AVAILABLE)_AX_TYPE_PLOT_OUT_TYPEzLipVertexError.plotc eZdZUdZdZeed<dZeed<dZeed<dZ e ed<e e ed <e e ed < dd e e d ed eddffd Zde de ddfdZde fdZ ddeee ee fdeedefdZxZS)LipVertexErrora/ Implements Lip Vertex Error (LVE) metric for 3D talking head evaluation. The Lip Vertex Error (LVE) metric evaluates the quality of lip synchronization in 3D facial animations by measuring the maximum Euclidean distance (L2 error) between corresponding lip vertices of the generated and ground truth meshes for each frame. The metric is defined as: .. math:: \text{LVE} = \frac{1}{N} \sum_{i=1}^{N} \max_{v \in \text{lip}} \|x_{i,v} - \hat{x}_{i,v}\|_2^2 where :math:`N` is the number of frames, :math:`x_{i,v}` represents the 3D coordinates of vertex :math:`v` in the lip region of the ground truth frame :math:`i`, and :math:`\hat{x}_{i,v}` represents the corresponding vertex in the predicted frame. The metric computes the maximum squared L2 distance between corresponding lip vertices for each frame and averages across all frames. A lower LVE value indicates better lip synchronization quality. As input to ``forward`` and ``update``, the metric accepts the following input: - ``preds`` (:class:`~torch.Tensor`): Predicted vertices tensor of shape (T, V, 3) where T is number of frames, V is number of vertices, and 3 represents XYZ coordinates - ``target`` (:class:`~torch.Tensor`): Ground truth vertices tensor of shape (T', V, 3) where T' can be different from T As output of ``forward`` and ``compute``, the metric returns the following output: - ``lve_score`` (:class:`~torch.Tensor`): A scalar tensor containing the mean Lip Vertex Error value across all frames. Args: mouth_map: List of vertex indices corresponding to the mouth region validate_args: bool indicating if input arguments and tensors should be validated for correctness. Set to ``False`` for faster computations. kwargs: Additional keyword arguments, see :ref:`Metric kwargs` for more info. Raises: ValueError: If the number of dimensions of `vertices_pred` or `vertices_gt` is not 3. If vertex dimensions (V) or coordinate dimensions (3) don't match If ``mouth_map`` is empty or contains invalid indices Example: >>> import torch >>> from torchmetrics.functional.multimodal import lip_vertex_error >>> vertices_pred = torch.randn(10, 100, 3, generator=torch.manual_seed(42)) >>> vertices_gt = torch.randn(10, 100, 3, generator=torch.manual_seed(43)) >>> mouth_map = [0, 1, 2, 3, 4] >>> lip_vertex_error(vertices_pred, vertices_gt, mouth_map) tensor(12.7688) Tis_differentiableFhigher_is_betterfull_state_updategplot_lower_boundvertices_pred_listvertices_gt_list mouth_map validate_argskwargsreturnNc t|di|||_||_|js t d|j dgd|j dgdy)Nzmouth_map cannot be empty.r)defaultdist_reduce_fxr)super__init__rr ValueError add_state)selfrrr __class__s p/Volumes/fast/ai/experiments/voice-extract-mac/.venv/lib/python3.12/site-packages/torchmetrics/multimodal/lve.pyr zLipVertexError.__init__Vs] "6""*~~9: : +RM )2dK vertices_pred vertices_gtc|jr|jdk7s|jdk7r&td|jd|jd|jdd|jddk7r&td|jd|jdt |j |jdk\r2td t |j d |jddt |jd |jd }|d|}|d|}|jj||jj|y) aeUpdate metric states with predictions and targets. Args: vertices_pred: Predicted vertices tensor of shape (T, V, 3) where T is number of frames, V is number of vertices, and 3 represents XYZ coordinates vertices_gt: Ground truth vertices tensor of shape (T', V, 3) where T' can be different from T zIExpected both vertices_pred and vertices_gt to have 3 dimensions but got z and z dimensions respectively.NzdExpected vertices_pred and vertices_gt to have same vertex and coordinate dimensions but got shapes .z5mouth_map contains invalid vertex indices. Max index z# is larger than number of vertices r) rndimr!shapemaxrminrappendr)r#r'r( min_framess r%updatezLipVertexError.updatefss   !!Q&+*:*:a*? _$))*% 0@0@/AAZ\""12&+*;*;AB*?? +112% 8I8I7J!M4>>"m&9&9!&<< KCPTP^P^L_K`a**7*=*=a*@)AD ,,Q/1B1B11EF %kz2 !+:.  &&}5 $$[1r&ct|j}t|j}t|||j|j S)zCompute the Lip Vertex Error over all accumulated states. Returns: torch.Tensor: A scalar tensor with the mean LVE value )r rrr rr)r#r'r(s r%computezLipVertexError.computes?%T%<%<= "4#8#89  {DNNDL^L^__r&valaxc&|j||S)aPlot a single or multiple values from the metric. Args: val: Either a single result from calling `metric.forward` or `metric.compute` or a list of these results. If no value is provided, will automatically call `metric.compute` and plot that result. ax: An matplotlib axis object. If provided will add plot to that axis Returns: Figure and Axes object Raises: ModuleNotFoundError: If `matplotlib` is not installed .. plot:: :scale: 75 >>> # Example plotting a single value >>> import torch >>> from torchmetrics.multimodal.lve import LipVertexError >>> metric = LipVertexError(mouth_map=[0, 1, 2, 3, 4]) >>> vertices_pred = torch.randn(10, 100, 3, generator=torch.manual_seed(42)) >>> vertices_gt = torch.randn(10, 100, 3, generator=torch.manual_seed(43)) >>> metric.update(vertices_pred, vertices_gt) >>> fig_, ax_ = metric.plot() .. plot:: :scale: 75 >>> # Example plotting multiple values >>> import torch >>> from torchmetrics.multimodal.lve import LipVertexError >>> metric = LipVertexError(mouth_map=[0, 1, 2, 3, 4]) >>> values = [] >>> for _ in range(10): ... vertices_pred = torch.randn(10, 100, 3, generator=torch.manual_seed(42+_)) ... vertices_gt = torch.randn(10, 100, 3, generator=torch.manual_seed(43+_)) ... values.append(metric(vertices_pred, vertices_gt)) >>> fig_, ax_ = metric.plot(values) )_plot)r#r6r7s r%plotzLipVertexError.plotsXzz#r""r&)T)NN)__name__ __module__ __qualname____doc__rbool__annotations__rrrfloatrrintrr r3r5rrrr rr: __classcell__)r$s@r%rrs/b#t""d"#t#!e!V $6l" #L9LL L  L 2F22D2B ` `_c,#E&(6*:":;<,#IQRZI[,# ,#r&rN)typingrrrrrtorchr&torchmetrics.functional.multimodal.lver torchmetrics.metricr torchmetrics.utilities.datar torchmetrics.utilities.importsr torchmetrics.utilities.plotr r__doctest_skip__rrr&r%rLs987C&4@@-.b#Vb#r&