iddlmZddlmZmZmZddlZddlmZddlm Z ddl m Z m Z m Z mZmZddlmZddlmZmZesd gZGd d e Zy) )Sequence)AnyOptionalUnionN)Tensor)Metric) PREDS_TYPE TARGETS_TYPE_squad_compute_squad_input_check _squad_update)_MATPLOTLIB_AVAILABLE)_AX_TYPE_PLOT_OUT_TYPEz SQuAD.plotceZdZUdZdZeed<dZeed<dZeed<dZ e ed<d Z e ed <e ed <e ed <e ed <de ddffd ZdededdfdZdeee ffdZ ddeee ee fdeedefdZxZS)SQuADaCalculate `SQuAD Metric`_ which is a metric for evaluating question answering models. This metric corresponds to the scoring script for version 1 of the Stanford Question Answering Dataset (SQuAD). As input to ``forward`` and ``update`` the metric accepts the following input: - ``preds`` (:class:`~Dict`): A Dictionary or List of Dictionary-s that map ``id`` and ``prediction_text`` to the respective values Example ``prediction``: .. code-block:: python {"prediction_text": "TorchMetrics is awesome", "id": "123"} - ``target`` (:class:`~Dict`): A Dictionary or List of Dictionary-s that contain the ``answers`` and ``id`` in the SQuAD Format. Example ``target``: .. code-block:: python { 'answers': [{'answer_start': [1], 'text': ['This is a test answer']}], 'id': '1', } Reference SQuAD Format: .. code-block:: python { 'answers': {'answer_start': [1], 'text': ['This is a test text']}, 'context': 'This is a test context.', 'id': '1', 'question': 'Is this a test?', 'title': 'train test' } As output of ``forward`` and ``compute`` the metric returns the following output: - ``squad`` (:class:`~Dict`): A dictionary containing the F1 score (key: "f1"), and Exact match score (key: "exact_match") for the batch. Args: kwargs: Additional keyword arguments, see :ref:`Metric kwargs` for more info. Example: >>> from torchmetrics.text import SQuAD >>> preds = [{"prediction_text": "1976", "id": "56e10a3be3433e1400422b22"}] >>> target = [{"answers": {"answer_start": [97], "text": ["1976"]}, "id": "56e10a3be3433e1400422b22"}] >>> squad = SQuAD() >>> squad(preds, target) {'exact_match': tensor(100.), 'f1': tensor(100.)} Fis_differentiableThigher_is_betterfull_state_updategplot_lower_boundgY@plot_upper_boundf1_score exact_matchtotalkwargsreturnNc nt|di||jdtjdtj d|jdtjdtj d|jdtjdtj dy) Nrr)dtypesum)namedefaultdist_reduce_fxrr)super__init__ add_statetorchtensorfloatint)selfr __class__s l/Volumes/fast/ai/experiments/voice-extract-mac/.venv/lib/python3.12/site-packages/torchmetrics/text/squad.pyr%zSQuAD.__init__hs{ "6" J Qekk0Rchi M5<<3Ufkl GU\\!599-M^cdpredstargetct||\}}t||\}}}|xj|z c_|xj|z c_|xj|z c_y)z*Update state with predictions and targets.N)r r rrr)r+r/r0 preds_dict target_dictrrrs r-updatez SQuAD.updatersS"4UF"C K'4Z'M$+u !  K' e r.cXt|j|j|jS)z5Aggregate the F1 Score and Exact match for the batch.)r rrr)r+s r-computez SQuAD.computezsdmmT-=-=tzzJJr.valaxc&|j||S)aPlot a single or multiple values from the metric. Args: val: Either a single result from calling `metric.forward` or `metric.compute` or a list of these results. If no value is provided, will automatically call `metric.compute` and plot that result. ax: An matplotlib axis object. If provided will add plot to that axis Returns: Figure and Axes object Raises: ModuleNotFoundError: If `matplotlib` is not installed .. plot:: :scale: 75 >>> # Example plotting a single value >>> from torchmetrics.text import SQuAD >>> metric = SQuAD() >>> preds = [{"prediction_text": "1976", "id": "56e10a3be3433e1400422b22"}] >>> target = [{"answers": {"answer_start": [97], "text": ["1976"]}, "id": "56e10a3be3433e1400422b22"}] >>> metric.update(preds, target) >>> fig_, ax_ = metric.plot() .. plot:: :scale: 75 >>> # Example plotting multiple values >>> from torchmetrics.text import SQuAD >>> metric = SQuAD() >>> preds = [{"prediction_text": "1976", "id": "56e10a3be3433e1400422b22"}] >>> target = [{"answers": {"answer_start": [97], "text": ["1976"]}, "id": "56e10a3be3433e1400422b22"}] >>> values = [ ] >>> for _ in range(10): ... values.append(metric(preds, target)) >>> fig_, ax_ = metric.plot(values) )_plot)r+r7r8s r-plotz SQuAD.plot~sTzz#r""r.)NN)__name__ __module__ __qualname____doc__rbool__annotations__rrrr)rrrr%r r r4dictstrr6rrrrrr; __classcell__)r,s@r-rr#s8t$t#!d!#t#!e!#e# Mee eJ Kc6k*K _c*#E&(6*:":;<*#IQRZI[*# *#r.r)collections.abcrtypingrrrr'r torchmetricsr"torchmetrics.functional.text.squadr r r r r torchmetrics.utilities.importsrtorchmetrics.utilities.plotrr__doctest_skip__rr#r.r-rLsE%'' A@$~E#FE#r.