i*ddlZddlmZddlmZmZmZmZmZddl Z ddl m Z ddl m Z ddl mZmZmZmZmZmZddlmZddlmZdd lmZmZdd lmZmZesd gZesd d gZGd d eZy)N)Sequence)AnyClassVarListOptionalUnion)Tensor)_load_tokenizer_and_model)$_ALLOWED_INFORMATION_MEASURE_LITERAL_get_dataloader_get_special_tokens_map_infolm_compute_infolm_update_InformationMeasure)Metric) dim_zero_cat)_MATPLOTLIB_AVAILABLE_TRANSFORMERS_GREATER_EQUAL_4_4)_AX_TYPE_PLOT_OUT_TYPEz InfoLM.plotInfoLMceZdZUdZdZeeed<eeed<eeed<eeed<dddddddddd Ze ed < d"d e e e jfd ed ededeedeedee e ej&fdeedededededee efdd ffd ZedefdZde e ee fde e ee fdd fdZde eeeefffdZ d#dee eeefd eedefd!Z xZ!S)$ru Calculate `InfoLM`_. InfoLM measures a distance/divergence between predicted and reference sentence discrete distribution using one of the following information measures: - `KL divergence`_ - `alpha divergence`_ - `beta divergence`_ - `AB divergence`_ - `Rényi divergence`_ - L1 distance - L2 distance - L-infinity distance - `Fisher-Rao distance`_ `InfoLM`_ is a family of untrained embedding-based metrics which addresses some famous flaws of standard string-based metrics thanks to the usage of pre-trained masked language models. This family of metrics is mainly designed for summarization and data-to-text tasks. The implementation of this metric is fully based HuggingFace ``transformers``' package. As input to ``forward`` and ``update`` the metric accepts the following input: - ``preds`` (:class:`~Sequence`): An iterable of hypothesis corpus - ``target`` (:class:`~Sequence`): An iterable of reference corpus As output of ``forward`` and ``compute`` the metric returns the following output: - ``infolm`` (:class:`~torch.Tensor`): If `return_sentence_level_score=True` return a tuple with a tensor with the corpus-level InfoLM score and a list of sentence-level InfoLM scores, else return a corpus-level InfoLM score Args: model_name_or_path: A name or a model path used to load ``transformers`` pretrained model. By default the `"bert-base-uncased"` model is used. temperature: A temperature for calibrating language modelling. For more information, please reference `InfoLM`_ paper. information_measure: A name of information measure to be used. Please use one of: ['kl_divergence', 'alpha_divergence', 'beta_divergence', 'ab_divergence', 'renyi_divergence', 'l1_distance', 'l2_distance', 'l_infinity_distance', 'fisher_rao_distance'] idf: An indication of whether normalization using inverse document frequencies should be used. alpha: Alpha parameter of the divergence used for alpha, AB and Rényi divergence measures. beta: Beta parameter of the divergence used for beta and AB divergence measures. device: A device to be used for calculation. max_length: A maximum length of input sequences. Sequences longer than ``max_length`` are to be trimmed. batch_size: A batch size used for model processing. num_threads: A number of threads to use for a dataloader. verbose: An indication of whether a progress bar to be displayed during the embeddings calculation. return_sentence_level_score: An indication whether a sentence-level InfoLM score to be returned. Example: >>> from torchmetrics.text.infolm import InfoLM >>> preds = ['he read the book because he was interested in world history'] >>> target = ['he was interested in world history because he read the book'] >>> infolm = InfoLM('google/bert_uncased_L-2_H-128_A-2', idf=False) >>> infolm(preds, target) tensor(-0.1784) Fpreds_input_idspreds_attention_masktarget_input_idstarget_attention_maskT) kl_divergencealpha_divergencebeta_divergence ab_divergencerenyi_divergence l1_distance l2_distancel_infinity_distancefisher_rao_distance%_information_measure_higher_is_betterNmodel_name_or_path temperatureinformation_measureidfalphabetadevice max_length batch_size num_threadsverbosereturn_sentence_level_scorekwargsreturnc  ht|di| ||_||_||_||_||_||_tj|xsd|_ | |_ | |_ | |_ | |_t||\|_|_t%||||_|xs |j"j(j*|_t-|j |_|j1dgd|j1dgd|j1dgd|j1dgdy) Ncpurcat)dist_reduce_fxrrr)super__init__r'r(r)r*r+r,torchr-_devicer/r0r1r2r tokenizermodelrinformation_measure_clsconfigr.r special_tokens_map add_state)selfr'r(r)r*r+r,r-r.r/r0r1r2r3 __class__s m/Volumes/fast/ai/experiments/voice-extract-mac/.venv/lib/python3.12/site-packages/torchmetrics/text/infolm.pyr;zInfoLM.__init__s "6""4&#6   ||FOe4 $& +F(%>?QSY%Z" ':;NPUW['\$$D (9(9(D(D"9$.."I ("UC -r%H )2eD .5Ic4|j|jS)zReturns a bool indicating whether a higher value of the information measure is better. Done this way as depends on if the information measure is positive or negative. )r&r))rDs rFhigher_is_betterzInfoLM.higher_is_betters99$:R:RSSrGpredstargetc*t|||j|j\}}}}|jj ||j j ||j j ||jj |y)z*Update state with predictions and targets.N)rr>r.rappendrrr)rDrJrKrrrrs rFupdatez InfoLM.updatesYg 64>>4??Z V-/?AV ##O4 !!(()=> $$%56 ""))*?@rGc Jtt|jt|j|j|j |j }tt|jt|j|j|j |j }t|j|||j|j|j|j|j}|jr|j!|fS|j!S)zLCalculate selected information measure using the pre-trained language model.) input_idsattention_maskr*r/ num_workers)r rrrr*r/r0rrrr?r(r@rBr1r2mean)rDpreds_dataloadertarget_dataloader info_lm_scores rFcomputezInfoLM.computes*"4#7#78'(A(AB((  ,"4#8#89'(B(BC((  ( JJ      HH  ( (  # # LL    + + %%'6 6!!##rGvalaxc&|j||S)a1Plot a single or multiple values from the metric. Args: val: Either a single result from calling `metric.forward` or `metric.compute` or a list of these results. If no value is provided, will automatically call `metric.compute` and plot that result. ax: An matplotlib axis object. If provided will add plot to that axis Returns: Figure and Axes object Raises: ModuleNotFoundError: If `matplotlib` is not installed .. plot:: :scale: 75 >>> # Example plotting a single value >>> from torchmetrics.text.infolm import InfoLM >>> metric = InfoLM('google/bert_uncased_L-2_H-128_A-2', idf=False) >>> preds = ['he read the book because he was interested in world history'] >>> target = ['he was interested in world history because he read the book'] >>> metric.update(preds, target) >>> fig_, ax_ = metric.plot() .. plot:: :scale: 75 >>> # Example plotting multiple values >>> from torchmetrics.text.infolm import InfoLM >>> metric = InfoLM('google/bert_uncased_L-2_H-128_A-2', idf=False) >>> preds = ["this is the prediction", "there is an other sample"] >>> target = ["this is the reference", "there is another one"] >>> values = [ ] >>> for _ in range(10): ... values.append(metric(preds, target)) >>> fig_, ax_ = metric.plot(values) )_plot)rDrXrYs rFplotz InfoLM.plotsTzz#r""rG) zbert-base-uncasedg?rTNNNN@rTF)NN)"__name__ __module__ __qualname____doc__is_differentiablerr __annotations__r&rrstrosPathLikefloatr boolrr<r-intdictrr;propertyrIrrNtuplerWrrr\ __classcell__)rEs@rFrr*sEN&\!v,&6l"<'  !$$ 7)8  7J!DS!% $59$(,1%J!#r{{"23%J%JB %J  %J  %Juo%JsELL012%JSM%J%J%J%J&*%JsCx.%J %JNT$TTAE#x}"45AuS(SV-EW?XA]aA$vuVV^'<<=$D_c*#E&(6*:":;<*#IQRZI[*# *#rG) recollections.abcrtypingrrrrrr<r 4torchmetrics.functional.text.helper_embedding_metricr #torchmetrics.functional.text.infolmr r r rrrtorchmetrics.metricrtorchmetrics.utilities.datartorchmetrics.utilities.importsrrtorchmetrics.utilities.plotrr__doctest_skip__rr9rGrFrws[ $77 Z'4a@%& -0a#Va#rG