iDddlZddlZddlmZddlmZddlmZmZm Z ddl m Z m Z m Z ddlmZddlmZded ed efd Z d#d ed ededededed efdZded efdZded efdZdee d e fdZde eeefdee eeefdedd ee eeefee eeefffdZ d#dede eeefdedededed e fdZ d$de eeefdee eeefdeddededededeee d ee fd Z d%de eeefdee eeefdedd!ededededed e e ee e fffd"Zy)&N)Sequence)inf)ListOptionalUnion)Tensorstacktensor)Literal)_validate_inputs preds_word target_wordreturnct||k7S)a.Distance measure used for substitutions/identity operation. Code adapted from https://github.com/rwth-i6/ExtendedEditDistance/blob/master/EED.py. Args: preds_word: hypothesis word string target_word: reference word string Return: 0 for match, 1 for no match )int)r rs u/Volumes/fast/ai/experiments/voice-extract-mac/.venv/lib/python3.12/site-packages/torchmetrics/functional/text/eed.py_distance_between_wordsres z[( ))hyprefalpharhodeletion insertionc dgt|dzz}dgt|dzz}d|d<tgt|dzz}tdt|dzD]} tt|dzD]Q} | dkDr?t|| dz |z|| dz t || dz || dz z|| |z|| <G|| dz|| <S|j t|} || xxdz cc<|| dz dk(r!||| z} |D cgc]} t| | }} |}tgt|dzz}|t d|Dz}td|d|ztt||zz Scc} w)a8Compute extended edit distance score for two lists of strings: hyp and ref. Code adapted from: https://github.com/rwth-i6/ExtendedEditDistance/blob/master/EED.py. Args: hyp: A hypothesis string ref: A reference string alpha: optimal jump penalty, penalty for jumps between characters rho: coverage cost, penalty for repetition of characters deletion: penalty for deletion of character insertion: penalty for insertion or substitution of character Return: Extended edit distance score as float ?r c3.K|] }|dk\r|ndyw)rrN).0xs r z _eed_function..sF5Ea1f!+5Es)lenrrangeminrindexsumfloat)rrrrrrnumber_of_visitsrownext_rowwi min_indexjumpr$coverages r _eed_functionr4us.ts3x!|, %3s8a< C CFuC1 %H 1c#hl #s3x!|$A1u!QUOh.AJ!8QUSQZ!PPFY& "!fsl %NN3x=1 #q(# q1u: 8I..D.67hAt hH75CHqL))$,SF5EFFFH q3r7X%%C/H*DE FF8sE2sentencecLt|tstdt|d|j }gd}|D]\}}|j ||}gd}|D]\}}t j|||}gd}|D]\}}|j ||}d|zdzS)zPreprocess english sentences. Copied from https://github.com/rwth-i6/ExtendedEditDistance/blob/master/util.py. Raises: ValueError: If input sentence is not of a type `str`. 6Only strings allowed during preprocessing step, found  instead)).z .)!z !)?z ?),z ,))z\s+r )z(\d) ([.,]) (\d)z\1\2\3)z#(Dr|Jr|Prof|Rev|Gen|Mr|Mt|Mrs|Ms) .z\1.))ze . g .ze.g.)zi . e .zi.e.)zU . S .zU.S.r ) isinstancestr ValueErrortyperstripreplaceresub)r5rules_interpunctionpattern replacementrules_res r_preprocess_enrIs h $QRVW_R`Qaaijkk H !4##G[9!4H !)66';9!) !4##G[9!4 >C rct|tstdt|d|j }t j d|S)zPreprocess japanese sentences. Copy from https://github.com/rwth-i6/ExtendedEditDistance/blob/master/util.py. Raises: ValueError: If input sentence is not of a type `str`. r7r8NFKC)r=r>r?r@rA unicodedata normalize)r5s r_preprocess_jarNsK h $QRVW_R`Qaaijkk H   22rsentence_level_scoresctt|dk(r tdSt|tt|z S)zReduction for extended edit distance. Args: sentence_level_scores: list of sentence-level scores as floats Return: average of scores as a tensor rr)r&r r*)rOs r _eed_computerQs8 !Q&c{ $ %s3H/I(J JJrpredstargetlanguage)enjac  t||\}}|dk(rt}n|dk(rt}ntd||Dcgc] }|| }}|Dcgc]}|Dcgc] }|| c}}}}||fScc}wcc}wcc}}w)auPreprocess strings according to language requirements. Args: preds: An iterable of hypothesis corpus. target: An iterable of iterables of reference corpus. language: Language used in sentences. Only supports English (en) and Japanese (ja) for now. Defaults to en Return: Tuple of lists that contain the cleaned strings for target and preds Raises: ValueError: If a different language than ``'en'`` or ``'ja'`` is used ValueError: If length of target not equal to length of preds ValueError: If objects in reference and hypothesis corpus are not strings )hypothesis_corpus ref_corpusrUrVz?Expected argument `language` to either be `en` or `ja` but got )r rIrNr?)rRrSrTpreprocess_functionpred referencers r_preprocess_sentencesr]s,%uPMFE4, T ,Z[cZdeff38 954  &5E 9OU Vv)9=9C"3'9=vF V &= := VsA5 A?A: )A?:A? target_wordsc bt}|D]}t||||||}||ks|}t|S)aCompute scores for ExtendedEditDistance. Args: target_words: An iterable of reference words preds_word: A hypothesis word alpha: An optimal jump penalty, penalty for jumps between characters rho: coverage cost, penalty for repetition of characters deletion: penalty for deletion of character insertion: penalty for insertion or substitution of character Return: best_score: best (lowest) sentence-level score as a Tensor )rr4r ) r r^rrrr best_scorer\scores r_compute_sentence_statisticsrb#sA,J! j)UC9U : J" * r sentence_eedc t|||\}}|g}dt|t|dfvr|St||D]&\}} t|| ||||} |j | (|S)aCompute scores for ExtendedEditDistance. Args: preds: An iterable of hypothesis corpus target: An iterable of iterables of reference corpus language: Language used in sentences. Only supports English (en) and Japanese (ja) for now. Defaults to en alpha: optimal jump penalty, penalty for jumps between characters rho: coverage cost, penalty for repetition of characters deletion: penalty for deletion of character insertion: penalty for insertion or substitution of character sentence_eed: list of sentence-level scores Return: individual sentence scores as a list of Tensors r)r]r&ziprbappend) rRrSrTrrrrrc hypothesisr^ras r _eed_updaterhCs4*%BME6  SZVAY (($'v$6 L,ZucS[]fgE"%7 rreturn_sentence_level_scorec tgd||||gD]:\}} t| trt| ts'| dks-td|dt |||||||} t | } |r | t | fS| S)uXCompute extended edit distance score (`ExtendedEditDistance`_) [1] for strings or list of strings. The metric utilises the Levenshtein distance and extends it by adding a jump operation. Args: preds: An iterable of hypothesis corpus. target: An iterable of iterables of reference corpus. language: Language used in sentences. Only supports English (en) and Japanese (ja) for now. Defaults to en return_sentence_level_score: An indication of whether sentence-level EED score is to be returned. alpha: optimal jump penalty, penalty for jumps between characters rho: coverage cost, penalty for repetition of characters deletion: penalty for deletion of character insertion: penalty for insertion or substitution of character Return: Extended edit distance score as a tensor Example: >>> from torchmetrics.functional.text import extended_edit_distance >>> preds = ["this is the prediction", "here is an other sample"] >>> target = ["this is the reference", "here is another one"] >>> extended_edit_distance(preds=preds, target=target) tensor(0.3078) References: [1] P. Stanchev, W. Wang, and H. Ney, “EED: Extended Edit Distance Measure for Machine Translation”, submitted to WMT 2019. `ExtendedEditDistance`_ )rrrrrz Parameter `z)` is expected to be a non-negative float.)rer=r+r?rhrQr ) rRrSrTrirrrr param_nameparamrOaverages rextended_edit_distancernmsP!!JUTWYaclLmn E%'Jue,DQR{:,6_`a ao(vxXW`a01G"3444 Nr)@333333?皙?r)rUrorprqrN)rUFrorprqr) rCrLcollections.abcrmathrtypingrrrtorchrr r typing_extensionsr #torchmetrics.functional.text.helperr r>rrr+r4rIrNrQtupler]rbrhboolrnr"rrrzs r $((''%@ * *# *# *& 7G 7G 7G 7G  7G  7G  7G 7Gt* S* S* Z3S3S3" KV K K # hsm# $# U3 -. /#j!# 5hsm# $huS(3-5G/H&I IJ #R Xc]*+        F%)+/' hsm# $' U3 -. /'j!'  '  '  ''4<(' &\'Z%)(-2 hsm# $2 U3 -. /2j!2"& 2  2  222 65( ()2r