i0e*ddlmZddlmZddlmZddlmZmZm Z ddl Z ddl m Z m Z ddl mZe dZed Zd ed ed eeee feee feee feee feee feee fffd Zdeded eefdZded eefdZded eefdZdeeded eeeeedfe fffdZded ed ededed eeeeeedfe ffeeeeedfe ffeee feee fff dZdeeeeedfe ffdeeeeedfe ffd eee ffdZdeee fdeee fd eee ffdZd eee fd!eee fd"eee fd#eee fd$eee fd%eee fd&e d'e d e fd(Z!d)eed*eeeeedfe ffd+eeeeedfe ffd,eee fd-eee fd ed ed&e d'e deded ee eee feee feee feee fffd.Z" d defaultdict)Sequence)chain)ListOptionalUnionN)Tensortensor)_validate_inputsgؗҜ?@[\]^_`{|}~ n_char_order n_word_orderreturnc t|Dcic]}|dztd}}t|Dcic]}|dztd}}t|Dcic]}|dztd}}t|Dcic]}|dztd}}t|Dcic]}|dztd}}t|Dcic]}|dztd}}||||||fScc}wcc}wcc}wcc}wcc}wcc}w)ahPrepare dictionaries with default zero values for total ref, hypothesis and matching character and word n-grams. Args: n_char_order: A character n-gram order. n_word_order: A word n-gram order. Return: Dictionaries with default zero values for total reference, hypothesis and matching character and word n-grams. )ranger ) r rntotal_preds_char_n_gramstotal_preds_word_n_gramstotal_target_char_n_gramstotal_target_word_n_gramstotal_matching_char_n_gramstotal_matching_word_n_gramss v/Volumes/fast/ai/experiments/voice-extract-mac/.venv/lib/python3.12/site-packages/torchmetrics/functional/text/chrf.py_prepare_n_grams_dictsr&s5 PUUaOb2cOb!1q5&+3EOb2cOTUaOb2cOb!1q5&+3EOb2cPUVbPc3dPc1AE6#;4FPc3dPUVbPc3dPc1AE6#;4FPc3dRWXdRe5fReQa!eVC[6HRe5fRWXdRe5fReQa!eVC[6HRe5f ! !!##  3d2c3d3d5f5fs"C'C,C1=C6"C;Dsentence whitespacecn|r t|St|jjddS)aSplit sentence into individual characters. Args: sentence: An input sentence to split. whitespace: An indication whether to keep whitespaces during character n-gram extraction. Return: A list of separated characters.  )liststripreplace)rrs r_get_charactersr%Gs0H~  ((b1 22wordc~t|dk(r|gS|dtvr |dd|dgS|dtvr |d|ddgS|gS)axSeparates out punctuation from beginning and end of words for chrF. Adapted from https://github.com/m-popovic/chrF and https://github.com/mjpost/sacrebleu/blob/master/sacrebleu/metrics/chrf.py. Args: word: An input word to be separated from a punctuation if present. Return: A list of a single word or a separated word and punctuation. rNr)len _PUNCTUATIONS)r's r_separate_word_and_punctuationr,Ws_ 4yA~v  Bx= Sb 48$$ Aw-Qab"" 6Mr&cttjd|jj DS)zSeparates out punctuation from beginning and end of words for chrF for all words in the sentence. Args: sentence: An input sentence to split Return: An aggregated list of separated words and punctuation. c32K|]}t|ywN)r,).0r's r z-_get_words_and_punctuation..xs#nUmT$B4$HUms)r"r from_iterabler#split)rs r_get_words_and_punctuationr4ns3 ###nU]UcUcUeUkUkUm#nn oor&char_or_word_list n_gram_order.ctd}td|dzD]Dfdttz dzDD]}||xxtdz cc<F|S)zCalculate n-gram counts. Args: char_or_word_list: A list of characters of words n_gram_order: The largest number of n-gram. Return: A dictionary of dictionaries with a counts of given n-grams. ctdS)NctdSNrr r&rz1_ngram_counts....s _efi_jr&rr<r&rr=z_ngram_counts..s ;WjKkr&rc3@K|]}t||zywr/)tuple)r0ir5rs rr1z _ngram_counts..s%iChae-a!a%89Chs)rrr*r )r5r6ngramsngramrs` @r _ngram_countsrC{sm8CCk7lF 1lQ& 'i5M^I_bcIcfgIgChiE 1Ie q ) j( Mr& lowercasecdtdtdtdtdttttttdft fftttttdft ffff fd }dtttttdft ffdttt ffd }|||||\}}||} ||} ||| | fS) aGet n-grams and total n-grams. Args: sentence: An input sentence n_char_order: A character n-gram order. n_word_order: A word n-gram order. lowercase: An indication whether to enable case-insensitivity. whitespace: An indication whether to keep whitespaces during character n-gram extraction. Return: char_n_grams_counts: A dictionary of dictionaries with sentence character n-grams. word_n_grams_counts: A dictionary of dictionaries with sentence word n-grams. total_char_n_grams: A dictionary containing a total number of sentence character n-grams. total_word_n_grams: A dictionary containing a total number of sentence word n-grams. rr rrDr.c|r|j}tt||}tt||}||fS)z@Get a dictionary of dictionaries with a counts of given n-grams.)lowerrCr%r4)rr rrDchar_n_grams_countsword_n_grams_countsrs r_char_and_word_ngrams_countszJ_get_n_grams_counts_and_total_ngrams.._char_and_word_ngrams_countssI ~~'H+OHj,QS_`+,Fx,PR^_"$777r&n_grams_countsctd}|D]=}t||jjj ||<?|S)z.Get total sum of n-grams over n-grams w.r.t n.ctdSr:r;r<r&rr=zQ_get_n_grams_counts_and_total_ngrams.._get_total_ngrams..svc{r&)rsumvaluesdetachclone)rK total_n_gramsrs r_get_total_ngramsz?_get_n_grams_counts_and_total_ngrams.._get_total_ngramssL+67J+K A">!#4#;#;#=>EEGMMOM!  r&)strintboolr?dictr ) rr rrDrrJrSrHrItotal_char_n_gramstotal_word_n_gramss ` r$_get_n_grams_counts_and_total_ngramsrZs288%(88;8HL8 tCeCHov5667c4cSVhY_H_C`>`9aa b8$sDsCx&9P4Q/Q*RW[\_ag\gWh0L, i0,,++>?*+>?  35GI[ [[r&hyp_n_grams_countsref_n_grams_countsc td}|D]^}||Dcgc]$}tj||||||&}}t|j j ||<`|Scc}w)zGet a number of n-gram matches between reference and hypothesis n-grams. Args: hyp_n_grams_counts: n-grams counts for hypothesis ref_n_grams_counts: n-grams counts for reference Return: matching_n_grams ctdSr:r;r<r&rr=z$_get_ngram_matches..sfSkr&)rtorchminrNrPrQ)r[r\matching_n_gramsrn_gram min_n_gramss r_get_ngram_matchesrds+66I*J btuvbw bwX^EII(+F35G5J65R Sbw  "+.557==?    s)A2rRn_gramsc4|D]}||xx||z cc<|S)aBAggregate total n-grams to keep corpus-level statistics. Args: total_n_grams: A dictionary containing a total corpus-level number of n-grams. n_grams: A dictionary containing a sentence-level number of n-grams. Return: A dictionary containing a total corpus-level number of n-grams. r<)rRrers r_sum_over_dictsrgs(aGAJ& r&matching_char_n_gramsmatching_word_n_gramshyp_char_n_gramshyp_word_n_gramsref_char_n_gramsref_word_n_gramsn_orderbetac Xdtttfdtttfdtttfdtdtttff d}|||||} |||||} t | j t | j zt |z S)aCalculate sentence-level chrF/chrF++ score. For given hypothesis and reference statistics (either sentence-level or corpus-level) the chrF/chrF++ score is returned. Args: matching_char_n_grams: A total number of matching character n-grams between the best matching reference and hypothesis. matching_word_n_grams: A total number of matching word n-grams between the best matching reference and hypothesis. hyp_char_n_grams: A total number of hypothesis character n-grams. hyp_word_n_grams: A total number of hypothesis word n-grams. ref_char_n_grams: A total number of reference character n-grams. ref_word_n_grams: A total number of reference word n-grams. n_order: A sum of character and word n-gram order. beta: A parameter determining an importance of recall w.r.t. precision. If `beta=1`, their importance is equal. Return: A chrF/chrF++ score. This function is universal both for sentence-level and corpus-level calculation. ra ref_n_grams hyp_n_gramsrorc |Dcic]!}|||dkDr ||||z n td#}}|Dcic]!}|||dkDr ||||z n td#}}|Dcic],}|tj|dz||z||zt.}}|Dcic]}|d|dzz||z||z||z }}|Scc}wcc}wcc}wcc}w)zGet n-gram level f-score.rrr)r r_max_EPS_SMOOTHING) rarqrrror precisionrecall denominatorf_scores r_get_n_gram_fscorez-_calculate_fscore.._get_n_gram_fscores> eu( dt_`A{1~7I"[^3vVY{ Zdt ( eu% dt_`A{1~7I"[^3vVY{ Zdt % Wg* VfQRAuyyq9Q</&);^L LVf * Sc& RbQAD!G y|+fQi7+a.H HRb & ( % * & s&B:&B?1C"C )rWrUr floatrNrOr ) rhrirjrkrlrmrnror{char_n_gram_f_scoreword_n_gram_f_scores r_calculate_fscorers@sF{+:>sF{:KZ^_bdj_jZksx c6k &--BDTVfhlm,-BDTVfhlm #**, -4G4N4N4P0Q QU[\cUd ddr&targetspred_char_n_grams_countspred_word_n_grams_countspred_char_n_gramspred_word_n_gramsc 0td} td} td} td}td}|D]P}t|||| | \}}}}t||}t||}t ||||||||}|| kDsG|} |} |} |}|}R| | | ||fS)aSCalculate the best sentence-level chrF/chrF++ score. For a given pre-processed hypothesis, all references are evaluated and score and statistics for the best matching reference is returned. Args: targets: An iterable of references. pred_char_n_grams_counts: A dictionary of dictionaries with hypothesis character n-grams. pred_word_n_grams_counts: A dictionary of dictionaries with hypothesis word n-grams. pred_char_n_grams: A total number of hypothesis character n-grams. pred_word_n_grams: A total number of hypothesis word n-grams. n_char_order: A character n-gram order. n_word_order: A word n-gram order. n_order: A sum of character and word n-gram order. beta: A parameter determining an importance of recall w.r.t. precision. If `beta=1`, their importance is equal. lowercase: An indication whether to enable case-insensitivity. whitespace: An indication whether to keep whitespaces during character n-gram extraction. Return: Return chrF/chrF++ score and statistics for the best matching hypothesis and reference. f_score: A sentence-level chrF/chrF++ score. matching_char_n_grams: A total number of matching character n-grams between the best matching reference and hypothesis. matching_word_n_grams: A total number of matching word n-grams between the best matching reference and hypothesis. target_char_n_grams: A total number of reference character n-grams. target_word_n_grams: A total number of reference word n-grams. rctdSr:r;r<r&rr=z6_calculate_sentence_level_chrf_score..Ls r&ctdSr:r;r<r&rr=z6_calculate_sentence_level_chrf_score..Mrr&ctdSr:r;r<r&rr=z6_calculate_sentence_level_chrf_score..NfSkr&ctdSr:r;r<r&rr=z6_calculate_sentence_level_chrf_score..Orr&)r rrZrdr)rrrrrr rrnrorDr best_f_scorebest_matching_char_n_gramsbest_matching_word_n_gramsbest_target_char_n_gramsbest_target_word_n_gramstargettarget_char_n_grams_countstarget_word_n_grams_countstarget_char_n_gramstarget_word_n_gramsrhrirzs r$_calculate_sentence_level_chrf_scorer sV#;L4?@S4T4?@S4T2=>Q2R2=>Q2R 1|U^`j k  & &   23MOg h 23MOg h# ! !         \ !"L)> &)> &': $': $5: ""   r&predsrrrrrrrsentence_chrf_scorect||\}}t||D]\}}t||| | | \}}}}t||}t||}t ||||||| | | | | \}}}}}| |j |j dt||}t||}t||}t||}|||||||fS)aUpdate function for chrf score. Args: preds: An iterable of hypothesis corpus. target: An iterable of iterables of reference corpus. total_preds_char_n_grams: A dictionary containing a total number of hypothesis character n-grams. total_preds_word_n_grams: A dictionary containing a total number of hypothesis word n-grams. total_target_char_n_grams: A dictionary containing a total number of reference character n-grams. total_target_word_n_grams: A dictionary containing a total number of reference word n-grams. total_matching_char_n_grams: A dictionary containing a total number of matching character n-grams between references and hypotheses. total_matching_word_n_grams: A dictionary containing a total number of total matching word n-grams between references and hypotheses. n_char_order: A character n-gram order. n_word_order: A word n-gram order. n_order: Sum of character and word n-gram order. beta: A parameter determining an importance of recall w.r.t. precision. If `beta=1`, their importance is equal. lowercase: An indication whether to enable case-insensitivity. whitespace: An indication whether to keep whitespaces during character n-gram extraction. sentence_chrf_score: A list of sentence-level chrF/chrF++ scores. Return: total_target_char_n_grams: number of reference character n-grams. total_target_word_n_grams: number of reference word n-grams. total_preds_char_n_grams: number of hypothesis character n-grams. total_preds_word_n_grams: number of hypothesis word n-grams. total_matching_char_n_grams: number of matching character n-grams between references and hypotheses. total_matching_word_n_grams: number of total matching word n-grams between references and hypotheses. sentence_chrf_score: A list of sentence-level chrF/chrF++ scores. Raises: ValueError: If length of ``preds`` and ``target`` differs. r)r ziprZrgrappend unsqueeze)rrrrrrrrr rrnrorDrr target_corpuspredrrrrrsentence_level_f_scorerhrirrs r_chrf_score_updatervs.x,FE:M5UM2 g 1|\S\^h i  $ $  #23KM^#_ #23KM^#_  1  $ $          " ! !    *  & &'='G'G'J K$34MOb$c!$34MOb$c!&56QSh&i#&56QSh&i#I3N ! !!## r&c &t||||||||S)akCompute chrF/chrF++ score based on pre-computed target, prediction and matching character and word n-grams. Args: total_preds_char_n_grams: number of hypothesis character n-grams. total_preds_word_n_grams: number of hypothesis word n-grams. total_target_char_n_grams: number of reference character n-grams. total_target_word_n_grams: number of reference word n-grams. total_matching_char_n_grams: number of matching character n-grams between references and hypotheses. total_matching_word_n_grams: number of total matching word n-grams between references and hypotheses. n_order: A sum of character and word n-gram order. beta: A parameter determining an importance of recall w.r.t. precision. If `beta=1`, their importance is equal. Return: A corpus-level chrF/chrF++ score. )r)rrrrrrrnros r_chrf_score_computers)6 ##  !!   r&return_sentence_level_scorect|tr|dkr tdt|tr|dkr td|dkr tdt||z}t ||\} } } } } }|rgnd}t ||| | | | | ||||||||\} } } } } }}t | | | | | |||}|r|tj|fS|S)uaCalculate `chrF score`_ of machine translated text with one or more references. This implementation supports both chrF score computation introduced in [1] and chrF++ score introduced in `chrF++ score`_. This implementation follows the implementations from https://github.com/m-popovic/chrF and https://github.com/mjpost/sacrebleu/blob/master/sacrebleu/metrics/chrf.py. Args: preds: An iterable of hypothesis corpus. target: An iterable of iterables of reference corpus. n_char_order: A character n-gram order. If `n_char_order=6`, the metrics refers to the official chrF/chrF++. n_word_order: A word n-gram order. If `n_word_order=2`, the metric refers to the official chrF++. If `n_word_order=0`, the metric is equivalent to the original chrF. beta: A parameter determining an importance of recall w.r.t. precision. If `beta=1`, their importance is equal. lowercase: An indication whether to enable case-insensitivity. whitespace: An indication whether to keep whitespaces during character n-gram extraction. return_sentence_level_score: An indication whether a sentence-level chrF/chrF++ score to be returned. Return: A corpus-level chrF/chrF++ score. (Optionally) A list of sentence-level chrF/chrF++ scores if `return_sentence_level_score=True`. Raises: ValueError: If ``n_char_order`` is not an integer greater than or equal to 1. ValueError: If ``n_word_order`` is not an integer greater than or equal to 0. ValueError: If ``beta`` is smaller than 0. Example: >>> from torchmetrics.functional.text import chrf_score >>> preds = ['the cat is on the mat'] >>> target = [['there is a cat on the mat', 'a cat is on the mat']] >>> chrf_score(preds, target) tensor(0.8640) References: [1] chrF: character n-gram F-score for automatic MT evaluation by Maja Popović `chrF score`_ [2] chrF++: words helping character n-grams by Maja Popović `chrF++ score`_ rzMExpected argument `n_char_order` to be an integer greater than or equal to 1.rzMExpected argument `n_word_order` to be an integer greater than or equal to 0.z.Expected argument `beta` to be greater than 0.N) isinstancerU ValueErrorr|rrrr_cat)rrr rrorDrrrnrrrrrrr chrf_f_scores r chrf_scorer s'n lC (L1,<hii lC (L1,<hii axIJJL</0G |\:  !!##9T"Y]    !!##    !!##&'  !!##  LUYY':;;; r&r/)rtg@FFF)& collectionsrcollections.abcr itertoolsrtypingrrr r_r r #torchmetrics.functional.text.helperr rvsetr+rUr?rWrrTrVr"r%r,r4rCrZrdrgr|rrrrrr<r&rrs.$$(( @89 %( ftCK($sF{*;T#v+=NPTUXZ`U`PacghkmshscttB 3c 3t 3S 3 c. p pc pT#Ycd3PTUZ[^`c[cUdflUlPmKmFn$0\0\!$0\470\DH0\VZ0\ d5c?F*+ +,d5c?F*+ +,ff0\fS$uS#X'>"??@S$uS#X'>"??@ #v+. 4V #4 tCK?P UYZ]_eZeUf  6eV ,6eV ,6e3;'6e3;' 6e 3;' 6e 3;' 6e6e 6e 6erS #YS"3U38_f-D(E#EFS#3U38_f-D(E#EFSCK( S CK( S  SSS SSS 64V $d3;&7c6k9JDQTV\Q\L] ]^SJ37l hsm# $l (3-(3-!88 9l#3;/l#3;/ l $CK0 l $CK0 l"&c6k!2l"&c6k!2llll lll"$v,/l  ffffff T&\ !l^$"3;/$"3;/$ $CK0$ $CK0 $ "&c6k!2 $ "&c6k!2 $$ $ $T(-r hsm# $r U3 -. /rr r  r  rr"&r 65( ()rr&