ilB ^ddlZddlmZddlmZmZddlmZdZdZ e dZ eGdd e eZ Gd d Zd eee eee fd ee ee fdeeee ee ffdZdee dee de fdZdee dfdee dffdZdee dfdeee e fee ee ffdZy)N)Sequence)Enumunique)Unioni'g7yACc$eZdZdZdZdZdZdZdZy)_EditOperationsz1Enumerations for the Levenhstein edit operations.insertdelete substitutenothing undefinedN) __name__ __module__ __qualname____doc__ OP_INSERT OP_DELETE OP_SUBSTITUTE OP_NOTHING OP_UNDEFINEDx/Volumes/fast/ai/experiments/voice-extract-mac/.venv/lib/python3.12/site-packages/torchmetrics/functional/text/helper.pyr r ,s;II MJLrr ceZdZdZ ddeededededdf dZd eedeeee d fffd Z d eed ed eeeee fdeeeeeee fee d fffdZ dedeeeee fdee d ffdZ d eedeeeee fddfdZ d eedeeeeeee fffdZdedeeee ffdZdedeeee ffdZy)_LevenshteinEditDistancea.A convenience class for calculating the Levenshtein edit distance. Class will cache some intermediate values to hasten the calculation. The implementation follows the implementation from https://github.com/mjpost/sacrebleu/blob/master/sacrebleu/metrics/lib_ter.py, where the most of this implementation is adapted and copied from. Args: reference_tokens: list of reference tokens op_insert: cost of insertion operation op_delete: cost of deletion operation op_substitute: cost of substitution operation reference_tokens op_insert op_delete op_substitutereturnNc||_t||_i|_d|_||_||_||_d|_t|_ y)Nr) rlen reference_lencache cache_sizerrr op_nothing _INT_INFINITY op_undefined)selfrrrr s r__init__z!_LevenshteinEditDistance.__init__FsM!1 !1213 ""*)rprediction_tokens.c|j|\}}|j|||\}}}|j||||fS)a(Calculate edit distance between self._words_ref and the hypothesis. Uses cache to skip some computations. Args: prediction_tokens: A tokenized predicted sentence. Return: A tuple of a calculated edit distance and a trace of executed operations. ) _find_cache_levenshtein_edit_distance _add_cache)r*r,start_positioncached_edit_distanceedit_distance_int edit_distancetraces r__call__z!_LevenshteinEditDistance.__call__UsW04/?/?@Q/R,,262Q2Q ~/C3 /=% )=9 %''rprediction_startr%ct|}t||z Dcgc]&}t|j|j(}}||z}|r|j|z nd}|dz t kDrt j|dz t znt } t|dz|dzD]} t j| |z} td| | z } | |k(r|jdznt|jdz| | z} t| | D]}|dk(r2|| dz |d|jztjf|| |<;|| dz |j|dz k(r|j}tj }n|j"}tj$}|| dz |dz d|z|f|| dz |d|jztjf|| |dz d|j&ztj(ff}|D]\}}|| |d|kDs||f|| |< |j+||}|ddd|t|d|fScc}w)aDynamic programming algorithm to compute the Levenhstein edit distance. Args: prediction_tokens: A tokenized predicted sentence. prediction_start: An index where a predicted sentence to be considered from. cache: A cached Levenshtein edit distance. Returns: Edit distance between the predicted sentence and the reference sentence g?rN)r#rangelist_get_empty_rowr$ _BEAM_WIDTHmathceilfloormaxminrr rrr'rr rrr _get_trace)r*r,r7r%prediction_len_ empty_rowsr4 length_ratio beam_widthi pseudo_diagmin_jmax_jjcost_substituteoperation_substitute operationsoperation_costoperation_namer5s rr/z3_LevenshteinEditDistance._levenshtein_edit_distancejs"./DIZjIjCk? CkaD$$T%7%78 9Ck ? BGAS >Ot))N:UX COQRBRU`B`TYY|a/+=>fq '!+^a-?@A**Q%56K;34E*+~*=""Q&3tGYGY\]G]_jmw_wCx 5%(6%a!e,Q/2T^^C'11+M!$Q' )Q/43H3HQ3OO*.///>/I/I,*.*<*</>/L/L,'q1u-a!e4Q7/IK_`&q1u-a03dnnDoF_F_`&q)!a%03dnnDoF_F_`"J ;E6(+A.q1NB2@.2PM!,Q/;E1)AF >R $Q's5z|)DeKK_? s+IrFr4cNd}|}|j}|dkDs|dkDr|||d}|g|}|tjtjfvr |dz}|dz}n@|tjk(r|dz}n'|tj k(r|dz}nt d||dkDr|dkDr|S)a@Get a trace of executed operations from the edit distance matrix. Args: prediction_len: A length of a tokenized predicted sentence. edit_distance: A matrix of the Levenshtedin edit distance. The element part of the matrix is a tuple of an edit operation cost and an edit operation itself. Return: A trace of executed operations returned as a tuple of `_EDIT_OPERATIONS` enumerates. Raises: ValueError: If an unknown operation has been applied. rrr:Unknown operation )r$r rrrr ValueError)r*rFr4r5rKrO operations rrEz#_LevenshteinEditDistance._get_traces&.0    !eq1u%a(+A.I''E_::O>xA)!,-a0D!.xy9=IID#4 %*-T 1$JE8D Jrc|j}d}|j|jg}|D])}||vr|dz }||\}}|j|&||fS||fS)a+Find the already calculated rows of the Levenshtein edit distance metric. Args: prediction_tokens: A tokenized predicted sentence. Return: A tuple of a start hypothesis position and `edit_distance` matrix. prediction_start: An index where a predicted sentence to be considered from. edit_distance: A matrix of the cached Levenshtedin edit distance. The element part of the matrix is a tuple of an edit operation cost and an edit operation itself. rr:)r%_get_initial_rowr$append)r*r,r]r1r4r_r`s rr.z$_LevenshteinEditDistance._find_caches}zzBFBWBWX\XjXjBkAl %Dt|!# J c$$S)},,&},,rlengthcZt|jtjfg|dzzS)aPrecomputed empty matrix row for Levenhstein edit distance. Args: length: A length of a tokenized sentence. Return: A list of tuples containing infinite edit operation costs and yet undefined edit operations. r:)intr)r r)r*res rr>z'_LevenshteinEditDistance._get_empty_rows-T&&')E)EFG6TU:VVrc~t|dzDcgc]!}||jztjf#c}Scc}w)a*First row corresponds to insertion operations of the reference, so 1 edit operation per reference word. Args: length: A length of a tokenized sentence. Return: A list of tuples containing edit operation costs of insert and insert edit operations. r:)r<rr r)r*rerKs rrcz)_LevenshteinEditDistance._get_initial_rows>JOvXYzIZ[IZAT^^#_%>%>?IZ[[[s&:)r:r:r:)rrrrr=strrgr+r\r r6r/rEr0r.r>rcrrrrr7s ij * $S  *69 *JM *be *  *($s)(c5Z]I]C^>^8_(*BL9BLBLDsO3456 BL sDeC$89:;U?TWCW=XX Y BLH$!$26tE#BVt|tr|g}td|Dr"t|dk(r|gn|Dcgc]}|gc}}|rLtd|Dr:t|t|k7r#t dt|dt|||fScc}w)aCheck and update (if needed) the format of reference and hypothesis corpora for various text evaluation metrics. Args: ref_corpus: An iterable of iterables of reference corpus. hypothesis_corpus: An iterable of hypothesis corpus. Return: ref_corpus: An iterable of iterables of reference corpus. hypothesis_corpus: An iterable of hypothesis corpus. Raises: ValueError: If length of `ref_corpus` and `hypothesis_corpus` differs. c3<K|]}t|tywN) isinstanceri.0refs r z#_validate_inputs..As 6:C:c3 :sr:c3 K|]}|ywrnrrps rrsz#_validate_inputs..Ds ;  s zCorpus has different size z != )roriallr#rW)rjrkrrs r_validate_inputsrv*s&#S)./ 6: 66%():%;q%@j\XbFcXbQTuXbFc S ; ;;JSVWhSi@i5c*o5Fd3O`KaJbcdd ( (( Gds Br,rc <tt|dzDcgc]}dgt|dzz}}tt|dzD] }|||d< tt|dzD] }||d|< tdt|dzD]y}tdt|dzD]\}||dz ||dz k(r||dz |dz |||<)t||dz ||||dz ||dz |dz dz|||<^{|ddScc}w)aDynamic programming algorithm to compute the edit distance. Args: prediction_tokens: A tokenized predicted sentence reference_tokens: A tokenized reference sentence Returns: Edit distance between the predicted sentence and the reference sentence r:rr;)r<r#rD)r,rrGdprKrOs r_edit_distanceryJsV6;3?P;QTU;U5V W5V1#%&* +5VB W 3()A- .1a/ 3'(1, -1a. 1c+,q0 1q#./!34A Q'+;AE+BBa!e9QU+1ar!a%y|RU1q5\2a!e9QU;KLqP1a 52 b6": XsDr5.ctjtjtjtjidtdtttfdtfdt fd|DS)aFlip the trace of edit operations. Instead of rewriting a->b, get a recipe for rewriting b->a. Simply flips insertions and deletions. Args: trace: A tuple of edit operations. Return: inverted_trace: A tuple of inverted edit operations. rX_flip_operationsr!c0||vr|j|S|Srn)get)rXr{s r_replace_operation_or_retainz1_flip_trace.._replace_operation_or_retaints$ ( (#'' 2 2rc30K|] }|ywrnr)rqrXr{r~s rrsz_flip_trace..{sb\ay-i9IJ\as)r rrdictr\)r5r{r~s @@r _flip_tracerbsk !!?#<#<!!?#<#<@ "6:?O;[6\  b\ab bbrcdx}}g}g}i}|D]}|tjk(r2|dz }|dz }|||<|jd|jdH|tjk(r2|dz }|dz }|||<|jd|jd|tjk(r|dz }|jd|tj k(r|dz }|||<|jdt d|d|||fS)aTransform trace of edit operations into an alignment of the sequences. Args: trace: A trace of edit operations as a tuple of `_EDIT_OPERATIONS` enumerates. Return: alignments: A dictionary mapping aligned positions between a reference and a hypothesis. reference_errors: A list of error positions in a reference. hypothesis_errors: A list of error positions in a hypothesis. Raises: ValueError: If an unknown operation is r;r:rrV.)r rrdrrrrW)r5reference_positionhypothesis_positionreference_errorshypothesis_errors alignmentsrXs r_trace_to_alignmentr~sB 021,"$#%!#J 22 2 1 $  ! # -@J) *  # #A &  $ $Q ' /77 7 1 $  ! # -@J) *  # #A &  $ $Q ' /33 3 1 $   $ $Q ' /33 3 ! # -@J) *  # #A &1)a@A A+. '): ::r)r@collections.abcrenumrrtypingrr?rZrgr(rir rr\rvr=ryrrrrrrrs>> $ D c4p\p\f)hsmXhsm%<<=)S(3-/0) 8HSM "HSM 12)@d3i49QT0cu_c12cu_c=Q7Rc8-;u_c%9:-;uT#s(^UYZ]U^`deh`iEi?j-;r