i&ddlZddlZddlmZddlmZmZmZddlm Z m Z ddl m Z e eefZeeeefZe eeee eeeeeeffffZeeeefZeee eeeeffee eeeeeefffZdgdgdd d d d d ZdedefdZdedeefdZdedede fdZdedede fdZdeeege fdedeede fdZdededee eefee eee eee eefffffdZde eefdee eee eee eefffdee e e ffdZd e d!e d"e de ee ffd#Zdedede ee ffd$Z y)%N)Counter)AnyCallableUnion)Tensortensor)rank_zero_warnzThis is a test text) answer_starttextzThis is a test context.1zIs this a test?z train test)answerscontextidquestiontitlesreturnc dtdtfd}dtdtfd}dtdtfd}dtdtfd}|||||S)zALower text and remove punctuation, articles and extra whitespace.r rc0tjdd|S)Nz\b(a|an|the)\b )resubr s w/Volumes/fast/ai/experiments/voice-extract-mac/.venv/lib/python3.12/site-packages/torchmetrics/functional/text/squad.pyremove_articlesz(_normalize_text..remove_articles,svv'd33c@dj|jS)Nr)joinsplitrs rwhite_space_fixz(_normalize_text..white_space_fix/sxx %%rcjttjdjfd|DS)Nc3,K|] }|vs| ywN).0chexcludes r z7_normalize_text..remove_punc..4s>DbBg,=rDs )setstring punctuationr)r r)s @r remove_puncz$_normalize_text..remove_punc2s(f(()ww>D>>>rc"|jSr%)lowerrs rr0z_normalize_text..lower6szz|r)str)rrr!r.r0s r_normalize_textr2)si4c4c4&c&c&?#?#?CC ?;uQx+@A BBrc<|sgSt|jS)z&Split a sentence into separate tokens.)r2r )rs r _get_tokensr4<s26OA.4466rpredicted_answer target_answerct|}t|}t|t|z}tt|j }t |dk(st |dk(rtt ||k(S|dk(r tdSd|ztt |z }d|ztt |z }d|z|z||zz S)z#Compute F1 Score for two sentences.rg?)r4rrsumvalueslenint)r5r6 target_tokenspredicted_tokenscommonnum_same precisionrecalls r_compute_f1_scorerDAs .M"#34 ] #g.>&? ?Fc&--/*+H =Q#&6"71"<c-+;;<==1}c{h,<(=!>>I 8^fS%78 8F MF "y6'9 ::r prediction ground_truthcTttt|t|k(S)z&Compute Exact Match for two sentences.)rr=r2)rErFs r_compute_exact_match_scorerHQs! #oj1_\5RRS TTr metric_fn ground_truthsc0tfd|DS)zJCalculate maximum score for a predicted answer with all reference answers.c30K|] }|ywr%r&)r'truthrIrEs rr*z1_metric_max_over_ground_truths..ZsGyU+s)max)rIrErJs`` r_metric_max_over_ground_truthsrOVs GG GGrpredstargetsct|tr|g}t|tr|g}|D]%}|j}d|vsd|vstd|D]G}|j}d|vsd|vrtdt|d}d|vs7tdt|Dcic] }|d|d }}d} d d |Dcgc] }| | c}igig} || fScc}wcc}w) zOCheck for types and convert the input to necessary format to compute the input.prediction_textrzExpected keys in a single prediction are 'prediction_text' and 'id'.Please make sure that 'prediction_text' maps to the answer string and 'id' maps to the key string.rzExpected keys in a single target are 'answers' and 'id'.Please make sure that 'answers' maps to a `SQuAD` format dictionary and 'id' maps to the key string. SQuAD Format: r zzExpected keys in a 'answers' are 'text'.Please make sure that 'answer' maps to a `SQuAD` format dictionary. SQuAD Format: cH|ddDcgc]}d|ic}|ddScc}w)Nrr r)rrr&)tgttxts rz$_squad_input_check..s23y>RXCY)ZCYC63-CY)Zbefjbkl)Zs  paragraphsqas) isinstancedictkeysKeyError SQuAD_FORMAT) rPrQpred pred_keystarget target_keysrrE preds_dict _fn_answer targets_dicts r_squad_input_checkrf]s@%'4 )IIK I -Y1Fu kkm K '4{+B! ." ;A:K  ! ." &UZZTYj*T"J/@$AATYJZlJ!Ug,VgFZ-?g,V$W#XYZL | ##[,Vs !C<Crac Vtd}td}td}|D]z}|dD]p}|dD]f}|dz }|d|vrtd|dd"|d Dcgc]}|d  } }||d} |tt| | z }|tt| | z }hr||||fScc}w) auCompute F1 Score and Exact Match for a collection of predictions and references. Args: preds: A dictionary mapping an `id` to the predicted `answer`. target: A list of dictionary mapping `paragraphs` to list of dictionary mapping `qas` to a list of dictionary containing `id` and list of all possible `answers`. Return: Tuple containing F1 score, Exact match score and total number of examples. Example: >>> from torchmetrics.functional.text.squad import _squad_update >>> preds = [{"prediction_text": "1976", "id": "56e10a3be3433e1400422b22"}] >>> target = [{"answers": {"answer_start": [97], "text": ["1976"]}, "id": "56e10a3be3433e1400422b22"}] >>> preds_dict = {pred["id"]: pred["prediction_text"] for pred in preds} >>> targets_dict = [ ... dict(paragraphs=[dict(qas=[dict(answers=[ ... {"text": txt} for txt in tgt["answers"]["text"]], id=tgt["id"]) for tgt in target ... ])]) ... ] >>> _squad_update(preds_dict, targets_dict) (tensor(1.), tensor(1.), tensor(1)) r8rrXrYr rzUnanswered question z will receive score 0.rr )rr rOrHrD) rPraf1 exact_matchtotalarticle paragraphqaxrJr_s r _squad_updateros: B+K 1IE .I& d85("%9"T(CY#Z[46yM BMq6M BRX=>XZ^`mnn 45Fm\\'/ {E !! !Cs B& rhrirjc,d|z|z }d|z|z }||dS)zAggregate the F1 Score and Exact match for the batch. Return: Dictionary containing the F1 score, Exact match score for the batch. gY@)rirhr&)rhrirjs r_squad_computerqs,+%-K e B&b 11rcZt||\}}t||\}}}t|||S)aCalculate `SQuAD Metric`_ . Args: preds: A Dictionary or List of Dictionary-s that map `id` and `prediction_text` to the respective values. Example prediction: .. code-block:: python {"prediction_text": "TorchMetrics is awesome", "id": "123"} target: A Dictionary or List of Dictionary-s that contain the `answers` and `id` in the SQuAD Format. Example target: .. code-block:: python { 'answers': [{'answer_start': [1], 'text': ['This is a test answer']}], 'id': '1', } Reference SQuAD Format: .. code-block:: python { 'answers': {'answer_start': [1], 'text': ['This is a test text']}, 'context': 'This is a test context.', 'id': '1', 'question': 'Is this a test?', 'title': 'train test' } Return: Dictionary containing the F1 score, Exact match score for the batch. Example: >>> from torchmetrics.functional.text.squad import squad >>> preds = [{"prediction_text": "1976", "id": "56e10a3be3433e1400422b22"}] >>> target = [{"answers": {"answer_start": [97], "text": ["1976"]},"id": "56e10a3be3433e1400422b22"}] >>> squad(preds, target) {'exact_match': tensor(100.), 'f1': tensor(100.)} Raises: KeyError: If the required keys are missing in either predictions or targets. References: [1] SQuAD: 100,000+ Questions for Machine Comprehension of Text by Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, Percy Liang `SQuAD Metric`_ . )rfrorq)rPrarc target_dictrhrirjs rsquadrts8n1?J *:{CB U "k5 11r)!rr, collectionsrtypingrrrtorchrrtorchmetrics.utilitiesr r[r1SINGLE_PRED_TYPElist PREDS_TYPEr=SINGLE_TARGET_TYPE TARGETS_TYPEUPDATE_METHOD_SINGLE_PRED_TYPEr^r2r4rDrHrOtuplerfrorqrtr&rrrs" '' 1S> #T*:%;; < #uS$sE$s)T#Y:N4O/O*P%PQQR'.@)AAB !&tDeCHo1E,F'GdSVX]^bcf^gimnqir^rXsSsNt't!u"#.C-DE( !   CsCsC&737497 ; ;C ;F ; U3UcUfU Hc F*+H9<HMQRUYH H($ ($ ,($ 4S>4S$tCd38n9M4M/N*O%O PQ QR($V," S>," c4S$tCH~*>%> ?@@A B," 666 !","^ 2v 2F 26 2d3PV;FW 29292\92d3;6G92r