izh0ddlZddlZddlZddlmZmZddlmZddlm Z m Z m Z m Z m Z mZmZddlZddlmZddlmZddlmZddlmZmZmZmZmZmZmZdd lmZdd l m!Z!m"Z"dd l#m$Z$m%Z%ed edfd Z&dZ'e%rddl(m)Z)m*Z*d?dZ+e!re"e+sdgZ,ndgZ, d@dede-dede ee.ej^fde e-de0de0de0de e ee1e.efgefd e eeffdZ2dede.ded efdZ3d ed!ed"ed#ed e eeeff d$Z4dAd%e e.de e-de0d e.fd&Z5d'e.d efd(Z6d)e.d efd*Z7 dBd+e.d%e e.d'e e.d)e e.d e ef d,Z8 dCd-ed.ed/ed0ede e-de0d e eeeffd1Z9d2e e.d3e ee.ee.fd e e e.e e.e e e e-e-fffd4Z:d-ed.ed/ed5e e e-e-fd e eeeff d6Z; dDd2ee.ee.e1e.effd3ee.ee.eee.e1e.effd%e e.de e-de0de ed7e de e ee1e.efgefde0de0de ee.ej^fd8e-d9e-d:e-d;e0d+e.dZ=y)EN)IteratorSequence)contextmanager)AnyCallableListOptionalTupleUnioncast)Tensor)Module) DataLoader) TextDatasetTokenizedDataset_check_shape_of_model_output_get_progress_bar_input_data_collator_output_data_collator*_process_attention_mask_for_special_tokens)rank_zero_warn)_SKIP_SLOW_DOCTEST_try_proceed_with_timeout)_TQDM_AVAILABLE_TRANSFORMERS_GREATER_EQUAL_4_4returnc#Ktjd}|j} |jtjd|j|y#|j|wxYww)z]Ignore irrelevant fine-tuning warning from transformers when loading the model for BertScore.ztransformers.modeling_utilsN)logging getLoggergetEffectiveLevelsetLevelERROR)loggeroriginal_levels v/Volumes/fast/ai/experiments/voice-extract-mac/.venv/lib/python3.12/site-packages/torchmetrics/functional/text/bert.py_ignore_log_warningr&(sW  < =F--/N( & ''s&A4#A A4A11A4z roberta-large) AutoModel AutoTokenizerct5tjtt jtdddy#1swYyxYw)zDownload intensive operations.N)r&r(from_pretrained_DEFAULT_MODELr'r%_download_model_for_bert_scorer.:s0 "  ) ). 9  % %n 5# " "s 3AA bert_score dataloader target_lenmodeldevice num_layers all_layersidfverboseuser_forward_fnc g} g} t||D]} tj5t| |} |sQ|s%|| d| dd} | j||nd} n||| } t | | d| j d} n\|r td|| d| dd} tj| jD cgc]} | j dc} d } ddd | jd j dz} t| | d|\} }t|}tjd | |} | j| j|r| d |zn|j| j }||j#dd z}| j|jtj| }tj| }||fScc} w#1swYxYw) a<Calculate sentence embeddings and the inverse-document-frequency scaling factor. Args: dataloader: dataloader instance. target_len: A length of the longest sequence in the data. Used for padding the model output. model: BERT model. device: A device to be used for calculation. num_layers: The layer of representation to use. all_layers: An indication whether representation from all model layers should be used for BERTScore. idf: An Indication whether normalization using inverse document frequencies should be used. verbose: An indication of whether a progress bar to be displayed during the embeddings' calculation. user_forward_fn: A user's own forward function used in a combination with ``user_model``. This function must take ``user_model`` and a python dictionary of containing ``"input_ids"`` and ``"attention_mask"`` represented by :class:`~torch.Tensor` as an input and return the model's output represented by the single :class:`~torch.Tensor`. Return: A tuple of :class:`~torch.Tensor`s containing the model's embeddings and the normalized tokens IDF. When ``idf = False``, tokens IDF is not calculated, and a matrix of mean weights is returned instead. For a single sentence, ``mean_weight = 1/seq_len``, where ``seq_len`` is a sum over the corresponding ``attention_mask``. Raises: ValueError: If ``all_layers = True`` and a model, which is not from the ``transformers`` package, is used. input_idsattention_maskT)output_hidden_statesNzQThe option `all_layers=True` can be used only with default `transformers` models.dimzblsd, bs -> blsd input_ids_idf)keepdim)rtorchno_gradr hidden_statesr unsqueeze ValueErrorcatnormrreinsumappendcputypedtypesum)r0r1r2r3r4r5r6r7r8embeddings_listidf_scale_listbatchoutor;processed_attention_maskrA embeddings idf_scales r%_get_embeddings_and_idf_scalerXFsN%'O#%N":w7 ]]_(7E&k 2E:J4KbfgC++*:PJVXYC)%7C0eK6HImmA&"$kE+.6F0G^bcii9J9J K9JAQ9J KQRS#& sxxBx))"--3C?O9PR\]^#Mn#] ll-s4LMswwy)BEE/ "%= =JbJgJghkhqhqJr  **2t*<< m//12A8D?+J .)I y  '!L#_sBG5G0  G50G55G? cos_simmetricrWc|dk(rdnd}|j|j}tjd||j d}|j ddj S) zOCalculate precision or recall, transpose it and scale it with idf_scale factor. precisionr?zbls, bs -> blsr=rr>)maxvaluesrCrJrO transposesqueeze)rYrZrWr@ress r%_get_scaled_precision_or_recallrds^$!!C ++#+  % %C ,,'i 8 < blprr\recallr^)rCrJrd masked_fillisnan)rerfrgrhrYr\rjf1_scores r%_get_precision_recall_f1rosv ll/1ACTUG/oVI ,Wh@P QF9}v%V);)opencsvreader enumeratefloatrCtensor)rufnamecsv_fileidxrowitem baseline_lists r%_read_csv_from_local_filers m ::e$GPQYGZfGZ83^ade^e#6#$%+#6GZ f  << &q!"u --7f  s3%B B B BB BB BB baseline_urlc tjj|5}t|Dcgc]O\}}|dkDrE|j j dj dDcgc] }t|c}Q}}}}tj|ddddfcdddScc}wcc}}}w#1swYyxYw)ziRead baseline from csv file from URL. This method is implemented to avoid `pandas` dependency. rzutf-8,Nr>) urllibrequesturlopenrzstripdecodesplitr{rCr|)r http_requestrrrrs r%_read_csv_from_urlrs    -&l3 3SQw&)YY[%7%7%@%F%Fs%K L%KTU4[%K L3  ||M*1ab51 . - L  . -s.B;=B4 -B/?B4 !B;/B4 4B;;Clangc|r t|}|S|r t|}|S|r|rd}|d|d|d}t|}|Stdy)z}%M O %l3 O $l"1TF!,>+?tD %l3 O _`r-r\rjrnrc||durd}tj|||gd}|r|jdn||}||z d|z z }|d|d|dfS)z).r).r>).r^)rCstackrF)r\rjrnrr4r5 all_metricsbaseline_scales r%_rescale_metrics_with_baselinersvjE1 ++y&(;DK.8X''*hz>RN/A4FGK v  F 3[5H HHr-predstargetctd|Ds tdtd|D}|rg}g}g}d}t||D]\}}t |t t frr|j|gt|z|jttt||j||t|zf|t|z }|j||jtt||j||dzf|dz }|||fS|ttt|dfS)aPreprocesses predictions and targets when dealing with multiple references. This function handles the case where a single prediction might have multiple reference targets (represented as a list/tuple of strings). Args: preds: A list of predictions target: A list of targets, where each item could be a string or a list/tuple of strings Returns: Tuple: (preds, target, ref_group_boundaries) - preds: Flattened list of `str` - target: Flattened list of `str` - ref_group_boundaries: List of tuples (start, end) indicating the boundaries of reference groups in the flattened lists or `None` c3<K|]}t|tywN) isinstancestr.0rs r% z2_preprocess_multiple_references..s7z$$sInvalid input provided.c3HK|]}t|ttfywr)rlisttuplers r%rz2_preprocess_multiple_references..sR64z$u >6s "rr>N) allrGanyziprrrextendlenr rrrK) rrhas_nested_sequencesref_group_boundaries new_preds new_targetcountpred ref_groups r%_preprocess_multiple_referencesrs1( 77 7233R6RR68! " "5&1OD))dE]3  $#i.!89!!$tCy)"<=$++UEC N4J,KLY'  &!!$sI"67$++UEAI,>?  2*&::: $tCy&)4 //r-rcbggg}}}|D]\}}|jdkDr|j|dd||fjdd|j|dd||fjdd|j|dd||fjdd|j|||j|j|||j|j|||j |jdkDrFtj|d}tj|d}tj|d}n?tj|}tj|}tj|}|||fS)a)Postprocesses metrics when dealing with multiple references. For each group of references that correspond to a single prediction, this function takes the maximum score among all references. Args: precision: Tensor of precision scores recall: Tensor of recall scores f1_score: Tensor of F1 scores ref_group_boundaries: List of tuples (start, end) indicating the boundaries of reference groups Returns: tuple: (precision, recall, f1_score) with updated metrics r>Nr?r)r@rKr_rCr) r\rjrnr max_precision max_recallmax_f1startends r% _postprocess_multiple_referencesr2s&)+Bv:M* s ==?Q   1eCi ?  5!5!9!9!; <   fU3/335 6 MM(5-113 4+}}KK 15 ZQ/;;v1-KK . Z(;;v& fh &&r-user_tokenizer max_length batch_size num_threads return_hashrescale_with_baseline truncationc d}t|tr|g}t|tr|g}t|ttfs t|}t|ttfs t|}t |t |k7r#t dt |dt |t|tr>> from pprint import pprint >>> from torchmetrics.functional.text.bert import bert_score >>> preds = ["hello there", "general kenobi"] >>> target = ["hello there", "master kenobi"] >>> pprint(bert_score(preds, target)) {'f1': tensor([1.0000, 0.9961]), 'precision': tensor([1.0000, 0.9961]), 'recall': tensor([1.0000, 0.9961])} Example: >>> from pprint import pprint >>> from torchmetrics.functional.text.bert import bert_score >>> preds = ["hello there", "general kenobi"] >>> target = [["hello there", "master kenobi"], ["hello there", "master kenobi"]] >>> pprint(bert_score(preds, target)) {'f1': tensor([1.0000, 0.9961]), 'precision': tensor([1.0000, 0.9961]), 'recall': tensor([1.0000, 0.9961])} NzLExpected number of predicted and reference sentences to be the same, but gotz and rz1Expected argument `idf` to be a boolean, but got .zcAn argument `verbose = True` requires `tqdm` package be installed. Install with `pip install tqdm`.z`bert_score` metric with default models requires `transformers` package be installed. Either install with `pip install transformers>=4.4` or `pip install torchmetrics[text]`.zThe argument `model_name_or_path` was not specified while it is required when default `transformers` model are used.It is, therefore, used the default recommended model - num_hidden_layersz num_layers=z is forbidden for z. Please use num_layers <= zhModel config does not have `num_hidden_layers` as an integer attribute. Unable to validate `num_layers`.zXIt was not possible to retrieve the parameter `num_layers` from the model specification.c3\K|]$}t|txrt|dk(&ywrN)rrrrtexts r%rzbert_score..s*aQ`:dD1Dc$i1nDQ`s*,c3K|]9}t|txr#t|dkDxrt|dt;ywr)rrrrrs r%rzbert_score..s=ZiRV 4M3t9q=MZQ5MMZis?Ac3fK|])}t|txrt|dt+yw)r:N)rdictr rs r%rzbert_score..s0UdT 4H:d;.?#HHUds/1z%Predictions and references are empty.rk)r\rjf1hash)r6r)r6 tokens_idfrr6)r6rr)r num_workersr,)'rrrrrrGrboolrModuleNotFoundErrorrrr+r&r(r*r'evaltohasattrconfigrintAttributeErrorrupdatertrrrrrrXrdatasetsorting_indicesrorr)&rrrpr4r5r2rr8r7r6r3rrrrrrrurrr tokenizer_are_empty_lists_are_valid_lists_are_valid_tensors output_dictrtarget_dataset preds_dataset target_loader preds_loaderrfrhrergr\rjrns& r%r/r/]sN=A%&# edD\ *U  ftTl +f 5zS[  Z5zl%F } .  %3u:>j6NSVW]S^abSb.MeU[.\+v+ c4 LSEQRSTT! q   }.%l   % JJXIYYZ\  ! "%556H6ZN[I--.@.RNSE# "#  JJL HHV s 5<?e%C     2DjRU(VW XYn~d$6 |TswH$VY Xbc#   %00!   )N#Ivx&D vx:z' # 68'&F vx)=' # 68 K FI.@*c$RST E# "( sqrss;Q=BQ!Q!Q98Q9)rN)NNFFFN)NNF)enNNN)NF)NNFNNNFFNi@rFrFNNF)>rxrrcollections.abcrr contextlibrtypingrrrr r r r rCr torch.nnrtorch.utils.datar4torchmetrics.functional.text.helper_embedding_metricrrrrrrrtorchmetrics.utilitiesrtorchmetrics.utilities.checksrrtorchmetrics.utilities.importsrrr&r+ transformersr'r(r.__doctest_skip__rrr3rrrXrdrortrrrrrrr{r/r,r-r%rs$  .%DDD '2W[(Xd^((!"56 ";$~26 $OSN!N!N! N! U3 ,- . N!  N!  N! N!N!hS&[0A'BF'JKLN! 66>N!b)V)S)V)X^)''17'JP'dj' 666 !"'6O(3-OHSMO_cOpsO .S .V . 2S 2V 2 (,#'"&    C=3-  f 8!% II II I  I  I 666 !"I$+0 9+0"5hsm);#<=+0 49d3i$uS#X*?!@ @A+0\('('%('17('OSTYZ]_bZbTcOd(' 666 !"('\)- $"OS15"'#'"&)q hsmT#v+%66 7q #x}hx}&=tCK?PP Qq! q q  q F  qqhS&[0A'BF'JKLqq q U3 ,- .qqqqq !q" #q$C=%q&3-'q()q* #uVT%[#-. ./+qr-