+ QV-j*Rt^RIt^RIt^RIHt^RIHtHt^RIH t H t H t H t H t HtHtHt^RIHtHtHtHt]P,!]4tRtRtRt!R R 4t!R R ]4tR tRtRt Rt!Rt"RRlt#]!] 4!RR]44t$]$t%R#)z Tokenization classes for python tokenizers. For fast tokenizers (provided by HuggingFace's tokenizers library) see tokenization_utils_tokenizers.py N) OrderedDict)Anyoverload)INIT_TOKENIZER_DOCSTRING AddedToken BatchEncoding EncodedInputPreTokenizedInputPreTrainedTokenizerBase TextInputTruncationStrategy)PaddingStrategy TensorTypeadd_end_docstringsloggingzspecial_tokens_map.jsonzadded_tokens.jsonztokenizer_config.jsoncZa]tRt^-toRtRtRtV3RlRltV3RlRltRt R t Vt R #) Triez Trie in Python. Creates a Trie out of a list of words. The trie is used to split on `added_tokens` in one pass Loose reference https://en.wikipedia.org/wiki/Trie c`/Vn\4VnRVnVP!V!R#)N)dataset_tokens_termination_charupdate)selfargss&*q/Volumes/fast/ai/experiments/ui-tars-smoke/.venv/lib/python3.14/site-packages/transformers/tokenization_python.py__init__ Trie.__init__3s& u !# Tc H\V!FpVPV4K R#)z} Updates the Trie with new tokens provided as arguments. Args: *args: Variable number of words to be added to the Trie. N)tupleadd)rrtokens&* rr Trie.update9sD\E HHUO"rc <V^8dQhRS[/#)wordstr)format __classdict__s"r __annotate__Trie.__annotate__Cs(((rc V'gR#VPPV4VPpVFpVPV/4W#&W#,pK! ^W P&R#)u% Passes over every char (utf-8 char) on word and recursively adds it to the internal `data` trie representation. The special key `""` in `self._termination_char` is used to represent termination. This function is idempotent, adding twice the same word will leave the trie unchanged Example: ```python >>> trie = Trie() >>> trie.add("Hello 友達") >>> trie.data {"H": {"e": {"l": {"l": {"o": {" ": {"友": {"達": {"": 1}}}}}}}}} >>> trie.add("Hello") >>> trie.data {"H": {"e": {"l": {"l": {"o": {"": 1, " ": {"友": {"達": {"": 1}}}}}}}}} ``` N)rr"r setdefaultr)rr'refchars&& rr"Trie.addCsX(  iiDtR0CI)C'( " "#rc6<V^8dQhRS[RS[S[,/#r&textreturn)r)list)r*r+s"rr,r-bs#W,W,#W,$s)W,rc \4p^.p^p\V4EFywrVV'd WT8dK\4pRpVP4FwrRV 9dVP4FwrW8dMW8dV^,p V^,pMTp TpV \ V48d W,MRpRV 9dT p T pT pW9gKYW,p V ^, p RV 9dT p T pT pV \ V48XdKW,pKB VP V 4VP X4RpM)Wj9dW,p WV &KVP V 4K V'd/pM VFp W)K WT8gEKQW`P9gEKdVPV,W%&EK| VP4F;wrRV 9gK\ V4pVP V 4VP V4M VPW4#)a Will look for the words added to the trie within `text`. Output is the original string split along the boundaries of the words found. This trie will match the longest possible word first ! Example: ```python >>> trie = Trie() >>> trie.split("[CLS] This is a extra_id_100") ["[CLS] This is a extra_id_100"] >>> trie.add("[CLS]") >>> trie.add("extra_id_1") >>> trie.add("extra_id_100") >>> trie.split("[CLS] This is a extra_id_100") ["[CLS]", " This is a ", "extra_id_100"] ``` FrNT) r enumerateritemslenappendr"rcut_text)rr5statesoffsetsskipcurrent current_char to_removeresetstart trie_pointer lookstartlooktrie_pointerlookahead_indexend next_chars&& rsplit Trie.splitbsB # %.t_ !G IE(.||~#%8>||~3 $,!&./6kO")A+C /6O")C=LsSWy=XD$9^b !11$-E"1C#2D';/?/J,+q0O!%55(1&5'6.#d); %(,(=IA8FHNN5)NN3' E!1$0#=L%15M MM%((6F&E ' <99#<"&))L"9y&5~$*<<> E\!$iu%s#$2}}T++rcVP\V44.p^pVF=pWE8d\PR4K WE8XdK(VPWV4TpK? V#)zbThere was a bug in Trie algorithm in tokenization. Attempting to recover. Please report it anyway.)r<r;loggererror)rr5r?tokensrErJs&&& rr= Trie.cut_textsi s4y!C{  MM$S/ *E r)rrrN) __name__ __module__ __qualname____firstlineno____doc__rrr"rLr=__static_attributes____classdictcell__)r+s@rrr-s5  ((>W,W,rrrchaa]tRtRtoV3RltV3RlRltV3RlRltV3RlRltR tVt V;t #) ExtensionsTrieic$<\SV`!V!R#N)superr)rr __class__s&*rrExtensionsTrie.__init__s $rc <V^8dQhRS[/#)r&prefixr()r*r+s"rr,ExtensionsTrie.__annotate__s111rc VPV4pVPV4pVUu.uF qAV,NK up#uupi)z Generates all extensions of a given prefix token in the Trie. Example: ```python >>> trie = Trie() >>> trie.add("apple") >>> trie.add("app") >>> trie.add("application") >>> trie.extensions("app") ['app', 'apple', 'application'] ``` ) _get_node_collect_tokens)rrc prefix_noderetr#s&& r extensionsExtensionsTrie.extensionss>nnV, "";/,/0C5C000s;c&<V^8dQhRS[RS[/#)r&r#r6)r)dict)r*r+s"rr,rd+sstrc RVPpVFpW29dV#W#,pK V#)z Retrieves the node corresponding to the given token in the Trie. Args: token (str): The token for which the corresponding node needs to be retrieved. Returns: dict: The node in the Trie corresponding to the given token. )r)rr#noder1s&& rrfExtensionsTrie._get_node+s7yyD :D   rc&<V^8dQhRS[RS[/#)r&ror6)rmr7)r*r+s"rr,rd=sDTrc VPV9dVP.M.pVP4FOwr4W0P8wgKVPV4pTPVUu.uF qcV,NK up4KQ V#uupi)z Generates all tokens in the Trie starting from a given node. Args: node (dict): The node in the Trie from which tokens need to be generated. Returns: list: List of tokens generated from the given node. )rr:rgextend)rrorRr# subtrie_head subtokenssubtokens&& rrgExtensionsTrie._collect_tokens=s.2-C-Ct-K$(()QS#'::< E... 00>  J Hx// JK$0 Ks(B ) rTrUrVrWrrjrfrgrYrZ __classcell__r`r+s@@rr\r\s- 11&$rr\cVR8XgVR8XgVR8XgVR8XdR#\P!V4pVR8XdR#R#)z0Checks whether `char` is a whitespace character.    TZsF) unicodedatacategoryr1cats& r_is_whitespacerOs= s{ddlddlddl   t $C d{ rcVR8XgVR8XgVR8XdR#\P!V4pVPR4'dR#R#)z-Checks whether `char` is a control character.r}r~rFCT)rr startswithrs& r _is_controlr[s? t|tt|tt|   t $C ~~c rc\V4pV^!8dV^/8:g+V^:8dV^@8:gV^[8dV^`8:gV^{8d V^~8:dR#\P!V4pVPR4'dR#R#)z1Checks whether `char` is a punctuation character.TPF)ordrrr)r1cprs& r_is_punctuationrgsm TB bR2X28bbBh2QS8Y[_bYbgimpgp   t $C ~~c rc|VR,p\\V4\V4,\V4,4#)zcChecks whether the last character in text is one of a punctuation, control or whitespace character.boolrrr)r5 last_chars& r_is_end_of_wordrvs0RI  I&)CCnU^F__ ``rc|V^,p\\V4\V4,\V4,4#)zdChecks whether the first character in text is one of a punctuation, control or whitespace character.r)r5 first_chars& r_is_start_of_wordr|s0aJ  J'/**EEWaHbb ccrcFV^8dQhR\\,R\/#)r& token_list new_tokenr7r))r*s"rr,r,s 4 4$s) 4 4rc\P!W4pV\V48dW,V8XdR#VPW!4R#)ze Inserts one token to an ordered list if it does not already exist. Note: token_list must be sorted. N)bisect bisect_leftr;insert)rr insertion_idxs&& r!_insert_one_token_to_ordered_listrs;&&z=Ms:&:+D +Q-3rcaa]tRtRtoRtV3Rlt]V3RlRl4t]V3RlRl4t]V3RlR l4t ] PV3R lR l4t V3R lR lt Rt Rt R;V3RlRlltRV3R0lR1llt!R?V3R2lR3llt"R@V3R4lR5llt#R4PythonBackend.added_tokens_encoder..s eijkelrkey)sortedrr:r)rrrs& radded_tokens_encoder"PythonBackend.added_tokens_encoders? *00J0J0P0P0RXl)mn)m 1 )mnnnsAc6<V^8dQhRS[S[S[3,/#r)rmrr)r*r+s"rr,rs ZZd3 ?&;Zrc `\\VPP4RR74#)z Returns the added tokens in the vocabulary as a dictionary of index to AddedToken. Returns: `dict[str, int]`: The added tokens. cV^,#rrxrs&rr4PythonBackend.added_tokens_decoder..s PTUVPWrr)rmrrr:rs&rr"PythonBackend.added_tokens_decoders&F455;;=CWXYYrcb<V^8dQhRS[S[S[S[,3,RS[S[S[3,/#)r&valuer6)rmrrr))r*r+s"rr,rs4 ( ($sJ4D/D*E ($sT^J_ (rc VP4Fwr#\V\\34'd\V\4'g=\ RVP VP 3 R\\\,3 24h\V\4'd \V4MTVPV&W P\V4&K VP4R#)z;The provided `added_tokens_decoder` has an element of type z, should be a dict of N) r: isinstancer)rr TypeErrorr`rr_update_total_vocab_size)rrindexr#s&& rrrs"KKMLEec:%677z%QT?U?UQRWRaRachcrcrRrQstJKNPZ]`P`K`JabFPPUWZE[E[ 50AafD & &u -5: & &s5z 2* %%'rc6<V^8dQhRS[S[S[3,/#rr)r*r+s"rr,rs * *c3h *rc VP#)a( Returns the added tokens in the vocabulary as a dictionary of token to index. Results might be different from the fast call because for now we always add the tokens even if they are already in the vocabulary. This is something we should change. Returns: `dict[str, int]`: The added tokens. )rrs&rget_added_vocabPythonBackend.get_added_vocabs)))rc \VP^8XdVP4VP#)z4 Size of the full vocabulary with the added tokens. )rrrs&r__len__PythonBackend.__len__s*  A %  ) ) +$$$rc B\VP44VnR#)a Update the size of the full vocabulary with the added tokens. Counts the `keys` and not the `values` because otherwise if there is a hole in the vocab, we will add tokenizers at a wrong index. This operation is slow and is only updated when adding tokens. N)r; get_vocabrrs&rr&PythonBackend._update_total_vocab_sizes !$DNN$4 5rFc\<V^8dQhRS[S[,S[S[,,RS[RS[/#)r& new_tokensrr6)r7r)rrr)r*r+s"rr,rs7LLd3i$z2B&BLTXLehLrc  ^pVfV#VP4P4p\V4pVEFp\V\\ 34'g\ RV R\V4 R24h\ V4R8XdKM\V\4'dAW`P9dKuW`P9;'gTp\ VRRV'*VR7pM'V'd VPRRR VP/4W`P9dKVP'gEVP'd3\VR R4'd VPP!4VnVPV9d!WS,pWVP&V^, pMWFP,pVP'd6\ V4VP9dVP"P%V4W`PV&WPVP&VP&'gEK\(P+R V R 24EK VP-4VP/4V#) a Add a list of new tokens to the tokenizer class. If the new tokens are not in the vocabulary, they are added to it with indices starting from length of the current vocabulary. Special tokens are sometimes already in the vocab which is why they have to be handled specifically. Args: new_tokens (`list[str]`or `list[tokenizers.AddedToken]`): Token(s) to add in vocabulary. A token is counted as added if it's not already in the vocabulary (tested by checking if the tokenizer assign the index of the `unk_token` to them). If a token is part of the vocabulary then we simply mark this token as an `AddedToken` which allows to control the stripping and normalization of this token. This is NOT possible in `tokenizers`. special_tokens (`bool`, *optional*, defaults to `False`): Whether or not the tokens should be added as special tokens. Returns: `int`: The number of tokens actually added to the vocabulary. Examples: ```python # Let's see how to increase the vocabulary of Bert model and tokenizer tokenizer = BertTokenizer.from_pretrained("google-bert/bert-base-uncased") model = BertModel.from_pretrained("google-bert/bert-base-uncased") num_added_toks = tokenizer.add_tokens(["new_tok1", "my_new-tok2"]) print("We have added", num_added_toks, "tokens") # Note: resize_token_embeddings expects to receive the full size of the new vocabulary, i.e. the length of the tokenizer. model.resize_token_embeddings(len(tokenizer)) ```zToken z is not a string but a .rF)rstriplstrip normalizedspecialrTr do_lower_casezAdding z to the vocabulary)rcopyr;rr)rrtyperr __setstate__rrrgetattrrlower_extra_special_tokensr<verboserPinfo _update_trier) rrr added_tokens current_vocabnew_idxr# is_special token_indexs &&& rrPythonBackend._add_tokenss<   (--/ m$Eec:%677&/FtE{mST UVV5zR%%%666"'*A*A!A!S!S^J&eE*n^hE ""It\5CSCS#TU222===U%5%5%5'$Y^:_:_ % 3 3 5 }}M1%4 /:emm,! +MM: }}}U43J3J!J**11%86; & &{ 38C & &u}} 5||| geW,>?@I L  %%'rNc><V^8dQhRS[S[,R,/#)r&unique_no_split_tokensNr)r*r+s"rr,rRs,,49t3C,rctVPP4FOpVPVPP9gK*VPP VP4KQ T;'g.F:pW PP9gKVPP V4K< R#r^)rvaluesrrrr")rrr#s&& rrPythonBackend._update_trieRs//668E}}D$4$4$<$<<  $$U]]39,11r1E,,444  $$U+2rc&<V^8dQhRS[RS[/#)r&pairr6)rr)r*r+s"rr,rZsffdfsfrc b.p.p\TPY!'d V44#R44#)a Returns the number of added tokens when encoding a sequence with special tokens. This encodes a dummy input and checks the number of added tokens, and is therefore not efficient. Do not put this inside your training loop. Args: pair (`bool`, *optional*, defaults to `False`): Whether the number of added tokens should be computed in the case of a sequence pair or a single sequence. Returns: `int`: Number of special tokens added to sequences. N)r; build_inputs_with_special_tokens)rr token_ids_0 token_ids_1s&& rnum_special_tokens_to_add'PythonBackend.num_special_tokens_to_addZs5&  488UYkdee_cdeerc6<V^8dQhRS[RS[S[,/#r4)r r7r))r*r+s"rr,rqs55Y5T#Y5rc VPRVP4pVP!V3/VBwrV'dVPV4#VPP V4pVP P4p\V4EFwrgWu9gKVPPVP V,4pV^8dWF^, ,MRp V\V4^, 8dWF^,,MRp \V\4'gKVP'd"V 'dV P4WF^,&VP'd"V 'dV P4WF^, &VP 'gEK V 'd1V R,R8wd#WF^, ;;,V, uu&RWF&EKEV 'gEKPV ^,R8wgEKaWtV^,,,WF^,&RWF&EK .p \#VP$4p VFLpV'gK Wu9gW|9dV P'V4K,V P)VPV44KN V #)z Converts a string into a sequence of tokens, using the tokenizer. Args: text: The sequence to be encoded. **kwargs: Passed along to the model-specific `prepare_for_tokenization` preprocessing method. Returns: The list of tokens. split_special_tokensNr|rr)rrprepare_for_tokenization _tokenizerrLrkeysr9rgetr;rrrr single_wordrrr<rs) rr5rrrRno_split_tokenir# tok_extendedleftrightresultall_special_tokens_sets &&, rtokenizePythonBackend.tokenizeqs &zz*@$B[B[\44TDVD  >>$' '!!''-3388:"&)HA&#99==d>X>XY^>_` ()Av!e}4)*S[1_)<1u $lJ77#***u(- 1u #***t(, 1u #///DHO"q5MU2M(*FI"UuQx3,11q5M,AFq5M(*FI#*(!$T%<%.get_input_ids"s$$$11$--2O2OPP$u ..4d1gs++Kd1gs++*#99-1[TTT]]4=ZSY=ZcS=ZST[  55d;;^_cdh_i^jkl l\s2$D T)pair_idsr+r< truncationr.r/r1r2r3prepend_batch_axisr5r4r6r7r8rrx)rr7r!r)r;r@zipr _encode_plusr DO_NOT_PADr:r/r<rpadrrprepare_for_model)rr5r*r+r,r-r.r/r0r1r2r3r4r5r6r7r8rr is_batchedpairs batch_outputs current_text current_pairis_paircurrent_outputrrrB first_ids second_idssf&&&&&&&f&&&&&&&&&l rrHPythonBackend._encode_pluss, tUm4  X 1 111 ^ ^SS,SSDGdE]1S ^ ^\\00\\ZQ#tUZI[5\  :$!)dE];;s9~QTUYQZ?Z$%qrr!*!6ITFSY* l |dE];;$ &|A<<$,L)Q.mm' Q=kkLYZO^bdi]jAkmm' Q=kkLYZO^bdi]jAk 5A2 l\*a/'3A 0(+McR^N_M``k)lmm!%!2!2"%"*"(:"&5%?%? " )< " * """)<"(,""&"$("+@"+0"/H"0J" #0!""$%"(#1"6"6"8JC!,,S"5< seq0 [seq1] ` (custom prefix/suffix stored on the tokenizer) Args: token_ids_0 (`list[int]`): List of IDs to which the special tokens will be added. token_ids_1 (`list[int]`, *optional*): Optional second list of IDs for sequence pairs. Returns: `list[int]`: List of input IDs with the appropriate special tokens. cls_sepzzCannot add special tokens following 'cls_sep' pattern because one or several special tokens are not defined (cls_token_id=z; sep_token_id=zZ)Set the required special tokens in tokenizer or update `tokenizer.special_tokens_pattern`eoszaCannot add special tokens following 'eos' pattern because eos token is not defined (eos_token_id=z[).Set the required special tokens in tokenizer or update `tokenizer.special_tokens_pattern`boszaCannot add special tokens following 'bos' pattern because bos token is not defined (bos_token_id=bos_eoszzCannot add special tokens following 'bos_eos' pattern because one or several special tokens are not defined (bos_token_id=z; eos_token_id=cls_double_sepzCannot add special tokens following 'cls_double_sep' pattern because one or several special tokens are not defined (cls_token_id= prefix_suffix prefix_tokens suffix_tokens)r cls_token_id sep_token_idr@ eos_token_id bos_token_idr)rrr rcrds&&& rr.PythonBackend.build_inputs_with_special_tokensas0  & &) 3  (T->->-F 5595F5F4GW[WhWhVijpp "))*[8D->-F 5595F5F4GW[WhWhVijpp"))*[8D->-F 5595F5F4GW[WhWhVijpp "))*[8DM#D/2>M"$2]BB .<}L L""", ,rc P<V^8dQhRS[RS[R,RS[RS[S[,/#)r&rr Nalready_has_special_tokensr6)r7rr)r*r+s"rr,rsANWNWNW.2TkNW^bNW cNWrc ^<V'd!Ve \R4h\SV` WRR7#VPR8XdkVf$^.^.\ V4,,^.,#^.^.\ V4,,^.,^.\ V4,,^.,#VPR8Xd[Vf^.\ V4,^.,#^.\ V4,^.,^.\ V4,,^.,#VPR8Xd[Vf^.^.\ V4,,#^.^.\ V4,,^.,^.\ V4,,#VPR8XdkVf$^.^.\ V4,,^.,#^.^.\ V4,,^.,^.\ V4,,^.,#VPR8XdlVf$^.^.\ V4,,^.,#^.^.\ V4,,^^.,^.\ V4,,^.,#VPR 8Xd\ \ VR .44p\ \ VR .44p^.V,^.\ V4,,pVeV^.\ V4,, pV^.V,, pV#^.V'd \ V4M^\ V4,,#) a Retrieves sequence ids from a token list that has no special tokens added. This method is called when adding special tokens using the tokenizer `prepare_for_model` or `encode_plus` methods. This method dynamically builds the special tokens mask based on the tokenizer's `special_tokens_pattern`: - `"none"`: No special tokens (default, returns all 0s) - `"cls_sep"`: [CLS] seq0 [SEP] or [CLS] seq0 [SEP] seq1 [SEP] - `"eos"`: seq0 [EOS] or seq0 [EOS] seq1 [EOS] - `"bos"`: [BOS] seq0 or [BOS] seq0 [BOS] seq1 - `"bos_eos"`: [BOS] seq0 [EOS] or [BOS] seq0 [EOS] seq1 [EOS] - `"cls_double_sep"`: [CLS] seq0 [SEP] or [CLS] seq0 [SEP] [SEP] seq1 [SEP] - `"prefix_suffix"`: ` seq0 [seq1] ` Args: token_ids_0 (`list[int]`): List of ids of the first sequence. token_ids_1 (`list[int]`, *optional*): List of ids of the second sequence. already_has_special_tokens (`bool`, *optional*, defaults to `False`): Whether or not the token list is already formatted with special tokens for the model. Returns: A list of integers in the range [0, 1]: 1 for a special token, 0 for a sequence token. zYou should not supply a second sequence if the provided sequence of ids is already formatted with special tokens for the model.T)rr rkr]r^r_r`rarbrcrd)r@r_get_special_tokens_maskrr;r)rrr rk prefix_len suffix_lenmaskr`s&&&& rrm%PythonBackend.get_special_tokens_masks6 && R 72']a3   & &) 3"sqcC $445;;31#K 001QC7A3[AQ;QRVWUXX X  ( (E 1"c+..1#55C#k**qc1aS3{;K5KLPQsR R  ( (E 1"sqcC $445531#K 001QC7A3[AQ;QR R  ( (I 5"sqcC $445;;31#K 001QC7A3[AQ;QRVWUXX X  ( (,< <"sqcC $445;;31#K 001QF:qcC DT>TUYZX[[ [  ( (O ;WT?B?@JWT?B?@J3#sS-='=>D&c+... QC*$ $DK3{3{+SEUUV Vrc,<V^8dQhRS[RS[RS[/#r&idsskip_special_tokensr6)rrr))r*r+s"rr,rs\\\4\TW\rcR#r^rxrrtrus&&&rconvert_ids_to_tokens#PythonBackend.convert_ids_to_tokenssY\rcL<V^8dQhRS[S[,RS[RS[S[,/#rs)r7rrr))r*r+s"rr,rs#hhchhZ^_bZchrcR#r^rxrws&&&rrxrysehrcl<V^8dQhRS[S[S[,,RS[RS[S[S[,,/#rsrr7rr))r*r+s"rr,r s7!!tCy!t!`cfjknfo`o!rc \V\4'd?WP9dVPV,P#VP V4#.pV'd\ VP 4M \ 4pVFcp\V4pWT9dKTPWPP9dVPV,PMVP V44Ke V#)a Converts a single index or a sequence of indices in a token or a sequence of tokens, using the vocabulary and added tokens. Args: ids (`int` or `list[int]`): The token id (or token ids) to convert to tokens. skip_special_tokens (`bool`, *optional*, defaults to `False`): Whether or not to remove special tokens in the decoding. Returns: `str` or `list[str]`: The decoded token(s). )rrrr_convert_id_to_tokenrall_special_idsr<)rrtrurR ids_to_skiprs&&& rrxry s c3  444**3/77 ..s3  3Fc$../CE EJE# MM666**5199..u5   rc&<V^8dQhRS[RS[/#)r&rr6)rr))r*r+s"rr,rCs""#"#"rc\hr^r)rrs&&rr"PythonBackend._convert_id_to_tokenCr(rc6<V^8dQhRS[S[,RS[/#)r&rRr6r)r*r+s"rr,rFs  tCy S rc$RPV4#)r|)join)rrRs&&rconvert_tokens_to_string&PythonBackend.convert_tokens_to_stringFsxxrc`<V^8dQhRS[S[S[,,RS[RS[R,RS[/#)r& token_idsruclean_up_tokenization_spacesNr6r})r*r+s"rr,rIs?c?"'+Tk  rc VPWR7p\V\4'dV.pVPV4pVeTM VPpV'dVP V4pV#)zDecode token ids to string.)ru)rxrr)rrclean_up_tokenization)rrrurrfiltered_tokensr5s&&&&, r_decodePythonBackend._decodeIsq44Y4h os + +./O,,_= ,7 )22 % (--d3D rc$X<V^8dQhRS[S[,RS[S[,R,RS[RS[S[,S[,RS[S[,S[,RS[R,RS[R S[R,R S[R,R S[S[,R,R S[R,R S[R,RS[RS[RS[RS[RS[RS[/#)r&rtrDNr+r<rEr.r/r1r2r3r4r5r6r7r8rrFr6)r7rrr)r r rr)r*r+s"rr,rbs%lplp #Ylps)d"lp! lp o- lp 3J!33 lp$Jlplp $JlpDjlpj(4/lp $d{lp $d{lp$(lp%)lp !lp"#lp$!%lp( )lprc VP!RRVRVRVRVRV/VBwppppV 'd%V\P8XdVe \R4hV fRVP9p V fR VP9p VRJpV'dVP VR 7M^p\ V4\ T;'g.4,V,p.pV\P8wd0V'd(VV8d!VPVVVV, VVR 7wrpV'd$VPW4pVPW4pM%Y'dTM.,p^.\ V4,pR V/pV 'dVVR&V'd0V'dVPW4M^.\ V4,VR &V 'd,V 'g$V'dVVR&V'd VV, M^VR&VPVR ,VV4V\P8wg V 'd"VPVVVP VV V R7pV'd\ VR ,4VR&\#VV VR7#)z Prepares a sequence of input ids so it can be used by the model. Adds special tokens, truncates, and pads. Args: ids: Tokenized input ids of the first sequence. pair_ids: Tokenized input ids of the second sequence (optional). r<rEr.r1rNzNot possible to return overflowing tokens for pair of sequences with the `longest_first`. Please select another truncation strategy than `longest_first`, for instance `only_second` or `only_first`.token_type_idsattention_mask)r)rDnum_tokens_to_remover-r/ input_idsspecial_tokens_maskr:r;)r.r<r1r2r5length)r=rFrx)"_get_padding_truncation_strategiesr LONGEST_FIRSTr@model_input_namesr r;DO_NOT_TRUNCATEtruncate_sequencesr$create_token_type_ids_from_sequencesrm&_eventual_warn_about_too_long_sequencer rIrJrr)rrtrDr+r<rEr.r/r1r2r3r4r5r6r7r8rrFrr,r-_r num_special total_lenr:sequencerencoded_inputss&&&&&&&&&&&&&&&&&&, rrKPythonBackend.prepare_for_modelbs:@D?f?f@ @ !@ "@  2 @  @  @ <-z1 &#'9'G'GG$>  ! ($48N8N$N ! ($48N8N$N !t#CUd44$4?[\ Hs8>>r22[@  "4"D"D DXadnXn040G0G!%.%;$7 1H1 -C- <++F'GN8 $^\noorc<V^8dQhRS[S[,RS[S[,R,RS[RS[S[,RS[RS[S[S[,S[S[,S[S[,3,/#)r&rtrDNrr-r/r6)r7rr)r r!)r*r+s"rr,rszA1A1 #YA1s)d"A1" A1 !#55 A1  A1 tCy$s)T#Y. /A1rc 6V^8:dW.3#\V\4'g \V4p.pV\P8XgV\P8XdIVfE\ \ V4WS,4pVP R8Xd VRVpWRpEMW)RpVRV)pEMvV\P8XEd\PR\PP R24\ V4V'd \ V4M^r\ \W, 4V4p W:, p W8d W^,,p W^,, p MV ^,p W,V ^,, p VP R8Xd(V ^8dVRV )MTpV'dV ^8dVRV )MTpMqWRpV'dW-RMRpM^V\P8XdJV'dB\ \ V4WS,4pVP R8Xd W')RpVRV)pM VRVpW#RpWV3#)z8Truncates sequences according to the specified strategy.NrzmBe aware, overflowing tokens are not returned for the setting you have chosen, i.e. sequence pairs with the 'zg' truncation strategy. So the returned list will always be empty even if some tokens have been removed.r) rr ONLY_FIRSTrminr;truncation_siderPwarningrabs ONLY_SECOND)rrtrDrr-r/r: window_lenlen_idslen_pair first_remove second_remove ids_to_movepair_ids_to_moves&&&&&& rr PythonBackend.truncate_sequencess6 1 $"$ $-/ABB"45H"I  "4"?"? ? #5#C#C CHXSXv'DEJ##v-%(*%5"/0%(%6"0001!$6$D$D D NN22D2R2R2X2X1YZ,,  !$C8#h-Xs8#568LML0?M!*a-?? #0A3E#E +q0 #/#?=TUCU#V ##w.,7!Oc-K<(;CHX[\H\8$6&6%67bj,':B8$56!$6$B$B BxS]F,IJJ##w.%-kl%;"#$:&:%:;%-kz%:"#$9:000rcj<V^8dQhRS[S[,RS[S[,R,RS[S[,/#rZr[)r*r+s"rr,rs6>/>/9>/379t3C>/ c>/rc VP'ddVf VPV4p\V4p^pM\VPW4pVPV4p\V4p\V4V, pM\V4pVe \V4M^pVPR8Xdd\\ VR.44\V4,pVeV\V4, pV\\ VR.44, p^.V,#VP R8XdVe^.V,^.V,,#^.WE,,#)a Create a mask from the two sequences passed to be used in a sequence-pair classification task. This method dynamically builds the token type IDs based on the tokenizer's configuration attributes: - `token_type_ids_pattern`: Pattern to use ("all_zeros" or "bert_style") - `token_type_ids_include_special_tokens`: Whether to account for special tokens in length calculation Args: token_ids_0 (`list[int]`): List of IDs. token_ids_1 (`list[int]`, *optional*): Optional second list of IDs for sequence pairs. Returns: `list[int]`: Token type IDs according to the configured pattern. Examples: ```python # All zeros pattern (default, used by RoBERTa, BART, etc.) tokenizer.token_type_ids_pattern = "all_zeros" # Returns: [0, 0, 0, ...] for both sequences # BERT-style pattern (first sequence gets 0s, second gets 1s) tokenizer.token_type_ids_pattern = "bert_style" # Returns: [0, 0, 0, ..., 1, 1, 1, ...] for sequence pairs ``` rbrcrdr)rrr;rrr) rrr rseq0_lenseq1_len full_sequenceseq0_with_specialrs &&& rr2PythonBackend.create_token_type_ids_from_sequencess0>  5 5 5"@@Mx= $ E Ek _ %)$I$I+$V!01}-8;'H+6+Bs;'H  & &/ 9GD/2>?#kBRRI&S-- WT?B?@ @I3? "  & &, 6;;R3>QC(N2 23(-. .rcN<V^8dQhRS[RS[R,RS[S[R3,/#)r&save_directoryfilename_prefixNr6.)r)r!)r*r+s"rr,rSs3:(:(c:(C$J:(Z_`ceh`hZi:(rc  d^RIp^RIp\VRR4;'g\VRR4pVfR#VPP V4'g\ P RV R24R#\VR/4pV'dV R2MRpVPPWVPR R 4,4p\VR R R 7;_uu_4p V PVPV^RRR7R,4RRR4\VRR4p V fV3#VPPWVPRR4,4p \V R R R 7;_uu_4p \VRR4'dV PR4^p \V P4RR7FVwrW8wd\ PRV R24Tp V PRPV4R,4V ^, p KX RRR4W3# +'giEL;i +'giY3#;i)at Default implementation for common vocabulary saving patterns. Saves self.encoder/self.vocab as JSON, optionally with self.bpe_ranks as merges. Returns empty tuple if no vocabulary exists. Override this method if your tokenizer needs custom saving logic (e.g., SentencePiece models, multiple vocabulary files, or special file formats). Args: save_directory (`str`): The directory in which to save the vocabulary. filename_prefix (`str`, *optional*): An optional prefix to add to the named of the saved files. Returns: `tuple[str, ...]`: Paths to the files saved, or empty tuple if no files saved. NencodervocabzVocabulary path (z) should be a directoryvocab_files_names-r vocab_filez vocab.jsonwzutf-8)encodingTF)indent sort_keys ensure_asciir~ bpe_ranks merges_filez merges.txtadd_bpe_version_headerz#version: 0.2 cV^,#)rx)kvs&rr/PythonBackend.save_vocabulary..s TVWXTYrrzSaving vocabulary to zZ: BPE merge indices are not consecutive. Please check that the tokenizer is not corrupted!r|rx)jsonosrpathisdirrPrQrropenwritedumpsrr:r)rrrrr vocab_attrrrcrfr merge_filewriterr bpe_tokensrs&&& rsave_vocabularyPythonBackend.save_vocabularySs$ T9d3SSwtWd7S  Iww}}^,, LL,^,</>/@:(:(:(rr)&rXrr collectionsrtypingrrtokenization_utils_baserrrrr r r r utilsr rrr get_loggerrTrPSPECIAL_TOKENS_MAP_FILEADDED_TOKENS_FILETOKENIZER_CONFIG_FILErr\rrrrrrrPreTrainedTokenizerrxrrrs #    LK   H %4'/ddN8T8v   a d 4,-}(+}(.}(B $r