+ QV-jRt^RIt^RIt^RIt^RIHt^RIHt^RIH t ^RI H t ^RI H t^RIHt^RIHtHt^RIHt^R IHt^R IHt^R IHtHt^R IHtH t H!t!H"t"^R I#H$t$^RI%H&t&^RI'H(t(^RI)H*t*^RI+H,t,H-t-H.t.H/t/H0t0H1t1H2t2^RI3H4t4H5t5H6t6]6Pn!]84t9Rt:Rt;Rt],R, t,R]R] R]!R]"/t?R]:R]=/t@]5!],4!RR ]/44tA]AtBR#)!z Tokenization classes for fast tokenizers (provided by HuggingFace's tokenizers library). For slow (python) tokenizers see tokenization_utils.py N) defaultdict)Iterable)copyfile)Any)is_offline_mode) AddedToken processors)Encoding) Tokenizer)Decoder)BPEUnigram) BpeTrainerUnigramTrainerWordLevelTrainerWordPieceTrainer) cached_file) SpmConverter)convert_gguf_tokenizer)load_gguf_checkpoint)INIT_TOKENIZER_DOCSTRING BatchEncodingPreTokenizedInputPreTrainedTokenizerBase TextInputTruncationStrategygenerate_merges)PaddingStrategyadd_end_docstringsloggingztokenizer.jsonzspecial_tokens_map.jsonztokenizer_config.jsonztokenizer.modelzadded_tokens.jsonu tokenizer_object ([`tokenizers.Tokenizer`]): A [`tokenizers.Tokenizer`] object from 🤗 tokenizers to instantiate from. See [Using tokenizers from 🤗 tokenizers](../fast_tokenizers) for more information. tokenizer_file ([`str`]): A path to a local JSON file representing a previously serialized [`tokenizers.Tokenizer`] object from 🤗 tokenizers. r r WordLevel WordPiecetokenizer_file vocab_filecraa]tRt^StoRt]tRtRt] RBRl4t V3Rlt ] V3RlRl4t ] V3RlR l4tRCV3R lR lltR t] R 4t] R4t]P&R4t]P&R4tRt] V3RlRl4tV3RlRlt] V3RlRl4t] V3RlRl4t] V3RlRl4t]t]tV3RlRltV3RlRltV3R lR!lt] V3R"lR#l4t] V3R$lR%l4t RDV3R'lR(llt!V3R)lR*lt"V3R+lR,lt#RBV3R-lR.llt$RBV3R/lR0llt%RBV3R1lR2llt&REV3R3lR4llt'V3R5lR6lt(RR&])PT]+PXR^RRRRRRRRRRR&R3V3R7lR8llt-V3R9lR:lt.RFV3R;lR<llt/RGV3R=lR>llt0RHR?lt1] RIR@l4t2RAt3Vt4V;t5#)JTokenizersBackenda5 Base class for all fast tokenizers (wrapping HuggingFace tokenizers library). Inherits from [`~tokenization_utils_base.PreTrainedTokenizerBase`]. Handles all the shared methods for tokenization and special tokens, as well as methods for downloading/caching/loading pretrained tokenizers, as well as adding tokens to the vocabulary. This class also contains the added tokens in a unified way on top of all tokenizers so we don't have to handle the specific vocabulary augmentation methods of the various underlying dictionary structures (BPE, sentencepiece...). NFc a(\V4pVPRR4pVed\PP V4'd?V\ JgRVP 9g V'd\P!V4VR&V#VEe?\PP V4'Ed\VRR7;_uu_4p\P!V4pRRR4XPR/4PR4pVR:9dc\V4p\VR,4p /V R &VR 8Xd.V R &WR&.VR &\P!\P!V44p M\P!V4p V PVR&V P VR&V P"VR&V P"eV P"VR&V P eV P VR&VPR4p V 'dV PRR4R8Xd V R,p M\%V \&4'gV .p V FCp V PR4R8XgKRV 9gK$^RIp V P+V R,4VR&M VPR/4PR R4pVP,f2\%V\&4'd\'\/\0V44pEM>VP,P2R 8Xd]\%V\&4'dFV'd>\%V^,\&\034'dVUu.uFp\1V4NK ppMVP,P2R8Xd"\5V4UUu/uF wppVVbK pppMVP,P2R 8XgVP,P2R8XdV\%V\&4'd@\5V4UUu/uF(wpp\%V\&4'd V^,MTVbK* pppWR &\7VRR4pR VPR/49dyV'dqVP2R 8Xd`VR,R ,pVUu.uF>p\%V\84'd\1VP;R44M \1V4NK@ ppVVR &V#VPR4pVPR4pVPR 4pVPR 4p\%V\84'dfVP=R4'dO\PP V4'd*^RIH pV!VR 7PCV4wVR &VR &V#\%V\84'Ed\PP V4'EdVP=R!4'Ed^R"IH"pV!V4pVPF!VP,3/VBp^R#IH$pVPVP24pVe%\KVR$4'dVPL!R;/VBp\KTR(4'dTPT!R;/TBpRT9EdT\ JgRTP 9EdTPR R4pTPR R4pTPR)4;'g/pTeT'dTPW4UUu/uF wppTTbK pppTPW4Fbwpp\YT4p\9T4pTPT4p T 'gK7T T8wgK@TT9gKITPT 4TT&TTT&Kd \ZP\!TP^TTR*7p!T!eT!TR&TP^P`p"T"Pb^8d'TPeR+T"Pf;'gR,4T"Ph^8d'TPeR-T"Pj;'gR.4T"Pl^8d'TPeR/T"Pn;'gR04T#VfJ\%V\84'd4\PP V4'dVVR &VR ,pVfJ\%V\84'd4\PP V4'dVVR &VR ,pVfVP,eVP,P2R 8Xdq\%V\4'd[R7V(3R8llo(.R eos_token unk_tokenzz+Could not extract SentencePiece model from z$ using sentencepiece library due to z%. Falling back to TikToken extractor.)TikTokenConverterextra_special_tokens)r#rLc\V^8dQhR\\,R\\,/#)valuesreturn)rrliststr)formats"{/Volumes/fast/ai/experiments/ui-tars-smoke/.venv/lib/python3.14/site-packages/transformers/tokenization_utils_tokenizers.py __annotate__@TokenizersBackend.convert_to_native_format..__annotate__)s" ! !Xc] !tCy !c<.pVFXpVfK \V\\34'dVPS!V44K>VP \ V44KZ V#N) isinstancerQtupleextendappendrR)rO collectedval_iter_special_tokenss& rTr`HTokenizersBackend.convert_to_native_format.._iter_special_tokens)sY') !C{ !#e}55!(()=c)BC!((S2 "! rW) skip_tokens)Nr ) pad_tokenrJrFrH sep_token cls_token mask_tokenadditional_special_tokensrL)=dictpopospathisfiler%__dict__ TokenizerFast from_fileopenjsonloadgetfrom_strdumpsr1padding truncationrZrQbase64 b64decoder,mapr[__name__ enumerategetattrrRsplitendswithconvert_slow_tokenizerr>extract_vocab_merges_from_modelr@extractrAhasattrrB ExceptionloggerwarningrCitemsintrbuild_tokenizer_from_spm_protorE trainer_specbos_id setdefault bos_pieceeos_id eos_pieceunk_id unk_piecerK convertedsetupdater))clstrust_remote_codekwargs local_kwargsfast_tokenizer_filetokenizer_handletokenizer_json model_typeminimal_tokenizer_json minimal_model tok_from_filenormalizer_configr6ryr.itemitokenr/merger#r=r>r@ extractorrAconverter_classerDtoken_id id_to_token new_token current_tokenr( proto_specrK converterspecial_tokens_keysrbkeyr`s)&&, @rTconvert_to_native_format*TokenizersBackend.convert_to_native_formatesI F| *../?F +233))Zs||-KO`/CY]gCg%DJDTDT&'=>E %@A#4#&&w377FEyy eT** UE!23E##y0eT**uE!HtUZm9\9\5:;UTU4[UE;##{22;E2BC2Bha2BC##u, 0B0Bk0QeT**_hin_op_oS[STV[E4)@)@U1XeQN_oEp$) ! gt4J>--gr:: zObObfkOk'0:kqrkqbgZs5K5K% C 01QVW\Q]]kqr)/ X& !%%l3 "&&}5   )!!(+ j# & &:+>+>}+M+MRTRYRYR`R`akRlRl @>*+E+E*J]J]^fJgJgF IJ3:> (00KlK  O&=&A&A#,,&OO&2wPb7c7c'6'G'G'W,'W 3 899#&#=#=#M #ML &\9,, #,,0N(,,Wd;E)--h=F,8+;+;jc::rww~~k?Z?Z%0L "!(+F >cii3 8J8Je8SXbchjnXoXo ! ! # %(EK*,&&&';\#=Nhh  h i6Ai11i6c .<VPRR4pVPRR4pVPRR4VPRR4pVPRR4pVPRR4pVPR/4pVPRR 4p VPR 4p VPR 4p VPR 4p Rp Ve\P!V4p EMVe>\P P V4'd\P!V4p EMVe\VPR R4V3/VBp\V4pVR,R,pVR,pVR,p\VV4wp pVPV4\V4^8dVPV4EMVPfV eV eX\V \ 4'dT M%\#V 4UUUu/uF wpwppVVbK upppp\\%VV RRR74p M\V \ 4'd\\%V .RRR74p M\V \&4'dQV 'dI\V ^,\(\&34'd&\\+WPR^4R74p MVPf \-R4hVf7Vf3VPf%VP/RR4VP/RR4V eWnVPf \-R4hVPRR4;'g!VPP0;'gTpVeVPP2!R7/VBVP/RVR,4VP/RVR ,4VP/R!VR!,4VP/R"VR#,4MVPP54VPR$R4;'g!VPP6;'gTpVeVPP8!R7/VBVP/R%VR%,4VP/R&VR',4VP/R(VR ,4VP/RVR),4VP/R*VR*,4R+V9dR,VR+&R-V9;'gR.V9pVPR-R 4VnVPR.R 4VnVPR/R4;p'dVVPnT;'gVPP>RJVn \BS&V`!R7/VBV eWn#Wn$VPJVPn&VPNUu0uFp\Q\SV44kK pp\UVPW4R0R17UUu.uF#wpp\Q\SV44V9gK!VNK% ppp\'VPXP[44VUu.uFp\]V4NK up,p VP^Pa4F5p!V!fK \]V!4V 9gKV!V9gK$VPcV!4K7 VPdF/p\]V4V 9gKVV9gKVPcV4K1 \V4^8d.p"VP^Pa4U#u.uFp#V#'gK \]V#4NK p$p#VFwp\V\\4'd\gVRR27pM?\V\f4'd*VPh'g\]V4V$9dRVn4V"PcV4Ky V"'dVPkV"4VPPm4p%V%R38d\qVPR4R4enVPRR4VPr!VPVPtPR R43R5VPtR6VPR6R4/VBVnVP@;'gVPP>RJVn VP@'dVPw4R#R#uupppiuupiuuppiuupiuup#i \nd^p%ELi;i)8r4Nr5r;r( gguf_filer"rDadd_prefix_spaceFr#r.r/ name_or_pathconfigr tokenizertokenizer_configT)r.r/fuse_unkdropoutr)r.ra9Couldn't instantiate the backend tokenizer from one of: (1) a `tokenizers` library serialization file, (2) a slow tokenizer instance to convert or (3) an equivalent slow tokenizer class to instantiate and convert. You need to have sentencepiece or tiktoken installed to convert a slow tokenizer to a fast one.rFrGrHrIz3The backend tokenizer is not correctly initialized.r3 max_lengthtruncation_side directionstridetruncation_strategystrategyr2rdpad_token_type_id pad_type_id padding_sidelengthpad_to_multiple_ofbackend tokenizers add_bos_token add_eos_tokenr1cV^,#rc)xs&rT,TokenizersBackend.__init__..s STUVSWrWr)speciali pre_tokenizer init_kwargsfix_mistral_regexrc)pp$//B\B\pp`p  " OO - - < <   lK ,E F   / J   h H(= >   3[5L M OO ) ) +::148ddDOO ?   2H=Q4R S F " ,F9 %4%>%[%[/U[B["$jj%@$jj%@#ZZ(8$? ?> ?-;DOO *-G-q-q4??KiKimqKq* "6"  !(O 0040I0I-DHD]D]$^D]5T$u+%6D]!$^!'';'A'A'C X X uDK (AA E X  t005578Ta;bTa5CJTa;bb$(#;#;#B#B#D "*&'w6;NVc;c$$%89 $E//E5z(U--G$$U+0 }  !F040H0H0O0O0QW0Q1UVA0Q W&eS))&ud;Ez22 ===SZ;K-K(,  e$'' 779J  74??OT#R#^ JJ{D )"77  $$^T:!,,#)**-@$"G   DO  . . X X$//2P2PTX2X *  - - -  & & ( .oDl~%_ c>)d ddc <V^8dQhRS[/#rNrPbool)rS __classdict__s"rTrUTokenizersBackend.__annotate__srWcR#)Trcrs&rTis_fastTokenizersBackend.is_fastsrWc <V^8dQhRS[/#r r )rSrs"rTrUrs   rWc RVP9dxVPR,PR4'dP\VR4'd<VP'd*\P P VP4#R#R#)z `bool`: Whether or not the slow tokenizer can be saved. For a sentencepiece based slow tokenizer, this can only be `True` if the original `"sentencepiece.model"` was not deleted. r#r?FT)vocab_files_namesrrr#rkrlrmrs&rTcan_save_slow_tokenizer)TokenizersBackend.can_save_slow_tokenizers` 411 1d6L6L\6Z6c6cdl6m6mt\**tww~~doo66rWcJ<V^8dQhRS[RS[R,RS[S[,/#)rNsave_directoryfilename_prefixNrP)rRr[)rSrs"rTrUrs- ! !c !C$J !Z_`cZd !rWc\PPV4'g\P RV R24R#\PP Y'd VR,MR\ R,,4p\PPVP4\PPV48wd\VPV4V3#)zVocabulary path (z) should be a directoryN-rr#) rkrlisdirrerrorjoinVOCAB_FILES_NAMESabspathr#r)rrrout_vocab_files&&& rTsave_vocabulary!TokenizersBackend.save_vocabularysww}}^,, LL,^,<ArWc:VPPRR7#)TrF)r get_vocabrs&rTrMTokenizersBackend.get_vocab]s((4(@@rWc6<V^8dQhRS[S[S[3,/#r rK)rSrs"rTrUras  tCH~ rWc"VP4#rY)rMrs&rTr.TokenizersBackend.vocab`s~~rWc6<V^8dQhRS[S[S[3,/#r rK)rSrs"rTrUresnnd38nnrWc \VPP4RR7UUu/uFwrVPVbK upp#uuppi)z Returns the sorted mapping from string to index. The added tokens encoder is cached for performance optimisation in `self._added_tokens_encoder` for the slow tokenizers. cV^,#rrcrs&rTr8TokenizersBackend.added_tokens_encoder..j dhijdkrWrrrDrcontentrvks& rTr&TokenizersBackend.added_tokens_encoderds? *00I0I0O0O0QWk)lm)l 1 )lmmmAc6<V^8dQhRS[S[S[3,/#r )rirr)rSrs"rTrUrms::d3 ?&;:rWc 6VPP4#)z Returns the added tokens in the vocabulary as a dictionary of index to AddedToken. Returns: `dict[str, int]`: The added tokens. )rget_added_tokens_decoderrs&rTrD&TokenizersBackend.added_tokens_decoderls7799rWc6<V^8dQhRS[S[S[3,/#r rK)rSrs"rTrUr{snnc3hnrWc \VPP4RR7UUu/uFwrVPVbK upp#uuppi)z Returns the added tokens in the vocabulary as a dictionary of token to index. Returns: `dict[str, int]`: The added tokens. cV^,#rrcrUs&rTr3TokenizersBackend.get_added_vocab..rWrWrrXrZs& rTget_added_vocab!TokenizersBackend.get_added_vocab{s?*00I0I0O0O0QWk)lm)l 1 )lmmmr^c <V^8dQhRS[/#r r )rSrs"rTrUrs$rWc R#)z> Returns True, to avoid expensive `assert tokenizer` gotchas. Trcrs&rT__bool__TokenizersBackend.__bool__srWc <V^8dQhRS[/#r rD)rSrs"rTrUrsFFFrWc :VPPRR7#)z4 Size of the full vocabulary with the added tokens. TrFrHrs&rT__len__TokenizersBackend.__len__s---EErWc <V^8dQhRS[/#r )ro)rSrs"rTrUrs=rWc VP#)zS `tokenizers.implementations.BaseTokenizer`: The Rust tokenizer used as a backend. )rrs&rTbackend_tokenizer#TokenizersBackend.backend_tokenizers rWc <V^8dQhRS[/#r ) DecoderFast)rSrs"rTrUrs'''rWc .VPP#)zE `tokenizers.decoders.Decoder`: The Rust decoder for this tokenizer. )rdecoderrs&rTrxTokenizersBackend.decoders &&&rWTc<V^8dQhRS[RS[R,RS[R,RS[RS[RS[RS[R S[R S[S[S[S[3,S[S[,3,/ #) rNr+return_token_type_idsNreturn_attention_maskreturn_overflowing_tokensreturn_special_tokens_maskreturn_offsets_mapping return_lengthverboserP) EncodingFastr r[rirRrrQ)rSrs"rTrUrs-(-(-( $d{-( $d{ -( $( -( %) -(!%-(-(-( tCH~tL11 2-(rWc  VfRVP9pVfRVP9pV'd$VPeV.VP,p MV.p \\4p V EFp V R,P V P 4V'd#V R,P V P 4V'd#V R,P V P4V'd#V R,P V P4V'd#V R,P V P4V'gKV R,P \V P 44EK W3#)ar Convert the encoding representation (from low-level HuggingFace tokenizer output) to a python Dict and a list of encodings, take care of building a batch from overflowing tokens. Overflowing tokens are converted to additional examples (like batches) so the output values of the dict are lists (overflows) of lists (tokens). Output shape: (overflows, sequence length) token_type_idsattention_mask input_idsspecial_tokens_maskoffset_mappingr) model_input_names overflowingrrQr]idstype_idsrroffsetsr) rr+r{r|r}r~rrr encodings encoding_dictrs &&&&&&&&& rT_convert_encoding#TokenizersBackend._convert_encodings( ! ($48N8N$N ! ($48N8N$N ! $)=)=)I! X%9%99I! I#D) A + & - -aee 4$./66qzzB$./66q7G7GH)34;;A This encodes a dummy input and checks the number of added tokens, and is therefore not efficient. Do not put this inside your training loop. Args: pair (`bool`, *optional*, defaults to `False`): Whether the number of added tokens should be computed in the case of a sequence pair or a single sequence. Returns: `int`: Number of special tokens added to sequences. )rnum_special_tokens_to_add)rr(s&&rTr+TokenizersBackend.num_special_tokens_to_adds&88>>rWcl<V^8dQhRS[S[S[,,RS[RS[S[S[,,/#)rNrskip_special_tokensrPrrQr rR)rSrs"rTrUrs7tCyt`cfjknfo`orWc F\V\4'dVPPV4#.pV'd\ VP 4M \ 4pVF@p\V4pWT9dKVP VPPV44KB V#)a Converts a single index or a sequence of indices in a token or a sequence of tokens, using the vocabulary and added tokens. Args: ids (`int` or `list[int]`): The token id (or token ids) to convert to tokens. skip_special_tokens (`bool`, *optional*, defaults to `False`): Whether or not to remove special tokens in the decoding. Returns: `str` or `list[str]`: The decoded token(s). )rZrrrrall_special_idsr])rrrr ids_to_skiprs&&& rTconvert_ids_to_tokens'TokenizersBackend.convert_ids_to_tokenss c3  ??..s3 33Fc$../CE EJE# MM$//55e< =   rWc P<V^8dQhRS[RS[R,RS[RS[S[,/#)rNtextr(NrrP)rRr rQ)rSrs"rTrUr s:vvSvd vtvjnorjsvrWc NVP!RRVRVRV/VBP4#)r text_pairrrc) _encode_plusr)rrr(rrs&&&&,rTtokenizeTokenizersBackend.tokenize s0  ldldlOaleklssuurWc Z<V^8dQhRS[RS[RS[RS[RS[R,RS[R,/#)rNpadding_strategyrrrrNr)rrrrR)rSrs"rTrUrsWI9I9)I90I9 I9  I9  $J I9DjI9rWc  VPPpVPPpV\P8Xd VeVPP 4MhRVRVRVP RVP/p VfRp M!V U u/uFqVPV R4bK p p W8wdVPP!R /V BV\P8Xd#VeVPP4R#R#V\P8XdTMRp RT RVeTM VPRVPRVP R VP"R V/p W8wdVPP$!R /V BR#R#uup i) a Define the truncation and the padding strategies for fast tokenizers (provided by HuggingFace tokenizers library) and restore the tokenizer settings afterwards. The provided tokenizer has no padding / truncation strategy before the managed section. If your tokenizer set a padding / truncation strategy before, then it will be reset to no padding / truncation when exiting the managed section. Args: padding_strategy ([`~utils.PaddingStrategy`]): The kind of padding that will be applied to the input truncation_strategy ([`~tokenization_utils_base.TruncationStrategy`]): The kind of truncation that will be applied to the input max_length (`int`): The maximum size of a sequence. stride (`int`): The stride to use when handling overflow. pad_to_multiple_of (`int`, *optional*): If set will pad the sequence to a multiple of the provided value. This is especially useful to enable the use of Tensor Cores on NVIDIA hardware with compute capability `>= 7.5` (Volta). padding_side (`str`, *optional*): The side on which the model should have padding applied. Should be selected between ['right', 'left']. Default value is picked from the class attribute of the same name. Nrrrrrpad_idrdrrrc)rrxrwrDO_NOT_TRUNCATErr;rrtrr DO_NOT_PAD no_padding MAX_LENGTHr pad_token_idrdrr) rrrrrrrrrtargetcurrentr\rs &&&&&&& rTset_truncation_and_padding,TokenizersBackend.set_truncation_and_paddingshBoo00 ??** "4"D"D D&--/j&/55T11 F"@FG1kooa66G 11;F; 99 9#**,$$47Q7Q#QZW[F&\-E\4K\K\$++T^^t55$&8 F!..88"%Hs E;c(<V^8dQhRS[S[,S[S[,,S[S[,,RS[S[,S[S[,,S[S[,,R,RS[RS[RS[RS[R,RS[R S[R S[R,R S[R,R S[R,R S[R,RS[R,RS[RS[RS[RS[RS[RS[R,RS[/#)rNrrNrrrrris_split_into_wordsrrreturn_tensorsr{r|r}r~rrrrrP) rrrQr rrrrRr)rSrs"rTrUrYs?XX++d9o=EV@WWX004 ?BTJ[E\\_ccX! X * X 0 X$JXX"X $JXDjXt X $d{X $d{X$(X %)!X"!%#X$%X&'X(#Tk)X, -XrWc RpV!V4'g \R4hVeV!V4'g \R4hV'dG\V\\34;'d(T;'d\V^,\\34pM\V\\34pV'd~\V\4'd \ R4hVe>\ V4\ V48wd%\R\ V4 R\ V4 R24hVe\\W44MTpMV'dW3.MV.p\V\\34'g\ R\V4 R24hVPVVVVV V R7Vf VPpVPPV8wdVVPn VPPVVVR 7pVUu.uFpVPVV V VVVVVR 7NK pp/pV^,^,F0pVUUUu.uFwppVV,FpVNK K p pppV VV&K2 VUUUu.uFwppVFpVNK K p!pppV'dA.p"\V4F*wp#wp$pV"V#.\ V$R ,4,, p"K, V"VR &VR ,Fp%VP!V%VV4K \#VV!V R 7p&V'g}V fyV'gq\#V&P%4UU'u/uF?wpp'T\ V'4^8d'\V'^,\4'd V'^,MT'bKA up'pV&P&4p&V&#uupiuupppiuupppiuup'pi)c\V\4'dR#\V\\34'Ed \ V4^8XdR#\V^,\4'dR#\V^,\\34'd\ V^,4^8Xg%\V^,^,\4'dR#\V^,^,\\34'dJ\ V^,^,4^8H;'g&\V^,^,^,\4#R#R#R#)TF)rZrRrQr[r)rs&rT_is_valid_text_input._is_valid_text_inputqs!S!!Ae}--q6Q;!c**!tUm441Q4yA~AaDGS)A)A##AaDGdE];;"1Q47|q0OOJqtAwqz34OO$ rWztext input must be of type `str` (single example), `list[str]` (batch or single pretokenized example) or `list[list[str]]` (batch of pretokenized examples) or `list[tuple[list[str], list[str]]]` (batch of pretokenized sequence pairs).zdwhen tokenizing batches of text, `text_pair` must be a list or tuple with the same length as `text`.zbatch length of `text`: z- does not match batch length of `text_pair`: .z:batch_text_or_text_pairs has to be a list or a tuple (got ))rrrrrr)ris_pretokenized)r+r{r|r}r~rrrroverflow_to_sample_mapping) tensor_type)rrZrQr[rR TypeErrorrzipr-rrrr encode_batchrr}&_eventual_warn_about_too_long_sequencerrr)(rrrrrrrrrrrrr{r|r}r~rrrrrr is_batchedbatch_text_or_text_pairsrr+tokens_and_encodingssanitized_tokensrrrrstacksanitized_encodingsrrtoksrbatched_outputr;s(&&&&&&&&&&&&&&&&&&&&, rTrTokenizersBackend._encode_plusYs0 ($D))W   )=i)H)HW  #D4-8hhThhjQUVWQX[_afZgFhJ#D4-8J )S))$Tc)n)D .s4yk:I'q*FOEZtC,@'A`d $?H(9':dV $2UDMBBLTRjMkLllmn  ''- 3!1% (  '#'#<#< ?? 0 04H H4HDOO 1OO00 $1/1 $& &  " "!&;&;*C+E'=+ # &  '*1--C&:N&:74DIIqQIQ&:EN$) S !.1ES0DWQdqdq0DS %)+ & )*> ? 9D!*qcC[8I4J.JJ*!@=W 9 :)+66I  7 7 :w W7''79LZhin4=V*'5&:&:&<&< Uc%j1nE!Hd9S9S%(Y^^&<(( NW "OS"s"M%M* M1 AM8 c6<V^8dQhRS[S[,RS[/#)rNrrP)rQrR)rSrs"rTrUrs  tCy S rWcVPPe&VPPPV4#RPV4#)Nr<)rsrxdecoder )rrs&&rTconvert_tokens_to_string*TokenizersBackend.convert_tokens_to_stringsJ%%--9  " " * * 1 1& 9 &! rWc`<V^8dQhRS[S[S[,,RS[RS[R,RS[/#)rN token_idsrclean_up_tokenization_spacesNrPr)rSrs"rTrUrs?))c?)")'+Tk ) )rWc VPRR4\V\4'dV.p\V\4'd VR,pVPP WR7pVeTM VP pV'd\VPP4PR8XdBVP'g0\PRVPP R24V#VPV4pV#)use_source_tokenizerNr)rr z=Ignoring clean_up_tokenization_spaces=True for BPE tokenizer aE. The clean_up_tokenization post-processing step is designed for WordPiece tokenizers and is destructive for BPE (it strips spaces before punctuation). Set clean_up_tokenization_spaces=False to suppress this warning, or set clean_up_tokenization_spaces_for_bpe_even_though_it_will_corrupt_output=True to force cleanup anyway.)rjrZrrirrrr-rsr,r|Gclean_up_tokenization_spaces_for_bpe_even_though_it_will_corrupt_outputr warning_oncerclean_up_tokenization)rrrrrrs&&&&, rT_decodeTokenizersBackend._decodes  )40 i % %" I i & &!+.I%%i%Y,7 )22 % ( T++112;;uDddd##//01-- 11$7 rWc <V^8dQhRS[S[P,RS[S[R3,RS[R,RS[R,RS[S[R3,/#)rNr file_names. legacy_formatNrrP)rRrkPathLiker[r )rSrs"rTrUr%s^bkk)#s(Od{  t  sCx rWc\V4p\PPY'd VR,MR\,4pVP P V4W%3,pV#)rr)rRrkrlr TOKENIZER_FILErssave)rrrrrr"s&&&&& rT_save_pretrained"TokenizersBackend._save_pretrained%s[^, o_s22Q__  ##N3"33 rWc \P!VPP44pVP R4pVP R4p Rp VR,R,R8Xd/VR,R&.VR,R&MVR,R,R 8XdoVR,R ,e\VR,R ,p VR,R,V ,^,p VeW9d WZ,p ^VR,R &V R ..VR,R&M?VR,R,R&9d/VR,R&M\ R VR,R, R 24hVeDRVR,9d6VR,R,V9d!WWR,R,,VR,R&\ P!\P!V44p .p VFpVP RR4pVP RR4pVR,R,R 8wd V'gKFVe!VR,V9dW^R,,VR&V P\R'/VB4K VeV PV4VR,R,R8Xd-RV9d&VR,R,eVR,R,VR&VR,R,R8Xd-RV9d&VR,R,eVR,R,VR&VR,R,R 8Xd V eWR&VR,eVR,R,R8XgzVR,R,R8XdRVR,9dx\;QJd*RVR,R,4F 'gK RM RM!RVR,R,44'd"\PP4VR&\ VR,R,,pV!R'RVRV /VBpV P#WVR7V Een\P!V P44pRV 9dV R,FpV R,V,R,pVe"VUu.uFpVP%VV4NK ppVV R,V,R&VF#pV P'V4pVeK\ R 4h VUu.uFpV P'V4NK upV R,V,R!&K R(FSpVV 9gK V V,wppVeVV9d VV,pV P'V4pVf \ R 4hVV.V V&KU V VR&\ P!\P!V44p VP(P+4p\,P.Fp\1VV4fK\1VV4pVeVV9d VV,pVP2P%VR4p\5V\4'd?\VVP6VP8VP:VP<RR"7VV&KVVV&K VP>'dVP>P+4M.pVeVPV4\AV4^8dVVR#&WR$&VPB!R'/VB#uupiuupi \DdIpR%\GT49d3TP R$R4TPB!R'/TBpT TnTuRp?#hRp?ii;i))u Trains a tokenizer on a new corpus with the same defaults (in terms of special tokens or tokenization pipeline) as the current one. Args: text_iterator (generator of `list[str]`): The training corpus. Should be a generator of batches of texts, for instance a list of lists of texts if you have everything in memory. vocab_size (`int`): The size of the vocabulary you want for your tokenizer. length (`int`, *optional*): The total number of sequences in the iterator. This is used to provide meaningful progress tracking new_special_tokens (list of `str` or `AddedToken`, *optional*): A list of new special tokens to add to the tokenizer you are training. special_tokens_map (`dict[str, str]`, *optional*): If you want to rename some of the special tokens this tokenizer uses, pass along a mapping old special token name to new special token name in this argument. kwargs (`dict[str, Any]`, *optional*): Additional keyword arguments passed along to the trainer from the 🤗 Tokenizers library. Returns: [`PreTrainedTokenizerFast`]: A new tokenizer of the same type as the original one, trained on `text_iterator`. r0r1Nr,r-r r.r/r rgz;This method does not support this type of tokenizer (found z-) only BPE, Unigram, WordLevel and WordPiece.rJridrYcontinuing_subword_prefixend_of_word_suffixr ByteLevelr7 pretokenizersc3:"TFpVR,R8HxK R#5i)r-rNrc).0 pretokenizers& rT .s!(X !(K7(XsTFinitial_alphabetrr))rtrainerrzQAttempted to set a token in the post processor that does not exist in the mappingr) single_wordlstriprstripr?rrLr(z7multiple values for keyword argument 'tokenizer_object')r r!rc)rsep)$rrloadsrto_strrjrrorurvr]rr\anypre_tokenizers_fastralphabetMODEL_TO_TRAINER_MAPPINGtrain_from_iteratorrtrrrrSPECIAL_TOKENS_ATTRIBUTESr~rrZrrrr?rLrrrrR)r text_iteratorrrnew_special_tokensspecial_tokens_maprrr0r1rJrrr) added_tokenrr trainer_classrtrained_tokenizer_jsonrrrr special_tokenspecial_token_fullrLr new_tokenizers&&&&&&, rTtrain_new_from_iterator)TokenizersBackend.train_new_from_iterator6sDDOO$:$:$<=%)).9 '++,<= ' "6 *e 3/1N7 #G ,02N7 #H - G $V , 9g&x0<'0:*73G4D3Ew'0 G $V ,0J J/1N7 #G ,Mn]dNeflNmMno>>   *~g66w' 48JJ3EU\F]^iFj3kN7 #K 0!**4::n+EF 'K!ooi6Gd+Ag&v.);G!-+i2HL^2^); 1)*:;;C+,<=cB8LF)5TZ![TZ5"4"8"8"FTZ![FLN#34S9(C!'#,#8#8#?#+", s#"(ouCuntejIDYDYZ_D`ntCuN#34S9%@<"0  N2-m>+F+ +{"\Dvj HCPQFR -t4 $ 8 8 +4 ($$ s0Z Z$ Z)) [<4<[70[<6[77[<c aa^RIo^RIHp ^RIHp ^RIHp HoV !^R7RVV3Rll4pV'g\4'dRpVEeV'gV'EgpV!V4'EdaV !VR VVVR R VR 7pR pVe\VR R 7;_uu_4p\P!V4pRRR4XPR4pVPR4pV'd<V PV4V PR48dV'dVe VR9dV#M/V'd(V PV4V PR48dV#RpV'gV'EgpV!V4'EdaV'dRV9d\VRVR,4V f<\VRR 4'g)\VRR 4\ P#RV R24V#V RJg\VRR 4'd\VRR4^RIpVP&P)VP+R4RR7pVP,p\/VVP&P04'dVVP,^&V#\/VVP&P24'dVP&P5R R R7pVP&P1VV.4VnV# +'giEL);i)a6 Patches mistral related tokenizers with incorrect regex if detected 1) Local file with an associated config saved next to it >> Model type one of the mistral models (on older versions) 2) Remote models on the hub from official mistral models >> Tags including `base_model:.*mistralai` N) lru_cache)version)rhf_api)maxsizec0V^8dQhR\R\/#)rNmodel_idrP)rRr )rSs"rTrU.__annotate__s  c d rWc<S!4PV4pTPe4SPRRP TP44'dR#R# \dR#i;i)Fzbase_model:.*mistralairT) model_infortagssearchr )rr,rres& rTis_base_mistral?TokenizersBackend._patch_mistral_regex..is_base_mistrals_ ++H5zz%995rwwuzz7JKK   sA A+*A+Tz config.jsonF) cache_dirrlocal_files_only%_raise_exceptions_for_missing_entries'_raise_exceptions_for_connection_errors _commit_hashr)r*transformers_versionrz5.0.0rz$The tokenizer you are loading from 'a' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue.z[^\r\n\p{L}\p{N}]?[\p{Lu}\p{Lt}\p{Lm}\p{Lo}\p{M}]*[\p{Ll}\p{Lm}\p{Lo}\p{M}]+|[^\r\n\p{L}\p{N}]?[\p{Lu}\p{Lt}\p{Lm}\p{Lo}\p{M}]+[\p{Ll}\p{Lm}\p{Lo}\p{M}]*|\p{N}| ?[^\s\p{L}\p{N}]+[\r\n/]*|\s*[\r\n]+|\s+(?!\S)|\s+isolated)patternbehavior)r use_regex)mistralmistral3voxtral ministralpixtral)r# functoolsr packagingrtransformers.utils.hubrrrrqrrrsrtparsesetattrr~rrrpre_tokenizersSplitRegexrrZr7 Metaspacer)rrpretrained_model_name_or_pathrr&r'r*is_localrrrrrrr$ _config_filemistral_config_detectedf_configr+transformers_model_typersplit_pretokenizercurrent_pretokenizerrr#s&&&&&&&&&&, @@rTr&TokenizersBackend._patch_mistral_regexs* '%> 3      00H ( 4 XX/:W*X*X&-#!16;8=) L', #',99Q"iilG:'.{{3I'J$*1++l*C' (GMM:N,ORYR_R_`gRh,h 3?3  )()gmm?\>]^eeH?'$.')EXZ_2`2`I':DA%)3)B)B)H)H * 0 0s!", *I*& ,5+B+B(!"6 8Q8Q8Z8Z[[5G //2"&&:Jrgs  #$7+-/16*^^.05=@?   H %"3/'( : ~!! &~|EXY,-j/j.j\),rW