+ QV-j^RIt^RIt^RIHtHtHtHtHt^RIH t H t H t H t H t HtHt^RIHt^RIHt!RR]4tR#)N)IteratorListOptionalUnionTuple) AddedTokenRegex Tokenizerdecoders normalizerspre_tokenizerstrainers)Unigram) BaseTokenizercaa]tRt^ toRtR V3RlV3RllltR V3RlRlltR V3RlRllt]V3RlR l4t R t Vt V;t #)SentencePieceUnigramTokenizerzrSentencePiece Unigram Tokenizer Represents the Unigram algorithm, with the pretokenization used by SentencePiece cb<V^8dQhRS[S[S[S[S[3,,,RS[RS[/#)vocab replacementadd_prefix_space)rrrstrfloatbool)format __classdict__s"ځ/Volumes/fast/ai/experiments/ui-tars-smoke/.venv/lib/python3.14/site-packages/tokenizers/implementations/sentencepiece_unigram.py __annotate__*SentencePieceUnigramTokenizer.__annotate__s<00U3:./000 0c <Ve\\V44pM\\44p\P!\P!4\P !4\P !\R4R4.4VnV'dRMRp\P!W%R7Vn \P!W%R7Vn RRRVR V/p\SV`=WF4R#) N {2,} alwaysneverrprepend_schememodelSentencePieceUnigramrr)r rr SequenceNmtNFKCReplacer normalizerr Metaspace pre_tokenizerr decodersuper__init__)selfrrr tokenizerr' parameters __class__s&&&& rr3&SentencePieceUnigramTokenizer.__init__s  !'%.1I!'),I*33 __  0 0 2K4G4GgX[4\ ] &67"0":":{"r $..;f  + ;  0 /r c <V^8dQhRS[S[S[S[,3,RS[RS[RS[S[S[S[S[3,,,RS[S[S[,,RS[S[,/#)rfiles vocab_size show_progressspecial_tokensinitial_alphabet unk_token)rrrintrrr)rrs"rrr,st.6.6S$s)^$.6.6 .6 !eCO& .6 #49- .6C=.6r c Vf.pVf.p\P!VVVVVR7p\V\4'dV.pVPP WR7R#)a1 Train the model using the given files Args: files (:obj:`List[str]`): A list of path to the files that we should use for training vocab_size (:obj:`int`): The size of the final vocabulary, including all tokens and alphabet. show_progress (:obj:`bool`): Whether to show progress bars while training. special_tokens (:obj:`List[Union[str, AddedToken]]`, `optional`): A list of special tokens the model should know of. initial_alphabet (:obj:`List[str]`, `optional`): A list of characters to include in the initial alphabet, even if not seen in the training dataset. If the strings contain more than one character, only the first one is kept. unk_token (:obj:`str`, `optional`): The unknown token to be used by the model. Nr;r=r<r>r?)trainer)rUnigramTrainer isinstancer _tokenizertrain)r4r:r;r<r=r>r?rCs&&&&&&& rrG#SentencePieceUnigramTokenizer.train,se<  !N  #! ))!)'-   eS ! !GE e5r c<V^8dQhRS[S[S[,S[S[S[,,3,RS[RS[RS[S[S[S[S[3,,,RS[S[S[,,RS[S[,RS[S[,/#)riteratorr;r<r=r>r?length)rrrr@rrrr)rrs"rrr\s4 4  x '>>?4 4  4 !eCO& 4 #49- 4 C=4  4 r c Vf.pVf.p\P!VVVVVR7pVPPVVVR7R#)a Train the model using the given iterator Args: iterator (:obj:`Union[Iterator[str], Iterator[Iterator[str]]]`): Any iterator over strings or list of strings vocab_size (:obj:`int`): The size of the final vocabulary, including all tokens and alphabet. show_progress (:obj:`bool`): Whether to show progress bars while training. special_tokens (:obj:`List[Union[str, AddedToken]]`, `optional`): A list of special tokens the model should know of. initial_alphabet (:obj:`List[str]`, `optional`): A list of characters to include in the initial alphabet, even if not seen in the training dataset. If the strings contain more than one character, only the first one is kept. unk_token (:obj:`str`, `optional`): The unknown token to be used by the model. length (:obj:`int`, `optional`): The total number of sequences in the iterator. This is used to provide meaningful progress tracking NrB)rCrK)rrDrFtrain_from_iterator) r4rJr;r<r=r>r?rKrCs &&&&&&&& rrM1SentencePieceUnigramTokenizer.train_from_iterator\s]D  !N  #! ))!)'-   ++  , r c <V^8dQhRS[/#)rfilename)r)rrs"rrrs1131r c|^RIpVPPR4^RIpTP 4pTP \TR4P44TPPpTPUu.uFqUPTP3NK ppTPPpTPP pTPP"p T^8wd \ R4hRp Rp \%\'YgT 44p T'dQ\(P*!\(P,!T4\(P.!\1R4R 4.4T nM:\(P*!\(P.!\1R4R 4.4T nT 'dR MR p \4P6!YR 7T n\:P6!YR 7T nR R/p\>P@!\BY4p\>PD!YT4T# \d \ R4hi;iuupi)rN.a\You don't seem to have the required protobuf file, in order to use this function you need to run `pip install protobuf` and `wget https://raw.githubusercontent.com/google/sentencepiece/master/python/src/sentencepiece/sentencepiece_model_pb2.py` for us to be able to read the intrinsics of your spm_file. `pip install sentencepiece` is not required.rbz]You're trying to run a `Unigram` model but you're file was trained with a different algorithm▁Tr"r#r$r%r&r(r))#syspathappendsentencepiece_model_pb2 Exception ModelProtoParseFromStringopenreadnormalizer_specprecompiled_charsmappiecespiecescore trainer_specunk_id model_type byte_fallbackr rr r* Precompiledr-r r.r r/r0r r1r__new__rr3)rPrUr(mr_rarrdrerfrrr5r'r6objs& rfrom_spm&SentencePieceUnigramTokenizer.from_spms   HHOOC 3     $x.3356 00EE9:B++u{{+B&&^^.. 44 ?o  ge]CD #.#7#7++,@A''g<$I $/#7#79L9LUSZ^]`9a8b#cI %57"0":":{"r $..;f  + ##$A9Ysz: U o  Cs#H H9H6)NrTT)@TNNN)rnTNNNN) __name__ __module__ __qualname____firstlineno____doc__r3rGrM staticmethodrk__static_attributes____classdictcell__ __classcell__)r7rs@@rrr sC 006.6.6`4 4 l111r r)jsonostypingrrrrr tokenizersrr r r r r rtokenizers.modelsrbase_tokenizerrrrmr rr~s. 99ddd%)yMyr