+
QV-j@ ^ RI Ht ^ RIt^ RIt^ RIt^ RIHt ^ RIHtH t H
t
]' d ^RI H t ! R R4 t
! R R ]
4 t ! R
R ]4 t ! R R
]4 t ! R R]4 tR# ) ) annotationsN)Queue)
TYPE_CHECKINGAnycast)PreTrainedTokenizerBasec & ] tR t^ tRtR tR tRtR# ) BaseStreamerz?
Base class from which `.generate()` streamers should inherit.
c
\ 4 h)z;Function that is called by `.generate()` to push new tokensNotImplementedErrorselfvalues &&r/Volumes/fast/ai/experiments/ui-tars-smoke/.venv/lib/python3.14/site-packages/transformers/generation/streamers.pyputBaseStreamer.put! !## c
\ 4 h)zHFunction that is called by `.generate()` to signal the end of generationr r s &r endBaseStreamer.end% r r N)__name__
__module__ __qualname____firstlineno____doc__r r __static_attributes__r r r r
r
s $$r r
c L ] tR t^*tRtR R R lltR tR tR R R lltR t R t
R
# ) TextStreameraT
Simple text streamer that prints the token(s) to stdout as soon as entire words are formed.
Parameters:
tokenizer (`AutoTokenizer`):
The tokenizer used to decode the tokens.
skip_prompt (`bool`, *optional*, defaults to `False`):
Whether to skip the prompt to `.generate()` or not. Useful e.g. for chatbots.
decode_kwargs (`dict`, *optional*):
Additional keyword arguments to pass to the tokenizer's `decode` method.
Examples:
```python
>>> from transformers import AutoModelForCausalLM, AutoTokenizer, TextStreamer
>>> tok = AutoTokenizer.from_pretrained("openai-community/gpt2")
>>> model = AutoModelForCausalLM.from_pretrained("openai-community/gpt2")
>>> inputs = tok(["An increasing sequence: one,"], return_tensors="pt")
>>> streamer = TextStreamer(tok)
>>> # Despite returning the usual output, the streamer will also print the generated text to stdout.
>>> _ = model.generate(**inputs, streamer=streamer, max_new_tokens=20)
An increasing sequence: one, two, three, four, five, six, seven, eight, nine, ten, eleven,
```
c $ V ^8 d QhRRRRRR/# ) tokenizerr skip_promptbool
decode_kwargsr r )formats "r __annotate__TextStreamer.__annotate__F s# + +"9 + +gj +r c T Wn W n W0n . V n ^ V n RV n R# )r TN)r% r& r( token_cache print_lennext_tokens_are_prompt)r r% r& r( s &&&,r __init__TextStreamer.__init__F s, "& * ')&*#r c
\ VP 4 ^8 d$ VP ^ , ^8 d \ R4 h\ VP 4 ^8 d
V^ , pV P ' d V P ' d
RV n R# V P
P
VP 4 4 \ \ V P P ! V P
3/ V P B 4 pVP R4 ' d W P R p. V n ^ V n M\ V4 ^ 8 dY V P \! VR, 4 4 ' d2 W P R pV ;P \ V4 ,
un MFW P VP# R4 ^, pV ;P \ V4 ,
un V P% V4 R# )z]
Receives tokens, decodes them, and prints them to stdout as soon as they form entire words.
z'TextStreamer only supports batch size 1FN
)lenshape
ValueErrorr& r/ r- extendtolistr strr% decoder( endswithr. _is_chinese_charordrfindon_finalized_text)r r textprintable_texts && r r TextStreamer.putP sf u{{ a EKKNQ$6 FG G
!!HE ; ; ;*/D '
/C ..t/?/?V4CUCUVW == !.."2 3N !D DN
Y]t 4 4Sb] C C !.."2 3N NNc. 1 1N "..4::c?Q3F GN NNc. 1 1N ~.r c
* \ V P 4 ^ 8 d] \ \ V P P
! V P 3/ V P B 4 pWP R p. V n ^ V n MRpRV n V P VRR7 R# )z;Flushes any remaining cache and prints a newline to stdout.N T)
stream_end)
r6 r- r r; r% r< r( r. r/ rA )r rB rC s & r r TextStreamer.endr s t 1 $T^^ 2 243C3C ZtGYGY Z[D !.."2 3N !D DN N&*# ~$ ?r c V ^8 d QhRRRR/# r$ rB r; rG r' r )r) s "r r* r+ s D Dc Dt Dr c
6 \ TRV' g RMRR7 R# )zNPrints the new text to stdout. If the stream is ending, also prints a newline.TrF N)flushr )printr rB rG s &&&r rA TextStreamer.on_finalized_text s
d$jBdCr c
VR8 d VR8: gc VR8 d VR8: gU VR8 d VR8: gG VR8 d VR8: g9 VR 8 d VR
8: g+ VR 8 d VR 8: g VR
8 d VR8: g VR8 d
VR8: d R# R# )z6Checks whether CP is the codepoint of a CJK character.i N i i 4 iM i iߦ i i? i@ i i i i i i i TFr )r cps &&r r> TextStreamer._is_chinese_char sr 6\bFlf v g
"-g
"-g
"-g
"-f v g
"-r )r( r/ r. r& r- r% NF) r r r r r r0 r r rA r> r r r r r" r" * s# 6+ /D @ Dr r" c R a ] tR t^tRtR R V 3R llltR
R R lltR tR tRt V ;t
# ) TextIteratorStreamera
Streamer that stores print-ready text in a queue, to be used by a downstream application as an iterator. This is
useful for applications that benefit from accessing the generated text in a non-blocking way (e.g. in an interactive
Gradio demo).
Parameters:
tokenizer (`AutoTokenizer`):
The tokenizer used to decode the tokens.
skip_prompt (`bool`, *optional*, defaults to `False`):
Whether to skip the prompt to `.generate()` or not. Useful e.g. for chatbots.
timeout (`float`, *optional*):
The timeout for the text queue. If `None`, the queue will block indefinitely. Useful to handle exceptions
in `.generate()`, when it is called in a separate thread.
decode_kwargs (`dict`, *optional*):
Additional keyword arguments to pass to the tokenizer's `decode` method.
Examples:
```python
>>> from transformers import AutoModelForCausalLM, AutoTokenizer, TextIteratorStreamer
>>> from threading import Thread
>>> tok = AutoTokenizer.from_pretrained("openai-community/gpt2")
>>> model = AutoModelForCausalLM.from_pretrained("openai-community/gpt2")
>>> inputs = tok(["An increasing sequence: one,"], return_tensors="pt")
>>> streamer = TextIteratorStreamer(tok)
>>> # Run the generation in a separate thread, so that we can fetch the generated text in a non-blocking way.
>>> generation_kwargs = dict(inputs, streamer=streamer, max_new_tokens=20)
>>> thread = Thread(target=model.generate, kwargs=generation_kwargs)
>>> thread.start()
>>> generated_text = ""
>>> for new_text in streamer:
... generated_text += new_text
>>> generated_text
'An increasing sequence: one, two, three, four, five, six, seven, eight, nine, ten, eleven,'
```
c ( V ^8 d QhRRRRRRRR/# r$ r% r r& r' timeout float | Noner( r r )r) s "r r* !TextIteratorStreamer.__annotate__ s2
*
r c b < \ SV ` ! W3/ VB \ 4 V n R V n W0n R # N)superr0 r
text_queue stop_signalrX )r r% r& rX r( __class__s &&&&,r r0 TextIteratorStreamer.__init__ s- A=A ' r c V ^8 d QhRRRR/# rJ r )r) s "r r* rZ s H Hc Ht Hr c
V P P WP R7 V' d4 V P P V P V P R7 R# R# )\Put the new text in the queue. If the stream is ending, also put a stop signal in the queue.rX N)r^ r rX r_ rN s &&&r rA &TextIteratorStreamer.on_finalized_text sD D,,7 OO 0 0$,, G r c V # r\ r r s &r __iter__ TextIteratorStreamer.__iter__ r c V P P V P R 7 pWP 8X d \ 4 hV# )re )r^ getrX r_
StopIterationr s & r __next__ TextIteratorStreamer.__next__ s6 ##DLL#9 $$ $ / !Lr )r_ r^ rX FNrS ) r r r r r r0 rA rh rn r
__classcell__r` s @r rU rU s( %N
H r rU c R a ] tR t^tRtR R V 3R llltR
R R lltR tR tRt V ;t
# ) AsyncTextIteratorStreamera
Streamer that stores print-ready text in a queue, to be used by a downstream application as an async iterator.
This is useful for applications that benefit from accessing the generated text asynchronously (e.g. in an
interactive Gradio demo).
Parameters:
tokenizer (`AutoTokenizer`):
The tokenizer used to decode the tokens.
skip_prompt (`bool`, *optional*, defaults to `False`):
Whether to skip the prompt to `.generate()` or not. Useful e.g. for chatbots.
timeout (`float`, *optional*):
The timeout for the text queue. If `None`, the queue will block indefinitely. Useful to handle exceptions
in `.generate()`, when it is called in a separate thread.
decode_kwargs (`dict`, *optional*):
Additional keyword arguments to pass to the tokenizer's `decode` method.
Raises:
TimeoutError: If token generation time exceeds timeout value.
Examples:
```python
>>> from transformers import AutoModelForCausalLM, AutoTokenizer, AsyncTextIteratorStreamer
>>> from threading import Thread
>>> import asyncio
>>> tok = AutoTokenizer.from_pretrained("openai-community/gpt2")
>>> model = AutoModelForCausalLM.from_pretrained("openai-community/gpt2")
>>> inputs = tok(["An increasing sequence: one,"], return_tensors="pt")
>>> # Run the generation in a separate thread, so that we can fetch the generated text in a non-blocking way.
>>> async def main():
... # Important: AsyncTextIteratorStreamer must be initialized inside a coroutine!
... streamer = AsyncTextIteratorStreamer(tok)
... generation_kwargs = dict(inputs, streamer=streamer, max_new_tokens=20)
... thread = Thread(target=model.generate, kwargs=generation_kwargs)
... thread.start()
... generated_text = ""
... async for new_text in streamer:
... generated_text += new_text
>>> print(generated_text)
>>> asyncio.run(main())
An increasing sequence: one, two, three, four, five, six, seven, eight, nine, ten, eleven,
```
c ( V ^8 d QhRRRRRRRR/# rW r )r) s "r r* &AsyncTextIteratorStreamer.__annotate__ s8 U U*U U U
Ur c f < \ SV ` ! W3/ VB \ P ! 4 V n R V n W0n \ P ! 4 V n \ \ RR 4 p\ P R8 ;' d \ V4 V n
V P ' d WPn R # R V n R # )NrX ) )r] r0 asyncior r^ r_ rX get_running_looploopgetattrsys version_infocallablehas_asyncio_timeoutasyncio_timeout)r r% r& rX r( timeout_contextr` s &&&&, r r0 "AsyncTextIteratorStreamer.__init__ s A=A!--/ ,,. !'9d;##3w#>#\#\8OC\ 262J2J2J PT r c V ^8 d QhRRRR/# rJ r )r) s "r r* rv ! s Y Yc Yt Yr c
V P P V P P V4 V' d= V P P V P P V P 4 R# R# )rd N)r| call_soon_threadsafer^
put_nowaitr_ rN s &&&r rA +AsyncTextIteratorStreamer.on_finalized_text! sL &&t'A'A4H II * *4??+E+EtGWGW X r c V # r\ r r s &r __aiter__#AsyncTextIteratorStreamer.__aiter__' rj r c V " V P ' dn V P e` V P V P 4 ;_uu_4 GR j xL
V P P 4 G R j xL
pR R R 4 GR j xL
MB\
P ! V P P 4 V P R7 G R j xL
pXV P 8X d \ 4 hV# L Lv Lh + GR j xL
' g i L<; i LC \
P d \ 4 hi ; i5i)Nre )
r r rX r^ rl rz wait_forr_ StopAsyncIteration TimeoutErrorr s & r __anext__#AsyncTextIteratorStreamer.__anext__* s '''D,@,@,L // ==="&//"5"5"7 7E >== & . .t/B/B/Ddll [[ ((((** > 7 >=== \## ! . !s D)AD C" D C(-C$.C(2 D =C&>AD DD D)"D $C(&D (D .C1/
D :D < D D&&D))r r r| r_ r^ rX rp rS ) r r r r r r0 rA r r r rq rr s @r rt rt s* ,\U U Y
r rt c Z a ] tR tRtRtR
R V 3R llltR tR tV 3R ltV 3R lt R t
V ;t # ) TextDiffusionStreameri: a
Streamer that prints text diffusion outputs. Intermediate diffusion steps (drafts) are temporary
and overwritten by subsequent drafts, and removed when confirmed text is printed.
If you're running on an environment like tmux, the draft text may fail to overwrite itself.
Parameters:
tokenizer (`AutoTokenizer`):
The tokenized used to decode the tokens.
skip_prompt (`bool`, *optional*, defaults to `False`):
Whether to skip the prompt to `.generate()` or not. Useful e.g. for chatbots.
sleep_time (`float`, *optional*):
Time to sleep between diffusion drafts, which may be helpful to visualize intermediate outputs.
decode_kwargs (`dict`, *optional*):
Additional keyword arguments to pass to the tokenizer's `decode` method.
Examples:
```python
>>> from transformers import DiffusionGemmaForBlockDiffusion, AutoProcessor, TextDiffusionStreamer
>>> model = DiffusionGemmaForBlockDiffusion.from_pretrained(
... "google/diffusiongemma-26B-A4B-it", device_map="auto",
... )
>>> processor = AutoProcessor.from_pretrained("google/diffusiongemma-26B-A4B-it")
>>> chat = [{"role": "user", "content": "Why is the sky blue?"},]
>>> input_ids = processor.apply_chat_template(
... chat, tokenize=True, return_tensors="pt", add_generation_prompt=True
... )
>>> streamer = TextDiffusionStreamer(tokenizer=processor.tokenizer)
>>> model.generate(input_ids.to(model.device), max_new_tokens=512, streamer=streamer)
```
c ( V ^8 d QhRRRRRRRR/# ) r$ r% r r& r'
sleep_timerY r( r r )r) s "r r* "TextDiffusionStreamer.__annotate__c s2
%
%*
%
% !
%
%r c R < \ SV ` ! W3/ VB R V n R V n W0n R# rp )r] r0
_has_draft
_takes_logitsr )r r% r& r r( r` s &&&&,r r0 TextDiffusionStreamer.__init__c s- A=A #$r c X V P ' d \ R RRR7 RV n R# R# )z8[JrF Tr rL FN)r rM r s &r _clear_draft"TextDiffusionStreamer._clear_draftr s# ??? -Rt 4 #DO r c
V P 4 \ VP 4 ^8 d$ VP ^ , ^8 d \ R4 h\ VP 4 ^8 d
V^ , pV P P
! V3/ V P B p\ RRRR7 \ RV R2RRR7 RV n V P e# \ P ! V P 4 R# R# ) zq
Receives the full sequence of draft tokens, decodes them, and prints them in yellow.
Overwrites previous draft.
z0TextDiffusionStreamer only supports batch size 1z7rF Tr z[33mz[0mN) r r6 r7 r8 r% r< r( rM r r timesleep)r r kwargsrB s &&, r put_draft TextDiffusionStreamer.put_draftx s
u{{ a EKKNQ$6 OP P
!!HE~~$$UAd.@.@A g2T*
g&Bd; ?? & JJt ' 'r c
F < V P 4 \ SV ` V4 R# )zEReceives confirmed tokens, clears draft, and prints them permanently.N)r r] r )r r r` s &&r r TextDiffusionStreamer.put s
Er c
D < V P 4 \ SV ` 4 R# )z1Flushes any remaining cache and prints a newline.N)r r] r )r r` s &r r TextDiffusionStreamer.end s
r )r r r rp ) r r r r r r0 r r r r r rq rr s @r r r : s, &P
%
% $ (,
r r )
__future__r rz r~ r queuer typingr r r tokenization_utils_baser r
r" rU rt r r r r r si #
+ + A $ $ p< pfB< BJU U p\L \r