._run sk
R,,'
B))%*:*:DA
R))%*:*:LQ. .
' .
.
sC ' (.r c " / VCRVRV/Cp\ VR4 ' d RVR& V P ! VP 3/ VB G Rj xL
pVR, P R
, pV^ VR13, p VP V RR 7 p
WV 3# LB5i) zNRun generation to completion via ``model.generate()`` on the inference thread.r rQ r r' r Nr Tr r )r rx r shapedecode) r0 rL re r r~ r8 generate_kwargs sequences input_lenrt r' s &&&&&& r! r &GenerateManager.generate_non_streaming s ^V]%8*kS\] 5, ' '17O - . ++ENNNoNN ;'--b1 !!YZ-0
4 H
-- Os AB BAB c & < V ^8 d QhRS[ RS[/# r% rh rM )r r )r) r* s "r! r+ r $ s 8 8 8v 8r c B V P P ! V.VO5/ VB # )rp )r` rq r0 rh r r s &&*,r! rq GenerateManager.submit$ s! ||""27777r c : < V ^8 d QhRS[ RS[P /# r )r r r )r) r* s "r! r+ r ( s > >x >W^^ >r c B V P P ! V.VO5/ VB # ru )r` rx r s &&*,r! rx GenerateManager.async_submit( s! ||((=d=f==r c < V ^8 d QhRR/# r r )r) r* s "r! r+ r , s
d
r c R # r. r r, s &r! r GenerateManager.stop, r r )r` rL )
r r r r r3 r1 r r rq rx r r r4 r5 s @r! r r s@ P) B. .(8 8> >
r r c a ] tR tRt o RtRV 3R lR lltV 3R lR ltV 3R lR ltV 3R
lR ltRV 3R lR
llt V 3R lR lt
] V 3R lR l4 t V 3R lR lt
RtV tR# )CBGenerateManageri0 aQ Continuous batching generation via paged attention.
Translates between the handler's text-level asyncio.Queue and CB's
token-level interface. Per-request: ``max_new_tokens``, ``eos_token_id``.
The CB manager is initialized lazily on the first request via
:meth:`ensure_initialized`, using that request's ``gen_config`` for shared
sampling params (temperature, top_p, do_sample).
.. todo:: Remove :meth:`init_cb` when CB supports per-request
generation config. At that point, ``gen_config`` can be passed directly
to ``add_request`` and the CB manager no longer needs a shared config.
Nc < V ^8 d QhRR/# )r% cb_config ContinuousBatchingConfig | Noner )r) r* s "r! r+ CBGenerateManager.__annotate__? s $ $"C $r c R V n Wn R # r. r;
_cb_config)r0 r s &&r! r1 CBGenerateManager.__init__? s 59#r c & < V ^8 d QhRRRRRR/# r} r )r) r* s "r! r+ r C s% . zl"TXXMaMaLbc
%E r c < V ^8 d QhRRRRRS[ RRRS[R S[ R
, R S[ R
, R S[S[P S[3, /# r )rO r( r r r r5 )r) r* s "r! r+ r c sq 8$ 8$ 8$ >8$ 8$
' 8$
8$ D[8$ +8$
w}}j( )8$r c
a
a V P pVf \ R4 hV P V4 \ P ! 4 p \ P
! 4 oVR, p
VP
V
VRVP VP R7 p\ VRV4 P p \ V P VV V SVVR7 o
V
V3R lp VP W\4 SS
3# )zFStart streaming CB generation. Registers a per-request output handler.3CB manager not initialized. Call `init_cb()` first.r T)r8 streamingmax_new_tokens eos_token_idrQ r c | < SP V 4 V P e7 SP \ V P 4 4 SP 4 R # V P 4 ' d SP 4 R # R # \ d/ pSP \ \ T4 4 4 R p?R # R p?ii ; ir. )r( errorr% r# rx is_finishedrf r( )r@ rk r
text_queues & r!
_on_output8CBGenerateManager.generate_streaming.._on_output s
< V$ <<+ )),v||*DE LLN'')) LLN *
<%%l3q6&:;;
/. /.
' /.
/.
sCc" #/.r c Z a" V P pVf \ R4 hV P V4 VR, p\ V4 p\ P
! 4 p V P
4 oV3R lp
VP WZ4 VP VVVP RVP R7 SG Rj xL
p V P eE VP e \ RV RV P 24 h\ R V RV P 24 hV P p VP V R
R 7 p
WV 3# Ly5i) zcRun non-streaming CB generation. Registers a handler that resolves an asyncio.Future on completion.Nr r c Z < SP 4 ' g SP V 4 R # R # r. )donere )rj ri s &r!
_on_result._on_result s! ;;==!!&) !r F)r8 r r r z CB worker died during request r zCB generation failed for Tr )r; r r r r rv rw r r r r r r r= rB r )r0 rL re r r~ r8 r r r r r rj rt r' ri s &&&&&& @r! r (CBGenerateManager.generate_non_streaming s0 XX
: TU U *%;' N '')##% * ""::
!%44#00
<< #~~)'*H TVW]WcWcVd(eff !::,b WX X //
4 H
-- s B,D+/D)0A:D+c < V ^8 d QhRR/# )r% rM r r )r) r* s "r! r+ r s 2 2; 2r c V P e V P P f \ R4 hV P P P # )z*The CB scheduler (for testing/monitoring).z.Continuous batching processor not initialized.)r; batch_processorr schedulerr, s &r! r CBGenerateManager.scheduler s? 88 txx 7 7 ? OP Pxx''111r c < V ^8 d QhRR/# r r )r) r* s "r! r+ r s 1 1d 1r c ` V P e V P P R^R7 R # R # )NT)blocktimeout)r; r r, s &r! r CBGenerateManager.stop s% 88 HHMMaM 0 r r r. rL )r r r r r3 r1 r r r r r propertyr r r r4 r5 s @r! r r 0 sh $ $ "@ @ 8$ 8$t/. /.b 2 2 1 1r r c a ] tR tRt o RtRV 3R lR lltV 3R lR ltRV 3R lR lltV 3R
lR ltV 3R lR
lt Rt
V t R# )GenerationStatei a Shared generation state across all handlers.
Manages per-model :class:`GenerateManager` instances (each with its own
:class:`InferenceThread` so different models can run concurrently while
``torch.compile`` / CUDA graphs require same-model-same-thread) and a
single :class:`CBGenerateManager` for continuous batching.
Args:
continuous_batching (`bool`, *optional*, defaults to `False`):
Whether to use continuous batching with paged attention instead of
sequential ``model.generate()`` calls.
Nc * < V ^8 d QhRS[ RS[ RR/# )r% continuous_batchingcompiler r r )r) r* s "r! r+ GenerationState.__annotate__ s) - - ! - - 5 -r c T Wn W n W0n / V n R V n R V n R # r. )_continuous_batching_compiler _generate_managers _cb_manager _cb_model_id)r0 r r r s &&&&r! r1 GenerationState.__init__ s, %8!
#>@ 59(,r c * < V ^8 d QhRRRS[ RS[/# )r% rL r modalityrM )r r )r) r* s "r! r+ r s$ -> ( W[ r c V P ' g R# \ VR4 ;' d V\ P 8H pV' g- \ P VP P R24 V# )a' Check if continuous batching can be used for this model and modality.
Args:
model (`PreTrainedModel`): The loaded model.
modality (`Modality`): The detected model modality (LLM, VLM, etc.).
Returns:
`bool`: ``True`` if CB is enabled and the model supports it, ``False`` otherwise.
Fr zM does not support continuous batching. Falling back to sequential generation.)r r r r logger warning_oncer r )r0 rL r cans &&& r! use_continuous_batching'GenerationState.use_continuous_batching sc (((e 78UUX =U ??++, -9 9
r c , < V ^8 d QhRS[ RS[RS[/# )r% r use_cbrM )r( r r{ )r) r* s "r! r+ r s# 1 1C 1 1BU 1r c | V' d| V P V8w d0 V P e" V P P 4 RV n V P f" \ V P R7 V n Wn V P # WP
9 d \
4 V P
V& V P
V, # )a6 Return a per-model generation manager, lazily created on first request.
Args:
model_id (`str`): The model ID in ``'model_id@revision'`` format.
use_cb (`bool`): Whether to return a CB manager or a sequential one.
Returns:
`BaseGenerateManager`: Either a `GenerateManager` or `CBGenerateManager`.
N)r )r r r r r r r )r0 r r s &&&r! get_managerGenerationState.get_manager s H,##/$$))+'+D$ '#4t#O $,!## # 22 20?0AD # #H -&&x00r c < V ^8 d QhRR/# r r )r) r* s "r! r+ r s $ $$ $r c h V P e$ V P P 4 RV n R# R# )z$Stop any active generation managers.N)r r r, s &r! shutdownGenerationState.shutdown s- ' ! ! # #D (r c < V ^8 d QhRS[ /# rn r )r) r* s "r! r+ r & s G GT Gr c b V P RJ ;' g V P P 4 # )zTWhether the CB worker is healthy. ``True`` if CB is disabled or not yet initialized.N)r r r, s &r! is_cb_aliveGenerationState.is_cb_alive& s* 4'FF4+;+;+D+D+FFr )r r r r r r )FFNF) r r r r r3 r1 r
r r r r r4 r5 s @r! r r s> - - (1 1.$ $ G Gr r c a ] tR tRt o RtRt]! 4 tRV 3R lR lltV 3R lR lt ]
V 3R lR l4 t V 3R
lR lt RV 3R lR
llt
]
V 3R lR l4 tV 3R ltRtV tR# ) BaseHandleri+ a Shared logic for chat completion and responses handlers.
Provides model resolution, generation config building, and SSE formatting.
Generation is delegated to the shared :class:`GenerationState`.
Args:
model_manager (`ModelManager`):
Handles model loading, caching, and lifecycle.
generation_state (`GenerationState`):
Shared state managing per-model generation managers.
Nc 8 < V ^8 d QhRRRS[ RS[R, /# )r%
model_managerr generation_statechat_template_kwargsN)r rO )r) r* s "r! r+ BaseHandler.__annotate__; s- ? ?%? *? #Tk ?r c @ Wn W n T;' g / V n R # r. )r r! r" )r0 r r! r" s &&&&r! r1 BaseHandler.__init__; s + 0 $8$>$>B!r c $ < V ^8 d QhRS[ RR/# )r% bodyrM NrN )r) r* s "r! r+ r# E s Y Yd Yt Yr c > ^ RI Hp \ VP 4 4 pV P e< V\ V P R\ 4 4 ,
pV' d V! RRV 2R7 hW0P , pV' d \ P RV 24 R# R# )zMValidate request fields against the handler's params class and unused fields.
HTTPExceptionN__mutable_keys__i z"Unexpected fields in the request: status_codedetailz,Ignoring unsupported fields in the request: ) fastapir* r0 keys_valid_params_classr` _unused_fieldsr
r )r0 r' r*
input_keys
unexpectedunuseds && r! _validate_request BaseHandler._validate_requestE s )%
# # /#gd.F.FHZ\_\a&bbJ#>`ak`lW3X Y $
.E F //7;L;[;[^b;b37!0 88M " .,1$}2E,F )T-()S0.3!+ 88G (&+DM&: # 88F ' 4< ( ) ) ).?.T.T.\5= 2 */ ' ! r c L < V ^8 d QhRS[ S[, RS[RS[ S[, /# )r% messagesr rM )rq rO r )r) r* s "r! r+ r# s2 C C T$Z C 8 C X\]aXb C r c R . pV EF pRVR, R. /pRV9 d . pVR, F p\ P ! V4 pVP R4 ;' g Tp\ VR, \ 4 ' d! \
P ! VR, 4 VR& VP V4 K WTR& RV9 d
VR, VR& RV9 d . MVP R4 ;' g . p\ V\ 4 ' d RRRV/.pV EF p V R, p
V
R9 d& VR, P RRRV R, /4 K9 V
R9 dl V\ P \ P 39 dG V R , p \ V \ 4 ' d
V R
, p VR, P RR R
V /4 K V
R 8X dw V\ P 8X db V R , p \ V \ 4 ' d V P R
R4 MRp
V R, pVR, P RRR
RV
RV 2/4 EK( V
R8X dS V\ P \ P 39 d. VR, P RRR
V R, R
, /4 EK V
R8X g EK V\ P 8X g EK VR, P RRR
V R, R
, /4 EK V\ P 8X d# RP R VR, 4 4 VR& VP V4 EK V# )a Convert OpenAI-format messages to the format expected by HF processors.
All modalities extract text. VLM additionally handles ``image_url`` and ``video_url``.
MULTIMODAL handles all of the above plus ``input_audio`` and ``audio_url``.
For LLMs, the content parts are collapsed into a plain text string.
Args:
messages (`list[dict]`): OpenAI-format chat messages.
modality (`Modality`): The model modality (LLM, VLM, or MULTIMODAL).
Returns:
`list[dict]`: Processor-compatible messages.
rolerz rU rk rI tool_call_idrA r' image_urlurlimage input_audior) wavdataaudioz data:audio/z;base64, video_urlvideo audio_url c 3 2 " T F
qR , x K R# 5i)r' Nr )rW r s & r! r[ ABaseHandler.get_processor_inputs_from_messages.. s ,R@Q1vYY@Qs )r'
input_text output_text)r] input_image)rP rQ rl rm r( rE rO r r r r rO r join)rY r processor_inputsmessageru rU tcrh raw_contentrz content_typer^ r` fmt audio_b64s && r! "get_processor_inputs_from_messages.BaseHandler.get_processor_inputs_from_messages s G gfoy"=F w&
!,//B r*B
+11rB!"[/377*.**R _*E; %%b) 0 (2|$ ()0 )@~& !- 7"gkk)>T>Z>ZXZK+s++ & D E & &v #HH9%,,fffgfo-VW!%AAhS[S_S_aiatatRuFu!+.C!#t,, !%j9%,,fguc-JK!]2x8CVCV7V")-"8K>HVZ>[>[+//(E:afC +F 3I9%,,fgu TWSXX`aj`kFl-mn![0X(,,PXPcPcAd5d9%,,fgugkFZ[`Fa-bc![0XATAT5T9%,,fgugkFZ[`Fa-bc- '2 8<<'$'HH,Ry@Q,R$Ry! # #F +e f r c P < V ^8 d Qh/ S[ R, ;R& S[S[, ;R& # )r% Nr1 r2 )rA r0 r( )r) r* s "r! r+ r# + s' + H$ r )r" r! r r. r )r r r r r3 r1 r0 r2 r1 r6 staticmethodr> rF rV ru __annotate_func__r r4 r5 s @r! r r + sx
(, "uN? ? Y Y G G * **0! 0!d C C m r r ) qwen2 qwen2_moeqwen2_vl
qwen2_5_vlqwen3 qwen3_moe
qwen3_nextqwen3_vl qwen3_vl_moe)qwen3_5 qwen3_5_moe)z
<|channel>thought
r. )Gr3 r rP enumrE r abcr r collections.abcr concurrent.futuresr r r typingr transformers.utilsr pydantic
tokenizersrP rN r
r r r
r :transformers.generation.continuous_batching.continuous_apir 4transformers.generation.continuous_batching.requestsr 5transformers.generation.continuous_batching.schedulerr r r
get_loggerr r
X_REQUEST_IDEnumr r# rf r7 r( r: r r= rc rg ro rv r r r r r r r r r r5 rR rX rZ r{ r r r r r r r! r sV # $ % & eUO+
H % tyy P9 PC
}
~ ? G fh
F;
}
~ \ G VX . +-l"
!1* ZB<".0 i[ : * )
a
,
w7 M D(82(
(
VDNR R jL2 L2^ !& &RA># A>HD
) D
Ng1+ g1TNG NGby y r