+ UV-jF ^RIt^RIt^RIt^RIt^RIt^RIHt^RIHtH t ^RI H t ^RI H t ^RIHtHtHt^RIHtHtHtHtHt^RIHt^RIHt^R IHt^R IH t!^R I"H#t#H$t$H%t%H&t&H't'H(t(H)t)H*t*H+t+H,t,H-t-H.t.H/t/H0t0H1t1^R I2H3t3H4t4^R I5H6t6H7t7^RI8H9t9^RI:H;t;Ht>^RI?H@t@^RIAHBtBHCtC^RIDHEtEHFtFHGtG^RIHHHtH]P!R4tJRtKRtLRtM^dtN^ tO!RR]P4tQRtRRtSRtTRtURtVRtWR R!ltXR"R#ltYR$R%ltZR&R'lt[R(R)lt\R*R+lt]!R,R-4t^R.RR/R/R0R1llt_R2t`R3taR4tbR5tcR6R7ltdR8teR9tfR:R;ltgR<thR=R>ltiR?R@ltjRAtkRBtlRCRDltmRERFltn!RGRH4toRIRJltpRKRLltqRMRRNRRORRPRRQRRR^RS^RTRRURRVRRWR/ RXRYlltrRZR[lts]!R\R]44tt]!R^R_44tu]!R`Ra44tv]!RbRc44twRdReltx!RfRg4ty!RhRi4tzR#)jN)deque) dataclassfield)Empty)Queue)EventLockThread)Callable GeneratorListOptionalTuple) HTTPException)make_logits_processors)apc)DEFAULT_KV_GROUP_SIZEDEFAULT_KV_QUANT_SCHEMEDEFAULT_MAX_TOKENSDEFAULT_PREFILL_STEP_SIZEDEFAULT_QUANTIZED_KV_STARTDEFAULT_REPETITION_CONTEXT_SIZE DEFAULT_SEEDDEFAULT_TEMPERATUREDEFAULT_THINKING_END_TOKENDEFAULT_THINKING_START_TOKEN DEFAULT_TOP_PBatchGenerator_chunked_prefill_enabled _make_cache_merge_prefill_prompt_kwargs)generation_stream wired_limit)diffusion_generation_familystream_diffusion_generate)top_p_sampling)make_speculative_prompt_cacherun_speculative_server_roundsspeculative_hidden_statespeculative_prefill_kwargs)ThinkingAwareLogitsProcessor)_ServerTokenStreamermake_streaming_detokenizer)ThinkingBudgetCriterialoadprepare_inputs)runtimezmlx_vlm.serverg@@Fc]tRt^:tRtRtR#)PromptTooLongErrorzCRaised when a request exceeds the configured server context budget.N)__name__ __module__ __qualname____firstlineno____doc____static_attributes__r5j/Volumes/fast/ai/experiments/ui-tars-smoke/.venv/lib/python3.14/site-packages/mlx_vlm/server/generation.pyr4r4:sMr<r4cj\PPR4pV'd \V4#R#)MLX_VLM_DRAFT_BLOCK_SIZENosenvirongetint)draft_block_size_strs r=_get_draft_block_size_from_envrF>s)::>>*DE(<3# $F$Fr<ctVFpVFpVPV4K K R# \dK3i;iN)put Exception)queuesitemsqueueitems&* r=_notify_queuesrOCs:D  $  s ' 77c\\\PPR\44#)PREFILL_STEP_SIZE)rDrArBrCrr5r<r=get_prefill_step_sizerRLs rzz~~13LM NNr<c\\\PPR\44#)MLX_VLM_MAX_TOKENS)rDrArBrCrr5r<r=get_server_max_tokensrUPs rzz~~24FG HHr<c\PPRR4pVR8Xd\#\ V4pT^8:dR#T# \ d%\ PRT\4\u#i;i)MLX_VLM_TOKEN_QUEUE_TIMEOUTzCKb**+ $!| N + J  ' +* +s A,A43A4c\PPR\\44p\ R\ V44R, # \d\R, u#i;i)MLX_VLM_SPEC_BATCH_COALESCE_MSg@@)rArBrCstr%DEFAULT_SPECULATIVE_BATCH_COALESCE_MSmaxrZr[raws r= get_speculative_batch_coalesce_srifsT **..(#.S*T C>3c #f,, >4v==>sA A&%A&c ^V^8dQhR\R\R\R\\,/#) prompt_kwargs batch_sizesequence_lengthreturn)dictrDr rd)formats"r= __annotate__rrps0   (+ >A  #Y r<c<T;'g/P4UUu.uFpwr4\V\P4'gK'VP^8gK:VP ^,V8XgKTVP ^,V8XgKnVNKr upp#uuppi)rk)rL isinstancemxarrayndimshape)rlrmrnkvs&$$ r=_sequence_aligned_prefill_keysr{ps #((b//1 1DA a "  FFaK  GGAJ* $ GGAJ/ ) 1  s$BBB4BBc^V^8dQhR\R\\,R\R\/#rkrlkeysnrorpr rdrD)rqs"r=rrrr}s,T#Y34r<c~V'gV#\V4pVF!pWC9gK W4,RRV1R3,W4&K# V#NNNN.rprlr~routkeys&&& r=_slice_prefill_kwargsr}sC  } C :x2A2s +CH Jr<c^V^8dQhR\R\\,R\R\/#r}r)rqs"r=rrrrs,DI#$r<c~V'gV#\V4pVF!pWC9gK W4,RVR1R3,W4&K# V#rrrs&&& r=_drop_prefill_kwargsrsA  } C :x12s +CH Jr<cV^8dQhR\PR\PR\R\R\\,R\ \ \P3,/#)rk input_ids inputs_embedsrlspeculative_kwargsprefill_step_sizero)rurvrprrDrobject)rqs"r=rrrrs`3$3$xx3$883$  3$  3$ }3$ 6288 3$r<c TpTp \T;'g/4p \V VP^,VP^,R7p VEe!V^8EdV P^,V8EdV P^,^8d\WiP^,^, 4p \ WV 4p \ P !V4;_uu_4V!VRRV 13,3RVRV RRV 13,RV /V BRRR4\ P!VUu.uFqPNK up4VRV R13,pV RV R13,p \WV 4p \ P!4K/V CVCpWR&\ P !V4;_uu_4V!V3RV/VBpRRR4VV3# +'giL;iuupi +'giXT3#;i)zHPrefill target cache in chunks, capturing speculative state only at end.)rmrnNrcacher n_to_process) rpr{rxminrrustreamevalstater clear_cache)lmrr prompt_cacherlrrr"remaining_input_idsremaining_embedsremaining_kwargs sequence_keysr chunk_kwargsc final_kwargsrs&&&&&&$$ r= _run_chunked_speculative_prefillrs$$M//R02??1%%++A.M %  !  " "1 %(9 9$$Q'!+02H2H2Ka2OPL0 L,--'=L=(89&#31m|m3C"D".  # . GGl3lWWl3 4"5a6F"G /<=0@A 3    NN =&=*<=L$4! $ % %$ILILI & # ##).-4 & % # ##s&F/G G/ F?  G cHV^8dQhR\R\R\R\/#)rkseedrow_idpositionrorD)rqs"r=rrrrs(cSSr<cP\V4R, R,pV\V4^,R,,R,pV\V4^,R,, R,pW3^, ,pVR,R,pW3^, ,p\VR,4#)lyn<llkJ l5.ei-5r)rrrxs&&& r=_position_seedrs TZ :-A c&kAo + +z9A s8}q J. /:=AbLA Z:%AbLA q:~ r<cV^8dQhR\R\\,R\\,R\P/#)rkrrow_ids positionsro)rDr rurv)rqs"r=rrrrs4tCyT#Y288r<c \P!\W4UUu.uF.wr4\PP \ WV44NK0 upp4#uuppirH)rustackziprandomrr)rrrrowposs&&& r=_position_keysrsP 88 3 3 IIMM.C8 93   s4A c~a]tRt^toRtV3RlRltV3RlRltV3RlRltV3RlR ltV3R lR lt R t Vt R #)_PositionedTargetSamplerzCServer sampler with stateless target draws for ragged verification.c<<V^8dQhRS[RS[RS[S[,/#)rk temperaturetop_pr)rZrrD)rq __classdict__s"r=rr%_PositionedTargetSampler.__annotate__s+@@u@U@(3-@r<c\V4Vn\V4VnVf\VnR#\ V4VnR#rH)rZrrrrDr)selfrrrs&$$$r=__init__!_PositionedTargetSampler.__init__s/ -5\ $(LL c$i r<cN<V^8dQhRS[PRS[P/#)rklogprobsrorurv)rqrs"r=rrrs'HHHbhhHr<cVP^8d2VPR8d!\WPVP4#\PP V^VP, ,4#)r?)rr&rrur categorical)rrs&&r=__call__!_PositionedTargetSampler.__call__sT ::>djj3.!(JJ8H8HI Iyy$$XT5E5E1E%FGGr<cz<V^8dQhRS[PRS[S[,RS[S[,RS[P/#)rkrrrro)rurvr rD)rqrs"r=rrrsI I I(( Ic I 9 I  Ir<cVP^,\V48wg\V4\V48wd \R4h\VPW#4pVP ^8d:VP R8d)\ P!VPRR7!W4#\ P!VPRR7!W4#)rz5row_ids and positions must match logprobs batch size.r)in_axes)rr) rxlenr[rrrruvmap_sample_top_p_one _sample_one)rrrrr~s&&$$ r= sample_target&_PositionedTargetSampler.sample_targets >>! G ,G I0NTU Udii< ::>djj3.774116B8R Rwwt''8HHr<ch<V^8dQhRS[PRS[PRS[P/#rkrrror)rqrs"r=rrrs3QQBHHQ288QQr<ct\PPV^VP, ,VR7#)r)rurrr)rrrs&&&r=r$_PositionedTargetSampler._sample_ones,yy$$XT5E5E1E%FC$PPr<ch<V^8dQhRS[PRS[PRS[P/#rr)rqrs"r=rrrs3 V V"(( V Vbhh Vr<cVP\P8Xd VP\P4p\P !WP , RR7p\P!VRR7p\P!W4RR7p\P!VRR7p\P!V^VP, 8V\P!V44p\PP\P!V4VR7p\P!WHR,RR7^,#)r)axisr).N)dtyperubfloat16astypefloat32softmaxrargsorttake_along_axiscumsumwherer zeros_likerrlog) rrrprobssorted_indices sorted_probscumulative_probs top_probs sampled_poss &&& r=r*_PositionedTargetSampler._sample_top_p_ones >>R[[ (rzz2H 8&6&66R@E3))%bI 99\;HH q4::~ -  MM, ' ii++BFF9,=3+G !!.i2HrRSTUUr<)rrrN) r6r7r8r9r:rrrrrr;__classdictcell__rs@r=rrsDM@@ HH I IQQ V Vr<rrrc V^8dQhR\PR\\P.\P3,R\\\ ,,R\\\ ,,/#)rklogitssamplerrr)rurvr rr rD)rqs"r=rrrrsZ   HH  rxxj"((* + d3i  S " r<cVR,pV\P!VRRR7, p\VRR4p\V4'dVeVe V!WBVR7#V!V4#)rT)rkeepdimsrNrrr)rrr)ru logsumexpgetattrcallable)rrrrrrs&&$$ r=_sample_last_tokenrs`H F V"tDDHG_d;M 7#69;PX)LL 8 r<cz\PPR4pVf\#VP 4R9#)MLX_VLM_ENABLE_THINKING)1trueyeson)rArBrCDEFAULT_ENABLE_THINKINGlowerrgs r=get_server_enable_thinkingrs2 **..2 3C {&& 99;4 44r<cb\PPR4pVfR#\V4#)MLX_VLM_THINKING_BUDGETNr@rgs r=get_server_thinking_budgetrs( **..2 3C;4,CH,r<c@\PPR4#)MLX_VLM_THINKING_START_TOKENrArBrCr5r<r=get_server_thinking_start_tokenr #s ::>>8 99r<c@\PPR4#)MLX_VLM_THINKING_END_TOKENrr5r<r=get_server_thinking_end_tokenr 's ::>>6 77r<c$V^8dQhR\/#rkmodelrd)rqs"r=rrrr+sr<c\\PPR^44pV^8XdR#RV9d\ RV R24R#V#)KV_BITSNqatModel z7 is quantization aware, KV cache will not be quantized.)rZrArBrCprint)rkv_bitss& r=get_quantized_kv_bitsr+sFBJJNN9a01G!| ~ ugTUV Nr<c\\\PPR\44#) KV_GROUP_SIZE)rDrArBrCrr5r<r=get_kv_group_sizer5s rzz~~o/DE FFr<cJ\PPR\4#)KV_QUANT_SCHEME)rArBrCrr5r<r=get_kv_quant_schemer9s ::>>+-D EEr<c$V^8dQhR\/#rr)rqs"r=rrrr=s3r<c\\PPR^44pV^8XdR#\ V4e\ RV R24R#V#) MAX_KV_SIZENrz. uses QuantizedKVCache, can't set max KV size.)rDrArBrCrr)r max_kv_tokenss& r=get_max_kv_sizer"=sL }a89MU#/ ugKLM r<cl\\PPR^44pT;'gR#)r N)rDrArBrC)r!s r=get_configured_context_limitr$Gs) }a89M  D r<c0V^8dQhR\R\/#rk raw_inputsro)rprD)rqs"r=rrrrLsLLTLcLr<cfVR,p\VR4'd VP#\V4#)rsize)hasattrr)r)r'rs& r=_count_prompt_tokensr+Ls+;'I$Y779>>KS^Kr<c0V^8dQhR\R\/#)rk prompt_tokens max_tokensr)rqs"r=rrrrQs   C  S  r<c \4pT\^\T;'g^44,pVe!W28d\RV RV RV RV R2 4hR#R#)rNzRequest needs z context tokens (z prompt + z% max generation), but MAX_KV_SIZE is .)r$rfrDr4)r-r. context_limitrequested_tokenss&& r= _check_configured_context_budgetr3Qss02M$s1c*//.B'CC %5%E  !j 5""/ 3  &F r<c\\\PPR\44#)QUANTIZED_KV_START)rDrArBrCrr5r<r=get_quantized_kv_startr6]s rzz~~24NO PPr<c\\PPR^44p\ ^\ V^44#)zMax per-token top_logprobs honored by the server (0 = disabled). Set via TOP_LOGPROBS_K env var. OpenAI caps this at 20. When 0, requests with top_logprobs>0 still succeed but the top_logprobs list stays empty. TOP_LOGPROBS_K)rDrArBrCrfr)rys r=get_top_logprobs_kr9as1 BJJNN+Q /0A q#a* r<chV^8dQhR\\,R\R\\,/#rk token_timeswindowror rZrDr)rqs"r=rrrrks) ' 'DK ' '% 'r<c\V4^8dR#W)Rp\V4^8dR#VR,V^,, pV^8:dR#\V4^, V, #rkNrrr<r=subsetelapseds&& r=_token_window_raterEks[ ;!  "F 6{QRj6!9$G!| K!Ow &&r<chV^8dQhR\\,R\R\\,/#r;r>)rqs"r=rrrrws* ' '$u+ 's 'xPU 'r<c\V4^8dR#VRVp\V4^8dR#VR,V^,, pV^8:dR#\V4^, V, #r@rArBs&& r=_token_window_rate_firstrHws[ ;! & !F 6{QRj6!9$G!| K!Ow &&r<ca]tRtRtoRt]3V3RlRlltRtV3RlRltV3RlR lt V3R lR lt V3R lR lt Rt Vt R#)ServerMetricsStoreiz=Rolling request metrics and lifetime counters for the server.c <V^8dQhRS[/#)rk history_limitr)rqrs"r=rrServerMetricsStore.__annotate__scr<c2WnVP4R#rH)rLreset)rrLs&&r=rServerMetricsStore.__init__s* r<cD\P!4Vn\4VnRVn\ VP R7Vn^Vn^Vn ^Vn ^Vn ^Vn ^Vn ^Vn^VnRVnRVnRVnRVnR#)N)maxlenrc)time started_atr _lock_latestrrL_recent_requests_started_requests_completed_requests_failed_streaming_requests _in_flight_prompt_tokens_total_completion_tokens_total_generated_tokens_total_request_time_total_s_decode_time_total_s_last_request_at _last_errorrs&r=rOServerMetricsStore.resets))+V '+ D$6$67 !"#$ !#$ $%!()%'($%("$'!15+/r<c,<V^8dQhRS[RS[RS[/#)rkendpointrrrdbool)rqrs"r=rrrMs"...S.$.r<c??VP;_uu_4V;P^, unV;P^, unV'dV;P^, unRRR4R# +'giR#;i)rN)rUrXr\r[)rrgrrs&$$$r= begin_request ServerMetricsStore.begin_requestsO e ZZZ  " "a ' " OOq O((A-( ZZZs AA55 B c <V^8dQhRS[/#)rkenveloper)rqrs"r=rrrMs W Wt Wr<c \V4pVP;_uu_4V;P^, un\^VP^, 4VnW nVP PV4VPR4Vn T;P\VPR4;'g^4, un T;P\VPR4;'g^4, un T;P\VPR4;'g^4, un T;P\VPR4;'gR4, unT;P \VPR4;'gR4, unRRR4R# +'giR#;i) rtimestamp_unixr-completion_tokensgenerated_tokensrequest_elapsed_srcdecode_elapsed_sN)rprUrYrfr\rVrWappendrCrbr]rDr^r_r`rZra)rrnpayloads&& r=record_success!ServerMetricsStore.record_successs<x. ZZZ  $ $ ) $!!T__q%89DO"L LL   ($+KK0@$AD !  % %W[[-I-N-NQ)O O %  ) )S=P1Q1V1VUV-W W )  ( (C (>(*<%t'<'<($*B*B)4+D+D-t/L/L,d.J.J(*:'):&(8% */ZXZZsAGG .EG G G! )r^rar_r\rcrbrVrUr]rWr`rYrZrXr[rLrTN)r6r7r8r9r:METRICS_HISTORY_LIMITrrOrkrwr|rr;rrs@r=rJrJsDG,A0$.. W W  ++r<rJchV^8dQhR\R\\,R\\,/#)rkr- prompt_tpsro)rDrrZ)rqs"r=rrrrs,&&&$,UO& e_&r<c:V^8:g VeV^8:dR#W, #rNr5)r-rs&&r=_prompt_eval_time_from_tpsrs#Z/:?  %%r<c V^8dQhR\\,R\R\\,R\\,/#)rkr<rrgeneration_tpsror>)rqs"r=rrrrs<  e  UO e_ r<c\V4^8d!VR,V^,, pV^8dV#V^8dVeV^8d W, #R#r@rA)r<rrrrDs&&& r=_decode_elapsed_from_metricsrsP  ;1b/KN2 Q;N! :~PQ?Q00 r<r<rrpeak_memory_gb finish_reason image_count audio_countstructured_outputthinking_enabled tool_parser tool_callsc*V^8dQhR\R\R\R\R\R\R\R\R \R \\ \,,R \\,R \\,R \\,R\\,R\R\R\R\R\\,R\R\ /#)rkrgrrbackendr-rqrrrsrequest_started_sr<rrrrrrrrrrro)rdrirDrZrr rp)rqs"r=rrrrsDDD D  D  D  DDDDD$u+&DDUODUODC=D !D"#D$%D&'D(#)D*+D, -Dr<cZT ;'g.p V 'd\RV ^,V, 4MRp\WV 4pRpV e V ^8dT pMVeV^8dV^8d VV, p\WJ4pV^8d WW, MRp/R\P!4bRVbRVbR\ V4bRVbR\ V4bR\ V4bR \ V4bR \^\ V4\ V4, 4bR \ V4\ V4,bR VbR V bRVbRVbRVbRVbRVbR\ V ^ 4R\ V ^@4R\V ^ 4R\V ^@4RV RV R\ V4R\ V4R\ V4R\ V4RVR\ V4R\PRJ/ C#) rcNrprgrrrr-rqrrreasoning_tokens total_tokensprompt_eval_time_s prefill_tok_sttft_srtrs request_tok_s decode_tok_ssliding_decode_tok_s_first_32sliding_decode_tok_s_first_64sliding_decode_tok_s_last_32sliding_decode_tok_s_last_64rrrrrrrr apc_enabled) rfrrrSrirDrHrEr1 apc_manager)rgrrrr-rqrrrsrr<rrrrrrrrrrrrtrrrs$$$$$$$$$$$$$$$$$$$$ r=_build_metrics_envelopers<.##K=HSk!n'88 9dF3~L!nq&8%  %*:Q*>CSVWCW'*:: 3MN1BQ1F-C $))+ H    $v,   7   ]+   S!23  C 01  C3'7#83?P;Q#QR  M*S1B-CC  0    &  ,  .  ! "  # $ ()A+r)R')A+r)R&(:;(K&(:;(K.s;'s;'T"34D!12{d:&w**$6= r<cFV^8dQhR\R\\,/#)rk model_path adapter_pathrdr)rqs"r=rrrrJs"RRSR Rr<c\RV 24V'd\RV 24\PPRR4P 4R8Hp\ WVR7wr4VP p\R4W4V3# \d<p\RT R T 24\P!4\R R T 2R 7hR p?ii;i)zV Loads model, processor, and config based on paths. Handles potential loading errors. zLoading model from: zLoading adapter from: MLX_TRUST_REMOTE_CODEfalser)trust_remote_codez(Model and processor loaded successfully.zError loading model z: izFailed to load model: ) status_codedetailN) rrArBrCrr/configrJ traceback print_excr)rrrr processorres&& r=load_model_resourcesrJs R $ZL12  *<.9 : JJNN2G < B B D N  8I  89'' R $ZL1#676LQC4PQQRsA>B C 6CCca]tRtRtoRt]t]t] t ^t Rt Rt RtRt]tRt]tRt]tRt]tRtRtRtRtRtV3RlRltV3RlR ltV3R lt R t!Vt"R#) GenerationArgumentsihz#Arguments for a generation request.rcNFc <V^8dQhRS[/#rr)rqrs"r=rr GenerationArguments.__annotate__s$$D$r<c  RVPRVPRVPRVPRVPRVP /pVP eVP VR&VPeVPVR&VPeVPVR &VPeVPVR &VPeVPVR &VPeVPVR &VPeVPVR &VPeVPVR&VPeVPVR&VPeVPVR&VP eVP VR&VP"eVP"VR&VP$eVP$VR&V#)z8Convert to kwargs dict for generate()/stream_generate().r.rrtop_kmin_penable_thinkingrrepetition_penaltyrepetition_context_sizepresence_penaltypresence_context_sizefrequency_penaltyfrequency_context_size logit_biasthinking_budgetthinking_start_tokenthinking_end_tokenlogits_processors apc_tenant)r.rrrrrrrrrrrrrrrrr tenant_idrkws& r=to_generate_kwargs&GenerationArguments.to_generate_kwargss $// 4++ TZZ TZZ TZZ t33   99 BvJ  " " .'+'>'>B# $  ' ' 3,0,H,HB( )  ,%)%:%:B! "  % % 1*.*D*DB& '  ! ! -&*&<&'>B# $  ! ! -&*&<&> %#~~B|  r<c <V^8dQhRS[/#rr)rqrs"r=rrrs  D r<c RVP/pVPeVPVR&VPeVPVR&VPeVPVR&V#)z,Convert to kwargs for apply_chat_template().rrrr)rrrrrs& r=to_template_kwargs&GenerationArguments.to_template_kwargssn!5!5 6    +$($8$8B !  $ $ 0)-)B)BB% &  " " .'+'>'>B# $ r<cB<V^8dQh/S[;R&S[;R&S[;R&S[;R&S[;R&S[S[,;R&S[;R&S[S[,;R&S[S[,;R &S[S[,;R &S[S[,;R &S[S[,;R &S[S[,;R &S[S[,;R&S[;R&S[S[,;R&S[S[,;R&S[S[,;R&S[S[S[S[PS[P.S[P3,,,;R&S[S[,;R&#)rkr.rrrrrrrrrrrrrrrrrrr) rDrZrrirprdr r rurv)rqrs"r=rrrhs](  ,    N  3-!.&c]Luo,$C=J - %SMK!"%#$3%&c])'(#3-.)*! ,+, Xrxx.BBHH.L%M NOV-4}#5r<r5)#r6r7r8r9r:rr.rrrrrrrrrrrrrrrrrrrrrrrrr__annotate_func__r;rrs@r=rrhs-(J,K EEEDH*.-L(,+J)-,K!%J3O%)O*.(,RV $I$$L  Er<rc0a]tRtRtoRtV3RltRtVtR#)GenerationContextiz*Context returned when a request is queued.c2<V^8dQh/S[;R&S[;R&#)rkuidr-r)rqrs"r=rrGenerationContext.__annotate__s H   r<r5N)r6r7r8r9r:rr;rrs@r=rrs4r<rcva]tRtRtoRt]!]R7tRt^t Rt Rt V3RlRlt V3RlR lt V3R ltR tVtR#) GenerationMetricsiz  $!$T%7%7]9K!LD  %r<c<V^8dQh/S[S[,;R&S[;R&S[;R&S[S[,;R&S[S[,;R&#)rkr<rrrrr>)rqrs"r=rrrsRe:     &UO*r<)rrrr)r6r7r8r9r:rrr<rrrrrrrr;rrs@r=rrsCF$T:KKM"&J&*N"" M Mr<rcDa]tRtRtoRtRtRtRt^t^t V3Rlt Rt Vt R#)StreamingTokenizA single token response during streaming generation. Diffusion models stream block-by-block: one StreamingToken per denoised block, with ``token_count`` carrying the number of tokens in the block. rcNc<V^8dQh/S[;R&S[;R&S[;R&S[S[,;R&S[;R&S[S[,;R&S[S[S[S[S[3,,,;R&S[;R&S[;R &#) rktexttokenrrrr top_logprobsr token_count)rdrDrZrr r)rqrs"r=rrStreamingToken.__annotate__s I JOC= &4c5j 123: !r<r5) r6r7r8r9r:rrrrrrr;rrs@r=rrs,K"&J6:LMK!r<rcV^8dQhRR/#)rkroz%Generator[StreamingToken, None, None]r5)rqs"r=rrrrs""(O"r<c #z".p^p^pVEF)pVP'dKVP'dVPVP4VPe\ VP4pVP 'gVP 'gKVP 'g V'dm\VPV, ^4pVPp\RPV4VRVP VPVPVR7x.pVP 'gEK)R# R#5i)aTGroup diffusion engine results into block-by-block streaming tokens. The diffusion engine emits a canvas's tokens right after that canvas finishes denoising, followed by a block-boundary marker. Each completed block becomes one StreamingToken; the final token carries the finish reason (plus any text flushed by detokenizer finalization). NrXrrrrrrr) is_draftrrurrDdiffusion_block_completerrfgeneration_tokensrjoinrr)results block_text last_tokenemitted_tokensrrs& r=_diffusion_block_chunksrsJJN ???  ;;;   fkk * << #V\\*J...v7K7K7K     :f66GKK#55N WWZ( $22"..!,,' J     /s*2D;AD;D;D;+D;3A;D;4D;cBa]tRtRtoRtRtRtRtRtRt Rt Vt R #) _TokenIteratorizCloseable iterator over queued tokens for one generation request. close() cancels unfinished requests and is safe while another thread is blocked in __next__ waiting for the next token. cpWnW nW0nW@nRVnRVn\ 4VnR#)FN)_rqueue_uid _cancel_fn_queue_timeout_ended_closedr rU)rrqueuer cancel_fn queue_timeouts&&&&&r=r_TokenIterator.__init__s.  #+  V r<cV#rHr5rds&r=__iter___TokenIterator.__iter__#s r<cVP'd\hVPPVPR7pTfRTn\h\T\4'd RTnTh\TR R4'dRTnT# \ dGpTP 4TPfRMRTPR R2p\RT R24ThRp?ii;i) r_Nzwithout a timeoutzfor gszTimed out waiting zq for the next generated token. Increase MLX_VLM_TOKEN_QUEUE_TIMEOUT for long prefills, or reduce the prompt size.Tr) r StopIterationrrCr QueueEmptyclose RuntimeErrorrtrJr)rrNexclabels& r=__next___TokenIterator.__next__&s ;;;  <<##D,?,?#@D <DK  dI & &DKJ 4$ / /DK -  JJL&&.$D//2!4  $'77    s&B CACCc VP;_uu_4VP'd RRR4R#RVnRRR4VP'gVPVP4R#R# +'giLA;iNT)rUrrrrrds&r=r,_TokenIterator.closeCsT ZZZ|||Z DL{{{ OODII & ZsA3A33 B cLVP4R# \dR#i;irH)r,rJrds&r=__del___TokenIterator.__del__Ks"  JJL   s  ##)rrrrUrrrN) r6r7r8r9r:rr$r0r,r6r;rrs@r=rrs( :'r<rca]tRtRtoRtRRR]]]^R3V3RlRlltRt R2V3RlRllt R t V3R lR lt R t R3V3R lRlltR4V3RlRlltV3RlRltR2V3RlRlltV3RlRltV3RlRltV3RlRltV3RlRltR2V3RlRlltRR R!R"/V3R#lR$lltR%tV3R&lR'ltR(tR)tR*tR2R+ltV3R,lR-ltR.tR3V3R/lR0llt R1t!Vt"R#)5ResponseGeneratoriSu@ Continuous batching for concurrent requests via a single GPU thread. A dedicated thread owns all GPU work (BatchGenerator). FastAPI async handlers submit requests to a queue and read tokens back from per-request queues. Multiple requests are batched together for higher throughput — same pattern as mlx-lm's server. NcJ<V^8dQhRS[RS[S[,RS[R,/#)rkrrrz_apc.APCManagerr)rqrs"r=rrResponseGenerator.__annotate__]s3"""sm"/0"r<c WnW nRVnRVnRVn\ 4VnW0nRVnW@n WPn W`n Wpn Wn WnRVn\!4VnRVn\'4VnRVn\ 4Vn\/4Vn\3VP4RR7VnVP6P94R#)NFT)targetdaemon)rrrrrset stop_tokens vision_cache draft_modelr kv_group_sizekv_quant_schemequantized_kv_starttop_logprobs_kr tokenizerrrequests_stopr_ready _load_error _cancelledr _cancel_lockr _run_threadstart) rrrrArrCrDrErFrs &&&&&&&&&&r=rResponseGenerator.__init__]s%(  5( *."4,&$w  g 04"u FTYYt<  r<cRVnVPPR4VPP RR7R#)TNr2r')rIrHrIrOrrds&r= stop_and_joinResponseGenerator.stop_and_joins0  $ #&r<c0<V^8dQhRS[S[,/#)rkr_)rrZ)rqrs"r=rrr;s777r<cVPPV4'g \R4hVPe VPhVPVP VP 3#)z6Timed out waiting for generation thread to load model.)rJwaitr-rKrrr)rr_s&&r=wait_until_ready"ResponseGenerator.wait_until_readysT{{((WX X    '"" "zz4>>4;;66r<cVP;_uu_4VPPV4RRR4R# +'giR#;irH)rMrLadd)rrs&&r=_cancelResponseGenerator._cancels/     OO   $   s ; A c <V^8dQhRS[/#r)r?)rqrs"r=rrr;scr<cVP;_uu_4VP\4uqnVuuRRR4# +'giR#;irH)rMrLr?)rpendings& r=_drain_cancellations&ResponseGenerator._drain_cancellationss2    '+ $G_   s = A c \VPVP4wrp\4p\ VR4'df\ VP \4'dVPVP 4M)VP eVPVP 4Rp\PPR4p\PPR4pV'dy^RI HpHp \!RT;'gR RV 24V!WvR7wrZVf\!R V : R 24MW8wd\!R V : R V : R V: R 24T pV !WV4\!R4WnW nW0nW@nWPnW`n\ VR4'dVP0VnR#TVnR# \"dp \!RT 24RpRpRp ? LtRp ? ii;i) eos_token_idNMLX_VLM_DRAFT_KINDMLX_VLM_DRAFT_MODEL) load_draftervalidate_drafter_compatibilityzLoading speculative drafter (autoz): )kindu! → auto-detected --draft-kind=r0u$ → drafter requires --draft-kind=z; using z instead of u/Drafter ready — speculative decoding enabled.zfSpeculative drafter is incompatible with the target model; falling back to autoregressive generation. rG)rrrr?r*rtrdrupdater[rArBrCspeculative.draftersrgrhrr[rrrr@rB draft_kindrG) rrrrr@rBrmdraft_model_pathrgrh resolved_kindrs & r=_initialize_model#ResponseGenerator._initialize_models#7 OOT..$  &e 6> * *&--t44""6#6#67$$0 3 34 ZZ^^$89 ::>>*?@   / 0D0Df/ES#$& *66F)X &K!9-9J!LM,:=:KL*-\*qJ'J I.u:NGH " &&$#*9k#B#BI   HQ ! "BBCF# !  "s F77 GGGc |<V^8dQhRS[RS[S[,RS[S[,RS[S[,RS[S[R3,/#)rkpromptimagesaudioargsror)rdrr rrr)rqrs"r=rrr;sW" " " " ~ " *+ "  "22 3 " r<cTVP4T;'g\\4R7pVPeVPe \ R4hVPeVP e \ R4h\4pVPWV4p\V4p\WtP4VPPWVWtV34VP4p\V\ 4'dVhV\#WXP$VP&\)443#)r.zFStructured response_format is not supported with speculative decoding.zIthinking_budget is not supported with speculative decoding in the server.)rXrrUrBrr[rr_cpu_preprocessr+r3r.rHrIrCrtrJrrr\r`) rrsrtrurvrr'r-ctxs &&&&& r=generateResponseGenerator.generates  NN*6K6MN    'D,B,B,NX     'D,@,@,L[ ))&%@ ,Z8 (H 6}FKLjjl c9 % %IN GGT\\+B+D   r<c <V^8dQhRS[/#rr)rqrs"r=rrr;s  $ r<c  VPPPR9d\VPRR4RJMRp\VPPRR4p\ VPVVVVVR7#)z9CPU-only: tokenize text, load/resize images. Thread-safe. chat_templateNTimage_token_index)rtrupromptsradd_special_tokens)gemma3gemma3ngemma4gemma4_unified)rr model_typerrr0)rrsrtrurrs&&&& r=ry!ResponseGenerator._cpu_preprocessszz  ++@A DNNOT :d B  $DJJ$5$57JDQ NN/1   r<c6<V^8dQhRS[RS[S[,/#rkrvro)rrr )rqrs"r=rrr;s!  "5 (8:L r<cVP^8XdR#\VPVPVPR7#)rN)rrr)rrrr)rrvs&&r= _make_samplerResponseGenerator._make_samplers7   q '((**  r<c <V^8dQhRS[RS[S[P,RS[S[S[PS[P.S[P3,,/#rkrvrro)rrrurvr r )rqrs"r=rrr;sN'4 >(; >c3h >r<cVPpVP;'g\pVP;'g\pVP VRR7R,pVP VRR7R,pWV3#)F)rr)rGrrrrencode)rrvrGrrthinking_start_token_idthinking_end_token_ids&& r=_thinking_token_ids%ResponseGenerator._thinking_token_ids&sNN #88XXU+VVJ 6{Q"););B? CCc <V^8dQhRS[RS[S[S[PS[P.S[P3,,RS[S[S[PS[P.S[P3,,/#)rkrvrro)rr r rurv)rqrs"r=rrr;Csi  ! 288RXX"6"@AB  h"((+RXX56 7  r<c VP;'g\pVP;'g\pVUu.uFp\ VVP VVRR7NK up#uupi)T)rrGrrr)rrrrr+rG)rrvrrrrs&&& r=r6ResponseGenerator._wrap_processors_until_thinking_doneCsq $88XX.append_items'|:::g"&K NN4 r<r')rH get_nowaitrCr+rSsleep)rrrrrr`rsf$$$ @@r=_collect_pending_requests+ResponseGenerator._collect_pending_requestss  ! DMM4467DMM--l-CD zA~ JJz " DMM4467 ##     ## s-B1 B1"B1 C1 B?>B? CCc VP4TPP 4\TP4pTeTPT4R#TPe$TPR8wdTP4R#\P!\P !44pRp/pTP"'Eg\%T4pT'g*TPeTPR8Xd \'4MRpTP)TTR7wrT 'dEM/TP+4p T 'dQTeMT FFp Y9gK TP-T 4TP/T 4p T R,P1R4KH T'd1Te-T'g%TP2'gTP54RpTEFewrpppTf\7TPP8TP:TP<TP?T4TP@TPBTPDTPF\%TPH4TPH'd TPJM^TTPLTPTP\O4TPP^8HR7pTPSTT4wpp\%TPUR4RJ4p\WTR R4'dTPXTR &T'd`\Z;QJd&R TP\4F 'gK R M R M!R TP\44'dTP_YE4TPaTT4pTPcTPe^4Pg4.TPhT.TPkTT4.T.R7wp T P1\mYR74RT R\oTPp\sTP:44RT'dTMRRRR^/Y[&EKh T'dTfEKTPuYE4EKTe1\\WTRR44'dTP54R#R#R# \dOpYnTPP 4\ RT 24\ P!4Rp?R#Rp?ii;i \dEKi;i \dpT P1T4Rp?EKURp?ii;i \dp\vPyR4\{TP}44FFp T R,P1T4T R,P1R4K6 \dKDi;i TP4Rp\P!4\P!4Rp?EK6Rp?ii;i)z?Single GPU thread: owns BatchGenerator, runs tight next() loop.z*Error loading model in generation thread: Nmtprcrrr)r@rrrCrDrEcompute_logprobsrFrrrBrmdraft_block_sizegreedy_samplingrr _apc_tenantc3"TF5pV^,;'dV^,PR4RJ'*xK7 R#5i)rN)rC).0r)s& r= )ResponseGenerator._run..s;*!>AqTKKadhh&?t&KLL!>s?(?TF)r.rlrthinking_budget_criteriarr-streamerrrrzError in generation threadr,)DrprJrKrJr?rrrr$r_run_diffusionrBrm_run_speculativerudefault_streamdefault_devicerIririrraremoverrIhas_workr,rlanguage_modelrr@rrrCrDrErrFrrFrrrCrranyunprocessed_prompts_flushrinsertsqueezerr.rrr,rGr-_stepr\ exceptionrvaluesclearrgccollectr)rrdiffusion_familyr" batch_genr active_batchr new_itemsr cancelledrinforr'r-rvrtrr has_embedsrs& r=rNResponseGenerator._runs    " " $ 7tzzB  '    0 1     'DOOu,D  ! ! # --b.?.?.AB ***{  $F| ) ,,8 OOu4 56 *.)G)G')*H*& !557 !6(=%,,S1#)::c?D% $X 2 24 8 )!6v$---!)$( GPCF tV ($2 JJ55 NN(,(8(8$($6$6t$<$(LL*.*<*<,0,@,@/3/F/F-1$---@BF---4+>+>UV#4(,(8(8(,(8(8'+-K-M,0,<,<,A!% *-1OOJ,O)Iz!%jnn_&ET&Q!RJt[$7748NN =1 "cc*!*!>!>*ccc*!*!>!>*'' I6!373V3V )40"+!1!1&..q188:;'++5, $ < qcB C    !   p$-% $%|%! 1  !*   !=> 1DX**1-X**40$ 2      sR AT&'T&? T&"T&3S( T& T&#T&5ET& T&T&8$T&T&/A%S;A&T&;T&T& S%AS  S%( S83T&7S88T&; T#TT&T##T&& W111W,#0VW, V$ W,#V$$AW,,W1c <V^8dQhRS[/#)rkfamilyr)rqrs"r=rrr;Ns''S'r<c $VR8Xd VPM VPp^p\4pVP'gVP RR7wrVV'dR#W@P 4,pVF^wrxrp V^, pTp VP \WR74V!WWV4VP R4\P!4K` KR# \dXp \PR4TP T 4TP R4Rp ? Le \dRp ? Lvi;iRp ? ii;i \dE\PR4\P!4\P!4EKbi;i)a%GPU thread loop for diffusion models. Diffusion generation runs one request at a time (batch size 1). Output streams back block-by-block: one queue item per denoised block, plus a final item carrying the finish reason. Non-streaming endpoints aggregate the same items into a single response. blockF)rrNzError in diffusion generationz$Error in diffusion generation thread)_generate_diffusion_generate_masked_diffusionr?rIrrarIrrJr\rrurrr)rrgenerate_request uid_counterrrrrr'r-rv_imagesrrs&& r=r ResponseGenerator._run_diffusionNsU   $ $00    *** )-)G)Gu)G)U& 6688 HQDF W1$K%CJJ0SVW !(j R 4(NN$IR %!(()HI!"JJqM"JJt,,(! ! !   !GH   sgEAE#C>E D=&D8<"D#E# D5.D8/E4D55D88D==EA FFcDVPe \R4hVPe*\PPVP4VP R4pVeVP ^8Xd VR,pVPp\VR4'd0VPPVPP4\VPVPTTVP R4VP R4VP VP"\%\'VRR4;'g.4VP R4R7 p\)VP\*.4;_uu_4\-V4FWp VP/V 4V P0'dM3WPP34,pW9gKFVP5V4M RRR4VP74R# +'giL";i TP74i;i) NBStructured response_format is not supported with diffusion models.rstopping_criteriarrall_special_idsmm_token_type_ids)r.rskip_special_token_idsr)rr[rrurrCrwrGr*rrOrrdr%rrr.rr?rr#r"rrIrradiscardr,) rrrr'rvrrrGrrs &&&&&& r=r%ResponseGenerator._generate_diffusionws  ! ! -T  99 IINN499 %NN;/  Y^^q%8!$INN 91 2 2  ' ' - -dkk.F.F G+ JJ NN   NN> * NN+ ,((#& #4d;AAr$)nn-@A   TZZ*;)<==4W=EJJu%***!:!:!<> MMO>= MMOs+#H 9A G: G: H : H H Hc aaaaaaaaaaVPe \R4hVPe*\PPVP4VP R4oSeSP ^8Xd SR,oSPo\SR4'd0SPPSPP4SPp/oRo^oRVVVVVV3RlloVVVV3Rlp/pRFwr\WjR4p V fKWV &K \SP\ .4;_uu_4SPP"P$!S3RVP&R \VR R4;'g^ R \VR R4;'g^ R VP(RVP*eVP*R8dRM VP*RRRRRSRRRSRV/ VBp \P,!V 4RRR4X ^,P/4p V 'd!SPV R,4'dRMRpS!WR7R# +'giL\;i)a Generate with a masked-diffusion text model (llada, nemotron). The model's own blocking generate loop runs the diffusion; an ``on_block`` callback streams each completed block back as one StreamingToken (models without the hook fall back to a single final chunk). NrrrrXcj<V'dS PVRR7MRpVPS4'dV\S4RpMS'gTpMVfRMTpV'g V'gR#SPR4;'gRpS P \ TV'd VR,M^RT\ P!4R, V^8dSPV, MR\\V4S, ^4R74VPS4'dVo\\V4S4oR#) T)skip_special_tokensrXN prompt_timerceAr r) decode startswithrrCrIrruget_peak_memoryr)rf) rrrdeltar  emitted_textr gen_statsrrrGs && r=flush;ResponseGenerator._generate_masked_diffusion..flushsIO9##F#EUWD|,,S.01!,3#-- 6==#K JJ(.&*A!"/ " 2 2 4s :8Ca 4T #CK.$@! D  |,,#  Vn=Nr<c<S!V4SPSP44SS9dSPS4R#R#)FT)rkrar)rrrrrs&r=on_block>ResponseGenerator._generate_masked_diffusion..on_blocks= &M   T668 9i!!#&r<r block_lengthdefault_block_lengthstepsdefault_diffusion_steps gen_lengthrreos_early_stopT visualizeFrGr statsrstoplength)rrH)) thresholddefault_diffusion_threshold) min_thresholddefault_diffusion_min_threshold)editing_threshold#default_diffusion_editing_threshold)num_to_transfer!default_diffusion_num_to_transfer)max_transfer_per_step'default_diffusion_max_transfer_per_step)max_post_steps default_diffusion_max_post_steps)stability_steps!default_diffusion_stability_stepsr)rr[rrurrCrwrGr*rrOrrdrr#rr"rr{rr.rrr)rrrr'rvrrr tuned_kwargsr config_attrvalue generatedrrrrrrrrGsfff&&f @@@@@@r=r,ResponseGenerator._generate_masked_diffusionsI  ! ! -T  99 IINN499 %NN;/  Y^^q%8!$INN 91 2 2  ' ' - -dkk.F.F G   > >>   ! CF6E $)S!! &7%8 9 9 11:: ,,%V-CTJPPbf&?FLL"   ??  $zz1TZZ35FtDJJ  $ $%) "I GGI !:$1$$&!pT>PT UU9u/uFvpp9\)T4TbK Muup9piup9p4\T>P4T=R4\P!4\P!4Rp=?=EKRp=?=ii;i)#a!GPU thread loop with DFlash, EAGLE-3, or MTP speculative decoding. Collects incoming requests, prefills them as a batch with the per-family hooks, then runs the matching round-loop for decode. Finished sequences are filtered out automatically by the round-loop's ``stop_check`` callback. ) make_samplerrN)tempFr _position_ids _rope_deltasrrc38"TFp\V4xK R#5irHrA)ridss& r=r5ResponseGenerator._run_speculative..Rs@-3c#hh-s)r)rmrm left_padding make_cache)rrrrBrmprefill_kwargs)rr"rr!r"rcr )rrrrrrc36<"TFpSV,xK R#5irHr5)rumax_tokens_maps& r=rr=s>AnQ//sc<SV,pVS9dR#VSP9dR#\SV,4SV,8dR#R#)TF)r@r)seq_idxtoken_idr finished_uidsrCr token_listsuidss&& r= stop_check6ResponseGenerator._run_speculative..stop_checksEw-Cm+#4#3#33#;s+,s0CC# r<c3J"TFwrpVP^8HxK R#5ir)r)r_ pending_argss& r=rr=s+(8?4Aq!1%00A58?s!#T) rm first_bonusr.rr token_dtyperJrshared_kv_states eos_token_idsr-r[z] batch=z tokens=c3H<"TFp\SV,4xK R#5irHrA)rrBrHs& r=rr=s%H4ac+a.&9&94s"z accept=z.2fz rounds=z(Error in speculative generation thread: )Hmlx_lm.sample_utilsr7rurrrrrBrmr*r?r@rFrIrrir*r9r:ridrur,rGr-rr.rrrIrrrrf enumeratervint32r!r'r rRrrSrrr)rQrrrrD _stream_textrrrCr[r(all accept_lenssumrupperfinalizerJrrrrkrOrrr)Crrr"rdrafterrmis_mtpr@eos_setrrr`rqueuesr stream_infosprompt_tokens_mapprompt_tps_map all_input_idsprompt_kwargs_listrr'r-rvrtrrrBmax_lenr<r>ipaddedinput_mxinputs_embeds_mxrlrr policy_kwargsprompt_startedrhiddenrQsample_row_idsrOprompt_elapsedfb_listjtokis_stopis_maxfinishrmax_tokrJ rounds_itertok_listrMralmean_ar error_queuesrGrCrHrIsCf @@@@r=r"ResponseGenerator._run_speculative s F--b.?.?.AB ZZ & &""__ u$3JH+1#d&&'tQ'9;***GGm '+'E'E ?A(F($ ! !#$&!!# " %'"2//'+B$2~..&*BOGNCFJ tV,0OOJ,O)IzV*CKK$#)GCL')K$"$8 NN6t~~F%)L% +///N3'-:%c*!(():):1)=)D)D)FG&--j9JJ0S VW"006OO-Q:OG#HO&I@-@@>KLms#c( 2 2m L>G >V>VFAsQC,q/)C//>V88F"((;2N& 3/ - =) !-*   %:$;! C= CN C /JJ&"2!- ')#0)-%!%!2!2!4 @$ !"&7&7 ! X2*cB;A3#7#7t "#q0JJ* cAg   VSZZ8!%!2!2!4~!EC$5c$:M)1,!1C&6!#3' !$ &,,.'oFAsgaj/C$++C0!T%5%55G S!12nS6IIF'.VHDF,,\#->VLDCL$$&!%"%%(*0FL(:(:(  ) ((.%))#.'.*}%T2>>>!!<JJ ) +&#%5 ")$'C(8?(CCC(8?(%&6")"**' *$/KHa"+H"53;$"1g-/$#C(//4!,S!1"%)9)9"9!$Vs0C!C+2FPT#00c1BCP ((*%)&)),.4JPB,>,>,@3,FVW+9+=+=c+B  "-#CL,,T2)--c29#6:=)SY6?$/D((!"gB/3r7:FJ,,./xs;""%%H4%H"H!IJ""(Xc"gY@ C-/+C0<EEG ((*%)&'),.6,.,>,>,@3,F+9+=+=c+B    ((. uf Mj @DE##%AHAQRAQv6 F 2AQR R##'$R'JFQRZ%7'$RS|224a>   s=%a2$a2.E0a2a'9a2 $a,-G6a2$A9a2"Aa2,A#a2B'a2:A9a28Aa2a2A9a2B a2' a22 e=5e2cec7 6Aeec T;'g/pVP!R /VBwrVVFUpVPV9gKVPW'P,R&\VR^4W'P,R&KW V'gR#VEFupVPV9dKW(P,p V R,p VPp V fV R,P 4p ^p M[\ V R4'd.V P4p VPWVP4p MVPWVP4p VPp T P\T T T VPVP'd\P!4R, M^V PR4\VRR4V PR^4R 74VPfEKXV PR4W(PEKx R#) z,One batch generation step: prefill + decode.rrNrrrNr r)rrrrrrrrr5)nextrrrrr^r*rNrYr token_logprobrIrrurrC)rrrrkwargsprompt_responses responsesprompt_responserrrrurlps&&&& r=rResponseGenerator._steps!!r&/nn&>v&>#/O""f,?ooo 2 2 4s :ST#xx 5!(ND!A"&((?A">  * 4 55MEr<cB<V^8dQhRS[RS[RS[S[,RS[/#)rkrrrro)rprDrrd)rqrs"r=rrr;Cs.>>>c>(3->TW>r<c 2VR,PW#4#)z:Convert one generated token into a streaming text segment.r)advance)rrrrs&&&&r=rYResponseGenerator._stream_textCsJ''==r<c PVP'dVPW4K%R#)zFDrain all pending text-only prompts before inserting an image request.N)has_pending_promptsr)rrrs&&&r=rResponseGenerator._flushGs+++ JJy ),r<c b<V^8dQhRS[RS[S[,RS[S[,RS[S[,/#)rkrsrtrurv)rdrr r)rqrs"r=rrr;LsA    ~  *+  r<c \4fR#VP4T;'g\\4R7pVP WV4p\ \ V4VP4R#)z:Validate request size before opening a streaming response.Nrx)r$rXrrUryr3r+r.)rrsrtrurvr's&&&&& r=validate_context_budget)ResponseGenerator.validate_context_budgetLs[ ( ) 1  NN*6K6MN))&%@ (  ,doo r<)rMrLrKrJrIrOrrrrmrBrrCrDrrrrErHr@rGrFrArH)NNN)NN)#r6r7r8r9r:rrrrrSrXr\rarpr{ryrrrrrrrrrNrrrrrrYrrr;rrs@r=r9r9Ss'++/537""H' 77% 6 p" " H  (  . > >%%"  $  "!%!%F$$" $$  $$$$L]~''R(Th3TEN/"b>>*    r<r9){rloggingrArSr collectionsr dataclassesrrrMrr+r threadingrr r typingr r r rrmlx.corecorerufastapirrUrrXrrr{rrrrrrrrrrrrrr r!generate.commonr"r#generate.diffusionr$r% sample_utilsr&speculative.utilsr'r(r)r* structuredr+tokenizer_utilsr,r-utilsr.r/r0r1 getLoggerr\rYrerrrr[r4rFrOrRrUr`rir{rrrrrrrrrr r rrrr"r$r+r3r6r9rErHrJrrrrrrrrrrr9r5r<r=rs  (%))==!6"=W) 6N@@   + ,#(+%NNG OI$> 3$l+V+V\ $(  &* 5- :8GF! L  Q ' 'ggT& D*.D#'D'+D'+D$(D !D"#D$$%D&#'D("&)D*+DNR< JJ JZ   MM M8  $"J==@H H r<