+ QV-jp^RIt^RIHt^RIHtRRlt]P PRRl4tR#)N)PagedAttentionCache)!lazy_import_paged_flash_attentionc*V^8dQhR\PPR\PR\PR\PR\PR,R\R\PR \P\ \ \P3,,R \R \\ \ \3,,R \PR,R \\PR3,/ #)moduleqkvattention_maskNcache cu_seq_lens_q cu_seq_lens_k max_seqlen_q max_seqlen_k block_tablereturn) torchnnModuleTensorrdictstrinttuple)formats"v/Volumes/fast/ai/experiments/ui-tars-smoke/.venv/lib/python3.14/site-packages/transformers/integrations/flash_paged.py __annotate__rsQQ HHOOQ ||Q ||Q || Q LL4' Q  Q<<Q<<$sELL'8"99QQS#X&Q$Q 5<< Qc \VPP4wr\VRR4'gRMVP^, ^3pVR8XdRMRp\ V\ 4'dW,pW,p V Ef VPVVVPV R,V R,R7wr#R V 9dR V PR 4/M/pV !VP^^4P^4P4VP4VP4VP\P4VP\P4P!4VV 3R VP"R R R V/VBp\ V\$4'd V^,pVR3#R V 9d R V R ,/M/p\'WW#WWWV 3 /VBpVR3#)a Performs the forward pass of attention with paged key-value cache. This function handles the cache updates and performs the attention computation. For decode-only batches (when block_table is provided), uses `flash_attn_with_kvcache` for fused attention + cache update. Otherwise uses `flash_attn_varlen_func`. See the [paged attention guide](https://huggingface.co/docs/transformers/en/paged_attention) for more details. Args: q: (1, nheads, total_q, headdim), where total_q = total number of query tokens in the batch. k: (1, nheads_k, total_k, headdim), where total_k = total number of key tokens in the batch. v: (1, nheads_k, total_k, headdim), where total_k = total number of key tokens in the batch. cu_seq_lens_q: (batch_size + 1,), dtype torch.int32. The cumulative sequence lengths of the sequences in the batch, used to index into q. cu_seq_lens_k: (batch_size + 1,), dtype torch.int32. The cumulative sequence lengths of the sequences in the batch, used to index into kv. max_seqlen_q: int. Maximum query sequence length in the batch. max_seqlen_k: int. Maximum key sequence length in the batch. block_table: (num_groups, batch_size, max_blocks_per_seq), dtype int32. Block table for paged KV cache. If provided, uses flash_attn_with_kvcache for fused attention + cache update. For each request, the block table is a vector of size (max_blocks_per_seq,) with indices indicating the physical location of the cache to read from and write to. The kernel, using the cache_seqlens for that request, knows how much cache to read and dispatches the read using the block table. Same for the write. If a request has fewer than max_blocks_per_seq blocks, the block table is padded with -1s to indicate that the block is not allocated. sliding_windowFfull_attentionsliding_attentionN read_index write_index) key_states value_states layer_idxr#r$s_aux softmax_scalecausalT window_size)r,)rconfig_attn_implementationgetattrr isinstancerupdater'get transposesqueeze contiguoustorint32clonescalingr_paged_decode_forward)rrr r r r r rrrrkwargsflash_attn_varlen_funcflash_attn_with_kvcacher layer_type custom_kwargs attn_output flash_kwargss&&&&&&&&&&&, rpaged_attention_forwardrBsJ7X **73 &-V5Eu%M%MXTZTiTilmTmopSqN%3x%?!EXJ-&&%1 #/ ||&&l+}-  ;BV:K&**W"56QS , KK1  % %a ( 3 3 5 LLN LLN   U[[ )   U[[ ) / / 1    !..    '     k5 ) )%a.K   6=5F1B + qU>\g kw   rcFV^8dQhR\PPR\PR\PR\PR\R\PR\ \ \ 3,R\PR \P/ #) rrrr r r rr rr)rrrrrrr)rs"rrr\s."." HHOO." ||." ||." || ."  ." << ."#s(O."." \\."rc  VPVP,wrVPV ,PR VPVP VP 4p VPV ,PR VPVP VP 4p VP^^^^4P4pVP^^^^4P4pVP^^^^4P4pVP^4pV^V^,VRV, ^, P\P4pW,WPV4&V!R RVRV RV RVRVRVRVPR R R V/ V Bp\!V\"4'd V^,pVP%^4#)zaDecode fast path using flash_attn_with_kvcache. Disabled because FA3 has issue with tracing this.Nrk_cachev_cacher r cache_seqlensr)r*Tr+r,)layer_index_to_group_indicesr' key_cacheview block_sizenum_key_value_headshead_dim value_cachepermuter5sizer6rr7get_block_table_keyr9r0rr4)rrr r r rr r=rrA group_idxlayer_idx_in_grouprErF batch_sizerGr@s&&&&&&&&&, rr:r:[s%*$F$FvGWGW$X!Ioo0166r5;K;KUMfMfhmhvhvwG 2388 E  e77G !Q1((*A !Q1((*A !Q1((*AJ"1zA~6{ 9SSVWW[[\a\g\ghMGRG]L**+BCD)        $ nn  #   K+u%%!!n   q !!r) rgeneration.continuous_batchingrmodeling_flash_attention_utilsrrBcompilerdisabler:rHrrrZs3 @NQh."."r