+ QV-j*^RIt^RIHt^RIHt]P !]4tRt^t ^t Rs RRlt ]PPRRR7RR l4t]P R 4tR R ltR tRRRlltR#)N)logging)sdpa_attention_forwardc$V^8dQhR\/#)attn_implementation)str)formats"x/Volumes/fast/ai/experiments/ui-tars-smoke/.venv/lib/python3.14/site-packages/transformers/integrations/msa_attention.py __annotate__r scc<\e\#^RIHpVPR4R ,pVP R4wr#pT!Y$;'gRV'dRM^RR7pR F0p\ \ WVR44'dK \RV RV R 24h Vs\#) agLoad the MSA hub kernel once and verify the expected callables are present. The ``attn_implementation`` string may carry a ``paged|`` prefix and/or an ``@`` pin (e.g. ``kernels-staging/msa@v0``); the build currently lives on the repo's ``v0`` branch. The loaded module is cached in a module-level global so registration happens once, not per call. N) get_kernel|@T)revisionversionallow_all_kernelszThe MSA kernel loaded from `z` does not expose a callable `zK`. Make sure you request a compatible build, e.g. `kernels-staging/msa@v0`.)sparse_atten_func build_k2q_csr) _MSA_KERNEL hub_kernelsrsplit partitioncallablegetattr ImportError)rrrepo_id_revkernelfn_names& r load_and_register_msa_kernelr# s'!'',R0G'',OG ++stPQei jF9677.wi7UV]U^_[[ :K r ztransformers_msa::sparse_atten) mutates_argscXV^8dQhR\PR\PR\PR\PR\PR\PR\R\R \R \R \R \R \R\R\P/#)rqkvq2k cu_seqlens_q cu_seqlens_ktopk block_sizetotal_k max_seqlen_q max_seqlen_k qheads_per_kvscalingimplreturn)torchTensorintfloatr)r s"r r r =s5$5$ ||5$ ||5$ ||5$  5$ ,, 5$ ,, 5$ 5$5$5$5$5$5$5$ 5$ \\5$r cd\V 4p\PPVP4;_uu_4VP VVVVVV V V R7wppVP VVVVVVVVV V VRV R7 pRRR4VP 4# +'giXP 4#;i)a Opaque wrapper around the CuTe-DSL CSR build + block-sparse kernel. Registered as a ``torch.library`` custom op so ``torch.compile(fullgraph=True)`` treats the whole CSR-build + attention as a single opaque node (no graph break) and ``reduce-overhead`` CUDA graphs can capture it. The internal ``build_k2q_csr`` output is data-dependent in shape, but it never escapes this op (only the fixed-shape ``[total_q, Hq, D]`` attention output does), so the fake/meta impl below is exact. The op is functional (no input mutation). )r.r0r/ qhead_per_kvT)r*r+r/r0blk_kvcausal softmax_scaleN)r#r5cudadevicerr contiguous)r&r'r(r)r*r+r,r-r.r/r0r1r2r3msa k2q_row_ptr k2q_q_indices attn_outputs&&&&&&&&&&&&&& r _msa_sparse_atten_oprE<s2 't ,C   188 $ $%(%6%6    %%&&7 & " ]++   %%%%!,  %4  ! ! ##5 % $4  ! ! ##s ;B B/ c.\P!V4#N)r5 empty_like)r&r'r(r)r*r+r,r-r.r/r0r1r2r3s&&&&&&&&&&&&&&r _msa_sparse_atten_fakerIus$   A r cHV^8dQhR\PR\RR/#)rquerydropoutr4N)r5r6r8)r s"r r r s%  ell U t r cVPPR8wg6\PP VP4^,^ 8wd \ R4hVP R ,\8wd\R\ R24hVPP\8wd\R\ R24hVR8wd \R4hVPPpV\9d\R\ R V R 24hR #) aValidate kernel capability, dropout and configured topk once per attention module. Mirrors the flash-attention integration, which checks capability/dropout at model init rather than on every forward. The check is cached on the module so the hot path never re-runs it. There is no SDPA fallback: a sparse layer either runs the MSA kernel or this raises. Serves both prefill (q_len > 1) and single-token decode (q_len == 1) -- decode is just a varlen call with one query slot, so there is no context-length threshold. r>zMSA block-sparse attention requires an SM100 / Blackwell CUDA device. Select a different `attn_implementation` on unsupported hardware.z2MSA block-sparse attention only supports head_dim .z4MSA block-sparse attention only supports block_size zYMSA block-sparse attention does not support attention dropout; set `attention_dropout=0`.z1MSA block-sparse attention only supports topk in z, got `z0`. Set `index_topk_blocks` to a supported value.Nr)r?typer5r>get_device_capability RuntimeErrorshapeMSA_SUPPORTED_HEAD_DIM ValueErrorindexerr-MSA_SUPPORTED_BLOCK_SIZE topk_blocksMSA_SUPPORTED_TOPK)modulerKrLr,s&&& r _validate_msa_initr[s ||F"ejj&F&Fu||&TUV&W[]&] P   {{200MNdMeefghh ~~  $<<OPhOiijkll#~tuu >> % %D %%?@R?SSZ[_Z`a< <  &r c(aVPwrrVP^,VP^,rW,pVPR,o\V3Rl\44pVS8wdHVP.VPRROVS, N5R4p\P !VV.RR7pVoVP ^^4PW,W4P4pVP ^^4PW,W4P4pVP ^^4PW,W4P4p\P!^V^,V ,V VP\PR7pV^8XdVeVR,^,P\P4P^4p\P !\P!^VP\PR7V.4pMA\P!^V^,V ,V VP\PR7pVP\P4pVPW,S4P^4PV RR4P4p\!VVVVVVSVW,V V VVVP"P$4pVPWW4#)c38<"TFqS8gK VxK R#5irG).0tr,s& r $_sparse_attention..sB"4QT qq"4s N)dim)r?dtyper)rSnextrYnew_fullr5cat transposereshaper@aranger?int32tozeros unsqueezeexpandrEconfig_attn_implementation)rZrKkeyvaluer2 block_indicesr-cache_positionbsz num_q_headsq_lenhead_dim num_kv_headsk_lenr1 padded_topkpadr&r'r(r*valid_kr+r)rDr,s&&&&&&&& @r _sparse_attentionrss(- %Ce))A, ! %/M   r "DB"4BBKd$$%T}':':3B'?%TtAS%TVXY =#"6B?  1%%ck;ITTVA a##CKHSSUA 1%%ck<JUUWA<<C!Gu#4eAHHTYT_T_`L  axN.!"%)--ekk:BB1Eyy%++a "TV]!^_ ||Aa5'8%X]XcXcd   5;; 'C ++ck4 ( 2 21 5 < <\2r R ] ] _C'         **K   s; AAr chV^8dQhR\PPR\PR\PR\PR\PR,R\R\R,R \PR,R \ \PR3,/ #) rrZrKrsrtattention_maskNrLr2rur4)r5nnModuler6r8tuple)r s"r r r s   HHOO  <<    <<  LL4'   T\ <<$&  5<<  r c (VfVPR,R ,pVf\WW#V3RVRV/VB#\VRR4'g\WV4RVnVP P p VPR4p \WW#WgW4p V R3#) zf TODO: this opens a door to per-layer attn implementation which is something we might want lalter on. NrLr2_msa_validatedFTrvrg) rSrrr[rrVr-getr) rZrKrsrtrrLr2rukwargsr-rvrDs &&&&&&&&, r msa_attention_forwardrs++b/T)% 3~ ?F PW [a  6+U 3 36'2 $**JZZ 01N#F3wWarK  r ) r_)NrONN)r5utilsrsdpa_attentionr get_logger__name__loggerrYrWrTrr#library custom_oprE register_fakerIr[rrr_r r rs 2   H %$ 89K5$L5$p##$( :9Bx  r