(tiTddlZddlZddlmZddlZddlmZddl m Z e je Z ddefdZej j"j$ddfdedefd ZGd d ej(ZGd d ej(ZGddej(ZGddej(ZGddej(Zy)N)loggingkey_chunk_sizecjdd\}jdt|tjz t j t jdfdfd}t jj|tjd| \}}} tj| dd } tj| | z } |tj| d z}|| z}|jd } tj|djd } | | z S) zBMulti-head dot product attention with a limited number of queries.NF) prevent_csec\tjd||}tj|dd}tjj |}tj ||z }tjd||}tjd|}||jd|fS) Nz...qhd,...khd->...qhk) precisionr Taxiskeepdimsz...vhf,...qhv->...qhfz ...qhk->...qhr)jnpeinsummaxjaxlax stop_gradientexpsum)querykeyvalue attn_weights max_score exp_weights exp_valuesr s n/Volumes/fast/ai/experiments/MLX_z-image/.venv/lib/python3.12/site-packages/diffusers/models/attention_flax.pysummarize_chunkz/_query_chunk_attention..summarize_chunk#szz"95#QZ[ GGLrDA GG)))4 gglY67 ZZ 7 W`a JJ : KOOO4i@@c ltjjdgjdz z|ddgzt j ddgz}tjj dg jdz z|ddgzt j dd gz}||S)Nrroperand start_indices slice_sizes)rr dynamic_slicendimlistshape) chunk_idx key_chunk value_chunk k_featuresrr num_headsrr! v_featuresrs r chunk_scannerz-_query_chunk_attention..chunk_scanner0sGG))#A.)Q1BBSYYs^, :/VV* gg++#a0Iq!3DDU[["-..)Z1XX, ui==r"r)fxsTr r)r,minrsqrt functoolspartialr checkpointrmaparangerr expand_dimsr)rrrr rnum_kvr3 chunk_values chunk_weights chunk_max global_max max_diffs all_values all_weightsr0r1r!r2s````` @@@@r _query_chunk_attentionrFs$$'IIbcN!FIzRJ0N CHHZ( (Es~~59 A: A>>".1WW[[=SZZXY[acqMr[-s*L-T:J J./ICOOIB77LYM!!q!)J//-488a8@K  ##r"iquery_chunk_sizec  jdd\   f d}tjj|ddt j z \}}t j|dS)a Flax Memory-efficient multi-head dot product attention. https://huggingface.co/papers/2112.05682v2 https://github.com/AminRezaei0x443/memory-efficient-attention Args: query (`jnp.ndarray`): (batch..., query_length, head, query_key_depth_per_head) key (`jnp.ndarray`): (batch..., key_value_length, head, query_key_depth_per_head) value (`jnp.ndarray`): (batch..., key_value_length, head, value_depth_per_head) precision (`jax.lax.Precision`, *optional*, defaults to `jax.lax.Precision.HIGHEST`): numerical precision for computation query_chunk_size (`int`, *optional*, defaults to 1024): chunk size to divide query array value must divide query_length equally without remainder key_chunk_size (`int`, *optional*, defaults to 4096): chunk size to divide key and value array value must divide key_value_length equally without remainder Returns: (`jnp.ndarray`) with shape of (batch..., query_length, head, value_depth_per_head) rNc  tjj dg jdz z|ddgzt j ddt gz}| zt| fS)Nrr$rr%)rrrr r)rrr)r*r+r,r6rF) r-_ query_chunkrrr1num_qr q_featuresrrGrs r r3z5jax_memory_efficient_attention..chunk_scannerfsgg++3%**q.1iA5FFU[["-.#6F2NPY[e1ff,  ( ( "!s%9]k   r"r)r4initr5lengthr)r,rrscanmathceilr concatenate) rrrr rGrr3rJresr1rLrMs `````` @@@r jax_memory_efficient_attentionrUOsr*$);;rs#3 E9j   WW\\   yy!112 FAs ??3R ((r"ceZdZUdZeed<dZeed<dZeed<dZe ed<d Z e ed <d Z e ed <e jZe jed <d ZdZdZddZy) FlaxAttentiona A Flax multi-head attention module as described in: https://huggingface.co/papers/1706.03762 Parameters: query_dim (:obj:`int`): Input hidden states dimension heads (:obj:`int`, *optional*, defaults to 8): Number of heads dim_head (:obj:`int`, *optional*, defaults to 64): Hidden states dimension inside each head dropout (:obj:`float`, *optional*, defaults to 0.0): Dropout rate use_memory_efficient_attention (`bool`, *optional*, defaults to `False`): enable memory efficient attention https://huggingface.co/papers/2112.05682 split_head_dim (`bool`, *optional*, defaults to `False`): Whether to split the head dimension into a new axis for the self-attention computation. In most cases, enabling this flag should speed up the computation for Stable Diffusion 2.x and Stable Diffusion XL. dtype (:obj:`jnp.dtype`, *optional*, defaults to jnp.float32): Parameters `dtype` query_dimheads@dim_headdropoutFuse_memory_efficient_attentionsplit_head_dimdtypec$tjd|j|jz}|jdz|_t j |d|jd|_t j |d|jd|_ t j |d|jd|_ t j |j|jd |_ t j|j |_y) NFlax classes are deprecated and will be removed in Diffusers v1.0.0. We recommend migrating to PyTorch classes or pinning your version of Diffusers.gFto_q)use_biasranameto_kto_vto_out_0)rarfrate)loggerwarningr\rZscalennDenserarrrrX proj_attnDropoutr^ dropout_layerself inner_dims r setupzFlaxAttention.setups [ MMDJJ. ]]D( XXi%tzzPVW 88ITZZfUXXi%tzzPVW $.. TZZT\\:r"c|j\}}}|j}|j|||||z}tj|d}|j||z|||z}|SN)rrr$r,rZreshaper transposerutensor batch_sizeseq_lendim head_sizes r reshape_heads_to_batch_dimz(FlaxAttention.reshape_heads_to_batch_dimse#)<< GSJJ  GYy@PQv|4 Y 6 AQR r"c|j\}}}|j}|j||z|||}tj|d}|j||z|||z}|Sryr{r~s r reshape_batch_dim_to_headsz(FlaxAttention.reshape_batch_dim_to_headssd#)<< GSJJ  i 7GSQv|4 i 7# /R r"Nc||n|}|j|}|j|}|j|}|jr|jd}t j ||d|j|jf}t j ||d|j|jf} t j ||d|j|jf} n3|j|}|j|} |j|} |jr|jddd}| jddd} | jddd} |jd} | dzdk(rt| dz } n9| dzdk(rt| dz } n"| dzdk(rt| dz } n t| } t|| | | d }|jddd}|j|}n|jrt jd | |} nt jd || } | |j z} t#j$| |jrdnd }|jrWt jd|| }|jd}t j ||d|j|jzf}n(t jd|| }|j|}|j'|}|j)||S)Nrr rzrrr[i@)rGrzb t n h, b f n h -> b n f tzb i d, b j d->b i jrzb n f t, b t n h -> b f n hzb i j, b j d -> b i d deterministic)rrrr`r,rr|rZr\rr_r}intrUrrrnrosoftmaxrqrs)ru hidden_statescontextr query_projkey_proj value_projb query_states key_states value_statesflatten_latent_dimrGattention_scoresattention_probss r __call__zFlaxAttention.__call__s#*?-ZZ . 88G$ZZ(   ##A&A;;zAr4::t}}3UVLX2tzz4==/QRJ;;zAr4::t}}3UVL:::FL88BJ:::FL  . .'11!Q:L#--aA6J'11!Q:L ".!3!3B!7 !B&!+#&'9B'>#? #b(A-#&'9B'>#? #a'1,#&'9A'=#> #&'9#: :j,IYjrM*33Aq!