iTddlmZddlmZmZmZmZmZddlm Z ddl m Z ddl mZddlmZmZmZmZddlmZmZddlmZdd lmZdd lmZeGd d eZGd de j<Ze j@de jBdee jBde"de"de"de#de$de%dee jBe jBffdZ&Gdde j<Z'Gdde j<Z(Gdde j<Z)Gdd e j<Z*Gd!d"e j<Z+Gd#d$e j<Z,Gd%d&e j<Z-y)') dataclass)AnyDictListOptionalTupleN)swiglu) BaseModelArgscreate_attention_maskcreate_ssm_maskscaled_dot_product_attention) ArraysCacheKVCache)gated_delta_update) MultiLinear) SwitchGLUceZdZUeed<eed<eed<eed<eed<eed<eed<eed<eed <eed <eeefed <eed <eed <eed<eed<dZ e eeefed<dZ e ed<dZ e eed<dZe eed<dZe eed<dZe ed<dZeed<dZeed<dZeed<dZe ed<d Zeed!<dZeed"<dZeed#<dZe ed$<dZeed%<dZeed&<y)' ModelArgs model_type vocab_size hidden_sizenum_hidden_layersnum_attention_headsnum_key_value_headsintermediate_sizehead_dim rope_theta rms_norm_epslinear_attn_configmodel_max_length num_expertsmoe_intermediate_size kv_lora_rankN rope_scalingFtie_word_embeddingsqk_nope_head_dimqk_rope_head_dim v_head_dim mla_use_noper num_experts_per_tokenrnum_shared_expertssigmoidmoe_router_activation_funcTmoe_renormalize?routed_scaling_factorfirst_k_dense_replacemoe_layer_frequse_grouped_topknum_expert_group topk_group)__name__ __module__ __qualname__str__annotations__intfloatrrr%rr&boolr'r(r)r*r+r,r.r/r1r2r3r4r5r6n/Volumes/fast/ai/experiments/voice-extract-mac/.venv/lib/python3.12/site-packages/mlx_lm/models/kimi_linear.pyrrs$OOMS#X&-1L(4S>*1 %%&*hsm*&*hsm* $J $L$!"3"&// OT #&5&!"3"NC!d!cJr@rcveZdZ ddedeedeeffd ZdejdejfdZ xZ S) KimiMLPargsrrct||xs |j}|xs |j}t j ||d|_t j ||d|_t j ||d|_yNFbias) super__init__rrnnLinear gate_projup_proj down_proj)selfrDrrdimhidden __class__s rArJzKimiMLP.__init__:sl -T--"~>NaPQQK $++fll33{f2gr]Cwwvqr266B6N j OOLa!e"Ec2A2qjQ ""    Y ' HHS -   FB+ ??F7  ;C%K HD    T ;F qy[jjb4j85@ +% // //r@c\eZdZdeffd Zdej dej fdZxZS) KimiSparseMoErDct|||_|j}|j}| t dt j||d|_t||j||_ tj|ftj|_|j r,|j|j z}t#|||_yd|_y)Nz,num_experts must be specified for MoE layersFrGro)r)rIrJrDrr"rqrKrLgaterr# switch_mlprZzerosfloat32e_score_correction_biasr,rCshared_experts)rPrDrRexperts shared_hiddenrSs rArJzKimiSparseMoE.__init__ys  !!"" ?KL LIIfgE: #FD,F,FP')xx "**'M$  " " 669P9PPM")$-"PD "&D r@rTrUc |j|}t||j|jj|jj |jj |jj|jj|jj\}}|j||}||dzjd}|j||j|z}|S)N.Nrmrk) rrrrDr+r5r6r1r/r.rrur)rPrTr{rweightsouts rArXzKimiSparseMoE.__call__s1,   ( ( II + + II & & II II + + II % % II 0 0  gooa&WY'',,",5    *++A..C r@) r7r8r9rrJrZr[rXr\r]s@rArrxs*'Y'$"((rxxr@rc eZdZdeffd Z ddej deej deedej fdZ xZ S) KimiMLAAttentionrDct|||_|j|_|j |_|j xs |j|_|jxsd|_|j |jz|_ |jxs |j|_ |j|_ |jdz|_ |j}tj||j|jzd|_tj||j|jzd|_tj$|j|j&|_t+|j |j|j|_t+|j|j|j|_tj|j|jz|d|_y)NrFrGeps)rIrJrDr num_headsrr'rr( q_head_dimr)r$scalerrKrLq_projkv_a_proj_with_mqaRMSNormrkv_a_layernormrembed_q unembed_outo_proj)rPrDrRrSs rArJzKimiMLAAttention.__init__s  11#'#;#; $ 5 5 F $ 5 5 ://$2G2GG//:T]] --__d* !!ii(HuU "$))     5 5 5#  !jj):):@Q@QR"  ! !4#4#4dnn  '   t ii @&uU r@rTmaskcacherUc |j\}}}|j|j|||j|j}|j dddd}t j||jgd\}} |j|} t j| |jgd\} } | j||d|jj dddd} |j| } t j| d} ||j| | \} } | |jz| j!ddz} |\t j"|| t j$t j&| j(j*| j(} |dk(r|j-|}| x}}n$|j-| d }|j/| }t1|||||j| }|dk(r|j/|}|j ddddj||d}|j3|S) Nrrjr rerkrmF) transpose)rrr)rirreshaperrrrZsplitr'rr$r(r expand_dimsupdate_and_fetchrswapaxeswherer[finforpminrrrr)rPrTrrBL_qq_nopeq_pe compressed_kvk_pe kv_latent pe_scoresr~voutputs rArXzKimiMLAAttention.__call__s  ''1a KKN " "1a I KK1a #xxD$9$9#:D //2  hh}t7H7H6IPRS t||Aq!T%:%:;EEaAqQ'' 6 NN915  #44YEOItDJJ&$--B*??  )//266 HI 6\\&)F A Y% 8A  +A- AqTZZi  6%%f-F!!!Q1-55aB?{{6""r@rY) r7r8r9rrJrZr[rrrXr\r]s@rArrsZVYV>$(#' -# 88-#rxx -# -#  -#r@rc eZdZdedeffd Zdej deej deej deej deej ej ff d Z xZ S) ShortConv1dchannels kernel_sizecpt|||_tj|||d|d|_y)NFr) in_channels out_channelsrrHgroupspadding)rIrJrrKConv1dconv)rPrrrSs rArJzShortConv1d.__init__s8 &II !#   r@rTstaterlengthsrUcZ|tj|d|d}|Ktj|jd|jdz |jdf|j }tj ||gd}tj|j|}|jdz }|btj|d|jd}|dddftj|zd} tj|| d} || fS|dd| dddf} || fS)Nrrr rerork) rZrrrirrp concatenaterKsilurcliparangerz) rPrTrrr conv_inputrn_keepends positions new_states rArXzShortConv1d.__call__s  i!Q/A =HHT--11772;?qwwE^^UAJQ7 ggdii +,!!A%  777Aqwwqz2Dag6)::IFI**:yqIII~#1vgh>2II~r@) r7r8r9r<rJrZr[rrrXr\r]s@rArrs|    3   88!rxx  "((#  rxx! " r@rc eZdZdedeffd Z d dejdeejdee dejfdZ xZ S) KimiDeltaAttentionrD layer_idxc Lt||j}||_|d|_|d|_|j dd|_|j|j z|_|j}t|j dz|_ tj||jd|_tj||jd|_tj||jd|_t#|j|j|_t#|j|j|_t#|j|j|_tj||j d|_tj|j |jd|_tj||jd|_tj||j d|_tj|j |jd|_t5j6t5j8t4j:j=dd |jf d |_t5j@|jf|_!tjD|j |jF |_$tj|j|d|_%y) Nrrshort_conv_kernel_sizerFrGr0g0@)lowhighri)rr rr)&rIrJr rrrget conv_kernelprojection_dimrr=rrKrLrk_projv_projrq_convk_convv_convf_a_projf_b_projb_projg_a_projg_b_projrZrlograndomuniformA_logrdt_biasrro_normr)rPrDrcfgrRrSs rArJzKimiDeltaAttention.__init__s- %%"[)J 77#;Q?"nnt}}<!!4==)T1 ii(;(;%H ii(;(;%H ii(;(;%H !$"5"5t7G7GH !$"5"5t7G7GH !$"5"5t7G7GH  &$--eD  $--1D1D5Q iiUC  &$--eD  $--1D1D5Q ^^ FF299$$4?P$Q R   xx!4!4 67 jjD4E4EF ii 3 3V%H r@rTrrrUc |j\}}}|j}||\}} } } |j} n d}d} d} d} d} |7tj||j dz |j f|} | }| } | } |j|j|||| \}}|j|j|| || \}} |j|j|| || \}} |||d<| |d<| |d<|j|||j|j}|j|||j|j}|j|||j|j}|j }|dztj"j%|ddz}|tj"j%|ddz}|j'|j)|j|||j|j}|j+|j|||j}t-||||||j.j|jd|j0j|j|j| ||j2  \}} || |d<|j5||j7|j9|j|||j|j}|j;|j|||j|jtj<|zj||d}|j?|S) Nr rorrjgư>)rr use_kernelrre) rirprrZrrrrrrrrrrrrrfastrms_normrrrrrrtrainingadvancerrrr-r)rPrTrrrTrrpq_statek_statev_state ssm_statersrrrrr~r inv_scalea_logitsb_logitsrrs rArXzKimiDeltaAttention.__call__:s  ''1a  38 0GWgymmGGGGIG ?!T--143F3FGuUAGGG++dkk!ngtWM++dkk!ngtWM++dkk!ngtWM  E!HE!HE!H NN1a ? NN1a ? NN1a ?JJ \RWW--at< < ((D$7 7==q!12:: q$..$-- ;;q>))!Q?+   JJ  t~~q 1 LL  ?==(  Y   E!H MM! }}T]]1-.66 q$..$--  KK Aq$..$--H Ijj  '!Q  {{3r@rY r7r8r9rr<rJrZr[rrrXr\r]s@rArrsg$IY$I3$IR$(# G 88G rxx G } G  G r@rc eZdZdedeffd Z d dejdeejdee dejfdZ xZ S) KimiDecoderLayerrDrct||jd}|dz|v|_|jrt |||_nt ||_|jdkDr2||jk\r#||jzdk(rt||_ nt||_ tj|j|j |_tj|j|j |_y)N kda_layersr rr)rIrJr is_linearr self_attnrr"r2r3rmlprCrKrrrinput_layernormpost_attention_layernorm)rPrDrrrSs rArJzKimiDecoderLayer.__init__s ,,\: #a-J6 >>/i@DN-d3DN   q T777D///14$T*DHt}DH!zz$*:*:@Q@QR(*   $"3"3) %r@rTrrrUc|dn|}|j|j|||}||z}|j|j|}||zSrW)rrrr)rPrTrr attn_cacheyhzs rArXzKimiDecoderLayer.__call__sW #]T NN4//2D* E E HHT2215 61u r@rYrr]s@rArrs^ Y 3 6$(#  88 rxx  }   r@rcpeZdZdeffd Z ddej deee dej fdZ xZ S)KimiLinearModelrDct|tj|j|j |_t|jDcgc]}t||c}|_ tj|j |j|_ |jd}|ddz |_tt!|jD]}|dz|vs ||_yycc}w)Nrrrr )rIrJrK Embeddingrr embed_tokensrangerrlayersrrnormr ssm_idxlenattn_idx)rPrDirrSs rArJzKimiLinearModel.__init__s LL$:J:JK:?@V@V:WX:WQ'a0:WX JJt//T5F5FG ,,\: !!}q( s4;;'(AAj( ! ) YsC1inputsrrUcV|j|}|dgt|jz}t|||j}t |||j d}t|j|D] \}}|jr|n|}||||}"|j|S)NT) return_array)rr) rrrr rr rziprr) rPrrr ssm_mask attn_masklayer layer_cachers rArXzKimiLinearModel.__call__s   f % =FS--E"1eDLL&9:)!U4==-APTU "%dkk5"9 E;$8IDad+6A#:yy|r@rW) r7r8r9rrJrZr[rrrrXr\r]s@rAr r sF Y &*S "  r@r ceZdZdeffd Z d dej deee dej fdZ e dZ dZ d eeej fdeeej ffd Ze d Ze d ZxZS)ModelrDct|||_|j|_t ||_|j rd|_ytj|j|jd|_yrF) rIrJrDrr modelr&lm_headrKrLrr)rPrDrSs rArJzModel.__init__sZ  //$T*  # #DL99T%5%5tUSDLr@rrrUc|j||}|j%|jjj|S|j|SrW)r#r$r as_linear)rPrrrs rArXzModel.__call__sH jj' << ::**44S9 9||C  r@c.|jjSrW)r#r)rPs rArz Model.layersszz   r@cg}|jD]C}|jr|jtd+|jt E|S)Nr)size)rrappendrr)rPcachesrs rA make_cachezModel.make_cachesB[[E kq12 gi( !  r@rc l|jDcic]\}}|jdr||}}}|jjr|j ddt |j D]\}}d|}t|jtr|d}|d}dD]t\} } |d| d} | |vst|jjD cgc]} |j |d | d | d} } tj| ||d | d<vd D])}|d |d}||vs|j |||d |d<+|d}||vr|j |||d<|d}||vr|j |||d<t|dd}t|tr|d}dD]O\}}|d |d}||vs|j |}|j dk(r|j#dd}|||d |d<Q|d}||vr.||j dkDrtj$||d||<|d}|d}||vs|jj&xs|jj(}|jj*xs|jj(}||z}|jj,}|d|v}|j |}|r|jj.}|j |d}|j |d} |j0ddz|z}!||j0dz}"tj2||| |!|"}|j%||d}tj4|ddd|ddfj7dd }#tj4|dd|dddf}$|rXtj8|#!"\}#}%}&tj8|$|!|"\}$}'}(|%||d!<|&||d"<|'||d#<|(||d$<|#||d%<|$||d&<|Scc}}wcc} w)'Nz model.mtpzlm_head.weightz model.layers.z.block_sparse_moez.mlp))w1rM)w2rO)w3rNz .experts.0.z.weightz .experts..z .switch_mlp.)rMrNrOz.shared_experts.z .gate.weightz.gate.e_score_correction_biasz.e_score_correction_biasrz .self_attn))q_conv1dr)k_conv1dr)v_conv1drrrjr z .conv.weightz.dt_bias)rez.kv_b_proj.weightz.kv_b_proj.scalesz.kv_b_proj.biasesre )bits group_sizermz.embed_q.scalesz.embed_q.biasesz.unembed_out.scalesz.unembed_out.biasesz.embed_q.weightz.unembed_out.weight)items startswithrDr&pop enumerater isinstancerrrr"rZstackgetattrrndimmoveaxisrr'rr)rr$ri dequantize contiguousrquantize))rPrr~rrrprefix src_prefix dst_prefixsrcdstkeyrstackednamesrc_keygate_keybias_keyattn attn_prefixsrc_namedst_namewdt_keykv_b_keyqk_nopev_headrr quantizeddimsscalesbiasesr6r7wkwvwk_swk_bwv_swv_bs) rAsanitizezModel.sanitizes$+MMOUODAq1<< ;T1a4OU 99 ( ( KK($ / )$++ 6 Iu$YK0F%))]3 &x'89 &xt_ !HC (L C5@Cg~&+499+@+@%A#%A$KK:,is!C5(PQ%A #LN88#L:,l3%w GH!BD!+ ,tfG LMB)\6w&;B;;x;PGzl,78(\)FGw&GN{{ HGzl*BCD5+t4D$ 23!' 3 +&Hh "- Qxj@G')#KK066Q; ! 1a 0AKL;-q , GH+(=1W$v++a/*,**WV_e*L#HJ/K%&78H7"))44J 8J8J--C1C1C"V+ II99 *m+<=H KK)9911D$[[K=8I)JKF$[[K=8I)JKFGGBK",5D!%b)9!9J 66AIIi26]]1Q!^#4#=#=b"#EF]]1Q!^#45%'[[$:%VNBd%'[[$:%VNBd?CG{m?;?@CGG{m+>?@;=;-78?A;-':;.predicateSs*(D0}}W%y)Ar@)r:rPrhs rAcast_predicatezModel.cast_predicateQs C r@c d}|S)Nc0|jdrdddSy)Nzmlp.gate@)r7r6Trf)rdrs rArhz(Model.quant_predicate..predicate^s}}Z(&(!44r@r?ris rAquant_predicatezModel.quant_predicate\s  r@rW)r7r8r9rrJrZr[rrrrXpropertyrr,rr:rbrjror\r]s@rAr!r!sTYT&*!!S "!  !!!cS"((] 3cS"((]8KcJr@r!). dataclassesrtypingrrrrrmlx.corecorerZmlx.nnrK activationsr baser r r rrrr gated_deltarmlar switch_layersrrModulerCcompiler[r<r=r>r:rrrrrrr r!r?r@rAr}si"33 (+$   DJbiiJ$)0 88)0 288 )0 )0 )0  )0 ! )0)0)0 288RXX )0 )0X#BII#LJ#ryyJ#Z%"))%Pn n b#ryy#Lbii@YBIIYr@