i-ddlZddlmZddlmZddlmZmZddlm Z ddl m Z ddl mZmZmZddlmZmZmZddlmZmZdd lmZdd lmZeGd d eZdd Zee j<dd dedefdZ Gdde jBZ"Gdde jBZ#Gdde jBZ$Gdde jBZ%Gdde jBZ&Gdde jBZ'y)!N) dataclass)partial)AnyOptional) shard_inplace shard_linear sum_gradients) BaseModelArgscreate_attention_maskscaled_dot_product_attention)KVCacheRotatingKVCache)initialize_rope) SwitchGLUceZdZUdZeed<dZeed<dZeed<dZ eed<d Z eed <d Z e ed <d Z eed<d Zeed<dZeed<dZeed<dZeed<dZeed<dZeed<dZeed<dZeed<y) ModelArgsgpt_oss model_type$num_hidden_layersnum_local_expertsnum_experts_per_toki vocab_sizegh㈵> rms_norm_epsi@ hidden_sizeintermediate_size@head_dimnum_attention_headsnum_key_value_headssliding_windowiI rope_thetaN rope_scaling layer_types)__name__ __module__ __qualname__rstr__annotations__rintrrrrfloatrrr!r"r$r%r&r'rr(listj/Volumes/fast/ai/experiments/voice-extract-mac/.venv/lib/python3.12/site-packages/mlx_lm/models/gpt_oss.pyrrsJs s   JL%K!s!Hc!!  NCJL#Kr2rctj|| |}|d| df}tj|||}||fS)zMLX equivalent of torch.topk)kthaxis.Nr6)mx argpartitiontake_along_axis)akr6partitioned_indices top_k_indices top_k_valuess r3mlx_topkr@&sH//!!$?'aRS1M%%aTBL  &&r2T) shapelessalphalimitctj|d|}tj|| |}||z}tj|}||z}||dzzS)N)a_mina_maxr )r8clipsigmoid)x_linearx_glurBrC glu_scaledsigout_glus r3swiglurN0sY GGEU 3EwwxvU;HJ **Z CckG hl ##r2c$eZdZfdZdZxZS)SwiGLUc"t|yN)super__init__)self __class__s r3rTzSwiGLU.__init__?s r2ct||SrR)rN)rUxgates r3__call__zSwiGLU.__call__Bsar2)r)r*r+rTrZ __classcell__rVs@r3rPrP>s r2rPcveZdZdeffd Zddej dej dej fdZxZS)AttentionBlockconfigct||j|_|j|_|j|_|j|jz|_t j|jf|_tj|j|j|jzd|_ tj|j|j|jzd|_ tj|j|j|jzd|_tj|j|jz|jd|_dt!j"|jz |_t'|j|j(d|j*|_y)NTbiasr F) traditionalscaling_config)rSrTr!r"r$num_key_value_groupsr8zerossinksnnLinearrq_projk_projv_projo_projmathsqrtsm_scalerr&r'roperUr_rVs r3rTzAttentionBlock.__init__Gsi  #)#=#= #)#=#=  & &&*D*D D !XXv99;< ii    : :T]] JQU  ii    : :T]] JQU  ii    : :T]] JQU  ii MMF66 68J8JQU  DIIfoo66 # MM   !..   r2rXmaskreturnc |j\}}}|j}|j}|j|j ||d|j dd} |j |j ||d|j dd} |j|j ||d|j dd} |P|j| |j} |j| |j} |j| | \} } n"|j| } |j| } t| | | ||j||j} |j| j ddj ||dS)Nr )offset)rsrg)shaper!r$rjreshapeswapaxesrkrlrqrxupdate_and_fetchr rprgrm) rUrXrscacheBL_DHkqr<vv_hats r3rZzAttentionBlock.__call__jsT''1a MM  % % KKN " "1aQ / 8 8A > KKN " "1aQ / 8 8A > KKN " "1aQ / 8 8A >   !ELL 1A !ELL 1A))!Q/DAq ! A ! A, q!UDMMDJJ {{5>>!Q/771bABBr2rR r)r*r+rrTr8arrayrZr[r\s@r3r^r^Fs:! y! FC"((C"((C288Cr2r^c\eZdZdeffd Zdej dej fdZxZS)MLPBlockr_cnt||j|_|j|_|j|_t |j|j |jtd|_tj|j|jd|_ d|_ y)NT) input_dims hidden_dims num_experts activationrbra) rSrTrrrrrrPexpertsrhriroutersharding_grouprrs r3rTzMLPBlock.__init__s !--!'!9!9#)#=#=  ))0000x   ii 2 2F4L4LSWX "r2rXrtc|jt|j|}|j|}t||jd\}}t j |dd}|j||}|t j|dz}|jd}|j+t jj||j}|S)Nrv)r<r6T)r6preciser7group) rr rr@rr8softmaxr expand_dimssum distributedall_sum)rUrXgrindicesexpert_weightsys r3rZzMLPBlock.__call__s    *2 d11215A KKN#A)A)AKG"dC LLG $ ~B7 7 EErEN    *&&q0C0C&DAr2rr\s@r3rrs*#y#""((rxxr2rcveZdZdeffd Zddej dej dej fdZxZS)TransformerBlockr_ct|t||_t ||_t j|j|j|_ t j|j|j|_ yrR) rSrTr^ self_attnrmlprhRMSNormrrinput_layernormpost_attention_layernormrrs r3rTzTransformerBlock.__init__sg '/F#!zz&*<*Q>QR(*    3 3) %r2rXrsrtc|}|j|}|j|||}||z}|}|j|}|j|}||z}|SrR)rrrr)rUrXrsr}residuals r3rZzTransformerBlock.__call__sb   # NN1dE * qL  ) )! , HHQK qLr2rRrr\s@r3rrs5 y  "(( "(( 288 r2rcheZdZdeffd Z ddej deej fdZxZ S)GptOssMoeModelargsc:t|tj|j|j |_tj|j |j|_ |jxsddg|jdzz|_ t|jDcgc] }t|c}|_|j|_|jj#d|_|jj#d|_ycc}w)Nsliding_attentionfull_attentionrw)rSrTrh Embeddingrr embed_tokensrrnormr(rrangerlayersr% window_sizeindexswa_idxga_idx)rUrrrVs r3rTzGptOssMoeModel.__init__s LL$:J:JKJJt//1B1BC ++*  0  # #q (0*8=T=S=S7TU7T!'-7TU ..''--.AB &&,,-=> Vs,Dinputsinput_embeddingsc||}n|j|}|dgt|jz}t|||j}t|||j |j }t|j||jD]\}}} | dk(r|n|} ||| |}|j|}|S)N)rr) rlenrr rrrzipr(r) rUrr}rrX full_maskswa_masklayerc layer_typerss r3rZzGptOssMoeModel.__call__s  ' A!!&)A =FS--E)!U4;;-?@ ( uT\\"0@0@ %( UD >9HDaq!A%N IIaLr2)NN) r)r*r+rrTr8rrrZr[r\s@r3rrs; ?Y ? /3 #288, r2rceZdZdeffd Zd dej fdZdZd de ejjfdZ e dZe d Zd ZxZS) Modelrct|||_|j|_t ||_t j|j|jd|_ y)NFra) rSrTrrrmodelrhrirrlm_head)rUrrVs r3rTzModel.__init__sJ  //#D) yy!1!14??O r2rcD|j|j||SrR)rr)rUrr}s r3rZzModel.__call__s||DJJvu566r2ctd|jDr|Si}|jD]\}}d|vrd|vrd|vr@|jtj j d}|jdd}d|vr|jdd}t j|d ddd ddf||jdd <t j|d d dd ddf||jdd <d|vred|vrad|vr@|jtj j d}|jdd}d|vr|jdd}|||<9d|vrbt j|d ddd f||jdd<t j|d d dd f||jdd<d|vr|||jdd<|||<|S)Nc3$K|]}d|v yw)zgate_proj.weightNr1).0r<s r3 z!Model.sanitize..s?1!Q&s gate_up_projrb_blocksrz.weight_scalesz.scales.rw gate_projr up_proj down_projgate_up_proj_biaszgate_proj.biasz up_proj.biasdown_proj_biaszdown_proj.bias) anykeysitemsviewr8uint32flattenreplace contiguous)rUweights new_weightsr<rs r3sanitizezModel.sanitizes ? ? ?N MMODAq"vQ>ryy)11"5A )Y7A> )Y7AFHmmc3Q3kNG AIInkBCEGMMc14a4lOE AIIni@A!fAo>ryy)11"5A )Y7A> )Y7A!" A$)MM!C1H+.AII&9;KLMMM!CAI,/AII&9>JK"Q&MN AII&68HIJ!" A=$@r2rc|xstjj}|j}|j }|j j D]b}t|jjd||j_ t|jjd||j_ t|jjd||j_ t|jjd||j_ |jxj|zc_ |jxj|zc_|jj|jjz|j_|jj |jj|z|jj|dzz|j_t#|j$j&j(d|t#|j$j&j*d||j$j&j*xj,|zc_t#|j$j&j.d|||j$_ey)Nzall-to-sharded)shardingrzsharded-to-allr r)r8rinitsizerankrrrrrjrkrlrmr"r$rergrrrrrrbrr)rUrNRrs r3shardz Model.shards.,,. JJL JJLZZ&&E%1&&1A&EOO "&2&&1A&EOO "&2&&1A&EOO "&2&&1A&EOO " OO / /A 5 / OO / /A 5 /33??667 OO 0 %*OO$9$933oo99q5%EOO ! %))++557Gu U %))++557Gu U II   ' ' , , 1 ,  !!))4DE (-EII $E'r2c.|jjSrR)rr)rUs r3rz Model.layersCszz   r2c d}|S)Nc0|jdrdddSy)Nrr r#) group_sizebitsT)endswith)pathrs r3 predicatez(Model.quant_predicate..predicateIs}}X&&(!44r2r1)rUrs r3quant_predicatezModel.quant_predicateGs  r2cg}|jjD]P}|dk(r|jt"|jt |j j R|S)Nr)max_size)rr(appendrrrr%)rUcacheslts r3 make_cachezModel.make_cachePsT**((B%% gi( otyy7O7OPQ )  r2rR)r)r*r+rrTr8rrZrrrGrouprpropertyrrrr[r\s@r3rrspPYP7rxx7%N'-8BNN$8$89'-R!!r2r)rv)gZd;?g@)(rn dataclassesr functoolsrtypingrrmlx.corecorer8mlx.nnrhmlx.nn.layers.distributedrrr baser r r r}rr rope_utilsr switch_layersrrr@compiler/rNModulerPr^rrrrr1r2r3rs ! PPTT+'$   &' t$ $5 $ $% $RYY9CRYY9Cx$ryy$Nryy.%RYY%PoBIIor2