i<>ddlmZmZddlmZmZddlmZddl m Z ddl m Z ddl mZmZmZmZddlmZmZmZddlmZdd lmZeGd d eZGd d e j6ZdZGdde j6ZGdde j6ZGdde j6Z Gdde j6Z!Gdde j6Z"Gdde j6Z#y)) dataclassfield)ListOptionalN)swiglu) BaseModelArgscreate_attention_maskcreate_ssm_maskscaled_dot_product_attention) ArraysCache CacheListKVCache)initialize_rope) ssm_updatec~eZdZUdZeed<dZeed<dZeed<dZ eed<d Z e ed <d Z e ed <d Z eed<dZe ed<dZeed<dZeed<dZe ed<dZeed<dZe ed<d Ze ed<dZe ed<dZe ed<dZe ed <d!Ze ed"<d#Ze ed$<dZeed%<dZeed&<dZeed'<dZeed(<d)Ze ed*<dZeed+<d,Z e ed-<e!d./Z"e#eed0<d1Z$e%ed2<d,Z&e ed3<d4Z'e ed5<dZ(e ed6<dZ)eed7<d8Z*eed9<dZ+eed:<d;Z,e-eed<<d=Z.eed><d?Z/eed@<e!dA/Z0e#eedB<dCZ1eedD<dEZ2e edF<dZ3eedG<y;)H ModelArgsFattention_biasg?attention_in_multiplierg?attention_out_multiplierg;f@embedding_multiplier@head_dimi hidden_sizeg{Gz?initializer_rangeiintermediate_sizeg?key_multiplierg?lm_head_multipliermamba_chunk_sizeTmamba_conv_bias mamba_d_conv mamba_d_headi mamba_d_ssm mamba_d_state mamba_expandrmamba_n_groups mamba_n_headsmamba_norm_before_gatemamba_proj_biasmamba_rms_norm mamba_use_mlpimax_position_embeddingsmlp_biasmlp_expansion_factorc ddgS)Ng H?g?r5l/Volumes/fast/ai/experiments/voice-extract-mac/.venv/lib/python3.12/site-packages/mlx_lm/models/falcon_h1.pyzModelArgs.2s !3Y ?r6)default_factorymlp_multipliers falcon_h1 model_typenum_attention_heads$num_hidden_layersnum_key_value_headsprojectors_biasgh㈵> rms_norm_epsrope_traditionalN rope_scalinggvH7B rope_thetag?ssm_in_multiplierc gdS)N);f?g?rHg?rHr5r5r6r7r8zModelArgs.?s! r6ssm_multipliersgg}+?ssm_out_multiplieri vocab_sizetie_word_embeddings)4__name__ __module__ __qualname__rbool__annotations__rfloatrrrintrrrrrr r!r#r$r%r&r(r)r+r,r-r.r/r0r1r3rr:rr<strr=r?r@rArBrCrDrrErFrIrJrKrLr5r6r7rrs ND %(U(&,e,"3%3HcK#u#!s!$NE$ ))c OT L#L#KM3L#NCM3#(D(!OT! ND M4#)S)Hd !#!#(?$OT%["J!  s  !OT!L%"d"$(L(5/(&J&#u##( $OT%[!43J $$r6rc(eZdZdfd ZddZxZS)FalconH1RMSNormGatedct|tj|f|_||_||_||_yN)super__init__mxonesweightvariance_epsilonn_groupsnorm_before_gate)selfrepsr_r` __class__s r7rZzFalconH1RMSNormGated.__init__Ms7 gg{n- #  0r6c|js| t||}tjj ||j |j }|jr| t||}|SrX)r`rr[fastrms_normr]r^)ra hidden_statesgates r7__call__zFalconH1RMSNormGated.__call__Tsc$$)9"47M(( 4;;(=(=   T%5"47Mr6)gư>rTrXrMrNrOrZri __classcell__rcs@r7rVrVLs 1 r6rVc F|j}|j|jz}|j}|||||g}t j t ||jDcgc]/\}}t jt j||f1c}}Scc}}wrX) r%r)r&r+r[ concatenateziprI broadcast_toarray)argsrgroups_time_state_size num_headssizessms r7compute_mup_vectorrxas((!0043E3EE""I  E >>E4#7#78 81 OOBHHQK! .8   s!4B c&eZdZfdZddZxZS)FalconH1Attentionct||j|_|j|_|j |_|j|_|jdz|_tj|j|j|jz|j|_ tj|j|j |jz|j|_ tj|j|j |jz|j|_tj|j|jz|j|j|_t!|j|j"|j$|j&|j(|_y)Ng࿩bias)rYrZrr=rtr@ num_kv_headsrscalennLinearrq_projk_projv_projo_projrrErCrDr0roperarrrcs r7rZzFalconH1Attention.__init__vsW ++11 44  ]]D( ii   dnnt}}<4CVCV  ii       -$$  ii       -$$  ii NNT]] *D,<,<4CVCV  $ MM OO  ! !     ( (   r6c2|j\}}}|j|}|j|}|j|} |j |||j dj dddd}|j |||jdj dddd}| j |||jdj dddd} |P|j||j}|j||j}|j|| \}} n"|j|}|j|}t||| ||j|} | j ddddj ||d} |j| S)Nrr'r)offset)maskrcache)shaperrrreshapert transposer~rrupdate_and_fetchr rr) raxrrBL_querieskeysvaluesoutputs r7rizFalconH1Attention.__call__sl''1a++a.{{1~Q//!Q;EEaAqQ||Aq$"3"3R8BB1aAN1d&7&7<FFq!QPQR  ii i=G99T%,,97D 11$?LD&ii(G99T?D- T6DJJe !!!Q1-55aB?{{6""r6NNrjrls@r7rzrzts  D#r6rzcNeZdZfdZdej deedeej dej fdZdej dej d ej d ej deedeej dej fd Z d deej fd Z xZ S) FalconH1Mixerct||j|_|j|_|j |_|j|_|j|_ |j|_ |j|_|j|j z|_|j|_|j$|_|j(|_dt-df|_d|_d|_|jd|j"z|j zz|_t7j8|j4|j4|j|j|j4|_|j|j4z|jz}t7j<|j||j>|_ tCjD|j|_#tCjHd|jdz}tCjJ||_&|jN|_'|jNr>Qyy    %%  wwt~~. IIa!+ ,VVAY "11   ,&&++!%!>:z*BKL**Q.F}}( &&q)wwu}}aV<!!T']RYYv->> J --lIANa'F78Q7a66Vd&;&;a&?%CVLLkk,/ ww{##r6rgrCdtc  |j\}}} |j|||j|j}|j|||j|j }|j|||j|j }|r|d} |j } nd\} } t||j|||j||j| |j|| \} } |r| |d<| j|||jS)Nrr) rrrtrr_rrrrrrrr) rargrrrrr batch_sizeseq_lenrstaterys r7_ssmzFalconH1Mixer._ssms"/!4!4 GQ%--   IIj'4==$:M:M N IIj'4==$:M:M N !HEmmG'NE7  JJ FF  LL      5 E!HyyWd.D.DEEr6cF|j|}tj||j|j|jzgd\}}}|j |||}tj||j|j|j |jzzgd\} } } |j| | | |||} |r|j| jd|jr|j| |} n t|| } |j| S)Nrr)rr)rr[splitrrrr_rradvancerr.rrr) ra input_statesrrprojected_statesrhrrrrgrrrs r7rizFalconH1Mixer.__call__3s << 5!xx   # #T%;%;dmm%K L j" jjUD9  hh &&&&9L9L)LL    q! IImQ2u4I @  MM!''!* %    !T"AtQA}}Qr6r) rMrNrOrZr[rqrr rrrirkrls@r7rrs6 p $HH $ $ $rxx $  $D#Fxx#F 88#F 88 #F HH #F  $ #Frxx #F #FJ x7I r6rc$eZdZfdZdZxZS) FalconH1MLPc<t||j}|j}t j |||j |_t j |||j |_t j |||j |_ y)Nr|) rYrZrrrrr1 gate_projup_proj down_proj)rarrrrrcs r7rZzFalconH1MLP.__init__Usp &&  22;0A Vyy.?dmmT #4k Vr6c|t|j||j|}|j|}|SrX)rrrr)rarrs r7rizFalconH1MLP.__call___s2 4>>!$dll1o 6 NN1 r6rjrls@r7rrSsWr6rceZdZfdZdej deej deej dej fdZxZS)FalconH1DecoderLayerct|t||_|j}|j |zd|j z|zz|_t||_ t||_ tj|j|j|_tj|j|j|_y)Nr'rrrb)rYrZr feed_forwardrr=r@ channels_attnrmambarz self_attnrRMSNormrrBinput_layernormpre_ff_layernorm)rarrrrcs r7rZzFalconH1DecoderLayer.__init__fs '-==  $ $x /$***X5 6  #- *40!zz$*:*:@Q@QR " 4+;+;ARAR Sr6h attn_mask mamba_maskrc|}|j|}|j||d|}|j|||d}||z|z}|}|j|}|j |}||zS)Nr)rrrr)rr)rrrrr)rarrrrresidualmamba_hattn_hs r7rizFalconH1DecoderLayer.__call__ws   #**!58**M (   w  '  ! !! $   a !|r6) rMrNrOrZr[rqrrirkrls@r7rresNT" 88BHH%  RXX&   r6rc&eZdZfdZddZxZS) FalconH1Modelct|||_|j|_|j|_t j |j|j|_t||_ t|jDcgc] }t|c}|_ t j|j|j|_ycc}w)Nr)rYrZrrrKrr Embedding embed_tokensrx _mup_vectorranger?rlayersrrBfinal_layernorm)rarrrrcs r7rZzFalconH1Model.__init__s  //++LL$:J:JK-d305d6L6L0M 0M1  &0M   "zz$*:*:@Q@QR sCc|j|}|}|dt|jzg}t||dd}t ||dd}t |j|D]\}}|||||}|j |S)Nrrr)rrr)rlenrr r ror)rainputsrrrrlayercs r7rizFalconH1Model.__call__s   f %  =!C $445E$Qa 4 )!U1Xa[9 DKK/HE1#% A0##A&&r6rXrjrls@r7rrs S'r6rcBeZdZfdZddZdZdZedZxZ S)Modelct|||_|j|_t ||_|j s2tj|j|jd|_ yy)NrFr|) rYrZrrr<rmodelrLrrrrKlm_headrs r7rZzModel.__init__sX  //"- ''99T%5%5tUSDL(r6c |j||}|jjrU|jjj |}||jj |jj z zS|j|S)N)r)rrrrLr as_linearrrr)rarrrgouts r7rizModel.__call__sn 6 7 99 ( (**))33MBC$))669W9WWX X<< . .r6c|d}|jd|jdkr|Si}|j}|jD]\}}|jdr||jz}n||jdr||j z}nZ|jds|jdr||j z}n'|jdr||j |jzz}n|jd r||jz}n|jd r||jz}n|jd r||jd z}n|jd r||jdz}no|jdrG||j|jjj|jdddfzz}nd|vr|j!d dd}|||<|S)Nz"model.layers.0.mamba.conv1d.weightrrzembed_tokens.weightzlm_head.weightz q_proj.weightz k_proj.weightzkey_proj.weightz o_proj.weightzout_proj.weightzgate_proj.weightrzdown_proj.weightzin_proj.weightz conv1d.weightr')rrritemsendswithrrrrrrJr:rFrrastyperr)raweightsc1dsanitized_weightsrrnameparams r7sanitizezModel.sanitizes:; 99R=CIIaL (Nyy"==?KD%}}23222/0000/4==3Q55501558K8KKK/6660100012--a0012--a00/0**jj,,33EKK@DIJ!D(1a0&+ d #3+4! r6ct|jjDcgc] }tt dt "c}Scc}w)Nr')size)rrrr?rr r)rars r7 make_cachezModel.make_cachesF499667 7 kq)79 57   s%A c.|jjSrX)rr)ras r7rz Model.layersszz   r6rX) rMrNrOrZrirrpropertyrrkrls@r7rrs-T/#!J !!r6r)$ dataclassesrrtypingrrmlx.corecorer[mlx.nnr activationsrbaser r r r rr rr rope_utilsrssmrrModulerVrxrzrrrrrr5r6r7r#s)! 32' 3% 3% 3%l299*&<# <#~] BII] @"))$*299*Z$'BII$'N?!BII?!r6