+ TV-j<>^RIHtHt^RIHtHt^RIHt^RI H t ^RI H t ^RI HtHtHtHt^RIHtHtHt^RIHt^RIHt]!R R ]44t!R R ] P64tR t!RR] P64t!RR] P64t!RR] P64t !RR] P64t!!RR] P64t"!RR] P64t#R#)) dataclassfield)ListOptionalN)swiglu) BaseModelArgscreate_attention_maskcreate_ssm_maskscaled_dot_product_attention) ArraysCache CacheListKVCache)initialize_rope) ssm_updateca]tRt^toRtRtRtRt^@tRt Rt Rt Rt R t ^tR t^t^@tR t^t^t^t^tRtRtRtR tR tRt^t]!R R7tRt ^t!^$t"^t#Rt$Rt%Rt&Rt'Rt(Rt)]!RR7t*Rt+Rt,R t-V3Rlt.Rt/Vt0R#) ModelArgsFg?g?g;f@ig{Gz?ig?g?Tiic RR.#)g H?g?rh/Volumes/fast/ai/experiments/ui-tars-smoke/.venv/lib/python3.14/site-packages/mlx_lm/models/falcon_h1.pyModelArgs.2s !3Y ?r)default_factory falcon_h1gh㈵>NgvH7Bg?c .RO#);f?)rg?r?rrrrrrr?s! rgg}+?ic6<V^8dQh/S[;R&S[;R&S[;R&S[;R&S[;R&S[;R&S[;R&S[;R&S[;R &S[;R &S[;R &S[;R &S[;R &S[;R&S[;R&S[;R&S[;R&S[;R&S[;R&S[;R&S[;R&S[;R&S[;R&S[;R&S[;R&S[;R&S[S[,;R&S[;R&S[;R&S[;R&S[;R&S[;R &S[;R!&S[;R"&S[S[,;R#&S[;R$&S[;R%&S[S[,;R&&S[;R'&S[;R(&S[;R)&#)*attention_biasattention_in_multiplierattention_out_multiplierembedding_multiplierhead_dim hidden_sizeinitializer_rangeintermediate_sizekey_multiplierlm_head_multipliermamba_chunk_sizemamba_conv_bias mamba_d_conv mamba_d_head mamba_d_ssm mamba_d_state mamba_expandmamba_n_groups mamba_n_headsmamba_norm_before_gatemamba_proj_biasmamba_rms_norm mamba_use_mlpmax_position_embeddingsmlp_biasmlp_expansion_factormlp_multipliers model_typenum_attention_headsnum_hidden_layersnum_key_value_headsprojectors_bias rms_norm_epsrope_traditional rope_scaling rope_thetassm_in_multiplierssm_multipliersssm_out_multiplier vocab_sizetie_word_embeddings)boolfloatintrstrr)format __classdict__s"r __annotate__ModelArgs.__annotate__sB #($,   3   #!$)  !"#$%&'()*!(+,!-. /012!)3456!78%[9>!?@ ABCD EF!GHIJ"KL5/(MN&OP#QR%[Sd3efgh$irr)1__name__ __module__ __qualname____firstlineno__r r!r"r#r$r%r&r'r(r)r*r+r,r-r.r/r0r1r2r3r4r5r6r7r8r9rr:r;r<r=r>r?r@rArBrCrDrErFrGrH__annotate_func____static_attributes____classdictcell__)rNs@rrrs N%(&,"3HK#!$N ) OLLKMLNM#(!O NM#)H !#(?$O"J  !OL"$(L&J##( $O!4J $irrc@aa]tRt^LtoRV3RlltRRltRtVtV;t#)FalconH1RMSNormGatedc<\SV`4\P!V34VnW nW0nW@nR#N)super__init__mxonesweightvariance_epsilonn_groupsnorm_before_gate)selfr%epsrbrc __class__s&&&&&rr]FalconH1RMSNormGated.__init__Ms1 gg{n- # 0rcVP'gVe \W!4p\PP WP VP 4pVP'dVe \W!4pV#r[)rcrr^fastrms_normr`ra)rd hidden_statesgates&&&r__call__FalconH1RMSNormGated.__call__Tsc$$$)9"47M(( ;;(=(=   T%5"47Mr)rbrcrar`)gư>Tr[ rQrRrSrTr]rmrVrW __classcell__rfrNs@@rrYrYLs1  rrYc TVPpVPVP,pVPpVVVVV.p\P !\ W@P4UUu.uF1wrV\P!\P!V4V34NK3 upp4#uuppir[) r.r1r/r2r^ concatenateziprE broadcast_toarray)argsr'groups_time_state_size num_headssizessms& rcompute_mup_vectorr~as((!0043E3EE""I  E >>E#7#78 8 OOBHHQK! .8   s%7B$ c<aa]tRt^ttoV3RltRRltRtVtV;t#)FalconH1Attentionc<\SV`4VPVnVPVnVP VnVPVnVPR,Vn\P!VPVPVP,VPR7Vn \P!VPVP VP,VPR7Vn \P!VPVP VP,VPR7Vn\P!VPVP,VPVPR7Vn\!VPVP"VP$VP&VP(4VnR#)rbiasNg)r\r]r%r<rzr> num_kv_headsr$scalennLinearr q_projk_projv_projo_projrrCrArBr7roperdrxrfs&&rr]FalconH1Attention.__init__vsW ++11 44  ]]D( ii   dnnt}}<4CVCV  ii       -$$  ii       -$$  ii NNT]] *D,<,<4CVCV  $ MM OO  ! !     ( (   rc  VPwrEpVPV4pVPV4pVPV4p VP WEVP R4P ^^^^4pVP WEVPR4P ^^^^4pV P WEVPR4P ^^^^4p VeMVPWsPR7pVPWPR7pVPW4wrM"VPV4pVPV4p\WxWVPVR7p V P ^^^^4P WER4p VPV 4#)ro)offset)maskrcache)shaperrrreshaperz transposerrrupdate_and_fetchr rr) rdxrrBL_querieskeysvaluesoutputs &&&& rrmFalconH1Attention.__call__sX''a++a.{{1~Q//!;EEaAqQ||A$"3"3R8BB1aANd&7&7<FFq!QPQR  ii i=G99T,,97D 11$?LD&ii(G99T?D- 6DJJe !!!Q1-55aB?{{6""r) r$r%rrzrrrrrrNNrprrs@@rrrts  D##rrclaa]tRt^toV3RltV3RlRltV3RlRltR V3RlRlltRtVt V;t #) FalconH1Mixerc$<\SV`4VPVnVPVnVP VnVPVnVPVn VPVn VPVnVPVP ,VnVPVnVP$VnVP(VnR\-R43VnRVnRVnVP^VP",VP ,,Vn\6P8!VP4VP4VPVPVP4R7VnVPVP4,VP,p\6P<!VPVVP>R7Vn \BPD!VP4Vn#\BPH!^VP^,4p\BPJ!V4Vn&VPNVn'VPN'd=\QVPVPVP"VPRR7Vn*\BPD!VP4Vn+\6P<!VPVPVPXR7Vn-R#)ginfgMbP?g?) in_channels out_channelsr kernel_sizegroupsr)rerbrcN).r\r]r2rzr%r/ssm_state_sizer,conv_kernel_sizer.r'r+ use_conv_biasr@layer_norm_epsilonr1ryrbr-r$r* chunk_sizerJtime_step_limit time_step_min time_step_maxconv_dimrConv1dconv1drr4in_projr^r_dt_biasarangelogA_logr5rYr3normDr?out_proj)rdrxprojection_sizeArfs&& rr]FalconH1Mixer.__init__s6 ++++"00 $ 1 1!%!1!1!11"&"3"3&*&9&9D>Qyy    %%  wwt~~. IIa!+ ,VVAY "11    ,&&++!%!>:*BKL**Q.F}}( &&q)wwu}}a<!']RYYv->> J --lANa'F78Q7a66Vd&;&;a&?%CVLLkk,/ ww{##rc<V^8dQhRS[PRS[PRS[PRS[PRS[S[,RS[S[P,RS[P/#)rrkrCdtrrrr)rMrNs"rrOrs|#F#Fxx#F 88#F 88 #F HH #F  $ #Frxx #F #Frc VPwrxp VPWxVPVP4pVPWxVPVP 4pVPWxVPVP 4pV'dV^,p VP p MRRr\VVPVVVPVVPV VPVV 4 wrV'dW^&V PWxVP4#)roN) rrrzr$rbrrrrrrrr') rdrkrrrrr batch_sizeseq_lenrstaterys &&&&&&& r_ssmFalconH1Mixer._ssms"/!4!4 Q%--   IIj4==$:M:M N IIj4==$:M:M N !HEmmG!47  JJ FF  LL       !Hyyd.D.DEErcD<V^8dQhRS[S[P,/#)rr)rr^rw)rMrNs"rrOr3s  x7I rc rVPV4p\P!VVPVPVP,.RR7wrVpVP WbV4p\P!VVPVPVP VP,,.RR7wrp VPWWW#R7p V'd#VPV P^,4VP'dVPW4p M \W\4p VPV 4#)ror)rr)rr^splitr'rrrbrradvancerr5rrr) rd input_statesrrprojected_statesrlrrrrkrrrs &&&& rrmFalconH1Mixer.__call__3s<< 5!xx   # #T%;%;dmm%K L " jjD9  hh &&&&9L9L)LL    ! IImuI @  MM!''!* %     !"AtA}}Qr)rrrrrrrryr$r%rr'rr5rbrrzrrrrrrr) rQrRrSrTr]rrrmrVrWrqrrs@@rrrs26 p $ $D#F#FJ   rrc8aa]tRtRtoV3RltRtRtVtV;t#) FalconH1MLPiSc><\SV`4VPpVPp\P !W#VP R7Vn\P !W#VP R7Vn\P !W2VP R7Vn R#)rN) r\r]r%r'rrr8 gate_projup_proj down_proj)rdrxr%r'rfs&& rr]FalconH1MLP.__init__Ush &&  22; VyydmmT #4 Vrc|\VPV4VPV44pVPV4pV#r[)rrrr)rdrrs&& rrmFalconH1MLP.__call___s2 4>>!$dll1o 6 NN1 r)rrrrprrs@@rrrSsWrrcDaa]tRtRtoV3RltV3RlRltRtVtV;t#)FalconH1DecoderLayeriec<\SV`4\V4VnVPpVP V,^VP ,V,,Vn\VR7Vn \V4Vn \P!VPVPR7Vn\P!VPVPR7VnR#)rrxreN)r\r]r feed_forwardr$r<r> channels_attnrmambar self_attnrRMSNormr%r@input_layernormpre_ff_layernorm)rdrxr$rfs&& rr]FalconH1DecoderLayer.__init__fs '-==  $ $x /$***X5 6  #- *40!zz$*:*:@Q@QR " 4+;+;ARAR Src<V^8dQhRS[PRS[S[P,RS[S[P,RS[P/#)rh attn_mask mamba_maskr)r^rwr)rMrNs"rrO!FalconH1DecoderLayer.__annotate__wsM 88BHH%  RXX&   rcTpVPV4pVPW^,VR7pVPVVV^,R7pWV,V,pTpVPV4pVP V4pWQ,#)r)rrr)rr)rrrrr)rdrrrrresidualmamba_hattn_hs&&&&& rrmFalconH1DecoderLayer.__call__ws   #**!8**M (     '  ! !! $   a |r)rrrrrrrprrs@@rrresT"rrc<aa]tRtRtoV3RltRRltRtVtV;t#) FalconH1Modelic<\SV`4WnVPVnVPVn\ P !VPVP4Vn\V4Vn \VP4Uu.uFp\V4NK upVn \ P!VPVPR7VnR#uupi)rN)r\r]rxrGr%r Embedding embed_tokensr~ _mup_vectorranger=rlayersrr@final_layernorm)rdrxrrfs&& rr]FalconH1Model.__init__s  //++LL$:J:JK-d305d6L6L0M 0M1  &0M   "zz$*:*:@Q@QR sCcFVPV4pTpVfR\VP4,.p\W2^,^,4p\ W2^,^,4p\ VPV4FwrgV!VVVVR7pK VP V4#)N)rrrr)rlenrr r rur)rdinputsrrrrlayercs&&& rrmFalconH1Model.__call__s   f %  =!C $445E$Qa 4 )!1Xa[9 DKK/HE#% A0##A&&r)rrxrrr%rrGr[rprrs@@rr r s S''rr cXaa]tRtRtoV3RltRRltRtRt]R4t Rt Vt V;t #) Modelic<\SV`4WnVPVn\ VR7VnVP 'g5\P!VPVPRR7Vn R#R#)rFrN) r\r]rxr;r modelrHrrr%rGlm_headrs&&rr]Model.__init__sY  //"- '''99T%5%5tUSDL(rc8VPWR7pVPP'd]VPPP V4pW@PP VPP , ,#VPV4#))r)rrxrHr as_linearr)r#r )rdrrrkouts&&& rrmModel.__call__sm 6 7 99 ( ( (**))33MBC))669W9WWX X<< . .rcVR,pVPR ,VP^,8:dV#/pVPpVP4EFwrVVPR4'dWdP,pEMVPR4'dWdP ,pEMVPR4'gVPR4'dWdP ,pEMsVPR4'd&WdP VP,,pEM7VPR4'dWdP,pEM VPR4'dWdP,pMVPR4'dWdP^,,pMVPR 4'dWdP^,,pMVPR 4'dPVVPVPPPVP4R ,,,pMR V9dVP!^^^4pWcV&EK V#)z"model.layers.0.mamba.conv1d.weightzembed_tokens.weightzlm_head.weightz q_proj.weightz k_proj.weightzkey_proj.weightz o_proj.weightzout_proj.weightzgate_proj.weightzdown_proj.weightzin_proj.weightz conv1d.weightrr)rrxitemsendswithr#r)r!r(r"rFr:rDrrastyperr)rdweightsc1dsanitized_weightsrxnameparams&& rsanitizeModel.sanitizes:; 99R=CIIaL (Nyy"==?KD}}233222/00000//4==3Q3Q555011558K8KKK//666011000122--a00122--a00/00**jj,,33EKK@IJ!D(1a0&+d #3+4! rc \VPP4Uu.uF!p\\ ^R7\ 44NK# up#uupi)r)size)rrxr=r r r)rdrs& r make_cacheModel.make_cachesF499667 7 kq)79 57   s'A c.VPP#r[)rr)rds&rr Model.layersszz   r)rxr rr;r[) rQrRrSrTr]rmr/r3propertyrrVrWrqrrs@@rrrs4T/#!J !!!rr)$ dataclassesrrtypingrrmlx.corecorer^mlx.nnr activationsrbaserr r r rr r r rope_utilsrssmrrModulerYr~rrrrr rrrrrBs)! 32' 3% 3% 3%l299*&<# <#~] BII] @"))$*299*Z$'BII$'N?!BII?!r