+ TV-jL0Rt^RIt^RIt^RIt^RIt^RIt^RIt^RIHt^RI H t ^RI H t H t Ht^RIt^RIHt^RIHt^RIt^RIHt^RIHt^RIHt^RIHt^R IHt^R I H!t!^R I"H#t#R t$R t%Rt&Rt'Rt(]!R4!RR]44t)Rt*R#)z9 Adapted from a PyTorch implementation by David Grangier N)version)Path)AnyCallableOptional)LM)register_model)tqdm)batch_generate)make_prompt_cache) make_sampler)loadi c\V4pVUu.uFq0PV4NK ppVUu.uF qU^8dTMTNK ppVR\V4#uupiuupi)zFLimit a string to the first occurrence of any substring in untils.N)lenfindmin)suntilslufxs&& `/Volumes/fast/ai/experiments/ui-tars-smoke/.venv/lib/python3.14/site-packages/mlx_lm/evaluate.py _rstrip_untilr!sZ AA"#FqFA#$%&Aq!e AA& Xs1v; $&s AAc^VPV4;pR8wdW\V4,R#V#)zHTruncate the prefix of the string after the first occurrence of pattern.N)rr)rpatternidxs&& r_lstripr)s1vvg2%s7|#%&& Hc \P!VUu.uFp\V4NK up4pVP4p\P!VUu.uF,p\P !V^V\V4, 34NK. up^R7p\ P!V4\ P!V43#uupiuupi)axis)nparrayrmaxstackpadmx)inputsrlengthsmaxlenpaddeds& r _pad_inputsr.0shh/1A/0G [[]F XX289&QAvA' (&9 F 88F RXXg. .. 0 :s B;2Cc aRRV3RlllpV#)Tc$V^8dQhR\/#returnstr)formats"r __annotate__&chat_template_fn..__annotate__;s  s rc T<VPP!V3RRRVRV'*/SB#)tokenizeFadd_generation_promptcontinue_final_message) tokenizerapply_chat_template)self chat_historyr; extra_kwargss&&&rr>-chat_template_fn..apply_chat_template;sF~~11   #8 (=#<     r)T)rAr>sl rchat_template_fnrD:s   rmlxlmcaa]tRt^Gto]!4tRV3RlV3RllltRV3RlRlltRV3RlRlltRt ] V3RlR l4t V3R lR lt V3R lR lt V3RlRltRtVtV;t#)MLXLMc<V^8dQhRS[RS[S[,RS[RS[S[,RS[RS[S[S[P .S[P 3,,RR/#) r2path_or_hf_repo max_tokens batch_sizeuse_chat_templatetrust_remote_codesamplerr3N)r5rintboolrr)r%)r6 __classdict__s"rr7MLXLM.__annotate__Lsr   SM   $D>    (BHH:rxx#789   rc<\SV`4RV'dRMR/p\WR7wVnVnW nW0nW@nVfVPPRJVnW`n R#)rMTN)tokenizer_config) super__init__r _modelr= _max_tokens _batch_sizerL chat_template_sampler) r?rIrJrKrLrMrNrT __class__s &&&&&&& rrVMLXLM.__init__Lsm /9JPTU&* ' # T^&%!2  $%)^^%A%A%MD " rc <V^8dQhRS[/#)r2 step_size)rO)r6rQs"rr7rRasrc\P!V4R,p\VP4p\ ^VP ^,V4FkpVPVRWDV,13,VR7p\P !VUu.uFqfPNK up4\P!4Km \P!XR,P\P44pWs3#uupi)NNNNcache)rarra) r)r%r rWrangeshapeevalstate clear_cachenn log_softmaxastypefloat32)r?promptr_rcilogitsclogprobss&&& r_process_promptMLXLM._process_promptas&!$'!$++.q&,,q/95A[[19}+<(>&"2"9"9"**"EF-s C= c6<V^8dQhRS[S[,RS[/#)r2rcr_)rrrO)r6rQs"rr7rRks**x}**rc \V4wrVRRR13,VR,rQT;'g\VP4p^p..r\^VP^,V4EFlp VRWV,13,p V P^,p VPWR7p \ P !V P\P44p \P!WRWV,1\P3,RR7R,pVRWV,13,\P!V RR78Hp\P!\P!WkV,4VR ,8VR4p\P!W4\P !4VP#V4VP#V4Wk, pEKo \P$!V^R7p\P$!V^R7pWtV3#) .Nrarbr"Fr).:NN).r!raN)r.r rWrdrerirjrkr)rltake_along_axisnewaxisargmaxwherearangerfrhappend concatenate)r?r*rcr_r+targetsoffsetscores is_greedyrninpTro log_probsscoreigs&&&& r _score_fnMLXLM._score_fnks%f- crc*F7O77*4;;7 q&,,q/95AAI --.C ! A[[[2Fv}}RZZ'@AI&&1ai-&7#CD2EAI --."))F2LLB"))FJ7'':JJBPUVB GGE  NN    R MM% KF'6*Q/NN915  ))rc VUu.uF7p\VPPW P'*R74NK9 up#uupi)add_special_tokens)tupler=encoderL)r?textsts&& r _tokenizeMLXLM._tokenizesP    %%a@V@V>>rcNVPPPRR4#)/__)r= name_or_pathreplace)r?s&rtokenizer_nameMLXLM.tokenizer_names~~**223==rcF<V^8dQhRS[S[S[S[3,,/#r1)listrfloatrP)r6rQs"rr7rRs%n>n>eE4K.@)An>rc \P!R\V4,4\PP 4p\ P!\4p\V4F?wrEW5P^,,PWEP^,34KA \VP44p.p.pVP4F0p \V !wrJVPV4VPV 4K2 WbP4RVP!41,pWrP4RVP!41,p^p ..r\#\Wg4\V4R7EFwrVP%V.4^,pTP%VUu.uF pVV,NK up4p\'RV44pVP(;'g\*p\'^VV, ^, 4p\V4p\'\V4V, ^4pV\V4V, RpV^8XdcV ^, p \,P/\1R4).\V4,4\2P/R.\V4,4EK2VP5V4wpp\P6!V4P94pVEF"pV\V4RpV PV^V^,3,P944V PV^,V8H4\V4^8XdKnVP;\P<!V4R ,\>P@!V4R7wpppV R ;;,\PB!V4P94, uu&V R ;;,\PD!V4P94,uu&EK% EK V ^8d"\P!RV R 2R ,4\V4p \PPG\V 4\PHR 7P94p!V ^.V!\V 4, ,,p V R.V!\V 4, ,,p \P<!V 4p \P<!V 4p \PPKV \PHR 7p \PPKV \PHR 7p \PL!W4.p"\OVP!44Fhp#VV#RVP!41,U$Uu.uF p$V$FqDNK K p%p$pV%V .V!\V%4, ,, p%V"P/V%4Kj \PP!\P<!V"44p&V RV V&,p V RV V&,p \\V PS4V PS444#uupiuupp$i)aCompute log-likelihood of generating a continuation from a context. Downstream tasks should attempt to use loglikelihood instead of other LM calls whenever possible. :param requests: list[Instance] A list of Instance objects, with property `args` which returns a tuple (context, continuation). `context: str` Context string. Implementations of LM must be able to handle an empty context string. `continuation: str` The continuation over which log likelihood will be calculated. If there is a word boundary, the space should be in the continuation. For example, context="hello" continuation=" world" is correct. :return: list[tuple[float, bool]] A list of pairs (logprob, isgreedy) `logprob: float` The log probability of `continuation`. `isgreedy`: Whether `continuation` would be generated by greedy sampling from `context`. z&Estimating loglikelihood for %d pairs.N)totalc38"TFp\V4xK R#5iNr).0rs& r &MLXLM.loglikelihood..s!A.Q#a&&.infFrbzPrefix eliminated for z requests with zcompletion longer than context.stream)Nrar)*logginginforr) distributedinit collections defaultdictr enumerateargsr}keysvalueszipranksizer rr&rXDEFAULT_MAX_TOKENS all_scoresextendr all_is_greedyrrrzitemrr%copydeepcopysumallall_maxcpu all_gatherrfrdargsorttolist)'r?requestsgroup group_reqsrreq questions responsesindicesvresplong_completionsrrqrsprefixrfull_sequencesmax_completed_lrJ truncation orig_prefix_lprefix_lrqrcmax_idxrr*r_r num_results per_group all_indicesrquestion rank_indicesinv_sorts'&& r loglikelihoodMLXLM.loglikelihoods(  =H MN##%!,,T2 !(+HC xx{ # * *C!+= >,*+  ""$AQIC NN3    T "% jjl:ejjl:; jjl:ejjl:;  #i33y>JJEA^^QC(+F!^^B,?BqQUUB,?@N!!A.!AAO))??-?JQ* \P!R\V4,4TPVUu.uFq"P^,NK up4p.p\ \ ^\V4VP44FpW5WPP,pVPV4wrxp \P!VPR,4VR,8p VPW,PRR7P44K V#uupi)aCompute full log-likelihood of a string, with no truncation, for perplexity computation - We will use the full max context length of the model. - For inputs that exceed the max context length, we divide the tokenized string into chunks of up to the max context length. - IMPORTANT: Each document's loglikelihood/perplexity is computed *separately*, unlike other implementations which may simply concatenate multiple documents together. - IMPORTANT: We maximize the amount of context for each prediction. Specifically, for inputs that we break into multiple chunks, the last input will still a full-sized context. Example: Input tokens: [ 0 1 2 3 4 5 6 7 8 9 ] Prefix: EOT Max context length: 4 Resulting input/prediction pairs: INPUT: EOT 0 1 2 PRED: 0 1 2 3 INPUT: 3 4 5 6 PRED: 4 5 6 7 INPUT: 5 6 7 8 PRED: 8 9 Observe that: 1. Each token is predicted exactly once 2. For the last pair, we provide the full context, but only score the last two tokens :param requests: list[Instance] A list of Instance objects with property `args` which returns a tuple (context,). string: str String for which we are computing overall loglikelihood :return: list[tuple[float]] A list of tuples (logprob,) logprob: float The log probability of `context` conditioned on the EOT token. z2Estimating loglikelihood rolling for %d sequences.r"rrw)rrrrrr rdrYrr)r|rerrr) r?rrr*rrnbatchrr+rmasks && rloglikelihood_rollingMLXLM.loglikelihood_rollings@  @3x= P  A! AB eAs6{D,<,<=>Aq#3#334E!%!6 FQ99V\\"-.1AAD   t}11r1:AAC D ? !BsDc0<V^8dQhRS[S[,/#r1)rr5)r6rQs"rr7rR5sUU$s)Urc  * \PP4p\V4pWP 4RVP 41,p\ P!R\V4,4\VUu.uFqDPNK up!wrVVUu.uF.pVPPWpP'*R7NK0 ppVUu.uF-pVP;'gVPR\4NK/ p p\!VP"VPVV RVP$R7P&p \)\W44F[wp wr\+WR,4W&VPP,'gK:\/WPP04W&K] VP 4^8Ed\P2!\P44;_uu_4W2P 4,^, VP 4,p V \V 4, pV Uu.uFp\7VPR44NK p p\P8!\;R V 444p\PP=V4P?4p\P8!V Uu.uFp\V4NK up^.V,,4p\P8!V Uu.uF#q^.V\V4, ,,NK% up^.V,.V,,\P@4p \PPCV R,4PE^^4PG^^4PI4p \PPCVR,4PE^^4PG^^4PI4pV RVp VRVp\V V4UUu.uF"wpp\KVRV4PM4NK$ p ppRRR4V #V #uupiuupiuupiuupiuupiuupiuuppi +'giT #;i) aGenerate greedily until a stopping sequence :param requests: list[Instance] A list of Instance objects with property `args` which returns a tuple (context, until). context: str Context string until: [str] The string sequences to generate until. These string sequences may each span across multiple tokens, or may be part of one token. :return: list[str] A list of strings continuation continuation: str The generated continuation. Nz)Generating continuation for %d sequences.rmax_gen_tokensT)modelr=promptsrJverboserNuntilzutf-8c38"TFp\V4xK R#5irr)rrps& rr'MLXLM.generate_until..ps&C{!s1vv{r)'r)rrrrrrrrrr=rrLrXgetrr rWr[rrr has_thinkingr think_endrrrr%r&rruint8rswapaxesflattenr bytearraydecode)r?rrtotal_requestsrcontextsoptionscontextoptrJ completionsetextpad_tor(rpmax_lenr+rs&& rgenerate_untilMLXLM.generate_until5s##%XJJL8EJJL89 @3x=PQh!?hs((h!?@$  $ NN ! !0F0F,F " $       M M(8:L M M  %++nn!MM   % (K(ABNA{*4W>KN~~***!(~~/G/G!H C ::>@((K#@KqCFK#@A39#LM hh;FG;a!#a& 0111;GsW}o+,HH NN--k$.?@Xa^WQ]VX NN--gdm<Xa^WQ]VX  */>: !/>2:=k7:S:S$!QIae$++-:S5#<{A"@   0M$AG$5#"<s]Q4Q"Q'1Q'rVrrrrpropertyrrrr__static_attributes____classdictcell__ __classcell__)r\rQs@@rrGrGGsr+,  ***B >>n>n>`++ZUUUrrGc \P!R4pVPRRRR7VPRRRR7VPRR R R 7VPR \^R R7VPR\RRR7VPR\RRR7VPRRR\R7VPR\^{RR7VPRRRRR7VPR\PR RR7VPR!\ P R"R#R7VPR$RR%RR7VPR&RR'R(7VPR)\R*R+R7VPR,\R-R.R7VPR/\^R0R7VP4p\VP4pVPRRR17R2\PR3&\PP!VP 4\P"P%4p\P&!\P"P)^\P*R474VP-4^8d3VP/4^8Xd\1R5VP-4 R624\3VP4VP6VP8R77p\;VP<VP>VP@VPBVPDVR87p\GRB/VPHB\:n!\JPL!VVPNVPPVPRVPTVPVVP VP VP VP VPXR97 pR:VP<P[R;R<4\]R=4.pVPTeWqPTR> ., pWqPN, pR num_fewshotlimit random_seednumpy_random_seedtorch_random_seedfewshot_random_seedconfirm_run_unsafe_coderfrrlm_eval02dresults)indentzResults:rC)3argparseArgumentParser add_argumentrOBooleanOptionalActionjsonloadsr parse_argsr output_dirmkdirosenvironr)randomseedrrrfall_sumrrrprintr rrrrGrrJrKr>rMrDchat_template_argsr)simple_evaluater r!rL num_shotsr#r(rrjoin write_textdumpsr) parserrr4worldrNlmr+ file_keysfilename output_pathresults rmainrIs9  $ $<F  (;dK  t< *N S"<P  CDUV  3   5    sCnM  .  --   ZZA  # N    8  uc@VW  sAQR  Q=MN    Ddoo&JTD1,3BJJ'(IINN499 NN   !EGGBNN " "1RVV " 45 zz|aEJJLA-  f56 YYjjjjG  ????2200  B!1 K43J3J KE%%jj!6600NNjjII)))) II $ < < G++C5wy7IJI ~~!,..  Ixx "H zz|q + tzz')*rXs  &** /$+& / BBBBJ t0r