+ UV-jzB^RIt^RIt^RIt^RIt^RIt^RIt^RIHt^RIH t ^RI H t ^RI H t ^RI Ht.ROt.ROtRtRR ltR tR tR tR tRt]R8Xd ]!4R#R#)N)dump) load_dataset)tqdm)load) inferencec\\V4P4PRR44# \d\T4P4u#i;i)z)Normalize numeric strings for comparison.,)floatstrstripreplace Exception)ss&c/Volumes/fast/ai/experiments/ui-tars-smoke/.venv/lib/python3.14/site-packages/mlx_vlm/evals/mmmu.pynormalize_numberr8sHSV\\^++C455 1v||~s14$AAc0V^8dQhR\R\/#)data eval_file)listr )formats"r __annotate__r@sVVDVSVc  /p/p^p^pVEFp\VR,4p\VR,4p\VPRR44p W9d ^W)&^W9&W9;;,^, uu&V^, pVP4P4p VP4P4p Rp VR.9d.R/Op RpR0pV Fawpp\P !VV \P 4pV'gK6VV8gK?V^,P4pTpV^ 8gKaM V'd W8XdRp EM*V'g/\V 4^8dV ^,R9dV ^,V 8XdRp MW9dRp MVPR R 4PR R 4PR R 4P4'dR\P !R V4p\V4pVF(p\\V4V, 4R8dRp MXK* MS\V P44p\V P44pV'dVPV4'dRp V 'd&V^, pW);;,^, uu&^VR&EK^VR&EK /pV^8d\!V4\!V4, MRVR&VVR&VVR&\#VP%44F^p W9,^8gK\!W),4\!W9,4, VRV R2&W),VRV R2&W9,VRV R2&K` \'R4\'R14\'RVPR4R0,PR4^, 24\'RV 24\'RV 24\'RVR,R R V RV R!24\'R14\'R"4\#VP%44F^p VPRV R2R4pVPRV R2^4pVPRV R2^4p\'R#V R$VR R V RV R!2 4K` \'R14VPR%R&4p\)VR'4;_uu_4p\+VV^R(7RRR4\)VR'R R)R*7;_uu_4pV'dO\,P.!VV^,P%4R+7pVP14VP3V4RRR4\4P6!R,V R-V 24R# \dEKji;i +'giL;i +'giLT;i)2z` Evaluate MMMU results by subject. Handles both multiple choice (A-F) and open-ended questions. predictionanswersubjectUnknownFNTabcdef.r -r z -?\d+\.?\d*g{Gz?scoreoverall_accuracy total_correcttotal_questionssubject_ _accuracy_correct_totalz MMMU Evaluation Results:zModel: /_MMMU_zTotal Questions: zTotal Correct: zOverall Accuracy: z.4fz ()zSubject Breakdown:z : z.csvz _score.jsonw)indentutf-8newlineencoding fieldnamesz+MMMU_eval successfully finished evaluating z, results saved in )ABCDEFI))zoption\s+([a-f])\b )zanswer\s+is:?\s+([a-f])\br?)zchoice\s+is:?\s+([a-f])\br?)z#correct\s+answer\s+is:?\s+([a-f])\br?)z'correct\s+option\s+is:?\s+\(?([a-f])\)?r?)z \(([a-f])\))z^([a-f])[.:\)]\sr@)z \b([a-f])\bP================================================================================)r getlowerr refindall IGNORECASElenrisdigitrabsrsetsplitissubsetr sortedkeysprintopenrcsv DictWriter writeheader writerowslogginginfo)rrsubject_scoressubject_countersr&r'linepredictrr predict_lower answer_lower is_correctpatterns best_match best_prioritypatternprioritymatchesnumbers answer_numnum_str answer_words predict_wordsresultsacccorrecttotal score_pthfwriters&& r MMMU_evalrr@s.NMOd<()T(^$dhhy)45  (&'N #()  % !Q&!1 --/ ||~++-   8 8 HJM&.!**WmR]]K7x-7!(!1!1!3J$,M2~&.j8!  &*!!$0 #|3!%J ,! R(00b9AA#rJRRTT**^W=-f5 &G/8:EFM)-J!N ' #<#5#5#78 #M$7$7$9 : L$9$9-$H$H!%J  Q M  #q ( #DMDMotG9H19Lmu_55RU  -GO!0G .--/0  $q (5:'6&/061Ghwiy1 25C4KGhwix0 12B2KGhwiv. / 1 &' (O GIOOC(,228 hwiv6: 7)2c#Yb 5';< 1  (O!!&-8I i   Wa"  ib7 ; ;q ^^A$q',,.AF      T " <  LL 5i[@ST]S^_}%j   < ;s+-"UUAU* UU U' * U: cVPRR4pVPRR4pV'd(\P!RRV4PR4MRpV'dU\ V\ 4'd?VR, p\ V4F&wr4\^AV,4pVRV R V 2, pK( \P!R RV4P4pV#) zZ Process MMMU question to format it properly. MMMU questions may have options and images. questionr optionsNz [\[\]"\'], z Options: . z ) rDrFsubrM isinstancer enumeratechrr )examplertruioptionletters& rprocess_questionrs {{:r*Hkk)T*G?Fbff\2w/55d;DG:gt,,N""7+IAa[F "VHBvh/ /H, vvnb(399;H Orc.pRV9d8VR,e-VR,PR4pVPV4V#\ ^^4FEpRV 2pWP9gKW,fKW,PR4pVPV4KG V# \dp\RT 24Rp?T#Rp?ii;i \dp\RT RT 24Rp?KRp?ii;i)zO Extract images from MMMU example. MMMU can have multiple images per question. imageNRGBz#Warning: Could not process image - image_z)Warning: Could not process image for key z - )convertappendrrQrange)r}imagesimger~img_keys& r get_imagesrs F'gg.: ='"**51C MM#  Mq!AqclG!g&6&B!*2259CMM#&  M = 7s; < < M =!EgYcRSQTUVs/)B*(B? B<#B77B<? C& C!!C&cH\R4\R4\R4\\^4Fwr\VR RV 24K \R4\R4\R4\\^4Fwr\VR RV 24K \R4R#) z"Print all available MMMU subjects.zMMMU Pro Subjects (3 total)2drxz"MMMU Available Subjects (30 total)NrCzQ ================================================================================zQ================================================================================ )rQr{MMMU_PRO_SUBJECTS MMMU_SUBJECTS)r~rs r list_subjectsr s / '( (O 115  2b "#6 / ./ (O q1  2b "#2 /rc \P!RRR7pVPR\RRR7VPR\RR R7VPR \R R R7VPR \RRR7VPR\RRRP \ R,4 R2R7VPRRRR7VPR\ RRR7VPR\RRR7VPR\R R!R7VPR"\R#R$R7VPR%\ ^RR&R'7VPR(RR)R7VPR*\ RR+R7VPR,RR-R7VPR.\RR/R7VPR0\R1R2R7VPR3\ ^*R4R7VP4#)5zCMMMU Evaluation - Massive Multi-discipline Multimodal UnderstandingzQUse --subset to evaluate a specific subject, or omit to evaluate all 30 subjects.) descriptionepilogz--modelz'mlx-community/Qwen2-VL-2B-Instruct-bf16z Model path)typedefaulthelpz--adapter_pathNz Adapter pathz --datasetz MMMU/MMMUz Dataset pathz--split validationzSplit to use for evaluationz--subsetz$Subset to use - one of 30 subjects: rvNrANz#... (see SUBJECTS.md for full list)z --streaming store_truezUse streaming dataset loading)actionrz --max-tokensi z$Maximum number of tokens to generatez --temperaturer$z)Temperature for sampling (0.0 for greedy)z--top-pg?zTop-p sampling parameterz--repetition-penaltyg?zRepetition penalty parameterz--resize-shapezResize shape for the image)rnargsrrz --verbosezVerbose outputz --max-samplesz3Maximum number of samples to evaluate (for testing)z--list-subjectsz'List all 30 available subjects and exitz--prediction-filezPath to the prediction filez --output-dirz results/mmmuz$Directory to save evaluation resultsz--seedz Random seed) argparseArgumentParser add_argumentr joinrintr parse_args)parsers rparse_argumentsrsE  $ $YbF  9    (sD~V  #{X \8U  3DIImB>O4P3QQt u   l1P   3     8     '     +     )   L?OP   B    6    *     3   sB]K    rc\4p\P!VP4\P!VP 'd\P M\PRR7RVPP49d\pM\pVP'd\P!RVP R24.p\VPRRR7;_uu_4p\P !V4pVFpVP#V4K RRR4\%W P4\P!R 4R#VP&'d \'4R#\P!R 4VP('dwVP(V9df\P*!R VP( R 24\P*!R RP-V4 R24\P*!R4R#\P!RVP R24VP('du\P!RVP( R24VP(\/VPVP(VP0VP2R7/pVP(pMS\P!R4/pVF3p\/VPVVP0VP2R7Wh&K5 RpVP6'd|VP94U U u/uF<wrWP;\=\?VP6\AV 4444bK> pp p \P!R\AV4 R24\P!R\AVPC44 R24\P!RVPD R24\GVPDVPHRR7wrV PJp\P!RV R24VPDP1R4RA,pVPL RV R V R!VP0 R"2p\NPP!VPLRR#7.p\SVP94R$R%7EF^wpp\U\SVR&V 2R%74EF9wpp\WV4p\YV4p\[V V VVVP\VP^VP`VP 4pR*VPeR*V4R+TR,VPeR,R)4R-VPeR-R.4R/VPeR/R.4R0VPeR0R.4R1TR2VPeR2R4;'gT/pVP#V4V^,^ ,^8Xg V^8gEK\P!R3V^, R4VR,, R5VR6, R724EK< EKa \cR84\UVR9,4FGwpp\cV^, R:VR+,R6, R;VR,, R7;_uu_4pV'dN\Pf!W2^,PC4R?7pVPi4VPkV4RRR4\P!R@V R24\%VV4\P!R 4R# +'giEL$;i \4d*p \P*!RT RT R24Rp ? EKRp ? ii;iuup p i \4dp \cR'TTR(T 4R)pRp ? ELyRp ? ii;i +'giL;i)Bz)%(asctime)s - %(levelname)s - %(message)s)levelrprozLoading predictions from zrr2)r5NzEvaluation completez!Starting MMMU EvaluationzError: Invalid subset 'z'zValid subjects are: rvz)See SUBJECTS.md for more detailszLoading dataset from zUsing subset: )rM streamingz#Evaluating all 30 subjects)namerMrzError loading dataset for r/allzLimited to z samples for testingzDataset subset size: zLoading model from T) adapter_pathtrust_remote_codez Config: r,r-_z_predictions.csv)exist_okzProcessing subjects)descz Processing zError during inference:zerror message:r idrtrsubfieldrtopic_difficulty question_typerrzSample z : Answer=z , Prediction=:N2Nz...z First 5 results:rz . Question: z... | Answer: z | Prediction: r0r3r6zSaved results to rB)6rrandomseedrW basicConfigverboseINFOWARNINGdatasetrErrprediction_filerXrRrS DictReaderrrrrsubseterrorrrrMrr max_samplesitemsselectrminrIrPmodelrrconfig output_dirosmakedirsrr{rrr max_tokens temperature resize_shaperQrDrTrUrV)argssubjectsrkrpreaderrowdatasets subset_namerrkvr processorr model_name result_fileridxr}rtrrresultr~rqs rmainrnsk  D KK  "lllgll:   ""$$$   89M9M8NgVW $&&g > >!^^A&Fs#? '//0 9;  LL:; {{{t{{(2 7 }HMN 4TYYx5H4IQR GI LL0gFG {{{ -dkk]'BC KK dkkt~~  kk  @BG $0LL **"nn %!   !( ( xxc$"2"2CF;<= =(    *3x=/9TUV LL0X]]_1E0FgNO LL.tzzl'BC !2!2dE\\F LL#F8734!!#&r*J__%Qzl& QtzzlRbcKKK$/G !18MNN%d7;wi9P&QRLC'0H(F &OO$$%%LL  gkk$,H'++h3GKK I>"GKK0BI$N_i!Hj7;;y$7BB7 F NN6 "a2~"cAg cAgYix0@/AzZ]N__bcGSOR wr{+ 6 se<z 23 78vhGWFXXghno{h|~AiBhCCF G , k3W = = ^^A!*//2CDF      W % >  LL,[MABg{# LL57U? > >`  8 A3gN   L /(FDTVWX  > > =sP71]!/];A^:^'A^> ]  ^ "^^  ^;^66^;> _ __main__) Accounting AgricultureArchitecture_and_EngineeringArt Art_TheoryBasic_Medical_ScienceBiology ChemistryClinical_MedicineComputer_ScienceDesign#Diagnostics_and_Laboratory_Medicine Economics ElectronicsEnergy_and_PowerFinance GeographyHistory LiteratureManage Marketing MaterialsMathMechanical_EngineeringMusicPharmacyPhysics Psychology Public_Health Sociology)visionzstandard (10 options)zstandard (4 options))rrSrWrrrFjsonrrrrmlx_vlmrmlx_vlm.evals.utilsrrrrrrrrrrr__name__rrrss  !) BVr.6 Qh^8B zFr