@ic9UdZddlmZddlmZmZddlZddlmZddl Z ddl m Z ddl Z ddlZddlZddlZddlmZmZmZdd lmZmZmZdd lmZdd lmZmZmZdd lm Z m!Z!dd l"m#Z#m$Z$m%Z%ddl&Z'eedZe jPe)gdfe*d<dZ+eGddZ,dZ-de j\defdZ/ede,fdedede)dede j`ede,de jbe fdZ2dZ3de jbe)dede jbe de jbe fd Z4de j\deded!e jbe d"e jPde5f d#Z6y)$z This defines the `dora grid` action, and provides a `run_grid` API that can be used from a notebook or any other script. When using the API, you can provide the equivalent of the command line flags with the `RunGridArgs` dataclass. )ProcessPoolExecutor) dataclassfieldN)partial)Path) SlurmConfig SubmitRulesupdate_from_args)ExplorerLauncherHerd) DecoratedMain)colorize simple_logfatal)SheepShepherd)import_or_fatalreliable_rmtreetry_loadzGrid:logcyN)argskwargss ^/Volumes/fast/ai/experiments/voice-extract-mac/.venv/lib/python3.12/site-packages/dora/grid.pyno_printr%sceZdZUdZeeZeje e d<dZ e e d<dZ ee d<dZej ee d <d Ze e d <d Ze e d <d Ze e d <d Ze e d<d Ze e d<d Zej e e d<d Ze e d<dZej ee d<dZej ee d<dZej ee d<d Ze e d<y) RunGridArgsay Arguments to tune the behavior of the `run_grid` function. Args: patterns (list[str]): List of patterns used to filter by name the XPs. monitor (bool): if True, will monitor the advances of the XPs every `interval` minutes, stopping only when all runs completed or failed. interval (float): interval in minutes to wait between updates. trim (int or None): if provided, will trim all XP logs to the epoch of the XP with the provided index. Useful to compare XP started at different times. trim_last (bool): if True, will trim all XP to the least advanced XP. silent (bool): if True, do not print anything (e.g. API usage). dry_run (bool): if True, Dora will simulate the run of the grid, without scheduling or canceling any XP. cancel (bool): if True, will cancel all XPs in the grid. If `patterns` is provided, only XP matching the patterns will be canceled. clear (bool): This will cancel any previous job, clear the XP folder, and reschedule a new experiment. )default_factorypatternsTmonitorintervalNtrimF trim_lastsilentdry_runcancelclearinitjupyterfolderrtail_from_commandline)__name__ __module__ __qualname____doc__rlistr$tpListstr__annotations__r%boolr'floatr(Optionalintr)r*r+r,r-r.r/r0rr1r2rr rr"r")s.#48Hbggcl8GTHe!D"++c !ItFDGTFDE4#D"++d #GT $FBKK # CS  !D"++c !#t#r r"c|jj}||jdz}t|}|jA|jj dddz}t |jj|z }|jjs>g}t |jj}tj|D]\}} } | D]} t || z j|} | jjds<| jjdrX| j| j z } |j#t%| j dd|j"jst'dd|dt'd d j)|t+j,d |dz|jz} t| } |j.}t5t6st3|d |S#t0$rt3| d Y9wxYw)Nz.grids./z.py_z No grid file z in package z. Maybe you made a typo?zPotential grids are: z, rz( has no exploration function `explorer`.z' must be an instance of `dora.Explorer`)dora grid_packagepackagergridreplacer__file__parentexistsoswalk relative_tonameendswith startswithstemappendr:rjoinsysexitexplorerAttributeErrorr isinstancer )rmainrEgrids grid_filename grid_file candidatespkg_rootrootfoldersfilesfilefullpath grid_namerGrWs r _get_explorerf[s99))L||h. L )E yy ))#s3e; (//-?  yy 0 0 2 '..$&GGH$5 D'5 J-::8D==))%09Q9QRU9V'>H%%c(m&;&;C&EF %6 99 )9)9); - l<.I)) * #DIIj$9#: ;<  s"TYY.I 9 %DF== h )  ABC O F  CDEFs9 H%%H?>H?rrZct||}|j}t||t}t||t }d|_t||t |||j|||y)NT)rfget_slurm_configr r r"r2run_gridrG)rrZrWslurmrules grid_argss r grid_actionrmsfD$'H  ! ! #EUD! MEUD! I"&IY% T8TYYui@r rWrerkrjreturnc t|tsJ||j}|jj|jj z |z }|j ddt}t|t}|jrjArtC| j<j>d| _Ng}g}|jED]]}|jF|j vs|jI| |jK|jF}|J|jI|_|j|jVs| D cic]} | j<jX| }} |jZD]}|Dcgc] }||vs||}}|s|d}|j\|j<jX}t/|dk(r|j_|||g|ja|5|D]} |j_| || ddd|D]}|jcr|j6J|j9|j6|je|j<}td|j6jTd|j<jXd||jVr| D]} | jcr| j6J|je| j<}td| j6jTd| j<jXd||j9| j6|j*s| D]} || j<jXz }|jAs|jgrH|jgr6|ji| j<j>jik(rJ|jk| j<j>|j;|D]}|jm|jnr"| D]} |jq| j<|jVr| S| s td| S|j>|j|jrfDcgc]}|| }}|rC|j0s tudt/|dk(sJ|d} | |} |je j<}|j>!ty| j<j>| S|jrZ| jjAst-d| jdt{j|ddd d!d"| jg| S| jjAst-d#|d tjt| jtj| S|jrt} ntx} | d$| |jr$|jsdd%lHmI}!|!jd&|jt|||| | r | S|js | St|jd'z}"| t|"D]?}#d(|"|#z d)d*}$|"|#z d+kr t|$d,}$| |$d-.tjdA| d/#1swYxYw#tL$r}td|jFd|||jjNjPz }tS|}|<|d}td|jTd|jFd|j9|Yd}~d}~wwxYwcc} wcc}w#1swY8xYwcc}w#tv$rt-d|j>YwxYw#t$rY| SwxYw)0a Run a grid search, this is the API underlying the `dora grid` command, so that it can be used from a notebook. You can also provide patterns to filter out XPs to be displayed. Args: main (DecoratedMain): main training function, decorated with Dora. explorer (Explorer): explorer instance that will define the XPs to launch. grid_name (str): this must be a unique name for the grid. rules (SubmitRules): see `dora.conf.SubmitRules`, those defines the rules for rescheduling failed XP etc. slurm (SlurmConfig or None): if provided, this will override the default Slurm config defined my the `main` argument. Returns: A list of `dora.shep.Sheep`. NT)exist_okparents)r)poolz'--dry_run is incompatible with --clear.zYou are about to restart z experiments from the grid z' from scratch. This cannot be reverted.zConfirm [yN]: yzAbort...zCanceling all current jobs...zDeleting XP folders...z$Error when trying to load old sheep z: rzCanceling job z from unloadable sheep rArz for no longer required sheep rBz for sheep zNo sheep to handle.zIThe folder, log, and tail flags are only supported from the command line.zInvalid index zLog z does not existr1z-n200z-fz"Log file does not exist for sheep zMonitoring Grid )display)wait<zNext update in z.0fz seconds 31 )endz< )RrYr rhrDdir_gridsmkdirrrr_slowrr completeupdater7sheepsvalues_filter_grid_sheepsr$r-r+rlenr2inputlowerjob cancel_lazycommitxpr0rKriterdirrOrSget_sheep_from_sig Exceptionshepjob_filerjob_idr,sig job_arrays slurm_configsmaybe_submit_lazy job_arrayis_doneget_name is_symlinkresolve symlink_tounlinkr.init_xpr1 RuntimeError IndexErrorprintrLexecvpshutil copyfileobjopenrUstdoutBrokenPipeErrorr*rr/IPythonrv clear_outputr%r?r'rangertimesleep)%rZrWrerkrjr grid_folderherdshepherdrslauncherrreplsheep to_unlink old_sheepschild old_sheeperrorrjobsr sheep_maprr array_sheepsfirstrOlinkactionactionsindex maybe_printrvrelaouts% rriris)* h )) ) }%%'))--$))"2"22Y>KtT2 6D#&H zz  #t%DAH X  MMO$ # HeT2 OO $++$$& 'F f =F zz << ; < 'F }4OPY{[5 5 6  ! !)*Dzz|s"j! +,Eyy$$$UYY/  $%Exx%%'0EI IJ$$& ::T[[ (   U # -$77 C !,,,!!),!'$ OO ;;6<=fUUXX\\5(f =I6?Tis3)CSIcNiLT OE&&uxx||4E< A%**5%?''.!- 225%G"./.)   "==, ,,   /==.D .!5!5 67"))*!D63 4   {{E==?yy,,,}}UXX.nUYY%5%5$6k%((,,qQUPVWX$$UYY/  <<E%((,,.D{{} 1(T\\^uxx?V?V?X-XXX0  E LLN yyE LL " {{  !" %)[[$((DII$F]$F&&J\v$FG]%% QR R7|q     25ME}}UXX& ;; " %((// " YY "99##%UYYK78 IIfvtUD%))D E 99##%:4&BC ""4 ?CJJ?  {{  "9+./  << '  d + 4x =  M||  MDMMB&' >#:#::)#q'C. 4KEJJT/.V^ 2 N4;;-0 1 2#   ss(a=b d( d-d-<d2.d?6d?,e 7e,=b d%Bd  d%2d< !e)(e), e98e9c|sy|D]:}d}|dddk(r|dd}d}tj|d|zdz}|r|s5y|r:yy)NTFr!*)fnmatch)rOr$patternnegresults r _match_namerPsb  2A;# abkGCsW}s':;  r r$rcVg}t|D].} |jt||j|0g}|D];}|j |j }t||s+|j|=|r|Dcgc]}|| }}|S#t$rYwxYwcc}wr)r7rSr?remove ValueErrorrrr) r$rZrindexesprrrOidxs rrrasG (^  NN3q6 " OOA   C}}UXX& tX & JJu #*+7Cs3x7+ J   ,sB B& B#"B#rrcZ|j|Dcgc]}|jc}\}}|Dcgc]}|j|j}}d} |jt ||j} n|j rt d|D} | |D cgc]} | d|  }} g} d} tt|||D]\} \}}}|j}|jsd} |d}n|dd}|| |jr|jjnd|jj|d}i}||d < |j||}|j#|| j%||r |d |t'j(dt'j*d |j-g|j/z }|t'j0| ||j3 | Scc}wcc}wcc} w#t$r|j!|}YwxYw) zSingle iteration of monitoring of the jobs in a Grid. Returns `True` if all jobs are done or failed, and `False` otherwise. Nc32K|]}t|ywr)r).0metricss r zmonitor..s9yG3w.# 6,,W5E 6s"G>"H HH  H*)H*)7r6concurrent.futuresr dataclassesrrr functoolsrrLpathlibrtypingr8rrUrconfr r r explorer r rrZrrrrrrrrutilsrrrrrCallabler:r;rr"rfAnyrmr>r9rirrr<r%rr rrs3(  <<--,,!== ' G <R[[#<  .$.$ .$b#L Abff AM A#.-SW!,@=@H@@@8: K8P@@35775>@F""''#,m "35775>(4"&&4 44QV44)-4r