i'GdZddlZddlZddlZddlZddlmZddlm Z ddl m Z ddl Z ddl m Z ddl m Z ddlmZdd lmZdd lmZdd lmZd d lmZGddZdZy)ah Pipeline Usage: pyannote-pipeline train [options] [(--forever | --iterations=)] pyannote-pipeline best [options] pyannote-pipeline apply [options] pyannote-pipeline -h | --help pyannote-pipeline --version Common options: Experimental protocol (e.g. "Etape.SpeakerDiarization.TV") --registry= Path to, comma-separated, database configuration files. [default: ~/.pyannote/db.yml] --subset= Set subset. Defaults to 'development' in "train" mode, and to 'test' in "apply" mode. "train" mode: Set experiment root directory. This script expects a configuration file called "config.yml" to live in this directory. See "Configuration file" section below for more details. --iterations= Number of iterations. [default: 1] --forever Iterate forever. --sampler= Choose sampler between RandomSampler or TPESampler [default: TPESampler]. --pruner= Choose pruner between MedianPruner or SuccessiveHalvingPruner. Defaults to no pruning. --pretrained= Use parameters in existing training directory to bootstrap the optimization process. In practice, this will simply run a first trial with this set of parameters. --average-case Optimize for average case instead of worst case. "apply" mode: Path to the directory containing trained hyper- parameters (i.e. the output of "train" mode). --use-filter Apply pipeline only to files that pass the filter. Configuration file: The configuration of each experiment is described in a file called /config.yml that describes the pipeline. ................... /config.yml ................... pipeline: name: Yin2018 params: sad: tutorials/pipeline/sad scd: tutorials/pipeline/scd emb: tutorials/pipeline/emb metric: angular # preprocessors can be used to automatically add keys into # each (dict) file obtained from pyannote.database protocols. preprocessors: audio: ~/.pyannote/db.yml # load template from YAML file video: ~/videos/{uri}.mp4 # define template directly # filters can be used to filter out some files from the protocol # (e.g. to only keep files with a specific number of speakers) filters: pyannote.audio.utils.protocol.FilterByNumberOfSpeakers: num_speakers: 2 # one can freeze some hyper-parameters if needed (e.g. when # only part of the pipeline needs to be updated) freeze: speech_turn_segmentation: speech_activity_detection: onset: 0.5 offset: 0.5 # pyannote.audio pipelines will run on CPU by default. # use `device` key to send it to GPU. device: cuda ................................................................... "train" mode: Tune the pipeline hyper-parameters /..yml "best" mode: Display current best loss and corresponding hyper-paramters. "apply" mode Apply the pipeline (with best set of hyper-parameters) N)Optional)Path)docopt)tqdm)datetime) FileFinder)registry) get_annotated)get_class_by_name) OptimizerceZdZdZdZdZdZeddede ddfdZ dd ede ffd Z dd e d e e d e edede e de e de fdZdd e d e fdZ dd e ded e e de fdZxZS) ExperimentzPipeline experiment Parameters ---------- experiment_dir : `Path` Experiment root directory. training : `bool`, optional Switch to training mode z{experiment_dir}/config.ymlz*{experiment_dir}/train/{protocol}.{subset}z{train_dir}/apply/{date} train_dirtrainingreturnc|jd}|||}|dz }tjtjj ||_|jj||S)a6Load pipeline from train directory Parameters ---------- train_dir : `Path` Path to train directory training : `bool`, optional Switch to training mode. Returns ------- xp : `Experiment` Pipeline experiment. r r params.yml) parentsr fromtimestampospathgetmtimemtime_ pipeline_ load_params)clsrrexperiment_dirxp params_ymls q/Volumes/fast/ai/experiments/voice-extract-mac/.venv/lib/python3.12/site-packages/pyannote/pipeline/experiment.pyfrom_train_dirzExperiment.from_train_dirsb #**1- ( 3- **277+;+;J+GH    , rc t|||_|jj |j}t |d5}t j|t j|_ dddi}|jjdijD]Q\}}t|tr,t|dd}|di|jdi||<B t| ||<S||_g|jjd ijD]'\}} t|}j%|di| )d t&ffd } | |_|jd d} t| d}|di|jd jdi|_d|jvr*|jd} |j*j-| d|jvr>ddl}|j1|jd}|j*j3|yy#1swYxYw#t $r} |} | ||<Yd} ~ d} ~ wwxYw)N)rrLoader preprocessorsnamezpyannote.pipeline)default_module_nameparams) database_ymlfiltersrc.tfdDS)Nc3.K|] }|yw)N).0fis r" z;Experiment.__init__..all_filters..s-WqtWs)all)r4r.s`r" all_filtersz(Experiment.__init__..all_filterss-W-- -r$pipelinezpyannote.pipeline.blocksfreezedevicerr1)super__init__r CONFIG_YMLformatopenyamlload SafeLoaderconfig_getitems isinstancedictr rFileNotFoundErrorpreprocessors_appendboolfilters_rr9torchr:to)selfrr config_ymlfpr)key preprocessorKlassetemplater,r7 pipeline_namerMr:r. __class__s @r"r<zExperiment.__init__s8 ,__++4;N;N+O *c "b99R@DL# !%!1!1/2!F!L!L!N C,-) (>Q&+%L\-=-=h-K%L c" .&0\%J c"#"O2,<<++Ir:@@BKC%c*E NN5?6? +C .d .$  Z08 ! /I Lj!9!=!=h!KL t|| #\\(+F NN ! !& ) t|| # \\$,,x"89F NN  f % $q# "0% .(%- c"  .s$ +H;,I;I I#II# protocol_namesubset pretrained n_iterationssamplerpruner average_casec t|jj|j||}|j ddt j ||j} d} t|j|dz | |||} |jjdk(rd nd } |d z } td d d}|jd|jd |rF|d z }t|d5}tj |tj"}dddd}nd}t%t'|j(t+| |}| j-||d} | j.}|d krt7j8n t;|}t=||D]g\}}|d}| |z| |zkr%|d}|}|jj?| ||dd|zdd}|j||jd iy#1swYxYw#t0$r}| t2j4z}Yd}~d}~wwxYw)ayTrain pipeline Parameters ---------- protocol_name : `str` Name of pyannote.database protocol to use. subset : `str`, optional Use this subset for training. Defaults to 'development'. pretrained : Path, optional Use parameters in "pretrained" training directory to bootstrap the optimization process. In practice this will simply run a first trial with this set of parameters. n_iterations : `int`, optional Number of iterations. Defaults to 1. sampler : `str`, optional Choose sampler between RandomSampler and TPESampler pruner : `str`, optional Choose between MedianPruner or SuccessiveHalvingPruner. average_case : `bool`, optional Optimise for average case. Defaults to False (i.e. worst case). rprotocolrZTrexist_okr)defaulttrials.journal)db study_namer]r^r_minimizer rtrialr)unitpositionleavezFirst trial in progressr&moder'Nr,) warm_start show_progressloss)r,rtz Best trial: dg%)desc) r TRAIN_DIRr>rmkdirr get_protocolrIr r get_directionrset_descriptionupdater?r@rArBlistfilterrLgetattr tune_iter best_loss ValueErrornpinf itertoolscountrangezip dump_params)rOrYrZr[r\r]r^r_rrbri optimizer directionr! progress_barpre_params_ymlrQ pre_paramsrrinputs iterationsrrUrr4statusrt best_paramsrxs r"trainzExperiment.trainsH> NN ! !#22& "   t4(( )<)<   NN++!%  557:EA2 - 1DA $$%>?A ',6Nn3/2!YYr$//B 0#H-JJfT]],EGHf,E,GHI(( z)  +!++I&2A%5 !5;NUJ/IAv&>D4)i"77$X.  **{+ "# /!!4A6D  ( (d ( 3    "0'0/ +!BFF*I +s$0&H) HH I%H==Ic~t|jj|j||}d}t |j |dz |} |j }|j}tdd|zd d tj|d } t| y#t$r}tdYd}~yd}~wwxYw) a Print current best pipeline Parameters ---------- protocol_name : `str` Name of pyannote.database protocol used for training. subset : `str`, optional Subset used for training. Defaults to 'development'. rarfrg)rhriz4Still waiting for at least one iteration to succeed.NzLoss = rurvz&% with the following hyper-parameters:F)default_flow_style) rryr>rr rrrprintrr@dump) rOrYrZrrirrrUrcontents r"bestzExperiment.bestXs NN ! !#22& "    NNy+;;   !++I  ++  i**PQR))KEB g  H I  s B B<' B77B< output_dir use_filtercHtj||j} |jj }|j dd|r"||d|d|jjz }n!||d|d|jjz }t|d5} tt||} |rt|j| } d |d |d } t| | d D]^} |j| } |jj| | | jdd}|d}|It!| }||| |}` ddd|j"dz }|j%r|j'|j)||d|d}t+|y|r ||d|dz }n ||d|dz }t|d5} | jt-|dddy#t $r }d}Yd}~d}~wwxYw#1swYxYw#1swYyxYw)zApply current best pipeline Parameters ---------- protocol_name : `str` Name of pyannote.database protocol to process. subset : `str`, optional Subset to process. Defaults to 'test' reNTrc.z _INCOMPLETE.wrpz Processing z ()file)iterablerxrm annotation)uemlatestzWFor some (possibly good) reason, the output of this pipeline could not be evaluated on z_INCOMPLETE.evalz.eval)r r{rIr get_metricNotImplementedErrorrz write_formatr?rrrrLrwriterDr parentexistsunlink symlink_torstr)rOrYrrZrrbmetricrU output_extrQfilesrx current_fileoutput referencer_rmsg output_evals r"applyzExperiment.apply}sK$(( )<)<   ^^..0F 5 "O1VHL9T9T8UVW   axq9T9T8UVV *3 '226245Et}}e4 r&;D $e$V L  5$$R0),,\4@ $!F>#L19f#6!M (0""X- ==? MMO*% >66C_AG  #J  $-&AQ'RRK$-&'GGK +s #r HHS[ !$ #s# F ( 'V$ #s0G3"B+H H3 H <HH  HH!)F) developmentNr NNF)r)testF)__name__ __module__ __qualname____doc__r=ry APPLY_DIR classmethodrrKr#r<rrintrrr __classcell__)rXs@r"rrs/Jz--subsetrrz --foreverrkz --iterationsz --samplerz--prunerz --pretrainedT)strictz--average-casezr)rZr\r[r]r^r_rF)rZrrz --use-filterz z %Y%m%d-%H%M%S)rdate)rZr)rrsplitr load_databaserr expanduserresolverrrr#rr>rstrftimer) argumentsr-rYrZrr]r^r[r_r experimentrrrs r"mainrs=w(;"F [ !JY~67JK(:&~. j)446>>d>KJ !12 i(:;<'224<  #!%   >"Fi(:;<'224<F~. =12 ((*22$2? ..y5.I   ' '#**;*;*D*D_*U (   :f  !r$)rros.pathr@numpyrtypingrpathlibrrrrrpyannote.databaserr r pyannote.core.utils.helperr rr rrr1r$r"rsN:Xt  (&+8 E"E"P E r$