+ QV-j^RIt^RIHtHt^RIHt^RIHtHt]P!] 4t !RR4t ]R4t RtR t]]P R R l44tR#) N)PoolRLock)tqdm) experimentalloggingc]tRt^ tRtRtR#)ParallelBackendConfigN)__name__ __module__ __qualname____firstlineno__ backend_name__static_attributes__r k/Volumes/fast/ai/experiments/ui-tars-smoke/.venv/lib/python3.14/site-packages/datasets/parallel/parallel.pyr r sLrr c b\Pf\WW#WEWgV4 #\WW#WEWgV4 #)a **Experimental.** Apply a function to iterable elements in parallel, where the implementation uses either multiprocessing.Pool or joblib for parallelization. Args: function (`Callable[[Any], Any]`): Function to be applied to `iterable`. iterable (`list`, `tuple` or `np.ndarray`): Iterable elements to apply function to. num_proc (`int`): Number of processes (if no backend specified) or jobs (using joblib). types (`tuple`): Additional types (besides `dict` values) to apply `function` recursively to their elements. disable_tqdm (`bool`): Whether to disable the tqdm progressbar. desc (`str`): Prefix for the tqdm progressbar. single_map_nested_func (`Callable`): Map function that applies `function` to an element from `iterable`. Takes a tuple of function, data_struct, types, rank, disable_tqdm, desc as input, where data_struct is an element of `iterable`, and `rank` is used for progress bar. )r r_map_with_multiprocessing_pool_map_with_joblib) functioniterablenum_procbatched batch_sizetypes disable_tqdmdescsingle_map_nested_funcs &&&&&&&&&r parallel_maprsD"))1- :lZp   Hz,Vl rc V\V48:dTM \V4p.p \V4Fmp \V4V,p \V4V,p W,\W4,p W,W8d^M^,pV PWWW4WZWg34Ko \V4\ RV 448wd+\ R\V4 R\ RV 44 24h\ PRT R\V4 RV Uu.uFp\V^,4NK up 24RRppV'g\43\Ppp\VVVR 7;_uu_4pVPW4pRRR4\ PR V R 24XUUu.uFpVFpVNK K ppp\ PR \V4 R 24V#uupi +'giLn;iuuppi)c3F"TFp\V^,4xK R#5ir!Nlen.0is& r 1_map_with_multiprocessing_pool..7s:z!C!IIz!zHError dividing inputs iterable among processes. Total number of objects z , length: c3F"TFp\V^,4xK R#5ir#r$r&s& rr)r*;s9j3qt99jr+z Spawning z processes for z objects in slices of N)initargs initializerz Finished z processesz Unpacked z objects) r%rangeminappendsum ValueErrorloggerinforrset_lockrmap)rrrrrrrrr split_kwdsindexdivmodstartendr(r-r.poolmappedproc_resobjs&&&&&&&&& rrr+s$s8}4x#h-HJx(mx'(mh& c%o-k%+Q158e%8'u]ipq ! 8}:z:::''*8}o69j99: <   KK H:_S]O;QfpRqfpabSVWXYZW[S\fpRqQrs!$kH !& DMM+ h{ C Ct0= D KK)H:Z01"( =&hHScHc&F = KK)CK=12 MSr D C>s G*G""G5" G2 c aaaaaa ^RIo S P\PVR7;_uu_4S P 4!VVVV VV3RlV44uuRRR4# +'giR#;i)rN)n_jobsc 3b<"TF$pSPS4!SVSSSRRR34xK& R#5i)NT)delayed)r'rArrrjoblibrrs& rr)#_map_with_joblib..Us?!  NN1 2Hc7JX]_ceiko3p q qs,/)rFparallel_backendr rParallel) rrrrrrrrrrFs f&&fff&&f@rrrMsV   !6!C!CH U U ! ! !   V U U Us %A$$ A5 c$V^8dQhR\/#)r)str)formats"r __annotate__rN]s2232rc#"V\nVR8Xd^RIHpV!4RxR\nR# R\ni;i5i)ag **Experimental.** Configures the parallel backend for parallelized dataset loading, which uses the parallelization implemented by joblib. Args: backend_name (str): Name of backend for parallelization implementation, has to be supported by joblib. Example usage: ```py with parallel_backend('spark'): dataset = load_dataset(..., num_proc=2) ``` spark)register_sparkN)r r joblibsparkrQ)rrQs& rrHrH[s; *6&w. 2 -1*T*s A4 A AA) contextlibmultiprocessingrr tqdm.autorutilsrr get_loggerr r4r rrrcontextmanagerrHr rrrYsn')   H %4D   22r