iq=UdZddlmZddlZddlZddlZddlZddlZddl Z ddl Z ddl Z ddl Z ddl mZddlmZmZmZmZmZddlmZddlZddlmZddlZddlmZddlmZdd l m!Z!m"Z"m#Z#m$Z$dd l%m&Z&m'Z'dd l(m)Z)m*Z*m+Z+m,Z,erdd l m-Z-gd Z.edZ/eddZ0ee1gdfZ2ee3e/gefZ4ejjjlZ6de7d<ejjjpZ8ejrjtZ:ejve<Z=GddZ>Gdde+Z?dZ@d#dZAdZBGddee0ZCGddZDGdd eDZEGd!d"eDZFy)$aDefinition of the DataLoader and associated iterators that subclass _BaseDataLoaderIter. To support these two classes, in `./_utils` we define many utility methods and functions to be run in multiprocessing. E.g., the data loading worker loop is in `./_utils/worker.py`. ) annotationsN)Callable)AnyGenericNoReturn TYPE_CHECKINGTypeVar)Self)ExceptionWrapper)_utils)!_IterDataPipeSerializationWrapper _MapDataPipeSerializationWrapper IterDataPipe MapDataPipe)DatasetIterableDataset) BatchSampler RandomSamplerSamplerSequentialSampler)Iterable) DataLoaderget_worker_infodefault_collatedefault_convert_T_T_coT) covariant _collate_fn_trc$eZdZdZdZedZy) _DatasetKindrc|tjk(r"tjj ||||Stjj ||||SN)r!Mapr fetch_MapDatasetFetcher_IterableDatasetFetcher)kinddatasetauto_collation collate_fn drop_lasts p/Volumes/fast/ai/experiments/voice-extract-mac/.venv/lib/python3.12/site-packages/torch/utils/data/dataloader.pycreate_fetcherz_DatasetKind.create_fetcherRsR <## #<<22Y <<77Y N)__name__ __module__ __qualname__r%r staticmethodr/r0r.r!r!Ns CHr0r!ceZdZdZdZy)_InfiniteConstantSamplerzxAnalogous to ``itertools.repeat(None, None)``. Used as sampler for :class:`~torch.utils.data.IterableDataset`. c#K dwr$r5selfs r.__iter__z!_InfiniteConstantSampler.__iter__dsJs N)r1r2r3__doc__r;r5r0r.r7r7^s  r0r7ctjr KK##22-!1!y!"r0ctjdtjj|}t |t j rt j|d||jS)Nr5dtype generatorr)srcgroup) rEemptyint64random_rJr> ProcessGroup broadcastitem)rZpg _shared_seeds r._share_dist_seedresS;;r5== =RL"d''( |"5    r0cpeZdZUdZded<ded<ded<ded <ded <d ed <d ed<ded<ded<ded<dZ d#ddddd d$dZd%dZedZ e jd&dZ d&fd Z d%dZ edZ ed Zd'd!Zd&d"ZxZS)(ra Data loader combines a dataset and a sampler, and provides an iterable over the given dataset. The :class:`~torch.utils.data.DataLoader` supports both map-style and iterable-style datasets with single- or multi-process loading, customizing loading order and optional automatic batching (collation) and memory pinning. See :py:mod:`torch.utils.data` documentation page for more details. Args: dataset (Dataset): dataset from which to load the data. batch_size (int, optional): how many samples per batch to load (default: ``1``). shuffle (bool, optional): set to ``True`` to have the data reshuffled at every epoch (default: ``False``). sampler (Sampler or Iterable, optional): defines the strategy to draw samples from the dataset. Can be any ``Iterable`` with ``__len__`` implemented. If specified, :attr:`shuffle` must not be specified. batch_sampler (Sampler or Iterable, optional): like :attr:`sampler`, but returns a batch of indices at a time. Mutually exclusive with :attr:`batch_size`, :attr:`shuffle`, :attr:`sampler`, and :attr:`drop_last`. num_workers (int, optional): how many subprocesses to use for data loading. ``0`` means that the data will be loaded in the main process. (default: ``0``) collate_fn (Callable, optional): merges a list of samples to form a mini-batch of Tensor(s). Used when using batched loading from a map-style dataset. pin_memory (bool, optional): If ``True``, the data loader will copy Tensors into device/CUDA pinned memory before returning them. If your data elements are a custom type, or your :attr:`collate_fn` returns a batch that is a custom type, see the example below. drop_last (bool, optional): set to ``True`` to drop the last incomplete batch, if the dataset size is not divisible by the batch size. If ``False`` and the size of dataset is not divisible by the batch size, then the last batch will be smaller. (default: ``False``) timeout (numeric, optional): if positive, the timeout value for collecting a batch from workers. Should always be non-negative. (default: ``0``) worker_init_fn (Callable, optional): If not ``None``, this will be called on each worker subprocess with the worker id (an int in ``[0, num_workers - 1]``) as input, after seeding and before data loading. (default: ``None``) multiprocessing_context (str or multiprocessing.context.BaseContext, optional): If ``None``, the default `multiprocessing context `_ # noqa: D401 of your operating system will be used. (default: ``None``) generator (torch.Generator, optional): If not ``None``, this RNG will be used by RandomSampler to generate random indexes and multiprocessing to generate ``base_seed`` for workers. (default: ``None``) prefetch_factor (int, optional, keyword-only arg): Number of batches loaded in advance by each worker. ``2`` means there will be a total of 2 * num_workers batches prefetched across all workers. (default value depends on the set value for num_workers. If value of num_workers=0 default is ``None``. Otherwise, if value of ``num_workers > 0`` default is ``2``). persistent_workers (bool, optional): If ``True``, the data loader will not shut down the worker processes after a dataset has been consumed once. This allows to maintain the workers `Dataset` instances alive. (default: ``False``) pin_memory_device (str, optional): Deprecated, the current :ref:`accelerator` will be used as the device if ``pin_memory=True``. in_order (bool, optional): If ``False``, the data loader will not enforce that batches are returned in a first-in, first-out order. Only applies when ``num_workers > 0``. (default: ``True``) .. warning:: If the ``spawn`` start method is used, :attr:`worker_init_fn` cannot be an unpicklable object, e.g., a lambda function. See :ref:`multiprocessing-best-practices` on more details related to multiprocessing in PyTorch. .. warning:: ``len(dataloader)`` heuristic is based on the length of the sampler used. When :attr:`dataset` is an :class:`~torch.utils.data.IterableDataset`, it instead returns an estimate based on ``len(dataset) / batch_size``, with proper rounding depending on :attr:`drop_last`, regardless of multi-process loading configurations. This represents the best guess PyTorch can make because PyTorch trusts user :attr:`dataset` code in correctly handling multi-process loading to avoid duplicate data. However, if sharding results in multiple workers having incomplete last batches, this estimate can still be inaccurate, because (1) an otherwise complete batch can be broken into multiple ones and (2) more than one batch worth of samples can be dropped when :attr:`drop_last` is set. Unfortunately, PyTorch can not detect such cases in general. See `Dataset Types`_ for more details on these two types of datasets and how :class:`~torch.utils.data.IterableDataset` interacts with `Multi-process data loading`_. .. warning:: See :ref:`reproducibility`, and :ref:`dataloader-workers-random-seed`, and :ref:`data-loading-randomness` notes for random seed related questions. .. warning:: Setting `in_order` to `False` can harm reproducibility and may lead to a skewed data distribution being fed to the trainer in cases with imbalanced data. Dataset[_T_co]r* int | None batch_sizeintrIbool pin_memoryr-floattimeoutzSampler | Iterablesamplerstrpin_memory_deviceprefetch_factorz_BaseDataLoaderIter | None _iteratorFNT)rrpersistent_workersrqin_ordercRtjjd|dkr td| dkr td|dk(r | td|dkDr|d}n||dkr td|r|dk(r td||_||_||_||_||_| |_ | |_ | |_ ||_ t|jtrt|j|_n4t|jt rt#|j|_t|t$rt&j(|_t|tr8|Htj,j.j0j3|| }n|d vrtd ||td ||.td |t5|}t&j6|_| |r td||dk7s|s|| r tdd}d} n| | r td|C|j*t&j(k(r t9}n|rt;|| }n t=|}|| t?||| }||_ | |_!||_"||_#| |_$|A|jJrtLjNjP}ntLjNjR}||_*||_+d|_,d|_-d|_.|j_tj`dddy)Nzpython.data_loaderrzXnum_workers option should be non-negative; use num_workers=0 to disable multiprocessing.z%timeout option should be non-negativezprefetch_factor option could only be specified in multiprocessing.let num_workers > 0 to enable multiprocessing, otherwise set prefetch_factor to None.z-prefetch_factor option should be non-negativez/persistent_workers option needs num_workers > 0)shuffle>FNzVDataLoader with IterableDataset: expected unspecified shuffle option, but got shuffle=zVDataLoader with IterableDataset: expected unspecified sampler option, but got sampler=zbDataLoader with IterableDataset: expected unspecified batch_sampler option, but got batch_sampler=z1sampler option is mutually exclusive with shuffler"z[batch_sampler option is mutually exclusive with batch_size, shuffle, sampler, and drop_lastFzVbatch_size=None option disables auto-batching and is mutually exclusive with drop_lastrYT DataloaderenabledTrue)1rE_C_log_api_usage_once ValueErrorr*rIrrrlrqrnrMmultiprocessing_contextrvrJrr rrrr!r _dataset_kindrFrGrKapply_shuffle_settingsrkr%r7rrrrir-ro batch_samplerrZ_auto_collationr collaterrr,ru_DataLoader__initialized_IterableDataset_len_calledrscheck_worker_number_rationality set_vital)r:r*riryrorrIr,rlr-rnrMrrZrrrurqrvs r.__init__zDataLoader.__init__sT* $$%9: ?@  Q;DE E !  ;h 1_!8O  (_q-@LM M +"2NO O &.$!2 ,'>$   dllL 1 0), but got num_workers=) rIrJrprEmultiprocessingget_all_start_methodsr get_contextpython_multiprocessingcontext BaseContext TypeErrorr)r:rvalid_start_methodss r.rz"DataLoader.multiprocessing_contexts " .!#5s;*/*?*?*U*U*W'.6II(FFYE\]77N6QS /4.C.C.O.O//+"+-C-K-K-W-W$33J2KM !##'#3#3"46 *A&r0c|jr)|dvr%t|d|jjdt|||y)N)rirror-r*ruz# attribute should not be set after z is initialized)rr __class__r1super __setattr__)r:attrvalrs r.rzDataLoader.__setattr__sU   $+ # &;DNN> !%%d+>> !%%' 'r0c|jduSr$)rr9s r.rzDataLoader._auto_collations!!--r0cJ|jr |jS|jSr$)rrror9s r._index_samplerzDataLoader._index_samplers#   %% %<< r0c*|jtjk(rbt|jx}|_|j 8ddlm}|jr||j z}|S|||j z }|St|jS)Nr)ceil) rr!rlenr*rrimathrr-r)r:lengthrs r.__len__zDataLoader.__len__s   !6!6 6"9>#t6FM"&4??":;FMt**+ +r0cd}|jr|jdk(ryd}d}ttdr! ttjd}d}|tj }||}|*tj|||j|dy|j|kDr*tj|||j|dyy#t $rYwxYw)NcL|dj||rdndnd}d|d|d}|S)Nz|Our suggested max number of worker in current system is {}{}, which is smaller than what this DataLoader is going to create.rtz% (`cpuset` is not taken into account)zUDataLoader is not able to compute a suggested max number of worker in current system.zThis DataLoader will create z worker processes in total. z Please be aware that excessive worker creation might get DataLoader running slow or even freeze, lower the worker number to avoid potential slowness/freeze if necessary.)format)num_worker_suggestnum_worker_createdcpuset_checkedsuggested_max_worker_msgwarn_msgs r._create_warning_msgzGDataLoader.check_worker_number_rationality.._create_warning_msgAsb&1Hf* .!H l! %*//A.BB^_w^xy[[  Or0rFsched_getaffinityTrx stacklevel) rIhasattrosrr Exception cpu_countwarningswarn)r:rmax_num_worker_suggestrrs r.rz*DataLoader.check_worker_number_rationality&s6 84#3#3q#8 "& 2* + ),R-A-A!-D)E&!% " ) I$)2& ! ) MM#*D,<,r%tAjBd|j>dd|j<r7tjDjsd}tAjB|d|j<xrtjDj|_#|jFr,tjDjIx} |jJnd|_&|jLdk(r d |_#d }tAjB|d|jN|_(|jR|_*tW|j0|_,tjZd tj\ j_|j ja|_1|jd|_3d|_4d|jjjld|_7y)Ngloo)backendznpin_memory_device is deprecated, the current accelerator will be used as the device,ignore pin_memory_device='z'.rxrzj'pin_memory' argument is set as true but no accelerator is found, then device pinned memory won't be used.mpsFzf'pin_memory' argument is set as true but not supported on MPS now, device pinned memory won't be used.r5rWrYrzenumerate(DataLoader)#z .__next__)8r*_datasetrd_pgrJrr>r?r@ new_grouprerZrE Generator manual_seedrFrGrKapply_random_seedrrrr- _drop_lastrrI _num_workersrC _world_size_rankrlrqrr accelerator _pin_memorycurrent_acceleratortype_pin_memory_devicern_timeoutr, _collate_fniter _sampler_iterr]r^r_rb _base_seedru_persistent_workers _num_yieldedrr1 _profile_name)r:loader shared_rngwsrankraccs r.rz_BaseDataLoaderIter.__init__s   dmm\ 2  "t':':'<>>&9 01A1A488 LD *J  " "4#4#4 5!KK,,;;MM zDM$11+1+M+M(%55 **$33"..,.D   !9!9 MM--3-E-E,FbJ    U%6%6%C%C%E;  MM(q 1",,Q1B1B1O1O1Q ))==??L HH   " "e +$D 6  MM(q 1 !,,!$"5"56 KK%++ . Wv//W 0 TV  $*#<#< 5dnn6M6M5NiXr0c|Sr$r5r9s r.r;z_BaseDataLoaderIter.__iter__s r0ct|j|_d|_|j|_t |j trt|j|j|_ tj}|j|jtjj j"j%|j ||_yyr)rrrrrrJrrrerZrrdrErrrFrGrKr)r:r first_iterrs r.rz_BaseDataLoaderIter._resets!$"5"56+1+M+M( dmm\ 2 01A1A488 LD *J  " "4#4#4 5!KK,,;;MM zDM 3r0c,t|jSr$)nextrr9s r. _next_indexz_BaseDataLoaderIter._next_indexsD&&''r0ctr$)NotImplementedErrorr9s r. _next_dataz_BaseDataLoaderIter._next_datas!!r0cHtjjj|j5|j |j |j}|xjdz c_|jtjk(rz|jn|j|jkDrUd|jd|jd|jd}|jdkDr|dz }tj |d |cdddS#1swYyxYw) Nr"zLength of IterableDataset z was reported to be z&(when accessing len(dataloader)), but z samples have been fetched. rzFor multiprocessing data-loading, this could be caused by not properly configuring the IterableDataset replica at each worker. Please see https://pytorch.org/docs/stable/data.html#torch.utils.data.IterableDataset for examples.rxr)rEautogradprofilerrecord_functionrrrrrrr!rrrrrr)r:rGrs r.__next__z_BaseDataLoaderIter.__next__s ^^ $ $ 4 4T5G5G H!!) ??$D    " ""l&;&;;44@%%(H(HH1?STXTtTtSuz:_MultiProcessingDataLoaderIter.__init__..s/A!%%s)r)Jrrrr_prefetch_factorrv _in_orderrrHrrErrM_worker_init_fnrJrrr functoolspartialrUrrQueue_worker_result_queue_worker_pids_set _shutdownEvent_workers_done_event _index_queues_workersrangecancel_join_threadProcessr worker _worker_looprrrrrrrddaemonpicklerstartrAttributeErrorrrappendr threading_pin_memory_thread_done_eventqueue _data_queuercurrent_device_indexThreadrl_pin_memory_loopr_pin_memory_threadatexitregisterr_clean_up_workersignal_handling_set_worker_pidsidtuple_set_SIGCHLD_handlerr) r:rri index_queuer rcurrent_device_idpin_memory_threadr+rs r.rz'_MultiProcessingDataLoaderIter.__init__]s  & 6 6    ! V   A % Z   ) ) 1&+&;&; #&,&D&D #%44 dmmlK%@ A#,#4#4($$   $D %<$A$A$C! %#:#@#@#B  t(()A1779K  * * ,'//}}11&&MM--,,(($$OOOO((%%,,%%0A&AH -      % %k 2 MM  #_*b   1:1BD . %{{}D  % 1 1 F F H  ) 0 0((99--$$%66++  ! (,  $  # # %'8D ##88D   # #(8(8 ]] > O OQRS# // tH // /  335 $ Ft ,s~}=  Q !  s >P88)Q!c\t |||d|_d|_i|_d|_t |jDcgc]}dc}|_t |jDcgc]}dc}|_ tjt |j|_ |st |jD]G}|j|jtj j#|j$I|j}|dkDrO|j'\}}t)|tj j"r| t+d|dz}|dkDrOt |j,|jzD]}|j/ycc}wcc}w)NrTz7Expected return_data to be None when resuming iterationr")rr _send_idx _rcvd_idx _task_info_tasks_outstandingrr_workers_status_workers_num_tasks itertoolscycle_worker_queue_idx_cyclerputr r_ResumeIterationrd _get_datarJrHr _try_put_index) r:rrr3idxresume_iteration_cnt return_idx return_data_rs r.rz%_MultiProcessingDataLoaderIter._resets vz*  /4D4E4E.FG.F.FG /4D4E4E.F"G.F1.F"G'0uT=N=N7O'P$T../""3'++MM2243D3DE0$(#4#4 &**...*:' Kj&--*H*HI".,U)A-('*t,,t/@/@@AA    !B1 H #Hs  F$- F)c |jj|}d|fS#t$r\}g}t|jD]H\}}|j |s|j r'|j||j|Jt|dkDr(djd|D}td|d|t|tjrYd}~yddl}ddl} d } t#j$5} t'| D]!} | j)| j+# ddd#1swYxYw#t,$r/}|j|j.k(r td dYd}~d}~wwxYwd}~wwxYw) NrnTrz, c3FK|]}t|jywr$)rprrs r.r z?_MultiProcessingDataLoaderIter._try_get_data..*s$HASZs!zDataLoader worker (pid(s) z) exited unexpectedly)FN aToo many open files. Communication with the workers is no longer possible. Please increase the limit using `ulimit -n` in the shell or change the sharing strategy by calling `torch.multiprocessing.set_sharing_strategy('file_system')` at the beginning of your code)r&getr enumeraterr<is_aliver"_mark_worker_as_unavailablerjoin RuntimeErrorrJr%Emptyerrnotempfile contextlib ExitStackr enter_contextNamedTemporaryFileOSErrorEMFILE) r:rnrGefailed_workersrPr pids_strrUrVfds_limit_marginstackrIs r. _try_get_dataz,_MultiProcessingDataLoaderIter._try_get_datas~, ##'''8D$< )  N )$-- 8 1'' 21::<"))!,44Y?!9>"Q&99$H$HH"0 :OP!U[[)$   $& ))+u"#34++$7795, ,  77ell*&9  +  ?) su" F,FF*A4F#F,E0D<3E;F<E EFE F%E;6F;FFFc|jdkDr;|j|j\}}|r|Std|jd|jrW|jj r2|j\}}|r|S|jj r2td |j\}}|r|S)NrzDataLoader timed out after z secondsz%Pin memory thread exited unexpectedly)rrbrSrr*rP)r:successrGs r.rCz(_MultiProcessingDataLoaderIter._get_datas ==1  ..t}}=MGT "1$--I  ))224 $ 2 2 4 K))224 ##JKK $ 2 2 4 Kr0c |j|jkr|jj|jd}|r:|d}t |dk(s|j |rnh|j|j=|xjdz c_|j|jkr|j s|jtt |j|jdk(rO|jj|j\}}|xjdz c_|j||S|js|jdkr td|j\}}|xjdzc_ |jt j"k(rwt%|t&j(j*rS|j rd|j |j,<n|j/|j,|j1#||jk7rU|j2s0|jj|d}|j||S|j|xx|fz cc<nE|jj|d}|xjdz c_|j||S)Nrrxr"zPInvalid iterator state: shutdown or no outstanding tasks when fetching next dataF)r9r8r:rNrr<r_shutdown_workers StopIterationpop _process_datarr;rHrCrr!rrJr r_IterableDatasetStopIterationrPrQrDr )r:rRrPrGrEs r.rz)_MultiProcessingDataLoaderIter._next_datasB..4>>1**4>>4@ $QID Q$*>*>y*I7!#..4>>1//**,## 4??4>>23q8"&//"5"5dnn"E 4!#))$ ::~~!8!8A!=$f(IC  # #q ( #!!\%:%::dFMM$O$OP//?D,,T^^<88H'')dnn$~~!% 3 3C 8 ;I--dI>>$/$ OO//4Q7 !#))$ ::yr0c|j|jz}|j|k\r td |j }t |jD]_}t|j}|j|s(|jrn-|j||t|jzks_ny|j|j|j|f|f|j |j<|j|xxdz cc<|xjdz c_|xjdz c_y#t $rYywxYw)Nz:Number of outstanding tasks exceeded maximum allowed tasksr")r rr;rHrrgrrr@r<r r=sumrrAr8r:)r: max_tasksrrIworker_queue_idxs r.rDz-_MultiProcessingDataLoaderIter._try_put_indexsC))D,=,==  " "i / L  $$&Et(()A#D$@$@A ##$45>>,,-=>c((OB *  +,00$..%1HI+;*='  01Q61 1$ !+   sE EEc|j|xxdzcc<|jt|tr|j |S)Nr")r=rDrJr reraise)r:rG worker_idxs r.riz,_MultiProcessingDataLoaderIter._process_data.s=  +q0+  d, - LLN r0c|j|s|js |s td|j|}|j dd|j|<|j j |k7r tdy)Nz=Worker status inconsistent when marking worker as unavailableFz6_workers_done_event state does not match shutdown flag)r<rrHrrAris_set)r:rPshutdownqs r.rQz:_MultiProcessingDataLoaderIter._mark_worker_as_unavailable5s $$Y/,, O    y ) d +0Y'  # # * * , 8 H  9r0ct"tjdustjy|jsd|_ t|dr|jj |j jd|jj|j j|j j|jj tt|jD]1}|j s|j"|s|j%|d3|jD]"}|jtj&$|j(D]"}|j|j$ |j*r/tj,j/t1|d|_|jD]#}|j3s|j5%yy#|j*r/tj,j/t1|d|_|jD]#}|j3s|j5%wxYw)NTr*)NN)rtrKF)r python_exit_statusrrr$setrrAr*rRrcloserrrrrr<rQMP_STATUS_CHECK_INTERVALrrr._remove_worker_pidsr0rP terminate)r:rPr rus r.rfz0_MultiProcessingDataLoaderIter._shutdown_workersYs N((D0((0 ~~!DN7 &4!5666::<--11,?++002--@@B--335((,,.!&s4=='9!:I //43G3G 3R88T8R";AFF6#B#BFC ' ++A((*GGI,((**>>r$xH,1D)Azz|  'g`((**>>r$xH,1D)Azz|  'sC%H %A7H AI;(I;c |jtj|jr|j yy#|jr|j wwxYw)NrK)rRr rzrPr|)r s r.r-z/_MultiProcessingDataLoaderIter._clean_up_workersJ  FF6::F ;zz| qzz| s A#A'c$|jyr$)rfr9s r.__del__z&_MultiProcessingDataLoaderIter.__del__s  r0rr)r1r2r3r<rrr rzrbrCrrDrirQrfr4r-rrrs@r.rr'sbTj E-N+"Z%+$C$C8v! F=;~@"HH&V!r0rr)Gr< __future__rrWrr>loggingrrrr%r#rcollections.abcrtypingrrrrr typing_extensionsr rEtorch.distributed distributedr>torch.utils.data.graph_settings torch._utilsr torch.utils.datar #torch.utils.data.datapipes.datapiper rrrtorch.utils.data.datasetrrtorch.utils.data.samplerrrrrr__all__rrrj_worker_init_fn_tlistrrrrrrr getLoggerr1loggerr!r7rCrUrerrrrr5r0r.rsU#0 $AA" &)# >(  T]4(cUD[) $r(S) "(!?!??..00--//   8 $   w".qqh@S@SF#6DF!%8F!r0