iQ ddlZddlZddlZddlZddlmZddlmZmZm Z m Z ddl m Z ddl mZmZmZmZgdZe dZe dd ZeeefZeed fZe d eeZGd deeZGddeee eZGddeeed fZGddeeZGddeeZGddeZ GddeeZ!efdeedee"e#zdedzde$e!efdZ%y) N)Sequence)castGenericIterableTypeVar) deprecated)default_generator GeneratorrandpermTensor)DatasetIterableDataset TensorDataset StackDataset ConcatDataset ChainDatasetSubset random_split_T_T_coT) covariant._T_stackc$eZdZdZdefdZddZy)r aAn abstract class representing a :class:`Dataset`. All datasets that represent a map from keys to data samples should subclass it. All subclasses should overwrite :meth:`__getitem__`, supporting fetching a data sample for a given key. Subclasses could also optionally overwrite :meth:`__len__`, which is expected to return the size of the dataset by many :class:`~torch.utils.data.Sampler` implementations and the default options of :class:`~torch.utils.data.DataLoader`. Subclasses could also optionally implement :meth:`__getitems__`, for speedup batched samples loading. This method accepts list of indices of samples of batch and returns list of samples. .. note:: :class:`~torch.utils.data.DataLoader` by default constructs an index sampler that yields integral indices. To make it work with a map-style dataset with non-integral indices/keys, a custom sampler must be provided. returnctd)Nz3Subclasses of Dataset should implement __getitem__.)NotImplementedErrorselfindexs m/Volumes/fast/ai/experiments/voice-extract-mac/.venv/lib/python3.12/site-packages/torch/utils/data/dataset.py __getitem__zDataset.__getitem__:s!"WXXct||gSN)rrothers r __add__zDataset.__add__AsdE]++r"N)r&zDataset[_T_co]rzConcatDataset[_T_co])__name__ __module__ __qualname____doc__rr!r'r"r r r 's$YEY,r"r c"eZdZdZdeefdZy)ra?An iterable Dataset. All datasets that represent an iterable of data samples should subclass it. Such form of datasets is particularly useful when data come from a stream. All subclasses should overwrite :meth:`__iter__`, which would return an iterator of samples in this dataset. When a subclass is used with :class:`~torch.utils.data.DataLoader`, each item in the dataset will be yielded from the :class:`~torch.utils.data.DataLoader` iterator. When :attr:`num_workers > 0`, each worker process will have a different copy of the dataset object, so it is often desired to configure each copy independently to avoid having duplicate data returned from the workers. :func:`~torch.utils.data.get_worker_info`, when called in a worker process, returns information about the worker. It can be used in either the dataset's :meth:`__iter__` method or the :class:`~torch.utils.data.DataLoader` 's :attr:`worker_init_fn` option to modify each copy's behavior. Example 1: splitting workload across all workers in :meth:`__iter__`:: >>> # xdoctest: +REQUIRES(env:TORCH_DOCTEST_DATALOADER) >>> # xdoctest: +SKIP("Fails on MacOS12") >>> class MyIterableDataset(torch.utils.data.IterableDataset): ... def __init__(self, start, end): ... super(MyIterableDataset).__init__() ... assert end > start, "this example only works with end >= start" ... self.start = start ... self.end = end ... ... def __iter__(self): ... worker_info = torch.utils.data.get_worker_info() ... if worker_info is None: # single-process data loading, return the full iterator ... iter_start = self.start ... iter_end = self.end ... else: # in a worker process ... # split workload ... per_worker = int(math.ceil((self.end - self.start) / float(worker_info.num_workers))) ... worker_id = worker_info.id ... iter_start = self.start + worker_id * per_worker ... iter_end = min(iter_start + per_worker, self.end) ... return iter(range(iter_start, iter_end)) ... >>> # should give same set of data as range(3, 7), i.e., [3, 4, 5, 6]. >>> ds = MyIterableDataset(start=3, end=7) >>> # Single-process loading >>> print(list(torch.utils.data.DataLoader(ds, num_workers=0))) [tensor([3]), tensor([4]), tensor([5]), tensor([6])] >>> # xdoctest: +REQUIRES(POSIX) >>> # Multi-process loading with two worker processes >>> # Worker 0 fetched [3, 4]. Worker 1 fetched [5, 6]. >>> # xdoctest: +IGNORE_WANT("non deterministic") >>> print(list(torch.utils.data.DataLoader(ds, num_workers=2))) [tensor([3]), tensor([5]), tensor([4]), tensor([6])] >>> # With even more workers >>> # xdoctest: +IGNORE_WANT("non deterministic") >>> print(list(torch.utils.data.DataLoader(ds, num_workers=12))) [tensor([3]), tensor([5]), tensor([4]), tensor([6])] Example 2: splitting workload across all workers using :attr:`worker_init_fn`:: >>> # xdoctest: +REQUIRES(env:TORCH_DOCTEST_DATALOADER) >>> class MyIterableDataset(torch.utils.data.IterableDataset): ... def __init__(self, start, end): ... super(MyIterableDataset).__init__() ... assert end > start, "this example only works with end >= start" ... self.start = start ... self.end = end ... ... def __iter__(self): ... return iter(range(self.start, self.end)) ... >>> # should give same set of data as range(3, 7), i.e., [3, 4, 5, 6]. >>> ds = MyIterableDataset(start=3, end=7) >>> # Single-process loading >>> print(list(torch.utils.data.DataLoader(ds, num_workers=0))) [3, 4, 5, 6] >>> >>> # Directly doing multi-process loading yields duplicate data >>> print(list(torch.utils.data.DataLoader(ds, num_workers=2))) [3, 3, 4, 4, 5, 5, 6, 6] >>> # Define a `worker_init_fn` that configures each dataset copy differently >>> def worker_init_fn(worker_id): ... worker_info = torch.utils.data.get_worker_info() ... dataset = worker_info.dataset # the dataset copy in this worker process ... overall_start = dataset.start ... overall_end = dataset.end ... # configure the dataset to only process the split workload ... per_worker = int(math.ceil((overall_end - overall_start) / float(worker_info.num_workers))) ... worker_id = worker_info.id ... dataset.start = overall_start + worker_id * per_worker ... dataset.end = min(dataset.start + per_worker, overall_end) ... >>> # Mult-process loading with the custom `worker_init_fn` >>> # Worker 0 fetched [3, 4]. Worker 1 fetched [5, 6]. >>> print(list(torch.utils.data.DataLoader(ds, num_workers=2, worker_init_fn=worker_init_fn))) [3, 5, 4, 6] >>> # With even more workers >>> print(list(torch.utils.data.DataLoader(ds, num_workers=12, worker_init_fn=worker_init_fn))) [3, 4, 5, 6] r&ct||gSr$)rr%s r r'zIterableDataset.__add__sT5M**r"N)r(r)r*r+r rr'r,r"r rrIsjX+WU^+r"rcHeZdZUdZeedfed<deddfdZdZde fdZ y) rzDataset wrapping tensors. Each sample will be retrieved by indexing tensors along the first dimension. Args: *tensors (Tensor): tensors that have the same size of the first dimension. .tensorsrNcRtfdDr td|_y)Nc3jK|]*}djd|jdk7,yw)rN)size).0tensorr0s r z)TensorDataset.__init__..s,J'wqzq!V[[^3's03zSize mismatch between tensors)anyAssertionErrorr0)rr0s `r __init__zTensorDataset.__init__s$ J'J J !@A A r"c@tfd|jDS)Nc3(K|] }| ywr$r,)r4r5rs r r6z,TensorDataset.__getitem__..s>vVE])tupler0rs `r r!zTensorDataset.__getitem__s>>>>r"c>|jdjdSNr)r0r3rs r __len__zTensorDataset.__len__s||A##A&&r") r(r)r*r+r=r __annotations__r9r!intrAr,r"r rrs<63; D ?''r"rc`eZdZUdZeezed<deedeeddfdZ dZ d e fd Z de fd Zy) raDataset as a stacking of multiple datasets. This class is useful to assemble different parts of complex input data, given as datasets. Example: >>> # xdoctest: +SKIP >>> images = ImageDataset() >>> texts = TextDataset() >>> tuple_stack = StackDataset(images, texts) >>> tuple_stack[0] == (images[0], texts[0]) >>> dict_stack = StackDataset(image=images, text=texts) >>> dict_stack[0] == {"image": images[0], "text": texts[0]} Args: *args (Dataset): Datasets for stacking returned as tuple. **kwargs (Dataset): Datasets for stacking returned as dict. datasetsargskwargsrNcV|rG|r tdt|d_tfd|Dr td|_y|rSt |j }t|d_tfd|Dr td|_ytd)NztSupported either ``tuple``- (via ``args``) or``dict``- (via ``kwargs``) like input/output, but both types are given.rc3NK|]}jt|k7ywr$_lengthlenr4datasetrs r r6z(StackDataset.__init__..sDtG4<<3w</t"%zSize mismatch between datasetsc3NK|]}jt|k7ywr$rJrMs r r6z(StackDataset.__init__..sCsG4<<3w</srOz%At least one dataset should be passed) ValueErrorrLrKr7rElistvalues)rrFrGtmps` r r9zStackDataset.__init__s  ^tAw.sA=WU^=r<) isinstancerEdictitemsr=)rrkrNs ` r r!zStackDataset.__getitem__s] dmmT *8< 8K8K8MN8M*!WAwu~%8MN NA4==AAAOsA+indicesc t|jtr|Dcgc]}i}}|jjD]\}}t t |ddrg|j |}t|t|k7r#tdt|dt|t||dD] \}}|||< t||dD] \} }|| ||<|S|Dcgc]}g} }|jD]}t t |ddrs|j |}t|t|k7r#tdt|dt|t|| dD]\}} | j|t|| dD]\} } | j|| | D cgc] } t| } } | Scc}wcc}wcc} w)N __getitems__z0Nested dataset's output size mismatch. Expected z, got Tstrict) rWrErXrYcallablegetattrr]rLrQzipappendr=)rr[_ dict_batchrZrNrYdatad_sampleidx list_batcht_samplesample tuple_batchs r r]zStackDataset.__getitems__s dmmT *5<(=WWJ(="mm113 7GG^TBC#009E5zS\1()),WfSZLJ+.eZ*Mh&* +N*-Wj)N X&-cl *O4 /6!6g"g !6}}G>?,,W5u:W-$%%(\N&U F'*%D&IND(OOD)'J&)*T%JMCOOGCL1&K%DN&N:uV}: &NA)>""7'Os G- GG c|jSr$)rKr@s r rAzStackDataset.__len__'s ||r")r(r)r*r+r=rXrBr rr9r!rRr]rCrAr,r"r rrsX$dlFgenFF4F(B #D#Jr"rceZdZUdZeeeed<eeed<e dZ de eddffd Z defdZ d Zeed e d ZxZS) rzDataset as a concatenation of multiple datasets. This class is useful to assemble different existing datasets. Args: datasets (sequence): List of datasets to be concatenated rEcumulative_sizescdgd}}|D]&}t|}|j||z||z }(|Sr?)rLrc)sequencersels r cumsumzConcatDataset.cumsum7s=11AAA HHQUO FAr"rNc t|t||_t |jdk(r t d|jD]}t |tst d|j|j|_ y)Nrz(datasets should not be an empty iterablez.ConcatDataset does not support IterableDataset) superr9rRrErLr8rWrrvro)rrEd __class__s r r9zConcatDataset.__init__@sq X t}}  " !KL LA!_-$%UVV!% DMM :r"c |jdS)Nror@s r rAzConcatDataset.__len__Js$$R((r"c|dkr(| t|kDr tdt||z}tj|j|}|dk(r|}n||j|dz z }|j ||S)Nrz8absolute value of index should not exceed dataset length)rLrQbisect bisect_rightrorE)rrh dataset_idx sample_idxs r r!zConcatDataset.__getitem__Ms 7tc$i Nd)c/C))$*?*?E ! Jt44[1_EEJ}}[)*55r"z>`cummulative_sizes` attribute is renamed to `cumulative_sizes`)categoryc|jSr$r}r@s r cummulative_sizeszConcatDataset.cummulative_sizes[s $$$r")r(r)r*r+rRr rrBrC staticmethodrvrr9rAr!propertyr FutureWarningr __classcell__rzs@r rr+s75>""3i;'!2;t;)) 6H%  %r"rcDeZdZdZdeeddffd ZdZdefdZ xZ S)ra_Dataset for chaining multiple :class:`IterableDataset` s. This class is useful to assemble different existing dataset streams. The chaining operation is done on-the-fly, so concatenating large-scale datasets with this class will be efficient. Args: datasets (iterable of IterableDataset): datasets to be chained together rErNc0t|||_yr$)rxr9rE)rrErzs r r9zChainDataset.__init__os   r"c#|K|jD]'}t|ts td|Ed{)y7w)N*ChainDataset only supports IterableDataset)rErWrr8)rrys r __iter__zChainDataset.__iter__ss6Aa1$%QRRLL s 0<:<c~d}|jD]+}t|ts td|t |z }-|S)Nrr)rErWrr8rL)rtotalrys r rAzChainDataset.__len__ys?Aa1$%QRR SVOE r") r(r)r*r+rr r9rrCrArrs@r rrds1!'!2!t! r"rceZdZUdZeeed<eeed<deedeeddfdZ dZ de ede efdZ defd Z y) ra^ Subset of a dataset at specified indices. .. note:: When subclassing `Subset` and overriding `__getitem__`, you **must** also override `__getitems__` to ensure `DataLoader` works correctly with your custom logic. If you override only `__getitem__`, a `NotImplementedError` will be raised when using `DataLoader`. A simple implementation of `__getitems__` can delegate to `__getitem__`: .. code-block:: python def __getitems__(self, indices): return [self.__getitem__(idx) for idx in indices] For better performance, consider implementing batch-aware logic in `__getitems__` instead of calling `__getitem__` multiple times. Args: dataset (Dataset): The whole Dataset indices (sequence): Indices in the whole set selected for subset rNr[rNc||_||_t|jtjurGt|j tj ur!t t|jdyy)Na2 overrides __getitem__ but not __getitems__. When subclassing Subset and overriding __getitem__, you must also override __getitems__ to ensure DataLoader works correctly with your custom logic. A simple implementation: def __getitems__(self, indices): return [self.__getitem__(idx) for idx in indices])rNr[typer!rr]rr()rrNr[s r r9zSubset.__init__su   J " "&*<*< <T ''6+>+>>%:&&'(HH ? =r"ct|tr*|j|Dcgc]}|j|c}S|j|j|Scc}wr$)rWrRrNr[)rrhis r r!zSubset.__getitem__sO c4 <<# >#Qa# >? ?||DLL-..!?sActt|jddr6|jj|Dcgc]}|j|c}S|Dcgc]}|j|j| c}Scc}wcc}w)Nr])r`rarNr]r[)rr[rhs r r]zSubset.__getitems__sv GDLL.$? @<<,,7-S7Cdll3.?7-ST T?FGwDLLc!23wG G.TGs B#Bc,t|jSr$)rLr[r@s r rAzSubset.__len__s4<<  r")r(r)r*r+r rrBrrCr9r!rRr]rAr,r"r rrsn0U^ c]#4$/ HDIH$u+H!!r"rrNlengths generatorrc Ztjt|drt|dkrg}t|D]P\}}|dks|dkDrt d|dtj t ||z}|j|Rt |t|z }t|D]}|t |z}||xxdz cc<|}t|D]&\}} | dk(s tjd|dd(t|t |k7r t d tt|| j} ttt|}t!t#j$||d D cgc]\} } t'|| | | z | c} } Scc} } w) a Randomly split a dataset into non-overlapping new datasets of given lengths. If a list of fractions that sum up to 1 is given, the lengths will be computed automatically as floor(frac * len(dataset)) for each fraction provided. After computing the lengths, if there are any remainders, 1 count will be distributed in round-robin fashion to the lengths until there are no remainders left. Optionally fix the generator for reproducible results, e.g.: Example: >>> # xdoctest: +SKIP >>> generator1 = torch.Generator().manual_seed(42) >>> generator2 = torch.Generator().manual_seed(42) >>> random_split(range(10), [3, 7], generator=generator1) >>> random_split(range(30), [0.3, 0.3, 0.4], generator=generator2) Args: dataset (Dataset): Dataset to be split lengths (sequence): lengths or fractions of splits to be produced generator (Generator): Generator used for the random permutation. rrzFraction at index z is not between 0 and 1zLength of split at index z- is 0. This might result in an empty dataset.) stacklevelzDSum of input lengths does not equal the length of the input dataset!)rTr^)mathisclosesum enumeraterQfloorrLrcrangewarningswarnr tolistrrrCrb itertools accumulater) rNrrsubset_lengthsrfracn_items_in_split remainder idx_to_add_atlengthr[offsets r rrs< ||CL!$W):$& )GAtax4!8 #5aS8O!PQQ#zz#g,*=>   ! !"2 3 * L3~#66 y!AN 33M = )Q . )"!"7+IAv{ /s3=> , 7|s7|# R  s7|y9@@BG8C='*G")"6"6w"?QUV VNFF w&9:V  sF')&rrrrcollections.abcrtypingrrrrtyping_extensionsrtorchr r r r __all__rrrXstr_T_dictr=_T_tuplerr rrrrrrrCfloatrRrr,r"r rs?  $43(A@  T]4( sEz    :x 1,gen,Dn+genhuon+h'GE&#+./'.T78$Tn6%GEN6%r?<<!WU^<!D#4> R[> cEk ">4> &* >r"