+
QV-j- ^ RI HtHt ^RIHtHtHt ^RIHtH t ^RI
H t ^RI H
t
HtHt ^RIHt ^RIHt ^RIHt ]P, ! ]4 t]! R ]]
4 tRR R lltRR
R lltR
# ) )OptionalTypeVar)Dataset _concatenate_map_style_datasets _interleave_map_style_datasets) DatasetDictIterableDatasetDict) DatasetInfo)IterableDataset _concatenate_iterable_datasets _interleave_iterable_datasets)
NamedSplit)logging)Literal DatasetTypeNc V ^8 d QhR\ \ , R\ \ \ , , R\ \ , R\ \
, R\ \ , R\ R, R\ /# ) datasets
probabilitiesseedinfosplitstopping_strategyreturnfirst_exhausted
all_exhausted!all_exhausted_without_replacement)listr r floatintr
r r )formats "a/Volumes/fast/ai/experiments/ui-tars-smoke/.venv/lib/python3.14/site-packages/datasets/combine.py __annotate__r$ s{ S
S
; S
DK(S
3-S
;
S
J S
O
S
S
c
^RI Hp ^RIHp V ' g \ R4 h\ V 4 F w r\
WV34 ' g \
V \ \ 34 ' dH V ' g \ RV R24 h\ RV R\ V 4 R\ \ V 4 4 R 24 h\ RV R
\ V 4 P R 24 hV^ 8X d \
W4 ' d Wg3MWv3w rK \
V X
4 ' d K \ R V
P R
X P RV R24 h VR9 d \ V R24 hX
VJ d \ WW#WER7 # \ V VVVVVR7 # )u
Interleave several datasets (sources) into a single dataset.
The new dataset is constructed by alternating between the sources to get the examples.
You can use this function on a list of [`Dataset`] objects, or on a list of [`IterableDataset`] objects.
- If `probabilities` is `None` (default) the new dataset is constructed by cycling between each source to get the examples.
- If `probabilities` is not `None`, the new dataset is constructed by getting examples from a random source at a time according to the provided probabilities.
The resulting dataset ends when one of the source datasets runs out of examples except when `oversampling` is `True`,
in which case, the resulting dataset ends when all datasets have ran out of examples at least one time.
Note for iterable datasets:
* The resulting dataset's `num_shards` is the minimum of each dataset's `num_shards` to ensure good parallelism.
If some of your datasets have a very low number of shards, you may use [`IterableDataset.reshard`].
* In a distributed setup or in PyTorch DataLoader workers, the stopping strategy is applied per process.
Therefore the "first_exhausted" strategy on an sharded iterable dataset can generate less samples in total (up to 1 missing sample per subdataset per worker).
Args:
datasets (`List[Dataset]` or `List[IterableDataset]`):
List of datasets to interleave.
probabilities (`List[float]`, *optional*, defaults to `None`):
If specified, the new dataset is constructed by sampling
examples from one source at a time according to these probabilities.
seed (`int`, *optional*, defaults to `None`):
The random seed used to choose a source for each example.
info ([`DatasetInfo`], *optional*):
Dataset information, like description, citation, etc.
split ([`NamedSplit`], *optional*):
Name of the dataset split.
stopping_strategy (`str`, defaults to `first_exhausted`):
Three strategies are proposed right now, `first_exhausted`, `all_exhausted` and `all_exhausted_without_replacement`.
By default, `first_exhausted` is an undersampling strategy, i.e the dataset construction is stopped as soon as one dataset has ran out of samples.
If the strategy is `all_exhausted`, we use an oversampling strategy, i.e the dataset construction is stopped as soon as every samples of every dataset has been added at least once.
When strategy is `all_exhausted_without_replacement` we make sure that each sample in each dataset is sampled only once.
Note that if the strategy is `all_exhausted`, the interleaved dataset size can get enormous:
- with no probabilities, the resulting dataset will have `max_length_datasets*nb_dataset` samples.
- with given probabilities, the resulting dataset will have more samples if some datasets have really low probability of visiting.
Returns:
[`Dataset`] or [`IterableDataset`]: Return type depends on the input `datasets`
parameter. `Dataset` if the input is a list of `Dataset`, `IterableDataset` if the input is a list of
`IterableDataset`.
Example:
For regular datasets (map-style):
```python
>>> from datasets import Dataset, interleave_datasets
>>> d1 = Dataset.from_dict({"a": [0, 1, 2]})
>>> d2 = Dataset.from_dict({"a": [10, 11, 12]})
>>> d3 = Dataset.from_dict({"a": [20, 21, 22]})
>>> dataset = interleave_datasets([d1, d2, d3], probabilities=[0.7, 0.2, 0.1], seed=42, stopping_strategy="all_exhausted")
>>> dataset["a"]
[10, 0, 11, 1, 2, 20, 12, 10, 0, 1, 2, 21, 0, 11, 1, 2, 0, 1, 12, 2, 10, 0, 22]
>>> dataset = interleave_datasets([d1, d2, d3], probabilities=[0.7, 0.2, 0.1], seed=42)
>>> dataset["a"]
[10, 0, 11, 1, 2]
>>> dataset = interleave_datasets([d1, d2, d3])
>>> dataset["a"]
[0, 10, 20, 1, 11, 21, 2, 12, 22]
>>> dataset = interleave_datasets([d1, d2, d3], stopping_strategy="all_exhausted")
>>> dataset["a"]
[0, 10, 20, 1, 11, 21, 2, 12, 22]
>>> d1 = Dataset.from_dict({"a": [0, 1, 2]})
>>> d2 = Dataset.from_dict({"a": [10, 11, 12, 13]})
>>> d3 = Dataset.from_dict({"a": [20, 21, 22, 23, 24]})
>>> dataset = interleave_datasets([d1, d2, d3])
>>> dataset["a"]
[0, 10, 20, 1, 11, 21, 2, 12, 22]
>>> dataset = interleave_datasets([d1, d2, d3], stopping_strategy="all_exhausted")
>>> dataset["a"]
[0, 10, 20, 1, 11, 21, 2, 12, 22, 0, 13, 23, 1, 10, 24]
>>> dataset = interleave_datasets([d1, d2, d3], probabilities=[0.7, 0.2, 0.1], seed=42)
>>> dataset["a"]
[10, 0, 11, 1, 2]
>>> dataset = interleave_datasets([d1, d2, d3], probabilities=[0.7, 0.2, 0.1], seed=42, stopping_strategy="all_exhausted")
>>> dataset["a"]
[10, 0, 11, 1, 2, 20, 12, 13, ..., 0, 1, 2, 0, 24]
For datasets in streaming mode (iterable):
>>> from datasets import interleave_datasets
>>> d1 = load_dataset('allenai/c4', 'es', split='train', streaming=True)
>>> d2 = load_dataset('allenai/c4', 'fr', split='train', streaming=True)
>>> dataset = interleave_datasets([d1, d2])
>>> iterator = iter(dataset)
>>> next(iterator)
{'text': 'Comprar Zapatillas para niña en chancla con goma por...'}
>>> next(iterator)
{'text': 'Le sacre de philippe ier, 23 mai 1059 - Compte Rendu...'
```
)r )r z/Unable to interleave an empty list of datasets.aExpected a list of Dataset objects or a list of IterableDataset objects, but element at position is an empty dataset dictionary.Dataset at position has at least one split: N
Please pick one to interleave with the other datasets, for example: dataset[''] is a .Unable to interleave a (at position 0) with a (at position K). Expected a list of Dataset objects or a list of IterableDataset objects.z: is not supported. Please enter a valid stopping_strategy.)r r r r )
arrow_datasetr iterable_datasetr
ValueError enumerate
isinstancer r r nextitertype__name__r r
) r r r r r r r r idataset dataset_type
other_types &&&&&& r# interleave_datasetsr@ s R '1 JKK )
'_#=>>'K1D#EFF ${|}{~ : : !*1#-FtG}o Vddhimnuivdwcxxz| stusvv| ~B CJ ~K ~T ~T }U UV W
6.8.J.J*Q`Pj
%L* G\22 ),*?*?)@@XYcYlYlXmm{|}{~ J K
) *. ii -..hijjw - TE
- /
r% c
V ^8 d QhR\ \ , R\ \ , R\ \ , R\
R\ /# )r dsetsr r axisr )r r r r
r r! )r" s "r# r$ r$ sW @X @X @X
;
@X J @X @X
@Xr% c
V ' g \ R4 h\ V 4 EF w rE\ V\ \ 34 ' g \ V\
\ 34 ' dH V' g \ RV R24 h\ RV R\ V4 R\ \ V4 4 R24 h\ RV R\ V4 P R 24 hV^ 8X d3 \ V\ 4 ' d
\ \ 3M \ \ 3w rgK \ VX4 ' d K \ R
VP R XP R V R
24 h X\ J d \ WW#R7 # \ WW#R7 # )a
Concatenate several datasets (sources) into a single dataset.
Use axis=0 to concatenate vertically (default), or axis=1 to concatenate horizontally.
Note for iterable datasets:
* if axis=0, the resulting dataset's `num_shards` is the sum of each dataset's `num_shards`.
* if axis=1, the resulting dataset has one (1) shard to not misalign data.
Args:
dsets (`List[datasets.Dataset]` or `List[datasets.IterableDataset]`):
List of Datasets to concatenate.
info (`DatasetInfo`, *optional*):
Dataset information, like description, citation, etc.
split (`NamedSplit`, *optional*):
Name of the dataset split.
axis (`{0, 1}`, defaults to `0`):
Axis to concatenate over, where `0` means over rows (vertically) and `1` means over columns
(horizontally).
Example:
```py
>>> ds3 = concatenate_datasets([ds1, ds2])
```
z0Unable to concatenate an empty list of datasets.r' r( r) r* r+ r, r- r. r/ r0 r1 r2 )r r rC )r5 r6 r7 r r r r r r8 r9 r: r; r r )rB r r rC r< r= r> r? s &&&& r# concatenate_datasetsrE s H KLL &
'G_#=>>'K1D#EFF ${|}{~ : : !*1#-FtG}o Vddhimnuivdwcxxz| stusvv| ~B CJ ~K ~T ~T }U UV W
6.8'.J.J/*Q`biPj
%L* G\22 ),*?*?)@@XYcYlYlXmm{|}{~ J K
) '. w .uuXX-eeWWr% )NNNNr )NNr ) typingr r r3 r r r dataset_dictr r r r
r4 r r r
splitsr utilsr utils.py_utilsr
get_loggerr; loggerr r@ rE r% r# rN sZ $ c c : l l #
H % mWo> S
l@X @Xr%