+ QV-j:^RIt^RIt^RIt^RIt^RIt^RIt^RIt^RI t ^RI t ^RI t ^RI H t ^RIHtHt^RIHtHt^RIHt^RIHt^RIHtHt^RIHt^R IHtHtHtHtH t H!t!^RI"t#^RI$t%^RI&t'^RI(t)^RI*t+^RI,H-t.^RI/H0t1^R I2H3t3H4t4H5t5H6t6H7t7^R I8H9t9^R I:H;t;^R It>^RI?H@t@HAtAHBtBHCtC^RIDHEtE^RIFHGtGHHtHHItIHJtJHKtKHLtLHMtMHNtNHOtO^RIPHQtQHRtRHStSHTtTHUtUHVtV^RIWHXtX^RIYHZtZ^RI[H\t\H]t]H^t^^RI_H`t`HataHbtbHctcHdtdHete^RIfHgth^RIiHjtj^RIkHltlHmtmHntn^RIoHptpHqtqHrtrHsts^RItHutu]>P];P!R48d^RIxHyty^RIzH{t{H|t|MRtyRt{]7t|]'d^RI}t}^RI~t^RIt^RIt^RIHt]j!]4t]!]]]]]3,R3,tR tR!R"ltR#R$ltRtR%R&lltR'R(ltR)R*ltRuR+R,lltR-R.lt!R/R04t!R1R2]4t!R3R4]4t!R5R6]4t!R7R8]4t!R9R:]4t!R;R<]4t!R=R>]4tR?R@lt!RARB]4t!RCRD]4tRERFlt!RGRH]4tRIRJltRKRLltRMRNlt!RORP]4t!RQRR]4t!RSRT]4t!RURV]4t!RWRX]4t!RYRZ]4tR[R\lt]!R]R^44t!R_R`]4t]!RaRb44tRctRdRelt!RfRg4t!RhRi]A4tRvRjRklltRwRlRmlltRnRoltRptRqtRrRsltR#)xN)Counter)IterableIterator)copydeepcopy) dataclass)partial)cycleislice)Path) TYPE_CHECKINGAnyBinaryIOCallableOptionalUnion) CommitInfoCommitOperationAddHfApi HfFileSystemHfFileSystemResolvedPath)RepositoryNotFoundError)version) __version__config)DatasetDatasetInfoMixin_push_to_bucket _push_to_repoFeatures) FeatureTypeListValue_align_features!_check_if_features_can_be_aligned%_fix_for_backward_compatible_features_visitcast_to_python_objectsrequire_decoding)ArrowFormatterPythonFormatterTableFormatterTensorFormatterget_format_type_from_alias get_formatter) DatasetInfo) _split_re) NamedSplitSplit SplitInfo)(_batch_accumulate_arrow_table_by_columns_batch_arrow_tablecast_table_to_featuresembed_table_storageread_schema_from_file table_cast)tqdm) get_logger)Literalconvert_file_size_to_intiflatmap_unordered)_merge_gen_kwargs_number_of_shards_in_gen_kwargs_shuffle_gen_kwargs_split_gen_kwargs)PathLikez1.6.0)BucketNotFoundError)HfFileSystemResolvedBucketPath"HfFileSystemResolvedRepositoryPathKey BuilderKeycV#N)xs&j/Volumes/fast/ai/experiments/ui-tars-smoke/.venv/lib/python3.14/site-packages/datasets/iterable_dataset.py identity_funcrQhs HcRV^8dQhR\R\\\3,/#)examplecolumn_mappingdictstr)formats"rP __annotate__r[ls"   d38n rRc a\;QJdV3RlV4F 'gK RM RM!V3RlV44'dO\R\V4 R\VP44 R\ V4\ S4, R24h\;QJd-V3RlVP44F 'gK RM" RM!V3RlVP444'd]\R\V4 R\VP44 R\ S4\ VP44, R24hVP 4UUu/uFwr#VSV,bK upp#uuppi) c3,<"TF qS9xK R#5irMrN.0colrUs& rP %_rename_columns_fn..ms 8#g TFzError when renaming  to z : columns z are not in the dataset.c3,<"TF qS9xK R#5irMrNr^s& rPrarbqs =%%\PPV4pVe,\V\P!VP 44p\P!VP4# \ \P\P3dLLi;irM) paTable from_pydictr;schematype TypeError ArrowInvalidArrowNotImplementedErrorr!from_arrow_schema)rwrxpa_tables&& rP_infer_features_from_batchrsxxx##E*H !(BIIl6G6G,HIH  % %hoo 662??B,G,GH   s*A//*BBcV^8dQhR\\\\3,,R\\\3,/#)rTexamplesry)rhrXrYr)rZs"rPr[r[s/##d38n!5#$sDy/#rRc VUUu/uF qFq"RbK K pppVUUu.uF"q Uu.uFqPV4NK upNK$ ppp\\W444#uuppiuupiuuppirM)getrXzip)rrUr`colsarrayss& rP_examples_to_batchrsh&. AX'#ICXD ADH IDSx8xG{{3x8DF I D! "" B8 IsA A+A& A+&A+cV^8dQhR\\\3,R\\\\3,,/#)rTrwry)rXrYrhrr)rZs"rPr[r[s/>>d39o>(4S>2J>rRc#"\V4^8Xd^M#\V\\V44,4p\V4F0pVP 4UUu/uFwr4W4V,bK uppxK2 R#uuppi5i)z3Convert a batch (dict of examples) to examples listN)lennextiterrangerk)rw n_examplesir`arrays& rP_batch_to_examplesrs^%jAo3uT$u+5F/G+HJ : /4{{}=}s!H}}===sABA:.Bc V^8dQhR\\\\3,,R\R\ R\ \\\P3,,/#)rTiterable batch_sizedrop_last_batchry) rtuplerJrXrsboolrr{r|)rZs"rPr[r[sVffuS$Y'(fffeCM"# frRc #"VeV^8:dER\PP\VUUu.uFwr4VNK uppRR743xR#\ V4pVFwrd\ WQ^, 4pWd3.\ V4,p\V4V8d V'dR#\V!wrRPRV 44p V \PP\V RR743xK R#uuppi5i)aConvert and group examples in Arrow tables of size `batch_size`. Args: iterable (`Iterable[Tuple[Key, dict]]`): An examples iterable containing tuples (example_key, example) of type (int/str, dict) batch_size (`Optional[int]`): Size of each sub-table to yield. If None or <= 0, yields the full table. drop_last_batch (`bool`, defaults to `False`): Drop the last batch if it is smaller than `batch_size`. NallT)only_1d_for_numpy_c38"TFp\V4xK R#5irMrYr_keys& rPra$_convert_to_arrow..s4t3s88t) r{r| from_pylistr)rr rhrrjoin) rrrrrUiteratorriterator_batchkey_examples_listkeysrnew_keys &&& rP_convert_to_arrowrsZ1_  HH !7S[8\S[ZQS[8\pt!u v   H~H q.9!^,tN/CC  !J .? /0((4t44rxx++,B8_c,deee! 9]s0D C<A"D"A Dc,V^8dQhRRR\RR/#)rT ex_iterable_BaseExamplesIterablevalueryrs)rZs"rPr[r[s# - -(? - -Pg -rRc$aaVV3RloS!V4#)zWe need to go through the ex_iterables recursively, create a new seed and return a new iterable, then set it to the containing ex_iterable.c<\VR4'dVPS4p\VR4'dS!VP4Vn\VR4'd(VPUu.uF pS!V4NK upVnV#uupi) shift_rngsr ex_iterables)hasattrrrr)reiset_seed_recursivelyrs& rPr4shift_ex_examples_rngs..set_seed_recursivelys~ ; - -%007K ; . .&:;;R;R&SK # ; / /KVKcKc'dKcR(._inner_load_state_dicts}$E4)@)@$C!7 IN!SEJ% &:eT+B+Bs5z*A5eh MEH+ I& &rR)rr)rr rs&&@rPload_state_dict%_BaseExamplesIterable.load_state_dicts) ' %d&6&6 CCrRc <V^8dQhRS[/#rr)rZrs"rPr[rsooDorRcfVP'd\VP4#\R4h)zPState dict is not initialized, please call ex_iterable._init_state_dict() first.)rr RuntimeErrorrs&rPr  _BaseExamplesIterable.state_dicts+    D,,- -mnnrRc\VR4'd VP#\VR4'dVP.M VPp\;QJdRV4F 'gK R# R#!RV44#)_sleep_on_threads_shutdownrc38"TFqPxK R#5irMsleep_on_threads_shutdownr_rs& rPraB_BaseExamplesIterable.sleep_on_threads_shutdown..s]P\<9!! n  -2-89  !45vs C&D!-4D!cL<V^8dQhRS[PPRR/#)rTrryr,r)rZrs"rPr[r1Cs&  bii.A.A FX rRc\VP\\V4VP4VP VP 4#rM)r,r.rCrr/r0rrs&&rPr%ExamplesIterable.shuffle_data_sourcesCs=  % %  3T[[ A  ( (  * *   rRc*<V^8dQhRS[RS[RR/#)rTrrryr,r)rZrs"rPr[r1Ks#   S    Rd  rRc \VPVPR7pVPWVR7p\ VUu.uF qdV,NK up4p\ VP VVPVP4#uupiKeep only the requested shard.rAr) rDr/rrrAr,r.r0rrrrrgen_kwargs_list shard_indicesrrequested_gen_kwargss&&&& rPr#ExamplesIterable.shard_data_sourcesKs}+DKKdooV:::Yc:d 0m1\m!2D2Dm1\]  % %  ( (  * *   2]sBc<V^8dQhRR/#)rTryr,rN)rZrs"rPr[r1Ws  &8 rRc  VP'g7\VPVPVPVP4#\ VPVP R7p\VUUu.uFpVP!R/VBFpVNK K upp4p\VPW0PVP4#uuppiz)Split shars into more shards if possible.rArN)r0r,r.r/rrDrrArrQrDnew_gen_kwargss& rPr%ExamplesIterable.reshard_data_sourcesWs+++#))4;;8T8TVZVtVt ,DKKdooV*#2 "1J&*&B&B&PZ&PN&P"1    % %~7S7SUYUsUs   s4$C c <V^8dQhRS[/#rr)rZrs"rPr[r1j<rRc#\"\4pVP'dVPR,M^p\\VPVP R7VR4EFEpVP'dVPR,M^p^pVP !R/VBFwrgV\V4,V8:dV\V4, pK0VP\PR7FkpVPV4p \V 4FHp WT8d7VP'd VPR;;,^, uu&Wj3xV^, pKJ Km K VP'gEKVPR;;,^, uu&^VPR&EKH R#5i)r:rANr; max_chunksizerN) r,rr rDr/rrgr to_readerr'ARROW_READER_BATCH_SIZE_IN_DATASET_ITER format_batchr) r formatterrC gen_kwagsrEr;rr pa_subtableformatted_batchrUs & rPrArrowExamplesIterable.__iter__sf#% ;?;K;K;K$**;7QR 1$++DOO \^mostIOSO_O_O_d&6&67J&Kef # ! !%!8!8!E9!E $s8}48OO%X6%#+#5#5FDrDr#5#sK&/&<&<[&IO#5o#F,G#/// $ 0 01D E J E"%,.)Q.) $G$t "F  -2-89  !45#usD&F,)AF,75F,c#"VP'dVPR,M^p\\VPVPR7VR4FpVP'dVPR,M^p^pVP !R/VBF^wrVV\ V4, pWC8:dKVP'd)VPR;;,\ V4, uu&WV3xK` VP'gKVPR;;,^, uu&^VPR&K R#5ir@)rr rDr/rrgr)rrCrxrEr;rrs& rPrl!ArrowExamplesIterable._iter_arrows;?;K;K;K$**;7QR 1$++DOO \^mostIOSO_O_O_d&6&67J&Kef # ! !%!8!8!E9!E !S]2!$?###$$%89S]J9m# "F  -2-89  !45usCD>AD> 4D>cL<V^8dQhRS[PPRR/#)rTrryrer)rZrs"rPr[rhs&  bii.A.A F] rRc\VP\\V4VP4VP VP 4#rM)rergrCrr/r0rrs&&rPr*ArrowExamplesIterable.shuffle_data_sourcess=$  # #  3T[[ A  ( (  * *   rRc*<V^8dQhRS[RS[RR/#)rTrrryrer)rZrs"rPr[rhs#  S  Ri rRc \VPVPR7pVPWVR7p\ VUu.uF qdV,NK up4p\ VP WpPVP4#uupirM) rDr/rrrArergr0rrPs&&&& rPr(ArrowExamplesIterable.shard_data_sourcessz+DKKdooV:::Yc:d 0m1\m!2D2Dm1\]$  # #%9;W;WY]YwYw  2]sBc<V^8dQhRR/#)rTryrerN)rZrs"rPr[rhs  &= rRc  VP'g,\VPVPVP4#\ VPVP R7p\ VUUu.uFpVP!R/VBFpVNK K upp4p\VPW0PVP4#uuppirW)r0rergr/rDrrArrXs& rPr*ArrowExamplesIterable.reshard_data_sourcess+++()@)@$++tOkOkl l+DKKdooV*#2 "1J&*&B&B&PZ&PN&P"1  %  # #^5Q5QSWSqSq   s)$C c <V^8dQhRS[/#rr)rZrs"rPr[rhr\rRc,\VP4#rMr^rs&rPr ArrowExamplesIterable.num_shardsr`rR)rrr0rgr/rar!)r"r#r$r%rr'rrrrlrrrrr(r)rbrcs@@rPrerenspDD"    :,:       <<#>#>$B]B]B]tgcggrRc.VPP#rMrrrs&rPr'RebatchedArrowExamplesIterable.is_typed(((rRc.VPP#rMrrrs&rPr'RebatchedArrowExamplesIterable.featuresrrRc <V^8dQhRS[/#rr)rZrs"rPr[r   $  rRc RVPP4RRR^R^R^RVPP/VnVP#)examples_iterableprevious_stateN batch_idxnum_chunks_since_previous_statecropped_chunk_lengthrrrr4r"rrs&rPr/RebatchedArrowExamplesIterable._init_state_dictsQ !1!1!B!B!D d  -q "A DNN++  rRc#:"VPRjxL R#L5irM)rrs&rPr'RebatchedArrowExamplesIterable.__iter__s####s cZ<V^8dQhRS[S[S[S[P3,,/#rrrrJr{r|)rZrs"rPr[rs,X@X@XeCM&:;X@rRc# "VP'dFVPR,'d-VPPVPR,4VPP'dVPP4pM5VP'd\ VP^R7pM \ R4hVPeVP^8:dVP'dVPR,^8dR#\P!VUUu.uFwr#VNK upp4pVP'd^VPR&RV3xR#.p.p^pVP'dVPR,M^pVP'dVPR,M^p VP'd)VPP4p WPR&VEFwr\VPVPR 74EFUwrV^8d V^,pKV^8XdV ^8Xd V^,pK1V^8Xd.V ^8d'V PV \V 4V , 4p ^p^p \V 4^8XdKwV\V 4,VP8d7VPV 4VPV 4V\V 4, pKV\V 4,VP8XEd VPV 4VPV 4R P!R V44pVP'dWVPR;;,^, uu&VPR;;,\V4, uu&^VPR&V\P"P%V43x.p.p^pVP'd)X VPR&V ^,VPR&EKEKVPV, pVPV R V R 24VPV P^V44R P!RV44pVP'dVVPR;;,^, uu&VPR;;,\V4, uu&WPR&V\P"P%V43xV RV R2.pV PV\V 4V, 4.p\V 4V, pVP'gEK8X VPR&WPR&EKX VP'gEKVPP4p EK VP&'gV'dR P!RV44pVP'dMX VPR&VPR;;,^, uu&^VPR&^VPR&V\P"P%V43xR#R#R#uuppi5i)z-Iterate over sub-tables of size `batch_size`.rrz_iter_arrow is not available in RebatchedArrowExamplesIterable, use an examples iterable that implements _iter_arrow() or pass force_convert_to_arrow=TrueNrrrrrrrc38"TFp\V4xK R#5irMrr__keys& rPra=RebatchedArrowExamplesIterable._iter_arrow../&I[Ts4yy[rz[:]c38"TFp\V4xK R#5irMrrs& rPrar?rr[z:]c38"TFp\V4xK R#5irMrrs& rPrarNsA[Ts4yy[r)rrrrrrrrr{ concat_tablesr  enumeratertslicerappendrr| from_batchesr)rrrr all_pa_table keys_buffer chunks_bufferchunks_buffer_sizenum_chunks_to_skipchunk_length_to_croprrrchunkrrs& rPrl*RebatchedArrowExamplesIterable._iter_arrows     0 01A B B    , ,T-=-=>N-O P    & & &''224H  ( ( (()9)9aHHm  ?? "doo&:D$4$4[$AA$E++,R+!X,RSL01  -% %   TXTdTdTdT--.OPjkKOK[K[K[t//0FGab    !--88:N1?  - .%MC:CHDVDVeietetDVDu:v6/%)&!+&'1,1E1J&!+&'1,1E1I!KK(_`E)*&+,(u:?%E 2T__D&&s+!((/&#e*4&'#e*4G&&s+!((/!hh&I[&IIG'''((5:5(()JKsS`OaaKCD(()?@!288#8#8#GGG"$K$&M)*&'''=K(()9:NmpqNq(()JK(,0??=O+O(&&#b1E0Fa'HI!((Q8L)MN!hh&I[&IIG'''((5:5(()JKsS`OaaKCW(()?@!288#8#8#GGG&)U!,@+A#D"EK%*[[1Es5zThGh%i$jM),U6J)J&'''=K(()9:Nm(()JKc;wd!%!1!1!i&j### hhA[AAG5C  !12  -2-FG  !BC;<  !7828800?? ?)6#E-Ssw+Y,AY,6,Y,#AY,7/Y,& Y& 3Y, .Y,<&Y,#&Y, FY,BY,,BY, CY,1Y,0Y,?Y,)Y,1A;Y,cL<V^8dQhRS[PPRR/#)rTrryrr)rZrs"rPr[rVs&  bii.A.A Ff rRc\VPPV4VPVPVP 4#rM)rrrrrrrs&&rPr3RebatchedArrowExamplesIterable.shuffle_data_sourcesVs<-    1 1) < OO   ' '   rRc*<V^8dQhRS[RS[RR/#)rTrrryrr)rZrs"rPr[r^#  S  Rr rRc\VPPWVR7VPVPVP 4#rO)rrrrrrrs&&&&rPr1RebatchedArrowExamplesIterable.shard_data_sources^sA-    / / j / Y OO   ' '   rRc<V^8dQhRR/#)rTryrrN)rZrs"rPr[rfs  &F rRc\VPP4VPVPVP 4#rM)rrrrrrrs&rPr3RebatchedArrowExamplesIterable.reshard_data_sourcesfs8-    1 1 3T__dFZFZ\`\w\w  rRc <V^8dQhRS[/#rr)rZrs"rPr[rl++C+rRc.VPP#rMrrrs&rPr)RebatchedArrowExamplesIterable.num_shardsk***rR)rrrrr)FFr!r"r#r$r%rr'rrrrrrlrrrrr(r)rbrcs@@rPrrs = =hh))))   $X@X@t      +++rRrcaa]tRtRtoV3RlV3Rllt]R4t]R4t]R4tV3RlRlt R t V3R lR lt V3R lR lt RV3RlRllt V3RlRlt]V3RlRl4tRtVtV;t#)SelectColumnsIterableipc6<V^8dQhRS[RS[S[,/#)rTr column_names)rrhrY)rZrs"rPr["SelectColumnsIterable.__annotate__qs ))$9)c)rRc<<\SV`4WnW nR#rM)r3rrr)rrrr4s&&&rPrSelectColumnsIterable.__init__qs &(rRcVVPP'd VP#R#rMrrrlrs&rPr SelectColumnsIterable.iter_arrowvs%    & & &## # 'rRc.VPP#rMrrs&rPrSelectColumnsIterable.is_typed{rrRc.VPP#rMrrs&rPrSelectColumnsIterable.featuresrrRc <V^8dQhRS[/#rr)rZrs"rPr[rr7rRcXVPP4VnVP#rMrrrrs&rPr&SelectColumnsIterable._init_state_dict%++<<>rRc#"VPF*wrYPUu/uF q3W#,bK up3xK, R#uupi5irM)rr)rrprowcs& rPrSelectColumnsIterable.__iter__s>((HC+<+<=+>XeCM&:;>rRc#"VPP4F5wr\V4^8gKWPVP43xK7 R#5irN)rrrselectr)rrprs& rPrl!SelectColumnsIterable._iter_arrowsD!--88:MC8}q ??4+<+<===;s -A$AcL<V^8dQhRS[PPRR/#)rTrryrr)rZrs"rPr[rs*jjbii.A.AjF]jrRc`\VPPV4VP4#rM)rrrrrs&&rPr*SelectColumnsIterable.shuffle_data_sourcess'$T%5%5%J%J9%UW[WhWhiirRc*<V^8dQhRS[RS[RR/#)rTrrryrr)rZrs"rPr[rs#  S  Ri rRcd\VPPWVR7VP4#r)rrrrrs&&&&rPr(SelectColumnsIterable.shard_data_sourcess1$    / / j / Y[_[l[l  rRc<V^8dQhRR/#)rTryrrN)rZrs"rPr[rsaa&=arRc^\VPP4VP4#rM)rrrrrs&rPr*SelectColumnsIterable.reshard_data_sourcess$$T%5%5%J%J%LdN_N_``rRc <V^8dQhRS[/#rr)rZrs"rPr[rrrRc.VPP#rMrrs&rPr SelectColumnsIterable.num_shardsrrR)rrrr!rrcs@@rPrrps)) $$))))  >>> jj  aa+++rRrcaa]tRtRtoV3RlV3Rllt]R4t]R4t]R4tV3RlRlt R t R t V3R lR lt RV3R lRllt V3RlRlt]V3RlRl4tRtVtV;t#)StepExamplesIterableic,<V^8dQhRS[RS[RS[/#)rTrstepoffset)rrs)rZrs"rPr[!StepExamplesIterable.__annotate__s#$9crRcH<\SV`4WnW nW0nR#rM)r3rrrr)rrrrr4s&&&&rPrStepExamplesIterable.__init__s &  rRcVVPP'd VP#R#rMrrs&rPrStepExamplesIterable.iter_arrow$#'#3#3#>#>#>tHDHrRc.VPP#rMrrs&rPrStepExamplesIterable.is_typedrrRc.VPP#rMrrs&rPrStepExamplesIterable.featuresrrRc <V^8dQhRS[/#rr)rZrs"rPr[r  $ rRcRVPP4R^RVPP/VnVP#)rsteppedrrrs&rPr%StepExamplesIterable._init_state_dictsB !1!1!B!B!D q DNN++  rRc#"\VP4p\\WP44p\ V4VP 8dW P ,xKOR#5iTN)rrrhr rrr)r ex_iteratorrws& rPrStepExamplesIterable.__iter__sK4++,  YY78E5zDKK'KK((sA'A)c #&"VP'dVPR,M^pVPP4Fwr#VP\P !\ VPV, VP,\V4VP4\P!4R74pV\V4,VP,pVP'dWPR&W$3xK R#5i)r rN) rrrtaker{rrrrrint64)rr rrstepped_pa_tables& rPrl StepExamplesIterable._iter_arrows151A1A1A$""9-q!--88:MC'}} g 5BCMSWS\S\]dfdldldno  X.$));G.5  +' ';sDDcL<V^8dQhRS[PPRR/#)rTrryrr)rZrs"rPr[rs&  bii.A.A F\ rRcx\VPPV4VPVPR7#rr)rrrrrrs&&rPr)StepExamplesIterable.shuffle_data_sourcess2#    1 1) <499UYU`U`  rRc*<V^8dQhRS[RS[RR/#)rTrrryrr)rZrs"rPr[rs#  S  Rh rRc|\VPPWVR7VPVPR7#)rOr)rrrrrrs&&&&rPr'StepExamplesIterable.shard_data_sourcess6#    / / j / Y;;  rRc<V^8dQhRR/#)rTryrrN)rZrs"rPr[rs  &< rRcv\VPP4VPVPR7#r)rrrrrrs&rPr)StepExamplesIterable.reshard_data_sourcess/#    1 1 3;;  rRc <V^8dQhRS[/#rr)rZrs"rPr[rrrRc.VPP#rMrrs&rPrStepExamplesIterable.num_shardsrrR)rrrrr!rrcs@@rPrrs II))))   (      +++rRrcaa]tRtRtoRV3RlV3Rlllt]R4t]R4t]R4tRt V3RlR lt R t R t V3R lR lt ]V3RlRl4tRV3RlRlltV3RlRltRtVtV;t#)#CyclingMultiSourcesExamplesIterableicD<V^8dQhRS[S[,RS[R,/#)rTrstopping_strategyfirst_exhausted all_exhausted!all_exhausted_without_replacement)rhrr>)rZrs"rPr[0CyclingMultiSourcesExamplesIterable.__annotate__s-  01 # S  rRc<\SV`4WnW nVR9d\P VnR#\P VnR#)r,N)r,r-)r3rrr)rrrfbool_strategy_func)rrr)r4s&&&rPr,CyclingMultiSourcesExamplesIterable.__init__sF (!2 ),bbBFF ikioio rRc<VP^,P#rrrrs&rPr,CyclingMultiSourcesExamplesIterable.is_typed  #,,,rRc<VP^,P#r3rrrs&rPr,CyclingMultiSourcesExamplesIterable.featuresr6rRc\;QJd&RVP4F 'dK RM RM!RVP44'd VP#R#)c38"TFqPxK R#5irMrrs& rPraACyclingMultiSourcesExamplesIterable.iter_arrow..s&cQb+'='=QbrFTNrrrlrs&rPr.CyclingMultiSourcesExamplesIterable.iter_arrowsD$'3&cQUQbQb&c333&cQUQbQb&c#c#ctmimmrRc #("VP'dVPR,M^p\\\\ VP 444V^,R4F)pVP'dW PR&VxTpK+ R#5iex_iterable_idxN)rr r rrr)rrBnext_ex_iterable_idxs& rP_get_indices_iterator9CyclingMultiSourcesExamplesIterable._get_indices_iterators|AEAQAQAQ$**+<=WX$*5s4;L;L7M1N+OQ`cdQdfj$k 6J  !23! !2O %lsBBc <V^8dQhRS[/#rr)rZrs"rPr[r.rrRc VPFpVP4K R^RR.\VP4,RR.\VP4,RVPP/VnVP #)rBprevious_statesN is_exhaustedFr)rrrr4r"rrrs& rPr4CyclingMultiSourcesExamplesIterable._init_state_dictsz,,K  ( ( *- q vD,=,=(>> UGc$*;*;&<< DNN++   rRc#( "R.\VP4,pVP'd\\VP44F[pVPR,V,fK!VPV,P VPR,V,4K] VPUu.uFq3P 4NK ppVPUu.uFq3P 4NK ppRp\PP\VP4R7pVUu.uFqPWh4NK p pVP4p VP'd(\P!VPR,4M)\P!\VP4R4p V EF^pVPV 4'dEMEW,'dVP R9dK?W,fW,P#4W&VP'dAXV,VPR,V&VPV,P 4WB&VPWeV,4W&W,p W,P#4W&VP'dAXV,VPR,V&VPV,P 4WB&W,RJdVPWeV,4W&MRW&VP'dRVPR,V&VP R9dVP'd\VPV,P%4VPV,P 4XV&RVPR,V&VPV,P 4WR&RW&VPWeV,4W&V RJgEKZV xEKa V Fp V P#4K V'dVP'4p?K\(;QJd&RVP4F 'gK RM RM!RVP44'd'\*P,!\.P04R#R#uupiuupiuupi T Fp T P#4K T'dTP'4p?K\(;QJd&RTP4F 'gK RM RM!RTP44'd&\*P,!\.P04ii;i5i) NrHc\VR4#rrrs&rPfetch_next_sampleJCyclingMultiSourcesExamplesIterable._iter_arrow..fetch_next_sample'%( (rR max_workersrIFTc38"TFqPxK R#5irMrrs& rPraBCyclingMultiSourcesExamplesIterable._iter_arrow..c^L][88L]rr-)rrrrrr r concurrentfuturesThreadPoolExecutorsubmitrDrrfullr0r)resultrpoprftimesleeprSLEEP_TIME_ON_THREADS_SHUTDOWNrnextsrrrH iteratorsrPexecutorrrZindices_iteratorrIr^futures& rPrl/CyclingMultiSourcesExamplesIterable._iter_arrowsXT..//    3t0012##$56q9E%%a(889I9IJ[9\]^9_`3LPK\K\]K\K557K\O]AEARARSAR+++-AR S )%%88SIZIZE[8\PYZPYH??#4?PYZ557;?:J:J:JBHHT%%n5 6PRPWPWX[\`\m\mXnpuPv 1 B%**<88??t'='=Af'f8#&z002EH'''APQRAS(():;A>-1->->q-A-L-L-N*!)1BaL!QGJ":,,.###=LQ=OD$$%67:)-):):1)=)H)H)JO&85(!)1BaL!QGJ'+LO'''>B((8;--5ZZ+++ --a0AAC151B1B11E1P1P1ROA.EID,,->?B'+'8'8';'F'F'H #'%-__5FRS %U & LO&T" "$==?s^DL]L]^sss^DL]L]^^^ 6@@A_A^S[d" "$==?s^DL]L]^sss^DL]L]^^^ 6@@A_sA*V1A V=SV%S=.fetch_next_samplerrRrRrSrIFT)waitc38"TFqPxK R#5irMrrs& rPra?CyclingMultiSourcesExamplesIterable.__iter__..rWrrX)rrrrrr rrYrZr[r\rDrrr]r0r)r^rr_shutdownrfr`rarrbrcs& rPr,CyclingMultiSourcesExamplesIterable.__iter__fstT..//    3t0012##$56q9E%%a(889I9IJ[9\]^9_`3LPK\K\]K\K557K\O]:>:K:KL:K;T+&:K L )%%88SIZIZE[8\PYZPYH??#4?PYZ557;?:J:J:JBHHT%%n5 6PRPWPWX[\`\m\mXnpuPv 2 B%**<88??t'='=Af'f8#&z002EH'''APQRAS(():;A>-1->->q-A-L-L-N*!)1BaL!QGJ":,,.###=LQ=OD$$%67:)-):):1)=)H)H)JO&85(!)1BaL!QGJ'+LO'''>B((8;--5ZZ+++ --a0AAC151B1B11E1P1P1ROA.EID,,->?B'+D,=,=a,@'A #'%-__5FRS %U & LO&T" "$==?   4  (s^DL]L]^sss^DL]L]^^^ 6@@A_C^L[d" "$==?   4  (s^DL]L]^sss^DL]L]^^^ 6@@A_sA*V-1A V-=S V-%S%9W[WhWhiWh 88CWh i2.O=Nk))=Nr)rrrs&rPr.CyclingMultiSourcesExamplesIterable.num_shardss*SWSdSdSdsOT=N=NOOkjkkrRc*<V^8dQhRS[RS[RR/#)rTrrryr'r)rZrs"rPr[r.s%&) .rRc  WP8dC\VPUu.uFpVPWVR7NK upVPR7#W P8dM\VPUu.uF pVPVPW#R7NK" upVPR7#\.VPR7#uupiuupi)rrOr))rr'rrr)rrrrrs&&&& rPr6CyclingMultiSourcesExamplesIterable.shard_data_sourcess  '6%)$5$5$5// j/Y$5#'"8"8  __ $6%)$5$5$5///^$5#'"8"8  7"&"8"8 s C6&C c<V^8dQhRR/#)rTryr'rN)rZrs"rPr[r.s  &K rRc\VPUu.uFqP4NK upVPR7#uupi)r)r'rrr)rrs& rPr8CyclingMultiSourcesExamplesIterable.reshard_data_sourcess<2=A=N=N O=N * * ,=N O"44  Os?)rr0rr))r+r!)r"r#r$r%rr'rrrrDrrlrrrrrr(r)rbrcs@@rPr'r's  $----nn3   IBVJBXYY ll4   rRr'caa]tRtRtoRtV3RlV3Rllt]R4t]R4t]R4t V3RlR lt R t R t V3R lR lt ]V3RlRl4tRV3RlRlltV3RlRltRtVtV;t#)2VerticallyConcatenatedMultiSourcesExamplesIterableias VerticallyConcatenatedMultiSourcesExamplesIterable simply chains the input iterables. It doesn't require the examples iterables to always yield the same columns. Instead, this is handled by the `IterableDataset` class or `FormattedExamplesIterable`. For information, `IterableDataset` merges the features of all the datasets to concatenate into one. We use `IterableDataset._resolve_features` to obtain the features of all the datasets to concatenate. Then for each example, `IterableDataset` and `FormattedExamplesIterable` automatically fill missing columns with None. This is done with `_apply_feature_types_on_example`. c0<V^8dQhRS[S[,/#rTrrhr)rZrs"rPr[?VerticallyConcatenatedMultiSourcesExamplesIterable.__annotate__))T*?%@)rRc0<\SV`4WnR#rMr3rrrrr4s&&rPr;VerticallyConcatenatedMultiSourcesExamplesIterable.__init__ (rRc<VP^,P#r3r4rs&rPr;VerticallyConcatenatedMultiSourcesExamplesIterable.is_typedr6rRc<VP^,P#r3r8rs&rPr;VerticallyConcatenatedMultiSourcesExamplesIterable.featuresr6rRc\;QJd&RVP4F 'dK RM RM!RVP44'd VP#R#)c3<"TFqPRJxK R#5irMr<rs& rPraPVerticallyConcatenatedMultiSourcesExamplesIterable.iter_arrow..sWEVk%%T1EVFTNr>rs&rPr=VerticallyConcatenatedMultiSourcesExamplesIterable.iter_arrowsA 3WTEVEVW333WTEVEVW W W## # XrRc <V^8dQhRS[/#rr)rZrs"rPr[rr rRcR^RVPUu.uFqP4NK upRVPP/VnVP#uupi)rBrrrrr4r"rrJs& rPrCVerticallyConcatenatedMultiSourcesExamplesIterable._init_state_dictsW q tO`O`aO` 99;O`a DNN++  bsA c#"VP'dVPR,M^p\VPVR4F@pTRjxL VP'gK!VPR;;,^, uu&KB R#L=5irA)rr rrex_iterable_idx_startrs& rPr;VerticallyConcatenatedMultiSourcesExamplesIterable.__iter__soGKGWGWGW 0 01B C]^!$"3"35JDQK" " "  !23q83R "sAB  B B "&B c#2"VP'dVPR,M^p\VPVR4FNpVP4RjxL VP'gK/VPR;;,^, uu&KP R#L=5irA)rr rrrs& rPrl>VerticallyConcatenatedMultiSourcesExamplesIterable._iter_arrow sxGKGWGWGW 0 01B C]^!$"3"35JDQK"--/ / /  !23q83R /sABBB0&BcL<V^8dQhRS[PPRR/#)rTrryrr)rZrs"rPr[rs+ ] ],, ] = ]rRc  \V4pVPUUu.uF<p\VP4F pVP VPVR7NK" K> pppVP V4\ V4#uuppi)zShuffle all shards.rr)rrrrrshuffler)rrrngrrsingle_shard_ex_iterabless&& rPrGVerticallyConcatenatedMultiSourcesExamplesIterable.shuffle_data_sourcessy! $00% 0 {556  * *k6L6LTY * Z6 [0 "% -.AB[\\ % sAA<c <V^8dQhRS[/#rr)rZrs"rPr[rsPPCPrRc:\RVP44#)c38"TFqPxK R#5irMryrs& rPraPVerticallyConcatenatedMultiSourcesExamplesIterable.num_shards..r{r)sumrrs&rPr=VerticallyConcatenatedMultiSourcesExamplesIterable.num_shardssOT=N=NOOOrRc*<V^8dQhRS[RS[RR/#)rTrrryrr)rZrs"rPr[r!s%     &)  =  rRc  *VPUUu.uF<p\VP4F pVPVPVR7NK" K> pppVP VXVR7p\ VUu.uF quV,NK up4#uuppiuupi)zKeep only the requested shardrrO)rrrrrr)rrrrrrrRrs&&&& rPrEVerticallyConcatenatedMultiSourcesExamplesIterable.shard_data_sources!s $00% 0 {556  * *k6L6LTY * Z6 [0 "% :::uYc:d A3@ A=aq ) )= A  %  Bs AB 2Bc<V^8dQhRR/#)rTryrrN)rZrs"rPr[r/s  &Z rRcp\VPUu.uFqP4NK up4#uupirM)rrrrs& rPrGVerticallyConcatenatedMultiSourcesExamplesIterable.reshard_data_sources/s3A=A=N=N O=N * * ,=N O  Os3rrr!)r"r#r$r%r&rr'rrrrrrlrrrrr(r)rbrcs@@rPrrs ))----$$  99 ] ]PP      rRrc:V^8dQhR\\,/#)rTr)rhrY)rZs"rPr[r[5s  d3i rRc0\V4p\;QJd*RVP44F 'dK RM RM!RVP444'g0VUu.uFq!V,^8gKVNK pp\RV R24hR#uupi)zBCheck the column names to make sure they don't contain duplicates.c3*"TF q^8HxK R#5irNrN)r_counts& rPra&_check_column_names..8s8'7ez'7sFTzAThe examples iterables can't have duplicated columns but columns z are duplicated.N)rrrirg)rcounterr`duplicated_columnss& rP_check_column_namesr5sl#G 38w~~'783338w~~'78 8 8-4IWc q8HccWIOPbOccs t   9Is &B:Bcaa]tRtRtoRtV3RlV3Rllt]R4t]R4t]R4t V3RlR lt R t R t V3R lR lt ]V3RlRl4tRV3RlRlltV3RlRltRtVtV;t#)4HorizontallyConcatenatedMultiSourcesExamplesIterablei?a  HorizontallyConcatenatedMultiSourcesExamplesIterable merges examples together for the input list of iterables. It also checks that there are no duplicate columns (otherwise we don't know which one to keep). This check is done once when yielding the first example. However it doesn't fill missing columns with None. Instead, this is handled by the `IterableDataset` class or `FormattedExamplesIterable`. For information, `IterableDataset` merges the features of all the datasets to concatenate into one. We use `IterableDataset._resolve_features` to obtain the features of all the datasets to concatenate. Then for each example, `IterableDataset` and `FormattedExamplesIterable` automatically fill missing columns with None. This is done with `_apply_feature_types_on_example`. c0<V^8dQhRS[S[,/#rr)rZrs"rPr[AHorizontallyConcatenatedMultiSourcesExamplesIterable.__annotate__OrrRc0<\SV`4WnR#rMrrs&&rPr=HorizontallyConcatenatedMultiSourcesExamplesIterable.__init__OrrRc\;QJd&RVP4F 'dK RM RM!RVP44'gi\VP4^8d[\;QJd&RVP4F 'dK RM RM!RVP44'd VP#R#)c3~"TF3p\V\4;'dVPPxK5 R#5irM)rrrrrs& rPraRHorizontallyConcatenatedMultiSourcesExamplesIterable.iter_arrow..Ws6#4K;(FGnnKLcLcLnLnn#4s==FTc38"TFqPxK R#5irMr<rs& rPrar[s2o]nk3I3I]nrN)rrrrlrs&rPr?HorizontallyConcatenatedMultiSourcesExamplesIterable.iter_arrowSss#'#4#4sss#'#4#4D%%&*ss2o]a]n]n2osss2o]a]n]n2o/o/o     rRc<VP^,P#r3r4rs&rPr=HorizontallyConcatenatedMultiSourcesExamplesIterable.is_typed_r6rRc<VP^,P#r3r8rs&rPr=HorizontallyConcatenatedMultiSourcesExamplesIterable.featurescr6rRc <V^8dQhRS[/#rr)rZrs"rPr[rgs  $ rRcRVPUu.uFqP4NK upRVPP/VnVP#uupi)rrrrJs& rPrEHorizontallyConcatenatedMultiSourcesExamplesIterable._init_state_dictgsR tO`O`aO` 99;O`a DNN++ bsAc #T"VPUu.uFp\V4NK pp\P!4Fp.p.p\ V4F3p\ V4wrxVP V4VP V4K5 V'dgV^8Xd%\VUU u.uF qFqNK K up p4/p VFpV PV4K RPRV44p W3xKR# R#uupi \dTPT4Ki;iuup pi5i)rrc38"TFp\V4xK R#5irMrrs& rPraPHorizontallyConcatenatedMultiSourcesExamplesIterable.__iter__..".rrN)rrrrrrhrrrrrrrrcolumns append_columnr)rrpa_table_iteratorsrr pa_tablespa_table_iteratorrrrjtable new_pa_tablerqr`rs& rPrl@HorizontallyConcatenatedMultiSourcesExamplesIterable._iter_arrows]PTPaPabPad;#9#9#;"6'5>hYRgRg;RgYh!*) 4HAAv', ),U-?-?)OID+3+A+A$+LL*P !5 ((")rZrs"rPr[8RandomlyCyclingMultiSourcesExamplesIterable.__annotate__sT + +01 +99&& + U , + # S +rRcR<\SV`W4\V4VnW0nR#rM)r3rrrr)rrrrr)r4s&&&&&rPr4RandomlyCyclingMultiSourcesExamplesIterable.__init__s$ 9!),*rRc$<V^8dQhRS[RR/#rTrryrr)rZrs"rPr[rs   (? rRc\VP4pVP^R4V, p\VP\ P PVR7VPVPR7#)rseed)rrrr)) rrintegersrrrr default_rngrr)rrrnew_seeds&& rPr6RandomlyCyclingMultiSourcesExamplesIterable.shift_rngssbt~~&<<7+e3:**ii+++:,,"44   rRc<VP^,P#r3r4rs&rPr4RandomlyCyclingMultiSourcesExamplesIterable.is_typedr6rRc<VP^,P#r3r8rs&rPr4RandomlyCyclingMultiSourcesExamplesIterable.featuresr6rRc#"\VP4p\VP4pRpVP'dVPR,M^pVP'd#VPR,VP nVPf\VP^W#R7VR4FjpV^,V,pVP'd9W@PR&V^8Xd$VP P VPR&\V4xKl K\VPW#VPR7VR4FjpV^,V,pVP'd9W@PR&V^8Xd$VP P VPR&\V4xKl K5i)bit_generator_index_offsetbit_generator_stateNsize)rp) rrrrr bit_generatorrrr rrschoice)rr num_sourcesrandom_batch_size index_offsetrs& rPrDARandomlyCyclingMultiSourcesExamplesIterable._get_indices_iteratorst~~&$++,  IMIYIYIYt''(DE_`    &*&6&67L&MC   #    % Q TVbdhiA$01$48I#IL'''IU(()EF'1,FIFWFWF]F]D,,-BCa&L jJJ{dFXFXJY[gimA%11$48I#IL'''IU(()EF'1,FIFWFWF]F]D,,-BCa&Ls A%G(EGc <V^8dQhRS[/#rr)rZrs"rPr[rs   $  rRc RVPFpVP4K RVPPPR^RR.\ VP4,RR.\ VP4,RVP P/VnVP#)rrrHNrIFr) rrrrrrr4r"rrJs& rPr>#?#?#E#E (! vD,=,=(>> UGc$*;*;&<< DNN++  rRcL<V^8dQhRS[PPRR/#)rTrryrr)rZrs"rPr[rs&  bii.A.A Fs rRc VPUu.uFq"PV4NK pp\VVVPVPR7#uupi)z;Shuffle the data sources of each wrapped examples iterable.rrr))rrrrr)rts&& rPr@RandomlyCyclingMultiSourcesExamplesIterable.shuffle_data_sourcessRW[WhWhiWh 88CWh i: ,,"44   jsA c*<V^8dQhRS[RS[RR/#)rTrrryrr)rZrs"rPr[r s%&) 6rRc  WP8dX\VPUu.uFpVPWVR7NK upVPVP VP 4#W P8db\VPUu.uF pVPVPW#R7NK" upVPVP VP 4#\.VPVP VP 4#uupiuupi)rrO)rrrrrrr)rs&&&& rPr>RandomlyCyclingMultiSourcesExamplesIterable.shard_data_sources s   '>%)$5$5$5// j/Y$5""&& __ $>%)$5$5$5///^$5""&& ?""&&  's D &D c<V^8dQhRR/#)rTryrrN)rZrs"rPr[r)s  &S rRc \VPUu.uFqP4NK upVPVPVP 4#uupi)r)rrrrrr)rs& rPr@RandomlyCyclingMultiSourcesExamplesIterable.reshard_data_sources)sL:=A=N=N O=N * * ,=N O NN     " "   OsA)rrr)Nr+r!)r"r#r$r%rrr'rrrDrrrrr(r)rbrcs@@rPrrst + +  ----!:     @   rRrc8V^8dQhR\P/#rr{r|)rZs"rPr[r[3s  bhh rRc\V\P4'dV#\V\P\P 34'd \PP V4#\P'dQR\P9d<^RI p\WPVP 34'dVP4#V#)polarsN) rr{r|pd DataFrameSeries from_pandasrPOLARS_AVAILABLEsysmodulesr-to_arrow)outputpls& rP_table_output_to_arrowr83s&"((## &2<<344xx##F++ 8s{{#: f||RYY7 8 8??$ $ MrRcaa]tRtRtoRV3RlV3Rlllt]R4t]R4t]R4tV3RlRlt R t R t RV3R lR llt V3R lRlt RV3RlRlltV3RlRlt]V3RlRl4tRtVtV;t#)MappedExamplesIterablei@c<V^8dQhRS[RS[RS[RS[S[S[,,RS[RS[S[,RS[RS[S[S[,,R S[S[,R S[R ,R S[S[,R S[S[,RS[/ #)rTrfunction with_indices input_columnsbatchedrrremove_columns fn_kwargs formattingFormattingConfigr/max_num_running_async_map_functions_in_parallel(is_batch_accumulate_arrow_table_function) rrrrrhrYrsrXr!)rZrs"rPr[#MappedExamplesIterable.__annotate__As4e4e*4e4e 4e  S * 4e  4eSM4e4e!c+4eD>4e/04e8$4e:B#4e374erRc <\SV`4WnW nWPnW`nWpnWnW0nW@n T ;'g/Vn Wn Wn T ;'g\PVnWnV 'EdUV P"'EdB\%V\&4'gT\)RV P*P-4 R\/V4P0 R\/V4P0 R24hVP2'gT\)RV P*P-4 R\/V4P0 R\/V4P0 R24hVP V'dTM^8wdY\)RV P*P-4 R\/V4P0 RV'dTM^ RVP : R2 4h.VnR#) zThe z -formatted z# has underlying iterable that is a z- instead of a RebatchedArrowExamplesIterable.z but doesnt' implement iter_arrow(), a possible fix could be to use RebatchedArrowExamplesIterable(..., force_convert_to_arrow=True).z has batch_size=z0 which is different from ex_iterable.batch_size=z from its underlying iterable.N)r3rrr<r?rrr@r=r>rArB _featuresr/MAX_NUM_RUNNING_ASYNC_MAP_FUNCTIONS_IN_PARALLELrDrEis_tablerrrg format_type capitalizerr"r_owned_loops_and_tasks)rrr<r=r>r?rrr@rArBrrDrEr4s&&&&&&&&&&&&&&rPrMappedExamplesIterable.__init__As &  $.,(*"b$! ; u uv?u?u <9a5 :*---k+IJJ :11<<>?{4PT:K^K^J_`!!%k!2!;!; <?{4PT:K^K^J_`!!%k!2!;!; <=__ '''JqI :11<<>?{4PT:K^K^J__oFpzLMpNN>&1&<&<%>>\^ ce#rRc~VP'd+VPP'd VP#R#R#rM)rBrJrlrs&rPr!MappedExamplesIterable.iter_arrowws- ???t777## # 8?rRcVPRJ#rMrrs&rPrMappedExamplesIterable.is_typed|s}}D((rRcVP#rMrHrs&rPrMappedExamplesIterable.features ~~rRc <V^8dQhRS[/#rr)rZrs"rPr[rFs  $ rRc RVPP4RRR^R^RVPP/VnVP#)rrN!num_examples_since_previous_stateprevious_state_example_idxrrrs&rPr'MappedExamplesIterable._init_state_dictsL !1!1!B!B!D d / (! DNN++  rRc#"VP'dXVPP'd<\4pVP^R7Fwr#W!P V43xK R#VP 4RjxL R#L5i)rrrN)rBrJr,rl format_row_iter)rrwrrs& rPrMappedExamplesIterable.__iter__se ???t777')I!%!1!1!1!B //999"Czz| # #s.BAB?BBc#4aaa a a a a aaaaaa"SP'dSPR,M^o SP'dZSPR,'dASPPSPR,4SPR,pM^p\SP4oSP'dE\ SPP 4p\V\4'd VPMRo MRo V V VV3Rlo V V3Rlo V3RloV3RloVV3RloVVV3R loVVV3R lo .o\P!SP4'd5\P!4oSP"P%SS34MRoVV V V V VVV3R lpV!4pSP&'d R V4pVFbwrVSP'd5SPR,e SPR;;,^, uu&V^8d V^,pK]WV3xKd R# \d\P !4oLi;i \(\*3dS'd\,P/R \1S4 R24SFpTP3RR7K SP5\P6!S!4h \P8\:3d\,P/R4hi;ihi;i5i)r[rrZNc3z<"S EF)wrS PeS P^8:dS M\S S P^, 4pW3.\V4,p\V!wrERP RV44pS P 'd=S Pe/S P^8d\ V4S P8dR#\V4pS 'd S !V4MTp\\ V44Uu.uF pS V,NK ppS \ V4, o WV33xEK, R#uupi5i)Nrc38"TFp\V4xK R#5irMrrs& rPraLMappedExamplesIterable._iter..iter_batched_inputs..s84Cs3xx4r) rr rhrrrrrr) rrUrrrrrwrindices current_idx format_dictrrs rPiter_batched_inputs9MappedExamplesIterable._iter..iter_batched_inputss ( .$//Q2F$//A*=> '*^$4tN7K$K!!$&7!8hh8488(((3!+H 7*84.9 E*u49#>O:P4QR4Qq;??4QRs7|+ U|++/!)*SsC?D;D6'D;c3j<"SF'wr\V4pS^, oS^, W33xK) R#5irr)rrUrfrs rP iter_inputs1MappedExamplesIterable._iter..iter_inputss8 ( w-q !Ao~55 !)s03c <SP'dV'd\\V44pVUu.uF,p\W,4\W,48wgK*VNK. ppV'dG\ RT RVUu.uFp\W,4NK up RV R\W,4 R2 4hR#R#R#uupiuupi)z!Column lengths mismatch: columns z have length z while z has length .N)r?rrrrg)processed_inputs first_colr`bad_colsrs& rPvalidate_function_output>MappedExamplesIterable._iter..validate_function_outputs||| 0 &6!78 #3#3Cs;K;P7QUXYiYtUu7uCC#3$;H:]{CTD{CsvTWXhXmTn{CTDSEE!!* <r=rXrA) rFrerrUr`fn_argsadditional_argsinputsrs && rPprepare_inputs4MappedExamplesIterable._iter..prepare_inputssw&LC#'#5#5#=wi\`\n\nCo\nUXCLL\nCoG O   :%']FOT^^C C DpsA0c<S!V4SP'd4SPF#pW19dWW ^,JgKW29gK!W#K% /VCVCpV#rr@)rFrwrortransformed_inputsrrrs&&& rPprepare_outputs5MappedExamplesIterable._iter..prepare_outputssa $%5 6""",,A{"I'q>9a>S,/ - "@F!?.>!? % %rRcZ<S!W4wr#rES P!.VOVO5/VBpS!WV4#)z8Utility to apply the function on a selection of columns.r< rFrerwrurvrArorxr}rs && rPapply_function4MappedExamplesIterable._iter..apply_functions<:H:^ 7F_#}}UgUU9U ";8HI IrRcv<"S!W4wr#rES P!.VOVO5/VBGRjxL pS!WV4#L 5i)zLUtility to apply the function on a selection of columns. Same code but asyncNrrs && rPasync_apply_function:MappedExamplesIterable._iter..async_apply_functionsI:H:^ 7F_%)]]%[G%[o%[QZ%[[ ";8HI I \s &979c 3\<"SP'dS !4MS!4p\P!SP4'EdSP'd?SP P 4pVSPR&RpSPR,p.pVEFwrVVPV4SPSPS !We444\S4SP8dSP\P!S\PR74wrxS'dT\V4SP8d:SP\P!S\PR74wrxK[\S4^ SP,8dSPS^,4S'dS^,P4'dVP!^4SP!^4rWYP#43xSP'gKmV XJgKuXSPR&^SPR&XSPR&RRr!KSP'gEKXeEKS'gEKSP P 4pSR,pS pEK S'dHV^,SPS^,43xVP!^4SP!^43KOR#SP'dXSP'dFSP P 4SPR&^SPR&S SPR&VFwrVSP'd"SP'gS SPR&VS !We43xSP'gKXSP'gKlSP P 4SPR&^SPR&S SPR&K R#5i)rNr[) return_whenrZ)r?inspectiscoroutinefunctionr<rrr r create_taskrrDrun_until_completeasynciormFIRST_COMPLETEDdoner_r^)inputs_iteratorrprevious_state_taskr[rerrFrpendingtaskrrrfrhrklooprtaskss rP iter_outputs2MappedExamplesIterable._iter..iter_outputssF7;|||13O**4==99###%)%5%5%@%@%BN9GD$$%56*.'151A1AB^1_.=?&5NANN1%LL!1!12F{2V!WX5zT%Y%YY(,(?(?#LLGTUD,,-PQMgD,,-IJBF,?''',?,GEE)-)9)9)D)D)F.3Bi+5@27'68!!*d&=&=eAh&GGGKKNEIIaL0###|||=A=M=M=X=X=Z(()9:PQ(()LMIT(()EF&5NA'''#|||MXD,,-IJ^K;;;'''<<TUD,,-PQMXD,,-IJ'6smAP, P,CP,5BP, P,&AP,.P,6AP,=P,P,1P,AP,P,.A/P,*P, P,!A P,c3P"TFwr\V4FpW3xK K R#5irM)r)r_rtransformed_batchtransformed_examples& rPra/MappedExamplesIterable._iter..Bs129./ABS/T+./T/29s$&z Canceling z async tasks.KeyboardInterrupt)msgzTasks canceled.)rrrrrBr0rKrr.recursive_tensorizerrr<rget_running_looprnew_event_looprMrr? ExceptionrloggerdebugrcancelrgatherCancelledErrorrg)rnum_examples_to_skiprwroutputsrrrrrrfrgrhrkrrrxr}rrrsf @@@@@@@@@@@@rPr_MappedExamplesIterable._itersHLHXHXHXd&&'CD^_     0 01A B B    , ,T-=-=>N-O P#'#3#34W#X #$ (() ???%doo&A&ABI;EiQ`;a;a)77gkKK , ,6 6  D & J  J %'  & &t}} 5 5 0//1  ' ' . .e} =D8 Y8 Yt "nG|||29 -4(###(8(89I(J(V$$%HIQNI'!+(A-(.. -4Q  0--/ 0^,-  z#e*]CD!DKK$7K8"4++GNNE,BC  .. ;4LL!23 4  s9L L"D L-H<-L0I" A1I":L< ILIL"L;=L9#KL0L LLLLcp<V^8dQhRS[S[,RS[S[S[S[P 3,,/#)rTrsry)rrsrrrJr{r|)rZrs"rPr[rFYs>TETE#TE(5QTVXV^V^Q^K_B`TErRc # "VP'd \VPP4M \4pVPP 'dVPP 4pM@\ VPVP'd VPM^VPR7pVP'dZVPR,'dAVPPVPR,4VPR,pM^pVP'd;Ve7VPP4VPR&^VPR&VP'dVPR,M^pVPP4pVP'd .pRpWvR&WR&VEFwrVP'd>VPe0\!V 4VP8dVP'dR#VP"fVP%V 4.M!VP"U u.uF qV ,NK up p VP&'d^VP'd;T P)\+\!V 44U u.uF qV ,NK up 4MV P)V4VP,!V /VBp\/V4p\1V\2P44'g3\7RVP8 R \;V4 R VP8 R 24hVP<'dQVP<F@pVVP>9gKVPAVP>PCV44pKB VfUV\!V 4, pVP'd)VPR;;,\!V 4, uu&W3xEK/\EVPGVR 74F]wp pV^, pVP'd VPR;;,^, uu&V^8d V^,pKRV R V 2V3xK_ VP'gEKVPP4VPR&^VPR&WPPR&EK VP'dX'dwVPIR4p \+\!V 44U u.uF qV ,NK pp V V3p VP,!V /VPBRVRVR,^,/BpRV3xR#R#R#uup iuup iuup i5i)rrrrrZNr[tables_accumulatorlengthz(Provided `function` which is applied to z returns a variable of type z*. Make sure provided `function` returns a z to update the dataset.rrr"last_batch_from_tables_accumulatorr)%rBr0rKr+rrrr?rrrrr rArrErr>rvr=rrr<r8rr{r|r table_typerr@r remove_columnrrrtr_)rrsrwrrrfrArrrrr` function_argsrr6 output_tablerrryres&& rPrl"MappedExamplesIterable._iter_arrowYsRVRaRaRaM$//2M2M$Ngugw    & & &''224H(  .2lll4?? $ 4 4H     0 01A B B    , ,T-=-=>N-O P#'#3#34W#X #$     9151A1A1L1L1ND  - .DED  @ AHLHXHXHXd&&'CD^_ NN'')  8 8 813 $(F.@* +"(h %MC OO/MDOO3(((%%-''12/3/A/AB/Asmm/AB     <<<!((5XCW)XCWa//CW)XY!((5]]M?Y?F1&9LlBHH55>y?S?S>TTpF|n$NyOcOcNdd{} """"11F!:!::'3'A'A,B[B[BaBabhBi'j 2$s8}, ###$$%ABc(mSB''&/ 0F0FUb0F0c&dNA{1$K'''(()LMQRRM+a/,1,  E1#, 33'e###9=9I9I9T9T9VD$$%56LMD$$%HIEP$$%ABg&h  8 8 8=O)--b1H05c(m0DE0D1Q0DGE%w/M=="&..EW`ghj`kno`oL7 D D>P 8QC*YLFsAU;A,U;U; AU;4A U;AU;AU;!1U;U,#U;8U; !U;+U1 \VPPV4VPVPVP VP VPVPVPVPVPVPVPR7 #)&Shuffle the wrapped examples iterable. r<r=r>r?rrr@rArBrrD)r:rrr<r=r>r?rrr@rArBrrDrs&&rPr+MappedExamplesIterable.shuffle_data_sourcessz%    1 1) <]]**,,LL 00..nn]]<@r?rrr@rArBrrDrs&&&&rPr)MappedExamplesIterable.shard_data_sourcess%    / / j / Y]]**,,LL 00..nn]]<@ rRc<\VPP4VPVPVP VP VPVPVPVPVPVPVPR7 #)r)r:rrr<r=r>r?rrr@rArBrrDrs&rPr+MappedExamplesIterable.reshard_data_sourcessx%    1 1 3]]**,,LL 00..nn]]<@rErDr@r=) FNFrFNNNNNFrMr!)r"r#r$r%rr'rrrrrr_rlrrrrr(r)rbrcs@@rPr:r:@s4e4el$$))  $AFTETEl  "  "  +++rRr:c V^8dQhR\\\P3,R\\\ \P \P\P3,R\/#)rTinputmaskmask_column_name) rrXr{r|rrhArray ChunkedArray BooleanScalarrY)rZs"rPr[r[sQ ( ( rxx  ( dBHHboor7G7GG H ( (rRc,\V\P4'ds\V\\P\P 34'g-\P !V.\P!4R7pVPW!4#W!/#)r) rr{r|rhrrrbool_r)rrrs&&&rP _add_maskrsd %""$rxx ABB88TF4D""#3:: ''rRcrV^8dQhR\R\\\P3,R\ /#rT mask_functionrrrrrXr{r|rY)rZs"rPr[r[s/44H4U4>-B4]`4rRc2V!V.VO5/VBp\WV4#rMrrrrargsr/rs&&$*, rPadd_maskrs$  0 0 0D U"2 33rRcrV^8dQhR\R\\\P3,R\ /#rr)rZs"rPr[r[s1444$)$.$94TW4rRcN"V!V.VO5/VBGRjxL p\WV4#L5irMrrs&&$*, rPasync_add_maskrs1u6t6v6 6D U"2 33 7s %#%caa]tRtRtoRtRV3RlV3RllltV3RltRV3RlV3RllltV3RlR ltRV3R lR llt V3R lR lt ] V3RlRl4t Rt VtV;t#)FilteredExamplesIterableiz ===MASK===c<V^8dQhRS[RS[RS[RS[S[S[,,RS[RS[S[,RS[S[,RS[R ,/#) rTrr<r=r>r?rrArBrC)rrrrrhrYrsrX)rZrs"rPr[%FilteredExamplesIterable.__annotate__sr  *     S *    SM D> /0 rRc B<W nVP'd0\/VPCVP\ R4/C4p MRp \ S T`T\\P!V4'd\M\VVPR7VVVVVVV R7 R#)rN)r) rr<r=r>r?rrArBr) rrr!rrr$r3rr rrrr) rrr<r=r>r?rrArBrr4s &&&&&&&&& rPr!FilteredExamplesIterable.__init__s&     ^;#7#7 ^9N9NPUV\P] ^_HH #")"="=h"G"GX!%!6!6 &'!!  rRc#<"\SV`4F8wr\V4pVPVP4'gK3W3xK: R#5irM)r3r_rXr_r)rrrUr4s& rPr_FilteredExamplesIterable._iter"s>!GMOLC7mG{{40011l",s =A Ac0<V^8dQhRS[S[,/#)rTrsrrs)rZrs"rPr[r(sII#IrRc#<"\SV`VR7FDwr#W0P,pW#PVP4P V43xKF R#5i)rrN)r3rlrdropfilter)rrsrrrr4s&& rPrl$FilteredExamplesIterable._iter_arrow(sQ"W0}0MMC112D}}T%:%:;BB4HH HNsAAc4<V^8dQhRS[S[,RR/#)rTrryrr)rZrs"rPr[r-s   #  ;U  rRc  \VPPV4VPVPVP VP VPVPVPR7#)rr<r=r>r?rrArB) rrrrr=r>r?rrArB)rrs&&rPr-FilteredExamplesIterable.shuffle_data_sources-sZ'    1 1$ 7''**,,LLnn  rRc*<V^8dQhRS[RS[RR/#)rTrrryrr)rZrs"rPr[r:s#   S    Rl  rRc  \VPPWVR7VPVPVP VP VPVPVPR7#)rNrOr) rrrrr=r>r?rrArBrs&&&&rPr+FilteredExamplesIterable.shard_data_sources:s_'    / / j / Y''**,,LLnn  rRc<V^8dQhRR/#)rTryrrN)rZrs"rPr[rGs   &@  rRc \VPP4VPVPVP VP VPVPVPR7#)r) rrrrr=r>r?rrArBrs&rPr-FilteredExamplesIterable.reshard_data_sourcesGsX'    1 1 3''**,,LLnn  rRc <V^8dQhRS[/#rr)rZrs"rPr[rTrrRc.VPP#rMrrs&rPr#FilteredExamplesIterable.num_shardsSrrR)r)FNFrNNrMr!)r"r#r$r%rrr_rlrrrr'rr(r)rbrcs@@rPrrs^#  @# II         +++rRrc,aa]tRtRtoV3RlV3RlltV3RlRlt]R4t]R4t]R4t V3R lR lt V3R lV3R llt ] RV3R lRll4t RtRtV3RlRltRV3RlRlltV3RlRlt]V3RlRl4tRtVtV;t#)BufferShuffledExamplesIterableiXcT<V^8dQhRS[RS[RS[PP/#)rTr buffer_sizer)rrsrrr)rZrs"rPr[+BufferShuffledExamplesIterable.__annotate__Ys0##$9##XZXaXaXkXk#rRcH<\SV`4WnW nW0nR#rM)r3rrrr)rrrrr4s&&&&rPr'BufferShuffledExamplesIterable.__init__Ys &&"rRc$<V^8dQhRS[RR/#rr)rZrs"rPr[r_s   (? rRc\VP4pVP^R4V, p\VPVP \ PPVR7R7#)rr)rrrr) rrrrrrrrrr s&& rPr)BufferShuffledExamplesIterable.shift_rngs_sYt~~&<<7+e3-((((ii+++:  rRc.VPP#rMrrs&rPr'BufferShuffledExamplesIterable.is_typedhrrRc.VPP#rMrrs&rPr'BufferShuffledExamplesIterable.featureslrrRcVVPP'd VP#R#rMrrs&rPr)BufferShuffledExamplesIterable.iter_arrowprrRc <V^8dQhRS[/#rr)rZrs"rPr[rts  $ rRcVPP4VnVP4VnVP#rM)rrrr _original_state_dictrs&rPr/BufferShuffledExamplesIterable._init_state_dictts4++<<>$(OO$5!rRc&<V^8dQhRS[RS[/#r r)rZrs"rPr[rys33$343rRc<VP'd&WP8wd\PR4\SV`V4#)zLoading a state dict of a shuffle buffer of a dataset without the buffer content.The shuffle buffer will be refilled before starting to yield new examples.)rrrwarningr3r)rr r4s&&rPr.BufferShuffledExamplesIterable.load_state_dictys?    666aw&z22rRcd<V^8dQhRS[PPRS[RS[S[,/#)rTrrry)rrrrsr)rZrs"rPr[rs8^^"))"5"5^C^dlmpdq^rRc#X"RVP^WR74RjxL K%L5i)Tc38"TFp\V4xK R#5irMr)r_rs& rPraFBufferShuffledExamplesIterable._iter_random_indices..s](\1A(\rrN)r)rrrs&&&rP_iter_random_indices3BufferShuffledExamplesIterable._iter_random_indicess#] Q (\] ] ]s *(*c#N"VPp\VP4pVPW!4p.pVPF?p\ V4V8Xd\ V4pWF,xWTV&K.VPV4KA VPV4TRjxL R#L5irM) rrrrrrrrr)rrrrg mem_bufferrOrs& rPr'BufferShuffledExamplesIterable.__iter__s&& t~~&44SF !!A:+-)* m# !1 !!!$ " JsBB%B#B%c#r"VPp\VP4pVPW!4p.pVPP 4FCwrV\ V4V8Xd\V4pWG,xWV3WG&K1VPWV34KE VPV4TRjxL R#L5irM) rrrrrrrrrr)rrrrgrrrrs& rPrl*BufferShuffledExamplesIterable._iter_arrows&& t~~&44SF !--88:MC:+-)* m#!$ !!3/2 ; JsB,B7.B5/B7cL<V^8dQhRS[PPRR/#)rTrryrr)rZrs"rPr[rs&  bii.A.A Ff rRc x\VPPV4VPVPR7#)zFShuffle the wrapped examples iterable as well as the shuffling buffer.rr)rrrrrrs&&rPr3BufferShuffledExamplesIterable.shuffle_data_sourcess4-    1 1) <$JZJZfjftft  rRc*<V^8dQhRS[RS[RR/#)rTrrryrr)rZrs"rPr[rrrRc |\VPPWVR7VPVPR7#)rNrOr)rrrrrrs&&&&rPr1BufferShuffledExamplesIterable.shard_data_sourcess8-    / / j / Y((nn  rRc<V^8dQhRR/#)rTryrrN)rZrs"rPr[rs  &F rRcv\VPP4VPVPR7#)r)rrrrrrs&rPr3BufferShuffledExamplesIterable.reshard_data_sourcess1-    1 1 3((nn  rRc <V^8dQhRS[/#rr)rZrs"rPr[rrrRc.VPP#rMrrs&rPr)BufferShuffledExamplesIterable.num_shardsrrR)rrrrr)rr!)r"r#r$r%rrr'rrrrr staticmethodrrrlrrrrr(r)rbrcs@@rPrrXs##   ))))II  33^^^""      +++rRrc]tRtRtRtRtR#)DataSourcesShufflingDisallowediz8skip() or take() freeze the order of data sources shardsrNN)r"r#r$r%r&r(rNrRrPr'r'sBrRr'caa]tRtRtoRV3RlV3Rlllt]R4t]R4t]R4tV3RlRlt R t R t ] R 4t V3R lR ltRV3RlRlltV3RlRlt]V3RlRl4tRtVtV;t#)SkipExamplesIterableic2<V^8dQhRS[RS[RS[RS[/#rTrn"block_sources_order_when_shufflingsplit_when_shardingrrsr)rZrs"rPr[!SkipExamplesIterable.__annotate__3 7 7* 7  7-1 7 " 7rRcT<\SV`4WnW nW0nW@nR#rMr3rrr,r-r.rrr,r-r.r4s&&&&&rPrSkipExamplesIterable.__init__' &2T/#6 rRcVVPP'd VP#R#rMrrs&rPrSkipExamplesIterable.iter_arrowrrRc.VPP#rMrrs&rPrSkipExamplesIterable.is_typedrrRc.VPP#rMrrs&rPrSkipExamplesIterable.featuresrrRc <V^8dQhRS[/#rr)rZrs"rPr[r0r rRcR^RVPP4RVPP/VnVP#)skippedrrrrs&rPr%SkipExamplesIterable._init_state_dictsB q !1!1!B!B!D DNN++  rRc#"VP'dVPR,M^pVPFLpV^,VP8:d.V^, pVP'dWPR&KFKHVxKN R#5ir?N)rrr,)rr?rFs& rPrSkipExamplesIterable.__iter__sr151A1A1A$""9-q++K{dff$1 ###29$$Y/$"! ,s A)B,Bc#z"VP'dVPR,M^pVPP4Fwr#\V4^8XdKV\V4,VP8:d7V\V4, pVP'dWPR&KlKnV^,VP8:dgVPV, pVPpVP'dWPR&W#P V\V4V, 43xKW#3xK R#5irB)rrrrr,r)rr?rrrs& rPrl SkipExamplesIterable._iter_arrows151A1A1A$""9-q!--88:MC8}!3x=(DFF23x=(###29$$Y/$1&')&&###29$$Y/>>&#h-&2HIIIm#;s C8D;;AD;cW,pW,pV.V,p\V4FpWE;;,^, uu&K V#rrnumr,quotient remainderr^rs&& rP split_number!SkipExamplesIterable.split_number:8G ay!A INI" rRcL<V^8dQhRS[PPRR/#)rTrryr)r)rZrs"rPr[r0&  bii.A.A F\ rRc VP'd \4h\VPP V4VP VPVP R7#)zeMay not shuffle the wrapped examples iterable since it would skip examples from other shards instead.r,r-r.)r-r'r)rrr,r.rs&&rPr)SkipExamplesIterable.shuffle_data_sourcesQ  2 2 202 2'  55i@&&373Z3Z$($<$<  rRc*<V^8dQhRS[RS[RR/#)rTrrryr)r)rZrs"rPr[r0s#  S  Rh rRc VP'd`\VPPWVR7VP VP V4V,VP VPR7#V#rNrOrR)r.r)rrrLr,r-rs&&&&rPr'SkipExamplesIterable.shard_data_sourcessg  # # #'  33JR\3]##DFFJ7>373Z3Z$($<$<  KrRc<V^8dQhRR/#)rTryr)rN)rZrs"rPr[r0&  &< rRc\VPP4VPVPVP R7#rR)r)rrr,r-r.rs&rPr)SkipExamplesIterable.reshard_data_sources&:#    1 1 3ff/3/V/V $ 8 8   rRc <V^8dQhRS[/#rr)rZrs"rPr[r0/rrRc.VPP#rMrrs&rPrSkipExamplesIterable.num_shards.rrRrr-rr,r.TTr!r"r#r$r%rr'rrrrrrlr%rLrrrrr(r)rbrcs@@rPr)r)s 7 7II))))  "$$      +++rRr)caa]tRtRtoRtV3RlV3RlltV3RlRltRtV3RlR ltRV3R lR llt V3R lR lt ] V3RlRl4t Rt VtV;t#)RepeatExamplesIterablei3zH Iterable that repeats the underlying iterable a given number of times. c6<V^8dQhRS[RS[S[,/#)rTr num_times)rrrs)rZrs"rPr[#RepeatExamplesIterable.__annotate__8s###*#C=#rRc<<\SV`4WnW nR#rM)r3rrrh)rrrhr4s&&&rPrRepeatExamplesIterable.__init__8s &"rRc <V^8dQhRS[/#rr)rZrs"rPr[riAr rRcR^RVPP4RVPP/VnVP#) repeat_indexrrrrs&rPr'RepeatExamplesIterable._init_state_dictAsB A !1!1!B!B!D DNN++  rRc#"VP'dVPR,M^pVPeV\VP^48dR#VPRjxL V^, pVP'gK]WPR&VPP 4VPR&KLX5i)rnNr)rrhmaxrr)rrns& rPrRepeatExamplesIterable.__iter__Is;?;K;K;Kt''7QR ~~)lc$..RS>T.T'' ' ' A L3?  08<8H8H8Y8Y8[  !45 (sA#C%B>&C8CcL<V^8dQhRS[PPRR/#)rTrryrfr)rZrs"rPr[riTs*rrbii.A.ArF^rrRc b\VPPV4VPR7#)z-Shuffle the underlying iterable, then repeat.rh)rfrrrhrs&&rPr+RepeatExamplesIterable.shuffle_data_sourcesTs'%d&6&6&K&KI&VbfbpbpqqrRc*<V^8dQhRS[RS[RR/#)rTrrryrfr)rZrs"rPr[riXs#  S  Rj rRc f\VPPWVR7VPR7#)zShard, then repeat shards.rOru)rfrrrhrs&&&&rPr)RepeatExamplesIterable.shard_data_sourcesXs/%    / / j / Ynn  rRc<V^8dQhRR/#)rTryrfrN)rZrs"rPr[ri_s  &> rRc`\VPP4VPR7#)ru)rfrrrhrs&rPr+RepeatExamplesIterable.reshard_data_sources_s(%    1 1 3nn  rRc <V^8dQhRS[/#rr)rZrs"rPr[rifrrRc.VPP#rMrrs&rPr!RepeatExamplesIterable.num_shardserrR)rrrhr!)r"r#r$r%r&rrrrrrr'rr(r)rbrcs@@rPrfrf3s]##   \rr    +++rRrfcaa]tRtRtoRV3RlV3Rlllt]R4t]R4t]R4tV3RlRlt R t R t ] R 4t V3R lR ltRV3RlRlltV3RlRlt]V3RlRl4tRtVtV;t#)TakeExamplesIterableijc2<V^8dQhRS[RS[RS[RS[/#r+r/)rZrs"rPr[!TakeExamplesIterable.__annotate__kr1rRcT<\SV`4WnW nW0nW@nR#rMr3r4s&&&&&rPrTakeExamplesIterable.__init__kr6rRcVVPP'd VP#R#rMrrs&rPrTakeExamplesIterable.iter_arrowxrrRc.VPP#rMrrs&rPrTakeExamplesIterable.is_typed|rrRc.VPP#rMrrs&rPrTakeExamplesIterable.featuresrrRc <V^8dQhRS[/#rr)rZrs"rPr[rr rRcR^RVPP4RVPP/VnVP#)takenrrrrs&rPr%TakeExamplesIterable._init_state_dictsB Q !1!1!B!B!D DNN++  rRc#6"VP'dVPR,M^pWP8dR#VPFKpV^,VP8:d0V^, pVP'dWPR&VxKJR# R#5irN)rr,r)rrrFs& rPrTakeExamplesIterable.__iter__sz-1-=-=-=  )1 FF? ++KqyDFF" ###05$$W-!!,s A;B>Bc#|"VP'dVPR,M^pWP8dR#VPP4Fwr#\ V4^8XdKV\ V4,VP8:d:V\ V4, pVP'dWPR&W#3xKqV^,VP8:dWVPV, pVPpVP'dWPR&W#P ^V43xKR# R#5ir)rr,rrrr)rrrrrs& rPrl TakeExamplesIterable._iter_arrows-1-=-=-=  )1 FF? !--88:MC8}!X&$&&0X&###05$$W-m#dff$%###05$$W->>!V444;s D D<,D<cW,pW,pV.V,p\V4FpWE;;,^, uu&K V#rrGrHs&& rPrL!TakeExamplesIterable.split_numberrNrRcL<V^8dQhRS[PPRR/#)rTrryrr)rZrs"rPr[rrPrRc VP'd \4h\VPP V4VP VPVP R7#)zeMay not shuffle the wrapped examples iterable since it would take examples from other shards instead.rR)r-r'rrrr,r.rs&&rPr)TakeExamplesIterable.shuffle_data_sourcesrTrRc*<V^8dQhRS[RS[RR/#)rTrrryrr)rZrs"rPr[rs#SRhrRc tVP'd`\VPPWVR7VP VP V4V,VP VPR7#\VPPWVR7VP VP VPR7#rW)r.rrrrLr,r-rs&&&&rPr'TakeExamplesIterable.shard_data_sourcess  # # #'  33JR\3]##DFFJ7>373Z3Z$($<$<  (  33JR\3]&&373Z3Z$($<$<  rRc<V^8dQhRR/#)rTryrrN)rZrs"rPr[rrZrRc\VPP4VPVPVP R7#r\)rrrr,r-r.rs&rPr)TakeExamplesIterable.reshard_data_sourcesr^rRc <V^8dQhRS[/#rr)rZrs"rPr[rrrRc.VPP#rMrrs&rPrTakeExamplesIterable.num_shardsrrRrbrcr!rdrcs@@rPrrjs 7 7II))))   *  "  +++rRrc V^8dQhR\R\R\\\\\R3,3,R\/#)rTrUrtoken_per_repo_idNry)rXr!rYrr)rZs"rPr[r[sB    % :>sE#tUY/DZ?Z:[   rRc\V4pVFpW09gK RW&K VPV4pVPWBR7pV#)Nr)rXencode_exampledecode_example)rUrrrencoded_exampledecoded_examples&&& rP_apply_feature_types_on_examplersP7mG  %#'G  --g6O--o-cO rRcda]tRtRto]V3RlRl4t]V3RlRl4tV3RltRtVt R#) rCic <V^8dQhRS[/#rr)rZrs"rPr[FormattingConfig.__annotate__sKK$KrRcH\\VP4\4#rM)rr0rKr-rs&rPrJFormattingConfig.is_tables-(8(89>JJrRc <V^8dQhRS[/#rr)rZrs"rPr[rsLL4LrRcH\\VP4\4#rM)rr0rKr.rs&rP is_tensorFormattingConfig.is_tensors-(8(89?KKrRc6<V^8dQh/S[S[,;R&#)rTrKrrY)rZrs"rPr[rs#rRrNN) r"r#r$r%r'rJr__annotate_func__r(r)r*s@rPrCrCs8KKLLrRrCcaa]tRtRtoRV3RlV3Rlllt]R4t]R4t]R4tV3RlRlt R t V3R lR lt V3R lR lt RV3RlRllt V3RlRlt]V3RlRl4tRtVtV;t#)FormattedExamplesIterableic <V^8dQhRS[RS[S[,RS[S[,RS[S[S[S[S[R3,3,RS[/#)rTrrBrrNforce_convert_to_python)rrrCr!rXrYrr)rZrs"rPr[&FormattedExamplesIterable.__annotate__s] ? ?* ?-. ?8$ ?  U3d?%; ;< ? "& ?rRc`<\SV`4WnW0nW nW@nWPnR#rM)r3rrrHrBrr)rrrBrrrr4s&&&&&&rPr"FormattedExamplesIterable.__init__s, &!$!2'>$rRc~VPP'd!VP'g VP#R#R#rM)rrrrlrs&rPr$FormattedExamplesIterable.iter_arrow s4    & & &t/K/K/K## #0L &rRcZVPP;'gVPRJ#rM)rrrHrs&rPr"FormattedExamplesIterable.is_typed s%((FFDNN$,FFrRcVP#rMrUrs&rPr"FormattedExamplesIterable.features rWrRc <V^8dQhRS[/#rr)rZrs"rPr[r r7rRcXVPP4VnVP#rMrrs&rPr*FormattedExamplesIterable._init_state_dict rrRc#"VP'dVPP'dA\VPP'g VP MRVP R7pMT\VPPVPP'g VP MRVP R7pVPP'dDVP4F-wr#VPV4p\V4FpW%3xK K/ R#\V\4'd VPMRpVPFiwr%VP 'd>VPP'g"\#WPP VP R7pV'd V!V4pW%3xKk R#5i)N)rrr)rBrJr,rrrHrr0rKrrlrvrrr.rrr)rrwrrrwrUrgs& rPr"FormattedExamplesIterable.__iter__ sV$//":":":'/3/?/?/H/H/Hd"&"8"8I &++/3/?/?/H/H/Hd"&"8"8I    & & &!%!1!1!3 !..x81%8G,& 9"4i99--  !% 0 0 ===)9)9)B)B)B=$BXBXG)'2Gl"!1s.GB/G!B%G(G0GcZ<V^8dQhRS[S[S[S[P3,,/#rr)rZrs"rPr[rD s&  XeCM&:; rRc#D"VP'g%VPP4RjxL R#VPPpVPP4Fwr#\ VP 4pVPFZpWT9gK \ PP\ P!4\V4R.4pVPWV4pK\ VPV8wd\W0P4pW#3xK R#L5irM)rrrl arrow_schemarjrr{ NullArray from_buffersnullrrr~r8)rr~rrrrr`s& rPrl%FormattedExamplesIterable._iter_arrowD s}}}''335 5 5 ++!--99;MC(//0G#}} -,,33BGGIs8}tfUC'55kGH -&(1(MMJ- < 6s0D DA"D BD cL<V^8dQhRS[PPRR/#)rTrryrr)rZrs"rPr[rT s&  bii.A.A Fa rRc \VPPV4VPVPVP VP R7#)rrrrBr)rrrrrrBrrs&&rPr.FormattedExamplesIterable.shuffle_data_sourcesT sC(    1 1) <]]"44$($@$@   rRc*<V^8dQhRS[RS[RR/#)rTrrryrr)rZrs"rPr[r^ s#  S  Rm rRc \VPPWVR7VPVPVP VP R7#)rNrOr)rrrrrrBrrs&&&&rPr,FormattedExamplesIterable.shard_data_sources^ sH(    / / j / Y]]"44$($@$@   rRc<V^8dQhRR/#)rTryrrN)rZrs"rPr[rh s  &A rRc\VPP4VPVPVP VP R7#)r)rrrrrrBrrs&rPr.FormattedExamplesIterable.reshard_data_sourcesh sA(    1 1 3]]"44$($@$@   rRc <V^8dQhRS[/#rr)rZrs"rPr[rr rrRc.VPP#rMrrs&rPr$FormattedExamplesIterable.num_shardsq rrR)rHrrrrBrrr!rrcs@@rPrrs ? ?$$GG  $#L        +++rRrc,a]tRtRtoV3RltRtVtR#)DistributedConfigiv c2<V^8dQh/S[;R&S[;R&#)rTrank world_sizer)rZrs"rPr[DistributedConfig.__annotate__v s IOrRrNN)r"r#r$r%rr(r)r*s@rPrrv srRrc\P'dp^RIpVPPP VP 9d;V;P VPPP 3, unR#R#R#)zNAdd torch.utils.data.IterableDataset as a parent class if 'torch' is availableN)rTORCH_AVAILABLEtorch.utils.datautilsdataIterableDataset __bases__)clstorchs& rP._maybe_add_torch_iterable_dataset_parent_classr| sY  ;;   + +3== @ MMekk..>>@ @M ArRcdV^8dQhR\\R3,R\\R3,/#)rTrz torch.Tensorry)rrs)rZs"rPr[r[ s1  eCllt*RV3R?lR@llt+V3RAlRBlt,V3RClRDlt-V3RElRFlt.V3RGlRHlt/RV3RIlRJllt0V3RKlRLlt1V3RMlRNlt2V3ROlRPlt3V3RQlRRlt4V3RSlRTlt5V3RUlRVlt6V3RWlRXlt7V3RYlRZlt8RV3R[lR\llt9V3R]lR^lt:R_t;RV3R`lRalltRV3RflRgllt?RV3RhlRillt@R}V3RjlRklltAR}V3RllRmlltBRV3RnlRolltCR}V3RplRqlltDV3RrlRsltEV3RtlRultFRV3RvlRwlltGRxtHVtIR#)ri z A Dataset backed by an iterable.Nc<V^8dQhRS[RS[S[,RS[S[,RS[S[,RS[S[,RS[S[S[S[S[S[ R3,3,,/#)rTrinfosplitrB distributedrN) rrr1r3rCrrXrYrr)rZrs"rPr[IterableDataset.__annotate__ sGG*G{#G # G -. G /0 G$DeCtO.D)D$EFGrRcnVeVP4M \4p\P!WVR7\V4VnW@nWPnT;'g/Vn\^4Vn RVn \PVn VP4\VP 4R#)Nrr )rr1rr _ex_iterable _formatting _distributed_token_per_repo_idr_epoch_starting_state_dictr_cache_IterableDataset__hffs_cache"_prepare_ex_iterable_for_iterationrr4)rrrr rBr rs&&&&&&&rPrIterableDataset.__init__ s#.tyy{KM!!$? -%'EVE\E\Z\2\]^2_ 48!(// //16t~~FrRc0<V^8dQhRS[S[,/#rr)rZrs"rPr[r  s E EXc] ErRc LVPfR#\VP4#)aNumber of columns in the dataset. This can be None if the dataset has unknown features (e.g. after a map() operation). Example: ```py >>> from datasets import load_dataset >>> ds = load_dataset("cornell-movie-review-data/rotten_tomatoes", split="validation") >>> ds.num_columns 2 ``` N)rrrs&rP num_columnsIterableDataset.num_columns s!}},tD#dmm2DDrRc@<V^8dQhRS[S[S[,,/#r)rrhrY)rZrs"rPr[r  s F FhtCy1 FrRc LVPfR#\VP4#)aBNames of the columns in the dataset. This can be None if the dataset has unknown features (e.g. after a map() operation). Example: ```py >>> from datasets import load_dataset >>> ds = load_dataset("cornell-movie-review-data/rotten_tomatoes", split="validation", streaming=True) >>> ds.column_names ['text', 'label'] ``` N)rrhrs&rPrIterableDataset.column_names s!}},tE$t}}2EErRc <V^8dQhRS[/#rr)rZrs"rPr[r  s3*3*D3*rRc ,\VP4#)ahGet the current state_dict of the dataset. It corresponds to the state at the latest example it yielded. Resuming returns exactly where the checkpoint was saved except in two cases: 1. examples from shuffle buffers are lost when resuming and the buffers are refilled with new data 2. combinations of `.with_format(arrow)` and batched `.map()` may skip one batch. Returns: `dict` Example: ```py >>> from datasets import Dataset, concatenate_datasets >>> ds = Dataset.from_dict({"a": range(6)}).to_iterable_dataset(num_shards=3) >>> for idx, example in enumerate(ds): ... print(example) ... if idx == 2: ... state_dict = ds.state_dict() ... print("checkpoint") ... break >>> ds.load_state_dict(state_dict) >>> print(f"restart from checkpoint") >>> for example in ds: ... print(example) ``` which returns: ``` {'a': 0} {'a': 1} {'a': 2} checkpoint restart from checkpoint {'a': 3} {'a': 4} {'a': 5} ``` ```py >>> from torchdata.stateful_dataloader import StatefulDataLoader >>> ds = load_dataset("deepmind/code_contests", streaming=True, split="train") >>> dataloader = StatefulDataLoader(ds, batch_size=32, num_workers=4) >>> # checkpoint >>> state_dict = dataloader.state_dict() # uses ds.state_dict() under the hood >>> # resume from checkpoint >>> dataloader.load_state_dict(state_dict) # uses ds.load_state_dict() under the hood ``` )rrrs&rPr IterableDataset.state_dict sf(())rRc$<V^8dQhRS[RR/#)rTr ryNr)rZrs"rPr[r  s0/0/$0/40/rRc WnR#)a\Load the state_dict of the dataset. The iteration will restart at the next example from when the state was saved. Resuming returns exactly where the checkpoint was saved except in two cases: 1. examples from shuffle buffers are lost when resuming and the buffers are refilled with new data 2. combinations of `.with_format(arrow)` and batched `.map()` may skip one batch. Example: ```py >>> from datasets import Dataset, concatenate_datasets >>> ds = Dataset.from_dict({"a": range(6)}).to_iterable_dataset(num_shards=3) >>> for idx, example in enumerate(ds): ... print(example) ... if idx == 2: ... state_dict = ds.state_dict() ... print("checkpoint") ... break >>> ds.load_state_dict(state_dict) >>> print(f"restart from checkpoint") >>> for example in ds: ... print(example) ``` which returns: ``` {'a': 0} {'a': 1} {'a': 2} checkpoint restart from checkpoint {'a': 3} {'a': 4} {'a': 5} ``` ```py >>> from torchdata.stateful_dataloader import StatefulDataLoader >>> ds = load_dataset("deepmind/code_contests", streaming=True, split="train") >>> dataloader = StatefulDataLoader(ds, batch_size=32, num_workers=4) >>> # checkpoint >>> state_dict = dataloader.state_dict() # uses ds.state_dict() under the hood >>> # resume from checkpoint >>> dataloader.load_state_dict(state_dict) # uses ds.load_state_dict() under the hood ``` N)r)rr s&&rPrIterableDataset.load_state_dict s `%/!rRcRVPPe.\VPPP44MR RVP R2#)z IterableDataset({ features: Unknownz, num_shards: z }))_inforrhrrrs&rP__repr__IterableDataset.__repr__R st3X\XbXbXkXkXwD9L9L9Q9Q9S4T~G4HH[\`\k\k[llqr rrRcVP#rM)__dict__rs&rP __getstate__IterableDataset.__getstate__U s }}rRcWn\VP4Vn\PVn\ VP4R#rM)r+rrrrrrr4)rds&&rP __setstate__IterableDataset.__setstate__X s1 @M (//6t~~FrRcJ\\VPVR744#)r)rr)rr,s&&rP_headIterableDataset._heada sDa0122rRc <V^8dQhRS[/#rr)rZrs"rPr[r e s  s rRc,\VP4#rM)rsrrs&rPepochIterableDataset.epochd s4;;rRc <V^8dQhRS[/#rr)rZrs"rPr[r i s,,C,rRc VP'dhVPPVPP,^8Xd2VPPVPP,#VPP#r3)rrrrrs&rPrIterableDataset.num_shardsh sh    !2!2!=!=@Q@Q@\@\!\`a!a$$//43D3D3O3OO O  +++rRc <V^8dQhRS[/#rr)rZrs"rPr[r o s#rRcVP#rMryrs&rPn_shardsIterableDataset.n_shardsn s rRc #"VP4p\PP4^RIpVP P P4pVP4'dVPVP8d\PRVP RVP RVPVP, R24\PRVP RVP R24VP'dR VPP R 2MR pVP!VPVP"R R 7pV'Ed\P%V RVP" R\'V4 RVP R24VP)VPVP"R R 7p\+WP"R7pRVP-4RVP./VnVP2'dEVP.VP2R,8Xd#VP5VP2R,4VP6'dVP8'gVP6P:'dZ\=VP6P>VP@R7pVP94FwrxVPCV4xK R#VF wryV xK \P%V RVP" R\'V4 RVP R24R#\P%V RVP" RVP RVP R24R#5i)rNzToo many dataloader workers:  (max is dataset.num_shards= ). Stopping z dataloader workers.zTo parallelize data loading, we give each process some shards (or data sources) to process. Therefore it's unnecessary to have a number of workers greater than dataset.num_shards=J. To enable more parallelism, please split the dataset in more files than zk or try `dataset = dataset.reshard()` which may increase `num_shards` depending on the dataset file format.znode# Frrrzdataloader worker#z, ': Starting to iterate over / shards.)rrrr7rRz, ': Finished iterating over z9, ': Stopping... Number of dataset shards < num_workers ( LL-1+..1AA_`cdr`s_ttuwBwMwMvNNVW &88&22+..UZ9K1[P^P^_K#[%A%A%C D (((TZZ4;T;TU\;]-]++D,E,EFY,Z[[%;%;%;t?O?O?X?X?X)$*:*:*F*FQUQ^Q^_ %0%;%;%=MC#..x88&>$/LC!M%0 LL-1+..1AA^_bcq_r^sstvAvLvLuMMUV  LL-1+..1AAz|G|R|R{SSTU`UlUlTmmop s'DNA N'DN.NNC)Nc VP'dVPP^8dR#R\P9d@^RIpVP P P4pVeVP^8dR#R#)rFrNT) rrr3r4rrrrMrP)rrrQs& rPrN IterableDataset._is_main_process sb    !2!2!7!7!!; ckk ! #++**::>A+=rRc,<V^8dQhRS[RS[RS[/#)rTrrry)rsrr)rZrs"rPr[r  s%777487 7rRc VPpVP'dNVP\PP VP44p\ W0P4pVP'EdVPPpVPPpVPV,^8XdhVP4'd>VPV,pV^8dRMRp\PRV RV RV R24VPWTRR7pMdVP4'dB\PR V R 24\PR V R VP R V 24\W5VR7pVP 'd\#W1VR7pM3I3I2JJdeodpr 3;X\]  ! ! !8OK   $"2"2";";";8OlpK     +2F2F$--2W3++"&"9"9 K !=!=!? TZZ   $ $ $t7P7PQX7Y)Y  ' '(A(ABU(V WrRc#"R\P9dv^RIpVPPP 4p\ WPPP4'dVeVP4RjxL R#VP4pVP'dVP'gVPP'dZ\VPPVPR7pVP4FwrVVP!V4xK R#VF wrWVxK R#L5i)rNrR)r3r4rrrrMrrrTrrrrJr0rKrr^)rrrQrrwrrrUs& rPrIterableDataset.__iter__ s ckk ! #++**::%#as%BE-E-9AE- E' E-/>E-c&<V^8dQhRS[RS[/#)rTrry)rYr)rZrs"rPr[r " s11s1~1rRc\W4#rMrrs&&rPrIterableDataset.__getitem__" rrRc V<V^8dQhRS[RS[S[,RS[S[,RS[RR/#)rTrrrDr ryr)rrr!rXr3)rZrs"rPr[r & sD4448$4TN4 4  4rRc B^RIHpV!WVRVR7P4#)a|Create an Iterable Dataset from a generator. Args: generator (`Callable`): A generator function that `yields` examples. features (`Features`, *optional*): Dataset features. gen_kwargs(`dict`, *optional*): Keyword arguments to be passed to the `generator` callable. You can define a sharded iterable dataset by passing the list of shards in `gen_kwargs`. This can be used to improve shuffling and when iterating over the dataset with multiple workers. split ([`NamedSplit`], defaults to `Split.TRAIN`): Split name to be assigned to the dataset. Returns: [`IterableDataset`] Example: ```py >>> def gen(): ... yield {"text": "Good", "label": 0} ... yield {"text": "Bad", "label": 1} ... >>> ds = IterableDataset.from_generator(gen) ``` ```py >>> def gen(shards): ... for shard in shards: ... with open(shard) as f: ... for line in f: ... yield {"line": line} ... >>> shards = [f"data{i}.txt" for i in range(32)] >>> ds = IterableDataset.from_generator(gen, gen_kwargs={"shards": shards}) >>> ds = ds.shuffle(seed=42, buffer_size=10_000) # shuffles the shards order + uses a shuffle buffer >>> from torch.utils.data import DataLoader >>> dataloader = DataLoader(ds.with_format("torch"), num_workers=4) # give each worker a subset of 32/4=8 shards ``` )GeneratorDatasetInputStreamT)rrrD streamingr ) io.generatorrkread)rrrDr rks&&&& rPfrom_generatorIterableDataset.from_generator% s'b >*zUYaf $& rRcN<V^8dQhRRRS[S[,RS[S[,RR/#)rTdfzpyspark.sql.DataFramer rryr)rr3r!)rZrs"rPr[r ] s;(( #( #(8$(  (rRc ^RIHp\PR8Xd \ R4hV!V3RVRVRR/VBP 4#)a1Create an IterableDataset from Spark DataFrame. The dataset is streamed to the driver in batches. Args: df (`pyspark.sql.DataFrame`): The DataFrame containing the desired data. split (`NamedSplit`, *optional*): Split name to be assigned to the dataset. features (`Features`, *optional*): Dataset features. Returns: [`IterableDataset`] Example: ```py >>> df = spark.createDataFrame( >>> data=[[1, "Elia"], [2, "Teo"], [3, "Fang"]], >>> columns=["id", "name"], >>> ) >>> ds = IterableDataset.from_spark(df) ``` )SparkDatasetReaderwin32z@IterableDataset.from_spark is not currently supported on Windowsr rrlT)io.sparkrtr3platformOSErrorrn)rrr rr/rts&&&, rP from_sparkIterableDataset.from_spark\ s^< 1 <<7 "\] ]!        $&  rRc$<V^8dQhRS[RR/#)rTfilenameryrr)rZrs"rPr[r  s f fC f$5 frRc \V4p\P!V4p\\P RV/R7p\ V\VR7R7#)zInstantiate a IterableDataset from Arrow table at filename. Args: filename (`str`): File name of the dataset. Returns: [`IterableDataset`] r|r/rR)rr)r:r!rrer _generate_tables_from_cache_filerr1)r|pa_table_schemainferred_featuresrs& rP from_fileIterableDataset.from_file sK09$66G+G,T,T^hjr]st ;[Rc=deerRc<V^8dQhRS[PRS[S[,RS[S[,RS[S[,RS[S[,RS[S[,RR/#) rTrrrrr preserve_indexrryr)r.r/rr!r1r3rrs)rZrs"rPr[r  sl:5:5 LL:58$:5{# :5  # :5 ! :5SM:5 :5rRc X\P!VVVVVR7PVR7#)a Convert `pandas.DataFrame` to a `pyarrow.Table` to create an [`IterableDataset`]. The column types in the resulting Arrow Table are inferred from the dtypes of the `pandas.Series` in the DataFrame. In the case of non-object Series, the NumPy dtype is translated to its Arrow equivalent. In the case of `object`, we need to guess the datatype by looking at the Python objects in this Series. Be aware that Series of the `object` dtype don't carry enough information to always lead to a meaningful Arrow type. In the case that we cannot infer a type, e.g. because the DataFrame is of length 0 or the Series only contains `None/nan` objects, the type is set to `null`. This behavior can be avoided by constructing explicit features and passing it to this function. Important: a dataset created with from_pandas() lives in memory. This may change in the future, but in the meantime if you want to reduce memory usage you should write it on disk and reload using e.g. to_parquet / from_parquet. Args: df (`pandas.DataFrame`): Dataframe that contains the dataset. features ([`Features`], *optional*): Dataset features. info (`DatasetInfo`, *optional*): Dataset information, like description, citation, etc. split (`NamedSplit`, *optional*): Name of the dataset split. preserve_index (`bool`, *optional*): Whether to store the index as an additional column in the resulting Dataset. The default of `None` will store the index as a column, except for `RangeIndex` which is stored as metadata only. Use `preserve_index=True` to force it to be stored as a column. num_shards (`int`, default to `1`): Number of shards to define when instantiating the iterable dataset. This is especially useful for big datasets to be able to shuffle properly, and also to enable fast parallel loading using a PyTorch DataLoader or in distributed setups for example. Returns: [`IterableDataset`] Example: ```py >>> ds = IterableDataset.from_pandas(df) ``` )rrr rry)rr1to_iterable_dataset)rrrrrr rrs&&&&&&&rPr1IterableDataset.from_pandas s7j"" )     4  5rRc t<V^8dQhRS[R,RS[S[,RS[S[,RS[S[,RR/#)rTrrrrr ryr pl.DataFramez pl.LazyFrame)rrr!r1r3)rZrs"rPr[r  sR0 0 0 10 8$0 {# 0  # 0  0 rRc ^RIpVe1Ve-VPV8wd\RV RVP 24hVeTMVe VPMRpVe \V4pVf \ 4pT;'gZ\ P !\WP4'dVP4M VPP44Vn\\\RV/R7VVR7#)ai Create an IterableDataset from a polars DataFrame or LazyFrame. Iterating over the dataset is mostly zero copy. Under the hood, the dataset iterates over the polars DataFrame batches/slices. Data types that do copy: * CategoricalType Args: df (`polars.DataFrame`): DataFrame to convert to Arrow Table features (`Features`, optional): Dataset features. info (`DatasetInfo`, optional): Dataset information, like description, citation, etc. split (`NamedSplit`, optional): Name of the dataset split. Returns: [`IterableDataset`] Examples: ```py >>> ds = IterableDataset.from_polars(df) ``` NzIFeatures specified in `features` and `info.features` can't be different:  rrr~r )r-rrgr'r1r!rr LazyFramecollect_schemar~r5rre_generate_tables_from_polars)rrrrrr r7s&&&&& rP from_polarsIterableDataset.from_polars s>    4(9R\]e\ffhimivivhwx  (38$JZ`d  $>$.r<<$@$@R   bii Y Y [%   !">bz R  rRc |<V^8dQhRS[RS[S[,RS[S[,RS[S[,RS[S[,RR/#rTmappingrrr rryr)rXrr!r1r3rs)rZrs"rPr[r  sZ" " " 8$" {# "  # " SM "  " rRc R\P!WW4R7PVR7#)a Convert `dict` to a `pyarrow.Table` to create an [`IterableDataset`]. Important: a dataset created with from_dict() lives in memory. This may change in the future, but in the meantime if you want to reduce memory usage you should write it back on disk and reload using e.g. to_parquet / from_parquet. Args: mapping (`Mapping`): Mapping of strings to Arrays or Python lists. features ([`Features`], *optional*): Dataset features. info (`DatasetInfo`, *optional*): Dataset information, like description, citation, etc. split (`NamedSplit`, *optional*): Name of the dataset split. num_shards (`int`, default to `1`): Number of shards to define when instantiating the iterable dataset. This is especially useful for big datasets to be able to shuffle properly, and also to enable fast parallel loading using a PyTorch DataLoader or in distributed setups for example. Returns: [`IterableDataset`] rrr ry)r from_dictrrrrrr rs&&&&&&rPrIterableDataset.from_dict s.B  $Thh!i  rRc <V^8dQhRS[S[,RS[S[,RS[S[,RS[S[,RS[S[,RR/#r)rhrXrr!r1r3rs)rZrs"rPr[r - s^$5$5d$58$$5{# $5  # $5 SM $5 $5rRc V\P!VVVVR7PVR7#)a, Convert a list of dicts to a `pyarrow.Table` to create an [`IterableDataset`]`. Note that the keys of the first entry will be used to determine the dataset columns, regardless of what is passed to features. Important: a dataset created with from_list() lives in memory. This may change in the future, but in the meantime if you want to reduce memory usage you should write it back on disk and reload using e.g. from_parquet / to_parquet. Args: mapping (`List[dict]`): A list of mappings of strings to row values. features (`Features`, optional): Dataset features. info (`DatasetInfo`, optional): Dataset information, like description, citation, etc. split (`NamedSplit`, optional): Name of the dataset split. num_shards (`int`, default to `1`): Number of shards to define when instantiating the iterable dataset. This is especially useful for big datasets to be able to shuffle properly, and also to enable fast parallel loading using a PyTorch DataLoader or in distributed setups for example. Returns: [`IterableDataset`] rry)r from_listrrs&&&&&&rPrIterableDataset.from_list, s4@       4  5rRc |<V^8dQhRS[S[S[S[,3,RS[S[,RS[S[,RS[RR/#)rT path_or_pathsr rkeep_in_memoryryr)rrErhrr3r!r)rZrs"rPr[r T sS((XtH~56( #(8$( (  (rRc L^RIHpV!V3RVRVRVRR/VBP4#)afCreate an IterableDataset from CSV file(s). Args: path_or_paths (`path-like` or list of `path-like`): Path(s) of the CSV file(s). split ([`NamedSplit`], *optional*): Split name to be assigned to the dataset. features ([`Features`], *optional*): Dataset features. keep_in_memory (`bool`, defaults to `False`): Whether to copy the data in-memory. **kwargs (additional keyword arguments): Keyword arguments to be passed to [`pandas.read_csv`]. Returns: [`IterableDataset`] Example: ```py >>> ds = IterableDataset.from_csv('path/to/dataset.csv') ``` )CsvDatasetReaderr rrrlT)io.csvrrn)rr rrr/rs&&&&, rPfrom_csvIterableDataset.from_csvS sQ@ -    *       $& rRc <V^8dQhRS[S[S[S[,3,RS[S[,RS[S[,RS[RS[S[,RR/#)rTrr rrfieldryr)rrErhrr3r!rrY)rZrs"rPr[r  sa,,XtH~56, #,8$, , } , ,rRc P^RIHpV!V3RVRVRVRVRR/VBP4#)aCreate an IterableDataset from JSON or JSON Lines file(s). Args: path_or_paths (`path-like` or list of `path-like`): Path(s) of the JSON or JSON Lines file(s). split ([`NamedSplit`], *optional*): Split name to be assigned to the dataset. features ([`Features`], *optional*): Dataset features. keep_in_memory (`bool`, defaults to `False`): Whether to copy the data in-memory. field (`str`, *optional*): Field name of the JSON file where the dataset is contained in. **kwargs (additional keyword arguments): Keyword arguments to be passed to [`JsonConfig`]. Returns: [`IterableDataset`] Example: ```py >>> ds = IterableDataset.from_json('path/to/dataset.json') ``` )JsonDatasetReaderr rrrrlT)io.jsonrrn)rr rrrr/rs&&&&&, rP from_jsonIterableDataset.from_json~ s[F /    *        $& rRcT<V^8dQhRS[S[S[S[,3,RS[S[,RS[S[,RS[RS[S[S[,,RS[S[S[PS[S[ ,S[S[S[ ,,3,,RS[S[P,RS[ R ,R R / #) rTrr rrrfiltersfragment_scan_options on_bad_filesryr)errorwarnskip) rrErhrr3r!rrYpds ExpressionrParquetFragmentScanOptionsr>)rZrs"rPr[r  sRRXtH~56R #R8$R R $s)$ R %U T$u+=N NOP R ((F(FGR56R RrRc \^RIHp V !V3RVRVRVRVRRRVR VR V/VBP4#) aCreate an IterableDataset from Parquet file(s). Args: path_or_paths (`path-like` or list of `path-like`): Path(s) of the Parquet file(s). split (`NamedSplit`, *optional*): Split name to be assigned to the dataset. features (`Features`, *optional*): Dataset features. keep_in_memory (`bool`, defaults to `False`): Whether to copy the data in-memory. columns (`List[str]`, *optional*): If not `None`, only these columns will be read from the file. A column name may be a prefix of a nested field, e.g. 'a' will select 'a.b', 'a.c', and 'a.d.e'. filters (`Union[pyarrow.dataset.Expression, list[tuple], list[list[tuple]]]`, *optional*): Return only the rows matching the filter. If possible the predicate will be pushed down to exploit the partition information or internal metadata found in the data source, e.g. Parquet statistics. Otherwise filters the loaded RecordBatches before yielding them. fragment_scan_options (`pyarrow.dataset.ParquetFragmentScanOptions`, *optional*) Scan-specific options for Parquet fragments. This is especially useful to configure buffering and caching. on_bad_files (`Literal["error", "warn", "skip"]`, *optional*, defaults to "error") Specify what to do upon encountering a bad file (a file that can't be read). Allowed values are : * 'error', raise an Exception when a bad file is encountered. * 'warn', raise a warning when a bad file is encountered and skip that file. * 'skip', skip bad files without raising or warning when they are encountered. **kwargs (additional keyword arguments): Keyword arguments to be passed to [`ParquetConfig`]. Returns: [`IterableDataset`] Example: ```py >>> ds = IterableDataset.from_parquet('path/to/dataset.parquet') ``` Load a subset of columns: ```python >>> ds = IterableDataset.from_parquet('path/to/dataset.parquet', columns=["col_0", "col_1"]) ``` Efficiently filter data, possibly skipping entire files or row groups: ```python >>> filters = [("col_0", "==", 0)] >>> ds = IterableDataset.from_parquet(parquet_files_list, filters=filters) ``` )ParquetDatasetReaderr rrrrlTrrr) io.parquetrrn) rr rrrrrrr/rs &&&&&&&&, rP from_parquetIterableDataset.from_parquet syL 5#       *        #8  &     $& rRc<V^8dQhRS[S[S[S[,3,RS[S[,RS[S[,RS[RS[RS[R ,RR/#) rTrr rrkeep_linebreaks sample_byryr)line paragraphdocument)rrErhrr3r!rr>)rZrs"rPr[r  sm11XtH~561 #18$1 1  1 :; 1 1rRc T^RIHpV!V3RVRVRVRRRVRV/VBP4#) a|Create an IterableDataset from text file(s). Args: path_or_paths (`path-like` or list of `path-like`): Path(s) of the text file(s). split (`NamedSplit`, *optional*): Split name to be assigned to the dataset. features (`Features`, *optional*): Dataset features. keep_in_memory (`bool`, defaults to `False`): Whether to copy the data in-memory. keep_linebreaks: (`bool`, defaults to False): Whether to keep line breaks. sample_by (`Literal["line", "paragraph", "document"]`, defaults to "line"): Whether to load data per line, praragraph or document. By default one row in the dataset = one line. **kwargs (additional keyword arguments): Keyword arguments to be passed to [`TextConfig`]. Returns: [`IterableDataset`] Example: ```py >>> ds = IterableDataset.from_text('path/to/dataset.txt') ``` )TextDatasetReaderr rrrlTrr)io.textrrn)rr rrrrr/rs&&&&&&, rP from_textIterableDataset.from_text seN /       *    ,        $& rRc4<V^8dQhRS[S[,RR/#)rTrryrr)rZrs"rPr[r 6 s#7 7 sm7  7 rRc  \V4p\VPVPP 4VP \ VR7\VP4VPR7#)a\ Return a dataset with the specified format. Args: type (`str`, *optional*): Either output type selected in `[None, 'numpy', 'torch', 'tensorflow', 'jax', 'arrow', 'pandas', 'polars']`. `None` means it returns python objects (default). Example: ```py >>> from datasets import load_dataset >>> from transformers import AutoTokenizer >>> ds = load_dataset("cornell-movie-review-data/rotten_tomatoes", split="validation", streaming=True) >>> tokenizer = AutoTokenizer.from_pretrained("bert-base-cased") >>> ds = ds.map(lambda x: tokenizer(x['text'], truncation=True, padding=True), batched=True) >>> ds = ds.with_format("torch") >>> next(iter(ds)) {'text': 'compassionately explores the seemingly irreconcilable situation between conservative christian parents and their estranged gay and lesbian children .', 'label': tensor(1), 'input_ids': tensor([ 101, 18027, 16310, 16001, 1103, 9321, 178, 11604, 7235, 6617, 1742, 2165, 2820, 1206, 6588, 22572, 12937, 1811, 2153, 1105, 1147, 12890, 19587, 6463, 1105, 15026, 1482, 119, 102, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]), 'token_type_ids': tensor([0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]), 'attention_mask': tensor([1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0])} ``` rKrrr rBr r) r/rrr'r_splitrCrrr)rrs&&rP with_formatIterableDataset.with_format6 s[X*$/))"++'D9 !2!23"55   rRc<V^8dQhRS[S[,RS[RS[S[S[S[S[,3,,RS[RS[S[,RS[RS[S[S[S[S[,3,,RS[S[,R S[S[,R R / #) rTr<r=r>r?rrr@rrAryr rrrrrYrhrsr!rX)rZrs"rPr[r o s] ] 8$] ]  c49n 56 ]  ] SM ] ] !sDI~!67] 8$] D>]  ] rRc  6VPVVVVVVVVV R7 #)a Apply a function to all the examples in the iterable dataset (individually or in batches) and update them. If your function returns a column that already exists, then it overwrites it. The function is applied on-the-fly on the examples when iterating over the dataset. You can specify whether the function should be batched or not with the `batched` parameter: - If batched is `False`, then the function takes 1 example in and should return 1 example. An example is a dictionary, e.g. `{"text": "Hello there !"}`. - If batched is `True` and `batch_size` is 1, then the function takes a batch of 1 example as input and can return a batch with 1 or more examples. A batch is a dictionary, e.g. a batch of 1 example is {"text": ["Hello there !"]}. - If batched is `True` and `batch_size` is `n` > 1, then the function takes a batch of `n` examples as input and can return a batch with `n` examples, or with an arbitrary number of examples. Note that the last batch may have less than `n` examples. A batch is a dictionary, e.g. a batch of `n` examples is `{"text": ["Hello there !"] * n}`. If the function is asynchronous, then `map` will run your function in parallel, with up to one thousand simulatenous calls. It is recommended to use a `asyncio.Semaphore` in your function if you want to set a maximum number of operations that can run at the same time. Args: function (`Callable`, *optional*, defaults to `None`): Function applied on-the-fly on the examples when you iterate on the dataset. It must have one of the following signatures: - `function(example: Dict[str, Any]) -> Dict[str, Any]` if `batched=False` and `with_indices=False` - `function(example: Dict[str, Any], idx: int) -> Dict[str, Any]` if `batched=False` and `with_indices=True` - `function(batch: Dict[str, List]) -> Dict[str, List]` if `batched=True` and `with_indices=False` - `function(batch: Dict[str, List], indices: List[int]) -> Dict[str, List]` if `batched=True` and `with_indices=True` For advanced usage, the function can also return a `pyarrow.Table`. If the function is asynchronous, then `map` will run your function in parallel. Moreover if your function returns nothing (`None`), then `map` will run your function and return the dataset unchanged. If no function is provided, default to identity function: `lambda x: x`. with_indices (`bool`, defaults to `False`): Provide example indices to `function`. Note that in this case the signature of `function` should be `def function(example, idx[, rank]): ...`. input_columns (`Optional[Union[str, List[str]]]`, defaults to `None`): The columns to be passed into `function` as positional arguments. If `None`, a dict mapping to all formatted columns is passed as one argument. batched (`bool`, defaults to `False`): Provide batch of examples to `function`. batch_size (`int`, *optional*, defaults to `1000`): Number of examples per batch provided to `function` if `batched=True`. `batch_size <= 0` or `batch_size == None` then provide the full dataset as a single batch to `function`. drop_last_batch (`bool`, defaults to `False`): Whether a last batch smaller than the batch_size should be dropped instead of being processed by the function. remove_columns (`[List[str]]`, *optional*, defaults to `None`): Remove a selection of columns while doing the mapping. Columns will be removed before updating the examples with the output of `function`, i.e. if `function` is adding columns with names in `remove_columns`, these columns will be kept. features (`[Features]`, *optional*, defaults to `None`): Feature types of the resulting dataset. fn_kwargs (`Dict`, *optional*, default `None`): Keyword arguments to be passed to `function`. Example: ```py >>> from datasets import load_dataset >>> ds = load_dataset("cornell-movie-review-data/rotten_tomatoes", split="train", streaming=True) >>> def add_prefix(example): ... example["text"] = "Review: " + example["text"] ... return example >>> ds = ds.map(add_prefix) >>> list(ds.take(3)) [{'label': 1, 'text': 'Review: the rock is destined to be the 21st century's new " conan " and that he's going to make a splash even greater than arnold schwarzenegger , jean-claud van damme or steven segal .'}, {'label': 1, 'text': 'Review: the gorgeously elaborate continuation of " the lord of the rings " trilogy is so huge that a column of words cannot adequately describe co-writer/director peter jackson's expanded vision of j . r . r . tolkien's middle-earth .'}, {'label': 1, 'text': 'Review: effective but too-tepid biopic'}] ``` ) r<r=r>r?rrr@rrA)_map) rr<r=r>r?rrr@rrAs &&&&&&&&&&rPmapIterableDataset.mapo s6fyy%'!+)  rRc<V^8dQhRS[S[,RS[RS[S[S[S[S[,3,,RS[RS[S[,RS[RS[S[S[S[S[,3,,RS[S[,R S[S[,R S[R R / #) rTr<r=r>r?rrr@rrArEryrr)rZrs"rPr[r  sU U 8$U U  c49n 56 U  U SM U U !sDI~!67U 8$U D>U 37U  U rRc \V\4'dV.p\V\4'dV.pVf\pV f/p Ve \V4pVPp V P 'd@VP Pe&VP PV P8XdRMVP Pp VP'dcVPP'dG\V \VP4V VPR7p \T V'dTM^VRR7p MVPP'd=VP'g V 'd#\VPV'dTM^VR7p VP'g V 'd.\V \VP4V VPRR7p \V VVVVVVVV VPVV R7 p VP P#4p Wn\%V V VP&VP\VP(4VPR7#)Nr\Tr[r)rBrrr) r<r=r>r?rrr@rArBrrEr)rrYrQr'rrr'rrrJrrrrrr:rrrrr)rr<r=r>r?rrr@rrArErinput_featuresrs&&&&&&&&&&& rPrIterableDataset._map s  mS ) )*OM nc * *,-N  $H  I  7'(8(89+&*&=&=,0  - %'!+)''5]  yy~~ #++'' !2!23"55   rRc <V^8dQhRS[S[,RS[S[S[S[S[,3,,RS[RS[S[,RS[S[,RR/#)rTr<r>r?rrAryr)rrrrYrhrrsrX)rZrs"rPr[r %snS S 8$S  c49n 56 S  S SM S D>S  S rRc  $\V\4'dV.pVPpVPP'gVP 'dV\ TVP VP'd VPMVPPVPR7p\VVVVVVVVP R7p\VVPVPVP \VP4VPR7#)a Apply a filter function to all the elements so that the dataset only includes examples according to the filter function. The filtering is done on-the-fly when iterating over the dataset. If the function is asynchronous, then `filter` will run your function in parallel, with up to one thousand simulatenous calls (configurable). It is recommended to use a `asyncio.Semaphore` in your function if you want to set a maximum number of operations that can run at the same time. Args: function (`Callable`): Callable with one of the following signatures: - `function(example: Dict[str, Any]) -> bool` if `with_indices=False, batched=False` - `function(example: Dict[str, Any], indices: int) -> bool` if `with_indices=True, batched=False` - `function(example: Dict[str, List]) -> List[bool]` if `with_indices=False, batched=True` - `function(example: Dict[str, List], indices: List[int]) -> List[bool]` if `with_indices=True, batched=True` If the function is asynchronous, then `filter` will run your function in parallel. If no function is provided, defaults to an always True function: `lambda x: True`. with_indices (`bool`, defaults to `False`): Provide example indices to `function`. Note that in this case the signature of `function` should be `def function(example, idx): ...`. input_columns (`str` or `List[str]`, *optional*): The columns to be passed into `function` as positional arguments. If `None`, a dict mapping to all formatted columns is passed as one argument. batched (`bool`, defaults to `False`): Provide batch of examples to `function`. batch_size (`int`, *optional*, default `1000`): Number of examples per batch provided to `function` if `batched=True`. fn_kwargs (`Dict`, *optional*, default `None`): Keyword arguments to be passed to `function`. Example: ```py >>> from datasets import load_dataset >>> ds = load_dataset("cornell-movie-review-data/rotten_tomatoes", split="train", streaming=True) >>> ds = ds.filter(lambda x: x["label"] == 0) >>> list(ds.take(3)) [{'label': 0, 'movie_review': 'simplistic , silly and tedious .'}, {'label': 0, 'movie_review': "it's so laddish and juvenile , only teenage boys could possibly find it funny ."}, {'label': 0, 'movie_review': 'exploitative and largely devoid of the depth or sophistication that would make watching such a graphic treatment of the crimes bearable .'}] ``` r\rr)rrYrr'rrrrrrrrrr)rr<r=r>r?rrArs&&&&&&& rPrIterableDataset.filter%sh mS ) )*OM'' ::   $"2"2"23++1<1E1E1E--4::K^K^"&"9"9 K/ %'!''  #++'' !2!23"55   rRch<V^8dQhRS[S[PP,RS[RS[RR/#)rTrrmax_buffer_input_shardsryr)rrrrrs)rZrs"rPr[r zsJ` ` BII//0`  ` "% `  ` rRc  ~Vf!\PPV4pM \V4pVPpVP V4pVP'd\V^R7pV^8dM\VPV4p\\V4Uu.uFpVPWgR7NK upRR7p\WSVR7p\VVP P#4VP$VP&\VP(4VP*R7# \ d^pLi;iuupi)a  Randomly shuffles the elements of this dataset. This dataset fills a buffer with `buffer_size` elements, then randomly samples elements from this buffer, replacing the selected elements with new elements. For perfect shuffling, a buffer size greater than or equal to the full size of the dataset is required. For instance, if your dataset contains 10,000 elements but `buffer_size` is set to 1000, then `shuffle` will initially select a random element from only the first 1000 elements in the buffer. Once an element is selected, its space in the buffer is replaced by the next (i.e. 1,001-st) element, maintaining the 1000 element buffer. If the dataset is made of several shards, it fills the buffer using up to `max_buffer_input_shards` shards at a time and also does shuffle the order of the shards. This greatly improves the quality of the shuffling. However if the order has been fixed by using [`~datasets.IterableDataset.skip`] or [`~datasets.IterableDataset.take`] then the order of the shards is kept unchanged and only one shard at a time is used to fill the buffer. Args: seed (`int`, *optional*, defaults to `None`): Random seed that will be used to shuffle the dataset. It is used to sample from the shuffle buffer and also to shuffle the data shards. generator (`numpy.random.Generator`, *optional*): Numpy random Generator to use to compute the permutation of the dataset rows. If `generator=None` (default), uses `np.random.default_rng` (the default BitGenerator (PCG64) of NumPy). buffer_size (`int`, defaults to `1000`): Size of the buffer. max_buffer_input_shards (`int`, defaults to `101000`): Maximum number of shards to use to feed the buffer at a time. Example: ```py >>> from datasets import load_dataset >>> ds = load_dataset("cornell-movie-review-data/rotten_tomatoes", split="train", streaming=True) >>> list(ds.take(3)) [{'text': 'the rock is destined to be the 21st century's new " conan " and that he's going to make a splash even greater than arnold schwarzenegger , jean-claud van damme or steven segal .', 'label': 1}, {'text': 'the gorgeously elaborate continuation of " the lord of the rings " trilogy is so huge that a column of words cannot adequately describe co-writer/director peter jackson's expanded vision of j . r . r . tolkien's middle-earth .', 'label': 1}, {'text': 'effective but too-tepid biopic', 'label': 1}] >>> shuffled_ds = ds.shuffle(seed=42) >>> list(shuffled_ds.take(3)) [{'text': "a sports movie with action that's exciting on the field and a story you care about off it .", 'label': 1}, {'text': 'at its best , the good girl is a refreshingly adult take on adultery . . .', 'label': 1}, {'text': "sam jones became a very lucky filmmaker the day wilco got dropped from their record label , proving that one man's ruin may be another's fortune .", 'label': 1}] >>> resharded_ds = ds.reshard() # useful to shard Parquet datasets by row group instead of by file, improving shuffling quality on dataset with one or few files. >>> shuffled_resharded_ds = resharded_ds.shuffle(seed=42) >>> list(shuffled_resharded_ds.take(3)) [{'text': 'this mistaken-identity picture is so film-culture referential that the final product is a ghost .', 'label': 0}, {'text': 'woody allen used to ridicule movies like hollywood ending . now he makes them .', 'label': 0}, {'text': "not only is undercover brother as funny , if not more so , than both austin powers films , but it's also one of the smarter , savvier spoofs to come along in some time .", 'label': 1}] ``` rrr-rr)rr rBr r)rrrrrrr'rrrrr'rrrrr'rrrrr)rrrrrrnum_shards_to_interleavers&&&&& rPrIterableDataset.shufflezs1H   --d3I +I''  (%::9EK  ! ! !8QRSK "Q &'*;+A+ACZ'[ $="''?!@!@ 22>V2d!@#F K5[eno  "++'' !2!23"55   . (&' # (sD'D:' D76D7c <V^8dQhRS[/#)rTr7r)rZrs"rPr[r s++s+rRcXV;PWP, , unR#rM)r)rr7s&&rP set_epochIterableDataset.set_epochs u{{** rRc$<V^8dQhRS[RR/#rTr,ryrr)rZrs"rPr[r s( ( c( /( rRc  \VPVVPRJR7p\VVPP 4VP VP\VP4VPR7#)a  Create a new [`IterableDataset`] that skips the first `n` elements. Args: n (`int`): Number of elements to skip. Example: ```py >>> from datasets import load_dataset >>> ds = load_dataset("cornell-movie-review-data/rotten_tomatoes", split="train", streaming=True) >>> list(ds.take(3)) [{'label': 1, 'text': 'the rock is destined to be the 21st century's new " conan " and that he's going to make a splash even greater than arnold schwarzenegger , jean-claud van damme or steven segal .'}, {'label': 1, 'text': 'the gorgeously elaborate continuation of " the lord of the rings " trilogy is so huge that a column of words cannot adequately describe co-writer/director peter jackson's expanded vision of j . r . r . tolkien's middle-earth .'}, {'label': 1, 'text': 'effective but too-tepid biopic'}] >>> ds = ds.skip(1) >>> list(ds.take(3)) [{'label': 1, 'text': 'the gorgeously elaborate continuation of " the lord of the rings " trilogy is so huge that a column of words cannot adequately describe co-writer/director peter jackson's expanded vision of j . r . r . tolkien's middle-earth .'}, {'label': 1, 'text': 'effective but too-tepid biopic'}, {'label': 1, 'text': 'if you sometimes like to go to the movies to have fun , wasabi is a good place to start .'}] ``` Nr.r) r)rrrr'rrrrrrr,rs&& rPrIterableDataset.skipso8+    $ 1 1T 9 #"++'' !2!23"55   rRc4<V^8dQhRS[S[,RR/#)rTrhryrr)rZrs"rPr[r  s ( (  ( 2C( rRc  \\VPVR7VPVPVP \ VP4VPR7#)a Create a new [`IterableDataset`] that repeats the underlying dataset `num_times` times. N.B. The effect of calling shuffle after repeat depends significantly on buffer size. With buffer_size 1, duplicate data is never seen in the same iteration, even after shuffling: ds.repeat(n).shuffle(seed=42, buffer_size=1) is equivalent to ds.shuffle(seed=42, buffer_size=1).repeat(n), and only shuffles shard orders within each iteration. With buffer size >= (num samples in the dataset * num_times), we get full shuffling of the repeated data, i.e. we can observe duplicates in the same iteration. Args: num_times (`int`) or (`None`): Number of times to repeat the dataset. If `None`, the dataset will be repeated indefinitely. Example: ```py >>> from datasets import load_dataset >>> ds = load_dataset("cornell-movie-review-data/rotten_tomatoes", split="train") >>> ds = ds.take(2).repeat(2) >>> list(ds) [{'label': 1, 'text': 'the rock is destined to be the 21st century's new " conan " and that he's going to make a splash even greater than arnold schwarzenegger , jean-claud van damme or steven segal .'}, {'label': 1, 'text': 'the gorgeously elaborate continuation of " the lord of the rings " trilogy is so huge that a column of words cannot adequately describe co-writer/director peter jackson's expanded vision of j . r . r . tolkien's middle-earth .'}, {'label': 1, 'text': 'effective but too-tepid biopic'}, {'label': 1, 'text': 'the rock is destined to be the 21st century's new " conan " and that he's going to make a splash even greater than arnold schwarzenegger , jean-claud van damme or steven segal .'}, {'label': 1, 'text': 'the gorgeously elaborate continuation of " the lord of the rings " trilogy is so huge that a column of words cannot adequately describe co-writer/director peter jackson's expanded vision of j . r . r . tolkien's middle-earth .'}, {'label': 1, 'text': 'effective but too-tepid biopic'}] ``` rur) rrfrr'rrrrr)rrhs&&rPrepeatIterableDataset.repeat sPB.t/@/@IV++'' !2!23"55   rRc$<V^8dQhRS[RR/#rr)rZrs"rPr[r 3s! ! c! /! rRc  \VPVVPRJR7p\VVPP 4VP VP\VP4VPR7#)a Create a new [`IterableDataset`] with only the first `n` elements. Args: n (`int`): Number of elements to take. Example: ```py >>> from datasets import load_dataset >>> ds = load_dataset("cornell-movie-review-data/rotten_tomatoes", split="train", streaming=True) >>> small_ds = ds.take(2) >>> list(small_ds) [{'label': 1, 'text': 'the rock is destined to be the 21st century's new " conan " and that he's going to make a splash even greater than arnold schwarzenegger , jean-claud van damme or steven segal .'}, {'label': 1, 'text': 'the gorgeously elaborate continuation of " the lord of the rings " trilogy is so huge that a column of words cannot adequately describe co-writer/director peter jackson's expanded vision of j . r . r . tolkien's middle-earth .'}] ``` Nrr) rrrrr'rrrrrrs&& rPrIterableDataset.take3so*+    $ 1 1T 9 #"++'' !2!23"55   rRc0<V^8dQhRS[RS[RS[RR/#)rTrrrryrrc)rZrs"rPr[r Vs36 6 6 6  6  6 rRc  VPPWVR7p\VVPP 4VP VP \VP4VPR7#)aAReturn the `index`-nth shard from dataset split into `num_shards` pieces. This shards deterministically. `dataset.shard(n, i)` splits the dataset into contiguous chunks, so it can be easily concatenated back together after processing. If `dataset.num_shards % n == l`, then the first `l` datasets each have `(dataset.num_shards // n) + 1` shards, and the remaining datasets have `(dataset.num_shards // n)` shards. `datasets.concatenate_datasets([dset.shard(n, i) for i in range(n)])` returns a dataset with the same order as the original. In particular, `dataset.shard(dataset.num_shards, i)` returns a dataset with 1 shard. Note: n should be less or equal to the number of shards in the dataset `dataset.num_shards`. On the other hand, `dataset.shard(n, i, contiguous=False)` contains all the shards of the dataset whose index mod `n = i`. Be sure to shard before using any randomizing operator (such as `shuffle`). It is best if the shard operator is used early in the dataset pipeline. Args: num_shards (`int`): How many shards to split the dataset into. index (`int`): Which shard to select and return. contiguous: (`bool`, defaults to `True`): Whether to select contiguous blocks of indices for shards. Example: ```py >>> from datasets import load_dataset >>> ds = load_dataset("fancyzhx/amazon_polarity", split="train", streaming=True) >>> ds IterableDataset({ features: ['label', 'title', 'content'], num_shards: 4 }) >>> ds.shard(num_shards=2, index=0) IterableDataset({ features: ['label', 'title', 'content'], num_shards: 2 }) ``` rFr) rrrr'rrrrrr)rrrrrs&&&& rPshardIterableDataset.shardVsf\''::jjt:u #"++'' !2!23"55   rRc<V^8dQhRR/#)rTryrrN)rZrs"rPr[r s& & *& rRc  VPP4p\VVPP 4VP VP \VP4VPR7#)aReshard the dataset if possible, i.e. split the current shards further into more shards. This increases the number of shards and the resulting dataset has num_shards >= previous_num_shards. Equality may happen if no shard can be split further. The resharding mechanism depends on the dataset file format: * Parquet: shard per row group instead of per file * Other: not implemented yet (contributions are welcome !) Be sure to reshard/shard before using any randomizing operator (such as `shuffle`). It is best if the shard operator is used early in the dataset pipeline. Example: ```py >>> from datasets import load_dataset >>> ds = load_dataset("fancyzhx/amazon_polarity", split="train", streaming=True) >>> ds IterableDataset({ features: ['label', 'title', 'content'], num_shards: 4 }) >>> ds.reshard() IterableDataset({ features: ['label', 'title', 'content'], num_shards: 3600 }) ``` r) rrrr'rrrrrrrJs& rPreshardIterableDataset.reshards]<''<<> #"++'' !2!23"55   rRcT<V^8dQhRS[RS[S[S[P3,RR/#)rTrqrrryr)rYrrhrr)rZrs"rPr[r s5 ] ]s ]E$.,A ]FW ]rRc FVP\\WR7RR7#)zAdd column to Dataset. Args: name (str): Column name. column (list or np.array): Column data to be added. Returns: `IterableDataset` )rqrrT)r=)rr ru)rrqrrs&&&rP add_columnIterableDataset.add_columnsxx DHW[x\\rRc*<V^8dQhRS[RS[RR/#)rTrlrmryrr)rZrs"rPr[r s(LL#LLPaLrRc &VPW/4#)a Rename a column in the dataset, and move the features associated to the original column under the new column name. Args: original_column_name (`str`): Name of the column to rename. new_column_name (`str`): New name for the column. Returns: `IterableDataset`: A copy of the dataset with a renamed column. Example: ```py >>> from datasets import load_dataset >>> ds = load_dataset("cornell-movie-review-data/rotten_tomatoes", split="train", streaming=True) >>> next(iter(ds)) {'label': 1, 'text': 'the rock is destined to be the 21st century's new " conan " and that he's going to make a splash even greater than arnold schwarzenegger , jean-claud van damme or steven segal .'} >>> ds = ds.rename_column("text", "movie_review") >>> next(iter(ds)) {'label': 1, 'movie_review': 'the rock is destined to be the 21st century's new " conan " and that he's going to make a splash even greater than arnold schwarzenegger , jean-claud van damme or steven segal .'} ``` )rename_columns)rrlrms&&&rP rename_columnIterableDataset.rename_columns8""$8#JKKrRc:<V^8dQhRS[S[S[3,RR/#)rTrVryrrW)rZrs"rPr[r s$T#s(^@QrRc  VPP'd%VPPP4MRpVP\ \ VR7\ V4R7pVeZ\VP4UUu/uF$wrEWAP49d W,MTVbK& upp4VPnV#uuppi)a) Rename several columns in the dataset, and move the features associated to the original columns under the new column names. Args: column_mapping (`Dict[str, str]`): A mapping of columns to rename to their new names Returns: `IterableDataset`: A copy of the dataset with renamed columns N)rVr{) r'rrrr rnrhr!rkr)rrVoriginal_features ds_iterabler`features&& rPrIterableDataset.rename_columnss;?**:M:M:MDJJ//446SWhh &~ FW[\jWk   ()1):(?(?(A(A ,/2E2E2G+GN'SRYY(A*K   &  s *C cJ<V^8dQhRS[S[S[S[,3,RR/#rTrryrrrYrh)rZrs"rPr[r s)!!5d3i+@!EV!rRc dVPP'd%VPPP4MRpVPVR7pVeXVP4VPnVP 4F$wrEWA9gK VPPVK& V#)a Remove one or several column(s) in the dataset and the features associated to them. The removal is done on-the-fly on the examples when iterating over the dataset. Args: column_names (`Union[str, List[str]]`): Name of the column(s) to remove. Returns: `IterableDataset`: A copy of the dataset object without the columns to remove. Example: ```py >>> from datasets import load_dataset >>> ds = load_dataset("cornell-movie-review-data/rotten_tomatoes", split="train", streaming=True) >>> next(iter(ds)) {'text': 'the rock is destined to be the 21st century's new " conan " and that he's going to make a splash even greater than arnold schwarzenegger , jean-claud van damme or steven segal .', 'label': 1} >>> ds = ds.remove_columns("label") >>> next(iter(ds)) {'text': 'the rock is destined to be the 21st century's new " conan " and that he's going to make a splash even greater than arnold schwarzenegger , jean-claud van damme or steven segal .'} ``` Nr{)r'rrrrk)rrrrr`rs&& rPr@IterableDataset.remove_columnss2;?**:M:M:MDJJ//446SWhhlh;  ():)?)?)AK   &+113&#))22374rRcJ<V^8dQhRS[S[S[S[,3,RR/#r r )rZrs"rPr[r s)0 0 5d3i+@0 EV0 rRc  \V\4'dV.pVP'd\VP4pVPPe\ V4\ VPPP 44, pV'dG\R\V4 R\VPPP 44 R24h\VUu/uFqDVPV,bK up4Vn\VPV4p\VXVPVPVPVP R7#uupi)aSelect one or several column(s) in the dataset and the features associated to them. The selection is done on-the-fly on the examples when iterating over the dataset. Args: column_names (`Union[str, List[str]]`): Name of the column(s) to select. Returns: `IterableDataset`: A copy of the dataset object with selected columns. Example: ```py >>> from datasets import load_dataset >>> ds = load_dataset("cornell-movie-review-data/rotten_tomatoes", split="train", streaming=True) >>> next(iter(ds)) {'text': 'the rock is destined to be the 21st century's new " conan " and that he's going to make a splash even greater than arnold schwarzenegger , jean-claud van damme or steven segal .', 'label': 1} >>> ds = ds.select_columns("text") >>> next(iter(ds)) {'text': 'the rock is destined to be the 21st century's new " conan " and that he's going to make a splash even greater than arnold schwarzenegger , jean-claud van damme or steven segal .'} ``` z Column name z- not in the dataset. Columns in the dataset: rnr)rrYr'rrrjrrgrhr!rrrrrrr)rrrmissing_columnsrrs&& rPselect_columnsIterableDataset.select_columnss#2 lC ( ((>L :::DJJ'Dzz""."%l"3c$**:M:M:R:R:T6U"U"$&tO'<&=>< 3 3 8 8 :;>> from datasets import load_dataset, Audio >>> ds = load_dataset("PolyAI/minds14", name="en-US", split="train", streaming=True) >>> ds.features {'audio': Audio(sampling_rate=8000, mono=True, decode=True, id=None), 'english_transcription': Value('string'), 'intent_class': ClassLabel(num_classes=14, names=['abroad', 'address', 'app_error', 'atm_limit', 'balance', 'business_loan', 'card_issues', 'cash_deposit', 'direct_debit', 'freeze', 'high_value_payment', 'joint_account', 'latest_transactions', 'pay_bill']), 'lang_id': ClassLabel(num_classes=14, names=['cs-CZ', 'de-DE', 'en-AU', 'en-GB', 'en-US', 'es-ES', 'fr-FR', 'it-IT', 'ko-KR', 'nl-NL', 'pl-PL', 'pt-PT', 'ru-RU', 'zh-CN']), 'path': Value('string'), 'transcription': Value('string')} >>> ds = ds.cast_column("audio", Audio(sampling_rate=16000)) >>> ds.features {'audio': Audio(sampling_rate=16000, mono=True, decode=True, id=None), 'english_transcription': Value('string'), 'intent_class': ClassLabel(num_classes=14, names=['abroad', 'address', 'app_error', 'atm_limit', 'balance', 'business_loan', 'card_issues', 'cash_deposit', 'direct_debit', 'freeze', 'high_value_payment', 'joint_account', 'latest_transactions', 'pay_bill']), 'lang_id': ClassLabel(num_classes=14, names=['cs-CZ', 'de-DE', 'en-AU', 'en-GB', 'en-US', 'es-ES', 'fr-FR', 'it-IT', 'ko-KR', 'nl-NL', 'pl-PL', 'pt-PT', 'ru-RU', 'zh-CN']), 'path': Value('string'), 'transcription': Value('string')} ``` r r'r'rrrrrrrrr)rrrrrs&&& rP cast_columnIterableDataset.cast_columnNskD8@zz  ' f))++'' !2!23"55   rRc$<V^8dQhRS[RR/#)rTrryrr )rZrs"rPr[r |s, , ,  , rRc  \V4pVPP4pWn\ VP VVP VP\VP4VPR7#)a Cast the dataset to a new set of features. Args: features ([`Features`]): New features to cast the dataset to. The name of the fields in the features must match the current column names. The type of the data must also be convertible from one type to the other. For non-trivial conversion, e.g. `string` <-> `ClassLabel` you should use [`~Dataset.map`] to update the Dataset. Returns: `IterableDataset`: A copy of the dataset with casted features. Example: ```py >>> from datasets import load_dataset, ClassLabel, Value >>> ds = load_dataset("cornell-movie-review-data/rotten_tomatoes", split="train", streaming=True) >>> ds.features {'label': ClassLabel(names=['neg', 'pos']), 'text': Value('string')} >>> new_features = ds.features.copy() >>> new_features["label"] = ClassLabel(names=["bad", "good"]) >>> new_features["text"] = Value("large_string") >>> ds = ds.cast(new_features) >>> ds.features {'label': ClassLabel(names=['bad', 'good']), 'text': Value('large_string')} ``` rrrrrs&& rPcastIterableDataset.cast|sdD9Bzz  ))++'' !2!23"55   rRc*<V^8dQhRS[RS[RR/#)rTenable num_threadsryr)rrs)rZrs"rPr[r s([[T[s[CT[rRc VP'g \R4hTpRRlpV'EdV^8dVPP4pVPP4p\V\ VR44\V\ VR44VP V4p\ PPV4p\ \WvP4pVPWR7p\VP\4'gQh^V,VPnV#VPP4p \V \ WA44VP V 4pV#)a Enable or disable the dataset features decoding for audio, image, video. When enabled (default), media types are decoded: * audio -> dict of "array" and "sampling_rate" and "path" * image -> PIL.Image * video -> torchcodec.decoders.VideoDecoder You can enable multithreading using `num_threads`. This is especially useful to speed up remote data streaming. However it can be slower than `num_threads=0` for local data on fast disks. Disabling decoding is useful if you want to iterate on the paths or bytes of the media files without actually decoding their content. To disable decoding you can use `.decode(False)`, which is equivalent to calling `.cast()` or `.cast_column()` with all the Audio, Image and Video types set to `decode=False`. Args: enable (`bool`, defaults to `True`): Enable or disable features decoding. num_threads (`int`, defaults to `0`): Enable multithreading for features decoding. Returns: `IterableDataset`: A copy of the dataset with casted features. Examples: Disable decoding: ```py >>> from datasets import load_dataset >>> ds = load_dataset("sshh12/planet-textures", split="train", streaming=True) >>> next(iter(ds)) {'image': , 'text': 'A distant celestial object with an icy crust, displaying a light blue shade, covered with round pits and rugged terrains.'} >>> ds = ds.decode(False) >>> ds.features {'image': Image(mode=None, decode=False, id=None), 'text': Value('string')} >>> next(iter(ds)) { 'image': { 'path': 'hf://datasets/sshh12/planet-textures@69dc4cef7a5c4b2cfe387727ec8ea73d4bff7302/train/textures/0000.png', 'bytes': None }, 'text': 'A distant celestial object with an icy crust, displaying a light blue shade, covered with round pits and rugged terrains.' } ``` Speed up streaming with multithreading: ```py >>> import os >>> from datasets import load_dataset >>> from tqdm import tqdm >>> ds = load_dataset("sshh12/planet-textures", split="train", streaming=True) >>> num_threads = min(32, (os.cpu_count() or 1) + 4) >>> ds = ds.decode(num_threads=num_threads) >>> for _ in tqdm(ds): # 20 times faster ! ... ... ``` zFeatures decoding is only available for datasets with known features, but features are Unknown. Please set the datasets features with `ds = ds.cast(features)`.c$V^8dQhR\/#)rTdecoder)rZs"rPr[,IterableDataset.decode..__annotate__s ( ( (rRc:\VR4'd WnR#R#)r"N)rr")r"rs&&rP set_decoding,IterableDataset.decode..set_decodingsw))!'*rRFTrR)rrgrr(r rmultiprocessingpool ThreadPool _apply_asyncrrrrr:rD) rrrdsr%disabled_decoding_featuresenabled_decoding_featuresr(funcrs &&& rPr"IterableDataset.decodes1@}}}R  ( 6kAo)-););)= &(, (:(:(< % -w|U/K L ,glD.I J34B"''22;?D</W/WXDABboo/EFF FFNOR]oBOO K  {{'')H 8W\: ;"B rRc*<V^8dQhRS[RS[RR/#)rTrrryrr)rZrs"rPr[r s#   #  s  /@  rRc \VPWR7p\VVPP 4VP VP \VP4VPR7#)rr) rrrr'rrrrrr)rrrrs&&& rP_stepIterableDataset._stepsY*4+<+<4W #"++'' !2!23"55   rRc VPeV#VPP'dVPPpM(\VP R4P 44pVP P4pWn\VPVVPVP\VP4VPR7#)Nr)rrrrrr3rrrrrrrrrs& rP_resolve_features!IterableDataset._resolve_featuress == $K    ' ' '((11H1$2B2B42H2N2N2PQHyy~~ ))++'' !2!23"55   rRcv<V^8dQhRS[S[,RS[S[S[S[S[,3,,RS[RR/#)rTr by_columnrryr)rrsrrYrhr)rZrs"rPr[r $sOA A SMA E#tCy.12A  A  A rRc  .VfVf \R4hVP'dC\VPP4UUu/uFwrEV\ V4bK upp4pMRpVe\ V\ 4'dV.MTpVPR4P\\VR7RRVVVRR7PVP'dVPP4pV#R4pV#VP'diVPP'dMVPR4P\RVVVR7PVPP4#VP\ RWVR7#uuppi)a/ Group samples from the dataset into batches. Args: batch_size (`int`, optional): The number of samples in each batch. by_column (`Union[str, list[str]`, optional): The column used to batch examples together. Successive examples with the same value for that column are in grouped the same batch. This can also be a list of columns if you want to batch by multiple columns. If batching by column, the batch_size is only used to control the size of the batches to group together or slice during acculumation. drop_last_batch (`bool`, defaults to `False`): Whether to drop the last incomplete batch. Example: ```py >>> ds = load_dataset("some_dataset", streaming=True) >>> batched_ds = ds.batch(batch_size=32) ``` NzDIterableDataset.batch() misses `batch_size` or `by_column` argument.arrow)rT)r=r?rrrrE)r?rrr)rgrr!rkr#rrYrrr r6rrKrJrr7 _batch_fn) rrr8rr`rrrr+s &&&& rPrwIterableDataset.batch$s:  )"3cd d === H[H[H] ^H] d7m!3H] ^_HH  %/ 3%?%?ykYG  )DgV!% )$3%=AT=M=M=MT--99X ITXX I     0 0 9 9 9  )& )$3% T--99: xx t fn  ?!_sF c b<V^8dQhRS[S[,RS[RS[S[S[S[,3,/#rTrr?ry)rrsrrrXr)rZrs"rPr[r gs>AA(3-AARWX\^fgk^lXlRmArRc# Z"V'dHVPR4PVR7F p\VRR7P4xK" R#\P !\ VPR4PRR744p\VRR7P4#5i)aOReturns the dataset as a Python dict. Can also return a generator for large datasets. Args: batch_size (`int`, *optional*): The size (number of rows) of the batches if `batched` is `True`. Defaults to `datasets.config.DEFAULT_MAX_BATCH_SIZE`. Returns: `dict` or `Iterator[dict]` Example: ```py >>> ds.to_dict() ``` r:runset fingerprintrN)rrrto_dictr{rrh)rrr?rs&&& rPrCIterableDataset.to_dictgs ))'277:7Ne9AACCO$$T$*:*:7*C*H*HTX*H*Y%Z[E5g6>>@ @sB)B+c <V^8dQhRS[/#r)rh)rZrs"rPr[r ~s = = =rRc \P!\VPR4P RR744p\ VRR7P 4#)zaReturns the dataset as a Python list. Returns: `list` Example: ```py >>> ds.to_list() ``` r:rrr@rA)r{rrhrrrto_list)rrs& rPrGIterableDataset.to_list~sI  d&6&6w&?&D&DPT&D&U!VWu'2::<)rrsrrr.r/r)rZrs"rPr[r sE&N&N"3-&N9=&N r||Xbll33 4&NrRc aaaSPe%\SPP4R7MRoRV3RlloV'd-VV3RlSPR4P VR74#\ P !SPR4P RR7Uu.uF pS!V4NK up4p\VSR R 7P4#uupi) a2Returns the dataset as a `pandas.DataFrame`. Can also return a generator for large datasets. Args: batch_size (`int`, *optional*): The size (number of rows) of the batches if `batched` is `True`. Defaults to `datasets.config.DEFAULT_MAX_BATCH_SIZE`. batched (`bool`): Set to `True` to return a generator that yields the dataset as batches of `batch_size` rows. Defaults to `False` (returns the whole datasets once). Returns: `pandas.DataFrame` or `Iterator[pandas.DataFrame]` Example: ```py >>> ds.to_pandas() ``` NrRc8V^8dQhR\P/#)rTrr+)rZs"rPr[/IterableDataset.to_pandas..__annotate__s  288 rRc<SPe<VPSPP8wd\VSP4#V#rM)rr~rr8)rrs&rPmaybe_cast_to_declared_featuresBIterableDataset.to_pandas..maybe_cast_to_declared_featuress8}}(U\\T]]=W=W-W-eT]]CCLrRc3h<"TF'p\S!V4SRR7P4xK) R#5i)r@rrBN)r to_pandas)r_rrrNs& rPra,IterableDataset.to_pandas..s4RE7>TW^_iikkRs/2r:rrr@rQ) rr1rrrr{rrrR)rrr?rrrNsf&& @@rPrRIterableDataset.to_pandass,>B]]=V{DMM$6$6$89\`   !--g6;;z;R  $$EIEUEUV]E^EcEcosEcEtuEtE07EtuE5tAKKM Mvs(Cc z<V^8dQhRS[S[,RS[RS[S[,RS[RS[RS[R,3,/#)rTrr?schema_overridesrechunkryr)rrsrrXrr)rZrs"rPr[r s[#u#uSM#u#u#4. #u  #u ~x77 8 #urRc# b"V'dJVPR4PVR7F"p\VRR7PW4R7xK$ R#\P !\ VPR4PRR744p\VRR7PW4R7#5i)aVReturns the dataset as a `polars.DataFrame`. Can also return a generator for large datasets. Args: batch_size (`int`, *optional*): The size (number of rows) of the batches if `batched` is `True`. Defaults to `datasets.config.DEFAULT_MAX_BATCH_SIZE`. batched (`bool`): Set to `True` to return a generator that yields the dataset as batches of `batch_size` rows. Defaults to `False` (returns the whole datasets once). schema_overrides (`dict`, *optional*): Support type specification or override of one or more columns; note that any dtypes inferred from the schema param will be overridden. rechunk (`bool`): Make sure that all data is in contiguous memory. Defaults to `True`. Returns: `polars.DataFrame` or `Iterator[polars.DataFrame]` Example: ```py >>> ds.to_polars() ``` r:rr@rA)rVrWrN)rrr to_polarsr{rrh)rrr?rVrWrs&&&&& rPrYIterableDataset.to_polarss< ))'277:7Ne9CCUeCwwO$$T$*:*:7*C*H*HTX*H*Y%Z[E5g6@@Rb@t tsB-B/ch<V^8dQhRS[S[S[3,RS[S[,RS[S[,RS[/#rT path_or_bufrstorage_optionsryrrErrrsrX)rZrs"rPr[r sE) ) 8X-.) SM) "$ ) ) rRc \P!\VPR4P RR744p\ VRR7P !V3RVRV/VB#)aExports the dataset to csv. This iterates on the dataset and loads it completely in memory before writing it. Args: path_or_buf (`PathLike` or `FileOrBuffer`): Either a path to a file (e.g. `file.csv`), a remote URI (e.g. `hf://datasets/username/my_dataset_name/data.csv`), or a BinaryIO, where the dataset will be saved to in the specified format. batch_size (`int`, *optional*): Size of the batch to load in memory and write at once. Defaults to `datasets.config.DEFAULT_MAX_BATCH_SIZE`. storage_options (`dict`, *optional*): Key/value pairs to be passed on to the file-system backend, if any. **to_csv_kwargs (additional keyword arguments): Parameters to pass to pandas's [`pandas.DataFrame.to_csv`](https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.to_csv.html). The parameter `index` defaults to `False` if not specified. If you would like to write the index, pass `index=True` and also set a name for the index column by passing `index_label`. Returns: `int`: The number of characters or bytes written. Example: ```py >>> ds.to_csv("path/to/dataset/directory") ``` r:rrr@rArr^)r{rrhrrrto_csv)rr]rr^ to_csv_kwargsrs&&&&, rPraIterableDataset.to_csvsmF  d&6&6w&?&D&DPT&D&U!VWu'299  ! ,    rRch<V^8dQhRS[S[S[3,RS[S[,RS[S[,RS[/#r\r_)rZrs"rPr[r sE4 4 8X-.4 SM4 "$ 4 4 rRc \P!\VPR4P RR744p\ VRR7P !V3RVRV/VB#)a:Export the dataset to JSON Lines or JSON. This iterates on the dataset and loads it completely in memory before writing it. The default output format is [JSON Lines](https://jsonlines.org/). To export to [JSON](https://www.json.org), pass `lines=False` argument and the desired `orient`. Args: path_or_buf (`PathLike` or `FileOrBuffer`): Either a path to a file (e.g. `file.json`), a remote URI (e.g. `hf://datasets/username/my_dataset_name/data.json`), or a BinaryIO, where the dataset will be saved to in the specified format. batch_size (`int`, *optional*): Size of the batch to load in memory and write at once. Defaults to `datasets.config.DEFAULT_MAX_BATCH_SIZE`. storage_options (`dict`, *optional*): Key/value pairs to be passed on to the file-system backend, if any. **to_json_kwargs (additional keyword arguments): Parameters to pass to pandas's [`pandas.DataFrame.to_json`](https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.to_json.html). Default arguments are `lines=True` and `orient="records". The parameter `index` defaults to `False` if `orient` is `"split"` or `"table"`. If you would like to write the index, pass `index=True`. Returns: `int`: The number of characters or bytes written. Example: ```py >>> ds.to_json("path/to/dataset/directory/filename.jsonl") ``` ```py >>> num_shards = dataset.num_shards >>> for index in range(num_shards): ... shard = dataset.shard(index, num_shards) ... shard.to_json(f"path/of/my/dataset/data-{index:05d}.jsonl") ``` r:rrr@rArr^)r{rrhrrrto_json)rr]rr^to_json_kwargsrs&&&&, rPrfIterableDataset.to_jsonsm\  d&6&6w&?&D&DPT&D&U!VWu'2::  ! ,    rRcZ<V^8dQhRS[RS[S[RRR3,RS[S[,RS[/#)rTrqconzsqlalchemy.engine.Connectionzsqlalchemy.engine.Enginezsqlite3.Connectionrry)rYrrrs)rZrs"rPr[r ;sN(q(q(q368RThh i(qSM (q (qrRc \P!\VPR4P RR744p\ VRR7P !W3RV/VB#)aExports the dataset to a SQL database. Args: name (`str`): Name of SQL table. con (`str` or `sqlite3.Connection` or `sqlalchemy.engine.Connection` or `sqlalchemy.engine.Connection`): A [URI string](https://docs.sqlalchemy.org/en/13/core/engines.html#database-urls) or a SQLite3/SQLAlchemy connection object used to write to a database. batch_size (`int`, *optional*): Size of the batch to load in memory and write at once. Defaults to `datasets.config.DEFAULT_MAX_BATCH_SIZE`. **sql_writer_kwargs (additional keyword arguments): Parameters to pass to pandas's [`pandas.DataFrame.to_sql`](https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.to_sql.html). The parameter `index` defaults to `False` if not specified. If you would like to write the index, pass `index=True` and also set a name for the index column by passing `index_label`. Returns: `int`: The number of records written. Example: ```py >>> # con provided as a connection URI string >>> ds.to_sql("data", "sqlite:///my_own_db.sql") >>> # con provided as a sqlite3 connection object >>> import sqlite3 >>> con = sqlite3.connect("my_own_db.sql") >>> with con: ... ds.to_sql("data", con) ``` r:rrr@rAr)r{rrhrrrto_sql)rrqrjrsql_writer_kwargsrs&&&&, rPrlIterableDataset.to_sql;sXN  d&6&6w&?&D&DPT&D&U!VWu'299$pPZp^opprRch<V^8dQhRS[S[S[3,RS[S[,RS[S[,RS[/#r\r_)rZrs"rPr[r esE. . 8X-.. SM. "$ . . rRc ^RIHpV!VP4;'g\Pp\ P !\VPR4PVR744p\VRR7P!V3RV/VB#)a<Exports the dataset to parquet Args: path_or_buf (`PathLike` or `FileOrBuffer`): Either a path to a file (e.g. `file.parquet`), a remote URI (e.g. `hf://datasets/username/my_dataset_name/data.parquet`), or a BinaryIO, where the dataset will be saved to in the specified format. batch_size (`int`, *optional*): Size of the batch to load in memory and write at once. Defaults to `datasets.config.DEFAULT_MAX_BATCH_SIZE`. storage_options (`dict`, *optional*): Key/value pairs to be passed on to the file-system backend, if any. **parquet_writer_kwargs (additional keyword arguments): Parameters to pass to PyArrow's `pyarrow.parquet.ParquetWriter`. Returns: `int`: The number of characters or bytes written. Example: ```py >>> ds.to_parquet("path/to/dataset/directory") ``` ```py >>> num_shards = dataset.num_shards >>> for index in range(num_shards): ... shard = dataset.shard(index, num_shards) ... shard.to_parquet(f"path/of/my/dataset/data-{index:05d}.parquet") ``` )get_arrow_writer_batch_size_from_featuresr:rr@rAr^) arrow_writerrrrrDEFAULT_MAX_BATCH_SIZEr{rrhrrr to_parquet)rr]rr^parquet_writer_kwargsrrrs&&&&, rPruIterableDataset.to_parquetesP L>t}}MnnQWQnQn   d&6&6w&?&D&DPZ&D&[!\]u'2==  )8 [c[c#[c %/0$s)S#EF G[crRc # aaaa"W,p W,p W,\W4,oSV ,W8d^M^,oVVV3Rl\SS, 44p \\PVR\ R7p ^p^p.p.pSP pV EFwppV 'dU^RIHpVPR4pVP\\SPR7RV!VP 4R7pV R V R VR R VR R 2p\P!R RR7pVP!V4VP#4\$P&!VP(4oVf/\*P,!SP.P144pVSP2, pV\5V3Rl\SP6444, pVP9V4\;V\<4'dqVP>'g_\AVVP(R7pV PCVPDV.VPFVPHVR7VP9V4M{\;V\J4'dXVPL'dVPLR ,V,pV POVPPVP(V3.R7M\SRV 24hVP#4\YVP(4P[4VR^3xEK VRVVVW33xR# \T\V3d6TP#4\YTP(4P[4hi;i5i)aPushes the dataset shards as Parquet files to the hub. Returns: additions (`List[CommitOperation]`): list of the `CommitOperationAdd` of the uploaded shards new_parquet_paths (`List[str]`): list of the paths of the uploaded parquet files features (`Features`): features of the uploaded dataset dataset_nbytes (`int`): approximate size in bytes of the uploaded dataset after uncompression num_examples (`int`): number of examples of th euploaded shards c3l<"TF)pSV,SPSS, VRR73xK+ R#5i)TrFN)r)r_rrrrs& rPraEIterableDataset._push_parquet_shards_to_hub_single..s2 _qZ[UQY cEkt T U_qs14datasetsendpointr} library_namelibrary_versionrqr:rT)r?rrG-05dz-of-z.parquetF)suffixdeleteNc3Z<"TF pSPV4PxK" R#5irM) row_grouptotal_byte_size)r_rparquet_metadatas& rPrars'&Kqa$..q1AAKqs(+) path_in_repopath_or_fileobj)repo_id additions repo_typerevisionr~) bucket_idaddz Bad HF path: ).rrrr HF_ENDPOINTrrrsrrrrr r9rtempfileNamedTemporaryFileruclosepq read_metadatarqr!rr~to_arrow_schemanum_rowsrnum_row_groupsrrrHrrpreupload_lfs_filesrrrrGpathbatch_bucket_filesrrrrr unlink)rryrzr{r|r r}r~rrrr index_shardsapidataset_nbytes num_examplesrnew_parquet_pathsrrrrrshard_path_in_repotmp_fileshard_additionrrrsf&&&&&&&&& @@@rP"_push_parquet_shards_to_hub_single2IterableDataset._push_parquet_shards_to_hub_singles..$# s6//ck&,QA6 _dehkpep_q V//u:grs .0 ')==(LE5#S))'2 /4CZCZ[ HX" %-:QugQuSkjQTEUU]!^ 22*USH#   * #%#3#3HMM#B #'99:J:Q:Q:a:a:cdH 0 9 99 #&KPQaQpQpKq&#"(();<35WXX0===%7EWiqiviv%wN++ 4 < <#1"2"6"@"@!5!>!>"+ ,$$^4 46TUU+000-A-F-F-LOa-a***"6"@"@ WiGjFk+. >R=S.TUU NN    & & (%" "e)hdY(98^bbb01  X]]#**, sGD!N(C-L;AL;3N4'L;AL; N!L;/A N;ANNc<V^8dQhRS[RS[RS[RS[S[,RS[S[,RS[S[S[S[3,,RS[S[,RS[R S[S[,R S[S[S[,S[S[,S[ S[ S[3,/ #) rTr{r|r r}r~max_shard_sizerrnum_procry) rrYrrrrsrrhrr!r5)rZrs"rPr[r stQtQ6tQtQ tQ } tQ D> tQ!sCx1tQSMtQ#tQ3-tQ t&'cHiL MtQrRc  V'dNVPPP4U U u.uFwr\V RR7'gKV NK up p M.p T;'d \ V 4pVfVfVP pM\ T;'g\P4p^p VPR4P\PR7FpWP, p K \W, 4^,p\Yy;'g^4p.p.p^p^p^pVPpT ;'g^pV^8:d"\P!RV RV R24Rp ^pM-VV8d'\P!RV R V R V R V R 2 4Tp Tp\#V4Uu.uF+pR VP%VVRR7RVRVRVRVRVRVRVRVRV/ NK- ppRpTV eV ^8dRV R2MR, p\'RVVR7pV eV ^8d\(P*!4M$\,P.!R4P1V 4;_uu_4pVf\2P4!R"/V^,BM\7V\2P4VR7pVFPwpppV'gVP9V4K"Vwppp p!p"VV, pVV, pVV!, pVV", pT pKR Ve!VP;4VP=4RRR4\?R V44p\AVVVR!7p#VVVV#V3#uup p iuupi +'giLB;i)#a?Pushes the dataset shards as Parquet files to the hub. Returns: additions (`List[CommitOperation]`): list of the `CommitOperationAdd` of the uploaded shards new_parquet_paths (`List[str]`): list of paths of the new files uploaded to the output path, relative to output path features (`features`): features of the uploaded dataset split_info (`int`): info of the uploaded split, including the approximate size in bytes of the uploaded dataset after uncompression uploaded_size (`int`): number of uploaded bytes to the repository or bucket T)ignore_decode_attributeNr:rzSetting num_proc from z back to 1 for the z@ split to disable multiprocessing as it only contains one shard.rdz for the z split as it only contains rHrrFryrzr{r|r r}r~rrzUploading the dataset shardsz (num_proc=)rEz shards)unittotaldescspawn)kwargs_iterablec3L"TFqPPxK R#5irM) upload_infor)r_additions& rPra>IterableDataset._push_parquet_shards_to_hub..dsPi(0055is"$) num_bytesrrN)!r'rrkr*rrr?rMAX_SHARD_SIZErrrtnbytesrsrqrr rrhf_tqdm contextlib nullcontextmp get_contextPoolrrr@rrrrr5)$rr{r|r r}r~rrrrkvdecodable_columnsestimated_nbytesrrr uploaded_sizerrrrzryrrpbarr( update_streamrcontent job_additionsjob_new_parquet_paths job_featuresjob_uploaded_sizejob_num_examples split_infos$&&&&&&&&&& rP_push_parquet_shards_to_hub+IterableDataset._push_parquet_shards_to_hubs8$ ::..446 l641:J1fj:k:kQQ6 l  4OO=N8O  %!__ !9.:a:aFLaLa!b#$ $ 0 0 9 > >&JgJg > hH$7$!i !1!BCaG  ]]; .0 ')  ====q ? NN( 2EeWMMN HH ( " NN( $zl)E7RmnxmyzBC "H!H / * hfQU V&H&(<HYj&(< *  . X-AhRSm+hZq)Y[[ 8a<  " " $(--h7 8 8;?< BBX_UVEWX'#FF$3 *7%gKK(nukM#8,HY[k.I%)>>%!%66M $44L+H*7  3 86PiPP u\Z +Xz=PPs mJ . 8 8sL1 L1 1L7 B8L<< M c P<V^8dQhRS[RS[RS[S[,RS[S[,RS[S[,RS[S[,RS[S[,RS[S[,R S[S[,R S[S[,R S[S[,R S[S[S[S[3,,R S[S[,RS[RS[S[,RS[/#)rTr config_name set_defaultr r|commit_messagecommit_descriptionprivater}rr~rrrrry)rYrrrrsr)rZrs"rPr[r hsAAAAd^ A } A 3- A! A%SMA$A}A3-AD>A!sCx1ASMA#A 3-!A" #ArRc fVeWP8dv\PRV RVP RWP, R24\PRVP RVP R24VPpVR8Xd \ R 4hV eV e \ R 4hVf&VP e\ VP 4MR p\P!\V4'g\ R \ R V R24hV'g VR8wdTMRp\\PV R\R7pVPR4'd\f \!R4hVP R4vppppVP#VR,V,4P$pRP+RV44p\-VVVVVVVV V V VVR7 #VP/VRR7P$pV e,V PR4'gVP7WRRR7\9VVVVVVVVV V V V V VVR7# \d2TP'TR,T,TRR7pTP(pLi;i \0d%TP3TRTRR7pTP4pLi;i)aPushes the dataset to the hub as a Parquet dataset. The dataset is pushed using HTTP requests and does not need to have neither git or git-lfs installed. The resulting Parquet files are self-contained by default. If your dataset contains [`Image`], [`Audio`] or [`Video`] data, the Parquet files will store the bytes of your images or audio files. You can disable this by setting `embed_external_files` to `False`. Args: repo_id (`str`): The ID of the repository to push to in the following format: `/` or `/`. Also accepts ``, which will default to the namespace of the logged-in user. It could also be a location inside a bucket, e.g. `buckets///...` config_name (`str`, defaults to "default"): The configuration name (or subset) of a dataset. Defaults to "default". set_default (`bool`, *optional*): Whether to set this configuration as the default one. Otherwise, the default configuration is the one named "default". split (`str`, *optional*): The name of the split that will be given to that dataset. Defaults to `self.split`. data_dir (`str`, *optional*): Directory name that will contain the uploaded data files. Defaults to the `config_name` if different from "default", else "data". commit_message (`str`, *optional*): Message to commit while pushing. Will default to `"Upload dataset"`. commit_description (`str`, *optional*): Description of the commit that will be created. Additionally, description of the PR if a PR is created (`create_pr` is True). private (`bool`, *optional*): Whether to make the repo private. If `None` (default), the repo will be public unless the organization's default is private. This value is ignored if the repo already exists. token (`str`, *optional*): An optional authentication token for the Hugging Face Hub. If no token is passed, will default to the token saved locally when logging in with `huggingface-cli login`. Will raise an error if no token is passed and the user is not logged-in. revision (`str`, *optional*): Branch to push the uploaded files to. Defaults to the `"main"` branch. create_pr (`bool`, *optional*, defaults to `False`): Whether to create a PR with the uploaded files or directly commit. max_shard_size (`int` or `str`, *optional*): Optional maximum size of the dataset shards to be uploaded to the hub. If expressed as a string, needs to be digits followed by a unit (like `"5MB"`). If not provided, shard count defaults to this dataset's `.num_shards`. num_shards (`int`, *optional*): Number of shards to write. If `max_shard_size` is provided and `num_shards` is not, then the number of shards is estimated from `max_shard_size`. embed_external_files (`bool`, defaults to `True`): Whether to embed file bytes in the shards. In particular, this will do the following before the push for the fields of type: - [`Audio`] and [`Image`]: remove local path information and embed file content in the Parquet files. num_proc (`int`, *optional*, defaults to `None`): Number of processes when preparing and uploading the dataset. This is helpful if the dataset is made of many samples and transformations. I uses "spawn" context to work with hf_xet, the rust client for fast uploads to HF. Multiprocessing is disabled by default. Return: huggingface_hub.CommitInfo Example: ```python >>> dataset.push_to_hub("/") >>> dataset_dict.push_to_hub("/", private=True) >>> dataset.push_to_hub("/", max_shard_size="1GB") >>> dataset.push_to_hub("/", num_shards=1024) ``` If your dataset has multiple splits (e.g. train/validation/test): ```python >>> train_dataset.push_to_hub("/", split="train") >>> val_dataset.push_to_hub("/", split="validation") >>> # later >>> dataset = load_dataset("/") >>> train_dataset = dataset["train"] >>> val_dataset = dataset["validation"] ``` If you want to add a new configuration (or subset) to a dataset (e.g. if the dataset has multiple tasks/versions/languages): ```python >>> english_dataset.push_to_hub("/", "en") >>> french_dataset.push_to_hub("/", "fr") >>> # later >>> english_dataset = load_dataset("/", "en") >>> french_dataset = load_dataset("/", "fr") ``` zToo many num_proc: rArBz processes.zTo parallelize data loading, we give each process some shards (or data sources) to process. Therefore it's unnecessary to have a number of processes greater than dataset.num_shards=rCrnrzN`config_name` cannot be 'data'. Please, choose another name for configuration.zXFailed to push_to_hub: please specify either max_shard_size or num_shards, but not both.trainzSplit name should match 'z ' but got 'z'.defaultrrzbuckets/z;Pushing datasets to buckets requires huggingface_hub>=1.6.0rGT)rexist_okc38"TFq'gK VxK R#5irMrN)r_rZs& rPra.IterableDataset.push_to_hub..s;~!AA~s  ) rrrrr r|r}rrrrdataset)r)rrrzrefs/pr/)branchrr)rrrr r|rrr}rr~rrrr)rrr rrgr rYrematchr2rrrr startswithrF ImportError bucket_inforP create_bucketrrr repo_infor create_repor create_branchr)rrrrr r|rrrr}rr~rrrrrr _namespace _bucket_name_path_segmentsr bucket_urlrrepo_urls&&&&&&&&&&&&&&&& rP push_to_hubIterableDataset.push_to_hubhsX  H$> NN%hZ/KDOOK\]$67{D  KKllpl{l{k|}[[_[j[jZkkln  H & mn n  %**@j  ='+zz'=C O7Exx 5))8 ;ugUWXY Y&1Y&>{FHV//u:grs   j ) )"*!"_``;B==;M 8Az<. 1OOJ,<|,KLOO 88;~;;D"#''!-%%9!   +--9-EHH#H,?,? ,K,K!!'iZ^!_ ''!-#5!#-%%9! C' 1 ..zC/?,/NX_jn.o &00  1*+ +??'#! + #** +s$)IJ9I>=I>,J0/J0) r+ __hffs_cacherrrrrrr)NNNNN))rFr)NN)NNNNr)NNN)NNNr)NNF)NNFN)NNFNNNr)NNFFrrM) NFNFrFNNN) NFNFrFNNNF)NFNFrN)NNr r!)Trra)NFNT)rNNNNNNNNFNNTN)Jr"r#r$r%r&rr'rrr rr(r,r0r3r7rr>rTrNrrrrr%r4TRAINroryr classmethodr1rrrrrrrrrrrrrrrrrrrrrr@rrrr"r2r5rwrCrGrRrYrarfrlrurrrr(r)r*s@rPrr s*GG, E E F F3*3*j0/0/drG3  ,, 6p 77r*??>11(,%)!KK 444l(((T f f:5:5:5x0 0 0 d" " " H$5$5$5L(((T,,,\RRRh111f7 7 r] ] ~U U nS S j` ` D++( ( T( ( T! ! F6 6 p& & P ] ]LL<2!!F0 0 d, , \, , \[[z    $A A FAA. = =&N&NP#u#uJ) ) V4 4 l(q(qT. . `[c[cztQtQlAAArRrc V^8dQhR\\,R\\,R\\,R\ R\/#)rTdsetsrr axisry)rhrrr1r3rs)rZs"rPr[r[,sPYY  Y ; Y J Y  Y  YrRc VUu.uFqDP4NK ppV^8Xd&\VUu.uFqUPNK up4M.\VUUu.uFqUPFqfNK K upp4\;QJdRV4F 'dK RM RM !RV44'dRpM\ ;QJdRV4F 'gK RM RM !RV44'dRp\ PR4MhVUu0uFqUPPkK pp\V4^8XdVP4p \V R7pMRp\ PR4\\VUu.uFqUPNK up4U U U u/uFqP4FwrWbK K up p p 4p VUu.uFp\!VP"4NK p pV^8Xd \%V 4pM\;QJdRV 4F 'dK RM RM !RV 44'dC^R IHpV!V 4;'g\*P,pV Uu.uFp\/VVR 7NK p p\1V 4pVf1\2P4!VUu.uFqDPNK up4pMVP74pWnVUUUu/uF*pVP8P4F wppVVbK K, pppp\;VVVVVR 7#uupiuupiuuppiuupiuupiuup p p iuupiuupiuupiuupppi) a Converts a list of `IterableDataset` with the same schema into a single `IterableDataset`. Missing data are filled with None values. Args: dsets (`List[datasets.IterableDataset]`): List of Datasets to concatenate. info (`DatasetInfo`, optional): Dataset information, like description, citation, etc. split (`NamedSplit`, optional): Name of the dataset split. axis (``{0, 1}``, default ``0``, meaning over rows): Axis to concatenate over, where ``0`` means over rows (vertically) and ``1`` means over columns (horizontally). *New in version 1.6.0* Example: ```py >>> ds3 = _concatenate_iterable_datasets([ds1, ds2]) ``` c3<"TFqPRJxK R#5irMrr_dsets& rPra1_concatenate_iterable_datasets..Qs 6  t #rFTNc3<"TFqPRJxK R#5irMrrs& rPrarSs 8%$   %%rzoSome of the datasets have disparate format or format not set. Resetting the format of the concatenated dataset.rc38"TFqPxK R#5irMr<rs& rPrarmsF+%%rrqr)rrr rrB)r5r&rrrrfrrrrKrr_rCr!r%rkrrrrsrrrrtrrr1 from_mergerrr)rrr rr/rcol_namerBformat_type_setrKrrrrrrrrrrr}rs&&&& rP_concatenate_iterable_datasetsr,s8-2 2Eq "EE 2 qy)U*KUT==U*KL%V%$ HX X%VW s 6 6sss 6 666  8% 8 8% 8 8 8  } EJJED++77EJ  1 $)--/K)kBJJ KKB  -.O}}.OPnP(]k]k]mUYUV]mPnH7<yy{M7<vuGSZSmSmSsSsSu%%Suuv  +  o 3+LVK/Pn='> ws:LL L% L+>L0 #L5 L<$MM0M cV^8dQhR\\,R\\\,,R\\,R\\ ,R\\ ,R\R,R\/#) rTrrrrr r)ryr*)rhrrrrsr1r3r>)rZs"rPr[r[s{RR?#RDK(R 3-R ;  R J  R O RRrRc VUu.uFqfP4NK pp\VUu.uFqwPNK up4\V4FVwrV^,PVP8wgK)\ RV^,P RVP RV 24h \ \VUu.uFqwPNK up4U U U u/uFqP4FwrWbK K up p p 4p VUu.uFp\VP4NK p p\;QJdRV 4F 'dK RM RM !RV 44'dV U u.uFp \V ^R7NK p p Vf\WR7p M.\PP!V4p\#V VVVR 7p Vf1\$P&!VUu.uFqfP(NK up4pMVP+4pWnVUUUu/uF)qP,P4F wppVVbK K+ pppp\/V VVVV^,PR 7#uupiuupiuupiuup p p iuupiuup iuupiuupppi) a Interleave several iterable datasets (sources) into a single iterable dataset. The new iterable dataset alternates between the sources to yield examples. If `probabilities = None` (default) the iterable dataset will cycles through the sources in order for each next example in the iteration. If `probabilities` is not `None, the iterable dataset will sample a random source according to the provided probabilities for each next examples in the iteration. Args: datasets (`List[IterableDataset]`): list of datasets to interleave probabilities (`List[float]`, optional, default None): If specified, the new iterable dataset samples examples from one source at a time according to these probabilities. seed (`int`, optional, default None): The random seed used to choose a source for each example. stopping_strategy (`str`, defaults to `first_exhausted`): Two strategies are proposed right now. By default, `first_exhausted` is an undersampling strategy, i.e the dataset construction is stopped as soon as one dataset has ran out of samples. If the strategy is `all_exhausted`, we use an oversampling strategy, i.e the dataset construction is stopped as soon as every samples of every dataset has been added at least once. Note that if the strategy is `all_exhausted`, the interleaved dataset size can get enormous: - with no probabilities, the resulting dataset will have max_length_datasets*nb_dataset samples. - with given probabilities, the resulting dataset will have more samples if some datasets have really low probability of visiting. Output: `datasets.IterableDataset` zJDatasets should be identically split_by_node before interleaving, but got z!=z at index 0 and c38"TFqPxK R#5irMr<rs& rPra0_interleave_iterable_datasets..s B\k ! !\rFTrrr")rrr rr )r5r&rrrrgr!r%rkrrrrr'rrrrr1rrrrr)rrrrr r)r/rrrrrrrrrrr}rs&&&&&& rP_interleave_iterable_datasetsr sfD088x!##%xH8&&J}}&JKX& A; # #t'8'8 8\]efg]h]u]u\vvxy}zKzKyLL\]^\_` '-.R}}.RSqS(`n`n`pX\XY`pSqH7??hHQ^^,hL? s B\ Bsss B\ BBBeqreqVa6{qQeq r9,l II))$/ A '/   |%%x&@x!vvx&@Ayy{M'/&.7E_E_EeEeEg>7EEgh  +QK,,  U9'K/Sq@r'A s.II#6I( #I- I4#I9I> /JcHV^8dQhR\R\R\R\/#)rTrrrry)rrs)rZs"rPr[r[s*_CUX]lrRc ^VP'd@W PP,V,pW PP,p\WR7p\ VP VP P4VPVPVVPR7#)a Split an iterable dataset for the node at rank `rank` in a pool of nodes of size `world_size`. If the dataset has a number of shards that is a factor of `world_size` (i.e. if `dataset.num_shards % world_size == 0`), then the shards are evenly assigned across the nodes, which is the most optimized. Otherwise, each node keeps 1 example out of `world_size`, skipping the other examples. Args: dataset ([`IterableDataset`]): The iterable dataset to split by node. rank (`int`): Rank of the current node. world_size (`int`): Total number of nodes. Returns: [`IterableDataset`]: The iterable dataset to be used on the node at rank `rank`. )rrr) rrrrrrr'rrrr)rrrr s&&& rP_split_by_node_iterable_datasetrs&00555<"6"6"A"AA #EK (( ]]   !nn&&!44  rRc"VPW34pVP4'dVP4#\P!^4GRjxL KGL5ir) apply_asyncreadyrrra)r(r.rOrhs&&& rPr*r*sC   dD )F  <<>>::< --" " "s)A)AAAcXVP4UUu/uFwrW.bK upp#uuppirM)rk) unbatchedrrs& rPr;r;s)(0 10tqAsF0 11 1s&cV^8dQhR\R,R\\R\P3,,/#)rTrrryrKr)rrrr{r|)rZs"rPr[r[ s8<r?r@utils.shardingrArBrCrD utils.typingrEHF_HUB_VERSIONparsehuggingface_hub.errorsrFhuggingface_hub.hf_file_systemrGrHsqlite3r-r7 sqlalchemyrrrJrKr"rrsrYrrQrnrurrrrrrr,rerrrr'rrrrr8r:rrrrrrr'r)rfrrrCrrrrrrrr rr*r;rrNrRrPr>s   .!#JJ:!TT   00#% wv" GMM'22:qq%)")A&* H  CeCHo| 34   7#>f@ -T^T^nJ<,J