+ QV-jWzRt^RIt^RIt^RIt^RIt^RIt^RIt^RIt^RIt^RI t ^RI H t ^RI H t ^RIHt^RIHt^RIHtHtHtHt^RIHt^RIt^RIt^RIHt^R IHt^R I H!t!^R I"H#t#H$t$^R I%H&t&^R I'H(t(H)t)^RI*H+t+H,t,H-t-^RI.H/t/H0t0H1t1^RI2H3t3H4t4^RI5H6t6^RI7H8t8H9t9^RI:H;t;Ht>H?t?H@t@^RIAHBtB^RICHDtDHEtE^RIFHGtG^RIHHItI^RIJHKtKHLtLHMtMHNtN^RIOHPtPHQtQ^RIRHStSHTtTHUtUHVtV^RIWHXtX^RIYHZtZ^RI$H[t[^RI$H\t]^R I^H_t_^R!I`Hata^R"IbHctcHdtdHete^R#IfHgtgHhthHitiHjtjHktkHltlHmtmHntn^R$IoHptpHqtq^R%IrHsts]'d^R&ItHutu][P!]w4tx!R'R(]y4tz] !R)R*44t{!R+R,4t|] !R-R.44t}!R/R0]|4t~!R1R2]|4t!R3R4]|4tR#)5zDatasetBuilder base class.N)Iterator) dataclass)partial)Path) TYPE_CHECKINGAnyOptionalUnion)patch) url_to_fs)Pool) thread_map)configutils)Dataset) ArrowReaderReadInstruction) ArrowWriter ParquetWriterSchemaInferenceError) DataFilesDictDataFilesPatternsDictsanitize_patterns) DatasetDictIterableDatasetDict)DownloadConfig)DownloadManager DownloadMode)StreamingDownloadManagerxjoin)DatasetGenerationCastErrorDatasetGenerationErrorFileFormatError)Features)is_remote_filesystemrename)Hasher DatasetInfo)ArrowExamplesIterableExamplesIterableIterableDataset_concatenate_iterable_datasets)"INVALID_WINDOWS_CHARACTERS_IN_PATHcamelcase_to_snakecase)Split SplitDictSplitGenerator SplitInfo)$extend_dataset_builder_for_streaming) CastError)logging)tqdm)FileLock) is_remote_url)VerificationModeverify_checksums verify_splits) classpropertyconvert_file_size_to_inthas_sufficient_disk_spaceiflatmap_unordered map_nestedmemoizesize_strtemporary_assignment)_number_of_shards_in_gen_kwargs_split_gen_kwargs) tracked_list) DatasetModulec]tRt^`tRtR#)InvalidConfigNameN)__name__ __module__ __qualname____firstlineno____static_attributes__rJa/Volumes/fast/ai/experiments/ui-tars-smoke/.venv/lib/python3.14/site-packages/datasets/builder.pyrIrI`srPrIca]tRt^dtoRtRt]P!R4tRt Rt Rt Rt Rt R V3RlRlltV3R lR ltV3R ltR tVtR#) BuilderConfigaBase class for `DatasetBuilder` data configuration. `DatasetBuilder` subclasses with data configuration options should subclass `BuilderConfig` and add their own properties. Attributes: name (`str`, defaults to `default`): The name of the configuration. version (`Version` or `str`, defaults to `0.0.0`): The version of the configuration. data_dir (`str`, *optional*): Path to the directory containing the source data. data_files (`str` or `Sequence` or `Mapping`, *optional*): Path(s) to source data file(s). description (`str`, *optional*): A human description of the configuration. default0.0.0Nc\F3pWP9gK\R\ RVP R24h VPeA\ VP\ \ 34'g\RVP 24hR#R#)z Bad characters from black list 'z ' found in 'z\'. They could create issues when creating a directory for this config on Windows filesystem.Nz/Expected a DataFilesDict in data_files but got )r-namerI data_files isinstancerr ValueError)self invalid_chars& rQ __post_init__BuilderConfig.__post_init__~s>Lyy('67Y6ZZfgkgpgpfqrpq? ?? &z$//M[pKq/r/rNtN_`a a0s &rPctaa\SPP44\SPP448wdR#\;QJd8VV3RlSPP44F 'dK R# R#!VV3RlSPP444#)Fc3\<"TF!q\SV43V\SV438HxK# R#5iN)getattr).0kor[s& rQ 'BuilderConfig.__eq__..s-]H\1wtQ'(Q1 ,>>H\s),T)set__dict__keysall)r[resffrQ__eq__BuilderConfig.__eq__su t}}!!# $AJJOO,=(> >s] HZHZH\]ss]s]s] HZHZH\]]]rPc<<V^8dQhRS[RS[S[,RS[/#) config_kwargscustom_featuresreturn)dictrr#str)format __classdict__s"rQ __annotate__BuilderConfig.__annotate__s.>>>"(+> >rPc dRpVP4pVPRR4VPRR4RV9dKVR,fVPRR4M,VR,p\PP V4pWTR&V'd\ V4Uu/uF qfWF,bK pp\ ;QJd*RVP44F 'dK RM RM!RVP444'dNRPR VP444p\V4^ 8d\P!V4pM\P!V4pVeE\4pV'dVPV4VPV4VP4pV'diVPR ,V,p\V4\ P"8d/VPR ,\P!V4,pV#VP#uupi) a The config id is used to build the cache directory. By default it is equal to the config name. However the name of a config is not sufficient to have a unique identifier for the dataset being generated since it doesn't take into account: - the config kwargs that can be used to overwrite attributes - the custom features used to write the dataset - the data_files for json/text/csv/pandas datasets Therefore the config id is just the config name with an optional suffix based on these. NrWversiondata_dirc3b"TF%p\V\\\\34xK' R#5ira)rYrtboolintfloat)rcvs& rQrf1BuilderConfig.create_config_id..s%kCja:a#tS%!899Cjs-/FT,c3"TFFwr\V4R,\PP\V44,xKH R#5i)=N)rturllibparse quote_plus)rcrdrs& rQrfrs9"OutqCFSL6<<#:#:3q6#BBBOusAA-)copypopospathnormpathsortedrkvaluesjoinitemslenr&hashupdate hexdigestrWr%MAX_DATASET_CONFIG_ID_READABLE_LENGTH) r[rprqsuffixconfig_kwargs_to_add_to_suffixr{rdm config_ids &&& rQcreate_config_idBuilderConfig.create_config_ids"!%)6););)=&&**648&**9d; 7 7-j9A.22:tD:*E77++H5=Ez: )?EEc>d.>d144>d +.skCaChChCjkssskCaChChCjkkk"OmOsOsOu"v;##[[)GHF%CD  &A HH_ %[[]F  C&0I9~ L LL IIOfkk&.AA  99 3.sH-c*<V^8dQhRS[RS[RR/#)ro base_pathdownload_configrrN)rtr)rurvs"rQrwrxs(RRSR>RVZRrPc\VP\4'dLVP'd\ WP4MTpVPP W4VnR#R#ra)rYrXrr{rresolve)r[rrs&&&rQ_resolve_data_files!BuilderConfig._resolve_data_filessG doo'< = =;?===i7iI"oo55iQDO >rPc<V^8dQh/S[;R&S[S[S[PS[3,,;R&S[S[,;R&S[S[S[S[3,,;R&S[S[,;R&#)rorWrzr{rX description)rtrr rVersionrr)rurvs"rQrwrxdsq( )*eEMM3./ 0I+,sm"-.}.CCDEL/0#%1rP)rXra)rKrLrMrN__doc__rWrrrzr{rXrr]rlrr__annotate_func__rO__classdictcell__rvs@rQrSrSdsY$D38==3IG"HHLJ!%K b^>>@RRYrPrSca]tRt^toRtRt]t.tRt Rt Rt R5V3RlRllt Rt RtV3RlR ltV3R lR ltR6V3R lR llt]]]!4V3RlRl444t]R4tV3RlRltR7V3RlRlltRt]P8V3RlRl4t]R4tV3RlRltR8V3RlRllt Rt!V3RlRlt"R t#R!t$R9V3R"lR#llt%R:V3R$lR%llt&]'PPR3V3R&lR'llt)V3R(lR)lt*R;V3R*lR+llt+V3R,lR-lt,]P8V3R.lR/l4t-]P8R Parameter `name` was renamed to `config_name`. hash (`str`, *optional*): Hash specific to the dataset builder code. Used to update the caching directory when the dataset builder code is updated (to avoid reusing old data). The typical caching directory (defined in `self._relative_data_dir`) is `name/version/hash/`. base_path (`str`, *optional*): Base path for relative paths that are used to download files. This can be a remote URL. features ([`Features`], *optional*): Features types to use with this dataset. It can be used to change the [`Features`] types of a dataset, for example. token (`str` or `bool`, *optional*): String or boolean to use as Bearer token for remote files on the Datasets Hub. If `True`, will get token from `"~/.huggingface"`. repo_id (`str`, *optional*): ID of the dataset repository. Used to distinguish builders with the same name but not coming from the same namespace, for example "rajpurkar/squad" and "lhoestq/squad" repo IDs. In the latter, the builder name would be "lhoestq___squad". data_files (`str` or `Sequence` or `Mapping`, *optional*): Path(s) to source data file(s). For builders like "csv" or "json" that need the user to specify data files. They can be either local or remote files. For convenience, you can use a `DataFilesDict`. data_dir (`str`, *optional*): Path to directory containing source data file(s). Use only if `data_files` is not passed, in which case it is equivalent to passing `os.path.join(data_dir, "**")` as `data_files`. For builders that require manual download, it must be the path to the local directory containing the manually downloaded data. storage_options (`dict`, *optional*): Key/value pairs to be passed on to the dataset file-system backend, if any. writer_batch_size (`int`, *optional*): Batch size used by the ArrowWriter. It defines the number of samples that are kept in memory before writing them and also the length of the arrow chunks. None means that the ArrowWriter will use its default value. **config_kwargs (additional keyword arguments): Keyword arguments to be passed to the corresponding builder configuration class, set on the class attribute [`DatasetBuilder.BUILDER_CONFIG_CLASS`]. The builder configuration class is [`BuilderConfig`] or a subclass of it. NFc<V^8dQhRS[S[,RS[S[,RS[S[,RS[S[,RS[S[,RS[S[,RS[S[,RS[S[S[S[3,,R S[S[,R S[S[S[S[S[S[3,,R S[S[,R S[S[,R S[S[ ,RS[S[,/#)ro cache_dir dataset_name config_namerrinfofeaturestokenrepo_idrXr{storage_optionswriter_batch_sizer) rrtr(r#r r}listrsrr~)rurvs"rQrwDatasetBuilder.__annotate__4ss3s3C=s3sms3c] s3 sm s3 C= s3{#s38$s3dCi()s3#s3U3dM#ABCs33-s3"$s3$C=s3C=s3rPc  \VPPR4R,4VnW@nWPnWnWnT ;'g/VnV'd \V4M VPVn T ;'g VPVn V eM\V \4'g7\P!\V 4V\!WPR7R7p R\"P$!VP&P(4P*9d VeWR&V eWR&V eWR&WnVP.!RRVRVR V/VBwVnVnVfVP54pVPVnVPVn VP0PVnVP0P:VnW`nVeWpP<n\AT;'g\0PB4Vn"\GVPD4'd VPDM(\HPJPMVPD4Vn"V'd0\NPP!VPD\0PR4M\A\0PT4Vn+\GVPV4'd VPVM(\HPJPMVPV4Vn+RVn,VP[4Vn.\GVPD4'Eg\HP^!VPDR R 7\HPJPQVPD\aVP\4Pc4PeR R 4R,4p\gV4;_uu_4\HPJPiVP\4'Ed\k\HPl!VP\44^8d\HPJPi\HPJPQVP\\0Pn44'd;\pPsR4\tPv!VP\4VnMP\pPyRVP\ RVP R24\HPz!VP\4RRR4VP\Vn>\~P!R4VnARVnBRVnCRVnD\V4R# +'giL^;i).NrrrrrrXr{rrqrTexist_ok/_z.lockz$>? "*.&//1T1122 KK,,t < $$d4??&;&D&D&F&N&NsTW&X[b&bI)$$77>>$//222::doo67!;77>>"'',,tHdHd*eff"LL)gh(3(B(B4??(SDI1$//1B-PTPaPaObcPQ1% ??.4.?.?.G"'! -T25%$s!B9U<B U<< V cVP#ra)rir[s&rQ __getstate__DatasetBuilder.__getstate__s }}rPc(Wn\V4R#ra)rir3)r[ds&&rQ __setstate__DatasetBuilder.__setstate__s ,T2rPc0<V^8dQhRS[S[,/#rorrrrt)rurvs"rQrwrs00Xc]0rPc VPPR4'Ed\VP4'EgVPP R8XEdj^RIHpVP'dCVPPR4^8d#VPPR4^,MRpVPeVPPRR4M VPpW0P\VPP 4R,pVPVP R4^,p\ P"!Vf VPMV RVP 2VR V4p\ P"!VPV4p\$P&P)V4'dV#R#R#R#R#) z]Check for the old cache directory template {cache_dir}/{namespace}___{builder_name} from 2.13 datasets.rT)_PACKAGED_DATASETS_MODULESrNz--missing___rU)rL startswithr8rrrWpackaged_modulesrrcountrrrrrgetrrrrisdir)r[r namespacerrrlegacy_relative_data_dirlegacy_cache_dirs& rQ_check_legacy_cache"DatasetBuilder._check_legacy_cachesc OO & &{ 3 3!$"6"677   I- D6:lllt||GYGYZ]G^abGb **3/2hlI=A\\=U$,,..sD9[_[l[lK#nnS9I9I5J5L&MMI-11$))YGJD'0~~%.%6!!ykTM^M^L_<` ( $ )~~d.B.BD\] ww}}-..///.8 4rPc4<V^8dQhRRRS[S[,/#)rodataset_modulerGrrr)rurvs"rQrwrs -0-0?-0xPS}-0rPc VPPR4'EdE\VP4'Eg'\ VP 4RR0, 'Eg^RIHp^RIH pRRlpVP'dCVPPR4^8d#VPPR4^,MR p\P!VR R 4;_uu_4VPP R ,\"P$!RVPP&/4,pR R R 4VP)VP R 4pVP*P,'dgVPP VP*P,9d8V!WqP*P,VPP ,4p\.P0!Vf VP2MV RVP2 2XRV4p\.P0!VPV4p \4P6P9V 4'dV#R #R #R #R # +'giEL7;i)zxCheck for the old cache directory template {cache_dir}/{namespace}___{dataset_name}/{config_name}-xxx from 2.14 and 2.15rrXr{)&_PACKAGED_DATASETS_MODULES_2_15_HASHES)Picklerc<V^8dQhR\R\R\/#)rorconfig_parametersrr)rtrs)rus"rQrw9DatasetBuilder._check_legacy_cache2..__annotate__s# % % %QU %Z] %rPc0Rmp\VP44UUu/uFwr4W29gK W4bK ppp\4pVPV4VPV4VP 4#uuppi)z Used to update hash of packaged modules which is used for creating unique cache directories to reflect different config parameters which are passed in metadata from readme. >rzrr)rrr&rr)rr params_to_excludeparamvalueparams_to_add_to_hashrs&& rQ"update_hash_with_config_parametersODatasetBuilder._check_legacy_cache2..update_hash_with_config_parameterssz %N!)//@/F/F/H(I)(I 5!EL(I&) H./{{}$)s A5A5rN_legacy_no_dict_keys_sortingTrrrrU)rLrr8rrhrprr  utils._dillr rrrr objectrrWr&rrXrbuilder_configs_parametersmetadata_configsrrrrrr) r[rr r rrrrrrs && rQ_check_legacy_cache2#DatasetBuilder._check_legacy_cache2s OO & &{ 3 3!$"6"677++, j/III P , % 7;lllt||GYGYZ]G^abGb **3/2hlIg'EtLL KK,,s2V[[,PTP[P[PfPfAg5hh M9==diiSD99JJJKK$$(Q(Q(b(bb9CCTTUYU`U`UeUef(1~~%.%6!!ykTM^M^L_<` ( $ )~~d.B.BD\] ww}}-..///OJ8 40MLLs ,AI)) I: c6<V^8dQhRS[S[S[3,/#r)tuplerSrt)rurvs"rQrwrs"Y)Y) }c! "Y)rPc BRpVEfXVP'EdEVPeVVPPVP4p\P RVP RVP 24M\VP4^8dV'g}RVP;'g VP RVP^,P R2p\R\VPP44 2RV R 2,4hMBVP^,p\P R VP RVP 24\V\4'ddVPPV4pVfEVP'd3\R V R \VPP44 24hV'gVeWR &M)VP'dV'gVPVR &RV9d4\VR4'd"VP 'dVP VR&VP"!R/VBpMnV'd\$P&!V4MTpVP)4F:wrxVfK \WW4'g\RV RV R24h\+WWV4K< VP'g\RVP 24hVP-VP.\1VP2VP4R7R7VfVP7VVR7pW0P9;'dVR8gp V 'd\P RV 24WS3#VPVP9dQWPPVP,8wd0\R\VPP44 24hVP8'g\RVP R24hWS3#)zCreate and validate BuilderConfig object as well as a unique config id for this config. Raises ValueError if there are multiple builder configs and config_name and DEFAULT_CONFIG_NAME are None. config_kwargs override the defaults kwargs in config Nz$No config specified, defaulting to: rzload_dataset('z', 'z')zEConfig name is missing. Please pick one among the available configs: z Example of usage: ``z6No config specified, defaulting to the single config: zBuilderConfig 'z' not found. Available: rWrzVERSIONzBuilderConfig z doesn't have a 'z' key.z$BuilderConfig must have a name, got rr)rqrTz Using custom data configuration zvCannot name a custom BuilderConfig the same as an available BuilderConfig. Change the name. Available BuilderConfigs: z must have a versionrJ)BUILDER_CONFIGSDEFAULT_CONFIG_NAMEbuilder_configsrrrrrWrrrZrrjrYrthasattrr!rrdeepcopyrsetattrrrrrrrrz) r[rrqrrpbuilder_configexample_of_usagekeyr is_customs &&&&, rQr%DatasetBuilder._create_builder_configs  4#7#7#7''3!%!5!5!9!9$:R:R!S B4CTCTBUUVWeWjWjVklmt++,q0(,T\\-N-NT=N=N,OtTXThThijTkTpTpSqqst))NNRSWSgSgSlSlSnNoMpr 89I8J!LM )&*%9%9!%*>*> %k]2J4PTPdPdPiPiPkKlJmn &(3f%)))-(,(@(@ f% -'$ 2J2Jt|||+/<< i(!66GGN?LT]]>:Q_N+113 $">77(>.9IIZ[^Z__e)fggN7 4 """CNDWDWCXYZ Z **nn*TMaMab +  &77 /8I&:&::VV Y@V  KK:9+F G((##t';';;"&:&:>;N;N&OO QQUVZVjVjVoVoVqQrPsu"))) >.2E2E1FFZ![\\((rPc6<V^8dQhRS[S[S[3,/#r)rsrtrS)rurvs"rQrwrUsS-%7 8rPc VPUu/uFqPVbK pp\V4\VP48wd3VPUu.uFqPNK pp\RV 24hV#uupiuupi)z@Dictionary of pre-defined configurations for this builder class.z5Names in BUILDER_CONFIGS must not be duplicated. Got )r"rWrrZ)clsrconfigsnamess& rQr$DatasetBuilder.builder_configsRs 695H5HI5H6;;&5HI w<3s223 3/2/B/BC/BV[[/BECTUZT[\] ] JCs A=BcVP#ra)rrs&rQrDatasetBuilder.cache_dir]s rPc<V^8dQhRR/#)rorrGrJ)rurvs"rQrwras+++rPcVPV4;'gVP4;'gRVnVP4VnVPVnR#ra)rrrrrr)r[rs&&rQ!_use_legacy_cache_dir_if_possible0DatasetBuilder._use_legacy_cache_dir_if_possibleasR  % %n 5 [ [9Q9Q9S [ [W[ &//1??rPc <V^8dQhRS[/#rrt)rurvs"rQrwris  s rPc VPeV'dV'd VP#VP'dCVPPR4^8d#VPPR4^,MRpVf VPMV RVP 2p\ P !W@P4pV'd5\ P !V\VPP44pV'dSVP'dA\VP\4'd!\ P !W@P4pV#)aORelative path of this dataset in cache_dir: Will be: self.dataset_name/self.config.version/self.hash/ or if a repo_id with a namespace has been specified: self.namespace___self.dataset_name/self.config.version/self.hash/ If any of these element is missing or if ``with_version=False`` the corresponding subfolders are dropped. Nrr) rrrrrrrrrtrrzrrY)r[ with_version with_hashrbuilder_data_dirs&&& rQ_relative_data_dir!DatasetBuilder._relative_data_diris  ) ) 5,911 126,,,4<>*:NNK (~~.>DKKDWDW@XY  z$))S'A'A(~~.> J rPc  (a\P!VPVPRR74o\P!VPVPRR74pV3Rlp\ S4'gV!4pV'dV^,^,pW@P P 8wd_R\V4 RVP RVP R\VP P 4 R 2 p\PV4V#) z2Return the data directory for the current version.F)r<Tc,<\PPS4'g.#.p\P!S4F+pVP \ P !V4V34K- VPRR7V# \dKQi;i)z"Returns previous versions on disk.T)reverse) rrrrappendrrrZsort)version_dirnamesdir_namer>s rQ_other_versions_on_disk@DatasetBuilder._build_cache_dir.._other_versions_on_disks77>>"233 ! JJ'78$++U]]8-Dh,OP9  ! !$ ! /# #"s'B BBzFound a different version z of dataset z in cache_dir z". Using currently defined version r) rrrr?r8rrzrtrrr)r[version_data_dirrH version_dirs other_versionwarn_msgr>s& @rQrDatasetBuilder._build_cache_dir}s$>>$*>*>@W@Wej@W@kl$>>$*>*>@W@Wei@W@jk $-..24L ,Q 2  KK$7$774S5G4H UYUfUfTgh%%)%9%9$::\t{{2234A7 NN8,rPc <V^8dQhRS[/#rr')rurvs"rQrwrs " "{ "rPc \h)zConstruct the DatasetInfo object. See `DatasetInfo` for details. Warning: This function is only called once and the result is cached for all following .info() calls. Returns: info: (DatasetInfo) The dataset information NotImplementedErrorrs&rQrDatasetBuilder._infos "!rPc \PP\P!\P !V444#)z8Return the path of the module of this class or subclass.)rrdirnamergetfile getmodule)r/s&rQget_imported_module_dir&DatasetBuilder.get_imported_module_dirs+wwww/@/@/EFGGrPc&<V^8dQhRS[RS[/#)rosrcdstr:)rurvs"rQrwrs##3#S#rPc2\VPW4R#ra)r%r)r[r[r\s&&&rQ_renameDatasetBuilder._renamestxx"rPc(<V^8dQhRS[S[,RS[S[,RS[S[S[S[3,,RS[S[S[S[3,,RS[S[,RS[S[,RS[RS[S[S[S[3,,R S[S[,R S[S[,/ #) ro output_dirr download_modeverification_moderr file_formatmax_shard_sizenum_procr) rrtrr rr9rr~rs)rurvs"rQrwrsYYSMY".1Y lC&7 89 Y $E*:C*?$@A Y _- YC=YY!sCx1Y3-Y"$YrPc  aaVeTM SPp\V3/T ;'g/BwrV Sn\SP4'gTMSPP V4Sn\ T;'g\ P4p\T;'g\P4pVeTM SPpVeVR'9d\RV R24hVSn SPPSP 4R8Xd8\RSP RSP SP, R24hVfVfS\!SP"V\ P$8HV\ P$8HRV SP&SP(R 7p\+SPVSP,P.VSP0R 7p\SP4'*oVSnS'dD\5SP 4P6P9R R R 7SP R ,p S'd \;X 4M\<P>!4;_uu_4SPPA\BPD!SP \,PF44pV'dfV\ P8XdQ\HPKRSP RSP R24SPM4Sn%RRR4R#\HPKRSP RSP R24S'd\OSPJPP;'g^\5SP 4P6R7'g\SR\USPJPP;'g^4 R\USPJPV;'g^4 R\USPJPX;'g^4 24h\<PZVV3Rl4pSPJPP'd\HPKRSP RSP,P\ R\USPJPV4 R\USPJPX4 R\USPJPP4 RSP R2 4MvS'd&SPPSP 4M SP p\HPKRSP RSP,P\ RV R24V!SP 4;_uu_4p\_SRV4;_uu_4RV/pVeVVR&V eV VR &SP`!R(R!VR"V/VBV B\cR#SPJPdPg444SPJn,VPh'd VPk4SPJn6SPJPVeASPJPXSPJPV,SPJn(SPo4RRR4RRR4\HPKR$SP R%SP R&24RRR4R# +'giLR;i +'giL];i +'giR#;i))a Downloads and prepares dataset for reading. Args: output_dir (`str`, *optional*): Output directory for the dataset. Default to this builder's `cache_dir`, which is inside `~/.cache/huggingface/datasets` by default. download_config (`DownloadConfig`, *optional*): Specific download configuration parameters. download_mode ([`DownloadMode`] or `str`, *optional*): Select the download/generate mode, default to `REUSE_DATASET_IF_EXISTS`. verification_mode ([`VerificationMode`] or `str`, defaults to `BASIC_CHECKS`): Verification mode determining the checks to run on the downloaded/processed dataset information (checksums/size/splits/...). dl_manager (`DownloadManager`, *optional*): Specific `DownloadManger` to use. base_path (`str`, *optional*): Base path for relative paths that are used to download files. This can be a remote url. If not specified, the value of the `base_path` attribute (`self.base_path`) will be used instead. file_format (`str`, *optional*): Format of the data files in which the dataset will be written. Supported formats: "arrow", "parquet". Default to "arrow" format. If the format is "parquet", then image and audio data are embedded into the Parquet files instead of pointing to local files. max_shard_size (`Union[str, int]`, *optional*): Maximum number of bytes written per shard, default is "500MB". The size is based on uncompressed data size, so in practice your shard files may be smaller than `max_shard_size` thanks to Parquet compression for example. num_proc (`int`, *optional*, defaults to `None`): Number of processes when downloading and generating the dataset locally. Multiprocessing is disabled by default. storage_options (`dict`, *optional*): Key/value pairs to be passed on to the caching file-system backend, if any. **download_and_prepare_kwargs (additional keyword arguments): Keyword arguments. Example: Download and prepare the dataset as Arrow files that can be loaded as a Dataset using `builder.as_dataset()`: ```py >>> from datasets import load_dataset_builder >>> builder = load_dataset_builder("cornell-movie-review-data/rotten_tomatoes") >>> builder.download_and_prepare() ``` Download and prepare the dataset as sharded Parquet files locally: ```py >>> from datasets import load_dataset_builder >>> builder = load_dataset_builder("cornell-movie-review-data/rotten_tomatoes") >>> builder.download_and_prepare("./output_dir", file_format="parquet") ``` Download and prepare the dataset as sharded Parquet files in a cloud storage: ```py >>> from datasets import load_dataset_builder >>> storage_options = {"key": aws_access_key_id, "secret": aws_secret_access_key} >>> builder = load_dataset_builder("cornell-movie-review-data/rotten_tomatoes") >>> builder.download_and_prepare("s3://my-bucket/my_rotten_tomatoes", storage_options=storage_options, file_format="parquet") ``` NzUnsupported file_format: z. Expected 'arrow' or 'parquet'z7Unable to download and prepare the dataset at the root z'. Please specify a subdirectory, e.g. ''F)rforce_download force_extractuse_etagrfrr)rrr{rrecord_checksumsT)parentsrz _builder.lockzFound cached dataset z ()zGenerating dataset ) directoryzNot enough disk space. Needed: z (download: z , generated: c3P<"S'g$SPPVRR7VxR#VR,p\P!VRR7Vx\PP V4'd\ P !V4\ P!W4\PPV4'd\ P !V4R#R# \PPT4'd\ P !T4ii;i5i)z4Create temporary dir for dirname and rename on exit.Trz .incompleteN) rrrrrshutilrmtreemover)rUtmp_diris_localr[s& rQincomplete_dir;DatasetBuilder.download_and_prepare..incomplete_dirPs HH%%g%=!M% 5GKK$73% 77==11"MM'2 G577>>'22"MM'23277>>'22"MM'23s$A D&(C%:,C%&?D&%>D##D&z"Downloading and preparing dataset rz , total: z) to z... to rrdrerfrrcc38"TFqPxK R#5ira) num_bytes)rcrs& rQrf6DatasetBuilder.download_and_prepare..s0hNgUNgsDataset z downloaded and prepared to z(. Subsequent calls will reuse this data.)arrowparquetrJ)8rr rr$unstrip_protocolrrREUSE_DATASET_IF_EXISTSr9 BASIC_CHECKSrrZr_strip_protocol RuntimeErrorrrrFORCE_REDOWNLOADrrrrr{rrrparentmkdirr7 contextlib nullcontextrrrrrr _load_infor> size_in_bytesOSErrorrB download_size dataset_sizecontextmanagerrWrC_download_and_preparesumsplitsrrmget_recorded_sizes_checksumsdownload_checksums _save_info)r[rarrbrcrrrdrerfrdownload_and_prepare_kwargsfsr data_existsrw_desttmp_output_dirprepare_split_kwargsrvsf&&&&&&&&&&, @rQdownload_and_prepare#DatasetBuilder.download_and_prepareszj$.#9Zt ":I/2G2GRI-A$((-K-K:QUQYQYQjQjkuQv$]%Z%Zl6Z6Z[ ,->-_-_BRB_B_`!*!6IDNN  "{:N'N8 Edef f' 88 # #D$4$4 5 ;I$JZJZI[\88<8H8H4K\K\8\7]]^`   &"0"88#0L4Q4Q#Q"/<3P3P"P"%**$($8$8#)!.. /--#!%!7!7 J,DHH55$  !! " ) ) / /t / L((?:I%-Xi *2H2H2J J J((//)..9I9I6KgKg*hiK} 0T0TT 3D4E4E3FbIYIYHZZ[\]!OO- K J KK-d.?.?-@4CSCSBTTUV W0II++00qDAQAQ'>?@dFVFVEWWZ\ GO001A1ABTXTdTd @ARAR@SSTUYU`U`UeUeTffjkpjqqtuv  0 011^*$ ~NN,9;+G(%1AO,-=>+;C,Z8..#-*;/6 .10hdiiN^N^NeNeNg0h-hDII*!2227A7^7^7` 4yy..:26))2H2H499KbKb2b /OO%'O20 KK4,,--I$JZJZI[\9: YK JnON21iK J Jsr9B,\=/7\='!\= A\=)\=)\=1A\=4B/\=$B\=,\* D \ \* 8\= \'"\* * \: 5\== ] c \VPR7pVPV4pVP!V3/VBpV\P 8XdBVP 'd0\VPPVP4R4VFp\VPP4P4R8Xd \R4h\ PRVPP R24VP#VP4VP$!V3/VBTP)4K V\P*8XgV\P 8Xd!\-VPP.V4W@PnVP0VPnR# \&d"p\'R\T4,4RhRp?ii;i) aDownloads and prepares dataset for reading. This is the internal implementation to overwrite called when user calls `download_and_prepare`. It should download all required data and generate the pre-processed datasets files. Args: dl_manager ([`DownloadManager`]): `DownloadManager` used to download and cache data. verification_mode ([`VerificationMode`]): if `ALL_CHECKS`, perform all the verifications including checksums. if `BASIC_CHECKS`, do not perform checksums, only perform split tests. if `NO_CHECKS`, do not perform any verification. prepare_split_kwargs: Additional options, such as `file_format`, `max_shard_size` )rzdataset source filesrkz{`all` is a special split keyword corresponding to the union of all splits, so cannot be used as key in ._split_generator(). Generating  splitNz(Cannot find data file. Original error: )r0r_make_split_generators_kwargs_split_generatorsr9 ALL_CHECKSrmr:rrrrt split_inforWlowerrZradd_prepare_splitrmanage_extracted_filesrr;rdownloaded_sizer) r[rrcr split_dictsplit_generators_kwargssplit_generatorssplit_generatores &&&, rQr$DatasetBuilder._download_and_prepares"D,=,=> "&"D"DEY"Z151G1G 1nVm1n  0 ; ; ; @[@[@[  ,,j.U.U.WYo  0O?--22399;uD + KK+o&@&@&E&E%FfM N NN?55 6 d##OL7KL  - - /! 0$  0 = = =ARVfVqVqAq $))**J 7& ","<"<  dORUVWRXXY_cc dsF:: G&G!!G&c <V^8dQhRS[/#rr')rurvs"rQrwrsffKfrPcn\P!VPVPPR7#)r)r(rrrrrs&rQrDatasetBuilder._load_infos%))$*:*:DHHLdLdeerPcb\VP4'g\VPR,4M\P !4pV;_uu_4VP PVPVPPR7RRR4R# +'giR#;i)z _info.lockrN) r$rr7rrrrwrite_to_directoryr)r[ file_locks& rQrDatasetBuilder._save_infoss(11 T%% 4 5'')  Y II ( ()9)9488KcKc ( dYYYs  in_memory (`bool`, defaults to `False`): Whether to copy the data in-memory. Returns: datasets.Dataset Example: ```py >>> from datasets import load_dataset_builder >>> builder = load_dataset_builder('cornell-movie-review-data/rotten_tomatoes') >>> builder.download_and_prepare() >>> ds = builder.as_dataset(split='train') >>> ds Dataset({ features: ['text', 'label'], num_rows: 8530 }) ``` r~zELoading a dataset not written in the "arrow" format is not supported.zLoading a dataset cached in a z is not supported.r}z: could not find data in z. Please make sure to call builder.download_and_prepare(), or use datasets.load_dataset() before trying to access the Dataset object.zConstructing Dataset for split z, z, from ) in_memoryT) map_tuple disable_tqdm)rr"r$rrRtyperKrrrrFileNotFoundErrorrrrrrrr@r_build_single_datasetrYrsr)r[rrsdatasetss&&& rQ as_datasetDatasetBuilder.as_datasetseF    (T->->'-I!"ij j  ) )%(FtDHH~G^G^F__q&rs sww~~d..//#4,,--FtGWGWFXYVV   6u7[7[ $))JZJZ@[6\\cdhdtdtcuvw =#'99#3#34#3aT#3E4 **#    h % %"8,H5s5 F c@<V^8dQhRS[S[S[S[3,RS[/#)rorr)r rtrr/r})rurvs"rQrwrs+S/501rPc \V\4'gQ\V4pVR8Xd4RPVPP P 44p\V4pVPVVR7pV#)zas_dataset for a single split.rk+)rr) rYrrtrrrrjr/ _as_dataset)r[rrdss&&& rQr$DatasetBuilder._build_single_datasetso %11JE~!1!1!6!6!89%LE    rPcB<V^8dQhRS[S[S[3,RS[RS[/#)rorrrr)r rr/r}r)rurvs"rQrwr)s3BB'=!>BY]BjqBrPc xVPPVP4pVPpVP 4'd VP p\ W0P4PVVVPPP4VR7pVPV4p\RRV/VB#)aConstructs a `Dataset`. This is the internal implementation to overwrite called when user calls `as_dataset`. It should read the pre-processed datasets files and generate the `Dataset` object. Args: split (`datasets.Split`): which subset of the data to read. in_memory (`bool`, defaults to `False`): Whether to copy the data in-memory. Returns: `Dataset` )rW instructions split_infosr fingerprintrJ) rrrrrrWrrreadrr_get_dataset_fingerprintr)r[rrrrdataset_kwargsrs&&& rQrDatasetBuilder._as_dataset)s HH,,T-=-=> ((  # # % %99L$Y :?? ((//1 @  33E: A;A.AArPc<<V^8dQhRS[S[S[3,RS[/#r)r rr/rt)rurvs"rQrwrFs&eOU4J.KPSrPc \4pVP\VP44P 44VP\ V44VP 4pV#)zThe dataset fingerprint is the hash of the relative directory dataset_name/config_name/version/hash, as well as the split specs.)r&rrr?rrtr)r[rhasherrs&& rQr'DatasetBuilder._get_dataset_fingerprintFsP d42245>>@A c%j!&&( rPc x<V^8dQhRS[S[,RS[S[,RS[S[S[S[3,S[3,/#)rorrrr)rrtr rsr+)rurvs"rQrwrNsE>>}>C=> tC()?: ; >rPc a\VP4'd-\R\VP4P R24h\ T;'g VP \VPVPR7VPVPPR7pVPV4Uu/uFqDPVbK upoVfSpEM\V4S9dS\V4,pEM\!VR4'd\#SP%4UUu/uFwrdWdP&bK upp4pVP)V4pVP+4p \,;QJdRV 4F 'gK RM RM !RV 44'd\/RV R \1S4 24h\3V U u.uF/p VP5SV P&P,4NK1 up VP6VR 7#\V4\\8P:48XdE\3SP=4Uu.uFq@P5V4NK upVP6VR 7#R \V49dR \V49dR \V49d\V4P?R 4U u.uFqPA4NK p p \B;QJdV3RlV 4F 'dK RM RM!V3RlV 44'g\/RV R \1S4 24h\3V U u.uFqP5SV ,4NK up VP6VR 7#\/RV R \1S4 24h\EVP4VRR7p \GV \H4'd \KV 4p V #uupiuuppiuup iuupiuup iuup i)z(Loading a streaming dataset cached in a z is not supported yet.r)rrrr{get_read_instructionc3<"TFqPRJxK R#5ira) slice_value)rcrs& rQrf6DatasetBuilder.as_streaming_dataset..hsT *))5 sTFz Bad split: z. Available splits: )rrr[]c3,<"TF qS9xK R#5irarJ)rc split_namesplits_generatorss& rQrfrzsU:%66s)r)&r$rrRrrKrrrrrrrr{rrWrtr%r0rrrget_list_sliced_split_infoanyrZrr,_as_streaming_dataset_singlerr/ALLrrstriprkr@rYrsr)r[rrrsgsplits_generatorrWrsplit_instructionrrr split_namesrrs&&& @rQas_streaming_dataset#DatasetBuilder.as_streaming_datasetNsa  ) )%:4>;R;R:SSij .114>>*TMaMab**[[))  483I3I*3UV3URWWb[3UV =0  Z, ,0U<  U2 3 3"HYH_H_Ha#bHaHDD--$7Ha#bcJ % : :: F +FFHKsT TsssT TTT ;ug5I$O`JaIb!cdd1'2&1 556G H]H]HbHb6cd&1YY  Z3uyy> )1ARAYAYA[\A[22226A[\YY  CJ 3c%j#8SE =R@CE @P@PQT@UV@U*++-@UKV3UU333UUUU ;ug5I$O`JaIb!cdd1dopdoV`223DZ3PQdopYY  {5'1EdK\F]E^_` `  - -   h % %*84H]W$c ] Wqs$<O *O 5OOO" O'c <V^8dQhRS[/#r)r+)rurvs"rQrwrs      rPcVPV4pVP'dVPVP/M/p\W PVP VR7#))rrtoken_per_repo_id) _get_examples_iterable_for_splitrrr+rrW)r[r ex_iterablers&& rQr+DatasetBuilder._as_streaming_dataset_singlesR;;,,,T\\4::6B ii/?/D/DXi  rPc6<V^8dQhRS[S[S[3,/#)ror)r rr)rurvs"rQrwrs,$,$E/C[2[,\,$rPc \4h)aSpecify feature dictionary generators and dataset splits. This function returns a list of `SplitGenerator`s defining how to generate data and what splits to use. Example: return [ datasets.SplitGenerator( name=datasets.Split.TRAIN, gen_kwargs={'file': 'train_data.zip'}, ), datasets.SplitGenerator( name=datasets.Split.TEST, gen_kwargs={'file': 'test_data.zip'}, ), ] The above code will first call `_generate_examples(file='train_data.zip')` to write the train data, then `_generate_examples(file='test_data.zip')` to write the test data. Datasets are typically split into different subsets to be used at various stages of training and evaluation. Note that for datasets without a `VALIDATION` split, you can use a fraction of the `TRAIN` data for evaluation as you iterate on your model so as not to overfit to the `TEST` data. For downloads and extractions, use the given `download_manager`. Note that the `DownloadManager` caches downloads, so it is fine to have each generator attempt to download the source data. A good practice is to download all data in this function, and then distribute the relevant parts to each split with the `gen_kwargs` argument Args: dl_manager (`Union[DownloadManager, StreamingDownloadManager]`): Download manager to download the data Returns: `list`. rQ)r[rs&&rQr DatasetBuilder._split_generatorssZ"##rPc h<V^8dQhRS[RS[RS[S[S[S[3,,RS[S[,/#)rorrdrerf)r1rtrr r~)rurvs"rQrwrsD$$'$$!sCx1 $ 3- $rPc \4h)a}Generate the examples and record them on disk. Args: split_generator (`SplitGenerator`): Split generator to process file_format (`str`, *optional*): format of the data files in which the dataset will be written. Supported formats: "arrow", "parquet". Default to "arrow" format. max_shard_size (`Union[str, int]`, *optional*): Maximum number of bytes written per shard, default is "500MB". The size is based on uncompressed data size, so in practice your shard files may be smaller than `max_shard_size` thanks to Parquet compression for example. num_proc (`int`, *optional*, defaults to `None`): Number of processes when downloading and generating the dataset locally. Multiprocessing is disabled by default. **kwargs: Additional kwargs forwarded from _download_and_prepare rQ)r[rrdrerfkwargss&&&&&,rQrDatasetBuilder._prepare_splits 8"##rPc&<V^8dQhRS[RS[/#rorrrr1r*)rurvs"rQrwrs$$$Sc$rPc \4h)ztGenerate the examples on the fly. Args: split_generator (`SplitGenerator`): Split generator to process rQr[rs&&rQr/DatasetBuilder._get_examples_iterable_for_splits "##rP)rirrrrrrrrrrrrrprrrrrWrrr)NNNNNNNNNNNNNN)NNN)TT) NNNNNNr~NNN)NF)F)NNr~NN)2rKrLrMrNrr!rSrr"r#rSLEEP_ON_THREADS_SHUTDOWNSrrrrrrr< classmethodrAr$propertyrr7r?rabcabstractmethodrrXr^rrrrrrrr/TRAINrrrrrrrrOrrs@rQrrsFRG)O !%"'s3s3j3 00.-0-0^Y)Y)v Y++  (  D  " "HH##YYv3=jffe @@D&BG`eBB:>>@    ,$,$\ $$$:$$rPrc2a]tRtRtoRtV3RltRtVtR#)KeyicD\VPVP34#ra)rtoriginal_shard_iditem_or_batch_idrs&rQ__str__ Key.__str__sD**D,A,ABCCrPc2<V^8dQh/S[;R&S[;R&#)ror r)r~)rurvs"rQrwKey.__annotate__srPrJN)rKrLrMrNrrrOrrs@rQr r s D rPr caa]tRtRtoRtV3RlRlt]PV3RlRl4tRV3RlRllt V3R lR lt V3R lt V3R lR lt Rt VtV;t#)GeneratorBasedBuilderiacBase class for datasets with data generation based on dict generators. `GeneratorBasedBuilder` is a convenience class that abstracts away much of the data writing and reading of `DatasetBuilder`. It expects subclasses to implement generators of feature dictionaries across the dataset splits (`_split_generators`). See the method docstrings for details. c\<V^8dQhRS[S[S[S[S[S[3,3,,/#rrr rtrsr)rurvs"rQrw"GeneratorBasedBuilder.__annotate__+$$HU3S#X;N5O,P$rPc \4haDefault function generating shards paths for each `SplitGenerator`. This function is useful to list the original shards from where the data comes from and is either converted to Arrow or streamed to an IterableDataset. This is optional and only used for certain utilities, but not in Dataset nor IterableDataset. E.g. it's used to map original shard files to Parquet files in the Dataset Viewer after conversion. Args: **kwargs (additional keyword arguments): Arguments forwarded from the SplitGenerator.gen_kwargs Yields: shard: generally a string representing the shard path, or a dict representing the shard in case of shards spanning intra or inter-files. rQr[rs&,rQ_generate_shards&GeneratorBasedBuilder._generate_shards $"##rPc\<V^8dQhRS[S[S[S[S[S[3,3,,/#r)rrr rsrtr)rurvs"rQrwrs+$$huS$sCx.=P7Q.R$rPc \4h)aDefault function generating examples for each `SplitGenerator`. This function preprocess the examples from the raw data to the preprocessed dataset files. This function is called once for each `SplitGenerator` defined in `_split_generators`. The examples yielded here will be written on disk. Args: **kwargs (additional keyword arguments): Arguments forwarded from the SplitGenerator.gen_kwargs Yields: key: `str` or `int`, a unique deterministic example identification key. * Unique: An error will be raised if two examples are yield with the same key. * Deterministic: When generating the dataset twice, the same example should have the same key. Good keys can be the image id, or line number if examples are extracted from a text file. The key will be hashed and sorted to shuffle examples deterministically, such as generating the dataset multiple times keep examples in the same order. example: `dict`, a feature dictionary ready to be encoded and written to disk. The example will be encoded with `self.info.features.encode_example({...})`. rQrs&,rQ_generate_examples(GeneratorBasedBuilder._generate_exampless :"##rPc b<V^8dQhRS[RS[S[,RS[S[S[S[3,,/#)rorrfre)r1rr~r rt)rurvs"rQrwr5s?\*\*'\*3- \* !sCx1 \*rPcP aa%a&a'\T;'g\P4pSPPe)SPPVP ,pM VP pRpSP RVP V RV 2p\P!SPV4o%V'dV^8dz\VP4pV^8:d*\PRV RVP R24^pM4W8d/\PRV RV RVP R V R 2 4Tp\R VP R VP R 2R7p RS%RVRVRV/p VeV^8XdRp VPp ^p T ;_uu_4SP"!R$RV RV /V BF#wrpV'dTp KV P%V4K% RRR4V fQR4hRV 4wpppo&pppEM'\'\)VPVR74U U u.uF wrRV RV /V CNK pp p \+V4pR.V,pR.V,pR.V,pR.V,o&R.V,pR.V,pR.V,p\-V4;_uu_4pT ;_uu_4\/VSP"VR7F@wrpV'd"VwVV &VV &VV &S&V &VV &VV &VV &K/V P%V4KB RRR4RRR4RV9g QRV R24h\1S&4o'\1V4p\1V4p\1V4pV^,pVVP nVVP n\P5RS' R 24S'^8d~RV%VV&V'3Rllp\'S&4U UUu.uFwp p\7V4FpVV 3NK K p pp p\9VV R^@R7VU!U"u.uFp!V!Fp"V"NK K up"p!VP nMJ^^p pSP=S%P?R VR! 4P?R"V R! 4S%P?VR#44V^8dC\P@'d-VU#U$u.uFp#V#Fp$V$NK K up$p#VP n!SPPDfVSPn"R#R# +'giEL ;iuup p i +'giEL;i +'giEL;iuuppp iuup"p!iuup$p#i)%N-JJJJJ-SSSSS-of-NNNNNrrSetting num_proc from  back to 1 for the @ split to disable multiprocessing as it only contains one shard.ry for the  split as it only contains  shards. examplesrrunittotaldescfpathrdrer gen_kwargsjob_id-Failed to retrieve results from prepare_splitc3&"TFq.xK R#5irarJrcitems& rQrf7GeneratorBasedBuilder._prepare_split..w+FDF max_num_jobskwargs_iterable;Failed to retrieve results from prepare_split: result list G still contains None - at least one worker failed to return its results Renaming c:V^8dQhR\\,/#)ro shard_and_jobrr~)rus"rQrw:GeneratorBasedBuilder._prepare_split..__annotate__s  U3Z rPc<Vwr\SRV4V,pSPSPRVR 4PRVR 4SPRVR 4PRSR 44R#NSSSSS05dJJJJJz JJJJJ-SSSSSNNNNNrr^r)rCshard_idr3global_shard_idr1r[shards_per_job total_shardss& rQ _rename_shard;GeneratorBasedBuilder._prepare_split.._rename_shards{#0 "%nWf&=">"I MM'hs^=EEgRXY\Q]_MM-OC3HJRRSZ_klo^prrPTdisable max_workersrHrIrJrhrJ)#r=rMAX_SHARD_SIZErrrWrrrrrrDr2rrhf_tqdm num_examples_prepare_split_singler enumeraterErr r?rr{rranger shard_lengthsr^rSAVE_ORIGINAL_SHARD_LENGTHSoriginal_shard_lengthsr)(r[rrdrfrerSUFFIXfnamenum_original_shardspbar_prepare_split_argsresultr2r3donecontentexamples_per_job bytes_per_jobfeatures_per_jobshard_lengths_per_joboriginal_shards_per_joboriginal_shard_lengths_per_jobkwargs_per_jobnum_jobspooltotal_original_shardstotal_num_examplestotal_num_bytesrrQ num_shardsrMshards_and_jobsr\ shard_lengthr^original_shard_lengthr1rOrPs(f&&&& @@@rQr$GeneratorBasedBuilder._prepare_split5s2.2Y2YFDYDYZ 99   '))/*>*>?J(33J($$%Q';';&*H $v0 !FX-M $v0 "Vh.N%)FX$5 !'+fx&7 #.2Vh-> *h4T1Cd88.2-g !( 0 8 -f 5 0 8 .v 6 5f = 7 ? >v F!KK0!2 (// MN^M_`gh />* #$; < !12m,#A&2D""//>"",  yh78 !   +4N*C*C&FJ %j 1H6" 1#*C  }otQS T3H82G\iL \i 2G8O & & 4 !!fH LL g(39AA'fUX\[ fb)  !1 $)K)K)K/MA.L*-C)&-C&.LAO & & = 99   %!)DII  &W("TT \8AsI>SS' T AS- =T1"T+T?T" S$ - S> 8 T T c<V^8dQhRS[RS[RS[RS[RS[RS[RS[S[S[S[S[S[S[S[S[S[S[3,3,,/#)ror2r1rdrerr3rr)rsrtr~r2rrr}r#)rurvs"rQrwrs}S S S S  S  S  S S  %T5c8S#s)J#KKL MS rPc #"VP!R /VBpVR8Xd\M\pVR8Hp .p .p ^^r^p^p^pV!VPPVP RVR 4P RVR 4VP VPPV R7p\P!4pVEFwpp\V\4'd VPpVeVPV8dVP4wppVP4V P!V4V V, p V V, p V^, pV!VP"VP RVR 4P RVR 4VP VPPV R7pVPPe&VPPP%V4MTpVP'V4\)V 4V8:d1V P+^.^V,\)V 4, ,4W;;,^, uu&V^, p\P!4V\,P.,8gEK\P!4pVRV3x^pEK VRV3xV^,pV^,pVP4wppVP4V P!V4V V, p V V, p TRT T TP"TT TT 33xR# TRT3xT^,pT^,pTP4wppTP4T P!T4T T, p T T, p i;i \0dBp\T\24'dTP4e TP4p\7R4ThRp?ii;i5i) rrHrIrJrrrrembed_local_filesNF.An error occurred while generating the datasetTrJ)r!rrrrrrrrtimerYr r  _num_bytesfinalizecloserD _featuresencode_examplewriterextendrPBAR_REFRESH_TIME_INTERVAL Exceptionr __context__r!)r[r2r1rdrerr3 generator writer_classr{r\r^rqrrrMr num_examples_progress_updatewriter_timer*recordrXr{examplersrars&&&&&&& rQrY+GeneratorBasedBuilder._prepare_split_singles++9j9 (3y(@}k '94 !#./O'($2 b!++]]7xn>FFwSYZ]R^`"&"9"9 $ 8 8"3 F% - #,KC!#s++,/,A,A)%1f6G6G.6X28//2C/ i %,,\:*l:*'94 A !-%+%5%5!&w8C.!J!R!RSZ_efi^j!l.2.E.E,0HH,D,D.? "LP99K]K]Kidii00??GouGLL)126GG.55qcQAR=RUXYoUp=p6qr*=B=0A50yy{UV-N-N%NN $ $e-III7845$-8e%AAA%\ &7!&;#*0//*;' i $$\2"l2"9,  "  #&   e%AAA%\ &7!&;#*0//*;' i $$\2"l2"9, b!122q}}7PMM()YZ`a a  bsJ8O A#M;GL;#LAM;=O A!M88M;; O.r9r:r;r=r?r@rAc:V^8dQhR\\,/#)roshard_id_and_jobrD)rus"rQrw6ArrowBasedBuilder._prepare_split..__annotate__s  c  rPc<Vwr\SRV4V,pSPSPRVR 4PRVR 4SPRVR 4PRSR 44R#rGrL)rrMr3rNr1r[rOrPs& rQrQ7ArrowBasedBuilder._prepare_split.._rename_shards{#3 "%nWf&=">"I MM'hs^=EEgRXY\Q]_MM-OC3HJRRSZ_klo^prrPTrSrHrIrJrhrJ)$r=rrVrrrWrrrrrrrDr2rrrWrXrYrrZrErr r?rr{rr[r r\r^rr]r^r)(r[rrdrfrerr_r`rarbrcrdr2r3rerfrgrhrirjrkrlrmrnrorprqrrrrQrsrMshard_ids_and_jobsr\rur^rvr1rOrPs(f&&&& @@@rQr ArrowBasedBuilder._prepare_splitcs2.2Y2YFDYDYZ 4))/*>*>?J)$$%Q';';&*H $v0 !FX-M $v0 "Vh.N%)FX$5 !'+fx&7 #.2Vh-> *h4T1Cd88.2-g !( 0 8 -f 5 0 8 .v 6 5f = 7 ? >v F!KK0!2 (// MN^M_`gh />* #$; < !12m,#A&2D""//>"",  yh78 !   +4N*C"*C&FJ %j 1H6" 1#*C " }&8$TV W3H82G\iL \i 2G8O & & 4 !!fH LL g(39AA'fUX\[ fb)  !1 $)K)K)K/MA.L*-C)&-C&.LAO & & = 99   %!)DII  &_ 4(33J 4H("TT \"8AsZ'R-.>S 5S/ S9:AS% S9 "T TT-SS S % S6 0 S99 T c<V^8dQhRS[RS[RS[RS[RS[RS[S[S[S[S[S[S[S[S[S[S[3,3,,/#)ror2r1rdrer3rr)rsrtr~rrr}r#)rurvs"rQrwrsh[ [ [ '*[ 9<[ NQ[ [^[ %T5c8S#s)J#KKL M[ rPc #"VP4UUu/uF)wrgT\V\4'd \V4MTbK+ pppVP!R /VBpVR8Xd\ M\ p VR8Hp .p .p ^^r^p^p^pV !VPPVPRVR 4PRVR 4VPVPPV R7p\P!4pVEFwpp\V\4'd VPpVeVP V8dVP#4wppVP%4V P'V4V V, p VV, pV^, pV !VP(VPRVR 4PRVR 4VPVPPV R7p\+V4^8XdVP-V4MVP/V4\+V 4V8XdV P'\+V44MV V;;,\+V4, uu&V\+V4, p\P!4V\:P<,8gEK\P!4pVRV3x^pEK VRV3xV^,pV^,pVP#4wppVP%4V P'V4V V, p VV, pTR T TTP(TT TT 33xR#uuppi \0d>p\2P4!TTPP6TTP8R7hRp?ii;i TRT3xT^,pT^,pTP#4wppTP%4T P'T4T T, p TT, pi;i \>dYp\T\@4'dTPBe TPBp\T\D4'dh\ER4ThRp?ii;i5i) rrHrIrJrzN) cast_errorrr2rFr|TrJ)#rrYrrFrrrrrrrrrr}r r r~rrrDrr write_row write_tabler4r from_cast_errorrrrrrrrr!)r[r2r1rdrer3rdrrrr{r\r^rqrrrMr rrrr*tablerXr{rrsrars&&&&&& rQrY'ArrowBasedBuilder._prepare_split_singlesT^ScScSefSe41aJq$,?,?aQFSe f))7J7 (3y(@}k '94 !#./O'($? b!++]]7xn>FFwSYZ]R^`"&"9"9 $ 8 8"3 F0 - "+JC!#s++,/,A,A)%1f6G6G.6X28//2C/ i %,,\:*l:*'94 A !-%+%5%5!&w8C.!J!R!RSZ_efi^j!l.2.E.E,0HH,D,D.? " u:?",,U3"..u5126GG.55c%jA./@ASZOA0CJ>0yy{UV-N-N%NN $ $e-III784K#,Ne%AAA%\ &7!&;#*0//*;' i $$\2"l2"9,  "  #&   YgV%8HH'1)-)?)?'1"&** "e%AAA%\ &7!&;#*0//*;' i $$\2"l2"9, b!122q}}7PMM!344()YZ`a a  bsQ/L?:QA#O4$C=N"!MNMBN##NAO4% Q N 8NN  NA!O11O44 Q?AQQQc&<V^8dQhRS[RS[/#rr)rurvs"rQrwr]s   Sc rPc r\VPVP\VRR4VPR7#)rN)rrsleep_on_threads_shutdown)r)rr2rbrrs&&rQr2ArrowBasedBuilder._get_examples_iterable_for_split]s7$  ! !"--$+D2Mt$T&*&E&E   rPrJr)rKrLrMrNrrrrrrrYrrOrrs@rQrr7sPk$$( $$([*[*z[ [ z  rPrca]tRtRto]P V3RlRl4tV3RlRltV3RlRltV3RlR lt R t Vt R #) _CountableBuilderMixinifc0<V^8dQhRS[S[,/#r)rr~)rurvs"rQrw#_CountableBuilderMixin.__annotate__hs$$(3-$rPc \4hrarQrs&,rQ_generate_num_examples-_CountableBuilderMixin._generate_num_examplesgs !##rPc<<V^8dQhRS[RS[S[S[3,/#)rorrr)rrsrtr~)rurvs"rQrwrks'uuuT#s(^urPcVP/4pVP!V3/VBpVUu/uFqDPVPV4bK up#uupira)rrrW_count_examples)r[rrrrs&& rQcount_examples%_CountableBuilderMixin.count_examplesksY"&"D"DR"H151G1G 1nVm1ncstcsP_$$d&:&:?&KKcsttts$Ac&<V^8dQhRS[RS[/#r)r1r~)rurvs"rQrwrps   ~  #  rPc \^ \P!4^,4p\\ VP \ VPV4^RVP 2R74#) zCounting rows for split=)delayr0) minr cpu_countrr _count_examples_singlerEr2rW)r[rrUs&& rQr&_CountableBuilderMixin._count_examplesps["bllnq01  ++!/"<">c3h>C>rPc8\VP!R/VB4#)NrJ)rr)r[r2s&&rQr-_CountableBuilderMixin._count_examples_single{s4..<<==rPrJN) rKrLrMrNrrrrrrrOrrs@rQrrfs=$$uu   >>rPr)rrrrrrrrrr}rcollections.abcr dataclassesr functoolsrpathlibrtypingrrrr unittest.mockr rpyarrowr fsspec.corer multiprocessr tqdm.contrib.concurrentr rhrr arrow_datasetr arrow_readerrr arrow_writerrrrrXrrr dataset_dictrrdownload.download_configrdownload.download_managerrr#download.streaming_download_managerrr exceptionsr r!r"rr# filesystemsr$r%rr&rr(iterable_datasetr)r*r+r,namingr-r.rr/r0r1r2 streamingr3rr4r5r6rWutils._filelockr7utils.file_utilsr8utils.info_utilsr9r:r;utils.py_utilsr<r=r>r?r@rArBrCutils.shardingrDrE utils.trackrFloadrG get_loggerrKrrZrIrSrr rrrrJrPrQrsd !    $!66 !."KJOO:4DP[[  O??;"%+OO   O%#   H %   nRnR nRbW$W$t  DD D{ N{ | l l ^ >^>rP