+ QV-j,^RIt^RIt^RIHt^RIHt^RIHtHt^RI H t H t H t ^RI t ^RIHt^RI Ht^RIHt^R IHt^R IHt^R IHt^R IHt^R IHt^RIHt^RIHt ^RI!H"t"H#t#H$t$H%t%H&t&^RI'H(t(] ])]*]*3,])]*,])R@,3,t+]*!]PX4t-]P\!]/4t0!RR]*4t1!RR]24t3Rt4]PXRR.]Pj.RAO]Pl.RBO/t7Rt8]Pr]Pt!R48d RR.t;.RCOtR?]R]*]T3,4tUR#uuppiuupppiuuppiuupppi)IN)partial) has_magic)PathPurePath)CallableOptionalUnion) url_to_fs) HfFileSystem)version) thread_map)config)DownloadConfig) _split_re)Split)logging)tqdm)!_prepare_path_and_storage_options is_local_pathis_relative_path xbasenamexjoin)string_to_dictc]tRt^!tRtR#)UrlN__name__ __module__ __qualname____firstlineno____static_attributes__rd/Volumes/fast/ai/experiments/ui-tars-smoke/.venv/lib/python3.14/site-packages/datasets/data_files.pyrr!r#rc]tRt^%tRtR#)EmptyDatasetErrorrNrrr#r$r'r'%r%r#r'zFdata/{split}-[0-9][0-9][0-9][0-9][0-9]-of-[0-9][0-9][0-9][0-9][0-9]*.*traintrainingz-._ 0-9z2023.9.0z**[{sep}/]{keyword}[{sep}]*z{keyword}[{sep}]*z 2023.12.0z**/*[{sep}/]{keyword}[{sep}]*z**/{keyword}[{sep}]*z**/*[{sep}]{keyword}[{sep}]*)keywordsepz**logsz **/*.evalz*[]c0V^8dQhR\R\/#)patternreturnstrbool)formats"r$ __annotate__r5us\\\\r#ca\;QJd#V3Rl\4F 'gK R# R#!V3Rl\44#)c3,<"TF qS9xK R#5iNr).0wildcard_characterr/s& r$ %contains_wildcards..vs[GZ1CW,GZsTF)anyWILDCARD_CHARACTERS)r/sfr$contains_wildcardsr?us. 3[GZ[33[3[3[GZ[ [[r#cV^8dQhR\\\\3,R\\\\\,R3,3,/#)r.patternsr0 DataFilesList)r dictlistr2)r4s"r$r5r5ys?&1&1dD#o 6&14U4PS9VeKeEf@f;g&1r#c \V\4'dHVP4UUu/uF*wr\V4\V\4'dTMV.bK, upp#\V\4'd \ V./#\V\4'EdK\ ;QJdRV4F 'gK RM RM !RV44'EdVFip\V\4'dE\V4^8Xd5RV9d.\VPR4\\34'dK]\RV 24h VUu.uF q3R,NK pp\\V44\V48wd\RV 24hVUu/uFDp\VR,4\VR,\4'd VR,M VR,.bKF up#\ V/#\\ V44#uuppiuupiuupi)a Take the data_files patterns from the user, and format them into a dictionary. Each key is the name of the split, and each value is a list of data files patterns (paths or urls). The default split is "train". Returns: patterns: dictionary of split_name -> list of patterns c3B"TFp\V\4xK R#5ir8) isinstancerC)r9r/s& r$r;$sanitize_patterns..sAWz'4((sTFsplitpathzInvalid format for data_files entry. Each item must be a dictionary with the structure {'split': , 'path': }. Received: z*Some splits are duplicated in data_files: ) rGrCitemsr2rDSANITIZED_DEFAULT_SPLITr=lenget ValueErrorsetsanitize_patterns)rAkeyvaluer/splitss& r$rQrQys(D!!ZbZhZhZjkZjJCC:eT#:#:%GZjkk Hc " "'(44 Hd # # 3AA333AA A A#w--G )7*"7;;v#6d DD$%&-I/$7??h7g&&hF?3v;3v;. #MfX!VWW ('GGG$%*WV_VZ:[:[wvbijpbqarr'  ,X6 6 h009l$@s0G2G8 A G=c<V^8dQhR\R\R\/#r.matched_rel_pathr/r0r1)r4s"r$r5r5s&SSSsStSr#c\\V4PPUu.uFq"PR4'gKVNK pp\V4PPUu.uFq"PR4'gKVNK pp\ V4\ V48g#uupiuupi)u When a path matches a pattern, we additionally check if it's inside a special directory we ignore by default (if it starts with a double underscore). Users can still explicitly request a filepath inside such a directory if "__pycache__" is mentioned explicitly in the requested pattern. Some examples: base directory: ./ └── __pycache__ └── b.txt >>> _is_inside_unrequested_special_dir("__pycache__/b.txt", "**") True >>> _is_inside_unrequested_special_dir("__pycache__/b.txt", "*/b.txt") True >>> _is_inside_unrequested_special_dir("__pycache__/b.txt", "__pycache__/*") False >>> _is_inside_unrequested_special_dir("__pycache__/b.txt", "__*/*") False __)rparentparts startswithrM)rWr/partdata_dirs_to_ignore_in_pathdata_dirs_to_ignore_in_patterns&& r$"_is_inside_unrequested_special_dirr`s85==M4N4U4U4[4["u4[D_n_nos_t444["u7?7H7O7O7U7U%o7UtYhYhimYndd7U"%o * +s3Q/R RR#v%osB$B$(B)B)c<V^8dQhR\R\R\/#rVr1)r4s"r$r5r5s);Q;QVY;Qdg;Qlp;Qr#c\V4PUu.uF0q"PR4'gK\V4R08XdK.VNK2 pp\V4PUu.uF0q"PR4'gK\V4R08XdK.VNK2 pp\ V4\ V48g#uupiuupi)u When a path matches a pattern, we additionally check if it's a hidden file or if it's inside a hidden directory we ignore by default, i.e. if the file name or a parent directory name starts with a dot. Users can still explicitly request a filepath that is hidden or is inside a hidden directory if the hidden part is mentioned explicitly in the requested pattern. Some examples: base directory: ./ └── .hidden_file.txt >>> _is_unrequested_hidden_file_or_is_inside_unrequested_hidden_dir(".hidden_file.txt", "**") True >>> _is_unrequested_hidden_file_or_is_inside_unrequested_hidden_dir(".hidden_file.txt", ".*") False base directory: ./ └── .hidden_dir └── a.txt >>> _is_unrequested_hidden_file_or_is_inside_unrequested_hidden_dir(".hidden_dir/a.txt", "**") True >>> _is_unrequested_hidden_file_or_is_inside_unrequested_hidden_dir(".hidden_dir/a.txt", ".*/*") False >>> _is_unrequested_hidden_file_or_is_inside_unrequested_hidden_dir(".hidden_dir/a.txt", ".hidden_dir/*") False base directory: ./ └── .hidden_dir └── .hidden_file.txt >>> _is_unrequested_hidden_file_or_is_inside_unrequested_hidden_dir(".hidden_dir/.hidden_file.txt", "**") True >>> _is_unrequested_hidden_file_or_is_inside_unrequested_hidden_dir(".hidden_dir/.hidden_file.txt", ".*/*") True >>> _is_unrequested_hidden_file_or_is_inside_unrequested_hidden_dir(".hidden_dir/.hidden_file.txt", ".*/.*") False >>> _is_unrequested_hidden_file_or_is_inside_unrequested_hidden_dir(".hidden_dir/.hidden_file.txt", ".hidden_dir/*") True >>> _is_unrequested_hidden_file_or_is_inside_unrequested_hidden_dir(".hidden_dir/.hidden_file.txt", ".hidden_dir/.*") False .)rr[r\rPrM)rWr/r]hidden_directories_in_pathhidden_directories_in_patterns&& r$?_is_unrequested_hidden_file_or_is_inside_unrequested_hidden_dirrfsl""2399"9__S=QZ]^bZchkglZl9""'*00%0OOC4HQTUYQZ_b^cQc0"% ) *c2O.P PP "%s"B6B6B6'B;B;B;cV^8dQhR\\.\\,3,R\\\\,3,/#)r.pattern_resolverr0)rr2rDrC)r4s"r$r5r5s@)d)dxtCy8H/I)ddSVX\]`XaSaNb)dr#c F\EFMpVPRR4pV!V4p\T4^8gK1\ 4pTF?p\ \ T4\ T44pTfQhTPTR,4KA \;QJdRT4F 'gK RM RM !RT44'd\R\ RT R 24h\Uu.uFqwT9gK \T4NK up\T\Uu0uFp\T4kK up, 4,pTUu/uFqwTPTR 7.bK upu# \Fvp .p V P!4F;wr{V F0pV!V4p\T4^8gKT P#T4K9 K= V 'gK^V Uu/uF qwW,bK upu# \R X R V 24h \dEKi;iuupiuupiuupi \dKi;iuupi) a Get the default pattern from a directory or repository by testing all the supported patterns. The first patterns to return a non-empty list of data files is returned. In order, it first tests if SPLIT_PATTERN_SHARDED works, otherwise it tests the patterns in ALL_DEFAULT_PATTERNS. z{split}*rIc3b"TF%p\P!\V4'*xK' R#5ir8)rematchr)r9rIs& r$r;+_get_data_files_patterns..s!Fverxx 5111vs-/TFzSplit name should match 'z '' but got 'z'.)rIzCouldn't resolve pattern z with resolver )ALL_SPLIT_PATTERNSreplaceFileNotFoundErrorrMrPrraddr=rOrDEFAULT_SPLITSr2sortedr4ALL_DEFAULT_PATTERNSrKappend) rh split_patternr/ data_filesrTpp_partsrI sorted_splits patterns_dictnon_empty_splitsrAs & r$_get_data_files_patternsr~s,+ '' 37 )'2J z?Q "uF(1y7OP*** 77+, sFvFsssFvFFF #>MZ Z%,(. ,224OE#!1'!:Jz?Q&$++E2$ 5  =MN=ME=//=MN N. 7yP`Oab cc;!   XAZ) OsAG*G=,G= H6H?H H* G:9G: H H c V^8dQhR\R\R\\\,,R\\,R\\,/#)r.r/ base_pathallowed_extensionsdownload_configr0)r2rrDr)r4s"r$r5r5-sSgg gg!c+gn- g  #Y gr#c a\V4'd \W4pMO\V4'd=\PP V4^,\P ,pMRp\WR7wr\V3/VBwrV\\4\V40, pRV9dVPR4^,M>\VP\4'd VPMVP^,pVR8wd VR,MRp /p VR8XdRV R&VPR4vr.p VP !V3R R /V BP#4FwrVR ,R8XgZVP%R 4'dS\PP'\PP)V44'd\V4V9dK~\+W4'dK\-W4'dKRV9dTMW,pV 'dRP/V.V ,4pV P1V4K SeV Uu.uFp\2;QJd>V3R l\V4PR4R,4F 'gK R M3 RM/!V3R l\V4PR4R,44'gKVNK pp\5V4\5V 48d@\7\V 4\V4, 4p\8P;RV RV 24MT pV'g+RV R2pSeVR\7S4 2, p\=V4hV#uupi)aY Resolve the paths and URLs of the data files from the pattern passed by the user. You can use patterns to resolve multiple local files. Here are a few examples: - *.csv to match all the CSV files at the first level - **.csv to match all the CSV files at any level - data/* to match all the files inside "data" - data/** to match all the files inside "data" and its subdirectories The patterns are resolved using the fsspec glob. In fsspec>=2023.12.0 this is equivalent to Python's glob.glob, Path.glob, Path.match and fnmatch where ** is unsupported with a prefix/suffix other than a forward slash /. More generally: - '*' matches any character except a forward-slash (to match just the file or directory name) - '**' matches any character including a forward-slash / Hidden files and directories (i.e. whose names start with a dot) are ignored, unless they are explicitly requested. The same applies to special directories that start with a double underscore like "__pycache__". You can still include one if the pattern explicitly mentions it: - to include a hidden file: "*/.hidden.txt" or "*/.*" - to include a hidden directory: ".hidden/*" or ".*/*" - to include a special directory: "__special__/*" or "__*/*" Example:: >>> from datasets.data_files import resolve_pattern >>> base_path = "." >>> resolve_pattern("docs/**/*.py", base_path) [/Users/mariosasko/Desktop/projects/datasets/docs/source/_config.py'] Args: pattern (str): Unix pattern or paths or URLs of the data files to resolve. The paths can be absolute or relative to base_path. Remote filesystems using fsspec are supported, e.g. with the hf:// protocol. base_path (str): Base path to use when resolving relative paths. allowed_extensions (Optional[list], optional): White-list of file extensions to use. Defaults to None (all extensions). For example: allowed_extensions=[".csv", ".json", ".txt", ".parquet"] download_config ([`DownloadConfig`], *optional*): Specific download configuration parameters. Returns: List[str]: List of paths or URLs to the local or remote files that match the patterns. rz://filehfF expand_infoz::detailTtypeislinkc3<<"TFpRV,S9xK R#5i)rcNr)r9suffixrs& r$r;"resolve_pattern..sgDf&3<#55Dfsrc:NNz Some files matched the pattern 'z-' but don't have valid data file extensions: zUnable to find ''z with any supported extension )rrrosrJ splitdriver+rr rPFILES_TO_IGNORErrIrGprotocolr2globrKrNisfilerealpathr`rfjoinrvr=rMrDloggerinforq)r/rrrstorage_optionsfs fs_patternfiles_to_ignorerprotocol_prefix glob_kwargs_ rest_hops matched_pathsfilepathroutinvalid_matched_files error_msgs&&f& r$resolve_patternr-s`   + w  GG&&w/2RVV;  @jGw:/:NB/*i.@-AAO G   eQ' S99bkkr{{1~ +3f*] ^_eiijs :cRV^8dQhR\R\\,R\/#)r. data_filerr0)r2rrSingleOriginMetadata)r4s"r$r5r5s+n-r#cVP\P4'd\\PVPR7pRV\ \P4^,R,pTP RVPR4'dRMR^4pTpM\WR7wr\V3/VBwr#\V\4'd<VPV4p\VR4'dVPVP3#VPV4pR FpWv9gK \Wg,43u# R #) )endpointtokenhf://Nz /resolve/z hf://buckets//@rrevision)ETagetagmtimer)r\r HF_ENDPOINTr rrMrprr rG resolve_pathhasattrrepo_idrrr2)rrrfs_pathr resolved_pathrrRs&& r$_get_single_origin_metadatarsF..// 6#5#5_=R=R SiF,>,>(?!(C(EFF %%k):N:N:_:_3ehjkl %Fy%r"  =_= "l##0 =* - - ((-*@*@@ @ 777 D( ; N$ $) Ir#c V^8dQhR\\,R\\,R\\,R\\ ,/#)r.rxr max_workersr0)rDr2rrintr)r4s"r$r5r5 sCS n-#  r#c VeTM\Pp\;QJdRV4F 'dK RM RM !RV44'd?\TR\ V4^8*;'gRR7Uu.uFp\ W1R7NK up#\ \\ VR7TT\R\ V4^8*;'gRR7#uupi)Nc3,"TF pRV9xK R#5i)rNr)r9rs& r$r;'_get_origin_metadata..s <I7i sFTzResolving data files)descdisabler)r tqdm_classrr)r&HF_DATASETS_MULTITHREADING_MAX_WORKERSallhf_tqdmrMrr r)rxrrrs&&& r$_get_origin_metadatar s "-!8+f>k>kK s < P!  r#c <V^8dQhRS[S[,RS[S[,RS[S[S[,,RS[S[,RR/#r.rArrrr0rBrDr2rr)r4rs"r$r5rOsS   s)  C=  %T#Y/  ".1    r#cVeTM%\4P4P4pVPWW4R7#)Nr)rresolveas_posixr)rrArrrs&&&&&r$from_local_or_remote"DataFilesList.from_local_or_remoteNs@"+!6IDFNNP!  r#c <V^8dQhRS[S[,RS[S[,RS[S[S[,,RS[S[,RR/#rr)r4rs"r$r5r\sS00s)0C=0%T#Y/ 0 ".1 0  0r#c VeTM%\4P4P4p.pVF"pVP\ VVVVR74K$ \WTR7pV!WW4# \ d\ T4'ghKZi;iNrr)rrrextendrrqrr)rrArrrrxr/rs&&&&& r$rDataFilesList.from_patterns[s"+!6IDFNN.zs"HZc299S>>Z#%z.*(z )(\..+)?$c3N"TFp\P!V4xK R#5ir8r)r9fns& r$r;r}s!E*B"))B--*rz.*[\/]?(z)$c3D<"TFqPS4xK R#5ir8)rm)r9r/rs& r$r;rs7i`hU\ i8P8P`hs TFr)rrvrlcompilerBr=rrD)rrrrA ext_pattern fn_patternrs&$$ `r$filterDataFilesList.filterus (("HZ"HHK OOBJJ[M'DE F !E*!EEJ OOBJJ(:,b'AB C  ,0jDyCC7i`h7iCCC7i`h7i4iDj $ 4 4  !dT=Q=QR R ks(D(8D(D(/D(r NNN)rrr r!__doc__rr classmethodrrrr r"__classdictcell__ __classcell__rrs@@r$rBrB's"//\\          0002S37S[_SSSr#rBca]tRtRtoRt]RV3RlRll4t]RV3RlRll4t]RV3RlR ll4tR RR R/V3R lR llt Rt Vt R#) DataFilesDictias Dict of split_name -> list of data files (absolute local paths or URLs). It has two construction methods given the user's data files patterns : - ``from_hf_repo``: resolve patterns inside a dataset repository - ``from_local_or_remote``: resolve patterns from a local path Moreover, each list is a DataFilesList. It is possible to hash the dictionary and get a different hash if and only if at least one file changed. For more info, see [`DataFilesList`]. This is useful for caching Dataset objects that are obtained from a list of data files. Changing the order of the keys of this dictionary also doesn't change its hash. Nc <V^8dQhRS[S[S[S[S[,S[3,3,RS[S[,RS[S[S[,,RS[S[,RR/#r.rArrrr0rrCr2r rDrBrr)r4rs"r$r5DataFilesDict.__annotate__gsE$s)]":;;<C=%T#Y/  ".1   r#cV!4pVP4F8wrg\V\4'dTM\PVVVVR7WV&K: V#r)rKrGrBrrrArrrrrRpatterns_for_keys&&&&& r$r"DataFilesDict.from_local_or_remotesae%-^^%5 !C. >>!"77$''9$3 8 H&6 r#c <V^8dQhRS[S[S[S[S[,S[3,3,RS[P PRS[S[,RS[S[S[,,RS[S[ ,RR/#)r.rArrrrr0r) rCr2r rDrBrrrrr)r4rs"r$r5rs}sE$s)]":;;<&,,88C=  %T#Y/  ".1  r#c V!4pVP4F9wrx\V\4'dTM\PVVVVVR7Wg&K; V#))rrrr)rKrGrBr) rrArrrrrrRrs &&&&&& r$rDataFilesDict.from_hf_reposde%-^^%5 !C. >>!"//$!-''9$3 0 H&6 r#c <V^8dQhRS[S[S[S[S[,S[3,3,RS[S[,RS[S[S[,,RS[S[,RR/#rr)r4rs"r$r5rrr#cV!4pVP4F8wrg\V\4'dTM\PVVVVR7WV&K: V#r)rKrGrBrrs&&&&& r$rDataFilesDict.from_patternssae%-^^%5 !C. >>!"00$''9$3 1 H&6 r#rrcj<V^8dQhRS[S[S[,,RS[S[S[,,RR/#)r.rrr0rr)r4rs"r$r5rs8%d3i0EMdSViEX r#c\V4!4pVP4FwrEVPWR7W4&K V#))rr)rrKr )rrrrrRdata_files_lists&$$ r$r DataFilesDict.filters<4jl$(JJL C&---[CH%1 r#rr) rrr r!rrrrrr r"rrs@r$rrst *.*37[_r#rcaa]tRtRtoRtV3RlV3RlltRt]R V3RlRll4tR V3RlR llt V3R lR lt R t Vt V;t #)DataFilesPatternsListiz List of data files patterns (absolute local paths or URLs). For each pattern there should also be a list of allowed extensions to keep, or a None ot keep all the files for the pattern. cf<V^8dQhRS[S[,RS[S[S[S[,,,/#)r.rArrDr2r)r4rs"r$r5"DataFilesPatternsList.__annotate__s155s)5!$s)!455r#c2<\SV`V4W nR#r8)rrr)rrArrs&&&r$rDataFilesPatternsList.__init__s ""4r#cX\.VOVOVPVP,4#r8)rBrrs&&r$rDataFilesPatternsList.__add__s(_t_e_d.E.EH`H`.`aar#cZ<V^8dQhRS[S[,RS[S[S[,,RR/#)r.rArr0r-r/)r4rs"r$r5r0s7CCCyC6>tCy6IC Cr#c4V!W.\V4,4#r8)rM)rrArs&&&r$r#DataFilesPatternsList.from_patternss81CMABBr#c:<V^8dQhRS[RS[S[,RR/#)r.rrr0rBr2rr)r4rs"r$r5r0s.:::".1:  :r#c NVeTM%\4P4P4p.p\WP4F$wrEVP \ VVVVR74K& \W2R7p\W64# \d\T4'ghK_i;ir) rrrziprrrrqrrrB)rrrrxr/rrs&&& r$rDataFilesPatternsList.resolves "+!6IDFNN list of data files patterns (absolute local paths or URLs). Ncp<V^8dQhRS[S[S[S[,3,RS[S[S[,,RR/#)r.rArr0rD)rCr2rDr)r4rs"r$r5"DataFilesPatternsDict.__annotate__ s;  CcN+ AI$s)AT   r#cV!4pVP4F6wrE\V\4'dTM\PVVR7W4&K8 V#)rB)rKrGr-r)rrArrrRrs&&& r$r#DataFilesPatternsDict.from_patternss\e%-^^%5 !C.0EFF!*88$'99 H&6 r#c:<V^8dQhRS[RS[S[,RR/#)r.rrr0rr9)r4rs"r$r5rF/s.".1  r#cr\4pVP4FwrEVPW4W4&K V#r8)rrKr)rrrrrRdata_files_patterns_lists&&& r$rDataFilesPatternsDict.resolve/s5 o-1ZZ\ )C/77 SCH.: r#c4<V^8dQhRS[S[,RR/#)r.rr0rDr>)r4rs"r$r5rF9s DI:Qr#c~\V4!4pVP4Fwr4VPV4W#&K V#r8)rrKr@)rrrrRrKs&& r$r@'DataFilesPatternsDict.filter_extensions9s74jl-1ZZ\ )C/AA*MCH.: r#rr8) rrr r!rrrrr@r"rr+s@r$rDrDs7   r#rDr) validationvaliddevval)testtestingeval evaluation)z {keyword}/**z{keyword}[{sep}]*/**z**[{sep}/]{keyword}/**z**[{sep}/]{keyword}[{sep}]*/**)z{keyword}/**/*z{keyword}[{sep}]*/**/*z**/*[{sep}/]{keyword}/**/*z"**/*[{sep}/]{keyword}[{sep}]*/**/*)z**/{keyword}/**z**/{keyword}[{sep}]*/**z**/*[{sep}]{keyword}/**z**/*[{sep}]{keyword}[{sep}]*/**)z README.mdz config.jsonzdataset_info.jsonzdataset_infos.jsonzdummy_data.zipzdataset_dict.json)NNr8)Vrrl functoolsrrrpathlibrrtypingrrr r fsspec.corer r packagingr tqdm.contrib.concurrentr rrdownloadrnamingrrTrutilsrrrutils.file_utilsrrrrrutils.py_utilsrtupler2rTRAINrL get_loggerrrrrqr'SPLIT_PATTERN_SHARDED VALIDATIONTESTSPLIT_KEYWORDSNON_WORDS_CHARSFSSPEC_VERSIONparse"KEYWORDS_IN_FILENAME_BASE_PATTERNS"KEYWORDS_IN_DIR_NAME_BASE_PATTERNSrsr4"DEFAULT_PATTERNS_SPLIT_IN_FILENAME"DEFAULT_PATTERNS_SPLIT_IN_DIR_NAMEDEFAULT_PATTERNS_ALLDEFAULT_PATTERNS_LOGSrorur>rr?rQr`rfr~rrrrrDrBrCrr-rD)rIr*r/s000r$rss' ",,!(.$"rr*U38_eCj%)CDekk*   H % #  ) a KK':& ; JJ9  7==44*GI\)]&*&  W]];77*IK^)_&*&+AB`)a&*&++u//< &    %e, ,G9G wO<9 =,   &" &    %e, ,G9G wO<9 =,   &" KK$ +/+,&&  \&1RSB;Q|)dXgTXjv0:^SDI^SBZDm+,Zz2 DI2 j#D&;!;<#] & &s0 K%K KK)%KK KK