+
QV-jN R t ^ RIt^ RIt^ RIt^ RIt^ RIt^ RIHt ^ RIHt ^ RI H
t
H t H t ^ RI
t
^ RIHt ^ RIHtHt ^RIHt ^RIHt ^R IHt ^R
IHt ^R IH t ^R I H t H t ] ! ] 4 t!] ! R
R4 4 t"] ! R R4 4 t# ! R R]$4 t% ! R R]$4 t&] ! R R4 4 t'] ! R R4 4 t( ! R R])]*](3, 4 t+R# )aw DatasetInfo record information we know about a dataset.
This includes things that we know about the dataset statically, i.e.:
- description
- canonical location
- does it have validation and tests splits
- size
- etc.
This also includes the things that can and should be computed once we've
processed the dataset as well:
- number of examples (in each split)
- etc.
N) dataclass)Path)ClassVarOptionalUnion) url_to_fs) DatasetCardDatasetCardData)config)Features) SplitDict)Version)
get_logger)asdict
unique_valuesc 4 a ] tR t^7t o RtRtV 3R ltRtV tR# )SupervisedKeysData c 2 < V ^8 d Qh/ S[ ;R& S[ ;R& # ) inputoutputstr)format
__classdict__s "^/Volumes/fast/ai/experiments/ui-tars-smoke/.venv/lib/python3.14/site-packages/datasets/info.py __annotate__ SupervisedKeysData.__annotate__7 s O
N) __name__
__module__ __qualname____firstlineno__r r __annotate_func____static_attributes____classdictcell__r s @r r r 7 s EF r r c 4 a ] tR t^=t o RtRtV 3R ltRtV tR# )DownloadChecksumsEntryDatar c 2 < V ^8 d Qh/ S[ ;R& S[ ;R& # )r keyvaluer )r r s "r r 'DownloadChecksumsEntryData.__annotate__= s M O r r N) r! r" r# r$ r, r- r% r& r' r( s @r r* r* = s CE r r* c ] tR t^CtRtRtR# ) MissingCachedSizesConfigErrorz;The expected cached sizes of the download file are missing.r Nr! r" r# r$ __doc__r& r r r r0 r0 C s Er r0 c ] tR t^GtRtRtR# )NonMatchingCachedSizesErrorz/The prepared split doesn't have expected sizes.r Nr1 r r r r4 r4 G s 9r r4 c V a ] tR t^Kt o RtRtR t]V 3R lR l4 tV 3R lt Rt
V t R# )PostProcessedInfoNc V P eJ \ V P \ 4 ' g( \ P ! V P 4 V n R # R # R # N)features
isinstancer from_dictselfs &r
__post_init__ PostProcessedInfo.__post_init__P s; == $Z
x-P-P $ . .t}} =DM .Q $r c $ < V ^8 d QhRS[ RR/# )r post_processed_info_dictreturnr6 dict)r r s "r r PostProcessedInfo.__annotate__V s ^ ^ ^:M ^r c
\ P ! V 4 Uu0 uF q"P kK ppT ! R/ VP 4 UUu/ uF w rEWC9 g K
WEbK uppB # u upi u uppi Nr dataclassesfieldsnameitems)clsrA f field_nameskvs && r r; PostProcessedInfo.from_dictU sb '2'9'9#'>?'>!vv'> ?]'?'E'E'G\'Gtq1K[dad'G\]] @\ A!
A&A&c R < V ^8 d Qh/ S[ S[, ;R& S[ S[, ;R& # )r r9 resources_checksums)r r rD )r r s "r r rE K s& x ' "$ . r )r9 ) r! r" r# r$ r9 rU r> classmethodr; r% r& r' r( s @r r6 r6 K s2 #'H*.>
^ ^ r r6 c a ] tR t^[t o Rt]P ! ]R7 t]P ! ]R7 t ]P ! ]R7 t
]P ! ]R7 t Rt Rt
RtRtRtRtRtRtRtRtRtRtRt. ROtR tRV 3R lR lltR R lt R t ] V 3R lR
l4 t ] R V 3R lR ll4 t ] V 3R
lR l4 t!R V 3R lR llt"V 3R lR lt#V 3R lR lt$] V 3R lR l4 t%R t&V 3R lt'Rt(V t)R# ) DatasetInfoa4 Information about a dataset.
`DatasetInfo` documents datasets, including its name, version, and features.
See the constructor arguments and properties for a full list.
Not all fields are known on construction and may be updated later.
Attributes:
description (`str`):
A description of the dataset.
citation (`str`):
A BibTeX citation of the dataset.
homepage (`str`):
A URL to the official homepage for the dataset.
license (`str`):
The dataset's license. It can be the name of the license or a paragraph containing the terms of the license.
features ([`Features`], *optional*):
The features used to specify the dataset's column types.
post_processed (`PostProcessedInfo`, *optional*):
Deprecated. Information regarding the resources of a possible post-processing of a dataset. For example, it can contain the information of an index.
supervised_keys (`SupervisedKeysData`, *optional*):
Specifies the input feature and the label for supervised learning if applicable for the dataset (legacy from TFDS).
builder_name (`str`, *optional*):
The name of the `GeneratorBasedBuilder` subclass used to create the dataset. It is also the snake_case version of the dataset builder class name.
config_name (`str`, *optional*):
The name of the configuration derived from [`BuilderConfig`].
version (`str` or [`Version`], *optional*):
The version of the dataset.
splits (`dict`, *optional*):
The mapping between split name and metadata.
download_checksums (`dict`, *optional*):
The mapping between the URL to download the dataset's checksums and corresponding metadata.
download_size (`int`, *optional*):
The size of the files to download to generate the dataset, in bytes.
post_processing_size (`int`, *optional*):
Deprecated. Size of the dataset in bytes after post-processing, if any.
dataset_size (`int`, *optional*):
The combined size in bytes of the Arrow tables for all splits.
size_in_bytes (`int`, *optional*):
The combined size in bytes of all files associated with the dataset (downloaded files + Arrow files).
**config_kwargs (additional keyword arguments):
Keyword arguments to be passed to the [`BuilderConfig`] and used in the [`DatasetBuilder`].
)default_factoryNc * V P eF \ V P \ 4 ' g& \ P ! V P 4 V n V P eE \ V P \
4 ' g% \
P V P 4 V n V P e \ V P \ 4 ' ga \ V P \ 4 ' d \ V P 4 V n M%\ P ! V P 4 V n V P eF \ V P \ 4 ' g& \ P ! V P 4 V n V P e \ V P \ 4 ' g] \ V P \ \ 34 ' d \ V P ! V n R # \ R/ V P B V n R # R # R # rG )r9 r: r r; post_processedr6 versionr
r splitsr from_split_dictsupervised_keysr tuplelistr< s &r r> DatasetInfo.__post_init__ sS == $Z
x-P-P $ . .t}} =DM *:d>Q>QSd3e3e"3"="=d>Q>Q"RD << #Jt||W,M,M$,,,, &t|| 4 & 0 0 > ;; ":dkk9+M+M#33DKK@DK +Jt?S?SUg4h4h$..
>>'94;O;O'P$'9'QD
Example:
```py
>>> from datasets import load_dataset
>>> ds = load_dataset("cornell-movie-review-data/rotten_tomatoes", split="validation")
>>> ds.info.write_to_directory("/path/to/directory/")
```
wb) pretty_printN)
r open posixpathjoinr
DATASET_INFO_FILENAME
_dump_infolicenseLICENSE_FILENAME
_dump_license)r= dataset_info_dirrh rd fs_rN s &&&& r write_to_directory DatasetInfo.write_to_directory s , +G0E0E2G
WWY^^$4f6R6RSUY
Z
Z^_ OOAO 9 [ <<< (8&:Q:QRTXYY]^""1% ZY [
Z ZYYs C ?C0 C- 0 D c TP \ P ! \ V 4 V' d ^MRR7 P R4 4 R# )zQDump info in `file` file-like object open in bytes mode (to support remote files)Nindentutf-8)writejsondumpsr encode)r= filerh s &&&r rm DatasetInfo._dump_info s-
4::fTl 1$OVVW^_`r c Z VP V P P R4 4 R# )zTDump license in `file` file-like object open in bytes mode (to support remote files)ry N)rz rn r} )r= r~ s &&r rp DatasetInfo._dump_license s
4<<&&w/0r c . < V ^8 d QhRS[ R, /# )r
dataset_infosrX )ra )r r s "r r re s
tM':
r c a S Uu. uF q"f K VP 4 NK upo\ S4 ^ 8 dJ \ ;QJ d V3R lS 4 F ' d K RM RM! V3R lS 4 4 ' d
S^ , # RP \ R S 4 4 4 P 4 pRP \ R S 4 4 4 P 4 pRP \ R S 4 4 4 P 4 pRP \ R S 4 4 4 P 4 pR pR pV ! VVVVVVR 7 # u upi )
Nc 3 < <" T F pS^ , V8H x K R# 5i) Nr ).0 dset_infor s & r )DatasetInfo.from_merge.. s )gYfI-*:i*GYfs FTz
c 3 8 " T F qP x K R # 5ir8 ) descriptionr infos & r r r s /[]T0@0@] c 3 8 " T F qP x K R # 5ir8 )citationr s & r r r ,U}t]]}r c 3 8 " T F qP x K R # 5ir8 )homepager s & r r r r r c 3 8 " T F qP x K R # 5ir8 )rn r s & r r r s +S]TLL]r )r r r rn r9 r_ )copylenallrk r strip) rM r r r r r rn r9 r_ s &f r
from_mergeDatasetInfo.from_merge s ;Hb=i) )=b
} !cc)gYf)gccc)gYf)g&g&g # # kk-/[]/["[\bbd ;;},U},U UV\\^;;},U},U UV\\^++m+S]+S STZZ\ # +
cs
EEc : < V ^8 d QhRS[ RS[S[, RR/# )r rq rd rB rX )r r rD )r r s "r r re s' 0 0c 0HTN 0^k 0r c \ V3/ T;' g / B vr4\ P RV 24 V' g \ R4 hVP \
P ! V\ P 4 RRR7 ;_uu_ 4 p\ P ! V4 pRRR4 V P X4 # + ' g i L!; i)a Create [`DatasetInfo`] from the JSON file in `dataset_info_dir`.
This function updates all the dynamically generated fields (num_examples,
hash, time of creation,...) of the [`DatasetInfo`].
This will overwrite all previous metadata.
Args:
dataset_info_dir (`str`):
The directory containing the metadata file. This
should be the root directory of a specific dataset version.
storage_options (`dict`, *optional*):
Key/value pairs to be passed on to the file-system backend, if any.
Example:
```py
>>> from datasets import DatasetInfo
>>> ds_info = DatasetInfo.from_directory("/path/to/directory/")
```
zLoading Dataset info from zECalling DatasetInfo.from_directory() with undefined dataset_info_dir.rry encodingN) r loggerdebug
ValueErrorri rj rk r
rl r{ loadr; )rM rq rd rr rs rN dataset_info_dicts &&& r from_directoryDatasetInfo.from_directory s 4 +G0E0E2G 12B1CDE de e
WWY^^$4f6R6RSUXcjW
k
kop $ ! l}} .// l
ks B22 C c $ < V ^8 d QhRS[ RR/# )r r rB rX rC )r r s "r r re s W W$ W= Wr c
\ P ! V 4 Uu0 uF q"P kK ppT ! R/ VP 4 UUu/ uF w rEWC9 g K
WEbK uppB # u upi u uppi rG rH )rM r rN rO rP rQ s && r r; DatasetInfo.from_dict sb '2'9'9#'>?'>!vv'> ?V'8'>'>'@U'@tqADTdad'@UVV @UrS c < V ^8 d QhRR/# )r other_dataset_inforX r )r r s "r r re s
r c V P pVP ! R/ VP P 4 UUu/ uF* w rEVf V' d K V\ P ! V4 bK, uppB R # u uppi rG )__dict__updaterL r deepcopy)r= r ignore_none self_dictrP rQ s &&& r r DatasetInfo.update sh MM
/ 7 7 = = ? ?DAM $4==## ?
s A0
A0c < V ^8 d QhRR/# )r rB rX r )r r s "r r re ( s Y Ym Yr c V P ! R/ V P P 4 UUu/ uF w rV\ P ! V4 bK uppB # u uppi rG ) __class__r rL r r )r= rP rQ s & r r DatasetInfo.copy( sE ~~X ATATAV WAVDMM!$4!4AV WXX Ws "Ac < V ^8 d QhRS[ /# )r rB rC )r r s "r r re + s t r c / p\ V 4 pV Fp pW0P 9 g K \ W4 p\ VR 4 ' d VP 4 W& KF \ VR4 ' d VP 4 W& Kl WAV& Kr V# )
_to_yaml_list_to_yaml_string)r _INCLUDED_INFO_IN_YAMLgetattrhasattrr r )r= yaml_dictr r, r- s & r
_to_yaml_dictDatasetInfo._to_yaml_dict+ s{ "4L$C111 *5/22%*%8%8%:IN U$566%*%:%:%?'>!vv'> ?Ny'8M'8tqA rt rm rp rV r r r; r r r r r r% r& r' r( s @r rX rX [ s< *Z #((=K %%c:H %%c:H $$S9G#'H26N48O #'L"&L!%K-1G"&F)-#'M*."&L#'M3R&& &:a1
. 0 0 0B W W
Y Y O O
Q r rX c a ] tR tRt o R V 3R lR llt]V 3R lR l4 t]V 3R lR l4 tV 3R lR ltR
t V t
R # )
DatasetInfosDictiN c < V ^8 d QhRR/# )r rB Nr )r r s "r r DatasetInfosDict.__annotate__O s 9 9\` 9r c
/ p\ P P V\ P 4 p\ P P V\ P
4 pV' g V P
V4 pVP V 4 \ P P V4 ' do \ VR RR7 ;_uu_ 4 pVP 4 UU u/ uF w rV\ V 4 bK p
pp \ P ! YV' d ^MRR7 RRR4 \ P P V4 ' d$ \ P ! V4 p V P p M Rp \# 4 p V' dV VP% V 4 V f# \ R\' V 4 , R, 4 MT p V P) \+ V4 4 R# R# u up pi + ' g i L; i)wry r Nrw z---
z
---
)ospathrk r
DATASETDICT_INFOS_FILENAMEREPOCARD_FILENAMEr r existsri rL r r{ dumpr r datar to_dataset_card_datar saver )
r= dataset_infos_dir overwriterh total_dataset_infosdataset_infos_pathdataset_readme_pathrN r r dataset_infos_dict dataset_carddataset_card_datas
&&&& r rt #DatasetInfosDict.write_to_directoryO sy WW\\*;V=^=^ _ ggll+V>V W"&"5"56G"H ""4(
77>>, - -(#@@AQdQjQjQl&Ql7M{K !22Ql # & ,\tT A 77>>- . .&++,?@L , 1 1 L / 1 4 45F GMYMa Gc*;&< >"'',,'8&:R:RS T T + 0 06G1H6KcKc1c d i i !22112CDD
77>>"'',,'8&:[:[\ ] ]bggll#4f6W6WXcjkkop ?Ciil>P>P>R>R :K $[%:%:;L%MM>R lk 5L
lkks *F<Tk; 7 7 99Xde#
5H4M4M4O0 5@M2 5P&'1,48>Q>X>X>Z9[4\!.1 / ? C CMSW X )+ &{9+N;9%n5 , 57!.1;ABUB[B[B];^7K*..}dC.;[-cLb-c*%n5<<=ST <_C * fs F/.F4r N)FF) r! r" r# r$ rt rV r r r r& r' r( s @r r r N sC 9 98 & $&U &Ur r ),r2 r rI r{ r rj r pathlibr typingr r r fsspec fsspec.corer huggingface_hubr r r r
r9 r r] r utilsr
utils.loggingr utils.py_utilsr r r! r r r* Exceptionr0 r4 r6 rX rD r r r r r r s
! , ,
! 8 % 1
H
FI F:) : ^ ^ ^ o
o
o
djUtC ,- jUr