+
QV-j+ * ^ RI t ^ RIHt ^ RIHtHtHt ^ RIt^ RI H
t ^ RI H
t ^ RIHt ^ RIt^ RIt^ RIHt ^ RIHt ]P. P0 P3 ]4 t] ! R R]P8 4 4 t ! R R
]P< 4 t R# ) N) dataclass)LiteralOptionalUnion)version)Key)
table_castc Z a a ] tR t^t oRtRtRtRtRtRt Rt
V 3R lt V3R lt Rt
VtV ;t# )
ParquetConfiga
BuilderConfig for Parquet.
Args:
batch_size (`int`, *optional*):
Size of the RecordBatches to iterate on.
The default is the row group size (defined by the first row group).
columns (`list[str]`, *optional*)
List of columns to load, the other ones are ignored.
All columns are loaded by default.
features: (`Features`, *optional*):
Cast the data to `features`.
filters (`Union[pyarrow.dataset.Expression, list[tuple], list[list[tuple]]]`, *optional*):
Return only the rows matching the filter.
If possible the predicate will be pushed down to exploit the partition information
or internal metadata found in the data source, e.g. Parquet statistics.
Otherwise filters the loaded RecordBatches before yielding them.
fragment_scan_options (`pyarrow.dataset.ParquetFragmentScanOptions`, *optional*)
Scan-specific options for Parquet fragments.
This is especially useful to configure buffering and caching.
on_bad_files (`Literal["error", "warn", "skip"]`, *optional*, defaults to "error")
Specify what to do upon encountering a bad file (a file that can't be read). Allowed values are :
* 'error', raise an Exception when a bad file is encountered.
* 'warn', raise a warning when a bad file is encountered and skip that file.
* 'skip', skip bad files without raising or warning when they are encountered.
Example:
Load a subset of columns:
```python
>>> ds = load_dataset(parquet_dataset_id, columns=["col_0", "col_1"])
```
Stream data and efficiently filter data, possibly skipping entire files or row groups:
```python
>>> filters = [("col_0", "==", 0)]
>>> ds = load_dataset(parquet_dataset_id, streaming=True, filters=filters)
```
Increase the minimum request size when streaming from 32MiB (default) to 128MiB and enable prefetching:
```python
>>> import pyarrow
>>> import pyarrow.dataset
>>> fragment_scan_options = pyarrow.dataset.ParquetFragmentScanOptions(
... cache_options=pyarrow.CacheOptions(
... prefetch_limit=1,
... range_size_limit=128 << 20
... ),
... )
>>> ds = load_dataset(parquet_dataset_id, streaming=True, fragment_scan_options=fragment_scan_options)
```
Nerrorc $ < \ SV ` 4 R # N)super
__post_init__)self __class__s &z/Volumes/fast/ai/experiments/ui-tars-smoke/.venv/lib/python3.14/site-packages/datasets/packaged_modules/parquet/parquet.pyr ParquetConfig.__post_init__Y s
c V < V ^8 d Qh/ S[ S[, ;R& S[ S[S[, , ;R& S[ S[P
, ;R& S[ S[S[P S[S[ , S[S[S[ , , 3, , ;R& S[ S[P , ;R& S[ R, ;R& # )
batch_sizecolumnsfeaturesfiltersfragment_scan_options on_bad_files)r warnskip) r intliststrdatasetsFeaturesr ds
ExpressiontupleParquetFragmentScanOptionsr )format
__classdict__s "r __annotate__ParquetConfig.__annotate__ s ~
$ @ d3i
'A B x(()0C D eBMM4;T%[8I IJ
KRE F $B$A$ABIG H 12 Parquet._info..g s R>Qsdkk 2 22>Qs %(TFz6The columns and features argument must match, but got z and )r ) r; r r setany
ValueErrorr# r$ DatasetInfo)r r= r s f r _info
Parquet._infoa s KK + $$0DKK''(C 0D0D,EEsRdkk>Q>QRsssRdkk>Q>QRRR L{{**+51E1E0FG
$,,X\XcXcXkXk-lXkQT4;;3G3G3L.LXk-lm{{++H##X66 .ms >&Fc
V P P ' g# \ RV P P 24 hRVP n VP V P P 4 p. pVP
4 F w rEV P P fk V Fd p \ VR4 ;_uu_ 4 p\ P P \ P ! V4 4 V P n RRR4 M V P P f# \ R
V P P 24 hVP1 \ P2 ! VRVRR.\5 V4 , /R7 4 K V P P6 e \9 V P P6 4 \9 V P P 4 8w dx \ P ! V P P P
4 U U
u/ uF# w rWP P6 9 g K! WbK% up
p 4 V P n V# + ' g i EK ; i \ P d pT P P" R8X d3 \$ P' RT R\) T4 P* RT 24 h T P P" R 8X d: \$ P- R
T R \) T4 P* RT R 24 Rp?EK \$ P/ R
T R \) T4 P* RT R 24 Rp?EK Rp?ii ; iu up
p i )z-We handle string, list and dicts in datafilesz=At least one data file must be specified, but got data_files=TNrbr z Failed to read schema from '
' with error : r zSkipping bad schema from ''. `zPAt least one valid data file must be specified, all the data_files are invalid: filesrow_groups_list)name
gen_kwargs) r;
data_filesrB download_configextract_on_the_flydownloaditemsinfor openr# r$ from_arrow_schemapq read_schemapa ArrowInvalidr loggerr typer. warningdebugappendSplitGeneratorlenr r@ ) r
dl_managerrP splits
split_namerL filefer= feats && r _split_generatorsParquet._split_generatorsr s {{%%% \]a]h]h]s]s\t uv v8<
""5 (()?)?@
!+!1!1!3 Jyy!!) !D i !$ - -191B1B1T1TUWUcUcdeUf1gDII . ! . - " yy!!) fgkgrgrg}g}f~
MM''#%ARUYTZ]`af]gTg0h
) "42 ;; *s4;;3F3F/G3tyyOaOaKb/b!)!2!2,0II,>,>,D,D,Fe,Fys#Q\Q\QdQdJd,Fe"DII
1 . - - ?? i ;;33w> "LL+Gv][_`a[b[k[kZllnopnq)r s ! ![[ 5 5 ? "NN-GvSQUVWQXQaQaPbbdefdggh+i j j "LL+EdV3tTUwO_O_N``bcdbeef)g h hi& fsJ *I/>AI I/ M&
;M&
I,
%I/,I//M#BM %2M M#c N < V ^8 d QhRS[ P RS[ P /# )r pa_tablereturn)rZ Table)r) r* s "r r+ Parquet.__annotate__ s# BHH r c V P P e* \ WP P P 4 pV# r )rU r r arrow_schema)r rm s &&r _cast_tableParquet._cast_table s3 99 ) "(II,>,>,K,KLHr c # r " V' g
T R j xL
R # \ W4 F
w r4RVRV/x K R # L$5i)Nfragment_data_filefragment_row_groups)zip)r rL rM rf
row_groupss &&& r _generate_shardsParquet._generate_shards s= $'$? ($): %@
s 75%7c
# B " V' d; \ ;QJ d R V 4 F ' g K RM RM
! R V 4 4 ' d \ P ! V P P R7 pV Fn p\ VR4 ;_uu_ 4 pVP
V4 pRV.VP , R\ VP 4 Uu. uF qw3NK up/x RRR4 Kp R# \ W4 F w rHRV.RV./x K R# u upi + ' g i K ; i5i)c 3 ( " T F qR J x K
R # 5ir r- )r< row_groups & r r> 4Parquet._generate_more_gen_kwargs.. s %Y I4&7 s TF default_fragment_scan_optionsrG rL rM N)
rA r% ParquetFileFormatr; r rV
make_fragmentnum_row_groupsrangerx ) r rL rM parquet_file_formatrf rg parquet_fragment row_group_idry s &&& r _generate_more_gen_kwargs!Parquet._generate_more_gen_kwargs s ##%Y %Y###%Y %Y"Y"Y"$"6"6UYU`U`UvUv"w $%%':'H'H'K$ $*:*I*I!I)@EFVFeFe@f,@f O@f, &% %($? (9J<HH %@ , &%%s@ D D D AD >D DD .D D D D c
# " V P P e V P P e \ R V P P P
4 4 \ V P P 4 8w d; \
RV P P RV P P R24 h\ V P P \ 4 ' d+ \ P ! V P P 4 MV P P p\ P ! V P P R7 p\ \! W4 4 EFk w pw rg \# VR4 ;_uu_ 4 pVP% V4 p Rp
Ve V P' VR7 p V P( ' d V P P* ;' g V P( ^ , P, p \ V P/ V V P P V^ ^ R 7 4 FC w r\0 P2 P5 V
.4 p\7 W\4 V P9 V4 3x KE R
p
V
' gJ \: P P< \> P@ ! R 4 8: d ? \B PD ! 4 R R R 4 EKn R # T
' gL \: P P< \> P@ ! R 4 8: d ? \B PD ! 4 i i i ; i + ' g i EK ; i \0 PF \ 3 d pT P PH R 8X d3 \J PM R
T R\O T4 PP RT 24 h T P PH R8X d: \J PS RT R\O T4 PP RT R24 R p?EK \J PU RT R\O T4 PP RT R24 R p?EK R p?ii ; i5i)Nc 3 8 " T F qP x K R # 5ir )rN )r< fields & r r> +Parquet._generate_tables.. s N.MUjj.Ms z)Tried to load parquet data with columns 'z ' with mismatching features ''r rG F)
row_group_ids)r r filterbatch_readaheadfragment_readaheadTz24.0.0r zFailed to read file 'rH rI r zSkipping bad file 'rJ rK )+r; r r sortedrU rr rB
isinstancer r! rX filters_to_expressionr% r r enumeraterx rV r subsetry r num_rows
to_batchesrZ ro from_batchesr rs r# PYARROW_VERSIONr parsegccollectr[ r r\ r r] r. r^ r_ )r rL rM filter_exprr file_idxrf ry rg r fragment_is_closedr batch_idx record_batchrm rh s &&& r _generate_tablesParquet._generate_tables s} ;; + 0C0C0ONdii.@.@.M.MNNRXY]YdYdYlYlRmm ? @S@S?TTqrvr{r{ sE sE rF FG H
$++--t44
$ $T[[%8%8 9$$
! 2 2QUQ\Q\QrQr s,5c%6Q,R (H(t$
Z$%%':'H'H'K$).&)%1/?/F/FU_/F/` ,+666)-)?)?)j)jCSC^C^_`CaCjCjJ;D 0 ; ;/9,0KK,?,?+64578 !< !"< 7 ,.88+@+@, +P '*(&>@P@PQY@Z&Z Z< 26 . 2hoo6U6UY`YfYfgoYp6p 0 JJL7 &% -S6 2hoo6U6UY`YfYfgoYp6p 0 JJL 7q 13 &%%8 OOZ0
Z;;++w6LL#8mDQRGL\L\K]]_`a_b!cd[[--7NN%8c$q'BRBRASSUVWUXXY#Z[[LL#6tfCQ@P@P?QQSTUSVVW!XYY
Zs E/Q2ML2'K K BK:AL2 MQAL/ /L22 M =MQMQ"BP<<Q2P<5Q<QQr- N)r. r/ r0 r1 r BUILDER_CONFIG_CLASSSLEEP_ON_THREADS_SHUTDOWNSrD rj rs rz r r r4 r5 )r* s @r r8 r8 ] s= (!% 7"$L I 1Z 1Zr r8 ) r dataclassesr typingr r r pyarrowrZ pyarrow.datasetdatasetr% pyarrow.parquetparquetrX packagingr r# datasets.configdatasets.builderr datasets.tabler utilslogging
get_loggerr. r\
BuilderConfigr ArrowBasedBuilderr8 r- r r r s ! + + %
* *8 4 F H** F F RMZh(( MZr