+ QV-jUaR$t50t^RIt^RIt^RIt^RIt^RIt^RIt^RIHt^RI H t ^RI H t H t HtHtHt^RIt^RIt^RIHt^RIHt^RIHt^RIHt] 'd^R IHt]!]4tR s Rs!]^kRs"]^k!R R 4t#R t$Rt%Rt&Rt'RRlt(RRlt)!RR4t*]PV!4t,/s-]^kRRlt.R%RRllt/Rt0R%RRllt1R&RRllt2R'R R!llt3R(R"R#llt4R#))N)wraps)Path) TYPE_CHECKINGAnyCallableOptionalUnion)config)"INVALID_WINDOWS_CHARACTERS_IN_PATH)dumps) get_logger)DatasetTc6a]tRt^-toRtRtRtRtRtVt R#) _TempCacheDirz A temporary directory for storing cached Arrow files with a cleanup that frees references to the Arrow files before deleting the directory itself to avoid permission errors on Windows. c0\PPR4;'gG\PPR4;'g \PPR4pV'd\PP V4p\PP V4'g3\P !VRR7\PRV 24M4\PPV4'g\RV R 24h\P!\PVR 7Vn\ P"!WP$4VnR # \dp\RT RT R24ThR p?ii;i) TMPDIRTEMPTMPT)exist_okzCreated TMPDIR directory: zTMPDIR is set to 'z=' but the directory does not exist and could not be created: z\. Please create it manually or unset TMPDIR to fall back to the default temporary directory.Nz' but it is not a directory. Please point TMPDIR to a writable directory or unset it to fall back to the default temporary directory.)prefixdir)osenvirongetpathnormpathexistsmakedirsloggerinfoOSErrorisdirtempfilemkdtempr TEMP_CACHE_DIR_PREFIXnameweakreffinalize_cleanup _finalizer)selftmpdires& e/Volumes/fast/ai/experiments/ui-tars-smoke/.venv/lib/python3.14/site-packages/datasets/fingerprint.py__init___TempCacheDir.__init__3s8)\\RZZ^^F-C\\rzz~~V[G\ WW%%f-F77>>&))KK6KK"D LLN? 77>>$)) $ $  dii( % YZ^ZcZcYdeAB s A66 B BB cjVPP4'dVP4R#R#N)r*detachr)r+s&r.cleanup_TempCacheDir.cleanup\s# ?? ! ! # # MMO $)r*r&N) __name__ __module__ __qualname____firstlineno____doc__r/r)r=__static_attributes____classdictcell__ __classdict__s@r.rr-s @< r?rc\fR#\f\P!4s\;QJd&RVP 4F 'gK RM RM!RVP 44'd\P V4R#R#)a This function registers the datasets that have cache files in _TEMP_DIR_FOR_TEMP_CACHE_FILES in order to properly delete them before deleting the temporary directory. The temporary directory _TEMP_DIR_FOR_TEMP_CACHE_FILES is used when caching is disabled. Nc3"TF;p\\P4\VR,4P9xK= R#5i)filenameN)r_TEMP_DIR_FOR_TEMP_CACHE_FILESr&parents).0 cache_files& r. ?maybe_register_dataset_for_temp_dir_deletion..ms8 -J + 0 01T*Z:P5Q5Y5YY-sAATF)rL _DATASETS_WITH_TABLE_IN_TEMP_DIRr'WeakSetany cache_filesadd)datasets&r.,maybe_register_dataset_for_temp_dir_deletionrXasr &-(/+2??+<( s !-- sss !--  ),,W5 r?c4\e\\4#.#r:)rRlistr?r.r2r2ts5U5a40 1igiir?c RsR#) When applying transforms on a dataset, the data are stored in cache files. The caching mechanism allows to reload an existing cache file if it's already been computed. Reloading a dataset is possible since the cache files are named using the dataset fingerprint, which is updated after each transform. If disabled, the library will no longer reload cached datasets files when applying transforms to the datasets. More precisely, if the caching is disabled: - cache files are always recreated - cache files are written to a temporary directory that is deleted when session closes - cache files are named using a random hash instead of the dataset fingerprint - use [`~datasets.Dataset.save_to_disk`] to save a transformed dataset or it will be deleted when session closes - caching doesn't affect [`~datasets.load_dataset`]. If you want to regenerate a dataset from scratch you should use the `download_mode` parameter in [`~datasets.load_dataset`]. TN_CACHING_ENABLEDr[r?r.enable_cachingr`xs $r?c RsR#)r]FNr^r[r?r.disable_cachingrbs $r?c$V^8dQhR\/#return)bool)formats"r. __annotate__ris""D"r?c \\4#)a When applying transforms on a dataset, the data are stored in cache files. The caching mechanism allows to reload an existing cache file if it's already been computed. Reloading a dataset is possible since the cache files are named using the dataset fingerprint, which is updated after each transform. If disabled, the library will no longer reload cached datasets files when applying transforms to the datasets. More precisely, if the caching is disabled: - cache files are always recreated - cache files are written to a temporary directory that is deleted when session closes - cache files are named using a random hash instead of the dataset fingerprint - use [`~datasets.Dataset.save_to_disk`]] to save a transformed dataset or it will be deleted when session closes - caching doesn't affect [`~datasets.load_dataset`]. If you want to regenerate a dataset from scratch you should use the `download_mode` parameter in [`~datasets.load_dataset`]. )rgr_r[r?r.is_caching_enabledrks$  !!r?c$V^8dQhR\/#rdstr)rhs"r.riris//S/r?cF\f \4s\P#)z7Return a directory that is deleted when session closes.)rLrr&r[r?r.#get_temporary_cache_files_directoryrps&-)6& ) . ..r?ca]tRt^toRt/tRt]V3RlRl4t]V3RlRl4t V3RlRlt V3R lR lt V3R lt R t VtR #)Hasherz-Hasher that accepts python objects as inputs.c:\P!4VnR#r:)xxhashxxh64mr<s&r.r/Hasher.__init__sr?cL<V^8dQhRS[S[S[S[,3,RS[/#revaluerf)r bytesrZrn)rhrHs"r.riHasher.__annotate__s)uUDK%78Sr?c\V\4'dV.MTp\P!4pVFpVP V4K VP 4#r:) isinstancer{rtruupdate hexdigest)clsrzrvxs&& r. hash_bytesHasher.hash_bytessC%eU33 LLNA HHQK{{}r?c&<V^8dQhRS[RS[/#ry)rrn)rhrHs"r.rir|s,,,,r?c6VP\V44#r:)rr )rrzs&&r.hash Hasher.hashs~~eEl++r?c$<V^8dQhRS[RR/#)rerzrfN)r)rhrHs"r.rir|s88C8D8r?cR\V4 R2pVPV4pVPPVP R44VPPVP R44R#)z==utf8zutf-8N)typerrvrencode)r+rzheader_for_updatevalue_for_updates&& r.r Hasher.updates\ e R099U+  '..v67  &--g67r?c <V^8dQhRS[/#rdrm)rhrHs"r.rir|s""3"r?c6VPP4#r:)rvrr<s&r.rHasher.hexdigestsvv!!r?c&<V^8dQh/S[;R&#)redispatch)dict)rhrHs"r.rir|sr?)rvN)r@rArBrCrDrr/ classmethodrrrr__annotate_func__rErFrGs@r.rrrrsU7H ,,88 ""5r?rrc(V^8dQhRRR\/#)rerWrrfrm)rhs"r.riris  )  r?cdVPp\4p\V4F4pVR8XdK VPV4VPW,4K6 VPF8pVP\ P PVR,44K: VP4#) _fingerprintrK) __dict__rrsortedrrUrrgetmtimer)rWstatehasherkeyrOs& r.generate_fingerprintrs   E XFe} .  c ej!  ))  bgg&&z*'=>?*    r?c0V^8dQhR\R\/#)renbitsrf)intrn)rhs"r.ririsBBsBCBr?cF\PV4RV^, R2 #)0r)fingerprint_rng getrandbits)rs&r.generate_random_fingerprintrs$))%05A:,a @Ar?c z\4pVPV4VPV4\T4F,pTPT4TPY$,4K. TP4# \'dZ\P RR4'g$\ P RT R24R\R&M3\ PRT R24M\ PRT R24\4u#;i \'dr\P RR4'g0\ P RT RY$, R T R24R\R&MK\ PRT RY$, R T R24M%\ PRT RY$, R T R24\4uu#;i) (update_fingerprint_transform_hash_failedFz Transform a couldn't be hashed properly, a random hash was used instead. Make sure your transforms and parameters are serializable with pickle or dill for the dataset fingerprinting and caching to work. If you reuse this transform, the caching mechanism will consider it to be different from the previous calls and recompute everything. This warning is only shown once. Subsequent hashing failures won't be shown.Tz= couldn't be hashed properly, a random hash was used instead.zn couldn't be hashed properly, a random hash was used instead. This doesn't affect caching since it's disabled.z Parameter 'z'=z of the transform ) rrrr_fingerprint_warningsrrwarningr rrr) fingerprint transformtransform_argsrrs&&& r.update_fingerprintrs XF MM+- i &n% c 1 MM.- .&.    S-  '++,VX]^^  ,cc TX$%OP j 3pqr KKYK(VW +,,  1+//0Z\abbNN%cU"^-@,AAST]S^_gg X\()STKK%cU"^-@,AAST]S^_\] !#b)<(==OPY{[IJ/0 0s%A9D9)D#A D)F:2BF:c$V^8dQhR\/#)rerrm)rhs"r.riri/s  c r?c \V\4'd V'g\RV R24h\FpW 9gK \R\ RV R24h \ V4V8d\RV RV R\ V4 R 24hR #) z Make sure the fingerprint is a non-empty string that is not longer that max_length=64 by default, so that the fingerprint can be used to name cache files without issues. zInvalid fingerprint 'z#': it should be a non-empty string.z5Invalid fingerprint. Bad characters from black list 'z ' found in 'z6'. They could create issues when creating cache files.z&Invalid fingerprint. Maximum lenth is z but 'z ' has length z2.It could create issues when creating cache files.N)r~rn ValueErrorr len)r max_length invalid_chars&& r.validate_fingerprintr/s k3 ' '{0 =`abb:  &GHjGkkwyDxEEFG ;  ;*$4ZL{mS`adepaq`rs@ @  %r?cRV^8dQhR\R\\,R\/#)refuncversionrf)rrrn)rhs"r.ririCs&8hsmWZr?c\VP RVP 2pVe VRV 2, pV#)zO Format a transform to the format that will be used to update the fingerprint. .@)rArB)rrrs&& r. format_transform_for_fingerprintrCs<??#1T%6%6$78Iq ]" r?cV^8dQhR\R\R\\\3,R\ \ \,,R\ \ \,,R\R\\\3,/#)rerargskwargs use_kwargs ignore_kwargsrandomized_functionrf)rtuplerrnrrrZrg)rhs"r.ririMsp*"*" *" *" cN*"c# *" DI& *"  *" #s(^*"r?cVP4pV'd\P!V4PP 4Uu.uF!qwVP 8wgKVP NK# ppVR,pVR,pVP\W44M3V\\\P!V4P44V'd,VP4U U u/uFwrW9gK WbK pp p V'd,VP4U U u/uFwrW9gK WbK pp p V'dVPR4fnVPR4f[\PP4vrrV R8d W,MV ^,p \PP!V 4VR&\P!V4PP 4Uu/uF:qwP"\P$8wgK#VP VP"bK< ppVP4F.wppW9gKWo,V8XgKVP'V4K0 V#uupiuup p iuup p iuupi)z] Format the kwargs of a transform to the format that will be used to update the fingerprint. NNseed generatorip)copyinspect signature parametersvalues VAR_KEYWORDr&rzipnextiteritemsrnprandom get_state default_rngdefault_emptypop)rrrrrrkwargs_for_fingerprintpparamskv_rposdefault_valuesdefault_varname default_values&&&&&& r.format_kwargs_for_fingerprintrMs$[[] ")"3"3D"9"D"D"K"K"Md"MQVWVcVcQc&!&&"MdBx%%c&&78 " g''-889 : 3I3O3O3Q!e3Q41UVUd$!$3Q!e3I3O3O3Q!l3Q41UVUk$!$3Q!l ! % %f - 5:P:T:TU`:a:i!yy224 AS #c 49tAwD24))2G2G2M "; / $+#4#4T#:#E#E#L#L#N#NaR[R[_f_m_mRm #N+9*>*>*@&  49O9`dq9q " & & 7+A "!;e"f!ls0 I6 I6- I;<I; J/J0 JJc V^8dQhR\R\\\,,R\\\,,R\\\,,R\R\\,/#)reinplacerrfingerprint_namesrr)rgrrZrn)rhs"r.ririzskff fc#fDI&f S * f  f c] fr?c4aaaaaaSe.\S\4'g\R\S4 24hSe.\S\4'g\R\S4 24hS'dS'd \R4hSeSMR.oVVVVVV3RlpV#)aJ Wrapper for dataset transforms to update the dataset fingerprint using ``update_fingerprint`` Args: inplace (:obj:`bool`): If inplace is True, the fingerprint of the dataset is updated inplace. Otherwise, a parameter "new_fingerprint" is passed to the wrapped method that should take care of setting the fingerprint of the returned Dataset. use_kwargs (:obj:`List[str]`, optional): optional white list of argument names to take into account to update the fingerprint to the wrapped method that should take care of setting the fingerprint of the returned Dataset. By default all the arguments are used. ignore_kwargs (:obj:`List[str]`, optional): optional black list of argument names to take into account to update the fingerprint. Note that ignore_kwargs prevails on use_kwargs. fingerprint_names (:obj:`List[str]`, optional, defaults to ["new_fingerprint"]): If the dataset transforms is not inplace and returns a DatasetDict, then it can require several fingerprints (one per dataset in the DatasetDict). By specifying fingerprint_names, one fingerprint named after each element of fingerprint_names is going to be passed. randomized_function (:obj:`bool`, defaults to False): If the dataset transform is random and has optional parameters "seed" and "generator", then you can set randomized_function to True. This way, even if users set "seed" and "generator" to None, then the fingerprint is going to be randomly generated depending on numpy's current state. In this case, the generator is set to np.random.default_rng(np.random.get_state()[1][0]). version (:obj:`str`, optional): version of the transform. The version is taken into account when computing the fingerprint. If a datase transform changes (or at least if the output data that are cached changes), then one should increase the version. If the version stays the same, then old cached data could be reused that are not compatible with the new transform. It should be in the format "MAJOR.MINOR.PATCH". z)use_kwargs is supposed to be a list, not z,ignore_kwargs is supposed to be a list, not z5fingerprint_names are only used when inplace is Falsenew_fingerprintc<aaS'gS\;QJdV3RlS4F 'dK RM RM!V3RlS44'g\RS RS R24hS'dURSPP9d\RS R24hR SPP9d\R S R24h\ SSR 7o\ S4VVVVVVV3R l4pR VnV#)c3T<"TFqSPP9xK R#5ir:)__code__ co_varnames)rNr&rs& r.rP>fingerprint_transform.._fingerprint..s "cQb4==+D+D#DQbs%(FTz function z is missing parameters z in signaturerz'seed' must be in z 's signaturerz'generator' must be in )rc <\SVVS S S R7pV'dV^,pVR,pMAVP\\\P !S4P 444pS 'd\VPS V4pMLSFFpVPV4f WRR&\VPS V4W&K5\W,4KH S!V.VO5/VBpS 'dXVnV#))rrrrfingerprint_name) rrrrrrrrrrr)rrrrWrroutrrrrrrrs*, r.wrapper._fingerprint..wrappers%B%+$7 & "#'7Bx#)::d48I8I$8O8Z8Z3[.\#]"4W5I5I9Vl"m(9$zz"23;EU/AB3E#00)=S40-V-EF):w000C'6$Jr?r)allrrrrr_decorator_name_) rrrrrrrrrs f @r.r+fingerprint_transform.._fingerprintsss"cQb"csss"cQb"cccy.EFWEXXefg g T]]666 #5dV-J)QbPc66p r?cV^8dQh/^\9d\R,;R&^\9d\\P,;R&^\9d\\ \ 3,;R&#)rerrLrRr)__conditional_annotations__rr'rSrrnrg)rhs"r.ririsZPA@ 9@QRCB(7??";BSN+*d39o*Or?)@r:)NNF)NNNFN)6rrrrr4r#r' functoolsrpathlibrtypingrrrrr numpyrrtr namingr utils._dillr utils.loggingr arrow_datasetrr@rr_rLrRrrXr2r`rbrkrprrRandomrrrrrrrrrri)rs@r.r s @@ 6%& H  <@@>B B11h6&j**"*/""D--/(** B/d (*"Zffr?