#ti.(ddlZddlZddlZddlZddlZddlZddlZddlmZddlm Z ddl m Z m Z ddl ZddlmZ ddlZddlmZddlmZddl mZmZmZdd lmZdd lmZmZm Z ejBd Z"Gd d e#Z$dZ%dZ&dZ'GddejPjRZ*GddejPjVZ,dZ-GddejPj\Z/GddeZ0GddeZ1y#e$r e sddlZYwxYw)N) lru_cache)chain) TYPE_CHECKINGLiteral)AbstractBufferedFile)AsyncFileSystem)DEFAULT_CALLBACK) filesystemopensplit_protocol)AsyncFileSystemWrapper) isfilelikemerge_offset_ranges other_pathszfsspec.referencec$eZdZfdZdZxZS)ReferenceNotReachablec:t||||_||_yN)super__init__ referencetarget)selfrrargs __class__s o/Volumes/fast/ai/experiments/MLX_z-image/.venv/lib/python3.12/site-packages/fsspec/implementations/reference.pyrzReferenceNotReachable.__init__#s $" c:d|jd|jS)Nz Reference "z" failed to fetch target )rrrs r__str__zReferenceNotReachable.__str__(sT^^,,Edkk]SSr)__name__ __module__ __qualname__rr __classcell__rs@rrr"s TrrcFtt|jSr)nextitervalues)ds r_firstr+,s QXXZ !!rc|j|}t|ttfr/t|dtr|drt |ddS|dSyyNr)get isinstancelisttuplestrr )path referencesrefs r_prot_in_referencesr60sW .. C#e}%*SVS*A,/F~c!f%a(>A>+B%rct|trt|||giSi}|D]/}t||}|j|gj |1|Sr)r/r2r6 setdefaultappend)pathsr4outr3protocols r_protocol_groupsr=6s[%#E:6@@ C&tZ8 x$++D1 JrceZdZdZy)RefsValuesViewc#K|jjjD]'}tj|j )|jj jEd{|jjD]c}|jj|}t|dk(r|j|dzA|jj|Ed{ey77 w)Nr/0) _mapping zmetadatar)jsondumpsencode_itemslistdir_get_chunk_sizeslen_generate_all_records)rvalfield chunk_sizess r__iter__zRefsValuesView.__iter__As==**113C**S/((* *4==''..000]]**,E--88?K;1$mmEDL11}}::5A A A - 1 Bs%A6D8C>9A=D6D7DDNr!r"r#rOrrr?r?@s Brr?ceZdZdZy) RefsItemsViewczt|jj|jjSr)ziprBkeysr)rs rrOzRefsItemsView.__iter__Ns)4==%%')=)=)?@@rNrPrQrrrSrSMsArrSchd}d}t|ddd|dddD]\}}|||zz }||z}|S)Nr)rU)idxsizesrLmultiss rravel_multi_indexr_RsN C DC"IuTrT{+1 q4x  , JrceZdZdZedZedZ ddedfdZdZ d Z e d d Z d Z d!d Zd ZdZdZdZdZdZdZdZdZdZe dZdZd"dZd"dZdZdZdZdZ y)#LazyReferenceMapperaoThis interface can be used to read/write references from Parquet stores. It is not intended for other types of references. It can be used with Kerchunk's MultiZarrToZarr method to combine references into a parquet store. Examples of this use-case can be found here: https://fsspec.github.io/kerchunk/advanced.html?highlight=parquet#parquet-storagecddl}|Sr-)numpy)rnps rrdzLazyReferenceMapper.npds  rcddl}|Sr-)pandas)rpds rrgzLazyReferenceMapper.pdjs  rNengine) fastparquetpyarrowc||_i|_||_||_||_|jdz|_|t jdn||_|jj|xs |j|_ ddl m }|jdk(r|d tdt|j|_td |j |_y) a This instance will be writable, storing changes in memory until full partitions are accumulated or .flush() is called. To create an empty lazy store, use .create() Parameters ---------- root : str Root of parquet store fs : fsspec.AbstractFileSystem fsspec filesystem object, default is local filesystem. cache_size : int, default=128 Maximum size of LRU cache, where cache_size*record_size denotes the total number of references that can be loaded in memory at once. categorical_threshold : int Encode urls as pandas.Categorical to reduce memory footprint if the ratio of the number of unique urls to total number of refs for each variable is greater than or equal to this number. (default 10) engine: Literal["fastparquet","pyarrow"] Engine choice for reading parquet files. (default is "fastparquet") z/{field}/refs.{record}.parqNfiler) find_specrjz)engine choice `pyarrow` is not installed.imaxsize)rootrN cat_threshrh cache_sizeurlfsspecr fsunstrip_protocolout_rootimportlib.utilrm ImportErrorrrH_key_to_record)rrprurwrrcategorical_thresholdrhrms rrzLazyReferenceMapper.__init__psB / $99<</1z&##F+r001FTYYG , ;;) # )(<(DIJ J #y{4<<0 5i5d6I6IJrc^|dvr|j|j|St|)N)rG record_sizerC)setup__dict__AttributeErrorritems r __getattr__zLazyReferenceMapper.__getattr__s. 9 9 JJL==& &T""rcPi_jjdjjdgjd<t j jd}|d_|d_tjfd}|_ y)N/ .zmetadatar}metadatarncrjj||}tjjj |} j j|j}|jDcic]}|||j}}|Scc}w#t$rd}Y|SwxYw)zcached parquet file loader)rMrecord)rhN) rsformatioBytesIOrucat_filerg read_parquetrhcolumnsto_numpyOSError)rMrr3datadfcrefsrs r open_refsz,LazyReferenceMapper.setup..open_refss88??v?>D::dgg..t45D WW))$t{{)C57ZZ@Z2a5>>++Z@KA K s$5B'B"B'"B'' B65B6) rGrurjoinrprDloadsr}rCrrrr)rmetrs` rr~zLazyReferenceMapper.setups $(GG$4$4 HHdii. /%  L!jj\23}-Z 4?? +  , #rc ni|d}|'tjj|fi|xsi\}}|j|r|j |d|j |d|j dj|dgtj|jt||fi|S)aJMake empty parquet reference set First deletes the contents of the given directory, if it exists. Parameters ---------- root: str Directory to contain the output; will be created storage_options: dict | None For making the filesystem to use for writing is fs is None fs: FileSystem | None Filesystem for writing record_size: int Number of references per parquet file kwargs: passed to __init__ Returns ------- LazyReferenceMapper instance rr}T recursiveexist_okrr) rtcore url_to_fsexistsrmmakedirspiperrDrErFra)rpstorage_optionsrur}kwargsrs rcreatezLazyReferenceMapper.creates,k: :{{,,TMo6KMHB 99T? EE$$E ' D4 ( $ -. 30F0F0HI"46v66rc>d|jD}t|S)zList top-level directoriesc3jK|]+}|jdr|jddd-yw).zrrXrN) startswithrsplit.0ps r z.LazyReferenceMapper.listdir..s-V^1<.s5ndTn c3 K|]}|ywrrQrs rrz)LazyReferenceMapper.ls..s2kdTkrF directoryrrtypesizerlc |dS)NrrQ)r^s rz(LazyReferenceMapper.ls..sAfIrkeyrY)rstriprHrlstriprrCrGupdatesortedrJrDrEr_keys_in_fieldr0rKrU)rr3detailpathdashdirnamesr*rfothersrdirinfofileinforMrVrecsrecrecinfos rlszLazyReferenceMapper.lss{{3!%4#:2<<> ||H%#QXXh5G*G   !N5dnn52dkk2A <<)c(9K.K   d#f~%RVWRV$k1ERVGW# #D!"4>>1 4>>$#78![[.#  'H,2EF F"nn Lnd4& 0KTn L $ K t$qz/Jt K L   t~~-JJt~~d34T*    ""5) U?<$t*, ,))%0!t_ , c1v63r7 ;,  '!!w X  MK  " s=-J -J% J8AJ3J! J!!J& ;J& AJ+-J0c||jvr|j|S||jvr0tj|j|j Sd|vs|j |r t ||jdd\}}|j|\}}}|jj||fij|d}|t |r|S|dk(ry |j||}gd} | D cgc]} | |vr|| |nd} } | d} | | S| d t d | dd ddgk(r| ddS| dd S#tttf$r} t || d} ~ wwxYwcc} w) zfGet the reference for one key Returns bytes, one-element list or three-element list. rrXFNrrr3offsetrrawrYz1This reference does not exist or has been deleted)rGrCrDrErF_is_metaKeyErrorrrzr.r ValueError TypeErrorFileNotFoundError)rrrM_rri chunk_sizemayberexcrr selectionrs r _load_one_keyz!LazyReferenceMapper._load_one_key,s $++ ;;s# # DNN "::dnnS1299; ; ^t}}S13- ::c1%q!%!4!4S!9J 488UC =N L 1_ )>>%0D4AHIAAIT!WR[47 Im ?J Q< NO O Qq>aV #Ra= !}I'89 )3-S ( )Js'EE-E* E%%E*c4|jdd\}}|j|}t|dk(ry|jdDcgc] }t |}}t ||}||j z}||j z} || t|fScc}w)z3Details needed to construct a reference for one keyrrXr)rrr.)rrIrJsplitintr_r}) rrrMchunkrNr chunk_idx chunk_numberrrs rrzz"LazyReferenceMapper._key_to_recordTszz#q) u++E2 { q %*[[%56%5SV%5 6(K@ !1!11 D,, ,r3{+++ 7s Bc||jvr`|j|d}t|d|dDcgc]\}}tj||z }}}|xsdg|j|<|j|Scc}}w)z6The number of chunks along each axis for a given fieldz/.zarrayshapechunksrX)rNrCrUmathceil)rrMzarrayr^r size_ratios rrIz$LazyReferenceMapper._get_chunk_sizes`s (( (^^ugX$67F-0&BR-S-STQ !a% -S '1&7QCD  U #&& s"A>c|j||}tt|j}t |dk(r d|DSt |dk(r|dSd|DS)z8The references for a given parquet file of a given fieldrc32K|]}t|ywrr0rts rrz7LazyReferenceMapper._generate_record..ps(RDGRrXrc3LK|]}|dst|ddn|dyw)rNrrs rrz7LazyReferenceMapper._generate_record..vs*Br!qtD2AK15rs"$)rr(rUr)rJ)rrMrrits r_generate_recordz$LazyReferenceMapper._generate_recordjs`~~eV, #t{{}% & t9>(R( ( Y!^; CrB Brc#Kd}|j|D]}||z} tj||jz }t |D]}|j ||Ed{y7w)zJLoad all the references within a field by iterating over the parquet filesrXN)rIrrr}ranger)rrMnrecchrs rrKz)LazyReferenceMapper._generate_all_recordsxsi''.B BJD/yy 0 001DkF,,UF; ; ;" ;sA#A/%A-&A/ct|Sr)r?rs rr)zLazyReferenceMapper.valuess d##rct|Sr)rSrs ritemszLazyReferenceMapper.itemss T""rct|Sr)idrs r__hash__zLazyReferenceMapper.__hash__s $xrc$|j|Sr)r)rrs r __getitem__zLazyReferenceMapper.__getitem__s!!#&&rc^d|vr|j|sy|jdd\}}|j|\}}}|jj ||fi}|||<t ||j k(r|j||yyt|dr|jj} n#t|tr|j} n|} | |j|<tj| } i|jj!|i| |j|<y)NrrXto_bytes)rrrzrGr8rJr}writehasattrrdecoder/bytesrDrrCr.) rrvaluerMrrr]rsubdictrL new_values r __setitem__zLazyReferenceMapper.__setitem__s #:dmmC0::c1-LE5..s3LFAqkk,,eV_bAGGAJ7|t/// 5&)0uj)nn&--/E5)lln"DKK  3I"NT^^%7%7R%@"NI"NDNN3 rc0|jdxsd|vS)Nrz/.zrrs rrzLazyReferenceMapper._is_metas~~d#3u|3rc||jvr|j|=y||jvr|j|=yd|vr|j|sy|jdd\}}|j |\}}}|jj ||fi}d||<t ||jk(r|j||yyd|j|<y)NrrX) rGrCrrrzr8rJr}r )rrrMrrr]rs r __delitem__zLazyReferenceMapper.__delitem__s $++  C DNN "s#cz$--"4::c1-q#2237 1++00%"E! w<4#3#33JJuf-4$( C rc tddl}ddl}ddl}|j||f}d} t ||j kr |j ||} | r| d} | d} | d} | d} n|j|j |jd} |j|j d } |j|j d } |j|j |jd} |jD]\}}t|trot| jd k(r/|d| jj vr| j#|d} |d| |<t |d kDst|d | |<|d | |<|d| |<d| |<d| |<d| |<|j$j'|| |<|j)| | | | d d}|j*j-|j*j/xsd z |j0kDr|dj3d |d<ddd}ddg}|xs |j4d|d|d}|j6j9|xs |j4d|d|j:dk(rddi}n.|j:dk(rd||d}nt=|j:d|j>|f|j:|xstA|j6ddddd||jC|jjE||fy#t$rYwxYw)NrFr3rrrO)dtypeint64categoryrXr)copyr utf8)rr3rz/refs.z.parqTrrjwrite_statisticsri)statsobject_encoding has_nullsz not supportedrzstd)rhr compressionindex)# kerchunk.dfrcrfrGrJr}rrfullnanzerosrr/r0r2r categoriesadd_categoriesr _proc_raw DataFramer3countnuniquerqastyperwrumkdirsrhNotImplementedError to_parquetgetattrclearpop)rrMrbase_urlrkerchunkrdrg partitionoriginalr:offsetsr[rawsjrrrr fndf_backend_kwargss rr zLazyReferenceMapper.writesEKK0  y>D,, , >>%8 V$Ex(GV$EE?DGGD,,bffCG@Ehht//wh?GHHT--WH=E774++RVV37?D (GAt$% $ 2Qu{{'='==!00a9E7at9q=!%aGAJ#AwE!Ha aQ#++//5Q')*\\!     77==?bggoo/41 5 GF**:6BvJ")6:UO )DMM*!E7& F (3dmm4AeW=M ;;) #!3U ;  [[M )#2&!  & ^&DE E  ;;+9tww 148      (Q  sL** L76L7ct|jD]-}t|ts|\}}|j ||||/t|jD]H}|dk7s d|vst j |jj||j|<J|j|jd}|jjt j|j|jd<|jjdj|xs |j dg|jd|j"j%y)zOutput any modified or deleted keys Parameters ---------- base_url: str Location of the output )r5rrrrrN)r0rGr/r1r rDrr4rCr}r3rErFrurrrwr cache_clear)rr5rthingrMrkrs rflushzLazyReferenceMapper.flush s$++&E%' % v %$3 'dkk"AL TQY$(JJt{{q/A$Bq!# >>$:J:JK $(JJsO$:$:$< L!  HHh/$--> ? KK % ""$rcd}|jD]@}|jdr|dz }|tj|j |z }B|t |j z }|td|jDz }|S)NrrrXc3BK|]}t|trdyw)rXN)r/r1)rrs rrz.LazyReferenceMapper.__len__..8sH 1:a3GQ s) rHrrprodrIrJrCsumrG)rr,rMs r__len__zLazyReferenceMapper.__len__.s\\^E$ 4#8#8#?@@ $ T^^$$ H HHH rc# Kt|j}|j|j|D]}t |t s||j D]!}|j|D] }||vs| #ywr)rrCrrGr/r2rHr)rmetasbitrMrAs rrOzLazyReferenceMapper.__iter__;srDNN# T[[!C#s# \\^E((/9G0$sAB 1B; BcF |j|y#t$rYywxYw)NTF)rrrs r __contains__z LazyReferenceMapper.__contains__Hs*    t $  s   c #K|j|}t|dk(r|dzytjd|D}|D]2}|dzdj |Dcgc] }t |c}z4ycc}ww)z|List key names in given field Produces strings like "field/x.y" appropriate from the chunking of the array rrANc32K|]}t|ywr)r)rr]s rrz5LazyReferenceMapper._keys_in_field..Xs"A[58[rrr)rIrJ itertoolsproductrr2)rrMrNindsindrs rrz"LazyReferenceMapper._keys_in_fieldOs ++E2 { q $,    "A["ABC#+#)>#Q#a&#)> ?? ?)>sABA; -B)NN ri)NNi')rTNN)!r!r"r#__doc__propertyrdrgrrrr~ staticmethodrrHrrrzrIrrKr)rrrrrrr rBrGrOrLrrQrrrara[sY   4A5K015Kn##077< @"D&P ,' C<$#'O(44("S)j!%F   @rraceZdZdZdZdZ d fd Zd!dZd!dZd!dZ dZ d Z e fd Z d"d Zd#d Zd$d ZdZd$dZdZdZdZd%dZd&dZdZdZdZd&dZd'fd ZdZdZdZd(dZ d(dZ!dZ"xZ#S))ReferenceFileSystemaView byte ranges of some other file as a file system Initial version: single file system target, which must support async, and must allow start and end args in _cat_file. Later versions may allow multiple arbitrary URLs for the targets. This FileSystem is read-only. It is designed to be used with async targets (for now). We do not get original file details from the target FS. Configuration is by passing a dict of references at init, or a URL to a JSON file containing the same; this dict can also contain concrete data for some set of paths. Reference dict format: {path0: bytes_data, path1: (target_url, offset, size)} https://github.com/fsspec/kerchunk/blob/main/README.md rFc Xt|di|||_| |_| |_i|_i|_i|_| |_| |_ t|trtdi|xs|xsid|i}tjj|fi|\}}d|vrL|j!ds|j#|r*t$j'd|t)||| |_nptj,|dfi|5}t$j'd|t/j0|}ddd|j3| n|j3|| t|tr|j5Dcic]A\}}|t|tr)tj6|j9d d d fi|n|Cc}}|_d|j vrt7d |j d<y|Dt|j:t<r|j:d n |j:}||j |<||j j?D]g}tA|r|}tjjC|\}}|s:||j vsIt7|fi|xsi}||j |<i||j*j?D]}tA|r|}t|tDs&|d s,tjjC|d \}}||j vs`t7|fi|xsi}||j |<n|r-||j vrt7|fi|xsi}||j |<|xs t7d |j d<|j j5D]Z\}}|jFs%tI||jJ |j |<7|jJ|jJz sQtMdy#1swYxYwcc}}w)a Parameters ---------- fo : dict or str The set of references to use for this instance, with a structure as above. If str referencing a JSON file, will use fsspec.open, in conjunction with target_options and target_protocol to open and parse JSON at this location. If a directory, then assume references are a set of parquet files to be loaded lazily. target : str For any references having target_url as None, this is the default file target to use ref_storage_args : dict If references is a str, use these kwargs for loading the JSON file. Deprecated: use target_options instead. target_protocol : str Used for loading the reference file, if it is a path. If None, protocol will be derived from the given path target_options : dict Extra FS options for loading the reference file ``fo``, if given as a path remote_protocol : str The protocol of the filesystem on which the references will be evaluated (unless fs is provided). If not given, will be derived from the first URL that has a protocol in the templates or in the references, in that order. remote_options : dict kwargs to go with remote_protocol fs : AbstractFileSystem | dict(str, (AbstractFileSystem | dict)) Directly provide a file system(s): - a single filesystem instance - a dict of protocol:filesystem, where each value is either a filesystem instance, or a dict of kwargs that can be used to create in instance for the given protocol If this is given, remote_options and remote_protocol are ignored. template_overrides : dict Swap out any templates in the references file with these - useful for testing. simple_templates: bool Whether templates can be processed with simple replace (True) or if jinja is needed (False, much slower). All reference sets produced by ``kerchunk`` are simple in this sense, but the spec allows for complex. max_gap, max_block: int For merging multiple concurrent requests to the same remote file. Neighboring byte ranges will only be merged when their inter-range gap is <= ``max_gap``. Default is 64KB. Set to 0 to only merge when it requires no extra bytes. Pass a negative number to disable merging, appropriate for local target files. Neighboring byte ranges will only be merged when the size of the aggregated range is <= ``max_block``. Default is 256MB. cache_size : int Maximum size of LRU cache, where cache_size*record_size denotes the total number of references that can be loaded in memory at once. Only used for lazily loaded references. kwargs : passed to parent class r<z.json)parqparquetrz$Open lazy reference dict from URL %s)rurrrbzRead reference from URL %sN:rXrrl) asynchronouszEReference-FS's target filesystem must have same value of asynchronousrQ)'rrrtemplate_overridessimple_templates templatesfss _dircachemax_gap max_blockr/r2dictrtrrendswithisdirloggerinforar4r rDload_process_referencesrr rr<r1r)callabler r0 async_implr r`r)rforref_storage_argstarget_protocoltarget_optionsremote_protocolremote_optionsrurarbrfrgrrrdicref_fsfo2rtextrAoptsr5r<rrs rrzReferenceFileSystem.__init__osR "6" "4 0 " b# #;~;GVC!++//:c:KFCc! 45c9J BBG"5)#[[T1S1QKK #-R[[%"@ Abkk )+DHH_ %  "~~,,.C=%C$kk88= ! 8#HG1E2GB)+DHHX& /  "--/C=%Cc4(SV"(++"<"OF0D"FB(*DHH_ %1z&1HHNN$DAq<<4QTEVEVW ""Q^^3 & %o21s:,P6AP&P#c  |j|}tjd| |j|}t |tr|j}t|dr|j}t |trGtjd|d|jdrtj|dd}|ddfSt|dk(r%tjd|d||d }||}}nV|\}} } tjd|d |d | d | | | z} ||d k\r| |z}n| |z}n| }||d k\r| |z}n| |z}n| }| |j }|||fS#t$r}t ||d}~wwxYw) Nzcat: rz Reference: z , type bytesbase64:rXz, whole file => rz => z , offset z, size )_strip_protocolrkdebugr4rrr/r2rFr rr rbase64 b64decoderJr) rr3startendpartrrsstart1end1start0rend0s r _cat_commonzReferenceFileSystem._cat_commons##D) uTF^$ 3??4(D dC ;;=D 4 $==?D dE " LL;tfL9 :z*''QR1t# # t9> LL;tf,>FcK|j|||\}}}t|tr|||St|\}} |j|j |||d{S7#t $r} t||| d} ~ wwxYwwN)rr)rr/r r rd _cat_file Exceptionr rr3rrr part_or_urlrrr<res rrzReferenceFileSystem._cat_file>s$($4$4TC$4$P! VT k5 )uS) )$[1 ! B(+556t6  B'k: A Bs:2B>c t|tr|rtt|tr|st d|Drtt ||j }i}|jD]\}}|j|} ggggf\} } } } |D]t} |j|\}}}t|ttfs0 | j|| j|| j|| j|vg}g}g}g}t}t| | | | D]u\}}}}t|tr|||<|!|j||j||j||j||j|wt| | | | D]S\}}}}| ||vs|j||j||j||j|Ut!t|t|t|d|j"|j$\}}}| j'|||}t| | | | D]\}}}}||vr t||||D]i\}}}}||k(r#||t|t(r|||<'|||||<0||k(s6||k\s<||ksBt|t(r|||<X|||z ||z xsd||<k|j+jD]\\} }!t|!t(s| |j vs&|| }"t-| |j | }#|"|#_|dk(r|#|dk7sX|#|| <^t1|dk(rt|trd|vr t3|S|S#t$r}|dk(r|dk7r|||<Yd}~/d}~wwxYw)Nc3$K|]}d|v yw)*NrQrs rrz*ReferenceFileSystem.cat..}s7O$Qq$sraiseomitT)sortrfrgrXr)r/r2r0r0anyr=r4rrdrr r9rrrUaddrrfrg cat_rangesrrr __cause__rJr+)$rr3ron_errorr proto_dictr;protor:ruurlsstartsends valid_pathsrur^rerrurls2starts2ends2paths2 whole_files new_paths new_startsnew_ends bytes_outrdnsnebrAvexnew_exs$ rrzReferenceFileSystem.catzs dC Y% % dD !yC7O$7O4O% %%dDOO< &,,.LE5%B.0"b"n +D&$ *"..q1GAq!%a%6 7KKNMM!$KKN&&q)#(EGEF%K!$kB 1aa'CFY OOA&LLONN1%LLOMM!$C"$kB 1a=Qk%9LLONN1%LLOMM!$ C/BU W U  .. / +Iz8 iXFI"$kB 1a8%(J)%TMBBQwBJ"*%a3%&CF%&qVCFqQ"Wb%a3%&CF%&q2vR0@D%ACF&UCy/VHHJ$$&DAq!Y'A,@V.q$//!2DE#% w& L'#CF' s8q=Zc2s$#;  [)%7*6)!$A %s+N O'N==Oc|jdd}||j|y|dk(r|j||ytd|)NversionrX)raz Unknown reference spec version: )r._process_references0_process_references1r)rr4raverss rrnz'ReferenceFileSystem._process_referencessS~~i. <  % %j 1 QY  % %jEW % X?vFG Grc t|trH|jDcic]-\}}|t|trtj|n|/}}}||_ycc}}w)z&Make reference dict for Spec Version 0N)r/rhrrDrEr4)rr4rrLs rrz(ReferenceFileSystem._process_references0sa j$ ' !+ 0 0 2 2HC 3(=TZZ_3F 2 % s2A!cjr jrddli_j |j dit dfd}|j dijD]/\}}t|trF|jdr%tj|ddj|<|j|<]t|tr#tj|j|<jr|d}d|vrQjr=|j!dd j!d d j"dij}n||}t%|d k(r|gn ||d |d gj|<!|j|<2jj'j)|j dgy)NrrcicZj|jdijSNrQ)Templaterenderrc)rjinja2rs r _render_jinjaz?ReferenceFileSystem._process_references1.._render_jinjas&,6??1%,,>t~~> >rrzbase64:r~{{{z}}}rXrgenrQ)rbrcrr4_process_templatesr.rrr/r2rrrrhrDrEreplacerrJr _process_gen)rr4rarrArrrs` @rrz(ReferenceFileSystem._process_references1s$$   {B ?@ 4 ?  ?NN62.446DAq!S!<< *)/)9)9!AB%)@DOOA&%&"At$%)ZZ]"aD19,,$AIIdC0$WT3/#V6&*nn6 *!,,/FaKaSa1qt_"%&")7* t00r1JKLrci|_|j|j|j|jD]1\}}d|vrddl|ffd |j|<#||j|<3y)NrrcFj|jdi|Sr)rr)temprrs rrz8ReferenceFileSystem._process_templates..s(>V__>&>#!>#r)rcrarrr)rtmprArrs @rrz&ReferenceFileSystem._process_templatessi  " " . JJt.. /IIKDAqqy01%#q!%&q! rc^ i}|D]}|djDcic]H\}}|t|tr|n/t|j dd|d|j ddJc}} fdt j jD}|D]}ddl}|j|djd i||j} |j|d jd i||j} d |vrd |vr{t|j|d jd i||j} t|j|d jd i||j} | | | g|| <d |vd |vz r td | g|| <|Scc}}w)N dimensionsrrstopsteprXc3dK|]'}ttj|)ywr)rhrUrV)rr) dimensions rrz3ReferenceFileSystem._process_gen..,s,DFS)623Ds-0rrsrlengthz^Both 'offset' and 'length' are required for a reference generator entry if either is provided.rQ)rr/r0rr.rOrPr)rrrrcrr)rgensr;rrArproductsprrrrsrrrs @rrz ReferenceFileSystem._process_gen!sC -335  6DAq !!T*quuWa0!F)QUU61=MNO 6 I'//1A1A1CDH8fooc%j188P2PP8fooc%j188P2PPO(c/ =H 6==UUdnnUF!=H 6==UUdnnUF!$VV4CH#o(c/:$K !$uCH'B AsA F)cdgi|_|jj}|D]H\}}t|tt fs t |dr t|}nt|dk(rd}n|\}}}d|vr|jdddnd}|}|g}|rA||jvr3|j|}|j||r||jvr3|jt||ddD]V\} } | |jvsJ| |jvsJ|j| j| dddg|j| <X|j|j|d|dKy) NrrrXrrrrrl) dircacher4rr/r r2r rJr_parentr9reverserU) rrr3rrrparpar0subdirsparentchilds r_dircache_from_itemsz(ReferenceFileSystem._dircache_from_itemsFseR __ " " $JD$$ -z1J4yTa! 1d,/4K$++c1%a(RCDfG4t}}4||D)t$4t}}4 OO !$Wgabk!: DMM111... f%,,"KC(* e$"; MM#  % %tVT&R S3rc |j|\}}}t|trtj|||St |\} } |&|$|j | j||f||d|St|||f||d|S)N) block_size cache_options) rr/r rrr rd_open ReferenceFile) rr3moderrrrrrr<rs rrzReferenceFileSystem._opends$($4$4T$:! VT k5 )::k&67 7$[1 ! >dl+488H%++&+        "'     rc tjd||j|}t|jt r |jj ||S|js|j|j|}| t||r|S|Dcgc]}|d c}S#t$rYnwxYwtd|dcc}w)Nzlist %s'z' is not a known keyr) rkrrr/r4rarrrrr_ls_from_cache)rr3rrr;os rrzReferenceFileSystem.ls~s Y%##D) doo': ; ))$77}}  % % '!!$' ;#D) ) J#&'3a& 3''  #av-A$BC C(sB0! C0 B<;B<c J|j|xs|j|Sr)rjisfilerr3rs rrzReferenceFileSystem.existsszz$44;;t#44rc|jr|jvSt|jtr|jj vSt fd|jDS)Nc3FK|]}|jdyw)rNr)rrr3s rrz,ReferenceFileSystem.isdir..s!IAq||tfAJ/s!)rr/r4rarHrrr3s `rrjzReferenceFileSystem.isdirsW ==4==( ( )< =4??2244 4III Irc||jvSrrrs rrzReferenceFileSystem.isfilest&&rc2K|j||fi|Swr)r)rr3rrs r_lszReferenceFileSystem._lsstwwtV.v..rc P|rt|f|||d|Sr0|jtfd|jD}nt|j}|r?|j s|j |Dcic]}||j|dc}S|Scc}w)N)maxdepthwithdirsrc3FK|]}|js|ywrr)rrAr3s rrz+ReferenceFileSystem.find..sH/QQ\\$5Gq/s!!r)rfindrrr4rrr) rr3r r rrrrArs ` rrzReferenceFileSystem.finds 7<'(6MS  ''-DH$//HHAt'A ==))+:;GD!"hhtn11$//$2G2JKDGFOAw9s 8 C3C3c,K|j|Swr)rlrs r_infozReferenceFileSystem._infosyyscxK|jj|d|jjywr)r4r4rr3rs r_rm_filezReferenceFileSystem._rm_files/  $  s8:cK|dk(r|j|rt||j|<|jj yw)Nr)rFileExistsErrorr4rr3)rr3rrrs r _pipe_filezReferenceFileSystem._pipe_files= 8  D 1! ! $ sAA cK|dk(r|j|rtt|d5}|j|j|<ddd|j j y#1swY$xYww)Nrr^)rrr readr4rr3)rrrrrrs r _put_filezReferenceFileSystem._put_files\ 8  E 2! ! % !%&VVXDOOE "  s)A8A, #A8,A51A8c i}|jjD]0\}}t|tr |j d||<,|||<2tj|dfi|5}|jtjd|djdddy#t $r,dt j|zj ||<YwxYw#1swYyxYw)z+Write modified references into new locationasciir}rrX)rrN)r4rr/r r UnicodeDecodeErrorr b64encodertr r rDrErF)rrsrr;rArrs r save_jsonzReferenceFileSystem.save_jsonsOO))+DAq!U#IXXg.CFA,[[d 6o 6! GGDJJ1c:;BBD E7 6 *I(6+;+;A+>>FFHCFI7 6sB&'6C&2CCC') NNNNNNNNTii@BrSrU)F)Frr)r^NN)T)NFF) overwrite)$r!r"r#rVr<cachablerrrrrrr rr.rrnrrrrrrrrrjrr rrlrrrrr!r$r%s@rrZrZ]s HH  aF*!X BB&/? , 2`DH %MB &#JT< 4($5J'/"*  FrrZcLeZdZ dfd ZedZfdZdZxZS)rc t |||f||||||d| |jj|j\} |_|_t| \} } |jj| |_ | |_ d|_ y)N)rr autocommitr cache_typer) rrrurr3rrr rdsrc_fssrc_path_f)rrur3rrr&r'rrrrr<rrs rrzReferenceFile.__init__s    !!!'  -1GG,?,? ,J) TZ$[1 !ggkk(+ # rc|j|jjrY|jj|jf|j |j |jdd|j|_|jS)Nnone)rrr&r') r*closedr(rr)r blocksizer&rrs rrzReferenceFile.fsj 77?dggnn'dkk'' YY>>??!  ++ DGwwrcl|j|jjt|Sr)r*closer)rrs rr0zReferenceFile.closes& 77  GGMMOw}rc||jz}t||jz|j}|jj ||jj ||z Sr)rminrrseekr)rrrs r _fetch_rangezReferenceFile._fetch_range sP "# "DHH-  Evv{{3;''r)r^defaultT readaheadNN) r!r"r#rrWrr0r4r$r%s@rrrs:  :   (rr)2r collectionsrrOloggingrr functoolsrrtypingrr fsspec.corert fsspec.specrujsonrDry fsspec.asynrfsspec.callbacksr r r r #fsspec.implementations.asyn_wrapperr fsspec.utilsrrr getLoggerrk RuntimeErrorrr+r6r=abc ValuesViewr? ItemsViewrSr_MutableMappingrarZrrQrrrHs   ), (-88F   - .TLT"?  B[__// BAKOO--A @+//88@DP F/P Ff4((4(& sD DD