#tiY?ddlmZddlZddlZddlZddlmZmZddlm Z ddl m Z ddl m Z GddeZGd d e ZGd d e Zy) ) annotationsN) HTTPAdapterRetry)override)AbstractFileSystem)AbstractBufferedFilec$eZdZdZdfd ZxZS)DatabricksExceptionz< Helper class for exceptions raised in this module. cNt||||_||_||_y)z Create a new DatabricksExceptionN)super__init__ error_codemessagedetails)selfrrr __class__s j/Volumes/fast/ai/experiments/MLX_z-image/.venv/lib/python3.12/site-packages/fsspec/implementations/dbfs.pyr zDatabricksException.__init__s% !$  N)__name__ __module__ __qualname____doc__r __classcell__rs@rr r srr ceZdZdZfdZeddZddZddZddZ ddZ ddZ dd Z d Z dd Zd Zd ZdZdfd ZxZS)DatabricksFileSystemz Get access to the Databricks filesystem implementation over HTTP. Can be used inside and outside of a databricks cluster. c d||_||_tj|_t ddgd|_|jjdt|j |jjjdd|jit|0d i|y ) a Create a new DatabricksFileSystem. Parameters ---------- instance: str The instance URL of the databricks cluster. For example for an Azure databricks cluster, this has the form adb-..azuredatabricks.net. token: str Your personal token. Find out more here: https://docs.databricks.com/dev-tools/api/latest/authentication.html g?)iiiiii)totalbackoff_factorstatus_forcelisthttps://) max_retries AuthorizationzBearer N) instancetokenrequestsSessionsessionrretriesmountrheadersupdater r )rr'r(kwargsrs rr zDatabricksFileSystem.__init__"s!  '') ;  :{t||'LM ##_ |6L$MN "6"rc&|jj|jdd|j|}||jvrG|j|D]*}|d|jdk(s|ddk7r|gcSgcSt |y)zCheck cache for listing Returns listing, if found (may be empty list for a directory that exists but contains nothing), None if not in cache. /Nnametype directory)dircachepoprstrip_parentFileNotFoundError)rpathparententrys r_ls_from_cachez#DatabricksFileSystem._ls_from_cache>s $++c*D1d# T]] "v.=DKK$44V} 3 %wI / $D) ) #rc  |j|}|sX |j ddd|i}|jdg}|Dcgc]}|d|drd nd |d d }}||j|<|r|S|Dcgc]}|d  c}S#t$r0|jj|j |dd}YwxYw#t $r+}|jdk(rt|j|d}~wwxYwcc}wcc}w)a" List the contents of the given path. Parameters ---------- path: str Absolute path detail: bool Return not only the list of filenames, but also additional information on file sizes and types. Ngetlistr;methodendpointjsonRESOURCE_DOES_NOT_EXISTfilesis_dirr5file file_size)r3r4sizer3) r>r:r6r7r9 _send_to_apir rrr@) rr;detailr0outrerGos rlszDatabricksFileSystem.lsPs5 %%d+C %% 6&EE'2&E  A fI+,X;KFkN  #&DMM$  J#&'3a& 3''=!  MM  dll40$ 7C  ' <<#<<+AII6A=   (s4BCC75 C<6B=<B= C4 &C//C4c||s& |jddd|itd|d |jd d d|i|j |j |y#t$r}|jdk(r Yd}~Vd}~wwxYw#t$r+}|jd k(rt|j|d}~wwxYw) aE Create a given absolute path and all of its parents. Parameters ---------- path: str Absolute path to create exist_ok: bool If false, checks if the folder exists before creating it (and raises an Exception if this is the case) r@z get-statusr;rBzPath z already existsrFNpostmkdirsRESOURCE_ALREADY_EXISTS)rLFileExistsErrorr rrinvalidate_cacher9)rr;exist_okrPs rmakedirszDatabricksFileSystem.makedirss !! >Cc "t||f||d|S)z Overwrite the base class method to make sure to create a DBFile. All arguments are copied from the base method. Only the default blocksize is allowed. )mode block_size)DatabricksFile)rr;rmrnr0s r_openzDatabricksFileSystem._opensdDUt UfUUrc|dk(r|jj}n*|dk(r|jj}ntd|tj j d|jd|}|||} |j|jS#tj$r=} |jj}n#t$r|dwxYwtdi||d}~wwxYw) a Send the given json to the DBFS API using a get or post request (specified by the argument `method`). Parameters ---------- method: str Which http method to use for communication; "get" or "post". endpoint: str Where to send the request to (last part of the API URL) json: dict Dictionary of information to send rTr@zDo not understand method r#z/api/2.0/dbfs/)rENr&)r+rTr@ ValueErrorurllibparseurljoinr'raise_for_statusr) HTTPErrorresponserE Exceptionr ) rrCrDrE session_callurlrOrPexception_jsons rrLz!DatabricksFileSystem._send_to_apis V <<,,L u_<<++L8AB Bll""Xdmm_N#KXV 4 ( ?   vvx!! ? "!"!2 "T! "&77Q > ?s*B C04CC+ CC++C0c |jdd||d}|dS#t$r+}|jdk(rt|j|d}~wwxYw)a{ Internal function to create a handle, which can be used to write blocks of a file to DBFS. A handle has a unique identifier which needs to be passed whenever written during this transaction. The handle is active for 10 minutes - after that a new write transaction needs to be created. Make sure to close the handle after you are finished. Parameters ---------- path: str Absolute path for this file. overwrite: bool If a file already exist at this location, either overwrite it or raise an exception. rTcreate)r; overwriterBhandlerVN)rLr rrWr)rr;rrOrPs r_create_handlez#DatabricksFileSystem._create_handle.sg$ !!!";"A X; " ||88%aii0a7   s A&A  Ac |jddd|iy#t$r+}|jdk(rt|j|d}~wwxYw)z Close a handle, which was opened by :func:`_create_handle`. Parameters ---------- handle: str Which handle to close. rTcloserrBrFN)rLr rr:r)rrrPs r _close_handlez"DatabricksFileSystem._close_handleMsR    VgXvDV  W" ||88' 29   s A &AA c2tj|j} |jdd||dy#t$rP}|j dk(rt |j||j dk(rt|j|d}~wwxYw)a Upload data to an already opened file handle (opened by :func:`_create_handle`). The maximal allowed data size is 1MB after conversion to base64. Remember to close the handle when you are finished. Parameters ---------- handle: str Which handle to upload data to. data: bytes Block of data to add to the handle. rTz add-blockrdatarBrFMAX_BLOCK_SIZE_EXCEEDEDN) base64 b64encodedecoderLr rr:rrr)rrrrPs r _add_datazDatabricksFileSystem._add_data^s%,,.    $ &5   # ||88' 29!:: +2   s= BA BBc  |jdd||||z d}tj|dS#t$rO}|jdk(rt |j ||jdvrt|j |d}~wwxYw) a Download data in bytes from a given absolute path in a block from [start, start+length]. The maximum number of allowed bytes to read is 1MB. Parameters ---------- path: str Absolute path to download data from start: int Start position of the block end: int End position of the block r@read)r;offsetlengthrBrrF)INVALID_PARAMETER_VALUEMAX_READ_SIZE_EXCEEDEDN)rLr b64decoder rr:rrr)rr;startendrOrPs r _get_datazDatabricksFileSystem._get_data|s !!"esU{K"A ##AfI. ." ||88' 29!VV +2   s25 B A BB c||jjn|jj|dt||yr)r6clearr7r rX)rr;rs rrXz%DatabricksFileSystem.invalidate_caches9 < MM   ! MM  dD )  &r)returnz!list[dict[str, str | int]] | None)TF)FN)rbdefaultr)rrrrr rr>rRrZr^rdrkrprLrrrrrXrrs@rrrsn #8**"-(^2B$"2<HL,>\V'R>"<<''rrcNeZdZdZdZ dfd ZdZd dZdZd dZ xZ S) rozH Helper class for files referenced in the DatabricksFileSystem. ic ||dk(r |j}||jk(s Jd|t |||f|||||xsid|y)zr Create a new instance of the DatabricksFile. The blocksize needs to be the default one. Nrz,Only the default block size is allowed, not )rmrn autocommit cache_type cache_options)DEFAULT_BLOCK_SIZEr r ) rfsr;rmrnrrrr0rs rr zDatabricksFile.__init__s|  y!800JT444 ::, G 4    !!!'-2  rcX|jj|j|_y)z(Internal function to start a file uploadN)rrr;r)rs r_initiate_uploadzDatabricksFile._initiate_uploadsgg,,TYY7 rc|jjd|jj}|jt |Dcgc] \}}||| }}}|D])}|j j |j|+|r'|j j|jyycc}}w)zrsH" 0&%, ) C'-C'L N')N'r