i+7dZddlZddlZddlmZddlmZddlmZddl m Z ddl m Z m Z mZddlmZdd lmZd d lmZmZmZmZmZmZd d lmZd d lmZmZe rd dlm Z ejBe"Z#ejHdZ%dZ&dddZ'eGddZ(e d9dee(de)dzde)de)de)dzde)dzde*e)e)fdzde+e)dzde,e+e*e+e*e)dzffdZ-Gdd eZ.Gd!d"eZ/ d:d#d$d%e*de)dzde*e)e)fdzde)dzddf d&Z0d'e*fd(Z1d)e*fd*Z2d+e*fd,Z3d#d$d-e)ddfd.Z4d#d$d/e*d0e5d-e)ddf d1Z6d/e*d2e(d0e5de+e)fd3Z7d4e+e*d5e)de/fd6Z8d#d$d7e+e)d0e5de+e*fd8Z9y);z.Git LFS related type definitions and utilitiesN)Iterable) dataclass)ceil)getsize) TYPE_CHECKINGBinaryIO TypedDict)unquote) constants)build_hf_headersfix_hf_endpoint_in_urlhf_raise_for_status http_backoffloggingvalidate_hf_hub_args) SliceFileObj)sha256 sha_fileobj)CommitOperationAddz^[0-9a-f]{40}$zlfs-multipart-uploadzapplication/vnd.git-lfs+json)Acceptz Content-TypecreZdZUdZeed<eed<eed<edefdZ edefdZ ed e fd Z y ) UploadInfoad Dataclass holding required information to determine whether a blob should be uploaded to the hub using the LFS protocol or the regular protocol Args: sha256 (`bytes`): SHA256 hash of the blob size (`int`): Size in bytes of the blob sample (`bytes`): First 512 bytes of the blob rsizesamplepathct|}t|d5}|jddd}t|}ddd||S#1swYxYw)Nrbrrr)ropenpeekr)clsrrfilershas h/Volumes/fast/ai/experiments/voice-extract-mac/.venv/lib/python3.12/site-packages/huggingface_hub/lfs.py from_pathzUploadInfo.from_pathIsTt} $ YYs^DS)Fd#CS88 s A  Adatacbt|j}|t||dd|S)Nr)rrr)rdigestlen)r#r(r%s r& from_byteszUploadInfo.from_bytesQs-Tl!!#D $t*SAAfileobjc|jd}|jdtjt |}|j }|jdtj||||S)Nrrr )readseekioSEEK_SETrtell)r#r.rr%rs r& from_fileobjzUploadInfo.from_fileobjVsXc" Q $'"||~ Q $S88r-N) __name__ __module__ __qualname____doc__bytes__annotations__int classmethodstrr'r,rr5r-r&rr6sj  M I M9S99BeBB9899r-r upload_infostoken repo_typerepo_idrevisionendpointheaders transfersreturnc||ntj}d}|tjvrtj|}|d||d} d||nddg|D cgc])} | jj | j d+c} d d } |d t |i| d <itt| |xsi}td| || } t| | j} | jdd}t|ts td| jd}t|t r|nd}|Dcgc]}d|vst#|c}|Dcgc]}d|vst%|c}|fScc} wcc}wcc}w)a Requests the LFS batch endpoint to retrieve upload instructions Learn more: https://github.com/git-lfs/git-lfs/blob/main/docs/api/batch.md Args: upload_infos (`Iterable` of `UploadInfo`): `UploadInfo` for the files that are being uploaded, typically obtained from `CommitOperationAdd.upload_info` repo_type (`str`): Type of the repo to upload to: `"model"`, `"dataset"` or `"space"`. repo_id (`str`): A namespace (user or an organization) and a repo name separated by a `/`. revision (`str`, *optional*): The git revision to upload to. headers (`dict`, *optional*): Additional headers to include in the request transfers (`list`, *optional*): List of transfer methods to use. Defaults to ["basic", "multipart"]. Returns: `LfsBatchInfo`: 3-tuple: - First element is the list of upload instructions from the server - Second element is a list of errors, if any - Third element is the chosen transfer adapter if provided by the server (e.g. "basic", "multipart", "xet") Raises: [`ValueError`](https://docs.python.org/3/library/exceptions.html#ValueError) If an argument is invalid or the server response is malformed. [`HfHubHTTPError`] If the server returned an error. N/z.git/info/lfs/objects/batchuploadbasic multipartoidrr) operationrGobjects hash_algonameref)rAPOSTrFjsonrRzMalformed response from servertransfererror)r ENDPOINTREPO_TYPES_URL_PREFIXESrhexrr LFS_HEADERSr rrrXget isinstancelist ValueErrorr>_validate_batch_actions_validate_batch_error)r@rArBrCrDrErFrG url_prefix batch_urlrLpayloadresp batch_inforRchosen_transferobjs r&post_lfs_batch_inforl`sX$/xY5G5GHJI55566yA *Aj\'2MNI"+"7Yg{=S '  '}}((*  '   G '("34   ' =bG  7 IDJnnY-G gt $9:: nnZ0O)3OS)IotO29O#G3r?r-r&rnrns O Ir-rnc,eZdZUdZeed<eeed<y)CompletionPayloadTz?Payload that will be sent to the Hub when uploading multi-part.rPpartsN)r6r7r8r9r>r;rarnr?r-r&rrrrsI H  r-rrrQrlfs_batch_actionc(t||jd}|$tjd|jdy|dd}t ||djd}| t ||jdi}|jd} t |d | } |  t| } t||| | n t|| |xt |t |d |} td| t|||jjj!|jj"d} t%| tj|jdy#ttf$rtd | d wxYw)a9 Handles uploading a given object to the Hub with the LFS protocol. Can be a No-op if the content of the file is already present on the hub large file storage. Args: operation (`CommitOperationAdd`): The add operation triggering this upload. lfs_batch_action (`dict`): Upload instructions from the LFS batch endpoint for this object. See [`~utils.lfs.post_lfs_batch_info`] for more details. headers (`dict`, *optional*): Headers to include in the request, including authentication and user agent headers. Raises: [`ValueError`](https://docs.python.org/3/library/exceptions.html#ValueError) If `lfs_batch_action` is improperly formatted [`HfHubHTTPError`] If the upload resulted in an error actionsNzContent of file z. is already present upstream - skipping uploadrLverifyheader chunk_sizehref)rEzTMalformed response from LFS batch endpoint: `chunk_size` should be an integer. Got 'z'.)rQrxry upload_url)rQr{rV)rArFrOrWz: Upload successful)rcr_loggerdebug path_in_repo_validate_lfs_actionrr<rb TypeError_upload_multi_part_upload_single_partrr upload_inforr]rr) rQrtrArFrErv upload_action verify_actionrxryr{ verify_url verify_resps r& lfs_uploadrs8,-""9-G ' (>(>'??mno%Y/9M'$Y/33H=M ]+  x ,FL)J' f(=QJ ZJ Yv*akliJG ]++M&,A8L "  $5'B"..5599;YEZEZE_E_`  K( LLI**++>?@'I& fgqfrrtu  s - E22F lfs_actionct|jdtr0|jd*t|jdts t d|S).validates response from the LFS batch endpointrzrxz"lfs_action is improperly formatted)r`r_r>dictrb)rs r&rr sK :>>&)3/ ^^H % -JNN8> r-lfs_batch_actionscPt|jdtrt|jdts t d|jdijd}|jdijd}| t || t ||S)rrPrz)lfs_batch_actions is improperly formattedrvrLrw)r`r_r>r<rbr)rrrs r&rcrcs (,,U3S 9jIZI^I^_eIfhk>lDEE%)))R8<t|jdtrt|jdts t d|S)rrPrz'lfs_batch_error is improperly formattedrZmessagecode)r`r_r>r<rbr)r error_infos r&rdrd"s **513 7JGZGZ[aGbdgz(_get_sorted_parts_urls..ds!A$r-)keyz0Invalid server response to upload large LFS file)sorteditemsisdigitr+r<rrrb)rxrrypart_numr{_sorted_part_upload_urls num_partss r&rr[s$-3LLN ,:(Hj##%#h-!*;Xr"J/,:      MAz   +,ID))J677KLL ""  s 6B  B%rrPcg}t|D]H\}}|jd}||dk(rtd|d|dz|j|dz|dJ||dS)NrprJzInvalid etag (`z`) returned for part r )rorp)rPrs) enumerater_rbappend)rrPrs part_numberrxrps r&rrms| "E()9: Vzz&! <42:tf4I+XY/IZ[\ \ )Ao  ; ''r-rc6g}|jd5}t|D]S\}}t|||z|5}td||}t ||j |j dddU ddd|S#1swYjxYw#1swY|SxYw)NTr) seek_from read_limitrr)rrrrrrrF) rQrryrFr.part_idxpart_upload_url fileobj_slicepart_upload_ress r&rr|sG   T  *g)23D)E %Ho$x/%".uoM"Z#O4667*F + N + Ns"#B5B- BB BB)NNNN)NNN):r9r2recollections.abcr dataclassesrmathros.pathrtypingrrr urllib.parser huggingface_hubr utilsr rrrrr utils._lfsr utils.sharr _commit_apir get_loggerr6r|compile OID_REGEXLFS_MULTIPART_UPLOAD_COMMANDr^rr>rratuplerlrnrrrrrcrdrr<rrrrr?r-r&rs5 $!55 %%*/   H % BJJ( ) 5-2  &9&9 &9R  %)"&S:&S :SS S Dj S Dj S#s(^d "SCy4S 4:tDz3: -.SSl9 %) DA#DADA :DA#s(^d " DA Dj DA  DANT t  4 &S&T&(("6((RU(cf(ko(.#4#j#c#VZ[^V_#$ (d4j (s (GY (#8