ikg`dZddlZddlZddlmZddlmZmZddlm Z ddl m Z ddl m Z mZddlZddlmZd d lmZmZmZmZmZmZmZd d lmZmZd d lmZmZm Z m!Z!m"Z"m#Z#m$Z$d dl%m&Z&m'Z'e$dZ(e dGddZ)ejTdZ+hdZ,hdZ-hdZ.ejTdZ/dddddZ0iZ1e2e.D]Z3e3e1e3<e3de1e3d<e3de1e3d <e d!e1e4e5"Z6e dGd#d$Z7e8eedzfZ9e:ee;e4fZd)ed&e8e:e4efe:e4e8eeffffd*Z?d)ed+e@d&e8eAe9efd?e e@ejd@fdAe eAe4dzejdBdCdDfdEedFe e6dzejdGfdHe eKdzejdIfd&dfdJZLe(jgdK;ddfdMe e@ejdNdOdPfdQe e@ejdRfdEed&df dSZNe(jdTdUg;ddfdMe e@ejdNdOdPfdQe e@ejdRfdEed&df dVZOe(jgdW;e jdddd9-?#@@AA 38_r/r"zA^(?P[a-zA-Z_]+)\s*(?P==|!=|>=|<=|>|<|=)\s*(?P.+)$><=>!=<=>=>refssizetypeaccessedmodified>namer8r:r;z.^(?P[a-zA-Z_]+)(?::(?Pasc|desc))?$descasc)r:r;r8r<z:asc_ascz:desc_desc SortOptions)r9modulec0eZdZUdZeed<eed<eed<y)CacheDeletionCountsz>Simple counters summarizing cache deletions for CLI messaging. repo_countpartial_revision_counttotal_revision_countN)r&r'r(__doc__intr+r.r/r0rDrDGsHOr/rDselected_by_reporeturncd}d}d}|jD]H\}}|t|z }t|t|jk(s6|dz }|t|z }J||z }t|||S)z(Summarize deletions across repositories.rr)itemslenr#rD)rJrEtotal_revisionsrevisions_in_full_reposrepor#rFs r0summarize_deletionsrRTsJO+113i3y>) y>S0 0 !OJ #s9~ 5 # 4 -/FF z+A? SSr/c t|jdD]}|jd|j}t||d}t |t |j k(rt jd|djt jd|d|D]\}djt|jxsd }t jd |jd |d |j^y )zBPretty-print selected cache revisions during confirmation prompts.cN|j|jjfSN repo_typerepo_idlowerrQs r0z0print_cache_selected_revisions..hs$..RVR^R^RdRdRfAgr/key/c|jSrU commit_hashrevs r0r[z0print_cache_selected_revisions..j3??r/ - z (entire repo): z (detached)z z [z] N) sortedkeysrWrXrNr#rtextjoinr7rasize_on_disk_str)rJrQrepo_keyr#revisionr7s r0print_cache_selected_revisionsrofs',,.4ghnn%Qt||n5+D17RS y>S0 0 HHtH:^4 5  4z#$!H88F8==12BlD HHvh2232dV2h>W>W=XY Z"ir/ hf_cache_infoci}i}|jD]Q}|jj}|||<|jD]!}||f||jj<#S||fS)zPCreate lookup tables so CLI commands can resolve repo ids and revisions quickly.)reposcache_idrYr#ra)rp repo_lookuprevision_lookuprQrmrns r0build_cache_indexrvuss.0KLNO##==&&( $ HH=A8.sQUQ]Q]QcQcQe@fr/r\c|jSrUr`rbs r0r[z'collect_cache_entries..rdr/NcN|dj|d|djfSdfS)Nrr)rsraentrys r0r[z'collect_cache_entries..s2a!!(-a(<a$$BDr/c |djS)Nr)rsr}s r0r[z'collect_cache_entries..suQx'8'8r/)rhrrr)r#r7appendsort)rprwentries repo_refs_map sorted_reposrQrnrefs r0collect_cache_entriesrs!#G!#M---3fgL'T^^(e^W_WdWdPSWd^(ef d "4>>7RSh/0T NND$< (     M !!  8 9 M !!)fsC exprrc v tj|j}|std|d|j dj |j d|j dj t vr"tdd|dtt d tvr"td d |dttd d k(rt   fd Sdvr>t j dtdtdzdtdtf fd }|Sdk(rG j dk7rtdddtdtdzdtdtffd }|Sdk7rtdd dtdtdzdtdtf fd }|S)zqConvert a `hf cache ls` filter expression into the yes/no test we apply to each cache entry before displaying it.zInvalid filter expression: 'z'.r]opvaluezUnsupported operator 'z ' in filter ''. Must be one of .zUnsupported filter key '' in 'r8cZt||jS|jSrU)_compare_numeric size_on_disk)rQrn_rsize_thresholds r0r[z&compile_cache_filter..s3)9%-%9H ! !  * ?C?P?P  * r/>r:r;rQrnNnowrKcdk(r |jn| |jn |j}|yt||z S)Nr:F) last_accessed last_modifiedr)rQrnr timestampr]rsecondss r0 _time_filterz*compile_cache_filter.._time_filtersW*$""'++''  #C)ORA Ar/r9r2z/Only '=' is supported for 'type' filters. Got 'rc>|jjk(SrU)rWrY)rQrnrexpecteds r0 _type_filterz*compile_cache_filter.._type_filters>>'')X5 5r/z.Only '=' is supported for 'refs' filters. Got c| |jnj|t}j|Dcgc]}|jc}vScc}wrU)r7getr)rY)rQrnrr7rr value_raws r0 _refs_filterz*compile_cache_filter.._refs_filtersO$,$88==m>O>OPTV_Va>bD??$(E(EE E(EsA)_FILTER_PATTERNmatchstrip ValueErrorgrouprY_ALLOWED_OPERATORSlist _FILTER_KEYSrrr rfloatbool) rrrrrrrr]rrrrs ` @@@@@@r0compile_cache_filterrs  ! !$**, /E 7vR@AA ++e  " " $C T B G$**,I ##1"]4&HZ[_`r[sZttuvww ,3C5tfDVW[\hWiVjjklmm f}#I.   && !23 B~ B9Kd9R BY^ Bcg B f}??$ 9NrdRTUV V 6~ 69Kd9R 6W\ 6ae 6 9MbTQRST T F~ F9Kd9R FW\ Fae Fr/leftrrightcl|y||k(||k7||kD||k||k\||kd}||vrtd|||S)z)Evaluate numeric comparisons for filters.F)r2r4r3r1r6r5z)Unsupported numeric comparison operator: )r)rrr comparisonss r0rrsd |U]em E\ E\emem K DRDIJJ r?r/ sort_expr.c tj|jj}|st d|d|j dj|j d}t vr"t dd|dtt d|r|nt}|d k(}d td ttd fffd }||fS)zConvert a `hf cache ls` sort expression into a key function for sorting entries. Returns: A tuple of (key_function, reverse_flag) where reverse_flag indicates whether to sort in descending order (True) or ascending order (False). zInvalid sort expression: 'z5'. Expected format: 'key' or 'key:asc' or 'key:desc'.r]orderzUnsupported sort key 'rrrr=r~rK.c|\}}dk(r|jj}|fSdk(r ||j}|fS|j}|fSdk(r |j|j}|fSd}|fSdk(r?| |j|j}|fSd}|fS|j |jnd}|fSt d)Nr<r8r:gr;zUnsupported sort key: )rsrYrrrr)r~rQrnrr]s r0 _sort_keyz%compile_cache_sort.._sort_key s h &=,,.E8O &=-5-AH))E8OHLGXGXE8O * +/*<*<*HD&&E8OORE8O * #2:2H2H2T..8O[^8O/3.@.@.L**RU8O1#788r/) _SORT_PATTERNrrrYrr _SORT_KEYSr_SORT_DEFAULT_ORDER CacheEntryr-r )rrexplicit_orderrreverserr]s @r0compile_cache_sortrs     1 7 7 9 :E 5i[@uvww ++e  " " $C[[)N *1#fYKGYZ^_iZjYkklmnn-N2Ec2JEvoG99c3h9< g r/targetsct|\}}tt}t}g}|D]}|j}|s|j } t j d| rZ|j| } | |j|a| \} } || j| |j| j|j| } | |j|| jD]1} || j| |j| j3|jD cic]\} }| t|}} }tt||t|Scc}} w)z8Resolve the deletion targets into a deletion resolution.z[0-9a-fA-F]{40}r#r$r%)rvrsetrrYre fullmatchrraddrar#rMr)r"r-)rprrtrur$r#r% raw_targettargetloweredrrQrn matched_reporevsfrozen_selecteds r0_resolve_deletion_targetsr+s^#4]#C K>I#>NH%IG !!# ,,. <<*G 4#''0E}z*"ND( TN  x ( MM(.. / "w/   NN: & $..H \ " & &x 0 MM(.. //+2@H~~?OP?OttYt_,?OOP I& g Qs<E5z list | ls)z hf cache lszhf cache ls --revisionsz*hf cache ls --filter "size>1GB" --limit 20zhf cache ls --format json)examplesF cache_dirz9Cache directory to scan (defaults to Hugging Face cache).r#zCInclude revisions in the output instead of aggregated repositories.filterz-fz--filterzZFilter entries (e.g. 'size>1GB', 'type=model', 'accessed>7d'). Can be used multiple times.formatraSort entries by key. Supported keys: 'accessed', 'modified', 'name', 'size'. Append ':asc' or ':desc' to explicitly set the order (e.g., 'modified:asc'). Defaults: 'accessed', 'modified', 'size' default to 'desc' (newest/biggest first); 'name' defaults to 'asc' (alphabetical).limitzSLimit the number of results returned. Returns only the top N entries after sorting.c t|}|xsg}t ||\} } |D cgc]} t | | } } tj} | D]"}| Dcgc]}||d|d| s|} }$|r, t|j\}}| j|||#|dkrtjd|d| d|} |r| Dcgc]t\}}|m|j|j|j |j"t|j$|j&|j(t+|j,d v}}}t/j0|gd d d d in| Dcgc]p\}}|j|j|j |j&|j2xsd|j(t+| j5|t7dr}}}t/j0|gddd d i| r| Dchc]\}}| }}}t9|}|r%t;d| D}t;d| D}n$t;d|D}t;d|D}t/j<t?j@d|d|dtC|dyy#t$r}td|j|d}~wwxYwcc} w#t $r$}tjt||d}~wwxYwcc}w#t $r$}tjt||d}~wwxYwcc}}wcc}}wcc}}w)z&List cached repositories or revisions.Cache directory not found: N)rwrr)r]rz&Limit must be a positive integer, got r)idrXrWrn snapshot_pathr8rr7)rrnr8rr7rnr8r)headersid_key alignmentsr|)rrXrWr8rrr7)rr8rrr7rc3,K|] \}}| dyw)Nrr..0rrcs r0 zls..s Lvq#COs c3@K|]\}}| |jywrUrrs r0rzls..sWG&!SsS--Gs c3FK|]}t|jywrU)rNr#rrQs r0rzls..s NT^^!4s!c34K|]}|jywrUrrs r0rzls..sH<4T..<z Found z repo(s) for a total of z revision(s) and z on disk.)"rrr rrrrtyper BadParameterr*timerrrrsrXrWrarrllast_modified_strrhr7rtablelast_accessed_strrr)rNsumrjr boldr)rr#rrrrrpexcfiltersrrr filter_fnsrfnr~ sort_key_fnrrQrnrMr unique_reposrErevision_count total_sizes r0lsrWsk`O&y1 lG2=T]^G]4LSTGD*4?G T ))+C&-MgUE!HeAh1L5gM  8#5djj#A K LL['L :   19$$'MeWTU%VW W&5/#*  #*h#mm<<!^^$00!$X%;%;!< 11!)!;!;x}}- #*    G(  "#  #amm<<!^^--!%!7!7!=2!%!7!7}00y{CD #    L(  ,34GqG 4&  L LLNWGWWJ N NNNH<HHJ  II:,&>~>NO$Z01<  M O4S]]ODE3NOU 4  S*34 N 8$$SX.C 7 8  *  (5sv J/KKKL0L;+LA9M1A5M  M/ K8KKK L 'LL  ML;;M)zhf cache rm model/gpt2zhf cache rm z hf cache rm model/gpt2 --dry-runzhf cache rm model/gpt2 --yeszQOne or more repo IDs (e.g. model/bert-base-uncased) or revision hashes to delete.yesz-yz--yeszSkip confirmation prompt.dry_runz,Preview deletions without removing anything.c 8 t|}t ||}|j r:dj d|j D}tjd|t|jdk(r+tjdtjd|jt|j} t!|j"} g} | j$r| j'| j$d | j(r| j'| j(d | s| j'| j*d d j | } tjd | d | j,dt/|j"|r9tj0dd| j$| j*| j,ytj2d|| j5t!|j"} tj0d| j$d| j*d| j,d| j$| j*| j,y#t$r}td|j|d}~wwxYw)z(Remove cached repositories or revisions.rN c3&K|] }d| ywreNr.)rr~s r0rzrm..sK8Jud5'N8JzCould not find in cache: rzNothing to delete.codez repo(s)z revision(s)z and About to delete z totalling rDry run: no files were deleted.T)rrrr#r8zProceed with deletion?rDeleted z repo(s) and z revision(s); freed ) repos_deletedrevisions_deletedfreed)rrr rrr%rkrwarningrNr#rjrExitdelete_revisionsrhrRr$rErrFrGexpected_freed_size_strroresultconfirmexecute) rrrrrrpr resolutiondetailsstrategycounts summary_parts summary_texts r0rmrsKLO&y1 +='BJ))K 8J8JKK 0 :; :   A% %&jja  -}--vj6J6J/KLH !4!4 5F!M  1 12(;< $$ = =>lKL  ; ;}t d|jD} | s"| ||<|jd| D@t|dk(rtjdytt ||d} |jt| j} t|} tjd | j d | j"d t%||r.tj&d d | j | j"ytj(d|| j+tj&d| j d| j"d| j | j"y#t$r}td|j|d}~wwxYw)z)Remove detached revisions from the cache.rNc3RK|]}t|jdk(s|!yw)rN)rNr7rrns r0rzprune..\s"`n(HMMHZ^_H_Xns''c34K|]}|jywrUr`rs r0rzprune..`sGh(--hrrz2No unreferenced revisions found. Nothing to prune.r.rrz unreferenced revision(s) (z total).rT)rr#r8zProceed?rrz! unreferenced revision(s); freed r)rr)rrr rrrrr)r#updaterNrrjr"rrhrRrGrrorrr) rrrrrprr$r#rQdetachedrr r s r0pruner;s2O&y1 EGH%I##`dnn`` !GhGG $ 9~ EF$I&J .}--vj6J6J/KLH  *FHH 66677RS[SsSsRtt|}#8, -1111  KK $ JJ 6..//PQYQqQqPrrst 55..S O4S]]ODE3NOs F(( G1G  G)zhf cache verify gpt2z)hf cache verify gpt2 --revision refs/pr/1z.hf cache verify my-dataset --repo-type datasetrXrWrnzXCache directory to use when verifying files from cache (defaults to Hugging Face cache). local_dirz?If set, verify files under this directory instead of the cache.fail_on_missing_filesz--fail-on-missing-filesz?Fail if some files exist on the remote but are missing locally.fail_on_extra_filesz--fail-on-extra-fileszLFail if some files exist locally but are not present on the remote revision.tokenc |-|+tjdtjdt |} | j |t |dr |jn t|||||} d} | jr.sF &1V9+[:r!K.9IQRS[Q\P] ^&s$&u<❌ Checksum verification failed for the following file(s): rc3&K|] }d| ywrr.rps r0rzverify..sI4Hq$qc 4Hru6❌ Missing files (present remotely, absent locally): zM remote file(s) are missing locally. Use --fail-on-missing-files for details.c3&K|] }d| ywrr.r!s r0rzverify..sG4Fq$qc 4Fru4❌ Extra files (present locally, absent remotely): zV local file(s) do not exist on the remote repo. Use --fail-on-extra-files for details.zVerification failed for 'z' (z) in z. Revision: z Verified z file(s) for z 'z'. All checksums match.)rXrWcheckedr)rerrorrrrverify_repo_checksumshasattrrr* mismatchesrkrj missing_pathsrrN extra_paths verified_pathrnr checked_count)rXrWrnrrrrrrapir exit_coder verified_locations r0verifyr0s7b!6 WXjja  5 !C  & &%,Y%@)//c)n 'FI )) &&   PQXPYZ[   iiIF4H4HIIG HHNwiX YI KKv++,-.;;   iiGF4F4FGGG HHLWIV WI KKv))*+,99  ,,A~ 'yIOO3DEJ[I\\klrl{l{k| } jji((JJ F(()y6Gr'Rij//$$ " # r/)RrHrr collectionsrcollections.abcrr dataclassesrenumrtypingrr rhuggingface_hub.errorsr utilsr r rrrrrutils._parsingrr _cli_utilsrrrrrrr_outputrr cache_clir"compilerrrrrr_sort_options_dictrhr]r*r&rArDr-rr,r) RepoRefsMaprRrorvrrrrrrrrcommandautoOptionrIrArgumentrrmodelr0r.r/r0rDsQO #-!! +vvv7sss. > ?  $ "**ab6? 5  LM     * C!s*-d|#d|$+.%u #e}%  ="43xP  $>#5#<< = >9S>12 Tni8J.KKLTT$ [W^YWiMj=j5k [pt [(( n eN$667 78:("""6:" 4 [ ()"4> >)> ~1D8%@$FG>B54<S4(3#3%*uSRUX9V0WY]0]*^3l&[&49&Qd&X   4 9 9 I}  d  L  }   V  }   S D  m   } * +} , d 7   -} >  d  f   ?} J K} } @ (    4 9 97P  S d  P d  L  P    ,   P*  ?  +P6 7P8 9PPf -/IJK    4 9 9+G d  L  G    ,   G  ?  G* +G, -GLGT )..    4 9 9?g ggg d  k   g d  R  g %  %R  !g.#  #_  /g< =g> ?g@ Aggr/