#ti-VddlZddlZddlZddlmZddlmZddlmZGddeZ y) N)AbstractFileSysteminfer_storage_options) MemoryFileceZdZdZdZdZdZdZ dfd Ze dZ e ddZ e d Z e d Ze d Zdd Zdd Ze fdZedZ ddZddZddZdZxZS)GithubFileSystemaInterface to files in github An instance of this class provides the files residing within a remote github repository. You may specify a point in the repos history, by SHA, branch or tag (default is current master). For files less than 1 MB in size, file content is returned directly in a MemoryFile. For larger files, or for files tracked by git-lfs, file content is returned as an HTTPFile wrapping the ``download_url`` provided by the GitHub API. When using fsspec.open, allows URIs of the form: - "github://path/file", in which case you must specify org, repo and may specify sha in the extra args - 'github://org:repo@/precip/catalog.yml', where the org and repo are part of the URI - 'github://org:repo@sha/precip/catalog.yml', where the sha is also included ``sha`` can be the full or abbreviated hex of the commit you want to fetch from, or a branch or tag name (so long as it doesn't contain special characters like "/", "?", which would have to be HTTP-encoded). For authorised access, you must provide username and token, which can be made at https://github.com/settings/tokens z9https://api.github.com/repos/{org}/{repo}/git/trees/{sha}zChttps://api.github.com/repos/{org}/{repo}/contents/{path}?ref={sha}github)<r c t |d i|||_||_|du|duz r t d||_||_|||_|bd}tj|j||fd|ji|j} | j| jd}||_|jd ddlm} | d i||_y#t&$r d|_YywxYw) Nz%Auth required both username and tokenz)https://api.github.com/repos/{org}/{repo}orgrepotimeoutdefault_branchr)HTTPFileSystem)super__init__rr ValueErrorusernametokenrrequestsgetformatkwraise_for_statusjsonrootlshttprhttp_fs ImportError) selfrrsharrrkwargsurr __class__s l/Volumes/fast/ai/experiments/MLX_z-image/.venv/lib/python3.12/site-packages/fsspec/implementations/github.pyrzGithubFileSystem.__init__,s "6"  $ /DE E    "DL ;;A St,6:llFJggA    &&(+,C    ,)3F3DL DL sCC.-C.cR|jrd|j|jfiSiS)Nauth)rrr&s r,rzGithubFileSystem.kwJs& ==T]]DJJ78 8 ctjdddg|d|d|j}|j|j Dcgc]}|d c}Scc}w)avList repo names for given org or user This may become the top level of the FS Parameters ---------- org_or_user: str Name of the github org or user to query is_org: bool (default True) Whether the name is an organisation (True) or user (False) Returns ------- List of string zhttps://api.github.com/usersorgs/z/repos)rname)rrrrr )cls org_or_useris_orgr*rs r,reposzGithubFileSystem.reposPsf" LL%w&7&?%@+f UKK  )*2V 222s Actjd|jd|jdfd|ji|j }|j |jDcgc]}|d c}Scc}w)zNames of tags in the repohttps://api.github.com/repos/r4z/tagsrr5rrrrrrrr r&r*ts r,tagszGithubFileSystem.tagshsv LL+DHH:Qtyyk G LL gg  #$668,8a& 8,,,* A9ctjd|jd|jdfd|ji|j }|j |jDcgc]}|d c}Scc}w)zNames of branches in the repor;r4z /branchesrr5r<r=s r,brancheszGithubFileSystem.branchesssv LL+DHH:Qtyyk K LL gg  #$668,8a& 8,,,r@c4|j|jdS)z#Named references, tags and branchesr?rBrDr/s r,refszGithubFileSystem.refs~s t}}==r0c L|j|}|dk(r|xs |j}||jdjd}d}|xs |j}|D]e}|j |d||} ||rd|zn|z }| D cgc] } | d|k(s | } } | s t || d} | dd k(r |r| gcS|cS| d }g||j vs||jdfvrtj|jj|j|j| fd |ji|j} | jd k(r t || j!d dd} | j#dD cgc]=} | d| vr4|r |dz| dzn| d| d| | d| jdd| d d?} } ||jdfvr| |j |<n|j |} |r| St%| D cgc]} | d c} Scc} wcc} wcc} w)a List files at given path Parameters ---------- path: str Location to list, relative to repo root detail: bool If True, returns list of dicts, one per file; if False, returns list of full filenames only sha: str (optional) List at the given point in the repo history, branch or tag name or commit SHA _sha: str (optional) List this specific tree object (used internally to descend into trees) rNr4T)r'_shar5rtypefiler')rrr'r directory)blobtreerMpathmodesize)r5rOrHrPr')_strip_protocolr!rstripsplitr"FileNotFoundErrordircacherrurlrrrrr status_coderr sorted)r&rNdetailr'rGr(partsso_farpartoutor*typesfs r,r"zGithubFileSystem.lss] ##D) 2:#$))D <KK$**3/EF#$))Dggfd$g?#*D8"%=#Q6f)( $3s HHAH H!c8|jjyN)rUclear)r&rNs r,invalidate_cachez!GithubFileSystem.invalidate_caches r0cht|}d|vrt| |S|djdS)NrrNr4)rrrQlstrip)r6rNoptsr+s r,rQz GithubFileSystem._strip_protocols:$T* T !7*40 0F|""3''r0cXt|}d|vriS|d|dd}|dr|d|d<|S)Nrpasswordrhostr'r)rNrgr]s r,_get_kwargs_from_urlsz&GithubFileSystem._get_kwargs_from_urlssF$T* T !I:&Z0@A <fCJ r0c R|dk7rt|jj|j|j||xs |j }t j|fd|ji|j}|jdk(r t||j|j} | dr6tj| d} | j!ds t#dd| S|j$ t'd|j$j(| df|||d |S) NrbrrrNr'rrJcontents#version https://git-lfs.github.com/zRPlease install fsspec[http] to access github files >1 MB or git-lfs tracked files. download_url)rO block_size cache_options)NotImplementedError content_urlrrrr!rrrrrWrTrr base64 b64decode startswithrr$r%open) r&rNrOrqrrr'r(rVr* content_jsonros r,_openzGithubFileSystem._opens0 4<% %%%tyyt9I &  LL >dll >dgg > ==C #D) ) vvx   "&&|I'>?G %%&LM!$g66 << , !t||   ( !'     r0cr|j|||}t|D]}|j||y)N) recursivemaxdepth)message) expand_pathreversedrm_file)r&rNr|r}r~ps r,rmzGithubFileSystem.rm s7 HM$A LLGL , r0c |js td|j|}|j|}|s|jj |j |j|jd|j}tj|fd|ji|j}|jdk(r t||j!|j#d}|jj |j |j||j}|j}|xsd||d|rd |ini} tj$|f| |jd |j}|j#jd d } t'j(d | r d} t| |j!|j+|y)aW Remove a file from a specified branch using a given commit message. Since Github DELETE operation requires a branch name, and we can't reliably determine whether the provided SHA refers to a branch, tag, or commit, we assume it's a branch. If it's not, the user will encounter an error when attempting to retrieve the file SHA or delete the file. Parameters ---------- path: str The file's location relative to the repository root. message: str, optional The commit message for the deletion. zAuthentication requiredr4rnrrJr'zDelete )r~r'branch)r rr~rzBranch .+ not foundzTRemove only works when the filesystem is initialised from a branch or default (None)N)rrrQ_get_sha_from_cachertrrrrfr!rrrrrWrTrr deleteresearchrd) r&rNr~r(r'rVr* delete_urlrdata error_messageerrors r,rzGithubFileSystem.rm_files"}}67 7##D)&&t,""))HH4994;;s3C*C SB$,,B$''BA}}#'--   &&(5/C%%,,tyyt- 2GD6"2 &,&!  OOJ ST4<< S477 S Y3 99+] ;jEU# #  d#r0c|jjD]0}|D])}|jd}|s||k(sd|vs"|dccS2y)Nr5r')rUvaluesr)r&rNentriesentry entry_paths r,rz$GithubFileSystem._get_sha_from_cacheGsP}}++-G "YYv. *"4% <'!. r0)NNNN)T)FNNrb)rmNNN)__name__ __module__ __qualname____doc__rVrtprotocolrrpropertyr classmethodr9r?rBrEr"rdrQ staticmethodrkrzrrr __classcell__)r+s@r,r r s6 FCWKHGGK < 33.---->>@4D((   1 f- 4$lr0r ) rurrspecrutilsrmemoryrr rr0r,rs& %)B)Br0