bbbfs

package
v0.1.26 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Sep 8, 2026 License: MIT Imports: 22 Imported by: 0

Documentation

Index

Constants

View Source 
const ACRScheme = "acr://"

ACRScheme is the scheme prefix for Azure Container Registry paths.

View Source 
const HFScheme = "hf://"

HFScheme is the scheme prefix for Hugging Face paths.

Variables

View Source 
var ErrWriteUnsupported = errors.New("bbbfs: write not supported")

ErrWriteUnsupported indicates that a backend does not support writes.

Functions

func AzAccountContainer added in v0.1.18 

func AzAccountContainer(p string) (account, container string, err error)

AzAccountContainer returns the account and container from an Azure path. Returns empty strings for non-Azure paths.

func BaseName added in v0.1.18 

func BaseName(p string) string

BaseName returns the base filename from any path.

func CanCopyServerSide added in v0.1.18 

func CanCopyServerSide(src, dst string) bool

CanCopyServerSide returns true when both src and dst can use server-side copy. Server-side copy is only valid within the same provider (Azure→Azure or S3→S3, GCS→GCS), never across providers.

func CanCopyServerSideFromURL added in v0.1.25 

func CanCopyServerSideFromURL(src, dst string) bool

CanCopyServerSideFromURL returns true when src can produce a public source URL and dst can copy from a URL server-side (e.g. Hugging Face → Azure).

func CanDownloadToFile added in v0.1.22 

func CanDownloadToFile(src string) bool

CanDownloadToFile reports whether src's backend supports an optimized download-to-local-file path.

func CanUploadFromFile added in v0.1.22 

func CanUploadFromFile(dst string) bool

CanUploadFromFile reports whether dst's backend supports an optimized upload-from-local-file path.

func ChildPath added in v0.1.18 

func ChildPath(parent, child string) string

ChildPath joins child to parent using the backend-specific separator.

func CopyServerSide added in v0.1.18 

func CopyServerSide(ctx context.Context, src, dst string, concurrency int, sizeHint int64, onProgress CopyProgress) error

CopyServerSide performs an optimised server-side copy (e.g. Azure→Azure). sizeHint, when > 0, avoids a HeadBlob round-trip for the source size. Returns an error if the backends do not support server-side copy.

func CopyServerSideFromURL added in v0.1.25 

func CopyServerSideFromURL(ctx context.Context, src, dst string, concurrency int, onProgress CopyProgress) error

CopyServerSideFromURL resolves a public source URL from src and asks dst to copy from it server-side, without streaming the bytes through this process.

func Delete added in v0.1.18 

func Delete(ctx context.Context, p string) error

Delete removes a file at the given path.

func DeletePrefix added in v0.1.18 

func DeletePrefix(ctx context.Context, p string) error

DeletePrefix deletes all files under the prefix. Only Azure supports this natively.

func DownloadToFile added in v0.1.22 

func DownloadToFile(ctx context.Context, src, localPath string, concurrency int, onProgress func(int64)) (int64, error)

DownloadToFile downloads the remote file at src into localPath using a backend-optimized transfer (parallel ranged GETs for Azure). onProgress, when non-nil, receives the cumulative number of bytes written. Returns the number of bytes downloaded.

func Exists added in v0.1.18 

func Exists(ctx context.Context, p string) (bool, error)

Exists checks whether a file exists at the given path.

func ExistsAsBlob added in v0.1.18 

func ExistsAsBlob(ctx context.Context, p string) (bool, error)

ExistsAsBlob checks whether the path points to an existing non-directory file. Returns false if the path is directory-like or does not exist.

func IsACR added in v0.1.26 

func IsACR(path string) bool

IsACR returns true if the path targets an Azure Container Registry backend.

func IsAz added in v0.1.18 

func IsAz(path string) bool

IsAz returns true if the path targets an Azure Blob Storage backend.

func IsDirLike added in v0.1.18 

func IsDirLike(ctx context.Context, p string) (bool, error)

IsDirLike checks whether the path is directory-like. For remote paths this is determined from the path structure; for local paths os.Stat is used.

func IsDirLikeFromPath added in v0.1.18 

func IsDirLikeFromPath(p string) bool

IsDirLikeFromPath checks if a path is directory-like without making any network calls. Uses path structure only.

func IsGS added in v0.1.26 

func IsGS(path string) bool

IsGS returns true if the path targets a Google Cloud Storage backend.

func IsHF added in v0.1.18 

func IsHF(path string) bool

IsHF returns true if the path targets a Hugging Face backend.

func IsNonRetryableHTTPErr added in v0.1.18 

func IsNonRetryableHTTPErr(err error) bool

IsNonRetryableHTTPErr reports whether err represents a failure that retrying cannot fix. That covers HTTP 401, 403 and 404 from any supported backend, plus, for ACR, a destination artifact that already exists, and any backend error exposing NotFound() == true.

A bare 409 or 412 is deliberately absent: HTTPStatusError does not say which request produced it, so treating those as final would also condemn a blob upload or an overwrite push that a retry would complete. The one conflict that is genuinely final — a create-only manifest write whose tag already holds something else — is reported as ErrArtifactExists instead.

A registry 404 is likewise not always final: an upload session the registry has forgotten answers BLOB_UPLOAD_UNKNOWN, and starting the upload again is exactly what recovers it, so the error code decides rather than the status.

func IsObjectStore added in v0.1.25 

func IsObjectStore(path string) bool

IsObjectStore returns true if the path targets a remote object-store backend with virtual-directory semantics and Stat-based existence checks (Azure Blob Storage, Amazon S3 or Google Cloud Storage).

func IsRemote added in v0.1.18 

func IsRemote(path string) bool

IsRemote returns true if the path targets a remote (non-local) backend.

func IsS3 added in v0.1.25 

func IsS3(path string) bool

IsS3 returns true if the path targets an Amazon S3 (or S3-compatible) backend.

func ListFilesFlat added in v0.1.18 

func ListFilesFlat(ctx context.Context, p string) ([]string, error)

ListFilesFlat returns a flat list of relative file names under the path.

func ListRecursive 

func ListRecursive(ctx context.Context, root string) <-chan ListResult

ListRecursive returns a channel that streams all files under the path. Entries are emitted as they are discovered; any listing error is sent as a ListResult with Err set. The channel is closed when listing completes or the context is cancelled. Callers should cancel the context if they stop consuming the channel early.

func ListRecursiveWithSizeStream added in v0.1.18 

func ListRecursiveWithSizeStream(ctx context.Context, p string, emit func(Entry) error) error

ListRecursiveWithSizeStream streams all entries recursively with their sizes via a callback. If the backend does not implement streaming, it falls back to collecting all entries and emitting them one by one.

func ListStream added in v0.1.18 

func ListStream(ctx context.Context, p string, fn func(Entry) error) error

ListStream lists entries via a streaming callback. Falls back to List when the backend does not provide a streaming implementation.

func MkDir added in v0.1.18 

func MkDir(ctx context.Context, p string) error

MkDir creates a directory or container at the given path.

func ParseShareInfo added in v0.1.18 

func ParseShareInfo(p string) (portal, direct string, err error)

ParseShareInfo returns shareable links for the path.

func PreAuthenticateAz added in v0.1.19 

func PreAuthenticateAz(ctx context.Context, paths ...string) error

PreAuthenticateAz eagerly authenticates to the storage accounts referenced by the given az:// paths. Call this before spawning parallel workers so that any interactive login popups happen sequentially.

func Register 

func Register(provider FS)

Register adds a filesystem provider.

func RegisterAzAccountRoles added in v0.1.19 

func RegisterAzAccountRoles(srcPaths, dstPaths []string)

RegisterAzAccountRoles tags source and destination storage accounts with their roles so that SRC_AZURE_* / DST_AZURE_* environment variables are used for authentication in multi-tenant environments.

If the same account appears in both srcPaths and dstPaths it is not tagged with any role. Identity credential lookup for untagged accounts still uses SRC_AZURE_* overrides and unprefixed AZURE_* defaults.

The function is safe to call repeatedly with a growing set of paths (as happens when task pairs are streamed): an account previously tagged with a role has its tag cleared once it is seen in both roles.

func ResolveDstPath added in v0.1.18 

func ResolveDstPath(dst, base string, mustBeDir bool) (string, error)

ResolveDstPath computes the final destination file path for a copy operation. If the destination is directory-like, base is appended.

func ScanConcurrency added in v0.1.18 

func ScanConcurrency(ctx context.Context) int

ScanConcurrency returns the scan concurrency stored in ctx, or 1 if unset.

func Touch added in v0.1.18 

func Touch(ctx context.Context, p string) error

Touch creates an empty file or updates the modification time.

func UploadArtifact added in v0.1.26 

func UploadArtifact(ctx context.Context, dst string, files []ArtifactFile, concurrency int, overwrite bool, onProgress func(name string, uploaded int64)) error

UploadArtifact publishes files atomically as one backend artifact.

onProgress, when non-nil, receives a file's name and how many of its bytes have been uploaded so far. The value is cumulative per file and may restart at zero when a transfer is retried, so callers must track a high-water mark per name instead of treating each call as a delta.

func UploadConcurrency added in v0.1.19 

func UploadConcurrency(ctx context.Context) int

UploadConcurrency returns the upload concurrency stored in ctx, or 1 if unset.

func UploadFromFile added in v0.1.22 

func UploadFromFile(ctx context.Context, localPath, dst string, concurrency int, onProgress func(int64)) (int64, error)

UploadFromFile uploads the local file at localPath to dst using a backend-optimized transfer (parallel StageBlock for Azure). onProgress, when non-nil, receives the cumulative number of bytes staged. Returns the number of bytes uploaded.

func UploadReader added in v0.1.25 

func UploadReader(ctx context.Context, dst string, r io.Reader, concurrency int, onProgress func(copied int64)) error

UploadReader writes r to dst using the backend's most reliable upload path, falling back to a plain streaming Write when the backend does not implement readerUploader. onProgress, when non-nil, receives cumulative bytes uploaded.

func WithScanConcurrency added in v0.1.18 

func WithScanConcurrency(ctx context.Context, n int) context.Context

WithScanConcurrency returns a context that carries the scan (listing) concurrency hint. Backends that support parallel prefix walking (e.g. Azure) use this to bound the number of concurrent listing goroutines.

func WithUploadConcurrency added in v0.1.19 

func WithUploadConcurrency(ctx context.Context, n int) context.Context

WithUploadConcurrency returns a context that carries the upload concurrency hint for streaming uploads.

Types

type ArtifactFile added in v0.1.26 

type ArtifactFile struct {
	Name string
	Size int64
	Open func() (io.ReadCloser, error)
}

ArtifactFile describes one file to publish in an artifact-oriented backend. Open must return a fresh reader for each call.

type CopyProgress added in v0.1.18 

type CopyProgress = func(copied, total int64)

CopyProgress is the callback type for copy progress reporting.

type Entry 

type Entry struct {
	Name    string
	Path    string
	Size    int64
	IsDir   bool
	ModTime time.Time
}

Entry is a filesystem entry metadata.

func ListRecursiveWithSize added in v0.1.18 

func ListRecursiveWithSize(ctx context.Context, p string) ([]Entry, error)

ListRecursiveWithSize lists all entries recursively with their sizes.

type FS 

type FS interface {
	Match(path string) bool
	Read(ctx context.Context, path string) (io.ReadCloser, error)
	Write(ctx context.Context, path string, r io.Reader) error
	List(ctx context.Context, path string) ([]Entry, error)
	Stat(ctx context.Context, path string) (Entry, error)
}

FS provides abstract access for supported backends.

func Resolve 

func Resolve(path string) FS

Resolve returns the first filesystem provider that matches the path.

type ListResult 

type ListResult struct {
	Entry Entry
	Err   error
}

ListResult wraps an Entry with an optional error from a listing operation.

type RecursiveSummary added in v0.1.26 

type RecursiveSummary struct {
	FileCount int64
	TotalSize int64
}

RecursiveSummary contains aggregate metadata for a recursive file listing.

func SummarizeRecursive added in v0.1.26 

func SummarizeRecursive(ctx context.Context, root string, onProgress func(count, size int64)) (RecursiveSummary, error)

SummarizeRecursive returns aggregate file count and size without retaining individual entries. Backends may optimize this operation independently.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.