Documentation
¶
Index ¶
- func AccountRole(account string) (string, bool)
- func ClearAccountRole(account string)
- func CopyBlobFromURLServerSide(ctx context.Context, dst AzurePath, sourceURL string, size int64, ...) error
- func CopyBlobServerSide(ctx context.Context, src AzurePath, dst AzurePath, concurrency int, ...) error
- func Delete(ctx context.Context, ap AzurePath) error
- func DeletePrefix(ctx context.Context, ap AzurePath) error
- func Download(ctx context.Context, ap AzurePath) ([]byte, error)
- func DownloadFile(ctx context.Context, ap AzurePath, file *os.File, concurrency int, ...) (int64, error)
- func DownloadStream(ctx context.Context, ap AzurePath) (io.ReadCloser, error)
- func HeadBlob(ctx context.Context, ap AzurePath) (int64, error)
- func IsBlobURL(raw string) bool
- func ListRecursiveStream(ctx context.Context, ap AzurePath, scanConcurrency int, ...) error
- func ListStream(ctx context.Context, ap AzurePath, cb func(BlobMeta) error) error
- func MkContainer(ctx context.Context, account, container string) error
- func PreAuthenticate(ctx context.Context, accounts ...string) error
- func RegisterAccountRole(account, role string)
- func SetHTTPTransport(rt http.RoundTripper)
- func SummarizeRecursive(ctx context.Context, ap AzurePath, scanConcurrency int, ...) (int64, int64, error)
- func Touch(ctx context.Context, ap AzurePath) error
- func Upload(ctx context.Context, ap AzurePath, data []byte) error
- func UploadFile(ctx context.Context, ap AzurePath, file *os.File, concurrency int, ...) error
- func UploadStream(ctx context.Context, ap AzurePath, reader io.Reader, concurrency int) error
- type AzurePath
- type BlobMeta
- type CopyProgress
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func AccountRole ¶ added in v0.1.19
AccountRole returns the role ("SRC" or "DST") registered for the given account, and a boolean indicating whether a role was registered.
func ClearAccountRole ¶ added in v0.1.19
func ClearAccountRole(account string)
ClearAccountRole removes the role registration and invalidates cached blob clients and delegation credentials created while that registration was active.
func CopyBlobFromURLServerSide ¶ added in v0.1.25
func CopyBlobFromURLServerSide(ctx context.Context, dst AzurePath, sourceURL string, size int64, concurrency int, onProgress CopyProgress) error
CopyBlobFromURLServerSide copies an external, publicly-readable HTTP(S) source URL directly into an Azure blob without streaming the bytes through this client. It is used for cross-backend server-side copy (e.g. Hugging Face → Azure), where the source URL is a public/signed CDN URL.
size is the exact source size in bytes when known; pass a negative value when the size is unknown. A known size uses the parallel StageBlockFromURL + CommitBlockList path (which must plan block IDs up front); an unknown size routes to Azure's async StartCopyFromURL, which transfers the whole blob server-side without a pre-known size.
func CopyBlobServerSide ¶ added in v0.1.12
func DeletePrefix ¶
DeletePrefix deletes all blobs under directory-like path
func DownloadFile ¶ added in v0.1.22
func DownloadFile(ctx context.Context, ap AzurePath, file *os.File, concurrency int, onProgress func(int64)) (int64, error)
DownloadFile downloads a blob to a local file using parallel ranged GETs. concurrency is the initial parallelism (and floor); an adaptive controller probes higher concurrency while throughput keeps improving, up to downloadHardConcurrencyCap (overridable via BBB_AZBLOB_DOWNLOAD_CONCURRENCY_MAX). The optional onProgress callback receives the cumulative number of bytes written. Returns the number of bytes downloaded.
func DownloadStream ¶ added in v0.1.8
func IsBlobURL ¶ added in v0.1.5
IsBlobURL performs a lightweight check whether the provided string is a blob endpoint URL.
func ListRecursiveStream ¶ added in v0.1.18
func ListStream ¶ added in v0.1.16
ListStream streams immediate children (non-recursive). If dir-like path provided, lists under it.
func MkContainer ¶
MkContainer creates a new Azure Blob container
func PreAuthenticate ¶ added in v0.1.19
PreAuthenticate eagerly authenticates to the given storage accounts sequentially. This ensures any interactive login popups happen one at a time before parallel workers start. It also pre-warms the blob client and UDC (User Delegation Credential) caches so that no credential acquisition happens during copy.
func RegisterAccountRole ¶ added in v0.1.19
func RegisterAccountRole(account, role string)
RegisterAccountRole tags the given storage account with a role ("SRC" or "DST"). When role-prefixed Azure identity environment variables are set (e.g. SRC_AZURE_TENANT_ID, DST_AZURE_CLIENT_ID, etc.), accounts tagged with the matching role will use those credentials instead of AzureCLI or interactive browser login. This makes authentication machine-friendly for CI/CD and multi-tenant environments. Changing the role invalidates cached blob clients and delegation credentials.
func SetHTTPTransport ¶ added in v0.1.21
func SetHTTPTransport(rt http.RoundTripper)
SetHTTPTransport installs a shared HTTP RoundTripper that will be used by every Azure SDK client constructed in this package. The RoundTripper is wrapped in an *http.Client so it satisfies policy.Transporter. Passing nil clears any previously configured transport.
This must be called before any SDK client is constructed; already-created clients keep their existing transport.
func SummarizeRecursive ¶ added in v0.1.26
func SummarizeRecursive(ctx context.Context, ap AzurePath, scanConcurrency int, onProgress func(count, size int64)) (int64, int64, error)
SummarizeRecursive counts blobs and bytes under ap. When the root contains multiple virtual directories, each directory is scanned independently so Azure continuation-token chains can advance in parallel.
func Touch ¶ added in v0.1.4
Touch ensures the blob exists by creating an empty object when missing.
func UploadFile ¶ added in v0.1.22
func UploadFile(ctx context.Context, ap AzurePath, file *os.File, concurrency int, onProgress func(int64)) error
UploadFile uploads a local file to a block blob using parallel ranged reads and StageBlock requests, mirroring azcopy's chunked upload for higher single-file throughput. concurrency is the initial parallelism (and floor); an adaptive controller probes higher concurrency while throughput keeps improving, up to uploadHardConcurrencyCap (overridable via BBB_AZBLOB_UPLOAD_CONCURRENCY_MAX). The optional onProgress callback receives the cumulative number of bytes staged.
If staging is rejected because stale uncommitted blocks poison the destination, the blob is cleared and the upload is retried once.
Types ¶
type AzurePath ¶
type AzurePath struct {
Account string
Container string
Blob string // may be empty or end with '/' for virtual directory
}
AzurePath represents an az:// path (account/container/blob)
func Parse ¶
Parse parses az://account/container[/blob] or https://account.blob.* URLs.
type BlobMeta ¶
BlobMeta minimal metadata for listing
func List ¶
List lists immediate children (non-recursive). If dir-like path provided, lists under it.
func ListContainers ¶
ListContainers lists all containers in the account
type CopyProgress ¶ added in v0.1.17
type CopyProgress func(copied, total int64)
CopyProgress is called during server-side copy with the number of bytes copied so far and the total size in bytes.
It may be invoked concurrently from multiple StageBlockFromURL goroutines, so implementations must be goroutine-safe. The copied value passed to a single CopyProgress is already clamped to be monotonically non-decreasing, even across an internal InvalidBlobOrBlock self-heal retry.
Source Files