transfer

package
v0.40.13 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Oct 28, 2026 License: MIT Imports: 26 Imported by: 0

Documentation

Overview

Package transfer provides minimal, fast blob transfer for tensor-based models.

This package is in x/ because the tensor model storage format is under development. It provides optimized transfer for models with many small blobs (tensor models) rather than few large blobs (typical LLMs).

TODO (jmorganca): Integrate into server/download.go and server/upload.go when stable.

Design Philosophy: This package is intentionally simpler than the main server's download/upload code. Key simplifications for many-small-blob workloads:

  • Whole-blob downloads: Each blob downloads as one unit, with HTTP Range resume for blobs >= 64MB on retry; small blobs restart from scratch.
  • Whole-blob uploads by default: A single PUT per blob. When the server returns a direct-upload URL the body goes straight to the storage backend; otherwise the body goes to the registry in one shot.
  • Multi-part upload fallback: If the server requires it, blobs are split into parts and sent via PATCH with a finalize PUT carrying a composite etag. This is a server-side compatibility path, not the fast path.
  • Inline hashing: digests computed during streaming.
  • Stall and speed detection (downloads): cancels on no data (stall) or speed below 10% of median.

For large models (multi-GB), use the server's download/upload code which has resumable downloads with JSON state files, async hashing from OS page cache, and per-part speed tracking with rolling median.

Index

Constants

View Source
const (
	DefaultDownloadConcurrency = 64
	DefaultUploadConcurrency   = 64
)

Default concurrency limits and settings

Variables

This section is empty.

Functions

func Download

func Download(ctx context.Context, opts DownloadOptions) error

Download downloads blobs in parallel with streaming hash verification.

func NewRedirectClient

func NewRedirectClient(baseURL string, allowPrivate bool) *http.Client

NewRedirectClient returns a client for download URLs supplied by a registry.

func Upload

func Upload(ctx context.Context, opts UploadOptions) error

Upload uploads blobs in parallel.

func ValidateRedirectScheme

func ValidateRedirectScheme(loc *url.URL, baseURL string) error

ValidateRedirectScheme rejects redirects that downgrade an https session to cleartext http, even when the target host is unchanged — a hostile or compromised registry must not be able to strip TLS off follow-up requests. It applies even under allowPrivate (the --insecure opt-in), which relaxes address checks but never scheme checks.

func ValidateRedirectTarget

func ValidateRedirectTarget(ctx context.Context, loc *url.URL, baseURL string, allowPrivate bool) error

ValidateRedirectTarget checks a registry-supplied target. When a proxy carries the request, or the target is a default-registry host we redirect to anyway, name resolution is deferred — the proxy enforces its own access rules and no local DNS is needed. Otherwise the target must resolve locally to public addresses. Explicit private IP literals are always rejected unless allowPrivate is set.

Types

type AuthChallenge

type AuthChallenge struct {
	Realm   string
	Service string
	Scope   string
}

AuthChallenge represents a parsed WWW-Authenticate challenge.

type Blob

type Blob struct {
	Digest string // sha256:...
	Size   int64

	// From enables cross-repository blob mounting (upload only).
	// When set, the upload will first attempt to mount the blob from this source
	// repository instead of uploading the data. This is a Docker Registry v2 API
	// feature that avoids re-uploading blobs that already exist elsewhere.
	//
	// Example: From="library/source-model" will add ?mount=<digest>&from=library/source-model
	// to the POST /blobs/uploads/ request. If the registry returns 201 Created,
	// the blob was mounted successfully and no upload is needed.
	//
	// See: https://distribution.github.io/distribution/spec/api/#cross-repository-blob-mount
	From string
}

Blob represents a content-addressed blob to transfer.

type DownloadOptions

type DownloadOptions struct {
	Blobs             []Blob                                                             // Blobs to download
	BaseURL           string                                                             // Registry base URL
	DestDir           string                                                             // Destination directory for blobs
	Repository        string                                                             // Repository path for blob URLs (e.g., "library/model")
	Concurrency       int                                                                // Max parallel downloads (default DefaultDownloadConcurrency)
	BodyConcurrency   int                                                                // Max simultaneous body-bearing transfers; 0 or negative serializes (capacity 1)
	Progress          func(completed, total int64)                                       // Progress callback (optional)
	Client            *http.Client                                                       // HTTP client (optional, uses default)
	Token             string                                                             // Auth token (optional)
	GetToken          func(ctx context.Context, challenge AuthChallenge) (string, error) // Token refresh callback
	Logger            *slog.Logger                                                       // Optional structured logger
	UserAgent         string                                                             // User-Agent header (optional, has default)
	StallTimeout      time.Duration                                                      // Timeout for stall detection (default 10s)
	AllowPrivateHosts bool                                                               // Permits cross-host redirects to local addresses
}

DownloadOptions configures a parallel download operation.

type UploadOptions

type UploadOptions struct {
	Blobs             []Blob                                                             // Blobs to upload
	BaseURL           string                                                             // Registry base URL
	SrcDir            string                                                             // Source directory containing blobs
	Concurrency       int                                                                // Max parallel uploads (default DefaultUploadConcurrency)
	BodyConcurrency   int                                                                // Max simultaneous body-bearing transfers; 0 or negative serializes (capacity 1)
	Progress          func(completed, total int64)                                       // Progress callback (optional)
	Client            *http.Client                                                       // HTTP client (optional, uses default)
	Token             string                                                             // Auth token (optional)
	GetToken          func(ctx context.Context, challenge AuthChallenge) (string, error) // Token refresh callback
	Logger            *slog.Logger                                                       // Optional structured logger
	UserAgent         string                                                             // User-Agent header (optional, has default)
	AllowPrivateHosts bool                                                               // Permits cross-host redirects to local addresses

	// Manifest fields (optional) - if set, manifest is pushed after all blobs complete
	Manifest          []byte // Raw manifest JSON to push
	ManifestMediaType string // Manifest Content-Type; required when Manifest is set
	ManifestRef       string // Tag or digest for the manifest (e.g., "latest", "sha256:...")
	Repository        string // Repository path for manifest URL (e.g., "library/model")
}

UploadOptions configures a parallel upload operation.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL