s3vfs

package
v0.12.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Jun 6, 2026 License: Apache-2.0 Imports: 14 Imported by: 0

Documentation

Overview

Package s3vfs implements a vfs.VFS backend backed by an S3-compatible object store (AWS S3, MinIO, Ceph RGW, Cloudflare R2, Backblaze B2, Aliyun OSS, Tencent COS, ...). It is meant to be plugged into httpcache-kit via NewVFSCacheWithConfig so that apt-proxy can serve its cached APT/RPM/Alpine packages straight from object storage with no changes to the proxy or distro logic.

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

This section is empty.

Types

type Config

type Config struct {
	// Endpoint is the host[:port] of the S3-compatible service, e.g.
	// "s3.amazonaws.com", "s3.us-west-2.amazonaws.com",
	// "minio.local:9000", "s3.cn-north-1.qiniucs.com".
	Endpoint string
	// Region is the optional region; required for AWS S3, ignored by most
	// MinIO-flavoured services.
	Region string
	// Bucket is the destination bucket. It must already exist; New()
	// performs a HeadBucket to validate.
	Bucket string
	// Prefix is an optional sub-path inside the bucket, e.g. "apt-proxy/",
	// allowing multiple cache instances to share one bucket. Trailing/leading
	// slashes are normalised away by the package.
	Prefix string
	// AccessKey / SecretKey are the static IAM credentials. Pass empty
	// strings to use anonymous access (rare) or the SDK's default chain.
	AccessKey string
	SecretKey string
	// SessionToken is the optional STS session token (for AssumeRole/IRSA);
	// leave empty for static long-lived credentials.
	SessionToken string
	// UseSSL toggles HTTPS for the S3 endpoint. Defaults to true; explicit
	// false is required only for plain-HTTP MinIO test deployments.
	UseSSL bool
	// UsePathStyle forces path-style URLs (https://endpoint/bucket/key)
	// instead of virtual-hosted-style (https://bucket.endpoint/key). MinIO,
	// Ceph and most third-party services need this set to true.
	UsePathStyle bool
	// InlineMaxBytes is the in-memory write buffer threshold above which a
	// write spills to a temporary file. Zero falls back to inlineSpillThreshold.
	InlineMaxBytes int64
	// TempDir is where spilled writes go. Empty means os.TempDir().
	TempDir string
	// MetaCacheSize controls how many StatObject results are kept in the LRU.
	// Zero falls back to defaultMetaCacheSize.
	MetaCacheSize int
	// MetaCacheTTL bounds how long a cached fileinfo is honored before we
	// re-check S3. Zero means no time-based invalidation; size eviction
	// alone is sufficient for most workloads.
	MetaCacheTTL time.Duration
	// HTTPClient lets tests inject a custom round-tripper. Production callers
	// leave this nil to get minio-go's default transport.
	HTTPClient *http.Client
}

Config configures the S3 backend. It mirrors the operator-facing config.S3Config; New() takes a Config rather than the apt-proxy struct so this package stays import-cycle-clean and individually unit-testable.

type S3VFS

type S3VFS struct {
	// contains filtered or unexported fields
}

S3VFS implements the vfs.VFS interface against an S3-compatible bucket.

func New

func New(ctx context.Context, cfg Config) (*S3VFS, error)

New constructs a new S3VFS. It validates connectivity by performing a BucketExists call; failing fast at startup is far less surprising than crashing on the first cache miss in production.

func (*S3VFS) Bucket

func (s *S3VFS) Bucket() string

Bucket returns the configured bucket name. Used by health checks.

func (*S3VFS) HealthCheck

func (s *S3VFS) HealthCheck(ctx context.Context) error

HealthCheck is a context-aware liveness probe suitable for plugging into health-kit. It performs a single HeadBucket round-trip.

func (*S3VFS) Lstat

func (s *S3VFS) Lstat(p string) (os.FileInfo, error)

Lstat is identical to Stat for S3: there are no symlinks.

func (*S3VFS) Mkdir

func (s *S3VFS) Mkdir(_ string, _ os.FileMode) error

Mkdir is a no-op on S3: there are no directories per se, just key prefixes. Returning nil keeps vfs.MkdirAll happy.

func (*S3VFS) Open

func (s *S3VFS) Open(p string) (vfs.RFile, error)

Open returns a readable file handle. The minio.Object returned by GetObject is lazy: the network request happens on first Read, which lets us cheaply construct a handle even if the caller decides to abort.

func (*S3VFS) OpenFile

func (s *S3VFS) OpenFile(p string, flag int, _ os.FileMode) (vfs.WFile, error)

OpenFile honours O_CREATE|O_TRUNC|O_WRONLY (the only flag combination httpcache-kit currently passes). Reading via the returned WFile is not supported because S3 has no append-or-update semantics that map cleanly onto a Go file handle.

func (*S3VFS) ReadDir

func (s *S3VFS) ReadDir(p string) ([]os.FileInfo, error)

ReadDir lists the immediate children of the given vfs directory using a delimited ListObjects. Both real objects and CommonPrefixes (synthetic subdirectories) are returned.

func (*S3VFS) Remove

func (s *S3VFS) Remove(p string) error

Remove deletes a single object by exact key. If the path looks like a directory (ends with "/" or is identifiable as a virtual dir), we recursively delete all keys under it. This lets httpcache-kit's Purge() do its job correctly.

func (*S3VFS) Stat

func (s *S3VFS) Stat(p string) (os.FileInfo, error)

Stat returns os.FileInfo for the given path. We treat both real S3 objects and synthetic "directories" (anything that has children) as valid Stat targets. Results are cached to absorb httpcache-kit's chatty Header() access pattern.

func (*S3VFS) String

func (s *S3VFS) String() string

String implements vfs.VFS. The format mirrors the local-FS String() output pattern so debug logs stay uniform.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL