Documentation
¶
Overview ¶
Package objectstore is the S3-compatible storage layer shared by log archive and other features that write to a connected storage destination.
Index ¶
- Constants
- Variables
- func AppPrefix(root, appName string) string
- func Classify(err error) (string, string)
- func IsArchiveKey(root, key string) bool
- func ObjectKey(root, resourceID string, startNs int64) string
- func PresetForProvider(provider string) string
- func ResourceID(appName string) string
- func ValidateEndpoint(endpoint string) error
- type ArchiveStore
- type Archiver
- type BuildCache
- func (b *BuildCache) Clear(ctx context.Context, app string) (ClearResult, error)
- func (b *BuildCache) Effective(ctx context.Context, app string) (store.BuildCacheSetting, bool, error)
- func (b *BuildCache) KeyPrefix(app string) (string, error)
- func (b *BuildCache) RecordBuild(ctx context.Context, app, warning string)
- func (b *BuildCache) S3Cache(ctx context.Context, app string) (*build.S3Cache, error)
- func (b *BuildCache) Stats(ctx context.Context, app string) (CacheStats, error)
- type BuildCacheOptions
- type BuildCacheResolver
- type BuildCacheStore
- type CacheStats
- type ClearResult
- type Client
- func (c *Client) Delete(ctx context.Context, key string) error
- func (c *Client) Get(ctx context.Context, key string) (io.ReadCloser, int64, error)
- func (c *Client) List(ctx context.Context, prefix, token string, limit int32) (ListPage, error)
- func (c *Client) Put(ctx context.Context, key string, body io.ReadSeeker, ...) error
- type Config
- type DumpRequest
- type HealthSource
- type ListPage
- type LogReader
- type Object
- type Options
- type Preset
- type ProbeResult
- type ProbeStep
- type Resolved
- type Resolver
- type SecretsResolver
- type TargetStore
Constants ¶
const ( EnvArchivePrefix = "APP_LOG_ARCHIVE_PREFIX" EnvArchiveMaxLines = "APP_LOG_ARCHIVE_MAX_LINES_PER_OBJECT" EnvArchiveMaxWindows = "APP_LOG_ARCHIVE_MAX_WINDOWS_PER_RUN" EnvArchiveBackfill = "APP_LOG_ARCHIVE_MAX_BACKFILL" EnvArchiveTick = "APP_LOG_ARCHIVE_TICK" )
Env vars tuning the archiver; unset or invalid values use the defaults.
const ( EnvBuildCachePrefix = "APP_BUILD_CACHE_PREFIX" EnvBuildCacheClearMaxObjects = "APP_BUILD_CACHE_CLEAR_MAX_OBJECTS" EnvBuildCacheStatsMaxObjects = "APP_BUILD_CACHE_STATS_MAX_OBJECTS" )
Env names for the build cache bounds.
const ( PresetAWS = "aws" PresetR2 = "r2" PresetB2 = "b2" PresetMinIO = "minio" PresetWasabi = "wasabi" PresetCustom = "custom" )
Preset identifies a well-known S3-compatible provider.
const ( ReasonInvalidCredentials = "invalid_credentials" //nolint:gosec // reason code, not a credential ReasonAccessDenied = "access_denied" ReasonBucketNotFound = "bucket_not_found" ReasonRegionMismatch = "region_mismatch" ReasonEndpointBlocked = "endpoint_blocked" ReasonTLS = "tls_error" ReasonUnreachable = "unreachable" ReasonIntegrity = "integrity_mismatch" ReasonUnknown = "unknown" )
Failure reason codes returned by Probe.
Variables ¶
var ErrBuildCacheNotConfigured = errors.New("objectstore: build cache is not configured for this app")
ErrBuildCacheNotConfigured is returned when an app has no enabled build cache setting.
var ErrInvalidBuildCacheApp = errors.New("objectstore: invalid app name for build cache")
ErrInvalidBuildCacheApp is returned for an app name that cannot be a key prefix.
var ErrInvalidDump = errors.New("objectstore: invalid dump request")
ErrInvalidDump is returned for a bad dump request.
Functions ¶
func Classify ¶
Classify maps an S3 or transport error to a stable reason code and a message safe to show (it never contains credentials).
func IsArchiveKey ¶
IsArchiveKey reports whether key is under the archive root and free of traversal.
func ObjectKey ¶
ObjectKey is root/<kind>/<name>/yyyy/mm/dd/hh/<start ns>.ndjson.gz. It is keyed by the chunk start only, so retrying a chunk overwrites it instead of leaving a duplicate; chunks are contiguous and sort by name.
func PresetForProvider ¶
PresetForProvider infers a preset for a backup target created without one.
func ResourceID ¶
ResourceID is the telemetry resource ID a log line of appName is stored under.
func ValidateEndpoint ¶
ValidateEndpoint checks scheme and host shape. Internal addresses are refused at dial time by netguard, not here, so DNS tricks cannot bypass it.
Types ¶
type ArchiveStore ¶
type ArchiveStore interface {
ListLogArchivePolicies(ctx context.Context) ([]store.LogArchivePolicy, error)
RecordLogArchivePolicyRun(ctx context.Context, id, at string, watermarkNs int64, runErr string) error
InsertLogArchiveRun(ctx context.Context, r store.LogArchiveRun) error
FinishLogArchiveRun(ctx context.Context, r store.LogArchiveRun) error
FailStaleLogArchiveRuns(ctx context.Context, at string) error
}
ArchiveStore is the persistence surface the archiver needs.
type Archiver ¶
type Archiver struct {
Store ArchiveStore
Clients *Resolver
Logs LogReader
Logger *slog.Logger
Opts Options
Now func() time.Time
// contains filtered or unexported fields
}
Archiver ships node-local logs to a storage destination as gzip NDJSON. One run at a time, so a slow bucket applies backpressure instead of piling up goroutines.
func NewArchiver ¶
func NewArchiver(st ArchiveStore, clients *Resolver, logs LogReader, logger *slog.Logger, opts Options) *Archiver
NewArchiver builds an Archiver.
func (*Archiver) Run ¶
Run checks for due policies every Opts.Tick until ctx ends. With no policies a tick is one small query, so idle cost stays near zero.
func (*Archiver) RunDue ¶
RunDue runs every enabled policy whose interval has elapsed since its last run.
func (*Archiver) RunPolicy ¶
RunPolicy archives everything a policy has not shipped yet, up to MaxWindowsPerRun hour windows, then advances its watermark. A failed window keeps the watermark in place so the next run retries it.
func (*Archiver) StartDump ¶
func (a *Archiver) StartDump(ctx context.Context, req DumpRequest) (store.LogArchiveRun, error)
StartDump validates a manual dump, records a running row, and archives the range in the background. Poll the run row (or Wait) for the outcome.
type BuildCache ¶
type BuildCache struct {
Store BuildCacheStore
Resolver BuildCacheResolver
Options BuildCacheOptions
Logger *slog.Logger
// ResolveHost is net.DefaultResolver.LookupNetIP unless a test overrides it.
ResolveHost func(ctx context.Context, network, host string) ([]netip.Addr, error)
}
BuildCache resolves per-app BuildKit remote cache settings against storage destinations and runs cache hygiene operations on the bucket.
func (*BuildCache) Clear ¶
func (b *BuildCache) Clear(ctx context.Context, app string) (ClearResult, error)
Clear deletes objects under the app's cache prefix, up to ClearMaxObjects.
func (*BuildCache) Effective ¶
func (b *BuildCache) Effective(ctx context.Context, app string) (store.BuildCacheSetting, bool, error)
Effective returns the setting an app builds with: its own row when present (an explicit disable wins), else the global default. The bool is false when no enabled setting applies.
func (*BuildCache) KeyPrefix ¶
func (b *BuildCache) KeyPrefix(app string) (string, error)
KeyPrefix is the per-app bucket prefix the cache lives under.
func (*BuildCache) RecordBuild ¶
func (b *BuildCache) RecordBuild(ctx context.Context, app, warning string)
RecordBuild stores the latest build outcome on the setting the app resolved to. Failures are logged, never returned: recording must not fail a build.
func (*BuildCache) S3Cache ¶
S3Cache builds the BuildKit s3 cache config for app, or nil when the app has no enabled setting. The result carries credentials.
func (*BuildCache) Stats ¶
func (b *BuildCache) Stats(ctx context.Context, app string) (CacheStats, error)
Stats lists the app's cache prefix, bounded by StatsMaxObjects.
type BuildCacheOptions ¶
BuildCacheOptions bounds the build cache bucket operations.
func BuildCacheOptionsFromEnv ¶
func BuildCacheOptionsFromEnv(lookup func(string) (string, bool)) BuildCacheOptions
BuildCacheOptionsFromEnv applies env overrides to the defaults.
func DefaultBuildCacheOptions ¶
func DefaultBuildCacheOptions() BuildCacheOptions
DefaultBuildCacheOptions returns the defaults.
type BuildCacheResolver ¶
type BuildCacheResolver interface {
Config(ctx context.Context, targetID string) (Config, store.BackupTarget, error)
Client(ctx context.Context, targetID string) (*Client, store.BackupTarget, error)
}
BuildCacheResolver turns a storage destination id into credentials and a client.
type BuildCacheStore ¶
type BuildCacheStore interface {
GetBuildCacheSetting(ctx context.Context, appName string) (store.BuildCacheSetting, error)
RecordBuildCacheResult(ctx context.Context, scope, at, result, warning string) error
RecordBuildCacheCleared(ctx context.Context, scope, at string) error
}
BuildCacheStore is the persistence surface the build cache service needs.
type CacheStats ¶
type CacheStats struct {
Prefix string `json:"prefix"`
Objects int `json:"objects"`
Bytes int64 `json:"bytes"`
LastModified time.Time `json:"last_modified,omitzero"`
// Truncated means the listing hit the bound, so counts are lower bounds.
Truncated bool `json:"truncated"`
}
CacheStats summarises the objects under an app's cache prefix.
type ClearResult ¶
type ClearResult struct {
Deleted int `json:"deleted"`
// More means the ClearMaxObjects bound was hit; run the clear again.
More bool `json:"more"`
}
ClearResult reports a cache clear.
type Client ¶
type Client struct {
// contains filtered or unexported fields
}
Client is a bucket-scoped S3 client.
type Config ¶
type Config struct {
Endpoint string
Region string
Bucket string
AccessKeyID string
SecretAccessKey string
PathStyle bool
// MaxAttempts bounds SDK retries per request; zero means 3.
MaxAttempts int
// HTTPClient defaults to netguard.NewClient(), which refuses internal addresses.
HTTPClient *http.Client
}
Config is everything needed to talk to one bucket.
type DumpRequest ¶
DumpRequest is a manual export of a time range.
type HealthSource ¶
type HealthSource struct {
Store interface {
ListLogArchivePolicies(ctx context.Context) ([]store.LogArchivePolicy, error)
}
}
HealthSource adapts stored policies to the alerting engine.
func (HealthSource) LogArchiveHealth ¶
func (h HealthSource) LogArchiveHealth(ctx context.Context) ([]alerting.LogArchiveHealth, error)
LogArchiveHealth implements alerting.LogArchiveSource.
type LogReader ¶
type LogReader interface {
DistinctLogResources(ctx context.Context, fromNs, toNs int64) ([]string, error)
StreamLogs(ctx context.Context, resourceID string, fromNs, toNs int64, maxLines int, fn func(telemetry.LogEntry) error) (telemetry.LogStreamResult, error)
}
LogReader is the node-local log store surface the archiver reads.
type Object ¶
type Object struct {
Key string `json:"key"`
Size int64 `json:"size"`
LastModified time.Time `json:"last_modified"`
}
Object is one listed key.
type Options ¶
type Options struct {
Prefix string
MaxLinesPerObject int
MaxWindowsPerRun int
Lag time.Duration
MaxBackfill time.Duration
MaxDumpRange time.Duration
Tick time.Duration
}
Options tune archive runs.
func DefaultOptions ¶
func DefaultOptions() Options
DefaultOptions returns the built-in archive tuning.
type Preset ¶
type Preset struct {
ID string `json:"id"`
Label string `json:"label"`
// EndpointTemplate may contain {account_id} and {region}. Empty means the
// SDK resolves the endpoint (AWS) or the caller supplies one.
EndpointTemplate string `json:"endpoint_template,omitempty"`
DefaultRegion string `json:"default_region,omitempty"`
PathStyle bool `json:"path_style"`
NeedsAccountID bool `json:"needs_account_id"`
NeedsRegion bool `json:"needs_region"`
NeedsEndpoint bool `json:"needs_endpoint"`
// Provider is the backup target provider this preset is stored as.
Provider string `json:"provider"`
}
Preset describes how a provider's endpoint, region and addressing work.
type ProbeResult ¶
type ProbeResult struct {
OK bool `json:"ok"`
Reason string `json:"reason,omitempty"`
Message string `json:"message,omitempty"`
Steps []ProbeStep `json:"steps"`
}
ProbeResult reports a put, get and delete round trip on a throwaway key.
type ProbeStep ¶
type ProbeStep struct {
Name string `json:"name"`
OK bool `json:"ok"`
Error string `json:"error,omitempty"`
}
ProbeStep is the outcome of one probe operation.
type Resolver ¶
type Resolver struct {
Store TargetStore
Secrets SecretsResolver
// HTTPClient overrides the SSRF-guarded default, for tests.
HTTPClient *http.Client
// MaxAttempts bounds SDK retries per request; zero means the default.
MaxAttempts int
}
Resolver builds a Client for a stored storage destination.
type SecretsResolver ¶
type SecretsResolver interface {
Resolve(ctx context.Context, serviceName, envKey string) (string, error)
}
SecretsResolver reads a stored credential.
type TargetStore ¶
type TargetStore interface {
GetBackupTarget(ctx context.Context, id string) (store.BackupTarget, error)
GetStorageOptions(ctx context.Context, targetID string) (store.StorageOptions, bool, error)
}
TargetStore is the storage-destination lookup surface the resolver needs.
Source Files
¶
Directories
¶
| Path | Synopsis |
|---|---|
|
Package objectstoretest is an in-memory, path-style S3 server for tests.
|
Package objectstoretest is an in-memory, path-style S3 server for tests. |