cmn

package
v1.3.30 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Jul 21, 2025 License: MIT Imports: 40 Imported by: 9

Documentation

Overview

Package cmn provides common constants, types, and utilities for AIS clients and AIStore.

  • Copyright (c) 2018-2025, NVIDIA CORPORATION. All rights reserved.

Package cmn provides common constants, types, and utilities for AIS clients and AIStore.

  • Copyright (c) 2018-2025, NVIDIA CORPORATION. All rights reserved.

Package cmn provides common constants, types, and utilities for AIS clients and AIStore.

  • Copyright (c) 2018-2025, NVIDIA CORPORATION. All rights reserved.

Package cmn provides common constants, types, and utilities for AIS clients and AIStore.

  • Copyright (c) 2018-2025, NVIDIA CORPORATION. All rights reserved.

Package cmn provides common constants, types, and utilities for AIS clients and AIStore.

  • Copyright (c) 2018-2025, NVIDIA CORPORATION. All rights reserved.

Package cmn provides common constants, types, and utilities for AIS clients and AIStore.

  • Copyright (c) 2018-2025, NVIDIA CORPORATION. All rights reserved.

Package cmn provides common constants, types, and utilities for AIS clients and AIStore.

  • Copyright (c) 2018-2025, NVIDIA CORPORATION. All rights reserved.

Package cmn provides common constants, types, and utilities for AIS clients and AIStore.

  • Copyright (c) 2018-2025, NVIDIA CORPORATION. All rights reserved.

Package cmn provides common constants, types, and utilities for AIS clients and AIStore.

  • Copyright (c) 2018-2025, NVIDIA CORPORATION. All rights reserved.

Package cmn provides common constants, types, and utilities for AIS clients and AIStore.

  • Copyright (c) 2018-2025, NVIDIA CORPORATION. All rights reserved.

Package cmn provides common constants, types, and utilities for AIS clients and AIStore.

  • Copyright (c) 2018-2025, NVIDIA CORPORATION. All rights reserved.

Package cmn provides common constants, types, and utilities for AIS clients and AIStore.

  • Copyright (c) 2018-2024, NVIDIA CORPORATION. All rights reserved.

Package cmn provides common constants, types, and utilities for AIS clients and AIStore.

  • Copyright (c) 2018-2024, NVIDIA CORPORATION. All rights reserved.

Package cmn provides common constants, types, and utilities for AIS clients and AIStore.

  • Copyright (c) 2018-2025, NVIDIA CORPORATION. All rights reserved.

Package cmn provides common constants, types, and utilities for AIS clients and AIStore.

  • Copyright (c) 2018-2024, NVIDIA CORPORATION. All rights reserved.

msgp -file cmn/objlist.go -tests=false -marshal=false -unexported Code generated by the command above where msgp is tinylib/msgp; see docs/msgp.md. DO NOT EDIT.

Package cmn provides common constants, types, and utilities for AIS clients and AIStore.

  • Copyright (c) 2018-2025, NVIDIA CORPORATION. All rights reserved.

Package cmn provides common constants, types, and utilities for AIS clients and AIStore.

  • Copyright (c) 2018-2024, NVIDIA CORPORATION. All rights reserved.

Package cmn provides common constants, types, and utilities for AIS clients and AIStore.

  • Copyright (c) 2018-2025, NVIDIA CORPORATION. All rights reserved.

Package cmn provides common constants, types, and utilities for AIS clients and AIStore.

  • Copyright (c) 2023-2025, NVIDIA CORPORATION. All rights reserved.

Package cmn provides common constants, types, and utilities for AIS clients and AIStore.

  • Copyright (c) 2018-2025, NVIDIA CORPORATION. All rights reserved.

Package cmn provides common constants, types, and utilities for AIS clients and AIStore.

  • Copyright (c) 2022-2025, NVIDIA CORPORATION. All rights reserved.

Index

Constants

View Source
const (
	PropBucketAccessAttrs  = "access"             // Bucket access attributes.
	PropBucketVerEnabled   = "versioning.enabled" // Enable/disable object versioning in a bucket.
	PropBucketCreated      = "created"            // Bucket creation time.
	PropBackendBck         = "backend_bck"
	PropBackendBckName     = PropBackendBck + ".name"
	PropBackendBckProvider = PropBackendBck + ".provider"
)
View Source
const (
	DfltDialupTimeout = 10 * time.Second
	DfltKeepaliveTCP  = 30 * time.Second
)
View Source
const (
	DefaultMaxIdleConns        = 0               // unlimited (in re: `http.errTooManyIdle`)
	DefaultMaxIdleConnsPerHost = 32              // (http.errTooManyIdleHost)
	DefaultIdleConnTimeout     = 6 * time.Second // (Go default is 90s)
	DefaultWriteBufferSize     = 64 * cos.KiB
	DefaultReadBufferSize      = 64 * cos.KiB
	DefaultSndRcvBufferSize    = 128 * cos.KiB
)

[NOTE] net/http.DefaultTransport has the following defaults:

- MaxIdleConns: 100, - MaxIdleConnsPerHost : 2 (via DefaultMaxIdleConnsPerHost) - IdleConnTimeout: 90 * time.Second, - WriteBufferSize: 4KB - ReadBufferSize: 4KB

See also: docs/idle_connections.md

Following are the defaults we use instead:

View Source
const (
	ObjSizeToAlwaysReplicate = -1 // (see `ObjSizeLimit` comment above)

	MinSliceCount = 1  // minimum number of data or parity slices
	MaxSliceCount = 32 // maximum --/--
)
View Source
const (
	IgnoreReaction = "ignore"
	WarnReaction   = "warn"
	AbortReaction  = "abort"
)
View Source
const (
	DfltTransportHeader = 4 * cos.KiB   // memsys.PageSize
	MaxTransportHeader  = 128 * cos.KiB // memsys.MaxPageSlabSize

	DfltTransportBurst = 256
	MaxTransportBurst  = 4096
)
View Source
const (
	// TODO:
	// config bloat vs specificity; consider phasing out `timeout.ec_streams_time` and
	// using `timeout.shared_streams_time` instead
	SharedStreamsEver = -time.Second
	SharedStreamsDflt = 10 * time.Minute
	SharedStreamsMin  = 5 * time.Minute
	SharedStreamsNack = max(SharedStreamsMin>>1, 3*time.Minute)

	LcacheEvictMin  = 20 * time.Minute
	LcacheEvictDflt = 2 * time.Hour
	LcacheEvictMax  = 16 * time.Hour

	ColdGetConflictDflt = 5 * time.Second
	ColdGetConflictMin  = time.Second
)
View Source
const (
	FmtErrIntegrity      = "[%s%d, for troubleshooting see %s/blob/main/docs/troubleshooting.md]"
	FmtErrUnmarshal      = "%s: failed to unmarshal %s (%s), err: %w"
	FmtErrMorphUnmarshal = "%s: failed to unmarshal %s (%T), err: %w"
	FmtErrBackwardCompat = "%v (backward compatibility is supported only one version back, e.g. 3.9 => 3.10)"

	BadSmapPrefix = "[bad cluster map]"

	StartupMayTimeout = "cluster startup is taking unusually long time..." // related ErrStartupTimeout
)
View Source
const (
	NetPublic       = "PUBLIC"
	NetIntraControl = "INTRA-CONTROL"
	NetIntraData    = "INTRA-DATA"
)
View Source
const (
	// source of the cold-GET and download; the values include all
	// 3rd party backend providers
	SourceObjMD = "source"

	// downloader' source is "web"
	WebObjMD = "web"

	VersionObjMD = "version" // "generation" for GCP, "version" for AWS but only if the bucket is versioned, etc.
	CRC32CObjMD  = cos.ChecksumCRC32C
	MD5ObjMD     = cos.ChecksumMD5
	ETag         = cos.HdrETag

	OrigURLObjMD = "orig_url"

	// RFC3339; see also: cos.HdrLastModified formatted RFC1123GMT
	LsoLastModified = "LastModified"

	// as the name implies
	OrigFntl = "orig_fntl"
)

LOM custom metadata stored under `lomCustomMD`.

View Source
const (
	RetryLogVerbose = iota
	RetryLogQuiet
	RetryLogOff
)
View Source
const (
	VersionAIStore = "3.30"
	VersionCLI     = "1.19"
	VersionLoader  = "1.13"
	VersionAuthN   = "1.2"
)
View Source
const (
	MetaverSmap  = 2 // Smap (cluster map) formatting version a.k.a. meta-version (see core/meta/jsp.go)
	MetaverBMD   = 2 // BMD (bucket metadata) --/--
	MetaverRMD   = 1 // Rebalance MD (jsp)
	MetaverVMD   = 2 // Volume MD (jsp)
	MetaverEtlMD = 2 // ETL MD (jsp)

	MetaverConfig      = 4 // Global Configuration (jsp)
	MetaverAuthNConfig = 1 // Authn config (jsp) // ditto
	MetaverAuthTokens  = 1 // Authn tokens (jsp) // ditto

	MetaverMetasync = 1 // metasync over network formatting version (jsp)

	MetaverJSP = jsp.Metaver // `jsp` own encoding version
)
View Source
const AwsHeaderMetaPrefix = "X-Amz-Meta-"

Due to import cycle we need to define "x-amz-meta" header here See also: - ais/s3/const.go - https://docs.aws.amazon.com/AmazonS3/latest/userguide/UsingMetadata.html#UserMetadata

View Source
const AwsMultipartDelim = "-"

from https://docs.aws.amazon.com/AmazonS3/latest/API/API_Object.html "The ETag may or may not be an MD5 digest of the object data. Whether or not it is depends on how the object was created and how it is encrypted..."

View Source
const (
	DfltMaxIdleTimeout = 30 * time.Second
)

see also: - docs/idle_connections.md - DefaultMaxIdle* in cmn/client - `listen` in ais/htcommon

View Source
const GitHubHome = "https://github.com/NVIDIA/aistore"
View Source
const HostnameListSepa = ","
View Source
const IterFieldNameSepa = "."
View Source
const MsgpLsoBufSize = 32 * cos.KiB
View Source
const (
	// NsGlobalUname is hardcoded here to avoid allocating it via Uname()
	// (the most common use case)
	NsGlobalUname = "@#"
)

Variables

View Source
var (
	// NsGlobal represents *this* cluster's global namespace that is used by default when
	// no specific namespace was defined or provided by the user.
	NsGlobal = Ns{}
	// NsAnyRemote represents any remote cluster. As such, NsGlobalRemote applies
	// exclusively to AIS (provider) given that other Backend providers are remote by definition.
	NsAnyRemote = Ns{UUID: string(apc.NsUUIDPrefix)}
)
View Source
var (
	ErrSkip             = errors.New("skip")
	ErrStartupTimeout   = errors.New("startup timeout") // related StartupMayTimeout
	ErrQuiesceTimeout   = errors.New("timed out waiting for quiescence")
	ErrNotEnoughTargets = errors.New("not enough target nodes")
	ErrNoMountpaths     = errors.New("no mountpaths")

	// aborts
	ErrXactRenewAbort   = errors.New("renewal abort")
	ErrXactUserAbort    = errors.New("user abort")              // via apc.ActXactStop
	ErrXactICNotifAbort = errors.New("IC(notifications) abort") // ditto

	ErrGetTxBenign = errors.New("Warning: failed to transmit GET response") //nolint:staticcheck // making an exception for Warning
)
View Source
var BackendHelpers = struct {
	Amazon backendFuncs
	Azure  backendFuncs
	Google backendFuncs
	OCI    backendFuncs
	HTTP   backendFuncs
}{
	Amazon: backendFuncs{
		EncodeVersion: func(v any) (string, bool) {
			switch x := v.(type) {
			case *string:
				if awsIsVersionSet(x) {
					return *x, true
				}
				return "", false
			case string:
				if awsIsVersionSet(&x) {
					return x, true
				}
				return x, false
			default:
				debug.FailTypeCast(v)
				return "", false
			}
		},

		EncodeETag: func(v any) (string, bool) {
			switch x := v.(type) {
			case *string:
				return *x, *x != ""
			case string:
				return x, x != ""
			default:
				debug.FailTypeCast(v)
				return "", false
			}
		},

		EncodeCksum: func(v any) (string, bool) {
			switch x := v.(type) {
			case *string:
				if x == nil || *x == "" {
					return "", false
				}
				if isS3MultipartEtag(*x) {
					return "", false
				}
				return UnquoteCEV(*x), true
			case string:
				if x == "" {
					return "", false
				}
				if isS3MultipartEtag(x) {
					return "", false
				}
				return UnquoteCEV(x), true
			default:
				debug.FailTypeCast(v)
				return "", false
			}
		},
		EncodeMetadata: func(metadata map[string]string) (header map[string]string) {
			if len(metadata) == 0 {
				return
			}
			header = make(map[string]string, len(metadata))
			for k, v := range metadata {
				key := http.CanonicalHeaderKey(AwsHeaderMetaPrefix + k)
				header[key] = v
			}
			return
		},

		DecodeMetadata: func(header http.Header) (metadata map[string]string) {
			for headerKey := range header {
				if strings.HasPrefix(headerKey, AwsHeaderMetaPrefix) {
					if metadata == nil {
						metadata = make(map[string]string)
					}
					key := strings.TrimPrefix(headerKey, AwsHeaderMetaPrefix)
					value := header.Get(headerKey)
					metadata[key] = value
				}
			}
			return
		},
	},
	Google: backendFuncs{
		EncodeVersion: func(v any) (string, bool) {
			switch x := v.(type) {
			case string:
				return x, x != ""
			case int64:
				return strconv.FormatInt(x, 10), true
			default:
				debug.FailTypeCast(v)
				return "", false
			}
		},
		EncodeETag: func(v any) (string, bool) {
			switch x := v.(type) {
			case string:
				return x, x != ""
			default:
				debug.FailTypeCast(v)
				return "", false
			}
		},
		EncodeCksum: func(v any) (string, bool) {
			switch x := v.(type) {
			case string:
				decoded, err := base64.StdEncoding.DecodeString(x)
				if err != nil {
					return "", false
				}
				return hex.EncodeToString(decoded), true
			case []byte:
				return hex.EncodeToString(x), true
			case uint32:

				b := []byte{byte(x >> 24), byte(x >> 16), byte(x >> 8), byte(x)}
				return base64.StdEncoding.EncodeToString(b), true
			default:
				debug.FailTypeCast(v)
				return "", false
			}
		},
	},
	OCI: backendFuncs{
		EncodeVersion: func(_ any) (string, bool) {
			return "", false
		},
		EncodeETag: func(v any) (string, bool) {
			switch x := v.(type) {
			case *string:
				if x == nil || *x == "" {
					return "", false
				}
				return UnquoteCEV(*x), true
			default:
				debug.FailTypeCast(v)
				return "", false
			}
		},
		EncodeCksum: func(v any) (string, bool) {
			switch x := v.(type) {
			case *string:
				if x == nil || *x == "" {
					return "", false
				}
				return UnquoteCEV(*x), true
			default:
				debug.FailTypeCast(v)
				return "", false
			}
		},

		DecodeMetadata: func(header http.Header) (metadata map[string]string) {
			for headerKey := range header {
				if strings.HasPrefix(headerKey, AwsHeaderMetaPrefix) {
					if metadata == nil {
						metadata = make(map[string]string)
					}
					key := strings.TrimPrefix(headerKey, AwsHeaderMetaPrefix)
					value := header.Get(headerKey)
					metadata[key] = value
				}
			}
			return
		},
	},
	HTTP: backendFuncs{
		EncodeETag: func(v any) (string, bool) {
			switch x := v.(type) {
			case string:

				return x, x != ""
			default:
				debug.FailTypeCast(v)
				return "", false
			}
		},
	},
}
View Source
var ConfigRestartRequired = [...]string{"auth.secret", "memsys", "net"}

assorted named fields that require (cluster | node) restart for changes to make an effect (used by CLI)

View Source
var GCO *gco
View Source
var Rom readMostly

Functions

func CheckDirNoRecurs added in v1.3.28

func CheckDirNoRecurs(prefix, relPath string) (addDirEntry bool, _ error)

no recursion (LsNoRecursion) helper function: check the level of nesting return:

  • true, to include virtual directory
  • filepath.SkipDir, to skip further recursion

func CopyHeaders added in v1.3.30

func CopyHeaders(dst, src http.Header)

Copies headers from original request(from client) to a new one(inter-cluster call)

func CopyProps added in v1.3.24

func CopyProps(src, dst any, asType string) error

CopyProps update dst with the values from src

func CustomMD2S

func CustomMD2S(md cos.StrKVs) string

[NOTE] - usage: LOM custom metadata => LsoEnt custom property - `OrigFntl` always excepted (and possibly other TBD internal keys)

func CustomProps2S added in v1.3.26

func CustomProps2S(nvs ...string) string

(compare w/ CustomMD2S above)

func DelBckFromQuery

func DelBckFromQuery(query url.Values) url.Values

func DirHasOrIsPrefix added in v1.3.16

func DirHasOrIsPrefix(dirPath, prefix string) bool

directory has to either: - include (or match) prefix, or - be contained in prefix - motivation: don't SkipDir a/b when looking for a/b/c an alternative name for this function could be smth. like SameBranch() see also: cos.TrimPrefix

func EnvToTLS added in v1.3.21

func EnvToTLS(sargs *TLSArgs)

EnvToTLS usage is limited to aisloader and tools NOTE that embedded intra-cluster clients utilize a similar method: `HTTPConf.ToTLS`

func FreeBuffer added in v1.3.18

func FreeBuffer(buf *bytes.Buffer)

func FreeHra

func FreeHra(a *HreqArgs)

func FreeHterr

func FreeHterr(a *ErrHTTP)

func HasSpecialSymbols added in v1.3.30

func HasSpecialSymbols(path string) bool

RFC 3986 defines a set of reserved characters that have special meaning within a URI (Uniform Resource Identifier). These include delimiters for different URI components (like '?' for query, '#' for fragment) and sub-delimiters. Additionally, certain characters like space ' ' are "unsafe" and must be percent-encoded when appearing in URI components like the path.

This function checks for the presence of such special/unsafe characters in a given path string. Its primary use case is to determine if a URL path (which usually contains bucket/object names) can be safely concatenated into a new URL, or if it requires robust URL parsing and re-encoding via `net/url.URL` to ensure correctness (e.g., proper percent-encoding of spaces, ampersands, etc. for the target URL).

The characters checked: - '%': Indicates percent-encoding (e.g., %20 for space) or a literal percent sign. - '?': Query string delimiter. - '#': Fragment identifier delimiter. - ' ': Space (unsafe, must be %20 encoded). - '+': Often used for space in x-www-form-urlencoded, or literal plus. Not checking: - '&': See '?' above - '=': Ditto

func Host2IP added in v1.3.23

func Host2IP(host string) (net.IP, error)

func HreqFree added in v1.3.26

func HreqFree(r *http.Request)

func Init added in v1.3.26

func Init(a string, b func(error))

func InitObjProps2Hdr added in v1.3.24

func InitObjProps2Hdr()

func IsErrAborted

func IsErrAborted(err error) bool

func IsErrBckNotFound

func IsErrBckNotFound(err error) bool

func IsErrBucketAlreadyExists

func IsErrBucketAlreadyExists(err error) bool

func IsErrBucketLevel

func IsErrBucketLevel(err error) bool

func IsErrBucketNought

func IsErrBucketNought(err error) bool

nought: not a thing

func IsErrBusy added in v1.3.28

func IsErrBusy(err error) bool

func IsErrCapExceeded added in v1.3.19

func IsErrCapExceeded(err error) bool

func IsErrEmptyProvider added in v1.3.26

func IsErrEmptyProvider(err error) bool

func IsErrFailedTo added in v1.3.26

func IsErrFailedTo(err error) bool

func IsErrGetCap added in v1.3.24

func IsErrGetCap(err error) bool

func IsErrInitMissingBackend added in v1.3.26

func IsErrInitMissingBackend(err error) bool

func IsErrLmetaCorrupted

func IsErrLmetaCorrupted(err error) bool

func IsErrLmetaNotFound

func IsErrLmetaNotFound(err error) bool

func IsErrMpathCheck added in v1.3.24

func IsErrMpathCheck(err error) bool

func IsErrMpathNewDisk added in v1.3.24

func IsErrMpathNewDisk(err error) bool

func IsErrMpathNotFound added in v1.3.24

func IsErrMpathNotFound(err error) bool

func IsErrObjLevel

func IsErrObjLevel(err error) bool

func IsErrObjNought

func IsErrObjNought(err error) bool

lom.Load

func IsErrRangeNotSatisfiable added in v1.3.22

func IsErrRangeNotSatisfiable(err error) bool

func IsErrRemoteBckNotFound

func IsErrRemoteBckNotFound(err error) bool

func IsErrStreamTerminated added in v1.3.16

func IsErrStreamTerminated(err error) bool

func IsErrTooManyRequests added in v1.3.28

func IsErrTooManyRequests(err error) bool

func IsErrWarning added in v1.3.24

func IsErrWarning(err error) bool

func IsErrXactNotFound added in v1.3.16

func IsErrXactNotFound(err error) bool

func IsErrXactUsePrev added in v1.3.18

func IsErrXactUsePrev(err error) bool

func IsFileAlreadyClosed added in v1.3.16

func IsFileAlreadyClosed(err error) bool

func IsNestedMpath

func IsNestedMpath(a string, la int, b string) (err error)

func IsStatusBadGateway

func IsStatusBadGateway(err error) (yes bool)

func IsStatusGone

func IsStatusGone(err error) (yes bool)

func IsStatusNotFound

func IsStatusNotFound(err error) (yes bool)

func IsStatusServiceUnavailable

func IsStatusServiceUnavailable(err error) (yes bool)

func IterFields

func IterFields(v any, updf updateFunc, opts ...IterOpts) error

IterFields walks the struct and calls `updf` callback at every leaf field that it encounters. The (nested) names are created by joining the json tag with dot. Iteration supports reading another, custom tag `list` with values:

  • `tagOmitempty` - omit empty fields (only for read run)
  • `tagOmit` - omit field
  • `tagReadonly` - field cannot be updated (returns error on `SetValue`)

Examples of usages for tags can be found in `BucketProps` or `Config` structs.

Passing additional options with `IterOpts` can for example call callback also at the non-leaf structures.

func KeepaliveRetryDuration

func KeepaliveRetryDuration(c *Config) time.Duration

func LoadConfig

func LoadConfig(globalConfPath, localConfPath, daeRole string, config *Config) error

is called at startup

func MakeRangeHdr added in v1.3.16

func MakeRangeHdr(start, length int64) string

Ref: https://www.rfc-editor.org/rfc/rfc7233#section-2.1 (compare w/ htrange.contentRange)

func NetworkIsKnown

func NetworkIsKnown(net string) bool

func NewBuffer added in v1.3.18

func NewBuffer() (buf *bytes.Buffer)

func NewClient

func NewClient(cargs TransportArgs) *http.Client

NOTE: `NewTransport` (below) fills-in certain defaults

func NewClientTLS added in v1.3.21

func NewClientTLS(cargs TransportArgs, sargs TLSArgs, intra bool) *http.Client

https client (ditto)

func NewDefaultClients added in v1.3.22

func NewDefaultClients(timeout time.Duration) (clientH, clientTLS *http.Client)

TODO -- FIXME: this call must get cert file and key to be used for the `clientTLS`

func NewErrRemoteMetadataMismatch added in v1.3.24

func NewErrRemoteMetadataMismatch(cause error) error

func NewIntraClientTLS added in v1.3.21

func NewIntraClientTLS(cargs TransportArgs, config *Config) *http.Client

func NewTLS added in v1.3.21

func NewTLS(sargs TLSArgs, intra bool) (tlsConf *tls.Config, err error)

func NewTransport

func NewTransport(cargs TransportArgs) *http.Transport

{TransportArgs + defaults} => http.Transport for a variety of ais clients NOTE: TLS below, and separately

func NormalizeProvider

func NormalizeProvider(provider string) (p string, err error)

func ObjHasPrefix added in v1.3.16

func ObjHasPrefix(objName, prefix string) bool

see also: cos.TrimPrefix

func OrigURLBck2Name

func OrigURLBck2Name(origURLBck string) (bckName string)

func ParseHost2IP added in v1.3.23

func ParseHost2IP(host string) (net.IP, error)

func ParsePort

func ParsePort(p string) (int, error)

func ParseRangeHdr added in v1.3.26

func ParseRangeHdr(contentRange string) (start, length, objectSize int64, err error)

Ref: https://www.rfc-editor.org/rfc/rfc7233#section-4.2

func ParseReadHeaderTimeout added in v1.3.22

func ParseReadHeaderTimeout() (_ time.Duration, isSet bool)

func ParseURL added in v1.3.18

func ParseURL(path string, itemsPresent []string, itemsAfter int, splitAfter bool) ([]string, error)

func ParseURLScheme

func ParseURLScheme(url string) (scheme, address string)

Splits url into [(scheme)://](address). It's not possible to use url.Parse as (from url.Parse() docs) 'Trying to parse a hostname and path without a scheme is invalid'

func PrependProtocol

func PrependProtocol(url string, protocol ...string) string

PrependProtocol prepends protocol in URL in case it is missing. By default it adds `http://` to the URL.

func PropToHeader

func PropToHeader(prop string) string

(compare with apc.PropToHeader)

func ReadBytes

func ReadBytes(r *http.Request) (b []byte, err error)

func ReadJSON

func ReadJSON(w http.ResponseWriter, r *http.Request, out any) (err error)

func S2CustomMD added in v1.3.22

func S2CustomMD(md cos.StrKVs, custom, version string)

func S2CustomVal added in v1.3.26

func S2CustomVal(custom, name string) (v string)

func SaveOverrideConfig

func SaveOverrideConfig(configDir string, toUpdate *ConfigToSet) error

func SortLso

func SortLso(entries LsoEntries)

func ToHeader

func ToHeader(oah cos.OAH, hdr http.Header, size int64, cksums ...*cos.Cksum)

may set headers: - standard cos.HdrContentLength ("Content-Length") & cos.HdrETag ("ETag") - atime, version, etc. - all the rest "ais-" prefixed

func TokenGreaterEQ

func TokenGreaterEQ(token, objName string) bool

Returns true if the continuation token >= object's name (in other words, the object is already listed and must be skipped). Note that string `>=` is lexicographic.

func TypeCodeHTTPErr added in v1.3.19

func TypeCodeHTTPErr(s string) (tcode string)

func UnquoteCEV added in v1.3.22

func UnquoteCEV(val string) string

unquote checksum, ETag, and version e.g., https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/ETag

func UpdateFieldValue

func UpdateFieldValue(s any, name string, value any) error

UpdateFieldValue updates the field in the struct with given value. Returns error if the field was not found or could not be updated.

func ValidateMpath

func ValidateMpath(mpath string) (string, error)

func ValidatePort

func ValidatePort(port int) (int, error)

func ValidateRemAlias

func ValidateRemAlias(alias string) (err error)

func WriteErr

func WriteErr(w http.ResponseWriter, r *http.Request, err error, opts ...int)

sends HTTP response header with the provided status (alloc/free via mem-pool)

func WriteErr405

func WriteErr405(w http.ResponseWriter, r *http.Request, methods ...string)

405 Method Not Allowed, see: * https://www.rfc-editor.org/rfc/rfc7231#section-6.5.5

func WriteErrJSON

func WriteErrJSON(w http.ResponseWriter, r *http.Request, out any, err error) error

func WriteErrMsg

func WriteErrMsg(w http.ResponseWriter, r *http.Request, msg string, opts ...int)

Create ErrHTTP (based on `msg` and `opts`) and write it into HTTP response.

Types

type Adaptive added in v1.3.28

type Adaptive struct {
	RateLimitBase
	NumRetries int `json:"num_retries"`
}

Adaptive rate limit (a.k.a. rate shaper): usage: handle status 429 and 503 from remote backends - up to max tokens (originally, `MaxTokens`) during one Interval with NumRetries and - `recompute` between Intervals - see cmn/cos/rate_limit

type AdaptiveToSet added in v1.3.28

type AdaptiveToSet struct {
	RateLimitBaseToSet      // NOTE: must be in the same exact order/position as above
	NumRetries         *int `json:"num_retries,omitempty"`
}

global config that can be used to manage: * adaptive rate limit vis-à-vis Cloud backend * rate limiting (bursty) user workloads on the front

type AllBsummResults

type AllBsummResults []*BsummResult

func (AllBsummResults) Aggregate

func (s AllBsummResults) Aggregate(from *BsummResult) AllBsummResults

func (AllBsummResults) Finalize

func (s AllBsummResults) Finalize(dsize map[string]uint64, testingEnv bool)

func (AllBsummResults) Len

func (s AllBsummResults) Len() int

func (AllBsummResults) Less

func (s AllBsummResults) Less(i, j int) bool

func (AllBsummResults) Swap

func (s AllBsummResults) Swap(i, j int)

type ArchConf added in v1.3.28

type ArchConf struct{ XactConf }

multi-object archive (multiple objects => shard)

type ArchConfToSet added in v1.3.28

type ArchConfToSet struct{ XactConfToSet }

global configuration

type ArchiveBckMsg added in v1.3.18

type ArchiveBckMsg struct {
	ToBck Bck `json:"tobck"`
	apc.ArchiveMsg
}

ArchiveBckMsg contains parameters to archive multiple objects from the specified (source) bucket. Destination bucket may the same as the source or a different one. -------------------- NOTE on terminology: --------------------- "archive" is any (.tar, .tgz/.tar.gz, .zip, .tar.lz4) formatted object often also called "shard"

See also: apc.PutApndArchArgs

func (*ArchiveBckMsg) Cname added in v1.3.18

func (msg *ArchiveBckMsg) Cname() string

type AuthConf

type AuthConf struct {
	Secret  string `json:"secret"`
	Enabled bool   `json:"enabled"`
}

global configuration

type AuthConfToSet added in v1.3.21

type AuthConfToSet struct {
	Secret  *string `json:"secret,omitempty"`
	Enabled *bool   `json:"enabled,omitempty"`
}

global configuration

type BackendBckToSet added in v1.3.21

type BackendBckToSet struct {
	Name     *string `json:"name"`
	Provider *string `json:"provider"`
}

type BackendConf

type BackendConf struct {
	Conf      map[string]any `json:"-"` // backend implementation-dependent (custom marshaling to populate this field)
	Providers map[string]Ns  `json:"-"` // conditional (build tag) providers set during validation (BackendConf.Validate)
}

global configuration

func (*BackendConf) EqualRemAIS

func (c *BackendConf) EqualRemAIS(o *BackendConf, sname string) bool

func (*BackendConf) Get added in v1.3.16

func (c *BackendConf) Get(provider string) (conf any)

func (*BackendConf) MarshalJSON

func (c *BackendConf) MarshalJSON() (data []byte, err error)

func (*BackendConf) Set added in v1.3.16

func (c *BackendConf) Set(provider string, newConf any)

func (*BackendConf) UnmarshalJSON

func (c *BackendConf) UnmarshalJSON(data []byte) error

func (*BackendConf) Validate

func (c *BackendConf) Validate() (err error)

type BackendConfAIS

type BackendConfAIS map[string][]string // cluster alias -> [urls...]

global configuration

func (BackendConfAIS) String

func (c BackendConfAIS) String() (s string)

type Bck

type Bck struct {
	Props    *Bprops `json:"-"`
	Name     string  `json:"name" yaml:"name"`
	Provider string  `json:"provider" yaml:"provider"` // NOTE: see api/apc/provider.go for supported enum
	Ns       Ns      `json:"namespace" yaml:"namespace" list:"omitempty"`
}

func ParseBckObjectURI

func ParseBckObjectURI(uri string, opts ParseURIOpts) (bck Bck, objName string, err error)

func ParseUname

func ParseUname(uname string) (b Bck, objName string)

unique name => Bck - MakeUname to make - compare w/ meta.ParseUname (that does more) and maybe use the latter

func (*Bck) AddToQuery

func (b *Bck) AddToQuery(query url.Values) url.Values

func (*Bck) AddUnameToQuery

func (b *Bck) AddUnameToQuery(query url.Values, uparam, objName string) url.Values

func (*Bck) Backend

func (b *Bck) Backend() *Bck

func (*Bck) Cname added in v1.3.16

func (b *Bck) Cname(objname string) (s string)

canonical name, with or without object

func (*Bck) Copy

func (b *Bck) Copy(src *Bck)

func (*Bck) DefaultProps

func (bck *Bck) DefaultProps(c *ClusterConfig) *Bprops

By default, created buckets inherit their properties from the cluster (global) configuration. Global configuration, in turn, is protected versioned, checksummed, and replicated across the entire cluster.

* Bucket properties can be changed at any time via `api.SetBprops`. * In addition, `api.CreateBucket` allows to specify (non-default) properties at bucket creation time. * Inherited defaults include checksum, LRU, etc. configurations - see below. * By default, LRU is disabled for AIS (`ais://`) buckets.

See also:

  • github.com/NVIDIA/aistore/blob/main/docs/bucket.md#default-bucket-properties
  • BpropsToSet (above)
  • ais.defaultBckProps()

func (*Bck) Equal

func (b *Bck) Equal(other *Bck) bool

func (*Bck) HasProvider

func (b *Bck) HasProvider() bool

func (*Bck) HasVersioningMD added in v1.3.22

func (b *Bck) HasVersioningMD() bool

A subset of remote backends that maintain assorted items of versioning information - the items including ETag, checksum, etc. - that, in turn, can be used to populate `ObjAttrs` * see related: `ObjAttrs.Equal`

func (*Bck) HashUname added in v1.3.28

func (b *Bck) HashUname(s string) uint64

alternative (one-way) uniqueness

func (*Bck) IsAIS

func (b *Bck) IsAIS() bool

func (*Bck) IsBuiltTagged added in v1.3.24

func (b *Bck) IsBuiltTagged() bool

func (*Bck) IsCloud

func (b *Bck) IsCloud() bool

func (*Bck) IsEmpty

func (b *Bck) IsEmpty() bool

func (*Bck) IsHT added in v1.3.24

func (b *Bck) IsHT() bool

func (*Bck) IsQuery

func (b *Bck) IsQuery() bool

QueryBcks (see below) is a Bck that _can_ have an empty Name.

func (*Bck) IsRemote

func (b *Bck) IsRemote() bool

func (*Bck) IsRemoteAIS

func (b *Bck) IsRemoteAIS() bool

func (*Bck) LenUnameGlob added in v1.3.21

func (b *Bck) LenUnameGlob(objName string) int

func (*Bck) Less

func (b *Bck) Less(other *Bck) bool

func (*Bck) MakeUname

func (b *Bck) MakeUname(objName string) []byte

Bck and optional objName => unique name (uname); note: - use ParseUname below to translate back - compare with HashUname

func (*Bck) RemoteBck

func (b *Bck) RemoteBck() *Bck

func (*Bck) SetQuery added in v1.3.26

func (b *Bck) SetQuery(q url.Values)

q = make(url.Values, 1) TODO -- FIXME

func (*Bck) Str added in v1.3.26

func (b *Bck) Str(sb *strings.Builder)

func (*Bck) String

func (b *Bck) String() string

func (*Bck) Validate

func (b *Bck) Validate() (err error)

func (*Bck) ValidateName

func (b *Bck) ValidateName() error

type Bcks

type Bcks []Bck

func (Bcks) Equal

func (bcks Bcks) Equal(other Bcks) bool

func (Bcks) Len

func (bcks Bcks) Len() int

func (Bcks) Less

func (bcks Bcks) Less(i, j int) bool

func (Bcks) Select

func (bcks Bcks) Select(query QueryBcks) (filtered Bcks)

func (Bcks) Swap

func (bcks Bcks) Swap(i, j int)

type Bprops added in v1.3.21

type Bprops struct {
	BackendBck  Bck             `json:"backend_bck,omitempty"`            // makes a remote bucket out of a given ais://
	WritePolicy WritePolicyConf `json:"write_policy"`                     // write object metadata (immediate | delayed | never)
	Provider    string          `json:"provider" list:"readonly"`         // backend provider
	Renamed     string          `list:"omit"`                             // non-empty iff the bucket has been renamed
	Cksum       CksumConf       `json:"checksum"`                         // this bucket's checksum (for supported enum, see cmn/cos.cksum)
	Extra       ExtraProps      `json:"extra,omitempty" list:"omitempty"` // e.g., AWS.Endpoint for this bucket
	RateLimit   RateLimitConf   `json:"rate_limit"`                       // frontend and backend rate limiting - bursty and adaptive, respectively
	EC          ECConf          `json:"ec"`                               // erasure coding
	Mirror      MirrorConf      `json:"mirror"`                           // n-way mirroring
	LRU         LRUConf         `json:"lru"`                              // LRU watermarks and enable/disable
	Access      apc.AccessAttrs `json:"access,string"`                    // access permissions
	Features    feat.Flags      `json:"features,string"`                  // to flip assorted enumerated defaults (e.g. "S3-Use-Path-Style"; see cmn/feat)
	BID         uint64          `json:"bid,string" list:"omit"`           // unique ID
	Created     int64           `json:"created,string" list:"readonly"`   // creation timestamp
	Versioning  VersionConf     `json:"versioning"`                       // see "inherit"
}

func (*Bprops) Apply added in v1.3.21

func (bp *Bprops) Apply(propsToSet *BpropsToSet)

func (*Bprops) Clone added in v1.3.21

func (bp *Bprops) Clone() *Bprops

func (*Bprops) Equal added in v1.3.21

func (bp *Bprops) Equal(other *Bprops) (eq bool)

func (*Bprops) SetProvider added in v1.3.21

func (bp *Bprops) SetProvider(provider string)

func (*Bprops) Validate added in v1.3.21

func (bp *Bprops) Validate(targetCnt int) error

type BpropsToSet added in v1.3.21

type BpropsToSet struct {
	BackendBck  *BackendBckToSet      `json:"backend_bck,omitempty"`
	Versioning  *VersionConfToSet     `json:"versioning,omitempty"`
	Cksum       *CksumConfToSet       `json:"checksum,omitempty"`
	LRU         *LRUConfToSet         `json:"lru,omitempty"`
	Mirror      *MirrorConfToSet      `json:"mirror,omitempty"`
	EC          *ECConfToSet          `json:"ec,omitempty"`
	Access      *apc.AccessAttrs      `json:"access,string,omitempty"`
	RateLimit   *RateLimitConfToSet   `json:"rate_limit,omitempty"`
	Features    *feat.Flags           `json:"features,string,omitempty"`
	WritePolicy *WritePolicyConfToSet `json:"write_policy,omitempty"`
	Extra       *ExtraToSet           `json:"extra,omitempty"`
	Force       bool                  `json:"force,omitempty" copy:"skip" list:"omit"`
}

Once validated, BpropsToSet are copied to Bprops. The struct may have extra fields that do not exist in Bprops. Add tag 'copy:"skip"' to ignore those fields when copying values.

func NewBpropsToSet added in v1.3.21

func NewBpropsToSet(nvs cos.StrKVs) (props *BpropsToSet, err error)

type BsummResult

type BsummResult struct {
	Bck
	apc.BsummResult
}

type Bursty added in v1.3.28

type Bursty struct {
	RateLimitBase
	Size int `json:"burst_size"`
}

rate limit that fails 'too-many requests' while permitting a certain level of burstiness - usage: to restrict the rate of user GET, PUT, and DELETE requests - see cmn/cos/rate_limit

type BurstyToSet added in v1.3.28

type BurstyToSet struct {
	RateLimitBaseToSet
	Size *int `json:"burst_size,omitempty"`
}

global config that can be used to manage: * adaptive rate limit vis-à-vis Cloud backend * rate limiting (bursty) user workloads on the front

type CksumConf

type CksumConf struct {
	// (note that `ChecksumNone` ("none") disables checksumming)
	Type string `json:"type"`

	// validate the checksum of the object that we cold-GET
	// or download from remote location (e.g., cloud bucket)
	ValidateColdGet bool `json:"validate_cold_get"`

	// - validate in-cluster object's checksum(s);
	// - upon any of the `cos.ErrBadCksum` errors try to recover from
	//   local redundant copies, and/or EC slices, and/or remote backend if exists;
	// - if all fails, remove the object and fail the GET.
	ValidateWarmGet bool `json:"validate_warm_get"`

	// validate checksums of objects migrated or replicated within the cluster
	ValidateObjMove bool `json:"validate_obj_move"`

	// EnableReadRange: Return read range checksum otherwise return entire object checksum.
	EnableReadRange bool `json:"enable_read_range"`
}

global configuration

func (*CksumConf) String

func (c *CksumConf) String() string

func (*CksumConf) Validate

func (c *CksumConf) Validate() (err error)

func (*CksumConf) ValidateAsProps

func (c *CksumConf) ValidateAsProps(...any) (err error)

type CksumConfToSet added in v1.3.21

type CksumConfToSet struct {
	Type            *string `json:"type,omitempty"`
	ValidateColdGet *bool   `json:"validate_cold_get,omitempty"`
	ValidateWarmGet *bool   `json:"validate_warm_get,omitempty"`
	ValidateObjMove *bool   `json:"validate_obj_move,omitempty"`
	EnableReadRange *bool   `json:"enable_read_range,omitempty"`
}

global configuration

type ClientConf

type ClientConf struct {
	Timeout        cos.Duration `json:"client_timeout"`
	TimeoutLong    cos.Duration `json:"client_long_timeout"`
	ListObjTimeout cos.Duration `json:"list_timeout"`
}

global configuration

func (*ClientConf) Validate

func (c *ClientConf) Validate() error

type ClientConfToSet added in v1.3.21

type ClientConfToSet struct {
	Timeout        *cos.Duration `json:"client_timeout,omitempty"` // readonly as far as intra-cluster
	TimeoutLong    *cos.Duration `json:"client_long_timeout,omitempty"`
	ListObjTimeout *cos.Duration `json:"list_timeout,omitempty"`
}

global configuration

type ClusterConfig

type ClusterConfig struct {
	Backend     BackendConf     `json:"backend" allow:"cluster"`
	Ext         any             `json:"ext,omitempty"` // reserved
	WritePolicy WritePolicyConf `json:"write_policy"`  // object metadata write policy: (immediate | delayed | never)
	LastUpdated string          `json:"lastupdate_time"`
	UUID        string          `json:"uuid"`
	Dsort       DsortConf       `json:"distributed_sort"`
	Proxy       ProxyConf       `json:"proxy" allow:"cluster"`
	Cksum       CksumConf       `json:"checksum"`
	Auth        AuthConf        `json:"auth"`
	Tracing     TracingConf     `json:"tracing"`
	TCB         TCBConf         `json:"tcb"`
	TCO         TCOConf         `json:"tco"`
	Arch        ArchConf        `json:"arch"`
	RateLimit   RateLimitConf   `json:"rate_limit"`
	Keepalive   KeepaliveConf   `json:"keepalivetracker"`
	Rebalance   RebalanceConf   `json:"rebalance" allow:"cluster"`
	Log         LogConf         `json:"log"`
	EC          ECConf          `json:"ec" allow:"cluster"`
	Net         NetConf         `json:"net"`
	Timeout     TimeoutConf     `json:"timeout"`
	Transport   TransportConf   `json:"transport"`
	Memsys      MemsysConf      `json:"memsys"`
	FSHC        FSHCConf        `json:"fshc"`
	Disk        DiskConf        `json:"disk"`
	Space       SpaceConf       `json:"space"`
	Periodic    PeriodConf      `json:"periodic"`
	Client      ClientConf      `json:"client"`
	Mirror      MirrorConf      `json:"mirror" allow:"cluster"`
	LRU         LRUConf         `json:"lru"`
	Downloader  DownloaderConf  `json:"downloader"`
	Features    feat.Flags      `json:"features,string" allow:"cluster"` // enumerated features to flip assorted global defaults (cmn/feat/feat.go)
	Version     int64           `json:"config_version,string"`
	Versioning  VersionConf     `json:"versioning" allow:"cluster"`
	Resilver    ResilverConf    `json:"resilver"`
}

global configuration

func (*ClusterConfig) Apply

func (c *ClusterConfig) Apply(updateConf *ConfigToSet, asType string) error

func (*ClusterConfig) JspOpts

func (*ClusterConfig) JspOpts() jsp.Options

func (*ClusterConfig) String

func (c *ClusterConfig) String() string

type Config

type Config struct {
	LocalConfig   `json:",inline"`
	ClusterConfig `json:",inline"`
	// contains filtered or unexported fields
}

Config is a single control structure (persistent and versioned) that contains both cluster (global) and node (local) configuration Naming convention for setting/getting values: (parent section json tag . child json tag) See also: `IterFields`, `IterFieldNameSepa`

func (*Config) SetRole

func (c *Config) SetRole(role string)

func (*Config) TestingEnv

func (c *Config) TestingEnv() bool

TestingEnv returns true if config is set to a development environment where a single local filesystem is partitioned between all (locally running) targets and is used for both local and Cloud buckets See also: `rom.testingEnv`

func (*Config) UpdateClusterConfig

func (c *Config) UpdateClusterConfig(updateConf *ConfigToSet, asType string) (err error)

func (*Config) Validate

func (c *Config) Validate() error

main config validator

type ConfigToSet added in v1.3.21

type ConfigToSet struct {
	// ClusterConfig
	Backend     *BackendConf          `json:"backend,omitempty"`
	Mirror      *MirrorConfToSet      `json:"mirror,omitempty"`
	EC          *ECConfToSet          `json:"ec,omitempty"`
	Log         *LogConfToSet         `json:"log,omitempty"`
	Periodic    *PeriodConfToSet      `json:"periodic,omitempty"`
	Tracing     *TracingConfToSet     `json:"tracing,omitempty"`
	Timeout     *TimeoutConfToSet     `json:"timeout,omitempty"`
	Client      *ClientConfToSet      `json:"client,omitempty"`
	Space       *SpaceConfToSet       `json:"space,omitempty"`
	LRU         *LRUConfToSet         `json:"lru,omitempty"`
	Disk        *DiskConfToSet        `json:"disk,omitempty"`
	Rebalance   *RebalanceConfToSet   `json:"rebalance,omitempty"`
	Resilver    *ResilverConfToSet    `json:"resilver,omitempty"`
	Cksum       *CksumConfToSet       `json:"checksum,omitempty"`
	Versioning  *VersionConfToSet     `json:"versioning,omitempty"`
	Net         *NetConfToSet         `json:"net,omitempty"`
	FSHC        *FSHCConfToSet        `json:"fshc,omitempty"`
	Auth        *AuthConfToSet        `json:"auth,omitempty"`
	Keepalive   *KeepaliveConfToSet   `json:"keepalivetracker,omitempty"`
	Downloader  *DownloaderConfToSet  `json:"downloader,omitempty"`
	Dsort       *DsortConfToSet       `json:"distributed_sort,omitempty"`
	Transport   *TransportConfToSet   `json:"transport,omitempty"`
	Memsys      *MemsysConfToSet      `json:"memsys,omitempty"`
	TCB         *TCBConfToSet         `json:"tcb,omitempty"`
	WritePolicy *WritePolicyConfToSet `json:"write_policy,omitempty"`
	Proxy       *ProxyConfToSet       `json:"proxy,omitempty"`
	RateLimit   *RateLimitConfToSet   `json:"rate_limit"`
	Features    *feat.Flags           `json:"features,string,omitempty"`

	// LocalConfig
	FSP *FSPConf `json:"fspaths,omitempty"`
}

global configuration

func (*ConfigToSet) FillFromKVS added in v1.3.21

func (ctu *ConfigToSet) FillFromKVS(kvs []string) (err error)

FillFromKVS populates `ConfigToSet` from key value pairs of the form `key=value`

func (*ConfigToSet) FillFromQuery added in v1.3.21

func (ctu *ConfigToSet) FillFromQuery(query url.Values) error

FillFromQuery populates ConfigToSet from URL query values

func (*ConfigToSet) JspOpts added in v1.3.21

func (*ConfigToSet) JspOpts() jsp.Options

func (*ConfigToSet) Merge added in v1.3.21

func (ctu *ConfigToSet) Merge(update *ConfigToSet)

type DiskConf

type DiskConf struct {
	DiskUtilLowWM   int64        `json:"disk_util_low_wm"`  // no throttling below
	DiskUtilHighWM  int64        `json:"disk_util_high_wm"` // throttle longer when above
	DiskUtilMaxWM   int64        `json:"disk_util_max_wm"`
	IostatTimeLong  cos.Duration `json:"iostat_time_long"`
	IostatTimeShort cos.Duration `json:"iostat_time_short"`
}

global configuration

func (*DiskConf) Validate

func (c *DiskConf) Validate() (err error)

type DiskConfToSet added in v1.3.21

type DiskConfToSet struct {
	DiskUtilLowWM   *int64        `json:"disk_util_low_wm,omitempty"`
	DiskUtilHighWM  *int64        `json:"disk_util_high_wm,omitempty"`
	DiskUtilMaxWM   *int64        `json:"disk_util_max_wm,omitempty"`
	IostatTimeLong  *cos.Duration `json:"iostat_time_long,omitempty"`
	IostatTimeShort *cos.Duration `json:"iostat_time_short,omitempty"`
}

global configuration

type DownloaderConf

type DownloaderConf struct {
	Timeout cos.Duration `json:"timeout"`
}

global configuration

func (*DownloaderConf) Validate

func (c *DownloaderConf) Validate() error

type DownloaderConfToSet added in v1.3.21

type DownloaderConfToSet struct {
	Timeout *cos.Duration `json:"timeout,omitempty"`
}

global configuration

type DsortConf added in v1.3.21

type DsortConf struct {
	XactConf
	DuplicatedRecords   string       `json:"duplicated_records"`
	MissingShards       string       `json:"missing_shards"` // cmn.SupportedReactions enum
	EKMMalformedLine    string       `json:"ekm_malformed_line"`
	EKMMissingKey       string       `json:"ekm_missing_key"`
	DefaultMaxMemUsage  string       `json:"default_max_mem_usage"`
	DsorterMemThreshold string       `json:"dsorter_mem_threshold"`
	CallTimeout         cos.Duration `json:"call_timeout"`
}

global configuration

func (*DsortConf) Validate added in v1.3.21

func (c *DsortConf) Validate() (err error)

func (*DsortConf) ValidateWithOpts added in v1.3.21

func (c *DsortConf) ValidateWithOpts(allowEmpty bool) (err error)

type DsortConfToSet added in v1.3.21

type DsortConfToSet struct {
	XactConfToSet
	DuplicatedRecords   *string       `json:"duplicated_records,omitempty"`
	MissingShards       *string       `json:"missing_shards,omitempty"`
	EKMMalformedLine    *string       `json:"ekm_malformed_line,omitempty"`
	EKMMissingKey       *string       `json:"ekm_missing_key,omitempty"`
	DefaultMaxMemUsage  *string       `json:"default_max_mem_usage,omitempty"`
	CallTimeout         *cos.Duration `json:"call_timeout,omitempty"`
	DsorterMemThreshold *string       `json:"dsorter_mem_threshold,omitempty"`
}

global configuration

type ECConf

type ECConf struct {
	XactConf

	// ObjSizeLimit is object size threshold _separating_ intra-cluster mirroring from
	// erasure coding.
	//
	// The value 0 (zero) indicates that objects of any size
	// are to be sliced, to produce (D) data slices and (P) erasure coded parity slices.
	// On the other hand, the value -1 specifies that absolutely all objects of any size
	// must be replicated as is. In effect, the (-1) option provides data protection via
	// intra-cluster (P+1)-way replication (a.k.a. mirroring).
	//
	// In all cases, a given (D, P) configuration provides node-level redundancy,
	// whereby P nodes can be lost without incurring loss of data.
	ObjSizeLimit int64 `json:"objsize_limit"`

	// Number of data (D) slices; the value 1 will have an effect of producing
	// (P) additional full-size replicas.
	DataSlices int `json:"data_slices"`

	// Depending on the object size and `ObjSizeLimit`, the value of `ParitySlices` (or P) indicates:
	// - a number of additional parity slices (generated or _computed_ from the (D) data slices),
	// or:
	// - a number of full object replicas (copies).
	// In all cases, the same rule applies: all slices and/or all full copies are stored on different
	// storage nodes (a.k.a. targets).
	ParitySlices int `json:"parity_slices"`

	Enabled  bool `json:"enabled"`   // EC is enabled
	DiskOnly bool `json:"disk_only"` // if true, EC does not use SGL - data goes directly to drives
}

global configuration

func (*ECConf) RequiredRestoreTargets

func (c *ECConf) RequiredRestoreTargets() int

func (*ECConf) String

func (c *ECConf) String() string

func (*ECConf) Validate

func (c *ECConf) Validate() error

func (*ECConf) ValidateAsProps

func (c *ECConf) ValidateAsProps(arg ...any) (err error)

type ECConfToSet added in v1.3.21

type ECConfToSet struct {
	XactConfToSet
	ObjSizeLimit *int64 `json:"objsize_limit,omitempty"`
	DataSlices   *int   `json:"data_slices,omitempty"`
	ParitySlices *int   `json:"parity_slices,omitempty"`
	Enabled      *bool  `json:"enabled,omitempty"`
	DiskOnly     *bool  `json:"disk_only,omitempty"`
}

global configuration

type ETLErrCtx added in v1.3.16

type ETLErrCtx struct {
	TID              string
	ETLName          string
	ETLTransformArgs string
	PodName          string
	SvcName          string
	k8s.PodStatus
}

assorted aistore errors

type ErrAborted

type ErrAborted struct {
	// contains filtered or unexported fields
}

assorted aistore errors

func AsErrAborted

func AsErrAborted(err error) (errAborted *ErrAborted)

func NewErrAborted

func NewErrAborted(what, ctx string, err error) *ErrAborted

func (*ErrAborted) Error

func (e *ErrAborted) Error() (s string)

func (*ErrAborted) Unwrap

func (e *ErrAborted) Unwrap() (err error)

type ErrBckNotFound

type ErrBckNotFound struct {
	// contains filtered or unexported fields
}

assorted aistore errors

func NewErrBckNotFound

func NewErrBckNotFound(bck *Bck) *ErrBckNotFound

func (*ErrBckNotFound) Error

func (e *ErrBckNotFound) Error() string

type ErrBucketAccessDenied

type ErrBucketAccessDenied struct {
	// contains filtered or unexported fields
}

assorted aistore errors

func NewBucketAccessDenied

func NewBucketAccessDenied(bucket, oper string, aattrs apc.AccessAttrs) *ErrBucketAccessDenied

func (*ErrBucketAccessDenied) Error

func (e *ErrBucketAccessDenied) Error() string

func (*ErrBucketAccessDenied) String

func (e *ErrBucketAccessDenied) String() string

type ErrBucketAlreadyExists

type ErrBucketAlreadyExists struct {
	// contains filtered or unexported fields
}

assorted aistore errors

func NewErrBckAlreadyExists

func NewErrBckAlreadyExists(bck *Bck) *ErrBucketAlreadyExists

func (*ErrBucketAlreadyExists) Error

func (e *ErrBucketAlreadyExists) Error() string

type ErrBusy added in v1.3.21

type ErrBusy struct {
	// contains filtered or unexported fields
}

assorted aistore errors

func NewErrBusy added in v1.3.21

func NewErrBusy(whereOrType, what string, detail ...string) *ErrBusy

func (*ErrBusy) Error added in v1.3.21

func (e *ErrBusy) Error() string

type ErrCapExceeded added in v1.3.19

type ErrCapExceeded struct {
	// contains filtered or unexported fields
}

assorted aistore errors

func NewErrCapExceeded added in v1.3.19

func NewErrCapExceeded(totalBytesUsed, totalBytes uint64, highWM, cleanupWM int64, usedPct int32, oos bool) *ErrCapExceeded

func (*ErrCapExceeded) Error added in v1.3.19

func (e *ErrCapExceeded) Error() string

type ErrCreateHreq added in v1.3.26

type ErrCreateHreq struct {
	// contains filtered or unexported fields
}

assorted aistore errors

func NewErrCreateHreq added in v1.3.26

func NewErrCreateHreq(err error) *ErrCreateHreq

func (*ErrCreateHreq) Error added in v1.3.26

func (e *ErrCreateHreq) Error() string

func (*ErrCreateHreq) Unwrap added in v1.3.26

func (e *ErrCreateHreq) Unwrap() (err error)

type ErrETL

type ErrETL struct {
	Reason string
	ETLErrCtx
	Ecode int
}

assorted aistore errors

func NewErrETL

func NewErrETL(ctx *ETLErrCtx, reason string, ecode ...int) *ErrETL

func NewErrETLf added in v1.3.24

func NewErrETLf(ctx *ETLErrCtx, format string, a ...any) *ErrETL

func (*ErrETL) Error

func (e *ErrETL) Error() string

func (*ErrETL) WithContext

func (e *ErrETL) WithContext(ctx *ETLErrCtx) *ErrETL

func (*ErrETL) WithPodName

func (e *ErrETL) WithPodName(name string) *ErrETL

type ErrEmptyProvider added in v1.3.26

type ErrEmptyProvider struct {
	// contains filtered or unexported fields
}

func (*ErrEmptyProvider) Error added in v1.3.26

func (e *ErrEmptyProvider) Error() string

type ErrFailedTo

type ErrFailedTo struct {
	// contains filtered or unexported fields
}

assorted aistore errors

func NewErrFailedTo

func NewErrFailedTo(actor fmt.Stringer, action string, what any, err error, ecode ...int) *ErrFailedTo

func (*ErrFailedTo) Error

func (e *ErrFailedTo) Error() string

func (*ErrFailedTo) Unwrap

func (e *ErrFailedTo) Unwrap() (err error)

type ErrGetCap added in v1.3.24

type ErrGetCap struct {
	// contains filtered or unexported fields
}

assorted aistore errors

func NewErrGetCap added in v1.3.24

func NewErrGetCap(err error) *ErrGetCap

func (*ErrGetCap) Error added in v1.3.24

func (e *ErrGetCap) Error() string

type ErrHTTP

type ErrHTTP struct {
	TypeCode   string `json:"tcode,omitempty"`
	Message    string `json:"message"`
	Method     string `json:"method"`
	URLPath    string `json:"url_path"`
	RemoteAddr string `json:"remote_addr"`
	Caller     string `json:"caller"`
	Node       string `json:"node"`

	Status int `json:"status"`
	// contains filtered or unexported fields
}

API error structure is returned to aistore client and carries one of the specific errors enumerated below

func InitErrHTTP

func InitErrHTTP(r *http.Request, err error, ecode int) (e *ErrHTTP)

uses `allocHterr` to allocate - caller must free via `FreeHterr`

func NewErrHTTP

func NewErrHTTP(r *http.Request, err error, ecode int) (e *ErrHTTP)

func Str2HTTPErr added in v1.3.18

func Str2HTTPErr(msg string) *ErrHTTP

func UnwrapErrHTTP added in v1.3.30

func UnwrapErrHTTP(err error) *ErrHTTP

func (*ErrHTTP) Error

func (e *ErrHTTP) Error() (s string)

func (*ErrHTTP) StringEx added in v1.3.18

func (e *ErrHTTP) StringEx() (s string)

Example: ErrBckNotFound: bucket "ais://abc" does not exist: HEAD /v1/buckets/abc (p[kWQp8080]: htrun.go:1035 <- prxtrybck.go:180 <- ...

type ErrInitBackend

type ErrInitBackend struct {
	Provider string
}

assorted aistore errors

func (*ErrInitBackend) Error

func (e *ErrInitBackend) Error() string

type ErrInvalidBackendProvider

type ErrInvalidBackendProvider struct {
	// contains filtered or unexported fields
}

assorted aistore errors

func (*ErrInvalidBackendProvider) Error

func (e *ErrInvalidBackendProvider) Error() string

func (*ErrInvalidBackendProvider) Is

type ErrInvalidCksum

type ErrInvalidCksum struct {
	// contains filtered or unexported fields
}

assorted aistore errors

func NewErrInvalidCksum

func NewErrInvalidCksum(eHash, aHash string) *ErrInvalidCksum

func (*ErrInvalidCksum) Error

func (e *ErrInvalidCksum) Error() string

func (*ErrInvalidCksum) Expected

func (e *ErrInvalidCksum) Expected() string

type ErrInvalidFSPathsConf

type ErrInvalidFSPathsConf struct {
	// contains filtered or unexported fields
}

assorted aistore errors

func NewErrInvalidFSPathsConf

func NewErrInvalidFSPathsConf(err error) *ErrInvalidFSPathsConf

func (*ErrInvalidFSPathsConf) Error

func (e *ErrInvalidFSPathsConf) Error() string

func (*ErrInvalidFSPathsConf) Unwrap

func (e *ErrInvalidFSPathsConf) Unwrap() (err error)

type ErrInvalidMountpath

type ErrInvalidMountpath struct {
	// contains filtered or unexported fields
}

assorted aistore errors

func NewErrInvalidaMountpath

func NewErrInvalidaMountpath(mpath, cause string) *ErrInvalidMountpath

func (*ErrInvalidMountpath) Error

func (e *ErrInvalidMountpath) Error() string

type ErrLimitedCoexistence

type ErrLimitedCoexistence struct {
	// contains filtered or unexported fields
}

assorted aistore errors

func NewErrLimitedCoexistence

func NewErrLimitedCoexistence(node, xaction, action, detail string) *ErrLimitedCoexistence

func (*ErrLimitedCoexistence) Error

func (e *ErrLimitedCoexistence) Error() string

type ErrLmetaCorrupted

type ErrLmetaCorrupted struct {
	// contains filtered or unexported fields
}

assorted aistore errors

func NewErrLmetaCorrupted

func NewErrLmetaCorrupted(err error) *ErrLmetaCorrupted

func (*ErrLmetaCorrupted) Error

func (e *ErrLmetaCorrupted) Error() string

func (*ErrLmetaCorrupted) Unwrap

func (e *ErrLmetaCorrupted) Unwrap() (err error)

type ErrLmetaNotFound

type ErrLmetaNotFound struct {
	// contains filtered or unexported fields
}

assorted aistore errors

func NewErrLmetaNotFound

func NewErrLmetaNotFound(name string, err error) *ErrLmetaNotFound

func (*ErrLmetaNotFound) Error

func (e *ErrLmetaNotFound) Error() string

func (*ErrLmetaNotFound) Unwrap

func (e *ErrLmetaNotFound) Unwrap() (err error)

type ErrMembershipChanges added in v1.3.28

type ErrMembershipChanges struct {
	// contains filtered or unexported fields
}

assorted aistore errors

func NewErrMembershipChanges added in v1.3.28

func NewErrMembershipChanges(info string) *ErrMembershipChanges

func (*ErrMembershipChanges) Error added in v1.3.28

func (e *ErrMembershipChanges) Error() string

type ErrMissingBackend

type ErrMissingBackend struct {
	Provider string
	Msg      string
}

assorted aistore errors

func (*ErrMissingBackend) Error

func (e *ErrMissingBackend) Error() string

type ErrMpathCheck added in v1.3.24

type ErrMpathCheck struct {
	// contains filtered or unexported fields
}

assorted aistore errors

func NewErrMpathCheck added in v1.3.24

func NewErrMpathCheck(err error) *ErrMpathCheck

func (*ErrMpathCheck) Error added in v1.3.24

func (e *ErrMpathCheck) Error() string

type ErrMpathLostDisk added in v1.3.24

type ErrMpathLostDisk struct {
	// contains filtered or unexported fields
}

assorted aistore errors

func NewErrMpathLostDisk added in v1.3.24

func NewErrMpathLostDisk(mpath, fsname, lostd string, disks, fsdisks []string) *ErrMpathLostDisk

func (*ErrMpathLostDisk) Error added in v1.3.24

func (e *ErrMpathLostDisk) Error() string

type ErrMpathNewDisk added in v1.3.24

type ErrMpathNewDisk struct {
	// contains filtered or unexported fields
}

assorted aistore errors

func NewErrMpathNewDisk added in v1.3.24

func NewErrMpathNewDisk(mpath, fsname string, disks, fsdisks []string) *ErrMpathNewDisk

func (*ErrMpathNewDisk) Error added in v1.3.24

func (e *ErrMpathNewDisk) Error() string

type ErrMpathNoDisks added in v1.3.24

type ErrMpathNoDisks struct {
	// contains filtered or unexported fields
}

assorted aistore errors

func NewErrMpathNoDisks added in v1.3.24

func NewErrMpathNoDisks(mpath, fsname string, err error) *ErrMpathNoDisks

func (*ErrMpathNoDisks) Error added in v1.3.24

func (e *ErrMpathNoDisks) Error() string

type ErrMpathNotFound added in v1.3.24

type ErrMpathNotFound struct {
	// contains filtered or unexported fields
}

assorted aistore errors

func NewErrMpathNotFound added in v1.3.24

func NewErrMpathNotFound(mpath, fqn string, disabled bool) *ErrMpathNotFound

func (*ErrMpathNotFound) Disabled added in v1.3.24

func (e *ErrMpathNotFound) Disabled() bool

func (*ErrMpathNotFound) Error added in v1.3.24

func (e *ErrMpathNotFound) Error() string

func (*ErrMpathNotFound) Mpath added in v1.3.24

func (e *ErrMpathNotFound) Mpath() string

type ErrNoNodes

type ErrNoNodes struct {
	// contains filtered or unexported fields
}

assorted aistore errors

func NewErrNoNodes

func NewErrNoNodes(role string, mmcount int) *ErrNoNodes

func (*ErrNoNodes) Error

func (e *ErrNoNodes) Error() (s string)

type ErrNotImpl

type ErrNotImpl struct {
	// contains filtered or unexported fields
}

assorted aistore errors

func NewErrNotImpl

func NewErrNotImpl(action, what string) *ErrNotImpl

func (*ErrNotImpl) Error

func (e *ErrNotImpl) Error() string

type ErrNotRemoteBck added in v1.3.22

type ErrNotRemoteBck struct {
	// contains filtered or unexported fields
}

assorted aistore errors

func ValidateRemoteBck added in v1.3.22

func ValidateRemoteBck(act string, bck *Bck) (err *ErrNotRemoteBck)

func (*ErrNotRemoteBck) Error added in v1.3.22

func (e *ErrNotRemoteBck) Error() string

type ErrObjDefunct

type ErrObjDefunct struct {
	// contains filtered or unexported fields
}

assorted aistore errors

func NewErrObjDefunct

func NewErrObjDefunct(name string, d1, d2 uint64) *ErrObjDefunct

func (*ErrObjDefunct) Error

func (e *ErrObjDefunct) Error() string

type ErrObjectAccessDenied

type ErrObjectAccessDenied struct {
	// contains filtered or unexported fields
}

assorted aistore errors

func NewObjectAccessDenied

func NewObjectAccessDenied(object, oper string, aattrs apc.AccessAttrs) *ErrObjectAccessDenied

func (*ErrObjectAccessDenied) Error

func (e *ErrObjectAccessDenied) Error() string

func (*ErrObjectAccessDenied) String

func (e *ErrObjectAccessDenied) String() string

type ErrRangeNotSatisfiable added in v1.3.22

type ErrRangeNotSatisfiable struct {
	// contains filtered or unexported fields
}

assorted aistore errors

func NewErrRangeNotSatisfiable added in v1.3.22

func NewErrRangeNotSatisfiable(err error, ranges []string, size int64) *ErrRangeNotSatisfiable

func (*ErrRangeNotSatisfiable) Error added in v1.3.22

func (e *ErrRangeNotSatisfiable) Error() string

type ErrRateLimitFrontend added in v1.3.28

type ErrRateLimitFrontend ErrTooManyRequests // to differentiate by tcode

assorted aistore errors

func NewErrRateLimitFrontend added in v1.3.28

func NewErrRateLimitFrontend() *ErrRateLimitFrontend

func (*ErrRateLimitFrontend) Error added in v1.3.28

func (e *ErrRateLimitFrontend) Error() string

type ErrRemoteBckNotFound

type ErrRemoteBckNotFound struct {
	// contains filtered or unexported fields
}

assorted aistore errors

func NewErrRemoteBckNotFound

func NewErrRemoteBckNotFound(bck *Bck) *ErrRemoteBckNotFound

func (*ErrRemoteBckNotFound) Error

func (e *ErrRemoteBckNotFound) Error() string

func (*ErrRemoteBckNotFound) Set added in v1.3.26

func (e *ErrRemoteBckNotFound) Set(ctx string)

type ErrRemoteBucketOffline

type ErrRemoteBucketOffline struct {
	// contains filtered or unexported fields
}

assorted aistore errors

func NewErrRemoteBckOffline

func NewErrRemoteBckOffline(bck *Bck) *ErrRemoteBucketOffline

func (*ErrRemoteBucketOffline) Error

func (e *ErrRemoteBucketOffline) Error() string

type ErrRemoteMetadataMismatch added in v1.3.24

type ErrRemoteMetadataMismatch struct {
	// contains filtered or unexported fields
}

assorted aistore errors

func (*ErrRemoteMetadataMismatch) Error added in v1.3.24

func (e *ErrRemoteMetadataMismatch) Error() string

type ErrStreamTerminated added in v1.3.16

type ErrStreamTerminated struct {
	// contains filtered or unexported fields
}

assorted aistore errors

func NewErrStreamTerminated added in v1.3.16

func NewErrStreamTerminated(stream string, err error, reason, detail string) *ErrStreamTerminated

func (*ErrStreamTerminated) Error added in v1.3.16

func (e *ErrStreamTerminated) Error() string

func (*ErrStreamTerminated) Unwrap added in v1.3.16

func (e *ErrStreamTerminated) Unwrap() (err error)

type ErrTooManyRequests added in v1.3.28

type ErrTooManyRequests struct {
	// contains filtered or unexported fields
}

assorted aistore errors

func NewErrTooManyRequests added in v1.3.28

func NewErrTooManyRequests(err error, status int) *ErrTooManyRequests

func (*ErrTooManyRequests) Error added in v1.3.28

func (e *ErrTooManyRequests) Error() string

type ErrUnsupp

type ErrUnsupp struct {
	// contains filtered or unexported fields
}

assorted aistore errors

func NewErrUnsupp

func NewErrUnsupp(action, what string) *ErrUnsupp

func NewErrUnsuppErr added in v1.3.25

func NewErrUnsuppErr(err error) *ErrUnsupp

func (*ErrUnsupp) Error

func (e *ErrUnsupp) Error() string

type ErrWarning added in v1.3.24

type ErrWarning struct {
	// contains filtered or unexported fields
}

assorted aistore errors

func NewErrWarning added in v1.3.24

func NewErrWarning(what string) *ErrWarning

func (*ErrWarning) Error added in v1.3.24

func (e *ErrWarning) Error() string

type ErrXactNotFound added in v1.3.16

type ErrXactNotFound struct {
	// contains filtered or unexported fields
}

assorted aistore errors

func NewErrXactNotFoundError

func NewErrXactNotFoundError(cause string) *ErrXactNotFound

func (*ErrXactNotFound) Error added in v1.3.16

func (e *ErrXactNotFound) Error() string

type ErrXactTgtInMaint added in v1.3.18

type ErrXactTgtInMaint struct {
	// contains filtered or unexported fields
}

assorted aistore errors

func NewErrXactTgtInMaint added in v1.3.18

func NewErrXactTgtInMaint(xaction, tname string) *ErrXactTgtInMaint

func (*ErrXactTgtInMaint) Error added in v1.3.18

func (e *ErrXactTgtInMaint) Error() string

type ErrXactUsePrev added in v1.3.18

type ErrXactUsePrev struct {
	// contains filtered or unexported fields
}

assorted aistore errors

func NewErrXactUsePrev added in v1.3.18

func NewErrXactUsePrev(xaction string) *ErrXactUsePrev

func (*ErrXactUsePrev) Error added in v1.3.18

func (e *ErrXactUsePrev) Error() string

type ExtraProps

type ExtraProps struct {
	HTTP ExtraPropsHTTP `json:"http,omitempty" list:"omitempty"`
	HDFS ExtraPropsHDFS `json:"hdfs,omitempty" list:"omitempty"` // NOTE: obsolete; rm with meta-version
	AWS  ExtraPropsAWS  `json:"aws,omitempty" list:"omitempty"`
}

func (*ExtraProps) ValidateAsProps

func (c *ExtraProps) ValidateAsProps(arg ...any) error

type ExtraPropsAWS

type ExtraPropsAWS struct {
	CloudRegion string `json:"cloud_region,omitempty"`

	// from https://github.com/aws/aws-sdk-go/blob/main/aws/config.go:
	// - "An optional endpoint URL (hostname only or fully qualified URI)
	// that overrides the default generated endpoint."
	Endpoint string `json:"endpoint,omitempty"`

	// from https://github.com/aws/aws-sdk-go/blob/main/aws/session/session.go:
	// - "Overrides the config profile the Session should be created from. If not
	// set the value of the environment variable will be loaded (AWS_PROFILE,
	// or AWS_DEFAULT_PROFILE if the Shared Config is enabled)."
	Profile string `json:"profile,omitempty"`

	// Amazon S3: 1000
	// - https://docs.aws.amazon.com/cli/latest/userguide/cli-usage-pagination.html#cli-usage-pagination-serverside
	// vs OpenStack Swift: 10,000
	// - https://docs.openstack.org/swift/latest/api/pagination.html
	MaxPageSize int64 `json:"max_pagesize,omitempty"`

	// Multipart upload size threshold must be greater or equal 5MB
	// - https://pkg.go.dev/github.com/aws/aws-sdk-go-v2/feature/s3/manager
	// - for the AIS default, see `DefaultPartSize` in ais/s3/const
	// - NOTE: the threshold is, effectively, one of the **performance tunables**
	MultiPartSize cos.SizeIEC `json:"multipart_size,omitempty"`
}

type ExtraPropsAWSToSet added in v1.3.21

type ExtraPropsAWSToSet struct {
	CloudRegion   *string      `json:"cloud_region,omitempty"`
	Endpoint      *string      `json:"endpoint,omitempty"`
	Profile       *string      `json:"profile,omitempty"`
	MaxPageSize   *int64       `json:"max_pagesize,omitempty"`
	MultiPartSize *cos.SizeIEC `json:"multipart_size,omitempty"`
}

type ExtraPropsHDFS

type ExtraPropsHDFS struct {
	// Reference directory.
	RefDirectory string `json:"ref_directory,omitempty"`
}

type ExtraPropsHDFSToSet added in v1.3.21

type ExtraPropsHDFSToSet struct {
	RefDirectory *string `json:"ref_directory"`
}

type ExtraPropsHTTP

type ExtraPropsHTTP struct {
	// Original URL prior to hashing.
	OrigURLBck string `json:"original_url,omitempty" list:"readonly"`
}

type ExtraPropsHTTPToSet added in v1.3.21

type ExtraPropsHTTPToSet struct {
	OrigURLBck *string `json:"original_url"`
}

type ExtraToSet added in v1.3.21

type ExtraToSet struct {
	AWS  *ExtraPropsAWSToSet  `json:"aws"`
	HTTP *ExtraPropsHTTPToSet `json:"http"`
	HDFS *ExtraPropsHDFSToSet `json:"hdfs"` // ditto
}

type FSHCConf

type FSHCConf struct {
	TestFileCount int `json:"test_files"` // number of files to read/write
	// critical and unexpected errors detected during FSHC run;
	// exceeding the limit "triggers" FSHC that may, in turn, disable the corresponding mountpath
	HardErrs int `json:"error_limit"`

	// - maximum number of I/O errors during the last `IOErrTime` interval;
	// - the number does not include network error (e.g., connection reset by peer)
	//   and errors returned by remote backends;
	// - exceeding this limit is also an FSHC-trggering event; subsequently,
	//   if FSHC confirms the problem it will disable the mountpath (see above)
	IOErrs int `json:"io_err_limit,omitempty"`
	// time interval (in seconds) to accumulate soft errors;
	// the total number by the end of the interval must not exceed `IOErrs` (above)
	IOErrTime cos.Duration `json:"io_err_time,omitempty"`

	// whether FSHC is enabled (note: disabling FSHC is _not_ recommended)
	Enabled bool `json:"enabled"`
}

global configuration

func (*FSHCConf) Validate added in v1.3.24

func (c *FSHCConf) Validate() error

type FSHCConfToSet added in v1.3.21

type FSHCConfToSet struct {
	TestFileCount *int          `json:"test_files,omitempty"`
	HardErrs      *int          `json:"error_limit,omitempty"`
	IOErrs        *int          `json:"io_err_limit,omitempty"`
	IOErrTime     *cos.Duration `json:"io_err_time,omitempty"`
	Enabled       *bool         `json:"enabled,omitempty"`
}

global configuration

type FSPConf

type FSPConf struct {
	Paths cos.StrKVs `json:"paths,omitempty" list:"readonly"`
}

ais node: fspaths (a.k.a. mountpaths) and their respective (optional) labels

func (*FSPConf) MarshalJSON

func (c *FSPConf) MarshalJSON() ([]byte, error)

func (*FSPConf) UnmarshalJSON

func (c *FSPConf) UnmarshalJSON(data []byte) error

func (*FSPConf) Validate

func (c *FSPConf) Validate(contextConfig *Config) error

type HTTPBckObj

type HTTPBckObj struct {
	Bck        Bck
	ObjName    string
	OrigURLBck string // HTTP URL of the bucket (object name excluded)
}

Represents the AIS bucket, object and URL associated with a HTTP resource

func NewHTTPObj

func NewHTTPObj(u *url.URL) *HTTPBckObj

func NewHTTPObjPath

func NewHTTPObjPath(rawURL string) (*HTTPBckObj, error)

type HTTPConf

type HTTPConf struct {
	Proto         string `json:"-"`             // http or https (set depending on `UseHTTPS`)
	Certificate   string `json:"server_crt"`    // HTTPS: X.509 certificate
	CertKey       string `json:"server_key"`    // HTTPS: X.509 key
	ServerNameTLS string `json:"domain_tls"`    // #6410
	ClientCA      string `json:"client_ca_tls"` // #6410
	// clients "idle" config: intra-cluster and backend
	IdleConnTimeout     cos.Duration `json:"idle_conn_time"`
	MaxIdleConnsPerHost int          `json:"idle_conns_per_host"`
	MaxIdleConns        int          `json:"idle_conns"`
	// cont-d
	ClientAuthTLS   int  `json:"client_auth_tls"`   // #6410 tls.ClientAuthType enum
	WriteBufferSize int  `json:"write_buffer_size"` // http.Transport.WriteBufferSize; zero defaults to 4KB
	ReadBufferSize  int  `json:"read_buffer_size"`  // http.Transport.ReadBufferSize; ditto
	UseHTTPS        bool `json:"use_https"`         // use HTTPS
	SkipVerifyCrt   bool `json:"skip_verify"`       // skip X.509 cert verification (used with self-signed certs)
	Chunked         bool `json:"chunked_transfer"`  // (https://tools.ietf.org/html/rfc7230#page-36; not used since 02/23)
}

global configuration

func (*HTTPConf) ToTLS added in v1.3.21

func (c *HTTPConf) ToTLS() TLSArgs

used intra-clients; see related: EnvToTLS()

func (*HTTPConf) Validate added in v1.3.22

func (c *HTTPConf) Validate() error

type HTTPConfToSet added in v1.3.21

type HTTPConfToSet struct {
	Certificate   *string `json:"server_crt,omitempty"`
	CertKey       *string `json:"server_key,omitempty"`
	ServerNameTLS *string `json:"domain_tls,omitempty"`
	ClientCA      *string `json:"client_ca_tls,omitempty"`
	// added v3.26
	IdleConnTimeout     *cos.Duration `json:"idle_conn_time,omitempty"`
	MaxIdleConnsPerHost *int          `json:"idle_conns_per_host,omitempty"`
	MaxIdleConns        *int          `json:"idle_conns,omitempty"`
	// cont-d
	WriteBufferSize *int  `json:"write_buffer_size,omitempty" list:"readonly"`
	ReadBufferSize  *int  `json:"read_buffer_size,omitempty" list:"readonly"`
	ClientAuthTLS   *int  `json:"client_auth_tls,omitempty"`
	UseHTTPS        *bool `json:"use_https,omitempty"`
	SkipVerifyCrt   *bool `json:"skip_verify,omitempty"`
	Chunked         *bool `json:"chunked_transfer,omitempty"`
}

global configuration

type HreqArgs

type HreqArgs struct {
	BodyR    io.ReadCloser
	Header   http.Header // request headers
	Query    url.Values  // query, e.g. ?a=x&b=y&c=z
	RawQuery string      // raw query
	Method   string
	Base     string // base URL, e.g. http://xyz.abc
	Path     string // path URL, e.g. /x/y/z
	Body     []byte
}

usage 1: initialize and fill out HTTP request. usage 2: intra-cluster control-plane (except streams) usage 3: PUT and APPEND API BodyR optimizes-out allocations - if non-nil and implements `io.Closer`, will always be closed by `client.Do`

func AllocHra

func AllocHra() (a *HreqArgs)

func (*HreqArgs) Req

func (u *HreqArgs) Req() (*http.Request, error)

NOTE: unlike standard http.NewRequest (above) - this method returns context-less request - it also does not assign req.GetBody - it is the caller's responsibility to assign one when and if required

func (*HreqArgs) ReqDeprecated added in v1.3.26

func (u *HreqArgs) ReqDeprecated() (*http.Request, error)

TODO: remaining usage - `api` package (remove when time permits)

func (*HreqArgs) ReqWith added in v1.3.26

TODO: HreqFree(original req) vs "shallow copy" produced by WithContext

func (*HreqArgs) URL

func (u *HreqArgs) URL() string

type IterField

type IterField interface {
	Value() any                          // returns the value
	String() string                      // string representation of the value
	SetValue(v any, force ...bool) error // `force` ignores `tagReadonly` (to be used with caution!)
}

Represents a single named field

type IterOpts

type IterOpts struct {
	// Skip fields based on allowed tag
	Allowed string
	// Visits all the fields, not only the leaves.
	VisitAll bool
	// Read-only walk is true by default (compare with `UpdateFieldValue`)
	// Note that `tagOmitempty` is limited to read-only - has no effect when `OnlyRead == false`.
	OnlyRead bool
}

type KeepaliveConf

type KeepaliveConf struct {
	Proxy       KeepaliveTrackerConf `json:"proxy"`       // how proxy tracks target keepalives
	Target      KeepaliveTrackerConf `json:"target"`      // how target tracks primary proxies keepalives
	NumRetries  int                  `json:"num_retries"` // default: `kaNumRetries`
	RetryFactor uint8                `json:"retry_factor"`
}

keepalive

func (*KeepaliveConf) Validate

func (c *KeepaliveConf) Validate() error

type KeepaliveConfToSet added in v1.3.21

type KeepaliveConfToSet struct {
	Proxy       *KeepaliveTrackerConfToSet `json:"proxy,omitempty"`
	Target      *KeepaliveTrackerConfToSet `json:"target,omitempty"`
	NumRetries  *int                       `json:"num_retries,omitempty"`
	RetryFactor *uint8                     `json:"retry_factor,omitempty"`
}

global configuration

type KeepaliveTrackerConf

type KeepaliveTrackerConf struct {
	Name     string       `json:"name"`     // "heartbeat"
	Interval cos.Duration `json:"interval"` // keepalive interval
	Factor   uint8        `json:"factor"`   // only average
}

global configuration

type KeepaliveTrackerConfToSet added in v1.3.21

type KeepaliveTrackerConfToSet struct {
	Interval *cos.Duration `json:"interval,omitempty"`
	Name     *string       `json:"name,omitempty" list:"readonly"`
	Factor   *uint8        `json:"factor,omitempty"`
}

global configuration

type L4Conf

type L4Conf struct {
	Proto         string `json:"proto"`           // tcp, udp
	SndRcvBufSize int    `json:"sndrcv_buf_size"` // SO_RCVBUF and SO_SNDBUF
}

global configuration

type LRUConf

type LRUConf struct {
	// DontEvictTimeStr denotes the period of time during which eviction of an object
	// is forbidden [atime, atime + DontEvictTime]
	DontEvictTime cos.Duration `json:"dont_evict_time"`

	// CapacityUpdTimeStr denotes the frequency at which AIStore updates local capacity utilization
	CapacityUpdTime cos.Duration `json:"capacity_upd_time"`

	// Enabled: LRU will only run when set to true
	Enabled bool `json:"enabled"`
}

global configuration

func (*LRUConf) String

func (c *LRUConf) String() string

func (*LRUConf) Validate

func (c *LRUConf) Validate() (err error)

type LRUConfToSet added in v1.3.21

type LRUConfToSet struct {
	DontEvictTime   *cos.Duration `json:"dont_evict_time,omitempty"`
	CapacityUpdTime *cos.Duration `json:"capacity_upd_time,omitempty"`
	Enabled         *bool         `json:"enabled,omitempty"`
}

global configuration

type LocalConfig

type LocalConfig struct {
	FSP       FSPConf        `json:"fspaths"`
	ConfigDir string         `json:"confdir"`
	LogDir    string         `json:"log_dir"`
	TestFSP   TestFSPConf    `json:"test_fspaths"`
	HostNet   LocalNetConfig `json:"host_net"`
}

local config

func (*LocalConfig) AddPath

func (c *LocalConfig) AddPath(mpath string)

func (*LocalConfig) DelPath

func (c *LocalConfig) DelPath(mpath string)

func (*LocalConfig) JspOpts

func (*LocalConfig) JspOpts() jsp.Options

func (*LocalConfig) TestingEnv

func (c *LocalConfig) TestingEnv() bool

type LocalNetConfig

type LocalNetConfig struct {
	Hostname             string `json:"hostname"`
	HostnameIntraControl string `json:"hostname_intra_control"`
	HostnameIntraData    string `json:"hostname_intra_data"`
	Port                 int    `json:"port,string"`               // listening port
	PortIntraControl     int    `json:"port_intra_control,string"` // --/-- for intra-cluster control
	PortIntraData        int    `json:"port_intra_data,string"`    // --/-- for intra-cluster data
	// omit
	UseIntraControl bool `json:"-"`
	UseIntraData    bool `json:"-"`
}

ais node: (local) network config

func (*LocalNetConfig) Validate

func (c *LocalNetConfig) Validate(contextConfig *Config) error

type LogConf

type LogConf struct {
	Level     cos.LogLevel `json:"level"`      // log level (aka verbosity)
	MaxSize   cos.SizeIEC  `json:"max_size"`   // exceeding this size triggers log rotation
	MaxTotal  cos.SizeIEC  `json:"max_total"`  // (sum individual log sizes); exceeding this number triggers cleanup
	FlushTime cos.Duration `json:"flush_time"` // log flush interval
	StatsTime cos.Duration `json:"stats_time"` // (not used)
	ToStderr  bool         `json:"to_stderr"`  // Log only to stderr instead of files.
}

global configuration

func (*LogConf) Validate

func (c *LogConf) Validate() error

type LogConfToSet added in v1.3.21

type LogConfToSet struct {
	Level     *cos.LogLevel `json:"level,omitempty"`
	ToStderr  *bool         `json:"to_stderr,omitempty"`
	MaxSize   *cos.SizeIEC  `json:"max_size,omitempty"`
	MaxTotal  *cos.SizeIEC  `json:"max_total,omitempty"`
	FlushTime *cos.Duration `json:"flush_time,omitempty"`
	StatsTime *cos.Duration `json:"stats_time,omitempty"`
}

global configuration

type LsoEnt added in v1.3.23

type LsoEnt struct {
	Name     string `json:"name" msg:"n"`                            // object name
	Checksum string `json:"checksum,omitempty" msg:"cs,omitempty"`   // checksum
	Atime    string `json:"atime,omitempty" msg:"a,omitempty"`       // last access time; formatted as ListObjsMsg.TimeFormat
	Version  string `json:"version,omitempty" msg:"v,omitempty"`     // e.g., GCP int64 generation, AWS version (string), etc.
	Location string `json:"location,omitempty" msg:"t,omitempty"`    // [tnode:mountpath]
	Custom   string `json:"custom-md,omitempty" msg:"m,omitempty"`   // custom metadata: ETag, MD5, CRC, user-defined ...
	Size     int64  `json:"size,string,omitempty" msg:"s,omitempty"` // size in bytes
	Copies   int16  `json:"copies,omitempty" msg:"c,omitempty"`      // ## copies (NOTE: for non-replicated object copies == 1)
	Flags    uint16 `json:"flags,omitempty" msg:"f,omitempty"`       // enum { EntryIsCached, EntryIsDir, EntryInArch, ...}
}

a single entry in LsoRes.Entries (below); contains list-objects results for the corresponding (listed) object or archived file; `Flags` is a bit field where `EntryStatusBits` bits [0-4] are reserved for object status (all statuses are mutually exclusive)

func (*LsoEnt) CopyWithProps added in v1.3.23

func (be *LsoEnt) CopyWithProps(propsSet cos.StrSet) (ne *LsoEnt)

func (*LsoEnt) DecodeMsg added in v1.3.23

func (z *LsoEnt) DecodeMsg(dc *msgp.Reader) (err error)

DecodeMsg implements msgp.Decodable

func (*LsoEnt) EncodeMsg added in v1.3.23

func (z *LsoEnt) EncodeMsg(en *msgp.Writer) (err error)

EncodeMsg implements msgp.Encodable

func (*LsoEnt) IsAnyFlagSet added in v1.3.26

func (be *LsoEnt) IsAnyFlagSet(fl uint16) bool

func (*LsoEnt) IsPresent added in v1.3.23

func (be *LsoEnt) IsPresent() bool

flags: - see above "LsoEntry Flags" enum - keeping IsPresent for client convenience, with Set/IsAnyFlag covering for the rest - terms "cached", "present" and "in-cluster" - are interchangeable

func (*LsoEnt) IsStatusOK added in v1.3.23

func (be *LsoEnt) IsStatusOK() bool

location _status_

func (*LsoEnt) Msgsize added in v1.3.23

func (z *LsoEnt) Msgsize() (s int)

Msgsize returns an upper bound estimate of the number of bytes occupied by the serialized message

func (*LsoEnt) SetFlag added in v1.3.26

func (be *LsoEnt) SetFlag(fl uint16)

func (*LsoEnt) Status added in v1.3.23

func (be *LsoEnt) Status() uint16

type LsoEntries

type LsoEntries []*LsoEnt

func (*LsoEntries) DecodeMsg added in v1.3.22

func (z *LsoEntries) DecodeMsg(dc *msgp.Reader) (err error)

DecodeMsg implements msgp.Decodable

func (LsoEntries) EncodeMsg added in v1.3.22

func (z LsoEntries) EncodeMsg(en *msgp.Writer) (err error)

EncodeMsg implements msgp.Encodable

func (LsoEntries) Msgsize added in v1.3.22

func (z LsoEntries) Msgsize() (s int)

Msgsize returns an upper bound estimate of the number of bytes occupied by the serialized message

type LsoRes added in v1.3.23

type LsoRes struct {
	UUID              string     `json:"uuid"`
	ContinuationToken string     `json:"continuation_token"`
	Entries           LsoEntries `json:"entries"`
	Flags             uint32     `json:"flags"`
}

`api.ListObjects` and `Backend.ListObjects` results

func (*LsoRes) DecodeMsg added in v1.3.23

func (z *LsoRes) DecodeMsg(dc *msgp.Reader) (err error)

DecodeMsg implements msgp.Decodable

func (*LsoRes) EncodeMsg added in v1.3.23

func (z *LsoRes) EncodeMsg(en *msgp.Writer) (err error)

EncodeMsg implements msgp.Encodable

func (*LsoRes) Msgsize added in v1.3.23

func (z *LsoRes) Msgsize() (s int)

Msgsize returns an upper bound estimate of the number of bytes occupied by the serialized message

type MemsysConf

type MemsysConf struct {
	MinFree        cos.SizeIEC  `json:"min_free"`
	DefaultBufSize cos.SizeIEC  `json:"default_buf"`
	SizeToGC       cos.SizeIEC  `json:"to_gc"`
	HousekeepTime  cos.Duration `json:"hk_time"`
	MinPctTotal    int          `json:"min_pct_total"`
	MinPctFree     int          `json:"min_pct_free"`
}

global configuration

func (*MemsysConf) Validate

func (c *MemsysConf) Validate() (err error)

type MemsysConfToSet added in v1.3.21

type MemsysConfToSet struct {
	MinFree        *cos.SizeIEC  `json:"min_free,omitempty"`
	DefaultBufSize *cos.SizeIEC  `json:"default_buf,omitempty"`
	SizeToGC       *cos.SizeIEC  `json:"to_gc,omitempty"`
	HousekeepTime  *cos.Duration `json:"hk_time,omitempty"`
	MinPctTotal    *int          `json:"min_pct_total,omitempty"`
	MinPctFree     *int          `json:"min_pct_free,omitempty"`
}

global configuration

type MirrorConf

type MirrorConf struct {
	Copies  int64 `json:"copies"`       // num copies
	Burst   int   `json:"burst_buffer"` // xaction channel (buffer) size
	Enabled bool  `json:"enabled"`      // enabled (to generate copies)
}

global configuration

func (*MirrorConf) String

func (c *MirrorConf) String() string

func (*MirrorConf) Validate

func (c *MirrorConf) Validate() error

func (*MirrorConf) ValidateAsProps

func (c *MirrorConf) ValidateAsProps(...any) error

type MirrorConfToSet added in v1.3.21

type MirrorConfToSet struct {
	Copies  *int64 `json:"copies,omitempty"`
	Burst   *int   `json:"burst_buffer,omitempty"`
	Enabled *bool  `json:"enabled,omitempty"`
}

global configuration

type NetConf

type NetConf struct {
	L4   L4Conf   `json:"l4"`
	HTTP HTTPConf `json:"http"`
}

global configuration

func (*NetConf) Validate

func (c *NetConf) Validate() (err error)

type NetConfToSet added in v1.3.21

type NetConfToSet struct {
	HTTP *HTTPConfToSet `json:"http,omitempty"`
}

global configuration

type Ns

type Ns struct {
	// UUID of other remote AIS cluster (for now only used for AIS). Note
	// that we can have different namespaces which refer to same UUID (cluster).
	// This means that in a sense UUID is a parent of the actual namespace.
	UUID string `json:"uuid" yaml:"uuid"`
	// Name uniquely identifies a namespace under the same UUID (which may
	// be empty) and is used in building FQN for the objects.
	Name string `json:"name" yaml:"name"`
}

Ns (or Namespace) adds additional layer for scoping the data under the same provider. It allows to have same dataset and bucket names under different namespaces what allows for easy data manipulation without affecting data in different namespaces.

func ParseNsUname

func ParseNsUname(s string) (n Ns)

Parses [@uuid][#namespace]. It does a little bit more than just parsing a string from `Uname` so that logic can be reused in different places.

func (Ns) IsAnyRemote

func (n Ns) IsAnyRemote() bool

func (Ns) IsGlobal

func (n Ns) IsGlobal() bool

func (Ns) IsRemote

func (n Ns) IsRemote() bool

func (Ns) Len added in v1.3.21

func (n Ns) Len() int

func (Ns) String

func (n Ns) String() string

func (Ns) Uname

func (n Ns) Uname() string

type OWT

type OWT int

Object Write Transaction (OWT) controls certain aspects of creating new objects in the cluster. In particular, OwtGet* group below simultaneously specifies cold-GET variations (that all involve reading from a remote backend) and the associated locking (that will always reflect a tradeoff between consistency and parallelism) NOTE: enum entries' order is important

const (
	// PUT and PUT-like transactions
	OwtPut       OWT = iota // PUT
	OwtPromote              // promote target-accessible files and directories
	OwtArchive              // multi-obj arch
	OwtTransform            // ETL
	OwtCopy                 // copy and move objects within cluster
	OwtRebalance            // NOTE: must be the last in PUT* group
	//
	// GET and friends
	//
	OwtGetTryLock      // if !try-lock(exclusive) { return error }; read from remote; ...
	OwtGetLock         // lock(exclusive); read from remote; ...
	OwtGet             // GET (with upgrading read-lock in the local-write path)
	OwtGetPrefetchLock // (used for maximum parallelism when prefetching)
	//
	// None of the above
	//
	OwtNone
)

func (*OWT) FromS

func (owt *OWT) FromS(s string)

func (OWT) String

func (owt OWT) String() (s string)

func (OWT) ToS

func (owt OWT) ToS() (s string)

type ObjAttrs

type ObjAttrs struct {
	Cksum    *cos.Cksum `json:"checksum,omitempty"`  // object checksum (cloned)
	CustomMD cos.StrKVs `json:"custom-md,omitempty"` // custom metadata: ETag, MD5, CRC, user-defined ...
	Ver      *string    `json:"version,omitempty"`   // object version
	Atime    int64      `json:"atime,omitempty"`     // access time (nanoseconds since UNIX epoch)
	Size     int64      `json:"size,omitempty"`      // object size (bytes)
}

see also apc.HdrObjAtime et al. @ api/apc/const.go (and note that naming must be consistent)

func (*ObjAttrs) AtimeUnix

func (oa *ObjAttrs) AtimeUnix() int64

func (*ObjAttrs) CheckEq added in v1.3.24

func (oa *ObjAttrs) CheckEq(rem cos.OAH) error

local <=> remote equality in the context of cold-GET and download. This function decides whether we need to go ahead and re-read the object from its remote location.

Other than a "binary" size and version checks, rest logic goes as follows: objects are considered equal if they have a) the same version and at least one matching checksum, or b) the same remote "source" and at least one matching checksum, or c) two matching checksums. (See also note below.)

Note that ETag, checksum, or version mismatch leads to immediate return with error specifying the exact cause.

Note version comparison may fail even when the objects are identical, content-wise: same size, ETag, and checksums may still "co-exist" with different versions.

TODO: count == 1 with matching checksum being xxhash - must be configurable :NOTE

func (*ObjAttrs) Checksum

func (oa *ObjAttrs) Checksum() *cos.Cksum

func (*ObjAttrs) CopyFrom

func (oa *ObjAttrs) CopyFrom(oah cos.OAH, skipCksum bool)

clone OAH => ObjAttrs (see also lom.CopyAttrs)

func (*ObjAttrs) CopyVersion added in v1.3.24

func (oa *ObjAttrs) CopyVersion(oah cos.OAH)

func (*ObjAttrs) DelStdCustom added in v1.3.26

func (oa *ObjAttrs) DelStdCustom()

func (*ObjAttrs) EqCksum added in v1.3.26

func (oa *ObjAttrs) EqCksum(cksum *cos.Cksum) bool

func (*ObjAttrs) FromHeader

func (oa *ObjAttrs) FromHeader(hdr http.Header) (cksum *cos.Cksum)

NOTE: returning checksum separately for subsequent validation

func (*ObjAttrs) GetCustomKey

func (oa *ObjAttrs) GetCustomKey(key string) (val string, exists bool)

func (*ObjAttrs) GetCustomMD

func (oa *ObjAttrs) GetCustomMD() cos.StrKVs

func (*ObjAttrs) Lsize added in v1.3.24

func (oa *ObjAttrs) Lsize(_ ...bool) int64

func (*ObjAttrs) SetCksum

func (oa *ObjAttrs) SetCksum(ty, val string)

func (*ObjAttrs) SetCustomKey

func (oa *ObjAttrs) SetCustomKey(k, v string)

func (*ObjAttrs) SetCustomMD

func (oa *ObjAttrs) SetCustomMD(md cos.StrKVs)

func (*ObjAttrs) SetSize added in v1.3.16

func (oa *ObjAttrs) SetSize(size int64)

func (*ObjAttrs) SetVersion added in v1.3.24

func (oa *ObjAttrs) SetVersion(ver string)

func (*ObjAttrs) String

func (oa *ObjAttrs) String() string

func (*ObjAttrs) Version

func (oa *ObjAttrs) Version(_ ...bool) string

func (*ObjAttrs) VersionPtr added in v1.3.24

func (oa *ObjAttrs) VersionPtr() *string

type ObjectProps

type ObjectProps struct {
	Bck Bck `json:"bucket"`
	ObjAttrs
	Name     string `json:"name"`
	Location string `json:"location"` // see also `GetPropsLocation`
	Mirror   struct {
		Paths  []string `json:"paths,omitempty"`
		Copies int      `json:"copies,omitempty"`
	} `json:"mirror"`
	EC struct {
		Generation   int64 `json:"generation"`
		DataSlices   int   `json:"data"`
		ParitySlices int   `json:"parity"`
		IsECCopy     bool  `json:"replicated"`
	} `json:"ec"`
	Present bool `json:"present"`
}

object properties NOTE: embeds system `ObjAttrs` that in turn includes custom user-defined NOTE: compare with `apc.LsoMsg`

type ParseURIOpts

type ParseURIOpts struct {
	DefaultProvider string // If set the provider will be used as provider.
	IsQuery         bool   // Determines if the URI should be parsed as query.
}

type PeriodConf

type PeriodConf struct {
	StatsTime     cos.Duration `json:"stats_time"`      // collect and publish stats; other house-keeping
	RetrySyncTime cos.Duration `json:"retry_sync_time"` // metasync retry
	NotifTime     cos.Duration `json:"notif_time"`      // (IC notifications)
}

NOTE: StatsTime is one important timer - a pulse

func (*PeriodConf) Validate

func (c *PeriodConf) Validate() error

type PeriodConfToSet added in v1.3.21

type PeriodConfToSet struct {
	StatsTime     *cos.Duration `json:"stats_time,omitempty"`
	RetrySyncTime *cos.Duration `json:"retry_sync_time,omitempty"`
	NotifTime     *cos.Duration `json:"notif_time,omitempty"`
}

global configuration

type PropsValidator

type PropsValidator interface {
	ValidateAsProps(arg ...any) error
}

type ProxyConf

type ProxyConf struct {
	PrimaryURL   string `json:"primary_url"`
	OriginalURL  string `json:"original_url"`
	DiscoveryURL string `json:"discovery_url"`
	NonElectable bool   `json:"non_electable"` // NOTE: deprecated, not used
}

global configuration

type ProxyConfToSet added in v1.3.21

type ProxyConfToSet struct {
	PrimaryURL   *string `json:"primary_url,omitempty"`
	OriginalURL  *string `json:"original_url,omitempty"`
	DiscoveryURL *string `json:"discovery_url,omitempty"`
}

global configuration

type QueryBcks

type QueryBcks Bck

func (*QueryBcks) AddToQuery

func (qbck *QueryBcks) AddToQuery(query url.Values)

func (QueryBcks) Contains

func (qbck QueryBcks) Contains(other *Bck) bool

NOTE: a named bucket with no provider is assumed to be ais://

func (*QueryBcks) Equal

func (qbck *QueryBcks) Equal(bck *Bck) bool

func (*QueryBcks) IsAIS

func (qbck *QueryBcks) IsAIS() bool

func (*QueryBcks) IsBucket

func (qbck *QueryBcks) IsBucket() bool

QueryBcks is a Bck that _can_ have an empty Name. (TODO: extend to support prefix and regex.)

func (*QueryBcks) IsCloud

func (qbck *QueryBcks) IsCloud() bool

func (*QueryBcks) IsEmpty

func (qbck *QueryBcks) IsEmpty() bool

func (*QueryBcks) IsHT added in v1.3.24

func (qbck *QueryBcks) IsHT() bool

func (*QueryBcks) IsRemoteAIS

func (qbck *QueryBcks) IsRemoteAIS() bool

func (*QueryBcks) SetQuery added in v1.3.26

func (qbck *QueryBcks) SetQuery(q url.Values)

func (QueryBcks) String

func (qbck QueryBcks) String() string

func (*QueryBcks) Validate

func (qbck *QueryBcks) Validate() (err error)

type RateLimitBase added in v1.3.28

type RateLimitBase struct {
	// optional per-operation MaxTokens override - a space-separated key:value list, e.g.:
	// - "put:3500"
	// - "get:5000 delete:1000"
	// optional; case-insensitive
	Verbs string `json:"per_op_max_tokens,omitempty"`
	//
	// mandatory parameters; for default values and min/max ranges, see RateLimitConf.Validate
	//
	Interval  cos.Duration `json:"interval"`
	MaxTokens int          `json:"max_tokens"`
	Enabled   bool         `json:"enabled"`
}

global config that can be used to manage: * adaptive rate limit vis-à-vis Cloud backend * rate limiting (bursty) user workloads on the front

type RateLimitBaseToSet added in v1.3.28

type RateLimitBaseToSet struct {
	Verbs     *string       `json:"per_op_max_tokens,omitempty"`
	Interval  *cos.Duration `json:"interval,omitempty"`
	MaxTokens *int          `json:"max_tokens,omitempty"`
	Enabled   *bool         `json:"enabled,omitempty"`
}

global config that can be used to manage: * adaptive rate limit vis-à-vis Cloud backend * rate limiting (bursty) user workloads on the front

type RateLimitConf added in v1.3.28

type RateLimitConf struct {
	Backend  Adaptive `json:"backend"`
	Frontend Bursty   `json:"frontend"`
}

global config that can be used to manage: * adaptive rate limit vis-à-vis Cloud backend * rate limiting (bursty) user workloads on the front

func (*RateLimitConf) Validate added in v1.3.28

func (c *RateLimitConf) Validate() error

func (*RateLimitConf) ValidateAsProps added in v1.3.28

func (c *RateLimitConf) ValidateAsProps(...any) error

type RateLimitConfToSet added in v1.3.28

type RateLimitConfToSet struct {
	Backend  *AdaptiveToSet `json:"backend,omitempty"`
	Frontend *BurstyToSet   `json:"frontend,omitempty"`
}

global config that can be used to manage: * adaptive rate limit vis-à-vis Cloud backend * rate limiting (bursty) user workloads on the front

type RebalanceConf

type RebalanceConf struct {
	XactConf
	DestRetryTime cos.Duration `json:"dest_retry_time"` // max wait for ACKs & neighbors to complete
	Enabled       bool         `json:"enabled"`         // true=auto-rebalance | manual rebalancing
}

global configuration

func (*RebalanceConf) String

func (c *RebalanceConf) String() string

func (*RebalanceConf) Validate

func (c *RebalanceConf) Validate() error

type RebalanceConfToSet added in v1.3.21

type RebalanceConfToSet struct {
	XactConfToSet
	DestRetryTime *cos.Duration `json:"dest_retry_time,omitempty"`
	Enabled       *bool         `json:"enabled,omitempty"`
}

global configuration

type ResilverConf

type ResilverConf struct {
	Enabled bool `json:"enabled"` // true=auto-resilver | manual resilvering
}

global configuration

func (*ResilverConf) String

func (c *ResilverConf) String() string

func (*ResilverConf) Validate

func (*ResilverConf) Validate() error

type ResilverConfToSet added in v1.3.21

type ResilverConfToSet struct {
	Enabled *bool `json:"enabled,omitempty"`
}

global configuration

type RetryArgs

type RetryArgs struct {
	Call    func() (int, error)
	IsFatal func(error) bool

	Action string
	Caller string

	SoftErr int // How many retries on ConnectionRefused or ConnectionReset error.
	HardErr int // How many retries on any other error.
	Sleep   time.Duration

	Verbosity int  // Verbosity level for logging.
	BackOff   bool // If true, retries will wait progressively longer between attempts (backoff).
	IsClient  bool // True when client side (with no access to cluster config timeouts)
}

func (*RetryArgs) Do added in v1.3.30

func (args *RetryArgs) Do() (ecode int, err error)

type SpaceConf

type SpaceConf struct {
	// Storage Cleanup watermark: used capacity (%) that triggers cleanup
	// (deleted objects and buckets, extra copies, etc.)
	CleanupWM int64 `json:"cleanupwm"`

	// LowWM: used capacity low-watermark (% of total local storage capacity)
	LowWM int64 `json:"lowwm"`

	// HighWM: used capacity high-watermark (% of total local storage capacity)
	// - LRU starts evicting objects when the currently used capacity (used-cap) gets above HighWM
	// - and keeps evicting objects until the used-cap gets below LowWM
	// - while self-throttling itself in accordance with target utilization
	HighWM int64 `json:"highwm"`

	// Out-of-Space: if exceeded, the target starts failing new PUTs and keeps
	// failing them until its local used-cap gets back below HighWM (see above)
	OOS int64 `json:"out_of_space"`
}

global configuration

func (*SpaceConf) String

func (c *SpaceConf) String() string

func (*SpaceConf) Validate

func (c *SpaceConf) Validate() (err error)

func (*SpaceConf) ValidateAsProps

func (c *SpaceConf) ValidateAsProps(...any) error

type SpaceConfToSet added in v1.3.21

type SpaceConfToSet struct {
	CleanupWM *int64 `json:"cleanupwm,omitempty"`
	LowWM     *int64 `json:"lowwm,omitempty"`
	HighWM    *int64 `json:"highwm,omitempty"`
	OOS       *int64 `json:"out_of_space,omitempty"`
}

global configuration

type TCBConf

type TCBConf struct{ XactConf }

bucket-to-bucket copy/transform

type TCBConfToSet added in v1.3.21

type TCBConfToSet struct{ XactConfToSet }

global configuration

type TCOConf added in v1.3.28

type TCOConf struct{ XactConf }

multi-object copy/transform

type TCOConfToSet added in v1.3.28

type TCOConfToSet struct{ XactConfToSet }

global configuration

type TCOMsg added in v1.3.24

type TCOMsg struct {
	ToBck Bck `json:"tobck"`
	apc.TCOMsg
}

Multi-object copy & transform (see also: TCBMsg)

type TLSArgs added in v1.3.21

type TLSArgs struct {
	ClientCA    string
	Certificate string
	Key         string
	SkipVerify  bool
}

type TestFSPConf

type TestFSPConf struct {
	Root     string `json:"root"`
	Count    int    `json:"count"`
	Instance int    `json:"instance"`
}

local config

func (*TestFSPConf) Validate

func (c *TestFSPConf) Validate(contextConfig *Config) (err error)

validate root and (NOTE: testing only) generate and fill-in counted FSP.Paths

func (*TestFSPConf) ValidateMpath

func (c *TestFSPConf) ValidateMpath(p string) (err error)

type TimeoutConf

type TimeoutConf struct {
	CplaneOperation cos.Duration `json:"cplane_operation"`  // read-mostly via global cmn.Rom.CplaneOperation
	MaxKeepalive    cos.Duration `json:"max_keepalive"`     // ditto, cmn.Rom.MaxKeepalive - see below
	MaxHostBusy     cos.Duration `json:"max_host_busy"`     // 2-phase transactions and more
	Startup         cos.Duration `json:"startup_time"`      // primary wait for joins at (primary's) startup; indirectly, cluster startup
	JoinAtStartup   cos.Duration `json:"join_startup_time"` // (join cluster at startup) timeout; (2 * Startup) when zero
	SendFile        cos.Duration `json:"send_file_time"`    // large file or blob and/or slow network
	// intra-cluster EC streams; default SharedStreamsDflt; never timeout when negative
	EcStreams cos.Duration `json:"ec_streams_time,omitempty"`
	// object metadata timeout; for training apps an approx. duration of 2 (two) epochs
	ObjectMD cos.Duration `json:"object_md,omitempty"`
	// prior to returning 409 conflict (ie., cmn.ErrBusy)
	ColdGetConflict cos.Duration `json:"cold_get_conflict,omitempty"`
}

maximum intra-cluster latencies (in the increasing order)

func (*TimeoutConf) Validate

func (c *TimeoutConf) Validate() error

type TimeoutConfToSet added in v1.3.21

type TimeoutConfToSet struct {
	CplaneOperation *cos.Duration `json:"cplane_operation,omitempty"`
	MaxKeepalive    *cos.Duration `json:"max_keepalive,omitempty"`
	MaxHostBusy     *cos.Duration `json:"max_host_busy,omitempty"`
	Startup         *cos.Duration `json:"startup_time,omitempty"`
	JoinAtStartup   *cos.Duration `json:"join_startup_time,omitempty"`
	SendFile        *cos.Duration `json:"send_file_time,omitempty"`
	EcStreams       *cos.Duration `json:"ec_streams_time,omitempty"`
	ObjectMD        *cos.Duration `json:"object_md,omitempty"`
	ColdGetConflict *cos.Duration `json:"cold_get_conflict,omitempty"`
}

global configuration

type TraceExporterAuthConf added in v1.3.26

type TraceExporterAuthConf struct {
	TokenHeader string `json:"token_header"` // header used to pass exporter auth token
	TokenFile   string `json:"token_file"`   // filepath from where auth token can be obtained
}

global configuration

func (TraceExporterAuthConf) IsEnabled added in v1.3.26

func (tac TraceExporterAuthConf) IsEnabled() bool

only with `oteltracing` build tag

type TraceExporterAuthConfToSet added in v1.3.26

type TraceExporterAuthConfToSet struct {
	TokenHeader *string `json:"token_header,omitempty"` // header used to pass exporter auth token
	TokenFile   *string `json:"token_file,omitempty"`   // filepath from where auth token can be obtained
}

global configuration

type TracingConf added in v1.3.26

type TracingConf struct {
	ExporterEndpoint  string                `json:"exporter_endpoint"`       // gRPC exporter endpoint
	ExporterAuth      TraceExporterAuthConf `json:"exporter_auth,omitempty"` // exporter auth config
	ServiceNamePrefix string                `json:"service_name_prefix"`     // service name prefix used by trace exporter
	ExtraAttributes   map[string]string     `json:"attributes,omitempty"`    // any extra-attributes to be added to traces

	// SamplerProbabilityStr is the percentage of traces to be sampled, expressed as a float64.
	// It's stored as a string to avoid potential floating-point precision issues during json unmarshal.
	// Valid values range from 0.0 to 1.0, where 1.0 means 100% sampling.
	SamplerProbabilityStr string `json:"sampler_probability,omitempty"`
	Enabled               bool   `json:"enabled"`
	SkipVerify            bool   `json:"skip_verify"` // allow insecure exporter gRPC connection

	SamplerProbability float64 `json:"-"`
}

TracingConf defines the configuration used for the OpenTelemetry (OTEL) trace exporter. It includes settings for enabling tracing, sampling ratio, exporter endpoint, and other parameters necessary for distributed tracing in AIStore.

func (*TracingConf) Validate added in v1.3.26

func (c *TracingConf) Validate() error

type TracingConfToSet added in v1.3.26

type TracingConfToSet struct {
	ExporterEndpoint      *string                     `json:"exporter_endpoint,omitempty"`   // gRPC exporter endpoint
	ExporterAuth          *TraceExporterAuthConfToSet `json:"exporter_auth,omitempty"`       // exporter auth config
	ServiceNamePrefix     *string                     `json:"service_name_prefix,omitempty"` // service name used by trace exporter
	ExtraAttributes       map[string]string           `json:"attributes,omitempty"`          // any extra-attributes to be added to traces
	SamplerProbabilityStr *string                     `json:"sampler_probability,omitempty"` // percentage of traces to be sampled
	Enabled               *bool                       `json:"enabled,omitempty"`
	SkipVerify            *bool                       `json:"skip_verify,omitempty"` // allow insecure exporter gRPC connection
}

NOTE: Updating TracingConfig requires restart.

type TransportArgs

type TransportArgs struct {
	DialTimeout      time.Duration
	Timeout          time.Duration
	IdleConnTimeout  time.Duration
	IdleConnsPerHost int
	MaxIdleConns     int
	SndRcvBufSize    int
	WriteBufferSize  int
	ReadBufferSize   int
	UseHTTPProxyEnv  bool
	LowLatencyToS    bool
}

assorted http(s) client options

func (*TransportArgs) ServerControl added in v1.3.26

func (args *TransportArgs) ServerControl(c syscall.RawConn)

type TransportConf

type TransportConf struct {
	MaxHeaderSize int `json:"max_header"`   // max transport header buffer (default=4K)
	Burst         int `json:"burst_buffer"` // num sends with no back pressure; see also AIS_STREAM_BURST_NUM
	// two no-new-transmissions durations:
	// * IdleTeardown: sender terminates the connection (to reestablish it upon the very first/next PDU)
	// * QuiesceTime:  safe to terminate or transition to the next (in re: rebalance) stage
	IdleTeardown cos.Duration `json:"idle_teardown"`
	QuiesceTime  cos.Duration `json:"quiescent"`
	// lz4
	// max uncompressed block size, one of [64K, 256K(*), 1M, 4M]
	// fastcompression.blogspot.com/2013/04/lz4-streaming-format-final.html
	LZ4BlockMaxSize  cos.SizeIEC `json:"lz4_block"`
	LZ4FrameChecksum bool        `json:"lz4_frame_checksum"`
}

global configuration

func (*TransportConf) Validate

func (c *TransportConf) Validate() (err error)

NOTE: uncompressed block sizes - the enum currently supported by the github.com/pierrec/lz4

type TransportConfToSet added in v1.3.21

type TransportConfToSet struct {
	MaxHeaderSize    *int          `json:"max_header,omitempty"`
	Burst            *int          `json:"burst_buffer,omitempty"`
	IdleTeardown     *cos.Duration `json:"idle_teardown,omitempty"`
	QuiesceTime      *cos.Duration `json:"quiescent,omitempty"`
	LZ4BlockMaxSize  *cos.SizeIEC  `json:"lz4_block,omitempty"`
	LZ4FrameChecksum *bool         `json:"lz4_frame_checksum,omitempty"`
}

global configuration

type Validator

type Validator interface {
	Validate() error
}

type VersionConf

type VersionConf struct {
	// Determines if versioning is enabled
	Enabled bool `json:"enabled"`

	// Validate remote version and, possibly, update in-cluster ("cached") copy.
	// Scenarios include (but are not limited to):
	// - warm GET
	// - prefetch bucket (*)
	// - prefetch multiple objects (see api/multiobj.go)
	// - copy bucket
	// Applies to Cloud and remote AIS buckets - generally, buckets that have remote backends
	// that in turn provide some form of object versioning.
	// (*) Xactions (ie., jobs) that read-access multiple objects (e.g., prefetch, copy-bucket)
	// may support operation-scope option to synchronize remote content (to aistore) - the option
	// not requiring changing bucket configuration.
	// See also:
	// - apc.QparamLatestVer, apc.PrefetchMsg, apc.CopyBckMsg
	ValidateWarmGet bool `json:"validate_warm_get"`

	// A stronger variant of the above that in addition entails:
	// - deleting in-cluster object if its remote ("cached") counterpart does not exist
	// See also: apc.QparamSync, apc.CopyBckMsg
	Sync bool `json:"synchronize"`
}

global configuration

func (*VersionConf) String

func (c *VersionConf) String() string

func (*VersionConf) Validate

func (c *VersionConf) Validate() error

type VersionConfToSet added in v1.3.21

type VersionConfToSet struct {
	Enabled         *bool `json:"enabled,omitempty"`
	ValidateWarmGet *bool `json:"validate_warm_get,omitempty"`
	Sync            *bool `json:"synchronize,omitempty"`
}

global configuration

type WritePolicyConf

type WritePolicyConf struct {
	Data apc.WritePolicy `json:"data"`
	MD   apc.WritePolicy `json:"md"`
}

global configuration

func (*WritePolicyConf) Validate

func (c *WritePolicyConf) Validate() (err error)

func (*WritePolicyConf) ValidateAsProps

func (c *WritePolicyConf) ValidateAsProps(...any) error

type WritePolicyConfToSet added in v1.3.21

type WritePolicyConfToSet struct {
	Data *apc.WritePolicy `json:"data,omitempty" list:"readonly"` // NOTE: NIY
	MD   *apc.WritePolicy `json:"md,omitempty"`
}

global configuration

type XactConf added in v1.3.28

type XactConf struct {
	Compression string `json:"compression"`       // enum { CompressAlways, ... } in api/apc/compression.go
	SbundleMult int    `json:"bundle_multiplier"` // stream-bundle multiplier: num streams to destination
	Burst       int    `json:"burst_buffer"`      // xaction channel (buffer) size
}

generic xaction --

func (*XactConf) Validate added in v1.3.28

func (c *XactConf) Validate() error

type XactConfToSet added in v1.3.28

type XactConfToSet struct {
	Compression *string `json:"compression,omitempty"`
	SbundleMult *int    `json:"bundle_multiplier,omitempty"`
	Burst       *int    `json:"burst_buffer,omitempty"`
}

global configuration

Directories

Path Synopsis
Package archive: write, read, copy, append, list primitives across all supported formats
Package archive: write, read, copy, append, list primitives across all supported formats
Package atomic provides simple wrappers around numerics to enforce atomic access.
Package atomic provides simple wrappers around numerics to enforce atomic access.
Package certloader loads and reloads X.509 certs.
Package certloader loads and reloads X.509 certs.
Package cos provides common low-level types and utilities for all aistore projects.
Package cos provides common low-level types and utilities for all aistore projects.
Package memsys provides memory management and slab/SGL allocation with io.Reader and io.Writer interfaces on top of scatter-gather lists of reusable buffers.
Package memsys provides memory management and slab/SGL allocation with io.Reader and io.Writer interfaces on top of scatter-gather lists of reusable buffers.
Package feat: global runtime-configurable feature flags
Package feat: global runtime-configurable feature flags
Package fname contains filename constants and common system directories
Package fname contains filename constants and common system directories
Package jsp (JSON persistence) provides utilities to store and load arbitrary JSON-encoded structures with optional checksumming and compression.
Package jsp (JSON persistence) provides utilities to store and load arbitrary JSON-encoded structures with optional checksumming and compression.
Package k8s: initialization, client, and misc.
Package k8s: initialization, client, and misc.
Package kvdb provides a local key/value database server for AIS.
Package kvdb provides a local key/value database server for AIS.
Package mono provides low-level monotonic time
Package mono provides low-level monotonic time
Package nlog - aistore logger, provides buffering, timestamping, writing, and flushing/syncing/rotating
Package nlog - aistore logger, provides buffering, timestamping, writing, and flushing/syncing/rotating
Package oom: serialized goroutine to go ahead and run GC _now_
Package oom: serialized goroutine to go ahead and run GC _now_
Package prob implements fully features dynamic probabilistic filter.
Package prob implements fully features dynamic probabilistic filter.
Package xoshiro256 implements the xoshiro256** RNG no-copyright
Package xoshiro256 implements the xoshiro256** RNG no-copyright

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL