Documentation
¶
Overview ¶
Package cmn provides common constants, types, and utilities for AIS clients and AIStore.
- Copyright (c) 2018-2025, NVIDIA CORPORATION. All rights reserved.
Package cmn provides common constants, types, and utilities for AIS clients and AIStore.
- Copyright (c) 2018-2025, NVIDIA CORPORATION. All rights reserved.
Package cmn provides common constants, types, and utilities for AIS clients and AIStore.
- Copyright (c) 2018-2025, NVIDIA CORPORATION. All rights reserved.
Package cmn provides common constants, types, and utilities for AIS clients and AIStore.
- Copyright (c) 2018-2025, NVIDIA CORPORATION. All rights reserved.
Package cmn provides common constants, types, and utilities for AIS clients and AIStore.
- Copyright (c) 2018-2025, NVIDIA CORPORATION. All rights reserved.
Package cmn provides common constants, types, and utilities for AIS clients and AIStore.
- Copyright (c) 2018-2025, NVIDIA CORPORATION. All rights reserved.
Package cmn provides common constants, types, and utilities for AIS clients and AIStore.
- Copyright (c) 2018-2025, NVIDIA CORPORATION. All rights reserved.
Package cmn provides common constants, types, and utilities for AIS clients and AIStore.
- Copyright (c) 2018-2025, NVIDIA CORPORATION. All rights reserved.
Package cmn provides common constants, types, and utilities for AIS clients and AIStore.
- Copyright (c) 2018-2025, NVIDIA CORPORATION. All rights reserved.
Package cmn provides common constants, types, and utilities for AIS clients and AIStore.
- Copyright (c) 2018-2025, NVIDIA CORPORATION. All rights reserved.
Package cmn provides common constants, types, and utilities for AIS clients and AIStore.
- Copyright (c) 2018-2025, NVIDIA CORPORATION. All rights reserved.
Package cmn provides common constants, types, and utilities for AIS clients and AIStore.
- Copyright (c) 2018-2024, NVIDIA CORPORATION. All rights reserved.
Package cmn provides common constants, types, and utilities for AIS clients and AIStore.
- Copyright (c) 2018-2024, NVIDIA CORPORATION. All rights reserved.
Package cmn provides common constants, types, and utilities for AIS clients and AIStore.
- Copyright (c) 2018-2025, NVIDIA CORPORATION. All rights reserved.
Package cmn provides common constants, types, and utilities for AIS clients and AIStore.
- Copyright (c) 2018-2024, NVIDIA CORPORATION. All rights reserved.
msgp -file cmn/objlist.go -tests=false -marshal=false -unexported Code generated by the command above where msgp is tinylib/msgp; see docs/msgp.md. DO NOT EDIT.
Package cmn provides common constants, types, and utilities for AIS clients and AIStore.
- Copyright (c) 2018-2025, NVIDIA CORPORATION. All rights reserved.
Package cmn provides common constants, types, and utilities for AIS clients and AIStore.
- Copyright (c) 2018-2024, NVIDIA CORPORATION. All rights reserved.
Package cmn provides common constants, types, and utilities for AIS clients and AIStore.
- Copyright (c) 2018-2025, NVIDIA CORPORATION. All rights reserved.
Package cmn provides common constants, types, and utilities for AIS clients and AIStore.
- Copyright (c) 2023-2025, NVIDIA CORPORATION. All rights reserved.
Package cmn provides common constants, types, and utilities for AIS clients and AIStore.
- Copyright (c) 2018-2025, NVIDIA CORPORATION. All rights reserved.
Package cmn provides common constants, types, and utilities for AIS clients and AIStore.
- Copyright (c) 2022-2025, NVIDIA CORPORATION. All rights reserved.
Index ¶
- Constants
- Variables
- func CheckDirNoRecurs(prefix, relPath string) (addDirEntry bool, _ error)
- func CopyHeaders(dst, src http.Header)
- func CopyProps(src, dst any, asType string) error
- func CustomMD2S(md cos.StrKVs) string
- func CustomProps2S(nvs ...string) string
- func DelBckFromQuery(query url.Values) url.Values
- func DirHasOrIsPrefix(dirPath, prefix string) bool
- func EnvToTLS(sargs *TLSArgs)
- func FreeBuffer(buf *bytes.Buffer)
- func FreeHra(a *HreqArgs)
- func FreeHterr(a *ErrHTTP)
- func HasSpecialSymbols(path string) bool
- func Host2IP(host string) (net.IP, error)
- func HreqFree(r *http.Request)
- func Init(a string, b func(error))
- func InitObjProps2Hdr()
- func IsErrAborted(err error) bool
- func IsErrBckNotFound(err error) bool
- func IsErrBucketAlreadyExists(err error) bool
- func IsErrBucketLevel(err error) bool
- func IsErrBucketNought(err error) bool
- func IsErrBusy(err error) bool
- func IsErrCapExceeded(err error) bool
- func IsErrEmptyProvider(err error) bool
- func IsErrFailedTo(err error) bool
- func IsErrGetCap(err error) bool
- func IsErrInitMissingBackend(err error) bool
- func IsErrLmetaCorrupted(err error) bool
- func IsErrLmetaNotFound(err error) bool
- func IsErrMpathCheck(err error) bool
- func IsErrMpathNewDisk(err error) bool
- func IsErrMpathNotFound(err error) bool
- func IsErrObjLevel(err error) bool
- func IsErrObjNought(err error) bool
- func IsErrRangeNotSatisfiable(err error) bool
- func IsErrRemoteBckNotFound(err error) bool
- func IsErrStreamTerminated(err error) bool
- func IsErrTooManyRequests(err error) bool
- func IsErrWarning(err error) bool
- func IsErrXactNotFound(err error) bool
- func IsErrXactUsePrev(err error) bool
- func IsFileAlreadyClosed(err error) bool
- func IsNestedMpath(a string, la int, b string) (err error)
- func IsStatusBadGateway(err error) (yes bool)
- func IsStatusGone(err error) (yes bool)
- func IsStatusNotFound(err error) (yes bool)
- func IsStatusServiceUnavailable(err error) (yes bool)
- func IterFields(v any, updf updateFunc, opts ...IterOpts) error
- func KeepaliveRetryDuration(c *Config) time.Duration
- func LoadConfig(globalConfPath, localConfPath, daeRole string, config *Config) error
- func MakeRangeHdr(start, length int64) string
- func NetworkIsKnown(net string) bool
- func NewBuffer() (buf *bytes.Buffer)
- func NewClient(cargs TransportArgs) *http.Client
- func NewClientTLS(cargs TransportArgs, sargs TLSArgs, intra bool) *http.Client
- func NewDefaultClients(timeout time.Duration) (clientH, clientTLS *http.Client)
- func NewErrRemoteMetadataMismatch(cause error) error
- func NewIntraClientTLS(cargs TransportArgs, config *Config) *http.Client
- func NewTLS(sargs TLSArgs, intra bool) (tlsConf *tls.Config, err error)
- func NewTransport(cargs TransportArgs) *http.Transport
- func NormalizeProvider(provider string) (p string, err error)
- func ObjHasPrefix(objName, prefix string) bool
- func OrigURLBck2Name(origURLBck string) (bckName string)
- func ParseHost2IP(host string) (net.IP, error)
- func ParsePort(p string) (int, error)
- func ParseRangeHdr(contentRange string) (start, length, objectSize int64, err error)
- func ParseReadHeaderTimeout() (_ time.Duration, isSet bool)
- func ParseURL(path string, itemsPresent []string, itemsAfter int, splitAfter bool) ([]string, error)
- func ParseURLScheme(url string) (scheme, address string)
- func PrependProtocol(url string, protocol ...string) string
- func PropToHeader(prop string) string
- func ReadBytes(r *http.Request) (b []byte, err error)
- func ReadJSON(w http.ResponseWriter, r *http.Request, out any) (err error)
- func S2CustomMD(md cos.StrKVs, custom, version string)
- func S2CustomVal(custom, name string) (v string)
- func SaveOverrideConfig(configDir string, toUpdate *ConfigToSet) error
- func SortLso(entries LsoEntries)
- func ToHeader(oah cos.OAH, hdr http.Header, size int64, cksums ...*cos.Cksum)
- func TokenGreaterEQ(token, objName string) bool
- func TypeCodeHTTPErr(s string) (tcode string)
- func UnquoteCEV(val string) string
- func UpdateFieldValue(s any, name string, value any) error
- func ValidateMpath(mpath string) (string, error)
- func ValidatePort(port int) (int, error)
- func ValidateRemAlias(alias string) (err error)
- func WriteErr(w http.ResponseWriter, r *http.Request, err error, opts ...int)
- func WriteErr405(w http.ResponseWriter, r *http.Request, methods ...string)
- func WriteErrJSON(w http.ResponseWriter, r *http.Request, out any, err error) error
- func WriteErrMsg(w http.ResponseWriter, r *http.Request, msg string, opts ...int)
- type Adaptive
- type AdaptiveToSet
- type AllBsummResults
- type ArchConf
- type ArchConfToSet
- type ArchiveBckMsg
- type AuthConf
- type AuthConfToSet
- type BackendBckToSet
- type BackendConf
- func (c *BackendConf) EqualRemAIS(o *BackendConf, sname string) bool
- func (c *BackendConf) Get(provider string) (conf any)
- func (c *BackendConf) MarshalJSON() (data []byte, err error)
- func (c *BackendConf) Set(provider string, newConf any)
- func (c *BackendConf) UnmarshalJSON(data []byte) error
- func (c *BackendConf) Validate() (err error)
- type BackendConfAIS
- type Bck
- func (b *Bck) AddToQuery(query url.Values) url.Values
- func (b *Bck) AddUnameToQuery(query url.Values, uparam, objName string) url.Values
- func (b *Bck) Backend() *Bck
- func (b *Bck) Cname(objname string) (s string)
- func (b *Bck) Copy(src *Bck)
- func (bck *Bck) DefaultProps(c *ClusterConfig) *Bprops
- func (b *Bck) Equal(other *Bck) bool
- func (b *Bck) HasProvider() bool
- func (b *Bck) HasVersioningMD() bool
- func (b *Bck) HashUname(s string) uint64
- func (b *Bck) IsAIS() bool
- func (b *Bck) IsBuiltTagged() bool
- func (b *Bck) IsCloud() bool
- func (b *Bck) IsEmpty() bool
- func (b *Bck) IsHT() bool
- func (b *Bck) IsQuery() bool
- func (b *Bck) IsRemote() bool
- func (b *Bck) IsRemoteAIS() bool
- func (b *Bck) LenUnameGlob(objName string) int
- func (b *Bck) Less(other *Bck) bool
- func (b *Bck) MakeUname(objName string) []byte
- func (b *Bck) RemoteBck() *Bck
- func (b *Bck) SetQuery(q url.Values)
- func (b *Bck) Str(sb *strings.Builder)
- func (b *Bck) String() string
- func (b *Bck) Validate() (err error)
- func (b *Bck) ValidateName() error
- type Bcks
- type Bprops
- type BpropsToSet
- type BsummResult
- type Bursty
- type BurstyToSet
- type CksumConf
- type CksumConfToSet
- type ClientConf
- type ClientConfToSet
- type ClusterConfig
- type Config
- type ConfigToSet
- type DiskConf
- type DiskConfToSet
- type DownloaderConf
- type DownloaderConfToSet
- type DsortConf
- type DsortConfToSet
- type ECConf
- type ECConfToSet
- type ETLErrCtx
- type ErrAborted
- type ErrBckNotFound
- type ErrBucketAccessDenied
- type ErrBucketAlreadyExists
- type ErrBusy
- type ErrCapExceeded
- type ErrCreateHreq
- type ErrETL
- type ErrEmptyProvider
- type ErrFailedTo
- type ErrGetCap
- type ErrHTTP
- type ErrInitBackend
- type ErrInvalidBackendProvider
- type ErrInvalidCksum
- type ErrInvalidFSPathsConf
- type ErrInvalidMountpath
- type ErrLimitedCoexistence
- type ErrLmetaCorrupted
- type ErrLmetaNotFound
- type ErrMembershipChanges
- type ErrMissingBackend
- type ErrMpathCheck
- type ErrMpathLostDisk
- type ErrMpathNewDisk
- type ErrMpathNoDisks
- type ErrMpathNotFound
- type ErrNoNodes
- type ErrNotImpl
- type ErrNotRemoteBck
- type ErrObjDefunct
- type ErrObjectAccessDenied
- type ErrRangeNotSatisfiable
- type ErrRateLimitFrontend
- type ErrRemoteBckNotFound
- type ErrRemoteBucketOffline
- type ErrRemoteMetadataMismatch
- type ErrStreamTerminated
- type ErrTooManyRequests
- type ErrUnsupp
- type ErrWarning
- type ErrXactNotFound
- type ErrXactTgtInMaint
- type ErrXactUsePrev
- type ExtraProps
- type ExtraPropsAWS
- type ExtraPropsAWSToSet
- type ExtraPropsHDFS
- type ExtraPropsHDFSToSet
- type ExtraPropsHTTP
- type ExtraPropsHTTPToSet
- type ExtraToSet
- type FSHCConf
- type FSHCConfToSet
- type FSPConf
- type HTTPBckObj
- type HTTPConf
- type HTTPConfToSet
- type HreqArgs
- type IterField
- type IterOpts
- type KeepaliveConf
- type KeepaliveConfToSet
- type KeepaliveTrackerConf
- type KeepaliveTrackerConfToSet
- type L4Conf
- type LRUConf
- type LRUConfToSet
- type LocalConfig
- type LocalNetConfig
- type LogConf
- type LogConfToSet
- type LsoEnt
- func (be *LsoEnt) CopyWithProps(propsSet cos.StrSet) (ne *LsoEnt)
- func (z *LsoEnt) DecodeMsg(dc *msgp.Reader) (err error)
- func (z *LsoEnt) EncodeMsg(en *msgp.Writer) (err error)
- func (be *LsoEnt) IsAnyFlagSet(fl uint16) bool
- func (be *LsoEnt) IsPresent() bool
- func (be *LsoEnt) IsStatusOK() bool
- func (z *LsoEnt) Msgsize() (s int)
- func (be *LsoEnt) SetFlag(fl uint16)
- func (be *LsoEnt) Status() uint16
- type LsoEntries
- type LsoRes
- type MemsysConf
- type MemsysConfToSet
- type MirrorConf
- type MirrorConfToSet
- type NetConf
- type NetConfToSet
- type Ns
- type OWT
- type ObjAttrs
- func (oa *ObjAttrs) AtimeUnix() int64
- func (oa *ObjAttrs) CheckEq(rem cos.OAH) error
- func (oa *ObjAttrs) Checksum() *cos.Cksum
- func (oa *ObjAttrs) CopyFrom(oah cos.OAH, skipCksum bool)
- func (oa *ObjAttrs) CopyVersion(oah cos.OAH)
- func (oa *ObjAttrs) DelStdCustom()
- func (oa *ObjAttrs) EqCksum(cksum *cos.Cksum) bool
- func (oa *ObjAttrs) FromHeader(hdr http.Header) (cksum *cos.Cksum)
- func (oa *ObjAttrs) GetCustomKey(key string) (val string, exists bool)
- func (oa *ObjAttrs) GetCustomMD() cos.StrKVs
- func (oa *ObjAttrs) Lsize(_ ...bool) int64
- func (oa *ObjAttrs) SetCksum(ty, val string)
- func (oa *ObjAttrs) SetCustomKey(k, v string)
- func (oa *ObjAttrs) SetCustomMD(md cos.StrKVs)
- func (oa *ObjAttrs) SetSize(size int64)
- func (oa *ObjAttrs) SetVersion(ver string)
- func (oa *ObjAttrs) String() string
- func (oa *ObjAttrs) Version(_ ...bool) string
- func (oa *ObjAttrs) VersionPtr() *string
- type ObjectProps
- type ParseURIOpts
- type PeriodConf
- type PeriodConfToSet
- type PropsValidator
- type ProxyConf
- type ProxyConfToSet
- type QueryBcks
- func (qbck *QueryBcks) AddToQuery(query url.Values)
- func (qbck QueryBcks) Contains(other *Bck) bool
- func (qbck *QueryBcks) Equal(bck *Bck) bool
- func (qbck *QueryBcks) IsAIS() bool
- func (qbck *QueryBcks) IsBucket() bool
- func (qbck *QueryBcks) IsCloud() bool
- func (qbck *QueryBcks) IsEmpty() bool
- func (qbck *QueryBcks) IsHT() bool
- func (qbck *QueryBcks) IsRemoteAIS() bool
- func (qbck *QueryBcks) SetQuery(q url.Values)
- func (qbck QueryBcks) String() string
- func (qbck *QueryBcks) Validate() (err error)
- type RateLimitBase
- type RateLimitBaseToSet
- type RateLimitConf
- type RateLimitConfToSet
- type RebalanceConf
- type RebalanceConfToSet
- type ResilverConf
- type ResilverConfToSet
- type RetryArgs
- type SpaceConf
- type SpaceConfToSet
- type TCBConf
- type TCBConfToSet
- type TCOConf
- type TCOConfToSet
- type TCOMsg
- type TLSArgs
- type TestFSPConf
- type TimeoutConf
- type TimeoutConfToSet
- type TraceExporterAuthConf
- type TraceExporterAuthConfToSet
- type TracingConf
- type TracingConfToSet
- type TransportArgs
- type TransportConf
- type TransportConfToSet
- type Validator
- type VersionConf
- type VersionConfToSet
- type WritePolicyConf
- type WritePolicyConfToSet
- type XactConf
- type XactConfToSet
Constants ¶
const ( PropBucketAccessAttrs = "access" // Bucket access attributes. PropBucketVerEnabled = "versioning.enabled" // Enable/disable object versioning in a bucket. PropBucketCreated = "created" // Bucket creation time. PropBackendBck = "backend_bck" PropBackendBckName = PropBackendBck + ".name" PropBackendBckProvider = PropBackendBck + ".provider" )
const ( DfltDialupTimeout = 10 * time.Second DfltKeepaliveTCP = 30 * time.Second )
const ( DefaultMaxIdleConns = 0 // unlimited (in re: `http.errTooManyIdle`) DefaultMaxIdleConnsPerHost = 32 // (http.errTooManyIdleHost) DefaultIdleConnTimeout = 6 * time.Second // (Go default is 90s) DefaultWriteBufferSize = 64 * cos.KiB DefaultReadBufferSize = 64 * cos.KiB DefaultSndRcvBufferSize = 128 * cos.KiB )
[NOTE] net/http.DefaultTransport has the following defaults:
- MaxIdleConns: 100, - MaxIdleConnsPerHost : 2 (via DefaultMaxIdleConnsPerHost) - IdleConnTimeout: 90 * time.Second, - WriteBufferSize: 4KB - ReadBufferSize: 4KB
See also: docs/idle_connections.md
Following are the defaults we use instead:
const ( ObjSizeToAlwaysReplicate = -1 // (see `ObjSizeLimit` comment above) MinSliceCount = 1 // minimum number of data or parity slices MaxSliceCount = 32 // maximum --/-- )
const ( IgnoreReaction = "ignore" WarnReaction = "warn" AbortReaction = "abort" )
const ( DfltTransportHeader = 4 * cos.KiB // memsys.PageSize MaxTransportHeader = 128 * cos.KiB // memsys.MaxPageSlabSize DfltTransportBurst = 256 MaxTransportBurst = 4096 )
const ( // TODO: // config bloat vs specificity; consider phasing out `timeout.ec_streams_time` and // using `timeout.shared_streams_time` instead LcacheEvictMin = 20 * time.Minute LcacheEvictDflt = 2 * time.Hour LcacheEvictMax = 16 * time.Hour ColdGetConflictDflt = 5 * time.Second ColdGetConflictMin = time.Second )
const ( FmtErrIntegrity = "[%s%d, for troubleshooting see %s/blob/main/docs/troubleshooting.md]" FmtErrUnmarshal = "%s: failed to unmarshal %s (%s), err: %w" FmtErrMorphUnmarshal = "%s: failed to unmarshal %s (%T), err: %w" FmtErrBackwardCompat = "%v (backward compatibility is supported only one version back, e.g. 3.9 => 3.10)" BadSmapPrefix = "[bad cluster map]" StartupMayTimeout = "cluster startup is taking unusually long time..." // related ErrStartupTimeout )
const ( NetPublic = "PUBLIC" NetIntraControl = "INTRA-CONTROL" NetIntraData = "INTRA-DATA" )
const ( // source of the cold-GET and download; the values include all // 3rd party backend providers SourceObjMD = "source" // downloader' source is "web" WebObjMD = "web" VersionObjMD = "version" // "generation" for GCP, "version" for AWS but only if the bucket is versioned, etc. CRC32CObjMD = cos.ChecksumCRC32C MD5ObjMD = cos.ChecksumMD5 ETag = cos.HdrETag OrigURLObjMD = "orig_url" // RFC3339; see also: cos.HdrLastModified formatted RFC1123GMT LsoLastModified = "LastModified" // as the name implies OrigFntl = "orig_fntl" )
LOM custom metadata stored under `lomCustomMD`.
const ( RetryLogVerbose = iota RetryLogQuiet RetryLogOff )
const ( VersionAIStore = "3.30" VersionCLI = "1.19" VersionLoader = "1.13" VersionAuthN = "1.2" )
const ( MetaverSmap = 2 // Smap (cluster map) formatting version a.k.a. meta-version (see core/meta/jsp.go) MetaverBMD = 2 // BMD (bucket metadata) --/-- MetaverRMD = 1 // Rebalance MD (jsp) MetaverVMD = 2 // Volume MD (jsp) MetaverEtlMD = 2 // ETL MD (jsp) MetaverConfig = 4 // Global Configuration (jsp) MetaverAuthNConfig = 1 // Authn config (jsp) // ditto MetaverAuthTokens = 1 // Authn tokens (jsp) // ditto MetaverMetasync = 1 // metasync over network formatting version (jsp) MetaverJSP = jsp.Metaver // `jsp` own encoding version )
const AwsHeaderMetaPrefix = "X-Amz-Meta-"
Due to import cycle we need to define "x-amz-meta" header here See also: - ais/s3/const.go - https://docs.aws.amazon.com/AmazonS3/latest/userguide/UsingMetadata.html#UserMetadata
const AwsMultipartDelim = "-"
from https://docs.aws.amazon.com/AmazonS3/latest/API/API_Object.html "The ETag may or may not be an MD5 digest of the object data. Whether or not it is depends on how the object was created and how it is encrypted..."
const (
DfltMaxIdleTimeout = 30 * time.Second
)
see also: - docs/idle_connections.md - DefaultMaxIdle* in cmn/client - `listen` in ais/htcommon
const GitHubHome = "https://github.com/NVIDIA/aistore"
const HostnameListSepa = ","
const IterFieldNameSepa = "."
const MsgpLsoBufSize = 32 * cos.KiB
const ( // NsGlobalUname is hardcoded here to avoid allocating it via Uname() // (the most common use case) NsGlobalUname = "@#" )
Variables ¶
var ( // NsGlobal represents *this* cluster's global namespace that is used by default when // no specific namespace was defined or provided by the user. NsGlobal = Ns{} // NsAnyRemote represents any remote cluster. As such, NsGlobalRemote applies // exclusively to AIS (provider) given that other Backend providers are remote by definition. NsAnyRemote = Ns{UUID: string(apc.NsUUIDPrefix)} )
var ( ErrSkip = errors.New("skip") ErrStartupTimeout = errors.New("startup timeout") // related StartupMayTimeout ErrQuiesceTimeout = errors.New("timed out waiting for quiescence") ErrNotEnoughTargets = errors.New("not enough target nodes") ErrNoMountpaths = errors.New("no mountpaths") // aborts ErrXactRenewAbort = errors.New("renewal abort") ErrXactUserAbort = errors.New("user abort") // via apc.ActXactStop ErrXactICNotifAbort = errors.New("IC(notifications) abort") // ditto ErrGetTxBenign = errors.New("Warning: failed to transmit GET response") //nolint:staticcheck // making an exception for Warning )
var BackendHelpers = struct { Amazon backendFuncs Azure backendFuncs Google backendFuncs OCI backendFuncs HTTP backendFuncs }{ Amazon: backendFuncs{ EncodeVersion: func(v any) (string, bool) { switch x := v.(type) { case *string: if awsIsVersionSet(x) { return *x, true } return "", false case string: if awsIsVersionSet(&x) { return x, true } return x, false default: debug.FailTypeCast(v) return "", false } }, EncodeETag: func(v any) (string, bool) { switch x := v.(type) { case *string: return *x, *x != "" case string: return x, x != "" default: debug.FailTypeCast(v) return "", false } }, EncodeCksum: func(v any) (string, bool) { switch x := v.(type) { case *string: if x == nil || *x == "" { return "", false } if isS3MultipartEtag(*x) { return "", false } return UnquoteCEV(*x), true case string: if x == "" { return "", false } if isS3MultipartEtag(x) { return "", false } return UnquoteCEV(x), true default: debug.FailTypeCast(v) return "", false } }, EncodeMetadata: func(metadata map[string]string) (header map[string]string) { if len(metadata) == 0 { return } header = make(map[string]string, len(metadata)) for k, v := range metadata { key := http.CanonicalHeaderKey(AwsHeaderMetaPrefix + k) header[key] = v } return }, DecodeMetadata: func(header http.Header) (metadata map[string]string) { for headerKey := range header { if strings.HasPrefix(headerKey, AwsHeaderMetaPrefix) { if metadata == nil { metadata = make(map[string]string) } key := strings.TrimPrefix(headerKey, AwsHeaderMetaPrefix) value := header.Get(headerKey) metadata[key] = value } } return }, }, Google: backendFuncs{ EncodeVersion: func(v any) (string, bool) { switch x := v.(type) { case string: return x, x != "" case int64: return strconv.FormatInt(x, 10), true default: debug.FailTypeCast(v) return "", false } }, EncodeETag: func(v any) (string, bool) { switch x := v.(type) { case string: return x, x != "" default: debug.FailTypeCast(v) return "", false } }, EncodeCksum: func(v any) (string, bool) { switch x := v.(type) { case string: decoded, err := base64.StdEncoding.DecodeString(x) if err != nil { return "", false } return hex.EncodeToString(decoded), true case []byte: return hex.EncodeToString(x), true case uint32: b := []byte{byte(x >> 24), byte(x >> 16), byte(x >> 8), byte(x)} return base64.StdEncoding.EncodeToString(b), true default: debug.FailTypeCast(v) return "", false } }, }, OCI: backendFuncs{ EncodeVersion: func(_ any) (string, bool) { return "", false }, EncodeETag: func(v any) (string, bool) { switch x := v.(type) { case *string: if x == nil || *x == "" { return "", false } return UnquoteCEV(*x), true default: debug.FailTypeCast(v) return "", false } }, EncodeCksum: func(v any) (string, bool) { switch x := v.(type) { case *string: if x == nil || *x == "" { return "", false } return UnquoteCEV(*x), true default: debug.FailTypeCast(v) return "", false } }, DecodeMetadata: func(header http.Header) (metadata map[string]string) { for headerKey := range header { if strings.HasPrefix(headerKey, AwsHeaderMetaPrefix) { if metadata == nil { metadata = make(map[string]string) } key := strings.TrimPrefix(headerKey, AwsHeaderMetaPrefix) value := header.Get(headerKey) metadata[key] = value } } return }, }, HTTP: backendFuncs{ EncodeETag: func(v any) (string, bool) { switch x := v.(type) { case string: return x, x != "" default: debug.FailTypeCast(v) return "", false } }, }, }
var ConfigRestartRequired = [...]string{"auth.secret", "memsys", "net"}
assorted named fields that require (cluster | node) restart for changes to make an effect (used by CLI)
var GCO *gco
var KnownNetworks = [...]string{NetPublic, NetIntraControl, NetIntraData}
var Rom readMostly
var SupportedReactions = []string{IgnoreReaction, WarnReaction, AbortReaction}
Functions ¶
func CheckDirNoRecurs ¶ added in v1.3.28
no recursion (LsNoRecursion) helper function: check the level of nesting return:
- true, to include virtual directory
- filepath.SkipDir, to skip further recursion
func CopyHeaders ¶ added in v1.3.30
Copies headers from original request(from client) to a new one(inter-cluster call)
func CustomMD2S ¶
[NOTE] - usage: LOM custom metadata => LsoEnt custom property - `OrigFntl` always excepted (and possibly other TBD internal keys)
func CustomProps2S ¶ added in v1.3.26
(compare w/ CustomMD2S above)
func DirHasOrIsPrefix ¶ added in v1.3.16
directory has to either: - include (or match) prefix, or - be contained in prefix - motivation: don't SkipDir a/b when looking for a/b/c an alternative name for this function could be smth. like SameBranch() see also: cos.TrimPrefix
func EnvToTLS ¶ added in v1.3.21
func EnvToTLS(sargs *TLSArgs)
EnvToTLS usage is limited to aisloader and tools NOTE that embedded intra-cluster clients utilize a similar method: `HTTPConf.ToTLS`
func FreeBuffer ¶ added in v1.3.18
func HasSpecialSymbols ¶ added in v1.3.30
RFC 3986 defines a set of reserved characters that have special meaning within a URI (Uniform Resource Identifier). These include delimiters for different URI components (like '?' for query, '#' for fragment) and sub-delimiters. Additionally, certain characters like space ' ' are "unsafe" and must be percent-encoded when appearing in URI components like the path.
This function checks for the presence of such special/unsafe characters in a given path string. Its primary use case is to determine if a URL path (which usually contains bucket/object names) can be safely concatenated into a new URL, or if it requires robust URL parsing and re-encoding via `net/url.URL` to ensure correctness (e.g., proper percent-encoding of spaces, ampersands, etc. for the target URL).
The characters checked: - '%': Indicates percent-encoding (e.g., %20 for space) or a literal percent sign. - '?': Query string delimiter. - '#': Fragment identifier delimiter. - ' ': Space (unsafe, must be %20 encoded). - '+': Often used for space in x-www-form-urlencoded, or literal plus. Not checking: - '&': See '?' above - '=': Ditto
func InitObjProps2Hdr ¶ added in v1.3.24
func InitObjProps2Hdr()
func IsErrAborted ¶
func IsErrBckNotFound ¶
func IsErrBucketLevel ¶
func IsErrCapExceeded ¶ added in v1.3.19
func IsErrEmptyProvider ¶ added in v1.3.26
func IsErrFailedTo ¶ added in v1.3.26
func IsErrGetCap ¶ added in v1.3.24
func IsErrInitMissingBackend ¶ added in v1.3.26
func IsErrLmetaCorrupted ¶
func IsErrLmetaNotFound ¶
func IsErrMpathCheck ¶ added in v1.3.24
func IsErrMpathNewDisk ¶ added in v1.3.24
func IsErrMpathNotFound ¶ added in v1.3.24
func IsErrObjLevel ¶
func IsErrRangeNotSatisfiable ¶ added in v1.3.22
func IsErrRemoteBckNotFound ¶
func IsErrStreamTerminated ¶ added in v1.3.16
func IsErrTooManyRequests ¶ added in v1.3.28
func IsErrWarning ¶ added in v1.3.24
func IsErrXactNotFound ¶ added in v1.3.16
func IsErrXactUsePrev ¶ added in v1.3.18
func IsFileAlreadyClosed ¶ added in v1.3.16
func IsStatusBadGateway ¶
func IsStatusGone ¶
func IsStatusNotFound ¶
func IterFields ¶
IterFields walks the struct and calls `updf` callback at every leaf field that it encounters. The (nested) names are created by joining the json tag with dot. Iteration supports reading another, custom tag `list` with values:
- `tagOmitempty` - omit empty fields (only for read run)
- `tagOmit` - omit field
- `tagReadonly` - field cannot be updated (returns error on `SetValue`)
Examples of usages for tags can be found in `BucketProps` or `Config` structs.
Passing additional options with `IterOpts` can for example call callback also at the non-leaf structures.
func KeepaliveRetryDuration ¶
func LoadConfig ¶
is called at startup
func MakeRangeHdr ¶ added in v1.3.16
Ref: https://www.rfc-editor.org/rfc/rfc7233#section-2.1 (compare w/ htrange.contentRange)
func NetworkIsKnown ¶
func NewClient ¶
func NewClient(cargs TransportArgs) *http.Client
NOTE: `NewTransport` (below) fills-in certain defaults
func NewClientTLS ¶ added in v1.3.21
func NewClientTLS(cargs TransportArgs, sargs TLSArgs, intra bool) *http.Client
https client (ditto)
func NewDefaultClients ¶ added in v1.3.22
TODO -- FIXME: this call must get cert file and key to be used for the `clientTLS`
func NewErrRemoteMetadataMismatch ¶ added in v1.3.24
func NewIntraClientTLS ¶ added in v1.3.21
func NewIntraClientTLS(cargs TransportArgs, config *Config) *http.Client
func NewTransport ¶
func NewTransport(cargs TransportArgs) *http.Transport
{TransportArgs + defaults} => http.Transport for a variety of ais clients NOTE: TLS below, and separately
func NormalizeProvider ¶
func ObjHasPrefix ¶ added in v1.3.16
see also: cos.TrimPrefix
func OrigURLBck2Name ¶
func ParseRangeHdr ¶ added in v1.3.26
func ParseReadHeaderTimeout ¶ added in v1.3.22
func ParseURLScheme ¶
Splits url into [(scheme)://](address). It's not possible to use url.Parse as (from url.Parse() docs) 'Trying to parse a hostname and path without a scheme is invalid'
func PrependProtocol ¶
PrependProtocol prepends protocol in URL in case it is missing. By default it adds `http://` to the URL.
func S2CustomMD ¶ added in v1.3.22
func S2CustomVal ¶ added in v1.3.26
func SaveOverrideConfig ¶
func SaveOverrideConfig(configDir string, toUpdate *ConfigToSet) error
func SortLso ¶
func SortLso(entries LsoEntries)
func ToHeader ¶
may set headers: - standard cos.HdrContentLength ("Content-Length") & cos.HdrETag ("ETag") - atime, version, etc. - all the rest "ais-" prefixed
func TokenGreaterEQ ¶
Returns true if the continuation token >= object's name (in other words, the object is already listed and must be skipped). Note that string `>=` is lexicographic.
func TypeCodeHTTPErr ¶ added in v1.3.19
func UnquoteCEV ¶ added in v1.3.22
unquote checksum, ETag, and version e.g., https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/ETag
func UpdateFieldValue ¶
UpdateFieldValue updates the field in the struct with given value. Returns error if the field was not found or could not be updated.
func ValidateMpath ¶
func ValidatePort ¶
func ValidateRemAlias ¶
func WriteErr405 ¶
func WriteErr405(w http.ResponseWriter, r *http.Request, methods ...string)
405 Method Not Allowed, see: * https://www.rfc-editor.org/rfc/rfc7231#section-6.5.5
func WriteErrJSON ¶
func WriteErrMsg ¶
Create ErrHTTP (based on `msg` and `opts`) and write it into HTTP response.
Types ¶
type Adaptive ¶ added in v1.3.28
type Adaptive struct {
RateLimitBase
NumRetries int `json:"num_retries"`
}
Adaptive rate limit (a.k.a. rate shaper): usage: handle status 429 and 503 from remote backends - up to max tokens (originally, `MaxTokens`) during one Interval with NumRetries and - `recompute` between Intervals - see cmn/cos/rate_limit
type AdaptiveToSet ¶ added in v1.3.28
type AdaptiveToSet struct {
RateLimitBaseToSet // NOTE: must be in the same exact order/position as above
NumRetries *int `json:"num_retries,omitempty"`
}
global config that can be used to manage: * adaptive rate limit vis-à-vis Cloud backend * rate limiting (bursty) user workloads on the front
type AllBsummResults ¶
type AllBsummResults []*BsummResult
func (AllBsummResults) Aggregate ¶
func (s AllBsummResults) Aggregate(from *BsummResult) AllBsummResults
func (AllBsummResults) Finalize ¶
func (s AllBsummResults) Finalize(dsize map[string]uint64, testingEnv bool)
func (AllBsummResults) Len ¶
func (s AllBsummResults) Len() int
func (AllBsummResults) Less ¶
func (s AllBsummResults) Less(i, j int) bool
func (AllBsummResults) Swap ¶
func (s AllBsummResults) Swap(i, j int)
type ArchConf ¶ added in v1.3.28
type ArchConf struct{ XactConf }
multi-object archive (multiple objects => shard)
type ArchConfToSet ¶ added in v1.3.28
type ArchConfToSet struct{ XactConfToSet }
global configuration
type ArchiveBckMsg ¶ added in v1.3.18
type ArchiveBckMsg struct {
ToBck Bck `json:"tobck"`
apc.ArchiveMsg
}
ArchiveBckMsg contains parameters to archive multiple objects from the specified (source) bucket. Destination bucket may the same as the source or a different one. -------------------- NOTE on terminology: --------------------- "archive" is any (.tar, .tgz/.tar.gz, .zip, .tar.lz4) formatted object often also called "shard"
See also: apc.PutApndArchArgs
func (*ArchiveBckMsg) Cname ¶ added in v1.3.18
func (msg *ArchiveBckMsg) Cname() string
type AuthConfToSet ¶ added in v1.3.21
type AuthConfToSet struct {
Secret *string `json:"secret,omitempty"`
Enabled *bool `json:"enabled,omitempty"`
}
global configuration
type BackendBckToSet ¶ added in v1.3.21
type BackendConf ¶
type BackendConf struct {
Conf map[string]any `json:"-"` // backend implementation-dependent (custom marshaling to populate this field)
Providers map[string]Ns `json:"-"` // conditional (build tag) providers set during validation (BackendConf.Validate)
}
global configuration
func (*BackendConf) EqualRemAIS ¶
func (c *BackendConf) EqualRemAIS(o *BackendConf, sname string) bool
func (*BackendConf) Get ¶ added in v1.3.16
func (c *BackendConf) Get(provider string) (conf any)
func (*BackendConf) MarshalJSON ¶
func (c *BackendConf) MarshalJSON() (data []byte, err error)
func (*BackendConf) Set ¶ added in v1.3.16
func (c *BackendConf) Set(provider string, newConf any)
func (*BackendConf) UnmarshalJSON ¶
func (c *BackendConf) UnmarshalJSON(data []byte) error
func (*BackendConf) Validate ¶
func (c *BackendConf) Validate() (err error)
type BackendConfAIS ¶
global configuration
func (BackendConfAIS) String ¶
func (c BackendConfAIS) String() (s string)
type Bck ¶
type Bck struct {
Props *Bprops `json:"-"`
Name string `json:"name" yaml:"name"`
Provider string `json:"provider" yaml:"provider"` // NOTE: see api/apc/provider.go for supported enum
Ns Ns `json:"namespace" yaml:"namespace" list:"omitempty"`
}
func ParseBckObjectURI ¶
func ParseBckObjectURI(uri string, opts ParseURIOpts) (bck Bck, objName string, err error)
func ParseUname ¶
unique name => Bck - MakeUname to make - compare w/ meta.ParseUname (that does more) and maybe use the latter
func (*Bck) AddUnameToQuery ¶
func (*Bck) DefaultProps ¶
func (bck *Bck) DefaultProps(c *ClusterConfig) *Bprops
By default, created buckets inherit their properties from the cluster (global) configuration. Global configuration, in turn, is protected versioned, checksummed, and replicated across the entire cluster.
* Bucket properties can be changed at any time via `api.SetBprops`. * In addition, `api.CreateBucket` allows to specify (non-default) properties at bucket creation time. * Inherited defaults include checksum, LRU, etc. configurations - see below. * By default, LRU is disabled for AIS (`ais://`) buckets.
See also:
- github.com/NVIDIA/aistore/blob/main/docs/bucket.md#default-bucket-properties
- BpropsToSet (above)
- ais.defaultBckProps()
func (*Bck) HasProvider ¶
func (*Bck) HasVersioningMD ¶ added in v1.3.22
A subset of remote backends that maintain assorted items of versioning information - the items including ETag, checksum, etc. - that, in turn, can be used to populate `ObjAttrs` * see related: `ObjAttrs.Equal`
func (*Bck) IsBuiltTagged ¶ added in v1.3.24
func (*Bck) IsRemoteAIS ¶
func (*Bck) LenUnameGlob ¶ added in v1.3.21
func (*Bck) MakeUname ¶
Bck and optional objName => unique name (uname); note: - use ParseUname below to translate back - compare with HashUname
func (*Bck) ValidateName ¶
type Bprops ¶ added in v1.3.21
type Bprops struct {
BackendBck Bck `json:"backend_bck,omitempty"` // makes a remote bucket out of a given ais://
WritePolicy WritePolicyConf `json:"write_policy"` // write object metadata (immediate | delayed | never)
Provider string `json:"provider" list:"readonly"` // backend provider
Renamed string `list:"omit"` // non-empty iff the bucket has been renamed
Cksum CksumConf `json:"checksum"` // this bucket's checksum (for supported enum, see cmn/cos.cksum)
Extra ExtraProps `json:"extra,omitempty" list:"omitempty"` // e.g., AWS.Endpoint for this bucket
RateLimit RateLimitConf `json:"rate_limit"` // frontend and backend rate limiting - bursty and adaptive, respectively
EC ECConf `json:"ec"` // erasure coding
Mirror MirrorConf `json:"mirror"` // n-way mirroring
LRU LRUConf `json:"lru"` // LRU watermarks and enable/disable
Access apc.AccessAttrs `json:"access,string"` // access permissions
Features feat.Flags `json:"features,string"` // to flip assorted enumerated defaults (e.g. "S3-Use-Path-Style"; see cmn/feat)
BID uint64 `json:"bid,string" list:"omit"` // unique ID
Created int64 `json:"created,string" list:"readonly"` // creation timestamp
Versioning VersionConf `json:"versioning"` // see "inherit"
}
func (*Bprops) Apply ¶ added in v1.3.21
func (bp *Bprops) Apply(propsToSet *BpropsToSet)
func (*Bprops) SetProvider ¶ added in v1.3.21
type BpropsToSet ¶ added in v1.3.21
type BpropsToSet struct {
BackendBck *BackendBckToSet `json:"backend_bck,omitempty"`
Versioning *VersionConfToSet `json:"versioning,omitempty"`
Cksum *CksumConfToSet `json:"checksum,omitempty"`
LRU *LRUConfToSet `json:"lru,omitempty"`
Mirror *MirrorConfToSet `json:"mirror,omitempty"`
EC *ECConfToSet `json:"ec,omitempty"`
Access *apc.AccessAttrs `json:"access,string,omitempty"`
RateLimit *RateLimitConfToSet `json:"rate_limit,omitempty"`
Features *feat.Flags `json:"features,string,omitempty"`
WritePolicy *WritePolicyConfToSet `json:"write_policy,omitempty"`
Extra *ExtraToSet `json:"extra,omitempty"`
Force bool `json:"force,omitempty" copy:"skip" list:"omit"`
}
Once validated, BpropsToSet are copied to Bprops. The struct may have extra fields that do not exist in Bprops. Add tag 'copy:"skip"' to ignore those fields when copying values.
func NewBpropsToSet ¶ added in v1.3.21
func NewBpropsToSet(nvs cos.StrKVs) (props *BpropsToSet, err error)
type BsummResult ¶
type BsummResult struct {
Bck
apc.BsummResult
}
type Bursty ¶ added in v1.3.28
type Bursty struct {
RateLimitBase
Size int `json:"burst_size"`
}
rate limit that fails 'too-many requests' while permitting a certain level of burstiness - usage: to restrict the rate of user GET, PUT, and DELETE requests - see cmn/cos/rate_limit
type BurstyToSet ¶ added in v1.3.28
type BurstyToSet struct {
RateLimitBaseToSet
Size *int `json:"burst_size,omitempty"`
}
global config that can be used to manage: * adaptive rate limit vis-à-vis Cloud backend * rate limiting (bursty) user workloads on the front
type CksumConf ¶
type CksumConf struct {
// (note that `ChecksumNone` ("none") disables checksumming)
Type string `json:"type"`
// validate the checksum of the object that we cold-GET
// or download from remote location (e.g., cloud bucket)
ValidateColdGet bool `json:"validate_cold_get"`
// - validate in-cluster object's checksum(s);
// - upon any of the `cos.ErrBadCksum` errors try to recover from
// local redundant copies, and/or EC slices, and/or remote backend if exists;
// - if all fails, remove the object and fail the GET.
ValidateWarmGet bool `json:"validate_warm_get"`
// validate checksums of objects migrated or replicated within the cluster
ValidateObjMove bool `json:"validate_obj_move"`
// EnableReadRange: Return read range checksum otherwise return entire object checksum.
EnableReadRange bool `json:"enable_read_range"`
}
global configuration
func (*CksumConf) ValidateAsProps ¶
type CksumConfToSet ¶ added in v1.3.21
type CksumConfToSet struct {
Type *string `json:"type,omitempty"`
ValidateColdGet *bool `json:"validate_cold_get,omitempty"`
ValidateWarmGet *bool `json:"validate_warm_get,omitempty"`
ValidateObjMove *bool `json:"validate_obj_move,omitempty"`
EnableReadRange *bool `json:"enable_read_range,omitempty"`
}
global configuration
type ClientConf ¶
type ClientConf struct {
Timeout cos.Duration `json:"client_timeout"`
TimeoutLong cos.Duration `json:"client_long_timeout"`
ListObjTimeout cos.Duration `json:"list_timeout"`
}
global configuration
func (*ClientConf) Validate ¶
func (c *ClientConf) Validate() error
type ClientConfToSet ¶ added in v1.3.21
type ClientConfToSet struct {
Timeout *cos.Duration `json:"client_timeout,omitempty"` // readonly as far as intra-cluster
TimeoutLong *cos.Duration `json:"client_long_timeout,omitempty"`
ListObjTimeout *cos.Duration `json:"list_timeout,omitempty"`
}
global configuration
type ClusterConfig ¶
type ClusterConfig struct {
Backend BackendConf `json:"backend" allow:"cluster"`
Ext any `json:"ext,omitempty"` // reserved
WritePolicy WritePolicyConf `json:"write_policy"` // object metadata write policy: (immediate | delayed | never)
LastUpdated string `json:"lastupdate_time"`
UUID string `json:"uuid"`
Dsort DsortConf `json:"distributed_sort"`
Proxy ProxyConf `json:"proxy" allow:"cluster"`
Cksum CksumConf `json:"checksum"`
Auth AuthConf `json:"auth"`
Tracing TracingConf `json:"tracing"`
TCB TCBConf `json:"tcb"`
TCO TCOConf `json:"tco"`
Arch ArchConf `json:"arch"`
RateLimit RateLimitConf `json:"rate_limit"`
Keepalive KeepaliveConf `json:"keepalivetracker"`
Rebalance RebalanceConf `json:"rebalance" allow:"cluster"`
Log LogConf `json:"log"`
EC ECConf `json:"ec" allow:"cluster"`
Net NetConf `json:"net"`
Timeout TimeoutConf `json:"timeout"`
Transport TransportConf `json:"transport"`
Memsys MemsysConf `json:"memsys"`
FSHC FSHCConf `json:"fshc"`
Disk DiskConf `json:"disk"`
Space SpaceConf `json:"space"`
Periodic PeriodConf `json:"periodic"`
Client ClientConf `json:"client"`
Mirror MirrorConf `json:"mirror" allow:"cluster"`
LRU LRUConf `json:"lru"`
Downloader DownloaderConf `json:"downloader"`
Features feat.Flags `json:"features,string" allow:"cluster"` // enumerated features to flip assorted global defaults (cmn/feat/feat.go)
Version int64 `json:"config_version,string"`
Versioning VersionConf `json:"versioning" allow:"cluster"`
Resilver ResilverConf `json:"resilver"`
}
global configuration
func (*ClusterConfig) Apply ¶
func (c *ClusterConfig) Apply(updateConf *ConfigToSet, asType string) error
func (*ClusterConfig) JspOpts ¶
func (*ClusterConfig) JspOpts() jsp.Options
func (*ClusterConfig) String ¶
func (c *ClusterConfig) String() string
type Config ¶
type Config struct {
LocalConfig `json:",inline"`
ClusterConfig `json:",inline"`
// contains filtered or unexported fields
}
Config is a single control structure (persistent and versioned) that contains both cluster (global) and node (local) configuration Naming convention for setting/getting values: (parent section json tag . child json tag) See also: `IterFields`, `IterFieldNameSepa`
func (*Config) TestingEnv ¶
TestingEnv returns true if config is set to a development environment where a single local filesystem is partitioned between all (locally running) targets and is used for both local and Cloud buckets See also: `rom.testingEnv`
func (*Config) UpdateClusterConfig ¶
func (c *Config) UpdateClusterConfig(updateConf *ConfigToSet, asType string) (err error)
type ConfigToSet ¶ added in v1.3.21
type ConfigToSet struct {
// ClusterConfig
Backend *BackendConf `json:"backend,omitempty"`
Mirror *MirrorConfToSet `json:"mirror,omitempty"`
EC *ECConfToSet `json:"ec,omitempty"`
Log *LogConfToSet `json:"log,omitempty"`
Periodic *PeriodConfToSet `json:"periodic,omitempty"`
Tracing *TracingConfToSet `json:"tracing,omitempty"`
Timeout *TimeoutConfToSet `json:"timeout,omitempty"`
Client *ClientConfToSet `json:"client,omitempty"`
Space *SpaceConfToSet `json:"space,omitempty"`
LRU *LRUConfToSet `json:"lru,omitempty"`
Disk *DiskConfToSet `json:"disk,omitempty"`
Rebalance *RebalanceConfToSet `json:"rebalance,omitempty"`
Resilver *ResilverConfToSet `json:"resilver,omitempty"`
Cksum *CksumConfToSet `json:"checksum,omitempty"`
Versioning *VersionConfToSet `json:"versioning,omitempty"`
Net *NetConfToSet `json:"net,omitempty"`
FSHC *FSHCConfToSet `json:"fshc,omitempty"`
Auth *AuthConfToSet `json:"auth,omitempty"`
Keepalive *KeepaliveConfToSet `json:"keepalivetracker,omitempty"`
Downloader *DownloaderConfToSet `json:"downloader,omitempty"`
Dsort *DsortConfToSet `json:"distributed_sort,omitempty"`
Transport *TransportConfToSet `json:"transport,omitempty"`
Memsys *MemsysConfToSet `json:"memsys,omitempty"`
TCB *TCBConfToSet `json:"tcb,omitempty"`
WritePolicy *WritePolicyConfToSet `json:"write_policy,omitempty"`
Proxy *ProxyConfToSet `json:"proxy,omitempty"`
RateLimit *RateLimitConfToSet `json:"rate_limit"`
Features *feat.Flags `json:"features,string,omitempty"`
// LocalConfig
FSP *FSPConf `json:"fspaths,omitempty"`
}
global configuration
func (*ConfigToSet) FillFromKVS ¶ added in v1.3.21
func (ctu *ConfigToSet) FillFromKVS(kvs []string) (err error)
FillFromKVS populates `ConfigToSet` from key value pairs of the form `key=value`
func (*ConfigToSet) FillFromQuery ¶ added in v1.3.21
func (ctu *ConfigToSet) FillFromQuery(query url.Values) error
FillFromQuery populates ConfigToSet from URL query values
func (*ConfigToSet) JspOpts ¶ added in v1.3.21
func (*ConfigToSet) JspOpts() jsp.Options
func (*ConfigToSet) Merge ¶ added in v1.3.21
func (ctu *ConfigToSet) Merge(update *ConfigToSet)
type DiskConf ¶
type DiskConf struct {
DiskUtilLowWM int64 `json:"disk_util_low_wm"` // no throttling below
DiskUtilHighWM int64 `json:"disk_util_high_wm"` // throttle longer when above
DiskUtilMaxWM int64 `json:"disk_util_max_wm"`
IostatTimeLong cos.Duration `json:"iostat_time_long"`
IostatTimeShort cos.Duration `json:"iostat_time_short"`
}
global configuration
type DiskConfToSet ¶ added in v1.3.21
type DiskConfToSet struct {
DiskUtilLowWM *int64 `json:"disk_util_low_wm,omitempty"`
DiskUtilHighWM *int64 `json:"disk_util_high_wm,omitempty"`
DiskUtilMaxWM *int64 `json:"disk_util_max_wm,omitempty"`
IostatTimeLong *cos.Duration `json:"iostat_time_long,omitempty"`
IostatTimeShort *cos.Duration `json:"iostat_time_short,omitempty"`
}
global configuration
type DownloaderConf ¶
global configuration
func (*DownloaderConf) Validate ¶
func (c *DownloaderConf) Validate() error
type DownloaderConfToSet ¶ added in v1.3.21
global configuration
type DsortConf ¶ added in v1.3.21
type DsortConf struct {
XactConf
DuplicatedRecords string `json:"duplicated_records"`
MissingShards string `json:"missing_shards"` // cmn.SupportedReactions enum
EKMMalformedLine string `json:"ekm_malformed_line"`
EKMMissingKey string `json:"ekm_missing_key"`
DefaultMaxMemUsage string `json:"default_max_mem_usage"`
DsorterMemThreshold string `json:"dsorter_mem_threshold"`
CallTimeout cos.Duration `json:"call_timeout"`
}
global configuration
func (*DsortConf) ValidateWithOpts ¶ added in v1.3.21
type DsortConfToSet ¶ added in v1.3.21
type DsortConfToSet struct {
XactConfToSet
DuplicatedRecords *string `json:"duplicated_records,omitempty"`
MissingShards *string `json:"missing_shards,omitempty"`
EKMMalformedLine *string `json:"ekm_malformed_line,omitempty"`
EKMMissingKey *string `json:"ekm_missing_key,omitempty"`
DefaultMaxMemUsage *string `json:"default_max_mem_usage,omitempty"`
CallTimeout *cos.Duration `json:"call_timeout,omitempty"`
DsorterMemThreshold *string `json:"dsorter_mem_threshold,omitempty"`
}
global configuration
type ECConf ¶
type ECConf struct {
XactConf
// ObjSizeLimit is object size threshold _separating_ intra-cluster mirroring from
// erasure coding.
//
// The value 0 (zero) indicates that objects of any size
// are to be sliced, to produce (D) data slices and (P) erasure coded parity slices.
// On the other hand, the value -1 specifies that absolutely all objects of any size
// must be replicated as is. In effect, the (-1) option provides data protection via
// intra-cluster (P+1)-way replication (a.k.a. mirroring).
//
// In all cases, a given (D, P) configuration provides node-level redundancy,
// whereby P nodes can be lost without incurring loss of data.
ObjSizeLimit int64 `json:"objsize_limit"`
// Number of data (D) slices; the value 1 will have an effect of producing
// (P) additional full-size replicas.
DataSlices int `json:"data_slices"`
// Depending on the object size and `ObjSizeLimit`, the value of `ParitySlices` (or P) indicates:
// - a number of additional parity slices (generated or _computed_ from the (D) data slices),
// or:
// - a number of full object replicas (copies).
// In all cases, the same rule applies: all slices and/or all full copies are stored on different
// storage nodes (a.k.a. targets).
ParitySlices int `json:"parity_slices"`
Enabled bool `json:"enabled"` // EC is enabled
DiskOnly bool `json:"disk_only"` // if true, EC does not use SGL - data goes directly to drives
}
global configuration
func (*ECConf) RequiredRestoreTargets ¶
func (*ECConf) ValidateAsProps ¶
type ECConfToSet ¶ added in v1.3.21
type ECConfToSet struct {
XactConfToSet
ObjSizeLimit *int64 `json:"objsize_limit,omitempty"`
DataSlices *int `json:"data_slices,omitempty"`
ParitySlices *int `json:"parity_slices,omitempty"`
Enabled *bool `json:"enabled,omitempty"`
DiskOnly *bool `json:"disk_only,omitempty"`
}
global configuration
type ETLErrCtx ¶ added in v1.3.16
type ETLErrCtx struct {
TID string
ETLName string
ETLTransformArgs string
PodName string
SvcName string
k8s.PodStatus
}
assorted aistore errors
type ErrAborted ¶
type ErrAborted struct {
// contains filtered or unexported fields
}
assorted aistore errors
func AsErrAborted ¶
func AsErrAborted(err error) (errAborted *ErrAborted)
func NewErrAborted ¶
func NewErrAborted(what, ctx string, err error) *ErrAborted
func (*ErrAborted) Error ¶
func (e *ErrAborted) Error() (s string)
func (*ErrAborted) Unwrap ¶
func (e *ErrAborted) Unwrap() (err error)
type ErrBckNotFound ¶
type ErrBckNotFound struct {
// contains filtered or unexported fields
}
assorted aistore errors
func NewErrBckNotFound ¶
func NewErrBckNotFound(bck *Bck) *ErrBckNotFound
func (*ErrBckNotFound) Error ¶
func (e *ErrBckNotFound) Error() string
type ErrBucketAccessDenied ¶
type ErrBucketAccessDenied struct {
// contains filtered or unexported fields
}
assorted aistore errors
func NewBucketAccessDenied ¶
func NewBucketAccessDenied(bucket, oper string, aattrs apc.AccessAttrs) *ErrBucketAccessDenied
func (*ErrBucketAccessDenied) Error ¶
func (e *ErrBucketAccessDenied) Error() string
type ErrBucketAlreadyExists ¶
type ErrBucketAlreadyExists struct {
// contains filtered or unexported fields
}
assorted aistore errors
func NewErrBckAlreadyExists ¶
func NewErrBckAlreadyExists(bck *Bck) *ErrBucketAlreadyExists
func (*ErrBucketAlreadyExists) Error ¶
func (e *ErrBucketAlreadyExists) Error() string
type ErrBusy ¶ added in v1.3.21
type ErrBusy struct {
// contains filtered or unexported fields
}
assorted aistore errors
func NewErrBusy ¶ added in v1.3.21
type ErrCapExceeded ¶ added in v1.3.19
type ErrCapExceeded struct {
// contains filtered or unexported fields
}
assorted aistore errors
func NewErrCapExceeded ¶ added in v1.3.19
func NewErrCapExceeded(totalBytesUsed, totalBytes uint64, highWM, cleanupWM int64, usedPct int32, oos bool) *ErrCapExceeded
func (*ErrCapExceeded) Error ¶ added in v1.3.19
func (e *ErrCapExceeded) Error() string
type ErrCreateHreq ¶ added in v1.3.26
type ErrCreateHreq struct {
// contains filtered or unexported fields
}
assorted aistore errors
func NewErrCreateHreq ¶ added in v1.3.26
func NewErrCreateHreq(err error) *ErrCreateHreq
func (*ErrCreateHreq) Error ¶ added in v1.3.26
func (e *ErrCreateHreq) Error() string
func (*ErrCreateHreq) Unwrap ¶ added in v1.3.26
func (e *ErrCreateHreq) Unwrap() (err error)
type ErrEmptyProvider ¶ added in v1.3.26
type ErrEmptyProvider struct {
// contains filtered or unexported fields
}
func (*ErrEmptyProvider) Error ¶ added in v1.3.26
func (e *ErrEmptyProvider) Error() string
type ErrFailedTo ¶
type ErrFailedTo struct {
// contains filtered or unexported fields
}
assorted aistore errors
func NewErrFailedTo ¶
func (*ErrFailedTo) Error ¶
func (e *ErrFailedTo) Error() string
func (*ErrFailedTo) Unwrap ¶
func (e *ErrFailedTo) Unwrap() (err error)
type ErrGetCap ¶ added in v1.3.24
type ErrGetCap struct {
// contains filtered or unexported fields
}
assorted aistore errors
func NewErrGetCap ¶ added in v1.3.24
type ErrHTTP ¶
type ErrHTTP struct {
TypeCode string `json:"tcode,omitempty"`
Message string `json:"message"`
Method string `json:"method"`
URLPath string `json:"url_path"`
RemoteAddr string `json:"remote_addr"`
Caller string `json:"caller"`
Node string `json:"node"`
Status int `json:"status"`
// contains filtered or unexported fields
}
API error structure is returned to aistore client and carries one of the specific errors enumerated below
func InitErrHTTP ¶
uses `allocHterr` to allocate - caller must free via `FreeHterr`
func Str2HTTPErr ¶ added in v1.3.18
func UnwrapErrHTTP ¶ added in v1.3.30
type ErrInitBackend ¶
type ErrInitBackend struct {
Provider string
}
assorted aistore errors
func (*ErrInitBackend) Error ¶
func (e *ErrInitBackend) Error() string
type ErrInvalidBackendProvider ¶
type ErrInvalidBackendProvider struct {
// contains filtered or unexported fields
}
assorted aistore errors
func (*ErrInvalidBackendProvider) Error ¶
func (e *ErrInvalidBackendProvider) Error() string
func (*ErrInvalidBackendProvider) Is ¶
func (*ErrInvalidBackendProvider) Is(err error) bool
type ErrInvalidCksum ¶
type ErrInvalidCksum struct {
// contains filtered or unexported fields
}
assorted aistore errors
func NewErrInvalidCksum ¶
func NewErrInvalidCksum(eHash, aHash string) *ErrInvalidCksum
func (*ErrInvalidCksum) Error ¶
func (e *ErrInvalidCksum) Error() string
func (*ErrInvalidCksum) Expected ¶
func (e *ErrInvalidCksum) Expected() string
type ErrInvalidFSPathsConf ¶
type ErrInvalidFSPathsConf struct {
// contains filtered or unexported fields
}
assorted aistore errors
func NewErrInvalidFSPathsConf ¶
func NewErrInvalidFSPathsConf(err error) *ErrInvalidFSPathsConf
func (*ErrInvalidFSPathsConf) Error ¶
func (e *ErrInvalidFSPathsConf) Error() string
func (*ErrInvalidFSPathsConf) Unwrap ¶
func (e *ErrInvalidFSPathsConf) Unwrap() (err error)
type ErrInvalidMountpath ¶
type ErrInvalidMountpath struct {
// contains filtered or unexported fields
}
assorted aistore errors
func NewErrInvalidaMountpath ¶
func NewErrInvalidaMountpath(mpath, cause string) *ErrInvalidMountpath
func (*ErrInvalidMountpath) Error ¶
func (e *ErrInvalidMountpath) Error() string
type ErrLimitedCoexistence ¶
type ErrLimitedCoexistence struct {
// contains filtered or unexported fields
}
assorted aistore errors
func NewErrLimitedCoexistence ¶
func NewErrLimitedCoexistence(node, xaction, action, detail string) *ErrLimitedCoexistence
func (*ErrLimitedCoexistence) Error ¶
func (e *ErrLimitedCoexistence) Error() string
type ErrLmetaCorrupted ¶
type ErrLmetaCorrupted struct {
// contains filtered or unexported fields
}
assorted aistore errors
func NewErrLmetaCorrupted ¶
func NewErrLmetaCorrupted(err error) *ErrLmetaCorrupted
func (*ErrLmetaCorrupted) Error ¶
func (e *ErrLmetaCorrupted) Error() string
func (*ErrLmetaCorrupted) Unwrap ¶
func (e *ErrLmetaCorrupted) Unwrap() (err error)
type ErrLmetaNotFound ¶
type ErrLmetaNotFound struct {
// contains filtered or unexported fields
}
assorted aistore errors
func NewErrLmetaNotFound ¶
func NewErrLmetaNotFound(name string, err error) *ErrLmetaNotFound
func (*ErrLmetaNotFound) Error ¶
func (e *ErrLmetaNotFound) Error() string
func (*ErrLmetaNotFound) Unwrap ¶
func (e *ErrLmetaNotFound) Unwrap() (err error)
type ErrMembershipChanges ¶ added in v1.3.28
type ErrMembershipChanges struct {
// contains filtered or unexported fields
}
assorted aistore errors
func NewErrMembershipChanges ¶ added in v1.3.28
func NewErrMembershipChanges(info string) *ErrMembershipChanges
func (*ErrMembershipChanges) Error ¶ added in v1.3.28
func (e *ErrMembershipChanges) Error() string
type ErrMissingBackend ¶
assorted aistore errors
func (*ErrMissingBackend) Error ¶
func (e *ErrMissingBackend) Error() string
type ErrMpathCheck ¶ added in v1.3.24
type ErrMpathCheck struct {
// contains filtered or unexported fields
}
assorted aistore errors
func NewErrMpathCheck ¶ added in v1.3.24
func NewErrMpathCheck(err error) *ErrMpathCheck
func (*ErrMpathCheck) Error ¶ added in v1.3.24
func (e *ErrMpathCheck) Error() string
type ErrMpathLostDisk ¶ added in v1.3.24
type ErrMpathLostDisk struct {
// contains filtered or unexported fields
}
assorted aistore errors
func NewErrMpathLostDisk ¶ added in v1.3.24
func NewErrMpathLostDisk(mpath, fsname, lostd string, disks, fsdisks []string) *ErrMpathLostDisk
func (*ErrMpathLostDisk) Error ¶ added in v1.3.24
func (e *ErrMpathLostDisk) Error() string
type ErrMpathNewDisk ¶ added in v1.3.24
type ErrMpathNewDisk struct {
// contains filtered or unexported fields
}
assorted aistore errors
func NewErrMpathNewDisk ¶ added in v1.3.24
func NewErrMpathNewDisk(mpath, fsname string, disks, fsdisks []string) *ErrMpathNewDisk
func (*ErrMpathNewDisk) Error ¶ added in v1.3.24
func (e *ErrMpathNewDisk) Error() string
type ErrMpathNoDisks ¶ added in v1.3.24
type ErrMpathNoDisks struct {
// contains filtered or unexported fields
}
assorted aistore errors
func NewErrMpathNoDisks ¶ added in v1.3.24
func NewErrMpathNoDisks(mpath, fsname string, err error) *ErrMpathNoDisks
func (*ErrMpathNoDisks) Error ¶ added in v1.3.24
func (e *ErrMpathNoDisks) Error() string
type ErrMpathNotFound ¶ added in v1.3.24
type ErrMpathNotFound struct {
// contains filtered or unexported fields
}
assorted aistore errors
func NewErrMpathNotFound ¶ added in v1.3.24
func NewErrMpathNotFound(mpath, fqn string, disabled bool) *ErrMpathNotFound
func (*ErrMpathNotFound) Disabled ¶ added in v1.3.24
func (e *ErrMpathNotFound) Disabled() bool
func (*ErrMpathNotFound) Error ¶ added in v1.3.24
func (e *ErrMpathNotFound) Error() string
func (*ErrMpathNotFound) Mpath ¶ added in v1.3.24
func (e *ErrMpathNotFound) Mpath() string
type ErrNoNodes ¶
type ErrNoNodes struct {
// contains filtered or unexported fields
}
assorted aistore errors
func NewErrNoNodes ¶
func NewErrNoNodes(role string, mmcount int) *ErrNoNodes
func (*ErrNoNodes) Error ¶
func (e *ErrNoNodes) Error() (s string)
type ErrNotImpl ¶
type ErrNotImpl struct {
// contains filtered or unexported fields
}
assorted aistore errors
func NewErrNotImpl ¶
func NewErrNotImpl(action, what string) *ErrNotImpl
func (*ErrNotImpl) Error ¶
func (e *ErrNotImpl) Error() string
type ErrNotRemoteBck ¶ added in v1.3.22
type ErrNotRemoteBck struct {
// contains filtered or unexported fields
}
assorted aistore errors
func ValidateRemoteBck ¶ added in v1.3.22
func ValidateRemoteBck(act string, bck *Bck) (err *ErrNotRemoteBck)
func (*ErrNotRemoteBck) Error ¶ added in v1.3.22
func (e *ErrNotRemoteBck) Error() string
type ErrObjDefunct ¶
type ErrObjDefunct struct {
// contains filtered or unexported fields
}
assorted aistore errors
func NewErrObjDefunct ¶
func NewErrObjDefunct(name string, d1, d2 uint64) *ErrObjDefunct
func (*ErrObjDefunct) Error ¶
func (e *ErrObjDefunct) Error() string
type ErrObjectAccessDenied ¶
type ErrObjectAccessDenied struct {
// contains filtered or unexported fields
}
assorted aistore errors
func NewObjectAccessDenied ¶
func NewObjectAccessDenied(object, oper string, aattrs apc.AccessAttrs) *ErrObjectAccessDenied
func (*ErrObjectAccessDenied) Error ¶
func (e *ErrObjectAccessDenied) Error() string
type ErrRangeNotSatisfiable ¶ added in v1.3.22
type ErrRangeNotSatisfiable struct {
// contains filtered or unexported fields
}
assorted aistore errors
func NewErrRangeNotSatisfiable ¶ added in v1.3.22
func NewErrRangeNotSatisfiable(err error, ranges []string, size int64) *ErrRangeNotSatisfiable
func (*ErrRangeNotSatisfiable) Error ¶ added in v1.3.22
func (e *ErrRangeNotSatisfiable) Error() string
type ErrRateLimitFrontend ¶ added in v1.3.28
type ErrRateLimitFrontend ErrTooManyRequests // to differentiate by tcode
assorted aistore errors
func NewErrRateLimitFrontend ¶ added in v1.3.28
func NewErrRateLimitFrontend() *ErrRateLimitFrontend
func (*ErrRateLimitFrontend) Error ¶ added in v1.3.28
func (e *ErrRateLimitFrontend) Error() string
type ErrRemoteBckNotFound ¶
type ErrRemoteBckNotFound struct {
// contains filtered or unexported fields
}
assorted aistore errors
func NewErrRemoteBckNotFound ¶
func NewErrRemoteBckNotFound(bck *Bck) *ErrRemoteBckNotFound
func (*ErrRemoteBckNotFound) Error ¶
func (e *ErrRemoteBckNotFound) Error() string
func (*ErrRemoteBckNotFound) Set ¶ added in v1.3.26
func (e *ErrRemoteBckNotFound) Set(ctx string)
type ErrRemoteBucketOffline ¶
type ErrRemoteBucketOffline struct {
// contains filtered or unexported fields
}
assorted aistore errors
func NewErrRemoteBckOffline ¶
func NewErrRemoteBckOffline(bck *Bck) *ErrRemoteBucketOffline
func (*ErrRemoteBucketOffline) Error ¶
func (e *ErrRemoteBucketOffline) Error() string
type ErrRemoteMetadataMismatch ¶ added in v1.3.24
type ErrRemoteMetadataMismatch struct {
// contains filtered or unexported fields
}
assorted aistore errors
func (*ErrRemoteMetadataMismatch) Error ¶ added in v1.3.24
func (e *ErrRemoteMetadataMismatch) Error() string
type ErrStreamTerminated ¶ added in v1.3.16
type ErrStreamTerminated struct {
// contains filtered or unexported fields
}
assorted aistore errors
func NewErrStreamTerminated ¶ added in v1.3.16
func NewErrStreamTerminated(stream string, err error, reason, detail string) *ErrStreamTerminated
func (*ErrStreamTerminated) Error ¶ added in v1.3.16
func (e *ErrStreamTerminated) Error() string
func (*ErrStreamTerminated) Unwrap ¶ added in v1.3.16
func (e *ErrStreamTerminated) Unwrap() (err error)
type ErrTooManyRequests ¶ added in v1.3.28
type ErrTooManyRequests struct {
// contains filtered or unexported fields
}
assorted aistore errors
func NewErrTooManyRequests ¶ added in v1.3.28
func NewErrTooManyRequests(err error, status int) *ErrTooManyRequests
func (*ErrTooManyRequests) Error ¶ added in v1.3.28
func (e *ErrTooManyRequests) Error() string
type ErrUnsupp ¶
type ErrUnsupp struct {
// contains filtered or unexported fields
}
assorted aistore errors
func NewErrUnsupp ¶
func NewErrUnsuppErr ¶ added in v1.3.25
type ErrWarning ¶ added in v1.3.24
type ErrWarning struct {
// contains filtered or unexported fields
}
assorted aistore errors
func NewErrWarning ¶ added in v1.3.24
func NewErrWarning(what string) *ErrWarning
func (*ErrWarning) Error ¶ added in v1.3.24
func (e *ErrWarning) Error() string
type ErrXactNotFound ¶ added in v1.3.16
type ErrXactNotFound struct {
// contains filtered or unexported fields
}
assorted aistore errors
func NewErrXactNotFoundError ¶
func NewErrXactNotFoundError(cause string) *ErrXactNotFound
func (*ErrXactNotFound) Error ¶ added in v1.3.16
func (e *ErrXactNotFound) Error() string
type ErrXactTgtInMaint ¶ added in v1.3.18
type ErrXactTgtInMaint struct {
// contains filtered or unexported fields
}
assorted aistore errors
func NewErrXactTgtInMaint ¶ added in v1.3.18
func NewErrXactTgtInMaint(xaction, tname string) *ErrXactTgtInMaint
func (*ErrXactTgtInMaint) Error ¶ added in v1.3.18
func (e *ErrXactTgtInMaint) Error() string
type ErrXactUsePrev ¶ added in v1.3.18
type ErrXactUsePrev struct {
// contains filtered or unexported fields
}
assorted aistore errors
func NewErrXactUsePrev ¶ added in v1.3.18
func NewErrXactUsePrev(xaction string) *ErrXactUsePrev
func (*ErrXactUsePrev) Error ¶ added in v1.3.18
func (e *ErrXactUsePrev) Error() string
type ExtraProps ¶
type ExtraProps struct {
HTTP ExtraPropsHTTP `json:"http,omitempty" list:"omitempty"`
HDFS ExtraPropsHDFS `json:"hdfs,omitempty" list:"omitempty"` // NOTE: obsolete; rm with meta-version
AWS ExtraPropsAWS `json:"aws,omitempty" list:"omitempty"`
}
func (*ExtraProps) ValidateAsProps ¶
func (c *ExtraProps) ValidateAsProps(arg ...any) error
type ExtraPropsAWS ¶
type ExtraPropsAWS struct {
CloudRegion string `json:"cloud_region,omitempty"`
// from https://github.com/aws/aws-sdk-go/blob/main/aws/config.go:
// - "An optional endpoint URL (hostname only or fully qualified URI)
// that overrides the default generated endpoint."
Endpoint string `json:"endpoint,omitempty"`
// from https://github.com/aws/aws-sdk-go/blob/main/aws/session/session.go:
// - "Overrides the config profile the Session should be created from. If not
// set the value of the environment variable will be loaded (AWS_PROFILE,
// or AWS_DEFAULT_PROFILE if the Shared Config is enabled)."
Profile string `json:"profile,omitempty"`
// Amazon S3: 1000
// - https://docs.aws.amazon.com/cli/latest/userguide/cli-usage-pagination.html#cli-usage-pagination-serverside
// vs OpenStack Swift: 10,000
// - https://docs.openstack.org/swift/latest/api/pagination.html
MaxPageSize int64 `json:"max_pagesize,omitempty"`
// Multipart upload size threshold must be greater or equal 5MB
// - https://pkg.go.dev/github.com/aws/aws-sdk-go-v2/feature/s3/manager
// - for the AIS default, see `DefaultPartSize` in ais/s3/const
// - NOTE: the threshold is, effectively, one of the **performance tunables**
MultiPartSize cos.SizeIEC `json:"multipart_size,omitempty"`
}
type ExtraPropsAWSToSet ¶ added in v1.3.21
type ExtraPropsHDFS ¶
type ExtraPropsHDFS struct {
// Reference directory.
RefDirectory string `json:"ref_directory,omitempty"`
}
type ExtraPropsHDFSToSet ¶ added in v1.3.21
type ExtraPropsHDFSToSet struct {
RefDirectory *string `json:"ref_directory"`
}
type ExtraPropsHTTP ¶
type ExtraPropsHTTP struct {
// Original URL prior to hashing.
OrigURLBck string `json:"original_url,omitempty" list:"readonly"`
}
type ExtraPropsHTTPToSet ¶ added in v1.3.21
type ExtraPropsHTTPToSet struct {
OrigURLBck *string `json:"original_url"`
}
type ExtraToSet ¶ added in v1.3.21
type ExtraToSet struct {
AWS *ExtraPropsAWSToSet `json:"aws"`
HTTP *ExtraPropsHTTPToSet `json:"http"`
HDFS *ExtraPropsHDFSToSet `json:"hdfs"` // ditto
}
type FSHCConf ¶
type FSHCConf struct {
TestFileCount int `json:"test_files"` // number of files to read/write
// critical and unexpected errors detected during FSHC run;
// exceeding the limit "triggers" FSHC that may, in turn, disable the corresponding mountpath
HardErrs int `json:"error_limit"`
// - maximum number of I/O errors during the last `IOErrTime` interval;
// - the number does not include network error (e.g., connection reset by peer)
// and errors returned by remote backends;
// - exceeding this limit is also an FSHC-trggering event; subsequently,
// if FSHC confirms the problem it will disable the mountpath (see above)
IOErrs int `json:"io_err_limit,omitempty"`
// time interval (in seconds) to accumulate soft errors;
// the total number by the end of the interval must not exceed `IOErrs` (above)
IOErrTime cos.Duration `json:"io_err_time,omitempty"`
// whether FSHC is enabled (note: disabling FSHC is _not_ recommended)
Enabled bool `json:"enabled"`
}
global configuration
type FSHCConfToSet ¶ added in v1.3.21
type FSHCConfToSet struct {
TestFileCount *int `json:"test_files,omitempty"`
HardErrs *int `json:"error_limit,omitempty"`
IOErrs *int `json:"io_err_limit,omitempty"`
IOErrTime *cos.Duration `json:"io_err_time,omitempty"`
Enabled *bool `json:"enabled,omitempty"`
}
global configuration
type FSPConf ¶
ais node: fspaths (a.k.a. mountpaths) and their respective (optional) labels
func (*FSPConf) MarshalJSON ¶
func (*FSPConf) UnmarshalJSON ¶
type HTTPBckObj ¶
type HTTPBckObj struct {
Bck Bck
ObjName string
OrigURLBck string // HTTP URL of the bucket (object name excluded)
}
Represents the AIS bucket, object and URL associated with a HTTP resource
func NewHTTPObj ¶
func NewHTTPObj(u *url.URL) *HTTPBckObj
func NewHTTPObjPath ¶
func NewHTTPObjPath(rawURL string) (*HTTPBckObj, error)
type HTTPConf ¶
type HTTPConf struct {
Proto string `json:"-"` // http or https (set depending on `UseHTTPS`)
Certificate string `json:"server_crt"` // HTTPS: X.509 certificate
CertKey string `json:"server_key"` // HTTPS: X.509 key
ServerNameTLS string `json:"domain_tls"` // #6410
ClientCA string `json:"client_ca_tls"` // #6410
// clients "idle" config: intra-cluster and backend
IdleConnTimeout cos.Duration `json:"idle_conn_time"`
MaxIdleConnsPerHost int `json:"idle_conns_per_host"`
MaxIdleConns int `json:"idle_conns"`
// cont-d
ClientAuthTLS int `json:"client_auth_tls"` // #6410 tls.ClientAuthType enum
WriteBufferSize int `json:"write_buffer_size"` // http.Transport.WriteBufferSize; zero defaults to 4KB
ReadBufferSize int `json:"read_buffer_size"` // http.Transport.ReadBufferSize; ditto
UseHTTPS bool `json:"use_https"` // use HTTPS
SkipVerifyCrt bool `json:"skip_verify"` // skip X.509 cert verification (used with self-signed certs)
Chunked bool `json:"chunked_transfer"` // (https://tools.ietf.org/html/rfc7230#page-36; not used since 02/23)
}
global configuration
type HTTPConfToSet ¶ added in v1.3.21
type HTTPConfToSet struct {
Certificate *string `json:"server_crt,omitempty"`
CertKey *string `json:"server_key,omitempty"`
ServerNameTLS *string `json:"domain_tls,omitempty"`
ClientCA *string `json:"client_ca_tls,omitempty"`
// added v3.26
IdleConnTimeout *cos.Duration `json:"idle_conn_time,omitempty"`
MaxIdleConnsPerHost *int `json:"idle_conns_per_host,omitempty"`
MaxIdleConns *int `json:"idle_conns,omitempty"`
// cont-d
WriteBufferSize *int `json:"write_buffer_size,omitempty" list:"readonly"`
ReadBufferSize *int `json:"read_buffer_size,omitempty" list:"readonly"`
ClientAuthTLS *int `json:"client_auth_tls,omitempty"`
UseHTTPS *bool `json:"use_https,omitempty"`
SkipVerifyCrt *bool `json:"skip_verify,omitempty"`
Chunked *bool `json:"chunked_transfer,omitempty"`
}
global configuration
type HreqArgs ¶
type HreqArgs struct {
BodyR io.ReadCloser
Header http.Header // request headers
Query url.Values // query, e.g. ?a=x&b=y&c=z
RawQuery string // raw query
Method string
Base string // base URL, e.g. http://xyz.abc
Path string // path URL, e.g. /x/y/z
Body []byte
}
usage 1: initialize and fill out HTTP request. usage 2: intra-cluster control-plane (except streams) usage 3: PUT and APPEND API BodyR optimizes-out allocations - if non-nil and implements `io.Closer`, will always be closed by `client.Do`
func (*HreqArgs) Req ¶
NOTE: unlike standard http.NewRequest (above) - this method returns context-less request - it also does not assign req.GetBody - it is the caller's responsibility to assign one when and if required
func (*HreqArgs) ReqDeprecated ¶ added in v1.3.26
TODO: remaining usage - `api` package (remove when time permits)
type IterField ¶
type IterField interface {
Value() any // returns the value
String() string // string representation of the value
SetValue(v any, force ...bool) error // `force` ignores `tagReadonly` (to be used with caution!)
}
Represents a single named field
type IterOpts ¶
type IterOpts struct {
// Skip fields based on allowed tag
Allowed string
// Visits all the fields, not only the leaves.
VisitAll bool
// Read-only walk is true by default (compare with `UpdateFieldValue`)
// Note that `tagOmitempty` is limited to read-only - has no effect when `OnlyRead == false`.
OnlyRead bool
}
type KeepaliveConf ¶
type KeepaliveConf struct {
Proxy KeepaliveTrackerConf `json:"proxy"` // how proxy tracks target keepalives
Target KeepaliveTrackerConf `json:"target"` // how target tracks primary proxies keepalives
NumRetries int `json:"num_retries"` // default: `kaNumRetries`
RetryFactor uint8 `json:"retry_factor"`
}
keepalive
func (*KeepaliveConf) Validate ¶
func (c *KeepaliveConf) Validate() error
type KeepaliveConfToSet ¶ added in v1.3.21
type KeepaliveConfToSet struct {
Proxy *KeepaliveTrackerConfToSet `json:"proxy,omitempty"`
Target *KeepaliveTrackerConfToSet `json:"target,omitempty"`
NumRetries *int `json:"num_retries,omitempty"`
RetryFactor *uint8 `json:"retry_factor,omitempty"`
}
global configuration
type KeepaliveTrackerConf ¶
type KeepaliveTrackerConf struct {
Name string `json:"name"` // "heartbeat"
Interval cos.Duration `json:"interval"` // keepalive interval
Factor uint8 `json:"factor"` // only average
}
global configuration
type KeepaliveTrackerConfToSet ¶ added in v1.3.21
type KeepaliveTrackerConfToSet struct {
Interval *cos.Duration `json:"interval,omitempty"`
Name *string `json:"name,omitempty" list:"readonly"`
Factor *uint8 `json:"factor,omitempty"`
}
global configuration
type L4Conf ¶
type L4Conf struct {
Proto string `json:"proto"` // tcp, udp
SndRcvBufSize int `json:"sndrcv_buf_size"` // SO_RCVBUF and SO_SNDBUF
}
global configuration
type LRUConf ¶
type LRUConf struct {
// DontEvictTimeStr denotes the period of time during which eviction of an object
// is forbidden [atime, atime + DontEvictTime]
DontEvictTime cos.Duration `json:"dont_evict_time"`
// CapacityUpdTimeStr denotes the frequency at which AIStore updates local capacity utilization
CapacityUpdTime cos.Duration `json:"capacity_upd_time"`
// Enabled: LRU will only run when set to true
Enabled bool `json:"enabled"`
}
global configuration
type LRUConfToSet ¶ added in v1.3.21
type LRUConfToSet struct {
DontEvictTime *cos.Duration `json:"dont_evict_time,omitempty"`
CapacityUpdTime *cos.Duration `json:"capacity_upd_time,omitempty"`
Enabled *bool `json:"enabled,omitempty"`
}
global configuration
type LocalConfig ¶
type LocalConfig struct {
FSP FSPConf `json:"fspaths"`
ConfigDir string `json:"confdir"`
LogDir string `json:"log_dir"`
TestFSP TestFSPConf `json:"test_fspaths"`
HostNet LocalNetConfig `json:"host_net"`
}
local config
func (*LocalConfig) AddPath ¶
func (c *LocalConfig) AddPath(mpath string)
func (*LocalConfig) DelPath ¶
func (c *LocalConfig) DelPath(mpath string)
func (*LocalConfig) JspOpts ¶
func (*LocalConfig) JspOpts() jsp.Options
func (*LocalConfig) TestingEnv ¶
func (c *LocalConfig) TestingEnv() bool
type LocalNetConfig ¶
type LocalNetConfig struct {
Hostname string `json:"hostname"`
HostnameIntraControl string `json:"hostname_intra_control"`
HostnameIntraData string `json:"hostname_intra_data"`
Port int `json:"port,string"` // listening port
PortIntraControl int `json:"port_intra_control,string"` // --/-- for intra-cluster control
PortIntraData int `json:"port_intra_data,string"` // --/-- for intra-cluster data
// omit
UseIntraControl bool `json:"-"`
UseIntraData bool `json:"-"`
}
ais node: (local) network config
func (*LocalNetConfig) Validate ¶
func (c *LocalNetConfig) Validate(contextConfig *Config) error
type LogConf ¶
type LogConf struct {
Level cos.LogLevel `json:"level"` // log level (aka verbosity)
MaxSize cos.SizeIEC `json:"max_size"` // exceeding this size triggers log rotation
MaxTotal cos.SizeIEC `json:"max_total"` // (sum individual log sizes); exceeding this number triggers cleanup
FlushTime cos.Duration `json:"flush_time"` // log flush interval
StatsTime cos.Duration `json:"stats_time"` // (not used)
ToStderr bool `json:"to_stderr"` // Log only to stderr instead of files.
}
global configuration
type LogConfToSet ¶ added in v1.3.21
type LogConfToSet struct {
Level *cos.LogLevel `json:"level,omitempty"`
ToStderr *bool `json:"to_stderr,omitempty"`
MaxSize *cos.SizeIEC `json:"max_size,omitempty"`
MaxTotal *cos.SizeIEC `json:"max_total,omitempty"`
FlushTime *cos.Duration `json:"flush_time,omitempty"`
StatsTime *cos.Duration `json:"stats_time,omitempty"`
}
global configuration
type LsoEnt ¶ added in v1.3.23
type LsoEnt struct {
Name string `json:"name" msg:"n"` // object name
Checksum string `json:"checksum,omitempty" msg:"cs,omitempty"` // checksum
Atime string `json:"atime,omitempty" msg:"a,omitempty"` // last access time; formatted as ListObjsMsg.TimeFormat
Version string `json:"version,omitempty" msg:"v,omitempty"` // e.g., GCP int64 generation, AWS version (string), etc.
Location string `json:"location,omitempty" msg:"t,omitempty"` // [tnode:mountpath]
Custom string `json:"custom-md,omitempty" msg:"m,omitempty"` // custom metadata: ETag, MD5, CRC, user-defined ...
Size int64 `json:"size,string,omitempty" msg:"s,omitempty"` // size in bytes
Copies int16 `json:"copies,omitempty" msg:"c,omitempty"` // ## copies (NOTE: for non-replicated object copies == 1)
Flags uint16 `json:"flags,omitempty" msg:"f,omitempty"` // enum { EntryIsCached, EntryIsDir, EntryInArch, ...}
}
a single entry in LsoRes.Entries (below); contains list-objects results for the corresponding (listed) object or archived file; `Flags` is a bit field where `EntryStatusBits` bits [0-4] are reserved for object status (all statuses are mutually exclusive)
func (*LsoEnt) CopyWithProps ¶ added in v1.3.23
func (*LsoEnt) IsAnyFlagSet ¶ added in v1.3.26
func (*LsoEnt) IsPresent ¶ added in v1.3.23
flags: - see above "LsoEntry Flags" enum - keeping IsPresent for client convenience, with Set/IsAnyFlag covering for the rest - terms "cached", "present" and "in-cluster" - are interchangeable
type LsoEntries ¶
type LsoEntries []*LsoEnt
func (*LsoEntries) DecodeMsg ¶ added in v1.3.22
func (z *LsoEntries) DecodeMsg(dc *msgp.Reader) (err error)
DecodeMsg implements msgp.Decodable
func (LsoEntries) EncodeMsg ¶ added in v1.3.22
func (z LsoEntries) EncodeMsg(en *msgp.Writer) (err error)
EncodeMsg implements msgp.Encodable
func (LsoEntries) Msgsize ¶ added in v1.3.22
func (z LsoEntries) Msgsize() (s int)
Msgsize returns an upper bound estimate of the number of bytes occupied by the serialized message
type LsoRes ¶ added in v1.3.23
type LsoRes struct {
UUID string `json:"uuid"`
ContinuationToken string `json:"continuation_token"`
Entries LsoEntries `json:"entries"`
Flags uint32 `json:"flags"`
}
`api.ListObjects` and `Backend.ListObjects` results
type MemsysConf ¶
type MemsysConf struct {
MinFree cos.SizeIEC `json:"min_free"`
DefaultBufSize cos.SizeIEC `json:"default_buf"`
SizeToGC cos.SizeIEC `json:"to_gc"`
HousekeepTime cos.Duration `json:"hk_time"`
MinPctTotal int `json:"min_pct_total"`
MinPctFree int `json:"min_pct_free"`
}
global configuration
func (*MemsysConf) Validate ¶
func (c *MemsysConf) Validate() (err error)
type MemsysConfToSet ¶ added in v1.3.21
type MemsysConfToSet struct {
MinFree *cos.SizeIEC `json:"min_free,omitempty"`
DefaultBufSize *cos.SizeIEC `json:"default_buf,omitempty"`
SizeToGC *cos.SizeIEC `json:"to_gc,omitempty"`
HousekeepTime *cos.Duration `json:"hk_time,omitempty"`
MinPctTotal *int `json:"min_pct_total,omitempty"`
MinPctFree *int `json:"min_pct_free,omitempty"`
}
global configuration
type MirrorConf ¶
type MirrorConf struct {
Copies int64 `json:"copies"` // num copies
Burst int `json:"burst_buffer"` // xaction channel (buffer) size
Enabled bool `json:"enabled"` // enabled (to generate copies)
}
global configuration
func (*MirrorConf) String ¶
func (c *MirrorConf) String() string
func (*MirrorConf) Validate ¶
func (c *MirrorConf) Validate() error
func (*MirrorConf) ValidateAsProps ¶
func (c *MirrorConf) ValidateAsProps(...any) error
type MirrorConfToSet ¶ added in v1.3.21
type MirrorConfToSet struct {
Copies *int64 `json:"copies,omitempty"`
Burst *int `json:"burst_buffer,omitempty"`
Enabled *bool `json:"enabled,omitempty"`
}
global configuration
type NetConfToSet ¶ added in v1.3.21
type NetConfToSet struct {
HTTP *HTTPConfToSet `json:"http,omitempty"`
}
global configuration
type Ns ¶
type Ns struct {
// UUID of other remote AIS cluster (for now only used for AIS). Note
// that we can have different namespaces which refer to same UUID (cluster).
// This means that in a sense UUID is a parent of the actual namespace.
UUID string `json:"uuid" yaml:"uuid"`
// Name uniquely identifies a namespace under the same UUID (which may
// be empty) and is used in building FQN for the objects.
Name string `json:"name" yaml:"name"`
}
Ns (or Namespace) adds additional layer for scoping the data under the same provider. It allows to have same dataset and bucket names under different namespaces what allows for easy data manipulation without affecting data in different namespaces.
func ParseNsUname ¶
Parses [@uuid][#namespace]. It does a little bit more than just parsing a string from `Uname` so that logic can be reused in different places.
func (Ns) IsAnyRemote ¶
type OWT ¶
type OWT int
Object Write Transaction (OWT) controls certain aspects of creating new objects in the cluster. In particular, OwtGet* group below simultaneously specifies cold-GET variations (that all involve reading from a remote backend) and the associated locking (that will always reflect a tradeoff between consistency and parallelism) NOTE: enum entries' order is important
const ( // PUT and PUT-like transactions OwtPut OWT = iota // PUT OwtPromote // promote target-accessible files and directories OwtArchive // multi-obj arch OwtTransform // ETL OwtCopy // copy and move objects within cluster OwtRebalance // NOTE: must be the last in PUT* group // // GET and friends // OwtGetTryLock // if !try-lock(exclusive) { return error }; read from remote; ... OwtGetLock // lock(exclusive); read from remote; ... OwtGet // GET (with upgrading read-lock in the local-write path) OwtGetPrefetchLock // (used for maximum parallelism when prefetching) // // None of the above // OwtNone )
type ObjAttrs ¶
type ObjAttrs struct {
Cksum *cos.Cksum `json:"checksum,omitempty"` // object checksum (cloned)
CustomMD cos.StrKVs `json:"custom-md,omitempty"` // custom metadata: ETag, MD5, CRC, user-defined ...
Ver *string `json:"version,omitempty"` // object version
Atime int64 `json:"atime,omitempty"` // access time (nanoseconds since UNIX epoch)
Size int64 `json:"size,omitempty"` // object size (bytes)
}
see also apc.HdrObjAtime et al. @ api/apc/const.go (and note that naming must be consistent)
func (*ObjAttrs) CheckEq ¶ added in v1.3.24
local <=> remote equality in the context of cold-GET and download. This function decides whether we need to go ahead and re-read the object from its remote location.
Other than a "binary" size and version checks, rest logic goes as follows: objects are considered equal if they have a) the same version and at least one matching checksum, or b) the same remote "source" and at least one matching checksum, or c) two matching checksums. (See also note below.)
Note that ETag, checksum, or version mismatch leads to immediate return with error specifying the exact cause.
Note version comparison may fail even when the objects are identical, content-wise: same size, ETag, and checksums may still "co-exist" with different versions.
TODO: count == 1 with matching checksum being xxhash - must be configurable :NOTE
func (*ObjAttrs) CopyVersion ¶ added in v1.3.24
func (*ObjAttrs) DelStdCustom ¶ added in v1.3.26
func (oa *ObjAttrs) DelStdCustom()
func (*ObjAttrs) FromHeader ¶
NOTE: returning checksum separately for subsequent validation
func (*ObjAttrs) GetCustomKey ¶
func (*ObjAttrs) GetCustomMD ¶
func (*ObjAttrs) SetCustomKey ¶
func (*ObjAttrs) SetCustomMD ¶
func (*ObjAttrs) SetVersion ¶ added in v1.3.24
func (*ObjAttrs) VersionPtr ¶ added in v1.3.24
type ObjectProps ¶
type ObjectProps struct {
Bck Bck `json:"bucket"`
ObjAttrs
Name string `json:"name"`
Location string `json:"location"` // see also `GetPropsLocation`
Mirror struct {
Paths []string `json:"paths,omitempty"`
Copies int `json:"copies,omitempty"`
} `json:"mirror"`
EC struct {
Generation int64 `json:"generation"`
DataSlices int `json:"data"`
ParitySlices int `json:"parity"`
IsECCopy bool `json:"replicated"`
} `json:"ec"`
Present bool `json:"present"`
}
object properties NOTE: embeds system `ObjAttrs` that in turn includes custom user-defined NOTE: compare with `apc.LsoMsg`
type ParseURIOpts ¶
type PeriodConf ¶
type PeriodConf struct {
StatsTime cos.Duration `json:"stats_time"` // collect and publish stats; other house-keeping
RetrySyncTime cos.Duration `json:"retry_sync_time"` // metasync retry
NotifTime cos.Duration `json:"notif_time"` // (IC notifications)
}
NOTE: StatsTime is one important timer - a pulse
func (*PeriodConf) Validate ¶
func (c *PeriodConf) Validate() error
type PeriodConfToSet ¶ added in v1.3.21
type PeriodConfToSet struct {
StatsTime *cos.Duration `json:"stats_time,omitempty"`
RetrySyncTime *cos.Duration `json:"retry_sync_time,omitempty"`
NotifTime *cos.Duration `json:"notif_time,omitempty"`
}
global configuration
type PropsValidator ¶
type ProxyConf ¶
type ProxyConf struct {
PrimaryURL string `json:"primary_url"`
OriginalURL string `json:"original_url"`
DiscoveryURL string `json:"discovery_url"`
NonElectable bool `json:"non_electable"` // NOTE: deprecated, not used
}
global configuration
type ProxyConfToSet ¶ added in v1.3.21
type ProxyConfToSet struct {
PrimaryURL *string `json:"primary_url,omitempty"`
OriginalURL *string `json:"original_url,omitempty"`
DiscoveryURL *string `json:"discovery_url,omitempty"`
}
global configuration
type QueryBcks ¶
type QueryBcks Bck
func (*QueryBcks) AddToQuery ¶
func (*QueryBcks) IsBucket ¶
QueryBcks is a Bck that _can_ have an empty Name. (TODO: extend to support prefix and regex.)
func (*QueryBcks) IsRemoteAIS ¶
type RateLimitBase ¶ added in v1.3.28
type RateLimitBase struct {
// optional per-operation MaxTokens override - a space-separated key:value list, e.g.:
// - "put:3500"
// - "get:5000 delete:1000"
// optional; case-insensitive
Verbs string `json:"per_op_max_tokens,omitempty"`
//
// mandatory parameters; for default values and min/max ranges, see RateLimitConf.Validate
//
Interval cos.Duration `json:"interval"`
MaxTokens int `json:"max_tokens"`
Enabled bool `json:"enabled"`
}
global config that can be used to manage: * adaptive rate limit vis-à-vis Cloud backend * rate limiting (bursty) user workloads on the front
type RateLimitBaseToSet ¶ added in v1.3.28
type RateLimitBaseToSet struct {
Verbs *string `json:"per_op_max_tokens,omitempty"`
Interval *cos.Duration `json:"interval,omitempty"`
MaxTokens *int `json:"max_tokens,omitempty"`
Enabled *bool `json:"enabled,omitempty"`
}
global config that can be used to manage: * adaptive rate limit vis-à-vis Cloud backend * rate limiting (bursty) user workloads on the front
type RateLimitConf ¶ added in v1.3.28
global config that can be used to manage: * adaptive rate limit vis-à-vis Cloud backend * rate limiting (bursty) user workloads on the front
func (*RateLimitConf) Validate ¶ added in v1.3.28
func (c *RateLimitConf) Validate() error
func (*RateLimitConf) ValidateAsProps ¶ added in v1.3.28
func (c *RateLimitConf) ValidateAsProps(...any) error
type RateLimitConfToSet ¶ added in v1.3.28
type RateLimitConfToSet struct {
Backend *AdaptiveToSet `json:"backend,omitempty"`
Frontend *BurstyToSet `json:"frontend,omitempty"`
}
global config that can be used to manage: * adaptive rate limit vis-à-vis Cloud backend * rate limiting (bursty) user workloads on the front
type RebalanceConf ¶
type RebalanceConf struct {
XactConf
DestRetryTime cos.Duration `json:"dest_retry_time"` // max wait for ACKs & neighbors to complete
Enabled bool `json:"enabled"` // true=auto-rebalance | manual rebalancing
}
global configuration
func (*RebalanceConf) String ¶
func (c *RebalanceConf) String() string
func (*RebalanceConf) Validate ¶
func (c *RebalanceConf) Validate() error
type RebalanceConfToSet ¶ added in v1.3.21
type RebalanceConfToSet struct {
XactConfToSet
DestRetryTime *cos.Duration `json:"dest_retry_time,omitempty"`
Enabled *bool `json:"enabled,omitempty"`
}
global configuration
type ResilverConf ¶
type ResilverConf struct {
Enabled bool `json:"enabled"` // true=auto-resilver | manual resilvering
}
global configuration
func (*ResilverConf) String ¶
func (c *ResilverConf) String() string
func (*ResilverConf) Validate ¶
func (*ResilverConf) Validate() error
type ResilverConfToSet ¶ added in v1.3.21
type ResilverConfToSet struct {
Enabled *bool `json:"enabled,omitempty"`
}
global configuration
type RetryArgs ¶
type RetryArgs struct {
Call func() (int, error)
IsFatal func(error) bool
Action string
Caller string
SoftErr int // How many retries on ConnectionRefused or ConnectionReset error.
HardErr int // How many retries on any other error.
Sleep time.Duration
Verbosity int // Verbosity level for logging.
BackOff bool // If true, retries will wait progressively longer between attempts (backoff).
IsClient bool // True when client side (with no access to cluster config timeouts)
}
type SpaceConf ¶
type SpaceConf struct {
// Storage Cleanup watermark: used capacity (%) that triggers cleanup
// (deleted objects and buckets, extra copies, etc.)
CleanupWM int64 `json:"cleanupwm"`
// LowWM: used capacity low-watermark (% of total local storage capacity)
LowWM int64 `json:"lowwm"`
// HighWM: used capacity high-watermark (% of total local storage capacity)
// - LRU starts evicting objects when the currently used capacity (used-cap) gets above HighWM
// - and keeps evicting objects until the used-cap gets below LowWM
// - while self-throttling itself in accordance with target utilization
HighWM int64 `json:"highwm"`
// Out-of-Space: if exceeded, the target starts failing new PUTs and keeps
// failing them until its local used-cap gets back below HighWM (see above)
OOS int64 `json:"out_of_space"`
}
global configuration
func (*SpaceConf) ValidateAsProps ¶
type SpaceConfToSet ¶ added in v1.3.21
type SpaceConfToSet struct {
CleanupWM *int64 `json:"cleanupwm,omitempty"`
LowWM *int64 `json:"lowwm,omitempty"`
HighWM *int64 `json:"highwm,omitempty"`
OOS *int64 `json:"out_of_space,omitempty"`
}
global configuration
type TestFSPConf ¶
type TestFSPConf struct {
Root string `json:"root"`
Count int `json:"count"`
Instance int `json:"instance"`
}
local config
func (*TestFSPConf) Validate ¶
func (c *TestFSPConf) Validate(contextConfig *Config) (err error)
validate root and (NOTE: testing only) generate and fill-in counted FSP.Paths
func (*TestFSPConf) ValidateMpath ¶
func (c *TestFSPConf) ValidateMpath(p string) (err error)
type TimeoutConf ¶
type TimeoutConf struct {
CplaneOperation cos.Duration `json:"cplane_operation"` // read-mostly via global cmn.Rom.CplaneOperation
MaxKeepalive cos.Duration `json:"max_keepalive"` // ditto, cmn.Rom.MaxKeepalive - see below
MaxHostBusy cos.Duration `json:"max_host_busy"` // 2-phase transactions and more
Startup cos.Duration `json:"startup_time"` // primary wait for joins at (primary's) startup; indirectly, cluster startup
JoinAtStartup cos.Duration `json:"join_startup_time"` // (join cluster at startup) timeout; (2 * Startup) when zero
SendFile cos.Duration `json:"send_file_time"` // large file or blob and/or slow network
// intra-cluster EC streams; default SharedStreamsDflt; never timeout when negative
EcStreams cos.Duration `json:"ec_streams_time,omitempty"`
// object metadata timeout; for training apps an approx. duration of 2 (two) epochs
ObjectMD cos.Duration `json:"object_md,omitempty"`
// prior to returning 409 conflict (ie., cmn.ErrBusy)
ColdGetConflict cos.Duration `json:"cold_get_conflict,omitempty"`
}
maximum intra-cluster latencies (in the increasing order)
func (*TimeoutConf) Validate ¶
func (c *TimeoutConf) Validate() error
type TimeoutConfToSet ¶ added in v1.3.21
type TimeoutConfToSet struct {
CplaneOperation *cos.Duration `json:"cplane_operation,omitempty"`
MaxKeepalive *cos.Duration `json:"max_keepalive,omitempty"`
MaxHostBusy *cos.Duration `json:"max_host_busy,omitempty"`
Startup *cos.Duration `json:"startup_time,omitempty"`
JoinAtStartup *cos.Duration `json:"join_startup_time,omitempty"`
SendFile *cos.Duration `json:"send_file_time,omitempty"`
EcStreams *cos.Duration `json:"ec_streams_time,omitempty"`
ObjectMD *cos.Duration `json:"object_md,omitempty"`
ColdGetConflict *cos.Duration `json:"cold_get_conflict,omitempty"`
}
global configuration
type TraceExporterAuthConf ¶ added in v1.3.26
type TraceExporterAuthConf struct {
TokenHeader string `json:"token_header"` // header used to pass exporter auth token
TokenFile string `json:"token_file"` // filepath from where auth token can be obtained
}
global configuration
func (TraceExporterAuthConf) IsEnabled ¶ added in v1.3.26
func (tac TraceExporterAuthConf) IsEnabled() bool
only with `oteltracing` build tag
type TraceExporterAuthConfToSet ¶ added in v1.3.26
type TraceExporterAuthConfToSet struct {
TokenHeader *string `json:"token_header,omitempty"` // header used to pass exporter auth token
TokenFile *string `json:"token_file,omitempty"` // filepath from where auth token can be obtained
}
global configuration
type TracingConf ¶ added in v1.3.26
type TracingConf struct {
ExporterEndpoint string `json:"exporter_endpoint"` // gRPC exporter endpoint
ExporterAuth TraceExporterAuthConf `json:"exporter_auth,omitempty"` // exporter auth config
ServiceNamePrefix string `json:"service_name_prefix"` // service name prefix used by trace exporter
ExtraAttributes map[string]string `json:"attributes,omitempty"` // any extra-attributes to be added to traces
// SamplerProbabilityStr is the percentage of traces to be sampled, expressed as a float64.
// It's stored as a string to avoid potential floating-point precision issues during json unmarshal.
// Valid values range from 0.0 to 1.0, where 1.0 means 100% sampling.
SamplerProbabilityStr string `json:"sampler_probability,omitempty"`
Enabled bool `json:"enabled"`
SkipVerify bool `json:"skip_verify"` // allow insecure exporter gRPC connection
SamplerProbability float64 `json:"-"`
}
TracingConf defines the configuration used for the OpenTelemetry (OTEL) trace exporter. It includes settings for enabling tracing, sampling ratio, exporter endpoint, and other parameters necessary for distributed tracing in AIStore.
func (*TracingConf) Validate ¶ added in v1.3.26
func (c *TracingConf) Validate() error
type TracingConfToSet ¶ added in v1.3.26
type TracingConfToSet struct {
ExporterEndpoint *string `json:"exporter_endpoint,omitempty"` // gRPC exporter endpoint
ExporterAuth *TraceExporterAuthConfToSet `json:"exporter_auth,omitempty"` // exporter auth config
ServiceNamePrefix *string `json:"service_name_prefix,omitempty"` // service name used by trace exporter
ExtraAttributes map[string]string `json:"attributes,omitempty"` // any extra-attributes to be added to traces
SamplerProbabilityStr *string `json:"sampler_probability,omitempty"` // percentage of traces to be sampled
Enabled *bool `json:"enabled,omitempty"`
SkipVerify *bool `json:"skip_verify,omitempty"` // allow insecure exporter gRPC connection
}
NOTE: Updating TracingConfig requires restart.
type TransportArgs ¶
type TransportArgs struct {
DialTimeout time.Duration
Timeout time.Duration
IdleConnTimeout time.Duration
IdleConnsPerHost int
MaxIdleConns int
SndRcvBufSize int
WriteBufferSize int
ReadBufferSize int
UseHTTPProxyEnv bool
LowLatencyToS bool
}
assorted http(s) client options
func (*TransportArgs) ServerControl ¶ added in v1.3.26
func (args *TransportArgs) ServerControl(c syscall.RawConn)
type TransportConf ¶
type TransportConf struct {
MaxHeaderSize int `json:"max_header"` // max transport header buffer (default=4K)
Burst int `json:"burst_buffer"` // num sends with no back pressure; see also AIS_STREAM_BURST_NUM
// two no-new-transmissions durations:
// * IdleTeardown: sender terminates the connection (to reestablish it upon the very first/next PDU)
// * QuiesceTime: safe to terminate or transition to the next (in re: rebalance) stage
IdleTeardown cos.Duration `json:"idle_teardown"`
QuiesceTime cos.Duration `json:"quiescent"`
// lz4
// max uncompressed block size, one of [64K, 256K(*), 1M, 4M]
// fastcompression.blogspot.com/2013/04/lz4-streaming-format-final.html
LZ4BlockMaxSize cos.SizeIEC `json:"lz4_block"`
LZ4FrameChecksum bool `json:"lz4_frame_checksum"`
}
global configuration
func (*TransportConf) Validate ¶
func (c *TransportConf) Validate() (err error)
NOTE: uncompressed block sizes - the enum currently supported by the github.com/pierrec/lz4
type TransportConfToSet ¶ added in v1.3.21
type TransportConfToSet struct {
MaxHeaderSize *int `json:"max_header,omitempty"`
Burst *int `json:"burst_buffer,omitempty"`
IdleTeardown *cos.Duration `json:"idle_teardown,omitempty"`
QuiesceTime *cos.Duration `json:"quiescent,omitempty"`
LZ4BlockMaxSize *cos.SizeIEC `json:"lz4_block,omitempty"`
LZ4FrameChecksum *bool `json:"lz4_frame_checksum,omitempty"`
}
global configuration
type VersionConf ¶
type VersionConf struct {
// Determines if versioning is enabled
Enabled bool `json:"enabled"`
// Validate remote version and, possibly, update in-cluster ("cached") copy.
// Scenarios include (but are not limited to):
// - warm GET
// - prefetch bucket (*)
// - prefetch multiple objects (see api/multiobj.go)
// - copy bucket
// Applies to Cloud and remote AIS buckets - generally, buckets that have remote backends
// that in turn provide some form of object versioning.
// (*) Xactions (ie., jobs) that read-access multiple objects (e.g., prefetch, copy-bucket)
// may support operation-scope option to synchronize remote content (to aistore) - the option
// not requiring changing bucket configuration.
// See also:
// - apc.QparamLatestVer, apc.PrefetchMsg, apc.CopyBckMsg
ValidateWarmGet bool `json:"validate_warm_get"`
// A stronger variant of the above that in addition entails:
// - deleting in-cluster object if its remote ("cached") counterpart does not exist
// See also: apc.QparamSync, apc.CopyBckMsg
Sync bool `json:"synchronize"`
}
global configuration
func (*VersionConf) String ¶
func (c *VersionConf) String() string
func (*VersionConf) Validate ¶
func (c *VersionConf) Validate() error
type VersionConfToSet ¶ added in v1.3.21
type VersionConfToSet struct {
Enabled *bool `json:"enabled,omitempty"`
ValidateWarmGet *bool `json:"validate_warm_get,omitempty"`
Sync *bool `json:"synchronize,omitempty"`
}
global configuration
type WritePolicyConf ¶
type WritePolicyConf struct {
Data apc.WritePolicy `json:"data"`
MD apc.WritePolicy `json:"md"`
}
global configuration
func (*WritePolicyConf) Validate ¶
func (c *WritePolicyConf) Validate() (err error)
func (*WritePolicyConf) ValidateAsProps ¶
func (c *WritePolicyConf) ValidateAsProps(...any) error
type WritePolicyConfToSet ¶ added in v1.3.21
type WritePolicyConfToSet struct {
Data *apc.WritePolicy `json:"data,omitempty" list:"readonly"` // NOTE: NIY
MD *apc.WritePolicy `json:"md,omitempty"`
}
global configuration
type XactConf ¶ added in v1.3.28
type XactConf struct {
Compression string `json:"compression"` // enum { CompressAlways, ... } in api/apc/compression.go
SbundleMult int `json:"bundle_multiplier"` // stream-bundle multiplier: num streams to destination
Burst int `json:"burst_buffer"` // xaction channel (buffer) size
}
generic xaction --
type XactConfToSet ¶ added in v1.3.28
type XactConfToSet struct {
Compression *string `json:"compression,omitempty"`
SbundleMult *int `json:"bundle_multiplier,omitempty"`
Burst *int `json:"burst_buffer,omitempty"`
}
global configuration
Source Files
¶
Directories
¶
| Path | Synopsis |
|---|---|
|
Package archive: write, read, copy, append, list primitives across all supported formats
|
Package archive: write, read, copy, append, list primitives across all supported formats |
|
Package atomic provides simple wrappers around numerics to enforce atomic access.
|
Package atomic provides simple wrappers around numerics to enforce atomic access. |
|
Package certloader loads and reloads X.509 certs.
|
Package certloader loads and reloads X.509 certs. |
|
Package cos provides common low-level types and utilities for all aistore projects.
|
Package cos provides common low-level types and utilities for all aistore projects. |
|
Package memsys provides memory management and slab/SGL allocation with io.Reader and io.Writer interfaces on top of scatter-gather lists of reusable buffers.
|
Package memsys provides memory management and slab/SGL allocation with io.Reader and io.Writer interfaces on top of scatter-gather lists of reusable buffers. |
|
Package feat: global runtime-configurable feature flags
|
Package feat: global runtime-configurable feature flags |
|
Package fname contains filename constants and common system directories
|
Package fname contains filename constants and common system directories |
|
Package jsp (JSON persistence) provides utilities to store and load arbitrary JSON-encoded structures with optional checksumming and compression.
|
Package jsp (JSON persistence) provides utilities to store and load arbitrary JSON-encoded structures with optional checksumming and compression. |
|
Package k8s: initialization, client, and misc.
|
Package k8s: initialization, client, and misc. |
|
Package kvdb provides a local key/value database server for AIS.
|
Package kvdb provides a local key/value database server for AIS. |
|
Package mono provides low-level monotonic time
|
Package mono provides low-level monotonic time |
|
Package nlog - aistore logger, provides buffering, timestamping, writing, and flushing/syncing/rotating
|
Package nlog - aistore logger, provides buffering, timestamping, writing, and flushing/syncing/rotating |
|
Package oom: serialized goroutine to go ahead and run GC _now_
|
Package oom: serialized goroutine to go ahead and run GC _now_ |
|
Package prob implements fully features dynamic probabilistic filter.
|
Package prob implements fully features dynamic probabilistic filter. |
|
Package xoshiro256 implements the xoshiro256** RNG no-copyright
|
Package xoshiro256 implements the xoshiro256** RNG no-copyright |