shard

package
v0.51.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Feb 4, 2026 License: GPL-3.0 Imports: 26 Imported by: 1

Documentation

Index

Constants

This section is empty.

Variables

View Source 
var ErrDegradedMode = logicerr.New("shard is in degraded mode")

ErrDegradedMode is returned when operation requiring metabase is executed in degraded mode.

View Source 
var ErrEndOfListing = meta.ErrEndOfListing

ErrEndOfListing is returned from object listing with cursor when storage can't return any more objects after provided cursor. Use nil cursor object to start listing again.

View Source 
var ErrInvalidMagic = logicerr.New("invalid magic")

ErrInvalidMagic is returned when dump format is invalid.

View Source 
var ErrLockObjectRemoval = meta.ErrLockObjectRemoval

ErrLockObjectRemoval is returned when inhume operation is being performed on lock object, and it is not a forced object removal.

View Source 
var ErrMetaWithNoObject = errors.New("got meta, but no object")

ErrMetaWithNoObject is returned when shard has metadata, but no object.

View Source 
var ErrMustBeReadOnly = logicerr.New("shard must be in read-only mode")
View Source 
var ErrReadOnlyMode = logicerr.New("shard is in read-only mode")

ErrReadOnlyMode is returned when it is impossible to apply operation that changes shard's memory due to the "read-only" shard's mode.

Functions

func IsErrNotFound added in v0.28.0 

func IsErrNotFound(err error) bool

IsErrNotFound checks if error returned by Shard Get/Head/GetRange method corresponds to missing object.

func IsErrObjectExpired added in v0.31.0 

func IsErrObjectExpired(err error) bool

IsErrObjectExpired checks if an error returned by Shard corresponds to expired object.

func IsErrOutOfRange added in v0.30.0 

func IsErrOutOfRange(err error) bool

IsErrOutOfRange checks if an error returned by Shard GetRange method corresponds to exceeding the object bounds.

func IsErrRemoved added in v0.28.0 

func IsErrRemoved(err error) bool

IsErrRemoved checks if error returned by Shard Exists/Get/Head/GetRange method corresponds to removed object.

Types

type ContainerPayments added in v0.50.0 

type ContainerPayments interface {
	// UnpaidSince must return an epoch starting from which, container is
	// considered as an unpaid one. Must return negative value if container
	// has been paid.
	UnpaidSince(cid.ID) (int64, error)
}

ContainerPayments defines containers' payment status checker.

type Cursor added in v0.27.0 

type Cursor = meta.Cursor

Cursor is a type for continuous object listing.

type DeletedLockCallback added in v0.30.0 

type DeletedLockCallback func([]oid.Address)

DeletedLockCallback is a callback handling list of deleted LOCK objects.

type Event added in v0.16.0 

type Event interface {
	// contains filtered or unexported methods
}

Event represents class of external events.

func EventNewEpoch added in v0.16.0 

func EventNewEpoch(e uint64) Event

EventNewEpoch returns new epoch event.

type ExpiredObjectsCallback added in v0.16.0 

type ExpiredObjectsCallback func([]oid.Address)

ExpiredObjectsCallback is a callback handling list of expired objects.

type ID 

type ID []byte

ID represents Shard identifier.

Each shard should have the unique ID within a single instance of local storage.

func NewIDFromBytes 

func NewIDFromBytes(v []byte) *ID

NewIDFromBytes constructs ID from byte slice.

func (ID) String 

func (id ID) String() string

type Info 

type Info struct {
	// Identifier of the shard.
	ID *ID

	// Shard mode.
	Mode mode.Mode

	// Information about the metabase.
	MetaBaseInfo meta.Info

	// Information about the storage.
	BlobStorInfo StorageInfo

	// Information about the Write Cache.
	WriteCacheInfo writecache.Info

	// ErrorCount contains amount of errors occurred in shard operations.
	ErrorCount uint32
}

Info groups the information about Shard.

type MetricsWriter added in v0.32.0 

type MetricsWriter interface {
	// SetObjectCounter must set object counter taking into account object type.
	SetObjectCounter(objectType string, v uint64)
	// AddToObjectCounter must update object counter taking into account object
	// type.
	// Negative parameter must decrease the counter.
	AddToObjectCounter(objectType string, delta int)
	// AddToContainerSize must add a value to the container size.
	// Value can be negative.
	AddToContainerSize(cnr string, value int64)
	// AddToPayloadSize must add a value to the payload size.
	// Value can be negative.
	AddToPayloadSize(value int64)
	// IncObjectCounter must increment shard's object counter taking into account
	// object type.
	IncObjectCounter(objectType string)
	// DecObjectCounter must decrement shard's object counter taking into account
	// object type.
	DecObjectCounter(objectType string)
	// SetShardID must set (update) the shard identifier that will be used in
	// metrics.
	SetShardID(id string)
	// SetReadonly must set shard readonly state.
	SetReadonly(readonly bool)
}

MetricsWriter is an interface that must store shard's metrics.

type ObjectStatus added in v0.39.0 

type ObjectStatus struct {
	Blob       StorageObjectStatus
	Metabase   meta.ObjectStatus
	Writecache writecache.ObjectStatus
	Errors     []error
}

ObjectStatus represents the status of an object in a storage system. It contains information about the object's status in various sub-components such as Blob storage, Metabase, and Writecache. Additionally, it includes a slice of errors that may have occurred at the object level.

type Option 

type Option func(*cfg)

Option represents Shard's constructor option.

func WithBlobstor added in v0.47.0 

func WithBlobstor(s common.Storage) Option

WithBlobstor provides storage.

func WithCompressObjects added in v0.47.0 

func WithCompressObjects(comp bool) Option

WithCompressObjects returns option to toggle compression of the stored objects.

If true, Zstandard algorithm is used for data compression.

If compressor (decompressor) creation failed, the uncompressed option will be used, and the error is recorded in the provided log.

func WithContainerPayments added in v0.50.0 

func WithContainerPayments(p ContainerPayments) Option

WithContainerPayments returns option to specify containers' payments status checher.

func WithDeletedLockCallback added in v0.30.0 

func WithDeletedLockCallback(v DeletedLockCallback) Option

WithDeletedLockCallback returns option to specify callback of the deleted LOCK objects handler.

func WithExpiredLocksCallback added in v0.28.0 

func WithExpiredLocksCallback(cb ExpiredObjectsCallback) Option

WithExpiredLocksCallback returns option to specify callback of the expired LOCK objects handler.

func WithExpiredObjectsCallback added in v0.16.0 

func WithExpiredObjectsCallback(cb ExpiredObjectsCallback) Option

WithExpiredObjectsCallback returns option to specify callback of the expired objects handler.

func WithGCRemoverSleepInterval added in v0.16.0 

func WithGCRemoverSleepInterval(dur time.Duration) Option

WithGCRemoverSleepInterval returns option to specify sleep interval between object remover executions.

func WithID 

func WithID(id *ID) Option

WithID returns option to set the default shard identifier.

func WithLogger 

func WithLogger(l *zap.Logger) Option

WithLogger returns option to set Shard's logger.

func WithMetaBaseOptions 

func WithMetaBaseOptions(opts ...meta.Option) Option

WithMetaBaseOptions returns option to set internal metabase options.

func WithMetricsWriter added in v0.32.0 

func WithMetricsWriter(v MetricsWriter) Option

WithMetricsWriter returns option to specify storage of the shard's metrics.

func WithMode added in v0.27.3 

func WithMode(v mode.Mode) Option

WithMode returns option to set shard's mode. Mode must be one of the predefined:

  • mode.ReadWrite;
  • mode.ReadOnly.

func WithRemoverBatchSize added in v0.16.0 

func WithRemoverBatchSize(sz int) Option

WithRemoverBatchSize returns option to set batch size of single removal operation.

func WithReportErrorFunc added in v0.36.0 

func WithReportErrorFunc(f func(selfID string, message string, err error)) Option

WithReportErrorFunc returns option to specify callback for handling storage-related errors in the background workers.

func WithResyncMetabase added in v0.44.0 

func WithResyncMetabase(v bool) Option

WithResyncMetabase returns option to set flag to refill the Metabase on Shard's initialization step.

func WithUncompressableContentTypes added in v0.47.0 

func WithUncompressableContentTypes(values []string) Option

WithUncompressableContentTypes returns option to disable decompression for specific content types as seen by object.AttributeContentType attribute.

func WithWriteCache 

func WithWriteCache(use bool) Option

WithWriteCache returns option to toggle write cache usage.

func WithWriteCacheOptions 

func WithWriteCacheOptions(opts ...writecache.Option) Option

WithWriteCacheOptions returns option to set internal write cache options.

type Shard 

type Shard struct {
	// contains filtered or unexported fields
}

Shard represents single shard of NeoFS Local Storage Engine.

func New 

func New(opts ...Option) *Shard

New creates, initializes and returns new Shard instance.

func (*Shard) Close 

func (s *Shard) Close() error

Close releases all Shard's components.

func (*Shard) CollectRawWithAttribute added in v0.51.0 

func (s *Shard) CollectRawWithAttribute(cnr cid.ID, attr string, val []byte) ([]oid.ID, error)

CollectRawWithAttribute looks up for raw objects with the given attribute and value.

func (*Shard) ContainerInfo added in v0.49.0 

func (s *Shard) ContainerInfo(cnr cid.ID) (meta.ContainerInfo, error)

func (*Shard) Delete 

func (s *Shard) Delete(addrs []oid.Address) error

Delete removes data from the shard's writeCache, metaBase and blobStor.

func (*Shard) DeleteContainer added in v0.40.0 

func (s *Shard) DeleteContainer(_ context.Context, cID cid.ID) error

DeleteContainer deletes any information related to the container including: - Metabase; - Blobstor; - Write-cache (if configured).

func (*Shard) Dump added in v0.27.5 

func (s *Shard) Dump(w io.Writer, ignoreErrors bool) (int, error)

Dump dumps all objects from the shard to a given stream.

Returns any error encountered and the number of objects written.

func (*Shard) DumpInfo 

func (s *Shard) DumpInfo() Info

DumpInfo returns information about the Shard.

func (*Shard) Exists 

func (s *Shard) Exists(addr oid.Address, ignoreExpiration bool) (bool, error)

Exists checks if object is presented in shard. ignoreExpiration flag allows to check for expired objects.

Returns any error encountered that does not allow to unambiguously determine the presence of an object.

Returns an error of type apistatus.ObjectAlreadyRemoved if object has been marked as removed. Returns the object.ErrObjectIsExpired if the object is presented but already expired.

If referenced object is a parent of some stored objects, Exists returns ierrors.ErrParentObject wrapping: - [*objectSDK.SplitInfoError] wrapping [objectSDK.SplitInfo] collected from stored parts; - iec.ErrParts if referenced object is EC.

func (*Shard) FlushWriteCache added in v0.33.0 

func (s *Shard) FlushWriteCache(ignoreErrors bool) error

FlushWriteCache flushes all data from the write-cache. If ignoreErrors is set will flush all objects it can irrespective of any errors.

func (*Shard) Get 

func (s *Shard) Get(addr oid.Address, skipMeta bool) (*object.Object, error)

Get reads an object from shard. skipMeta flag allows to fetch object from the blobstor directly.

Returns any error encountered that did not allow to completely read the object part.

Returns an error of type apistatus.ObjectNotFound if the requested object is missing in shard. Returns an error of type apistatus.ObjectAlreadyRemoved if the requested object has been marked as removed in shard. Returns the object.ErrObjectIsExpired if the object is presented but already expired.

If referenced object is a parent of some stored objects, Get returns ierrors.ErrParentObject wrapping: - *object.SplitInfoError wrapping [objectSDK.SplitInfo] collected from stored parts; - iec.ErrParts if referenced object is EC.

func (*Shard) GetBytes added in v0.41.0 

func (s *Shard) GetBytes(addr oid.Address) ([]byte, error)

GetBytes reads object from the Shard by address into memory buffer in a canonical NeoFS binary format. Returns apistatus.ObjectNotFound if object is missing.

func (*Shard) GetBytesWithMetadataLookup added in v0.41.0 

func (s *Shard) GetBytesWithMetadataLookup(addr oid.Address) ([]byte, error)

GetBytesWithMetadataLookup works similar to shard.GetBytes, but pre-checks object presence in the underlying metabase: if object cannot be accessed from the metabase, GetBytesWithMetadataLookup returns an error.

If referenced object is a parent of some stored objects, GetBytesWithMetadataLookup returns ierrors.ErrParentObject wrapping: - *object.SplitInfoError wrapping [objectSDK.SplitInfo] collected from stored parts; - iec.ErrParts if referenced object is EC.

func (*Shard) GetECPart added in v0.49.0 

func (s *Shard) GetECPart(cnr cid.ID, parent oid.ID, pi iec.PartInfo) (object.Object, io.ReadCloser, error)

GetECPart looks up for object that carries EC part produced within cnr for parent object and indexed by pi in the underlying metabase, checks its availability and reads it from the underlying BLOB storage. The result is a header and a payload stream that must be closed by caller after processing.

If the object is not EC part but of object.TypeTombstone, object.TypeLock or object.TypeLink type, GetECPart this object instead.

If object is found in the metabase but unreadable from the BLOB storage, GetECPart wraps ierrors.ObjectID with the object ID along with the failure cause.

If write-cache is enabled, GetECPart tries to get the object from it first.

If object has expired, GetECPart returns meta.ErrObjectIsExpired.

If object exists but tombstoned (e.g. via [Shard.Inhume] or stored tombstone object), GetECPart returns apistatus.ErrObjectAlreadyRemoved.

If object is marked as garbage (e.g. via Shard.MarkGarbage), GetECPart returns apistatus.ErrObjectNotFound.

If object is locked (e.g. via [Shard.Lock] or stored locker object), GetECPart ignores expiration, tombstone and garbage marks.

func (*Shard) GetECPartRange added in v0.50.0 

func (s *Shard) GetECPartRange(cnr cid.ID, parent oid.ID, pi iec.PartInfo, off, ln int64) (uint64, io.ReadCloser, error)

GetECPartRange looks up for object that carries EC part produced within cnr for parent object and indexed by pi in the underlying metabase, checks its availability and reads it from the underlying BLOB storage. Returns full payload len. If zero, GetECPartRange returns (0, nil, nil). Otherwise, range-cut payload stream is also returned. In this case, the stream must be finally closed by the caller.

If object is missing, GetECPartRange returns apistatus.ErrObjectNotFound.

If object is found in the metabase but unreadable from the BLOB storage, GetECPartRange wraps ierrors.ObjectID with the object ID along with the failure cause.

If object has expired, GetECPartRange returns meta.ErrObjectIsExpired.

If object exists but tombstoned (e.g. via [Shard.Inhume] or stored tombstone object), GetECPartRange returns apistatus.ErrObjectAlreadyRemoved.

If object is marked as garbage (e.g. via Shard.MarkGarbage), GetECPartRange returns apistatus.ErrObjectNotFound.

If object is locked (e.g. via [Shard.Lock] or stored locker object), GetECPartRange ignores expiration, tombstone and garbage marks.

If the range is out of payload bounds, GetECPartRange returns apistatus.ErrObjectOutOfRange.

func (*Shard) GetMode added in v0.27.3 

func (s *Shard) GetMode() mode.Mode

GetMode returns mode of the shard.

func (*Shard) GetRangeStream added in v0.50.0 

func (s *Shard) GetRangeStream(cnr cid.ID, id oid.ID, off, ln int64) (uint64, io.ReadCloser, error)

GetRangeStream reads specified range of payload of the referenced object from the underlying BLOB storage. First return is object's full payload length. If zero, GetRangeStream returns (0, nil, nil). Otherwise, range-cut payload stream is also returned. The stream must be finally closed by the caller.

If write-cache is enabled, GetRangeStream looks up there first. Its errors are logged and never returned.

If object is missing, GetECPartRange returns apistatus.ErrObjectNotFound.

If the range is out of payload bounds, GetRangeStream returns apistatus.ErrObjectOutOfRange.

func (*Shard) GetRangeStreamWithMetadataLookup added in v0.51.0 

func (s *Shard) GetRangeStreamWithMetadataLookup(addr oid.Address, off, ln uint64, skipMeta bool) (io.ReadCloser, error)

GetRangeStreamWithMetadataLookup reads payload range of the referenced object from s. Both zero off and ln mean full payload. The stream must be finally closed by the caller.

If object is missing, GetRangeStreamWithMetadataLookup returns apistatus.ErrObjectNotFound.

If the range is out of payload bounds, GetRangeStreamWithMetadataLookup returns apistatus.ErrObjectOutOfRange.

If object exists in underlying metabase but cannot be read from underlying storage, GetRangeStreamWithMetadataLookup returns ErrMetaWithNoObject along with storage error.

If skipMeta flag is set, GetRangeStreamWithMetadataLookup attempts to access object bypassing metabase.

func (*Shard) GetStream added in v0.49.0 

func (s *Shard) GetStream(addr oid.Address, skipMeta bool) (*object.Object, io.ReadCloser, error)

GetStream reads an object from shard as a stream. skipMeta flag allows to fetch object from the blobstor directly.

Returns the object header and a reader for the payload. On success, the reader is non-nil and must be closed; a nil reader is only returned with a non‑nil error.

Returns any error encountered that did not allow to completely read the object part. Returns an error of type apistatus.ObjectNotFound if the requested object is missing in shard. Returns an error of type apistatus.ObjectAlreadyRemoved if the requested object has been marked as removed in shard. Returns the object.ErrObjectIsExpired if the object is present but already expired.

If referenced object is a parent of some stored objects, GetStream returns ierrors.ErrParentObject wrapping: - *object.SplitInfoError wrapping [objectSDK.SplitInfo] collected from stored parts; - iec.ErrParts if referenced object is EC.

func (*Shard) Head 

func (s *Shard) Head(addr oid.Address, raw bool) (*object.Object, error)

Head reads header of the object from the shard. raw flag controls split object handling, if unset, then virtual object header is returned, otherwise SplitInfo of this object.

Returns any error encountered.

Returns an error of type apistatus.ObjectNotFound if object is missing in Shard. Returns an error of type apistatus.ObjectAlreadyRemoved if the requested object has been marked as removed in shard. Returns the object.ErrObjectIsExpired if the object is presented but already expired.

func (*Shard) HeadECPart added in v0.50.0 

func (s *Shard) HeadECPart(cnr cid.ID, parent oid.ID, pi iec.PartInfo) (object.Object, error)

HeadECPart is similar to Shard.GetECPart but returns only the header.

func (*Shard) ID 

func (s *Shard) ID() *ID

ID returns Shard identifier.

func (*Shard) InhumeContainer added in v0.40.0 

func (s *Shard) InhumeContainer(cID cid.ID) error

InhumeContainer marks every object in a container as removed. Any further [StorageEngine.Get] calls will return apistatus.ObjectNotFound errors. There is no any LOCKs, forced GC marks and any relations checks, every object that belongs to a provided container will be marked as a removed one.

func (*Shard) Init 

func (s *Shard) Init() error

Init initializes all Shard's components.

func (*Shard) IsLocked added in v0.36.0 

func (s *Shard) IsLocked(addr oid.Address) (bool, error)

IsLocked checks object locking relation of the provided object. Not found object is considered as not locked. Requires healthy metabase, returns ErrDegradedMode otherwise.

func (*Shard) List 

func (s *Shard) List() ([]oid.Address, error)

List returns all objects physically stored in the Shard.

func (*Shard) ListContainers added in v0.15.0 

func (s *Shard) ListContainers() ([]cid.ID, error)

ListContainers enumerates all containers known to this shard.

func (*Shard) ListWithCursor added in v0.27.0 

func (s *Shard) ListWithCursor(count int, cursor *Cursor, attrs ...string) ([]objectcore.AddressWithAttributes, *Cursor, error)

ListWithCursor lists physical objects available in shard starting from cursor. Includes regular, tombstone and storage group objects. Does not include inhumed objects. Use cursor value from response for consecutive requests.

Optional attrs specifies attributes to include in the result. If object does not have requested attribute, corresponding element in the result is empty.

Returns ErrEndOfListing if there are no more objects to return or count parameter set to zero.

func (*Shard) MarkGarbage added in v0.44.0 

func (s *Shard) MarkGarbage(addrs ...oid.Address) error

MarkGarbage marks objects to be physically removed from shard. It's a forced mark that overrides any restrictions imposed on object deletion (to be used by control service and other manual intervention cases). Otherwise similar to [Shard.Inhume], but doesn't need a tombstone.

func (*Shard) NeedsCompression added in v0.47.0 

func (s *Shard) NeedsCompression(obj *object.Object) bool

NeedsCompression returns true if the object should be compressed. For an object to be compressed 2 conditions must hold: 1. Compression is enabled in settings. 2. Object MIME Content-Type is allowed for compression.

func (*Shard) NotificationChannel added in v0.33.0 

func (s *Shard) NotificationChannel() chan<- Event

NotificationChannel returns channel for shard events.

func (*Shard) ObjectStatus added in v0.39.0 

func (s *Shard) ObjectStatus(address oid.Address) (ObjectStatus, error)

ObjectStatus returns the status of the object in the Shard. It contains status of the object in Blob storage, Metabase and Writecache.

func (*Shard) Open 

func (s *Shard) Open() error

Open opens all Shard's components.

func (*Shard) Put 

func (s *Shard) Put(obj *object.Object, objBin []byte) error

Put saves the object in shard. objBin parameter is optional and used to optimize out object marshaling.

Returns any error encountered that did not allow to completely save the object.

Returns ErrReadOnlyMode error if shard is in "read-only" mode.

Returns apistatus.ErrObjectAlreadyRemoved if obj is of object.TypeLock type and there is an object of object.TypeTombstone type associated with the same target.

func (*Shard) Reload added in v0.34.0 

func (s *Shard) Reload(opts ...Option) error

Reload reloads configuration portions that are necessary. If a config option is invalid, it logs an error and returns nil. If there was a problem with applying new configuration, an error is returned.

func (*Shard) Restore added in v0.27.5 

func (s *Shard) Restore(r io.Reader, ignoreErrors bool) (int, int, error)

Restore restores objects from the dump prepared by Dump. If ignoreErrors is set any restore errors are ignored (corrupted objects are just skipped).

Returns two numbers: successful and failed restored objects, as well as any error encountered.

func (*Shard) ReviveObject added in v0.44.0 

func (s *Shard) ReviveObject(addr oid.Address) (meta.ReviveStatus, error)

ReviveObject try to revive object in Shard, by remove records from graveyard and garbage.

Returns meta.ReviveStatus of object and error.

func (*Shard) Search added in v0.45.0 

func (s *Shard) Search(cnr cid.ID, fs []objectcore.SearchFilter, attrs []string, cursor *objectcore.SearchCursor, count uint16) ([]client.SearchResultItem, []byte, error)

Search performs Search op on the underlying metabase if it is not disabled.

func (*Shard) Select 

func (s *Shard) Select(cnr cid.ID, filters object.SearchFilters) ([]oid.Address, error)

Select selects the objects from shard that match select parameters.

Returns any error encountered that did not allow to completely select the objects.

Returns object.ErrInvalidSearchQuery if specified query is invalid.

func (*Shard) SetMode 

func (s *Shard) SetMode(m mode.Mode) error

SetMode sets mode of the shard.

Returns any error encountered that did not allow setting shard mode.

func (*Shard) UpdateID added in v0.27.7 

func (s *Shard) UpdateID() (err error)

UpdateID reads shard ID saved in the metabase and updates it if it is missing.

type StorageInfo added in v0.47.0 

type StorageInfo struct {
	Type string
	Path string
}

StorageInfo contains information about storage component.

type StorageObjectStatus added in v0.47.0 

type StorageObjectStatus struct {
	Type  string
	Path  string
	Error error
}

StorageObjectStatus represents the status of the object in the storage, containing the type and path of the storage and an error if it occurred.

Source Files

View all Source files

Directories

Path Synopsis

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.