Documentation
¶
Overview ¶
Package fstree implements a storage subsystem that saves objects as files in FS tree.
The main concept behind it is rather simple: each object is stored as a file in a directory. Given that handling many files in the same directory is usually problematic for file systems objects are being put into subdirectories by their IDs. This directory tree can have different FSTree.Depth and each component of the path (single directory name) is a DirNameLen number of ID bytes from its string representation. File name then is a leftover of object ID (after stripping FSTree.Depth bytes off of it) concatenated with container ID.
For example, an object with ID of WCdSV7F9TnDHFmbgKY7BNCbPs6g7meaVbh6DMNXbytB from Hh6qJ2Fa9WzSK7PESResS4mmuoZ2a47Z7F6ZvmR7AEHU container will be stored in W/C/d/S/V directory and a name of 7F9TnDHFmbgKY7BNCbPs6g7meaVbh6DMNXbytB.Hh6qJ2Fa9WzSK7PESResS4mmuoZ2a47Z7F6ZvmR7AEHU if the depth is 5. The overall structure may look like this (a part of it):
/W/ ├── 7 │ └── 2 │ └── F │ └── 6 │ └── 32vhigDXRPkSwaCTw3FtxWbKjoDGoDvVTJqxBb.J4SkhNifjANvYGr56vh4NbGBRCxvk8PT9YE5EgFSb7cc ├── C │ └── d │ └── S │ └── V │ └── 7F9TnDHFmbgKY7BNCbPs6g7meaVbh6DMNXbytB.Hh6qJ2Fa9WzSK7PESResS4mmuoZ2a47Z7F6ZvmR7AEHU
Binary file format can differ depending on the FSTree version. The basic format that was used from the beginning is storing serialized protobuf representation of the object as is. In this case file can be decoded into an object directly.
Version 0.44.0 of the node has introduced a new file format that is used for small files, the so-called "combined" one. It has a special prefix (0x7F) that can not collide with protobuf-encoded or ZSTD-compressed data. Files using this prefix can contain an unspecified number of objects (configured via WithCombinedCountLimit) that all follow the same serialization pattern:
- the first byte is magic 0x7F described above
- one byte version of the subsequent structure (currently zero only)
- 32-byte OID of the next object then
- 4-byte BE integer length of the next object
- followed by protobuf-encoded object data (of the length specified in the previous field)
Overall the structure is like this:
[0x7F 0x00 [OID A] [uint32 size]][object A][0x7F 0x00 [OID B] [uint32 size]][object B]...
Even though this file contains several objects it has hard links for all of them in the FS tree, so finding a file containing some object doesn't require any additional effort. Finding it in the file contents however requires reading the prefix described above, comparing the target OID and either skipping the object length specified there or reading it after the prefix.
Index ¶
- Constants
- type FSTree
- func (t *FSTree) CleanUpTmp() error
- func (t *FSTree) Close() error
- func (t *FSTree) Delete(addr oid.Address) error
- func (t *FSTree) Exists(addr oid.Address) (bool, error)
- func (t *FSTree) Get(addr oid.Address) (*object.Object, error)
- func (t *FSTree) GetBytes(addr oid.Address) ([]byte, error)
- func (t *FSTree) GetRangeStream(addr oid.Address, rng blobstor.PayloadRange, readHeader bool) (*object.Object, uint64, io.ReadCloser, error)
- func (t *FSTree) GetStream(addr oid.Address) (*object.Object, io.ReadSeekCloser, error)
- func (t *FSTree) Head(addr oid.Address) (*object.Object, error)
- func (t *FSTree) Init(id blobstor.ID) error
- func (t *FSTree) InitPut(addr oid.Address, headerLen uint64, payloadLen uint64, headerW io.WriterTo) (blobstor.PutStream, func(), error)
- func (t *FSTree) Iterate(objHandler func(addr oid.Address, data []byte) error, ...) error
- func (t *FSTree) IterateAddresses(f func(addr oid.Address) error, ignoreErrors bool) error
- func (t *FSTree) IterateSizes(f func(addr oid.Address, size uint64) error, ignoreErrors bool) error
- func (t *FSTree) Open(ro bool) error
- func (t *FSTree) Path() string
- func (t *FSTree) Put(addr oid.Address, data []byte) error
- func (t *FSTree) PutBatch(objs map[oid.Address][]byte) error
- func (t *FSTree) ReadHeader(addr oid.Address, buf []byte) (int, error)
- func (t *FSTree) ReadObject(addr oid.Address, buf []byte) (int, io.ReadCloser, error)
- func (t *FSTree) ReadObjectParts(buf []byte, addr oid.Address, rng blobstor.PayloadRange, ...) (int, io.ReadCloser, error)
- func (t *FSTree) ReadPayloadRange(addr oid.Address, off, ln uint64, hdrBuf []byte, ...) (io.ReadCloser, error)
- func (t *FSTree) ShardID() blobstor.ID
- func (*FSTree) Type() string
- type Info
- type Option
- func WithAllowDepthChange(allow bool) Option
- func WithCombinedCountLimit(limit int) Option
- func WithCombinedSizeLimit(size int) Option
- func WithCombinedSizeThreshold(size int) Option
- func WithCombinedWriteInterval(t time.Duration) Option
- func WithDepth(d uint64) Option
- func WithLogger(l *zap.Logger) Option
- func WithNoSync(noSync bool) Option
- func WithPath(p string) Option
- func WithPerm(p fs.FileMode) Option
- func WithReshapeProgressTracker(f func(string) ReshapeProgressTracker) Option
- func WithSubtype(st string) Option
- type ReshapeProgressTracker
Constants ¶
const ( // DirNameLen is how many bytes is used to group keys into directories. DirNameLen = 1 // in bytes // MaxDepth is maximum depth of nested directories. 58^8 is 128e12 of // directories, enough for a single FSTree. MaxDepth = 8 )
const SubtypeBlobstor = "blobstor"
const Type = "fstree"
Type is fstree storage type used in logs and configuration.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type FSTree ¶
type FSTree struct {
Info
Depth uint64
AllowDepthChange bool
// contains filtered or unexported fields
}
FSTree represents an object storage as a filesystem tree.
func (*FSTree) CleanUpTmp ¶ added in v0.44.0
CleanUpTmp removes all temporary files garbage.
func (*FSTree) Exists ¶
Exists returns the path to the file with object contents if it exists in the storage and an error otherwise.
func (*FSTree) GetBytes ¶ added in v0.41.0
GetBytes reads object from the FSTree by address into memory buffer in a canonical NeoFS binary format. Returns apistatus.ObjectNotFound if object is missing.
func (*FSTree) GetRangeStream ¶ added in v0.51.0
func (t *FSTree) GetRangeStream(addr oid.Address, rng blobstor.PayloadRange, readHeader bool) (*object.Object, uint64, io.ReadCloser, error)
GetRangeStream reads the requested payload range of the referenced object from t. It optionally returns the object header parsed from the same read. The stream must be finally closed by the caller.
If object is missing, GetRangeStream returns apistatus.ErrObjectNotFound.
If the range is out of payload bounds, GetRangeStream returns apistatus.ErrObjectOutOfRange.
func (*FSTree) GetStream ¶ added in v0.48.0
GetStream returns an object from the storage by address as a stream. It returns the object with header only and a seekable reader for the payload. The reader only supports forward seeking with io.SeekStart and io.SeekCurrent. Its initial position, and position 0 from io.SeekStart, are at the beginning of the payload, not at the beginning of the physical object file. Seeking from io.SeekEnd is not supported. On success, the reader is non-nil and must be closed; a nil reader is only returned with a non‑nil error.
func (*FSTree) Head ¶ added in v0.48.0
Head returns an object's header from the storage by address without reading the full payload.
func (*FSTree) InitPut ¶ added in v0.57.0
func (t *FSTree) InitPut(addr oid.Address, headerLen uint64, payloadLen uint64, headerW io.WriterTo) (blobstor.PutStream, func(), error)
InitPut opens write stream for object with given properties. If stream is successfully opened, InitPut writes header to it using headerW. It must write exactly headerLen bytes.
Resulting stream accepts payload passed to blobstor.PutStream.Write / blobstor.PutStream.WriteBuffers until blobstor.PutStream.Close call. It is caller's responsibility to ensure that the payload being written matches payloadLen parameter. Stream should not be used after any stream method error.
If the object is fully buffered, it is more efficient to use FSTree.Put.
InitPut never does a combined write. Use FSTree.Put instead if needed.
Returned function allows to rollback whole operation and free all allocated resources if any. It should not be called multiple times, after blobstor.PutStream.Close or failed blobstor.PutStream.Write / blobstor.PutStream.WriteBuffers.
Either blobstor.PutStream.Close or abort function must be finally called. All functions must not be called concurrently.
If the device runs out of space, InitPut or resulting stream calls return blobstor.ErrNoSpace.
func (*FSTree) Iterate ¶
func (t *FSTree) Iterate(objHandler func(addr oid.Address, data []byte) error, errorHandler func(addr oid.Address, err error) error) error
Iterate iterates over all stored objects.
func (*FSTree) IterateAddresses ¶ added in v0.44.1
IterateAddresses iterates over all objects stored in the underlying storage and passes their addresses into f. If f returns an error, IterateAddresses returns it and breaks. ignoreErrors allows to continue if internal errors happen.
func (*FSTree) IterateSizes ¶ added in v0.45.0
IterateSizes iterates over all objects stored in the underlying storage and passes their addresses and sizes into f. If f returns an error, IterateSizes returns it and breaks. ignoreErrors allows to continue if internal errors happen.
func (*FSTree) ReadHeader ¶ added in v0.52.0
ReadHeader reads first bytes of the referenced object's binary containing its full header from t into buf. Returns number of bytes read.
Read part may include payload prefix.
If object is missing, ReadHeader returns apistatus.ErrObjectNotFound.
Passed buf must have 2*objectwire.NonPayloadFieldsBufferLength bytes len at least.
func (*FSTree) ReadObject ¶ added in v0.52.0
ReadObject reads first bytes of the referenced object's binary containing its full header from t into buf. Returns number of bytes read and stream of remaining bytes. The stream must be finally closed by the caller.
Read part may include payload prefix.
If object is missing, ReadObject returns apistatus.ErrObjectNotFound.
Passed buf must have 2*objectwire.NonPayloadFieldsBufferLength bytes len at least.
func (*FSTree) ReadObjectParts ¶ added in v0.56.0
func (t *FSTree) ReadObjectParts(buf []byte, addr oid.Address, rng blobstor.PayloadRange, interceptHeaderBinaryFn func([]byte) error) (int, io.ReadCloser, error)
ReadObjectParts reads first bytes of the referenced object's binary containing its full header from t into buf. If optional interceptHeaderBinaryFn is specified, it's called with read header. On error, ReadObjectParts returns it immediately. ReadObjectParts also returns payload stream depending on range parameter. If kind is blobstor.PayloadRangeModeNone, full payload including field prefix is returned. Otherwise, stream contains requested range bytes only. The stream must be finally closed by the caller.
If object is missing, ReadObjectParts returns apistatus.ErrObjectNotFound.
If partial out-of-bounds range is requested, ReadObjectParts returns apistatus.ErrObjectOutOfRange.
Passed buf must have 2*objectwire.NonPayloadFieldsBufferLength bytes len at least.
func (*FSTree) ReadPayloadRange ¶ added in v0.53.0
func (t *FSTree) ReadPayloadRange(addr oid.Address, off, ln uint64, hdrBuf []byte, interceptHeaderBinaryFn func([]byte) error) (io.ReadCloser, error)
ReadPayloadRange is FSTree.ReadObject analogue for payload range reading. Zero range means full payload.
If given range is out of payload bounds, ReadPayloadRange returns apistatus.ErrObjectOutOfRange.
If interceptHeaderBinaryFn is specified, it's called instantly once header is read (never concurrently). If it returns an error, whole operation is aborted with this error.
type Info ¶
type Info struct {
// Permission bits of the root directory.
Permissions fs.FileMode
// Full path to the root directory.
RootPath string
}
Info groups the information about file storage.
type Option ¶ added in v0.32.0
type Option func(*FSTree)
func WithAllowDepthChange ¶ added in v0.57.0
WithAllowDepthChange permits changing the FSTree layout depth on initialization.
func WithCombinedCountLimit ¶ added in v0.44.0
func WithCombinedSizeLimit ¶ added in v0.44.0
func WithCombinedSizeThreshold ¶ added in v0.44.0
func WithCombinedWriteInterval ¶ added in v0.44.0
func WithLogger ¶ added in v0.53.0
func WithNoSync ¶ added in v0.36.0
func WithReshapeProgressTracker ¶ added in v0.57.0
func WithReshapeProgressTracker(f func(string) ReshapeProgressTracker) Option
WithReshapeProgressTracker sets the FSTree layout reshape progress tracker factory.
func WithSubtype ¶ added in v0.53.0
type ReshapeProgressTracker ¶ added in v0.57.0
type ReshapeProgressTracker interface {
SetReshapeProgress(float64)
}
ReshapeProgressTracker receives FSTree layout reshape progress percentage.