objstorage

package
v1.0.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Oct 6, 2023 License: BSD-3-Clause Imports: 16 Imported by: 7

Documentation

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func IsNotExistError

func IsNotExistError(err error) bool

IsNotExistError indicates whether the error is known to report that a file or directory does not exist.

func TestingCheckMaxReadahead

func TestingCheckMaxReadahead(rh ReadHandle) bool

TestingCheckMaxReadahead returns true if the ReadHandle has switched to OS-level read-ahead.

Types

type CreateOptions

type CreateOptions struct {
	// PreferSharedStorage causes the object to be created on shared storage if
	// the provider has shared storage configured.
	PreferSharedStorage bool
}

CreateOptions contains optional arguments for Create.

type CreatorID

type CreatorID = sharedobjcat.CreatorID

CreatorID identifies the DB instance that originally created a shared object. This ID is incorporated in backing object names. Must be non-zero.

type NoopReadHandle

type NoopReadHandle struct {
	io.ReaderAt
}

NoopReadHandle can be used by Readable implementations that don't support read-ahead.

func MakeNoopReadHandle

func MakeNoopReadHandle(r io.ReaderAt) NoopReadHandle

MakeNoopReadHandle initializes a NoopReadHandle.

func (*NoopReadHandle) Close

func (*NoopReadHandle) Close() error

Close is part of the ReadHandle interface.

func (*NoopReadHandle) MaxReadahead

func (*NoopReadHandle) MaxReadahead()

MaxReadahead is part of the ReadHandle interface.

func (*NoopReadHandle) RecordCacheHit

func (*NoopReadHandle) RecordCacheHit(offset, size int64)

RecordCacheHit is part of the ReadHandle interface.

type ObjectMetadata

type ObjectMetadata struct {
	FileNum  base.FileNum
	FileType base.FileType

	// The fields below are only set if the object is on shared storage.
	Shared struct {
		// CreatorID identifies the DB instance that originally created the object.
		CreatorID CreatorID
		// CreatorFileNum is the identifier for the object within the context of the
		// DB instance that originally created the object.
		CreatorFileNum base.FileNum
	}
}

ObjectMetadata contains the metadata required to be able to access an object.

func (*ObjectMetadata) IsShared

func (meta *ObjectMetadata) IsShared() bool

IsShared returns true if the object is on shared storage.

func (*ObjectMetadata) SharedObjectBacking

func (meta *ObjectMetadata) SharedObjectBacking() (SharedObjectBacking, error)

SharedObjectBacking encodes the shared object metadata.

type PreallocatedReadHandle

type PreallocatedReadHandle struct {
	// contains filtered or unexported fields
}

PreallocatedReadHandle is used to avoid an allocation in NewReadHandle; see UsePreallocatedReadHandle.

func (*PreallocatedReadHandle) Close

func (rh *PreallocatedReadHandle) Close() error

Close is part of the objstorage.ReadHandle interface.

func (*PreallocatedReadHandle) MaxReadahead

func (rh *PreallocatedReadHandle) MaxReadahead()

MaxReadahead is part of the objstorage.ReadHandle interface.

func (*PreallocatedReadHandle) ReadAt

func (rh *PreallocatedReadHandle) ReadAt(p []byte, offset int64) (n int, err error)

ReadAt is part of the objstorage.ReadHandle interface.

func (*PreallocatedReadHandle) RecordCacheHit

func (rh *PreallocatedReadHandle) RecordCacheHit(offset, size int64)

RecordCacheHit is part of the objstorage.ReadHandle interface.

type Provider

type Provider struct {
	// contains filtered or unexported fields
}

Provider is a singleton object used to access and manage objects.

An object is conceptually like a large immutable file. The main use of objects is for storing sstables; in the future it could also be used for blob storage.

The Provider can only manage objects that it knows about - either objects created by the provider, or existing objects the Provider was informed about via AddObjects.

Objects are currently backed by a vfs.File.

func Open

func Open(settings Settings) (p *Provider, _ error)

Open creates the Provider.

func (*Provider) AttachSharedObjects

func (p *Provider) AttachSharedObjects(objs []SharedObjectToAttach) ([]ObjectMetadata, error)

AttachSharedObjects registers existing shared objects with this provider.

func (*Provider) Close

func (p *Provider) Close() error

Close the provider.

func (*Provider) Create

func (p *Provider) Create(
	fileType base.FileType, fileNum base.FileNum, opts CreateOptions,
) (w Writable, meta ObjectMetadata, err error)

Create creates a new object and opens it for writing.

The object is not guaranteed to be durable (accessible in case of crashes) until Sync is called.

func (*Provider) LinkOrCopyFromLocal

func (p *Provider) LinkOrCopyFromLocal(
	srcFS vfs.FS, srcFilePath string, dstFileType base.FileType, dstFileNum base.FileNum,
) (ObjectMetadata, error)

LinkOrCopyFromLocal creates a new object that is either a copy of a given local file or a hard link (if the new object is created on the same FS, and if the FS supports it).

The object is not guaranteed to be durable (accessible in case of crashes) until Sync is called.

func (*Provider) List

func (p *Provider) List() []ObjectMetadata

List returns the objects currently known to the provider. Does not perform any I/O.

func (*Provider) Lookup

func (p *Provider) Lookup(fileType base.FileType, fileNum base.FileNum) (ObjectMetadata, error)

Lookup returns the metadata of an object that is already known to the Provider. Does not perform any I/O.

func (*Provider) OpenForReading

func (p *Provider) OpenForReading(fileType base.FileType, fileNum base.FileNum) (Readable, error)

OpenForReading opens an existing object.

func (*Provider) OpenForReadingMustExist

func (p *Provider) OpenForReadingMustExist(
	fileType base.FileType, fileNum base.FileNum,
) (Readable, error)

OpenForReadingMustExist is a variant of OpenForReading which causes a fatal error if the file does not exist. The fatal error message contains information helpful for debugging.

func (*Provider) Path

func (p *Provider) Path(meta ObjectMetadata) string

Path returns an internal, implementation-dependent path for the object. It is meant to be used for informational purposes (like logging).

func (*Provider) Remove

func (p *Provider) Remove(fileType base.FileType, fileNum base.FileNum) error

Remove removes an object.

The object is not guaranteed to be durably removed until Sync is called.

func (*Provider) SetCreatorID

func (p *Provider) SetCreatorID(creatorID CreatorID) error

SetCreatorID sets the CreatorID which is needed in order to use shared objects. Shared object usage is disabled until this method is called the first time. Once set, the Creator ID is persisted and cannot change.

Cannot be called if shared storage is not configured for the provider.

func (*Provider) Size

func (p *Provider) Size(meta ObjectMetadata) (int64, error)

Size returns the size of the object.

func (*Provider) Sync

func (p *Provider) Sync() error

Sync flushes the metadata from creation or removal of objects since the last Sync.

type ReadHandle

type ReadHandle interface {
	io.ReaderAt
	io.Closer

	// MaxReadahead configures the implementation to expect large sequential
	// reads. Used to skip any initial read-ahead ramp-up.
	MaxReadahead()

	// RecordCacheHit informs the implementation that we were able to retrieve a
	// block from cache.
	RecordCacheHit(offset, size int64)
}

ReadHandle is used to perform reads that are related and might benefit from optimizations like read-ahead.

func UsePreallocatedReadHandle

func UsePreallocatedReadHandle(readable Readable, rh *PreallocatedReadHandle) ReadHandle

UsePreallocatedReadHandle is equivalent to calling readable.NewReadHandle() but uses the existing storage of a PreallocatedReadHandle when possible (currently this happens if we are reading from a local file). The returned handle still needs to be closed.

type Readable

type Readable interface {
	io.ReaderAt
	io.Closer

	// Size returns the size of the object.
	Size() int64

	// NewReadHandle creates a read handle for ReadAt requests that are related
	// and can benefit from optimizations like read-ahead.
	//
	// The ReadHandle must be closed before the Readable is closed.
	//
	// Multiple separate ReadHandles can be used.
	NewReadHandle() ReadHandle
}

Readable is the handle for an object that is open for reading.

type Settings

type Settings struct {
	Logger base.Logger

	// Local filesystem configuration.
	FS        vfs.FS
	FSDirName string

	// FSDirInitialListing is a listing of FSDirName at the time of calling Open.
	//
	// This is an optional optimization to avoid double listing on Open when the
	// higher layer already has a listing. When nil, we obtain the listing on
	// Open.
	FSDirInitialListing []string

	// Cleaner cleans obsolete files from the local filesystem.
	//
	// The default cleaner uses the DeleteCleaner.
	FSCleaner base.Cleaner

	// NoSyncOnClose decides whether the implementation will enforce a
	// close-time synchronization (e.g., fdatasync() or sync_file_range())
	// on files it writes to. Setting this to true removes the guarantee for a
	// sync on close. Some implementations can still issue a non-blocking sync.
	NoSyncOnClose bool

	// BytesPerSync enables periodic syncing of files in order to smooth out
	// writes to disk. This option does not provide any persistence guarantee, but
	// is used to avoid latency spikes if the OS automatically decides to write
	// out a large chunk of dirty filesystem buffers.
	BytesPerSync int

	// Fields here are set only if the provider is to support shared objects
	// (experimental).
	Shared struct {
		Storage shared.Storage
	}
}

Settings that must be specified when creating the Provider.

func DefaultSettings

func DefaultSettings(fs vfs.FS, dirName string) Settings

DefaultSettings initializes default settings (with no shared storage), suitable for tests and tools.

type SharedObjectBacking

type SharedObjectBacking []byte

SharedObjectBacking encodes the metadata necessary to incorporate a shared object into a different Pebble instance.

type SharedObjectToAttach

type SharedObjectToAttach struct {
	// FileNum is the file number that will be used to refer to this object (in
	// the context of this instance).
	FileNum  base.FileNum
	FileType base.FileType
	// Backing contains the metadata for the share dobject backing (normally
	// generated from a different instance).
	Backing SharedObjectBacking
}

SharedObjectToAttach contains the arguments needed to attach an existing shared object.

type Writable

type Writable interface {
	// Write writes len(p) bytes from p to the underlying object. The data is not
	// guaranteed to be durable until Finish is called.
	//
	// Note that Write *is* allowed to modify the slice passed in, whether
	// temporarily or permanently. Callers of Write need to take this into
	// account.
	Write(p []byte) error

	// Finish completes the object and makes the data durable.
	// No further calls are allowed after calling Finish.
	Finish() error

	// Abort gives up on finishing the object. There is no guarantee about whether
	// the object exists after calling Abort.
	// No further calls are allowed after calling Abort.
	Abort()
}

Writable is the handle for an object that is open for writing. Either Finish or Abort must be called.

func NewFileWritable

func NewFileWritable(file vfs.File) Writable

NewFileWritable returns a Writable that uses a file as underlying storage.

Directories

Path Synopsis

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL