Documentation
¶
Index ¶
- func IsNotExistError(err error) bool
- func TestingCheckMaxReadahead(rh ReadHandle) bool
- type CreateOptions
- type CreatorID
- type NoopReadHandle
- type ObjectMetadata
- type PreallocatedReadHandle
- type Provider
- func (p *Provider) AttachSharedObjects(objs []SharedObjectToAttach) ([]ObjectMetadata, error)
- func (p *Provider) Close() error
- func (p *Provider) Create(fileType base.FileType, fileNum base.FileNum, opts CreateOptions) (w Writable, meta ObjectMetadata, err error)
- func (p *Provider) LinkOrCopyFromLocal(srcFS vfs.FS, srcFilePath string, dstFileType base.FileType, ...) (ObjectMetadata, error)
- func (p *Provider) List() []ObjectMetadata
- func (p *Provider) Lookup(fileType base.FileType, fileNum base.FileNum) (ObjectMetadata, error)
- func (p *Provider) OpenForReading(fileType base.FileType, fileNum base.FileNum) (Readable, error)
- func (p *Provider) OpenForReadingMustExist(fileType base.FileType, fileNum base.FileNum) (Readable, error)
- func (p *Provider) Path(meta ObjectMetadata) string
- func (p *Provider) Remove(fileType base.FileType, fileNum base.FileNum) error
- func (p *Provider) SetCreatorID(creatorID CreatorID) error
- func (p *Provider) Size(meta ObjectMetadata) (int64, error)
- func (p *Provider) Sync() error
- type ReadHandle
- type Readable
- type Settings
- type SharedObjectBacking
- type SharedObjectToAttach
- type Writable
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func IsNotExistError ¶
IsNotExistError indicates whether the error is known to report that a file or directory does not exist.
func TestingCheckMaxReadahead ¶
func TestingCheckMaxReadahead(rh ReadHandle) bool
TestingCheckMaxReadahead returns true if the ReadHandle has switched to OS-level read-ahead.
Types ¶
type CreateOptions ¶
type CreateOptions struct {
// the provider has shared storage configured.
PreferSharedStorage bool
}
CreateOptions contains optional arguments for Create.
type CreatorID ¶
type CreatorID = sharedobjcat.CreatorID
CreatorID identifies the DB instance that originally created a shared object. This ID is incorporated in backing object names. Must be non-zero.
type NoopReadHandle ¶
NoopReadHandle can be used by Readable implementations that don't support read-ahead.
func MakeNoopReadHandle ¶
func MakeNoopReadHandle(r io.ReaderAt) NoopReadHandle
MakeNoopReadHandle initializes a NoopReadHandle.
func (*NoopReadHandle) Close ¶
func (*NoopReadHandle) Close() error
Close is part of the ReadHandle interface.
func (*NoopReadHandle) MaxReadahead ¶
func (*NoopReadHandle) MaxReadahead()
MaxReadahead is part of the ReadHandle interface.
func (*NoopReadHandle) RecordCacheHit ¶
func (*NoopReadHandle) RecordCacheHit(offset, size int64)
RecordCacheHit is part of the ReadHandle interface.
type ObjectMetadata ¶
type ObjectMetadata struct {
FileNum base.FileNum
FileType base.FileType
Shared struct {
// CreatorID identifies the DB instance that originally created the object.
CreatorID CreatorID
// CreatorFileNum is the identifier for the object within the context of the
// DB instance that originally created the object.
CreatorFileNum base.FileNum
}
}
ObjectMetadata contains the metadata required to be able to access an object.
func (*ObjectMetadata) IsShared ¶
func (meta *ObjectMetadata) IsShared() bool
IsShared returns true if the object is on shared storage.
func (*ObjectMetadata) SharedObjectBacking ¶
func (meta *ObjectMetadata) SharedObjectBacking() (SharedObjectBacking, error)
SharedObjectBacking encodes the shared object metadata.
type PreallocatedReadHandle ¶
type PreallocatedReadHandle struct {
// contains filtered or unexported fields
}
PreallocatedReadHandle is used to avoid an allocation in NewReadHandle; see UsePreallocatedReadHandle.
func (*PreallocatedReadHandle) Close ¶
func (rh *PreallocatedReadHandle) Close() error
Close is part of the objstorage.ReadHandle interface.
func (*PreallocatedReadHandle) MaxReadahead ¶
func (rh *PreallocatedReadHandle) MaxReadahead()
MaxReadahead is part of the objstorage.ReadHandle interface.
func (*PreallocatedReadHandle) RecordCacheHit ¶
func (rh *PreallocatedReadHandle) RecordCacheHit(offset, size int64)
RecordCacheHit is part of the objstorage.ReadHandle interface.
type Provider ¶
type Provider struct {
// contains filtered or unexported fields
}
Provider is a singleton object used to access and manage objects.
An object is conceptually like a large immutable file. The main use of objects is for storing sstables; in the future it could also be used for blob storage.
The Provider can only manage objects that it knows about - either objects created by the provider, or existing objects the Provider was informed about via AddObjects.
Objects are currently backed by a vfs.File.
func (*Provider) AttachSharedObjects ¶
func (p *Provider) AttachSharedObjects(objs []SharedObjectToAttach) ([]ObjectMetadata, error)
AttachSharedObjects registers existing shared objects with this provider.
func (*Provider) Create ¶
func (p *Provider) Create( fileType base.FileType, fileNum base.FileNum, opts CreateOptions, ) (w Writable, meta ObjectMetadata, err error)
Create creates a new object and opens it for writing.
The object is not guaranteed to be durable (accessible in case of crashes) until Sync is called.
func (*Provider) LinkOrCopyFromLocal ¶
func (p *Provider) LinkOrCopyFromLocal( srcFS vfs.FS, srcFilePath string, dstFileType base.FileType, dstFileNum base.FileNum, ) (ObjectMetadata, error)
LinkOrCopyFromLocal creates a new object that is either a copy of a given local file or a hard link (if the new object is created on the same FS, and if the FS supports it).
The object is not guaranteed to be durable (accessible in case of crashes) until Sync is called.
func (*Provider) List ¶
func (p *Provider) List() []ObjectMetadata
List returns the objects currently known to the provider. Does not perform any I/O.
func (*Provider) Lookup ¶
Lookup returns the metadata of an object that is already known to the Provider. Does not perform any I/O.
func (*Provider) OpenForReading ¶
OpenForReading opens an existing object.
func (*Provider) OpenForReadingMustExist ¶
func (p *Provider) OpenForReadingMustExist( fileType base.FileType, fileNum base.FileNum, ) (Readable, error)
OpenForReadingMustExist is a variant of OpenForReading which causes a fatal error if the file does not exist. The fatal error message contains information helpful for debugging.
func (*Provider) Path ¶
func (p *Provider) Path(meta ObjectMetadata) string
Path returns an internal, implementation-dependent path for the object. It is meant to be used for informational purposes (like logging).
func (*Provider) Remove ¶
Remove removes an object.
The object is not guaranteed to be durably removed until Sync is called.
func (*Provider) SetCreatorID ¶
SetCreatorID sets the CreatorID which is needed in order to use shared objects. Shared object usage is disabled until this method is called the first time. Once set, the Creator ID is persisted and cannot change.
Cannot be called if shared storage is not configured for the provider.
type ReadHandle ¶
type ReadHandle interface {
io.ReaderAt
io.Closer
// MaxReadahead configures the implementation to expect large sequential
// reads. Used to skip any initial read-ahead ramp-up.
MaxReadahead()
// RecordCacheHit informs the implementation that we were able to retrieve a
// block from cache.
RecordCacheHit(offset, size int64)
}
ReadHandle is used to perform reads that are related and might benefit from optimizations like read-ahead.
func UsePreallocatedReadHandle ¶
func UsePreallocatedReadHandle(readable Readable, rh *PreallocatedReadHandle) ReadHandle
UsePreallocatedReadHandle is equivalent to calling readable.NewReadHandle() but uses the existing storage of a PreallocatedReadHandle when possible (currently this happens if we are reading from a local file). The returned handle still needs to be closed.
type Readable ¶
type Readable interface {
io.ReaderAt
io.Closer
// Size returns the size of the object.
Size() int64
// NewReadHandle creates a read handle for ReadAt requests that are related
// and can benefit from optimizations like read-ahead.
//
// The ReadHandle must be closed before the Readable is closed.
//
// Multiple separate ReadHandles can be used.
NewReadHandle() ReadHandle
}
Readable is the handle for an object that is open for reading.
type Settings ¶
type Settings struct {
Logger base.Logger
// Local filesystem configuration.
FS vfs.FS
FSDirName string
// FSDirInitialListing is a listing of FSDirName at the time of calling Open.
//
// This is an optional optimization to avoid double listing on Open when the
// higher layer already has a listing. When nil, we obtain the listing on
// Open.
FSDirInitialListing []string
// Cleaner cleans obsolete files from the local filesystem.
//
// The default cleaner uses the DeleteCleaner.
FSCleaner base.Cleaner
// NoSyncOnClose decides whether the implementation will enforce a
// close-time synchronization (e.g., fdatasync() or sync_file_range())
// on files it writes to. Setting this to true removes the guarantee for a
// sync on close. Some implementations can still issue a non-blocking sync.
NoSyncOnClose bool
// BytesPerSync enables periodic syncing of files in order to smooth out
// writes to disk. This option does not provide any persistence guarantee, but
// is used to avoid latency spikes if the OS automatically decides to write
// out a large chunk of dirty filesystem buffers.
BytesPerSync int
// (experimental).
Shared struct {
Storage shared.Storage
}
}
Settings that must be specified when creating the Provider.
type SharedObjectBacking ¶
type SharedObjectBacking []byte
SharedObjectBacking encodes the metadata necessary to incorporate a shared object into a different Pebble instance.
type SharedObjectToAttach ¶
type SharedObjectToAttach struct {
// FileNum is the file number that will be used to refer to this object (in
// the context of this instance).
// generated from a different instance).
Backing SharedObjectBacking
}
SharedObjectToAttach contains the arguments needed to attach an existing shared object.
type Writable ¶
type Writable interface {
// Write writes len(p) bytes from p to the underlying object. The data is not
// guaranteed to be durable until Finish is called.
//
// Note that Write *is* allowed to modify the slice passed in, whether
// temporarily or permanently. Callers of Write need to take this into
// account.
Write(p []byte) error
// Finish completes the object and makes the data durable.
// No further calls are allowed after calling Finish.
Finish() error
// Abort gives up on finishing the object. There is no guarantee about whether
// the object exists after calling Abort.
// No further calls are allowed after calling Abort.
Abort()
}
Writable is the handle for an object that is open for writing. Either Finish or Abort must be called.
func NewFileWritable ¶
NewFileWritable returns a Writable that uses a file as underlying storage.