litefs

package module
v0.3.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Dec 26, 2022 License: Apache-2.0 Imports: 24 Imported by: 1

README

LiteFS GitHub release (latest by date) Status GitHub

LiteFS is a FUSE-based file system for replicating SQLite databases across a cluster of machines. It works as a passthrough file system that intercepts writes to SQLite databases in order to detect transaction boundaries and record changes on a per-transaction level in LTX files.

This project is actively maintained but is currently in a beta state. Please report any bugs as an issue on the GitHub repository.

You can find a Getting Started guide on LiteFS' documentation site. Please see the ARCHITECTURE.md design document for details about how LiteFS works.

Contributing

LiteFS contributions work a little different than most GitHub projects. If you have a small bug fix or typo fix, please PR directly to this repository.

If you would like to contribute a feature, please follow these steps:

  1. Discuss the feature in an issue on this GitHub repository.
  2. Create a pull request to your fork of the repository.
  3. Post a link to your pull request in the issue for consideration.

This project has a roadmap and features are added and tested in a certain order. Additionally, it's likely that code style, implementation details, and test coverage will need to be tweaked so it's easier to for me to grab your implementation as a starting point when implementing a feature.

Documentation

Index

Constants

View Source 
const (
	WALHeaderSize      = 32
	WALFrameHeaderSize = 24
	WALIndexHeaderSize = 136
	WALIndexBlockSize  = 32768
)

SQLite constants

View Source 
const (
	PENDING_BYTE  = 0x40000000
	RESERVED_BYTE = (PENDING_BYTE + 1)
	SHARED_FIRST  = (PENDING_BYTE + 2)
	SHARED_SIZE   = 510
)

SQLite rollback journal lock constants.

View Source 
const (
	WAL_WRITE_LOCK   = 120
	WAL_CKPT_LOCK    = 121
	WAL_RECOVER_LOCK = 122
	WAL_READ_LOCK0   = 123
	WAL_READ_LOCK1   = 124
	WAL_READ_LOCK2   = 125
	WAL_READ_LOCK3   = 126
	WAL_READ_LOCK4   = 127
)

SQLite WAL lock constants.

View Source 
const (
	F_OFD_GETLK  = 36
	F_OFD_SETLK  = 37
	F_OFD_SETLKW = 38
)

Open file description lock constants.

View Source 
const (
	JournalModeDelete   = "DELETE"
	JournalModeTruncate = "TRUNCATE"
	JournalModePersist  = "PERSIST"
	JournalModeWAL      = "WAL"
)
View Source 
const (
	FileTypeNone = FileType(iota)
	FileTypeDatabase
	FileTypeJournal
	FileTypeWAL
	FileTypeSHM
	FileTypePos
)

Database file types.

View Source 
const (
	StreamFrameTypeLTX   = StreamFrameType(1)
	StreamFrameTypeReady = StreamFrameType(2)
	StreamFrameTypeEnd   = StreamFrameType(3)
)
View Source 
const (
	SQLITE_DATABASE_HEADER_STRING = "SQLite format 3\x00"

	// Location of the database size, in pages, in the main database file.
	SQLITE_DATABASE_SIZE_OFFSET = 28

	/// Magic header string that identifies a SQLite journal header.
	/// https://www.sqlite.org/fileformat.html#the_rollback_journal
	SQLITE_JOURNAL_HEADER_STRING = "\xd9\xd5\x05\xf9\x20\xa1\x63\xd7"

	// Size of the journal header, in bytes.
	SQLITE_JOURNAL_HEADER_SIZE = 28
)
View Source 
const (
	// Database file locks
	LockTypePending  = LockType(0x40000000) // 1073741824
	LockTypeReserved = LockType(0x40000001) // 1073741825
	LockTypeShared   = LockType(0x40000002) // 1073741826

	// SHM file locks
	LockTypeWrite   = LockType(120)
	LockTypeCkpt    = LockType(121)
	LockTypeRecover = LockType(122)
	LockTypeRead0   = LockType(123)
	LockTypeRead1   = LockType(124)
	LockTypeRead2   = LockType(125)
	LockTypeRead3   = LockType(126)
	LockTypeRead4   = LockType(127)
	LockTypeDMS     = LockType(128)
)
View Source 
const (
	DBModeRollback = DBMode(0)
	DBModeWAL      = DBMode(1)
)

Database journal modes.

View Source 
const (
	RWMutexStateUnlocked = RWMutexState(iota)
	RWMutexStateShared
	RWMutexStateExclusive
)
View Source 
const (
	DefaultReconnectDelay = 1 * time.Second
	DefaultDemoteDelay    = 10 * time.Second

	DefaultRetention                = 10 * time.Minute
	DefaultRetentionMonitorInterval = 1 * time.Minute
)

Default store settings.

View Source 
const IDLength = 24

IDLength is the length of a node ID, in bytes.

View Source 
const RWMutexInterval = 10 * time.Microsecond

RWMutexInterval is the time between reattempting lock acquisition.

View Source 
const TraceLogFlags = log.LstdFlags | log.LUTC

TraceLogFlags are the flags to be used with TraceLog.

Variables

View Source 
var (
	ErrDatabaseNotFound = fmt.Errorf("database not found")
	ErrDatabaseExists   = fmt.Errorf("database already exists")

	ErrNoPrimary     = errors.New("no primary")
	ErrPrimaryExists = errors.New("primary exists")
	ErrLeaseExpired  = errors.New("lease expired")

	ErrReadOnlyReplica = fmt.Errorf("read only replica")
)

LiteFS errors

View Source 
var NativeEndian = binary.LittleEndian

NativeEndian is always set to little endian as that is the only endianness used by supported platforms for LiteFS. This may be expanded in the future.

View Source 
var TraceLog = log.New(io.Discard, "", TraceLogFlags)

TraceLog is a log for low-level tracing.

Functions

func ContainsLockType added in v0.3.0 

func ContainsLockType(a []LockType, typ LockType) bool

ContainsLockType returns true if a contains typ.

func JournalChecksum added in v0.3.0 

func JournalChecksum(data []byte, initial uint32) uint32

JournalChecksum returns the checksum used by the journal format.

func TrimName 

func TrimName(name string) string

TrimName removes "-journal", "-shm" or "-wal" from the given name.

func WALChecksum added in v0.3.0 

func WALChecksum(bo binary.ByteOrder, s0, s1 uint32, b []byte) (uint32, uint32)

WALChecksum computes a running SQLite WAL checksum over a byte slice.

func WriteStreamFrame 

func WriteStreamFrame(w io.Writer, f StreamFrame) error

WriteStreamFrame writes the stream type & frame to the writer.

Types

type Client 

type Client interface {
	// Stream starts a long-running connection to stream changes from another node.
	Stream(ctx context.Context, rawurl string, id string, posMap map[string]Pos) (io.ReadCloser, error)
}

Client represents a client for connecting to other LiteFS nodes.

type DB 

type DB struct {

	// Returns the current time. Used for mocking time in tests.
	Now func() time.Time
	// contains filtered or unexported fields
}

DB represents a SQLite database.

func NewDB 

func NewDB(store *Store, name string, path string) *DB

NewDB returns a new instance of DB.

func (*DB) AcquireWriteLock added in v0.3.0 

func (db *DB) AcquireWriteLock(ctx context.Context) (_ *GuardSet, err error)

AcquireWriteLock acquires the appropriate locks for a write depending on if the database uses a rollback journal or WAL.

func (*DB) ApplyLTX added in v0.2.0 

func (db *DB) ApplyLTX(ctx context.Context, path string) error

ApplyLTX applies an LTX file to the database.

func (*DB) CanLock added in v0.3.0 

func (db *DB) CanLock(ctx context.Context, owner uint64, lockTypes []LockType) (bool, RWMutexState)

CanLock returns true if all locks can acquire a write lock. If false, also returns the mutex state of the blocking lock.

func (*DB) CanRLock added in v0.3.0 

func (db *DB) CanRLock(ctx context.Context, owner uint64, lockTypes []LockType) bool

CanRLock returns true if all locks can acquire a read lock.

func (*DB) CloseDatabase added in v0.3.0 

func (db *DB) CloseDatabase(ctx context.Context, f *os.File, owner uint64) error

CloseDatabase closes a handle associated with the database file.

func (*DB) CloseJournal added in v0.3.0 

func (db *DB) CloseJournal(ctx context.Context, f *os.File, owner uint64) error

CloseJournal closes a handle associated with the journal file.

func (*DB) CloseSHM added in v0.3.0 

func (db *DB) CloseSHM(ctx context.Context, f *os.File, owner uint64) error

CloseSHM closes a handle associated with the SHM file.

func (*DB) CloseWAL added in v0.3.0 

func (db *DB) CloseWAL(ctx context.Context, f *os.File, owner uint64) error

CloseWAL closes a handle associated with the WAL file.

func (*DB) CommitJournal 

func (db *DB) CommitJournal(ctx context.Context, mode JournalMode) (err error)

CommitJournal deletes the journal file which commits or rolls back the transaction.

func (*DB) CommitWAL added in v0.3.0 

func (db *DB) CommitWAL(ctx context.Context) (err error)

commitWAL is called when the client releases the WAL_WRITE_LOCK(120). The transaction data is copied from the WAL into an LTX file and committed.

func (*DB) CreateGuardSetIfNotExists added in v0.3.0 

func (db *DB) CreateGuardSetIfNotExists(owner uint64) *GuardSet

CreateGuardSetIfNotExists returns a guard set for the given owner. Creates a new guard set if one is not associated with the owner.

func (*DB) CreateJournal 

func (db *DB) CreateJournal() (*os.File, error)

CreateJournal creates a new journal file on disk.

func (*DB) CreateSHM added in v0.3.0 

func (db *DB) CreateSHM() (*os.File, error)

CreateSHM creates a new shared memory file on disk.

func (*DB) CreateWAL added in v0.3.0 

func (db *DB) CreateWAL() (*os.File, error)

CreateWAL creates a new WAL file on disk.

func (*DB) DatabasePath added in v0.1.1 

func (db *DB) DatabasePath() string

DatabasePath returns the path to the underlying database file.

func (*DB) EnforceRetention added in v0.2.0 

func (db *DB) EnforceRetention(ctx context.Context, minTime time.Time) error

EnforceRetention removes all LTX files created before minTime.

func (*DB) GuardSet added in v0.2.0 

func (db *DB) GuardSet(owner uint64) *GuardSet

GuardSet returns a guard set for the given owner, if it exists.

func (*DB) Import added in v0.3.0 

func (db *DB) Import(ctx context.Context, r io.Reader) error

Import replaces the contents of the database with the contents from the r. NOTE: LiteFS does not validate the integrity of the imported database!

func (*DB) InWriteTx 

func (db *DB) InWriteTx() bool

InWriteTx returns true if the RESERVED lock has an exclusive lock.

func (*DB) JournalPath added in v0.1.1 

func (db *DB) JournalPath() string

JournalPath returns the path to the underlying journal file.

func (*DB) LTXDir 

func (db *DB) LTXDir() string

LTXDir returns the path to the directory of LTX transaction files.

func (*DB) LTXPath 

func (db *DB) LTXPath(minTXID, maxTXID uint64) string

LTXPath returns the path of an LTX file.

func (*DB) Name 

func (db *DB) Name() string

Name of the database name.

func (*DB) Open 

func (db *DB) Open() error

Open initializes the database from files in its data directory.

func (*DB) OpenDatabase added in v0.3.0 

func (db *DB) OpenDatabase(ctx context.Context) (*os.File, error)

OpenDatabase returns a handle for the database file.

func (*DB) OpenJournal added in v0.3.0 

func (db *DB) OpenJournal(ctx context.Context) (*os.File, error)

OpenJournal returns a handle for the journal file.

func (*DB) OpenLTXFile 

func (db *DB) OpenLTXFile(txID uint64) (*os.File, error)

OpenLTXFile returns a file handle to an LTX file that contains the given TXID.

func (*DB) OpenSHM added in v0.3.0 

func (db *DB) OpenSHM(ctx context.Context) (*os.File, error)

OpenSHM returns a handle for the shared memory file.

func (*DB) OpenWAL added in v0.3.0 

func (db *DB) OpenWAL(ctx context.Context) (*os.File, error)

OpenWAL returns a handle for the write-ahead log file.

func (*DB) Path 

func (db *DB) Path() string

Path of the database's data directory.

func (*DB) Pos 

func (db *DB) Pos() Pos

Pos returns the current transaction position of the database.

func (*DB) ReadDatabaseAt added in v0.3.0 

func (db *DB) ReadDatabaseAt(ctx context.Context, f *os.File, data []byte, offset int64, owner uint64) (int, error)

ReadDatabaseAt reads from the database at the specified index.

func (*DB) ReadJournalAt added in v0.3.0 

func (db *DB) ReadJournalAt(ctx context.Context, f *os.File, data []byte, offset int64, owner uint64) (int, error)

ReadJournalAt reads from the journal at the specified offset.

func (*DB) ReadLTXDir added in v0.2.0 

func (db *DB) ReadLTXDir() ([]fs.DirEntry, error)

ReadLTXDir returns DirEntry for every LTX file.

func (*DB) ReadSHMAt added in v0.3.0 

func (db *DB) ReadSHMAt(ctx context.Context, f *os.File, data []byte, offset int64, owner uint64) (int, error)

ReadSHMAt reads from the shared memory at the specified offset.

func (*DB) ReadWALAt added in v0.3.0 

func (db *DB) ReadWALAt(ctx context.Context, f *os.File, data []byte, offset int64, owner uint64) (int, error)

ReadWALAt reads from the WAL at the specified index.

func (*DB) Recover added in v0.3.0 

func (db *DB) Recover(ctx context.Context) error

Recover forces a rollback (journal) or checkpoint (wal).

func (*DB) RemoveJournal added in v0.3.0 

func (db *DB) RemoveJournal(ctx context.Context) error

RemoveJournal deletes the journal file from disk.

func (*DB) RemoveSHM added in v0.3.0 

func (db *DB) RemoveSHM(ctx context.Context) error

RemoveSHM removes the SHM file from disk.

func (*DB) RemoveWAL added in v0.3.0 

func (db *DB) RemoveWAL(ctx context.Context) (err error)

RemoveWAL deletes the WAL file from disk.

func (*DB) SHMPath added in v0.3.0 

func (db *DB) SHMPath() string

SHMPath returns the path to the underlying shared memory file.

func (*DB) Store added in v0.2.0 

func (db *DB) Store() *Store

Store returns the store that the database is a member of.

func (*DB) SyncDatabase added in v0.3.0 

func (db *DB) SyncDatabase(ctx context.Context) (err error)

SyncDatabase fsync's the database file.

func (*DB) SyncJournal added in v0.3.0 

func (db *DB) SyncJournal(ctx context.Context) (err error)

SyncJournal fsync's the journal file.

func (*DB) SyncSHM added in v0.3.0 

func (db *DB) SyncSHM(ctx context.Context) (err error)

SyncSHM fsync's the shared memory file.

func (*DB) SyncWAL added in v0.3.0 

func (db *DB) SyncWAL(ctx context.Context) (err error)

SyncWAL fsync's the WAL file.

func (*DB) TXID 

func (db *DB) TXID() uint64

TXID returns the current transaction ID.

func (*DB) TruncateDatabase added in v0.3.0 

func (db *DB) TruncateDatabase(ctx context.Context, size int64) (err error)

TruncateDatabase sets the size of the database file.

func (*DB) TruncateJournal added in v0.3.0 

func (db *DB) TruncateJournal(ctx context.Context) error

TruncateJournal sets the size of the journal file.

func (*DB) TruncateSHM added in v0.3.0 

func (db *DB) TruncateSHM(ctx context.Context, size int64) error

TruncateSHM sets the size of the the SHM file.

func (*DB) TruncateWAL added in v0.3.0 

func (db *DB) TruncateWAL(ctx context.Context, size int64) (err error)

TruncateWAL sets the size of the WAL file.

func (*DB) TryLocks added in v0.3.0 

func (db *DB) TryLocks(ctx context.Context, owner uint64, lockTypes []LockType) (bool, error)

TryLocks attempts to lock one or more locks on the database for a given owner. Returns an error if no locks are supplied.

func (*DB) TryRLocks added in v0.3.0 

func (db *DB) TryRLocks(ctx context.Context, owner uint64, lockTypes []LockType) bool

TryRLocks attempts to read lock one or more locks on the database for a given owner. Returns an error if no locks are supplied.

func (*DB) Unlock added in v0.3.0 

func (db *DB) Unlock(ctx context.Context, owner uint64, lockTypes []LockType)

Unlock unlocks one or more locks on the database for a given owner.

func (*DB) UnlockDatabase added in v0.3.0 

func (db *DB) UnlockDatabase(ctx context.Context, owner uint64)

UnlockDatabase unlocks all locks from the database file.

func (*DB) UnlockSHM added in v0.3.0 

func (db *DB) UnlockSHM(ctx context.Context, owner uint64)

UnlockSHM unlocks all locks from the SHM file.

func (*DB) WALPath added in v0.3.0 

func (db *DB) WALPath() string

WALPath returns the path to the underlying WAL file.

func (*DB) WriteDatabaseAt added in v0.3.0 

func (db *DB) WriteDatabaseAt(ctx context.Context, f *os.File, data []byte, offset int64, owner uint64) error

WriteDatabaseAt writes data to the main database file at the given index.

func (*DB) WriteJournalAt added in v0.3.0 

func (db *DB) WriteJournalAt(ctx context.Context, f *os.File, data []byte, offset int64, owner uint64) (err error)

WriteJournal writes data to the rollback journal file.

func (*DB) WriteSHMAt added in v0.3.0 

func (db *DB) WriteSHMAt(ctx context.Context, f *os.File, data []byte, offset int64, owner uint64) (int, error)

WriteSHMAt writes data to the SHM file.

func (*DB) WriteSnapshotTo added in v0.2.0 

func (db *DB) WriteSnapshotTo(ctx context.Context, dst io.Writer) (header ltx.Header, trailer ltx.Trailer, err error)

WriteSnapshotTo writes an LTX snapshot to dst.

func (*DB) WriteWALAt added in v0.3.0 

func (db *DB) WriteWALAt(ctx context.Context, f *os.File, data []byte, offset int64, owner uint64) (err error)

WriteWALAt writes data to the WAL file. On final commit write, an LTX file is generated for the transaction.

type DBMode added in v0.3.0 

type DBMode int

DBMode represents either a rollback journal or WAL mode.

func (DBMode) String added in v0.3.0 

func (m DBMode) String() string

String returns the string representation of m.

type EndStreamFrame added in v0.3.0 

type EndStreamFrame struct{}

func (*EndStreamFrame) ReadFrom added in v0.3.0 

func (f *EndStreamFrame) ReadFrom(r io.Reader) (int64, error)

func (*EndStreamFrame) Type added in v0.3.0 

func (f *EndStreamFrame) Type() StreamFrameType

func (*EndStreamFrame) WriteTo added in v0.3.0 

func (f *EndStreamFrame) WriteTo(w io.Writer) (int64, error)

type FileType 

type FileType int

FileType represents a type of SQLite file.

func (FileType) IsValid 

func (t FileType) IsValid() bool

IsValid returns true if t is a valid file type.

type GuardSet added in v0.2.0 

type GuardSet struct {
	// contains filtered or unexported fields
}

GuardSet represents a set of mutex guards by a single owner.

func (*GuardSet) Ckpt added in v0.3.0 

func (s *GuardSet) Ckpt() *RWMutexGuard

Ckpt returns a reference to the CKPT mutex guard.

func (*GuardSet) DMS added in v0.3.0 

func (s *GuardSet) DMS() *RWMutexGuard

DMS returns a reference to the DMS mutex guard.

func (*GuardSet) Guard added in v0.2.0 

func (s *GuardSet) Guard(lockType LockType) *RWMutexGuard

Guard returns a guard by lock type. Panic on invalid lock type.

func (*GuardSet) Pending added in v0.3.0 

func (s *GuardSet) Pending() *RWMutexGuard

Pending returns a reference to the PENDING mutex guard.

func (*GuardSet) Read0 added in v0.3.0 

func (s *GuardSet) Read0() *RWMutexGuard

Read0 returns a reference to the READ0 mutex guard.

func (*GuardSet) Read1 added in v0.3.0 

func (s *GuardSet) Read1() *RWMutexGuard

Read1 returns a reference to the READ1 mutex guard.

func (*GuardSet) Read2 added in v0.3.0 

func (s *GuardSet) Read2() *RWMutexGuard

Read2 returns a reference to the READ2 mutex guard.

func (*GuardSet) Read3 added in v0.3.0 

func (s *GuardSet) Read3() *RWMutexGuard

Read3 returns a reference to the READ3 mutex guard.

func (*GuardSet) Read4 added in v0.3.0 

func (s *GuardSet) Read4() *RWMutexGuard

Read4 returns a reference to the READ4 mutex guard.

func (*GuardSet) Recover added in v0.3.0 

func (s *GuardSet) Recover() *RWMutexGuard

Recover returns a reference to the RECOVER mutex guard.

func (*GuardSet) Reserved added in v0.3.0 

func (s *GuardSet) Reserved() *RWMutexGuard

Reserved returns a reference to the RESERVED mutex guard.

func (*GuardSet) Shared added in v0.3.0 

func (s *GuardSet) Shared() *RWMutexGuard

Shared returns a reference to the SHARED mutex guard.

func (*GuardSet) Unlock added in v0.2.0 

func (s *GuardSet) Unlock()

Unlock unlocks all the guards in reversed order that they are acquired by SQLite.

func (*GuardSet) UnlockDatabase added in v0.3.0 

func (s *GuardSet) UnlockDatabase()

UnlockDatabase unlocks all the database file guards.

func (*GuardSet) UnlockSHM added in v0.3.0 

func (s *GuardSet) UnlockSHM()

UnlockSHM unlocks all the SHM file guards.

func (*GuardSet) Write added in v0.3.0 

func (s *GuardSet) Write() *RWMutexGuard

Write returns a reference to the WRITE mutex guard.

type Invalidator added in v0.1.1 

type Invalidator interface {
	InvalidateDB(db *DB) error
	InvalidateDBRange(db *DB, offset, size int64) error
	InvalidateSHM(db *DB) error
	InvalidatePos(db *DB) error
	InvalidateEntry(name string) error
}

Invalidator is a callback for the store to use to invalidate the kernel page cache.

type JournalMode 

type JournalMode string

JournalMode represents a SQLite journal mode.

type JournalReader added in v0.3.0 

type JournalReader struct {
	// contains filtered or unexported fields
}

JouralReader represents a reader of the SQLite journal file format.

func NewJournalReader added in v0.3.0 

func NewJournalReader(f *os.File, pageSize uint32) *JournalReader

JournalReader returns a new instance of JournalReader.

func (*JournalReader) DatabaseSize added in v0.3.0 

func (r *JournalReader) DatabaseSize() int64

DatabaseSize returns the size of the database before the journal transaction, in bytes.

func (*JournalReader) IsValid added in v0.3.0 

func (r *JournalReader) IsValid() bool

IsValid returns true if at least one journal header was read.

func (*JournalReader) Next added in v0.3.0 

func (r *JournalReader) Next() (err error)

Next reads the next segment of the journal. Returns io.EOF if no more segments exist.

func (*JournalReader) ReadFrame added in v0.3.0 

func (r *JournalReader) ReadFrame() (pgno uint32, data []byte, err error)

ReadFrame returns the page number and page data for the next frame. Returns io.EOF after the last frame. Page data should not be retained.

type LTXStreamFrame 

type LTXStreamFrame struct {
	Name string // database name
}

func (*LTXStreamFrame) ReadFrom 

func (f *LTXStreamFrame) ReadFrom(r io.Reader) (int64, error)

func (*LTXStreamFrame) Type 

func (*LTXStreamFrame) Type() StreamFrameType

Type returns the type of stream frame.

func (*LTXStreamFrame) WriteTo 

func (f *LTXStreamFrame) WriteTo(w io.Writer) (int64, error)

type Lease 

type Lease interface {
	RenewedAt() time.Time
	TTL() time.Duration

	// Renew attempts to reset the TTL on the lease.
	// Returns ErrLeaseExpired if the lease has expired or was deleted.
	Renew(ctx context.Context) error

	// Close attempts to remove the lease from the server.
	Close() error
}

Lease represents an acquired lease from a Leaser.

type Leaser 

type Leaser interface {
	io.Closer

	AdvertiseURL() string

	// Acquire attempts to acquire the lease to become the primary.
	Acquire(ctx context.Context) (Lease, error)

	// PrimaryInfo attempts to read the current primary data.
	// Returns ErrNoPrimary if no primary currently has the lease.
	PrimaryInfo(ctx context.Context) (PrimaryInfo, error)
}

Leaser represents an API for obtaining a lease for leader election.

type LockType 

type LockType int

LockType represents a SQLite lock type.

func ParseDatabaseLockRange added in v0.3.0 

func ParseDatabaseLockRange(start, end uint64) []LockType

ParseDatabaseLockRange returns a list of SQLite database locks that are within a range.

func ParseSHMLockRange added in v0.3.0 

func ParseSHMLockRange(start, end uint64) []LockType

ParseSHMLockRange returns a list of SQLite WAL locks that are within a range.

func (LockType) String added in v0.3.0 

func (t LockType) String() string

String returns the name of the lock type.

type Pos 

type Pos struct {
	TXID              uint64
	PostApplyChecksum uint64
}

Pos represents the transactional position of a database.

func (Pos) IsZero 

func (p Pos) IsZero() bool

IsZero returns true if the position is empty.

func (Pos) String added in v0.2.0 

func (p Pos) String() string

String returns a string representation of the position.

type PrimaryInfo added in v0.2.0 

type PrimaryInfo struct {
	Hostname     string `json:"hostname"`
	AdvertiseURL string `json:"advertise-url"`
}

PrimaryInfo is the JSON object stored in the Consul lease value.

func (*PrimaryInfo) Clone added in v0.2.0 

func (info *PrimaryInfo) Clone() *PrimaryInfo

Clone returns a copy of info.

type RWMutex 

type RWMutex struct {

	// If set, this function is called when the state transitions.
	// Must be set before use of the mutex or its guards.
	OnLockStateChange func(prevState, newState RWMutexState)
	// contains filtered or unexported fields
}

RWMutex is a reader/writer mutual exclusion lock. It wraps the sync package to provide additional capabilities such as lock upgrades & downgrades. It only supports TryLock() & TryRLock() as that is what's supported by our FUSE file system.

func (*RWMutex) Guard added in v0.2.0 

func (rw *RWMutex) Guard() RWMutexGuard

Guard returns an unlocked guard for the mutex.

func (*RWMutex) State 

func (rw *RWMutex) State() RWMutexState

State returns whether the mutex has a exclusive lock, one or more shared locks, or if the mutex is unlocked.

type RWMutexGuard 

type RWMutexGuard struct {
	// contains filtered or unexported fields
}

RWMutexGuard is a reference to a mutex. Locking, unlocking, upgrading, & downgrading operations are all performed via the guard instead of directly on the RWMutex itself as this works similarly to how POSIX locks work.

func (*RWMutexGuard) CanLock 

func (g *RWMutexGuard) CanLock() (canLock bool, mutexState RWMutexState)

CanLock returns true if the guard can become an exclusive lock. Also returns the current state of the underlying mutex to determine if the lock is blocked by a shared or exclusive lock.

func (*RWMutexGuard) CanRLock added in v0.2.0 

func (g *RWMutexGuard) CanRLock() bool

CanRLock returns true if the guard can become a shared lock.

func (*RWMutexGuard) Lock added in v0.2.0 

func (g *RWMutexGuard) Lock(ctx context.Context) error

Lock attempts to obtain a exclusive lock for the guard. Returns an error if ctx is done.

func (*RWMutexGuard) RLock 

func (g *RWMutexGuard) RLock(ctx context.Context) error

RLock attempts to obtain a shared lock for the guard. Returns an error if ctx is done.

func (*RWMutexGuard) State added in v0.3.0 

func (g *RWMutexGuard) State() RWMutexState

State returns the current state of the guard.

func (*RWMutexGuard) TryLock 

func (g *RWMutexGuard) TryLock() bool

TryLock upgrades the lock from a shared lock to an exclusive lock. This is a no-op if the lock is already an exclusive lock. This function will trigger OnLockStateChange on the mutex, if set, and if state changes.

func (*RWMutexGuard) TryRLock added in v0.2.0 

func (g *RWMutexGuard) TryRLock() bool

TryRLock attempts to obtain a shared lock on the mutex for the guard. This will upgrade an unlocked guard and downgrade an exclusive guard. Shared guards are a no-op.

func (*RWMutexGuard) Unlock 

func (g *RWMutexGuard) Unlock()

Unlock unlocks the underlying mutex.

type RWMutexState 

type RWMutexState int

RWMutexState represents the lock state of an RWMutex or RWMutexGuard.

func (RWMutexState) String added in v0.2.0 

func (s RWMutexState) String() string

String returns the string representation of the state.

type ReadyStreamFrame added in v0.2.0 

type ReadyStreamFrame struct{}

func (*ReadyStreamFrame) ReadFrom added in v0.2.0 

func (f *ReadyStreamFrame) ReadFrom(r io.Reader) (int64, error)

func (*ReadyStreamFrame) Type added in v0.2.0 

func (f *ReadyStreamFrame) Type() StreamFrameType

func (*ReadyStreamFrame) WriteTo added in v0.2.0 

func (f *ReadyStreamFrame) WriteTo(w io.Writer) (int64, error)

type StaticLease added in v0.2.0 

type StaticLease struct {
	// contains filtered or unexported fields
}

StaticLease represents a lease for a fixed primary.

func (*StaticLease) Close added in v0.2.0 

func (l *StaticLease) Close() error

func (*StaticLease) Renew added in v0.2.0 

func (l *StaticLease) Renew(ctx context.Context) error

Renew is a no-op.

func (*StaticLease) RenewedAt added in v0.2.0 

func (l *StaticLease) RenewedAt() time.Time

RenewedAt returns the Unix epoch in UTC.

func (*StaticLease) TTL added in v0.2.0 

func (l *StaticLease) TTL() time.Duration

TTL returns the duration until the lease expires which is a time well into the future.

type StaticLeaser added in v0.2.0 

type StaticLeaser struct {
	// contains filtered or unexported fields
}

StaticLeaser always returns a lease to a static primary.

func NewStaticLeaser added in v0.2.0 

func NewStaticLeaser(isPrimary bool, hostname, advertiseURL string) *StaticLeaser

NewStaticLeaser returns a new instance of StaticLeaser.

func (*StaticLeaser) Acquire added in v0.2.0 

func (l *StaticLeaser) Acquire(ctx context.Context) (Lease, error)

Acquire returns a lease if this node is the static primary. Otherwise returns ErrPrimaryExists.

func (*StaticLeaser) AdvertiseURL added in v0.2.0 

func (l *StaticLeaser) AdvertiseURL() string

AdvertiseURL returns the primary URL if this is the primary. Otherwise returns blank.

func (*StaticLeaser) Close added in v0.2.0 

func (l *StaticLeaser) Close() (err error)

Close is a no-op.

func (*StaticLeaser) IsPrimary added in v0.2.0 

func (l *StaticLeaser) IsPrimary() bool

IsPrimary returns true if the current node is the primary.

func (*StaticLeaser) PrimaryInfo added in v0.2.0 

func (l *StaticLeaser) PrimaryInfo(ctx context.Context) (PrimaryInfo, error)

PrimaryInfo returns the primary's info. Returns ErrNoPrimary if the node is the primary.

type Store 

type Store struct {

	// Client used to connect to other LiteFS instances.
	Client Client

	// Leaser manages the lease that controls leader election.
	Leaser Leaser

	// Time to wait after disconnecting from the primary to reconnect.
	ReconnectDelay time.Duration

	// Time to wait after manually demoting trying to become primary again.
	DemoteDelay time.Duration

	// Length of time to retain LTX files.
	Retention                time.Duration
	RetentionMonitorInterval time.Duration

	// Callback to notify kernel of file changes.
	Invalidator Invalidator

	// If true, computes and verifies the checksum of the entire database
	// after every transaction. Should only be used during testing.
	StrictVerify bool
	// contains filtered or unexported fields
}

Store represents a collection of databases.

func NewStore 

func NewStore(path string, candidate bool) *Store

NewStore returns a new instance of Store.

func (*Store) Candidate added in v0.2.0 

func (s *Store) Candidate() bool

Candidate returns true if store is eligible to be the primary.

func (*Store) Close 

func (s *Store) Close() error

Close signals for the store to shut down.

func (*Store) CreateDB 

func (s *Store) CreateDB(name string) (db *DB, f *os.File, err error)

CreateDB creates a new database with the given name. The returned file handle must be closed by the caller. Returns an error if a database with the same name already exists.

func (*Store) CreateDBIfNotExists added in v0.2.0 

func (s *Store) CreateDBIfNotExists(name string) (*DB, error)

CreateDBIfNotExists creates an empty database with the given name.

func (*Store) DB 

func (s *Store) DB(name string) *DB

DBByName returns a database by name. Returns nil if the database does not exist.

func (*Store) DBDir 

func (s *Store) DBDir() string

DBDir returns the folder that stores all databases.

func (*Store) DBPath added in v0.2.0 

func (s *Store) DBPath(name string) string

DBPath returns the folder that stores a single database.

func (*Store) DBs 

func (s *Store) DBs() []*DB

DBs returns a list of databases.

func (*Store) Demote added in v0.3.0 

func (s *Store) Demote()

Demote instructs store to destroy its primary lease, if any. Store will wait momentarily before attempting to become primary again.

func (*Store) EnforceRetention added in v0.2.0 

func (s *Store) EnforceRetention(ctx context.Context) (err error)

EnforceRetention enforces retention of LTX files on all databases.

func (*Store) ID added in v0.2.0 

func (s *Store) ID() string

ID returns the unique identifier for this instance. Available after Open(). Persistent across restarts if underlying storage is persistent.

func (*Store) IsPrimary 

func (s *Store) IsPrimary() bool

IsPrimary returns true if store has a lease to be the primary.

func (*Store) MarkDirty 

func (s *Store) MarkDirty(name string)

MarkDirty marks a database dirty on all subscribers.

func (*Store) Open 

func (s *Store) Open() error

Open initializes the store based on files in the data directory.

func (*Store) Path 

func (s *Store) Path() string

Path returns underlying data directory.

func (*Store) PosMap 

func (s *Store) PosMap() map[string]Pos

PosMap returns a map of databases and their transactional position.

func (*Store) PrimaryCtx added in v0.2.0 

func (s *Store) PrimaryCtx(ctx context.Context) context.Context

PrimaryCtx wraps ctx with another context that will cancel when no longer primary.

func (*Store) PrimaryInfo added in v0.2.0 

func (s *Store) PrimaryInfo() *PrimaryInfo

PrimaryInfo returns info about the current primary.

func (*Store) ReadyCh added in v0.2.0 

func (s *Store) ReadyCh() chan struct{}

ReadyCh returns a channel that is closed once the store has become primary or once it has connected to the primary.

func (*Store) Recover added in v0.3.0 

func (s *Store) Recover(ctx context.Context) (err error)

Recover forces a rollback (journal) or checkpoint (wal) on all open databases. This is done when switching the primary/replica state.

func (*Store) Subscribe 

func (s *Store) Subscribe() *Subscriber

Subscribe creates a new subscriber for store changes.

func (*Store) Unsubscribe 

func (s *Store) Unsubscribe(sub *Subscriber)

Unsubscribe removes a subscriber from the store.

type StoreVar added in v0.2.0 

type StoreVar Store

func (*StoreVar) String added in v0.2.0 

func (v *StoreVar) String() string

type StreamFrame 

type StreamFrame interface {
	io.ReaderFrom
	io.WriterTo
	Type() StreamFrameType
}

func ReadStreamFrame 

func ReadStreamFrame(r io.Reader) (StreamFrame, error)

ReadStreamFrame reads a the stream type & frame from the reader.

type StreamFrameType 

type StreamFrameType uint32

type Subscriber 

type Subscriber struct {
	// contains filtered or unexported fields
}

Subscriber subscribes to changes to databases in the store.

It implements a set of "dirty" databases instead of a channel of all events as clients can be slow and we don't want to cause channels to back up. It is the responsibility of the caller to determine the state changes which is usually just checking the position of the client versus the store's database.

func (*Subscriber) Close 

func (s *Subscriber) Close() error

Close removes the subscriber from the store.

func (*Subscriber) DirtySet 

func (s *Subscriber) DirtySet() map[string]struct{}

DirtySet returns a set of database IDs that have changed since the last call to DirtySet(). This call clears the set.

func (*Subscriber) MarkDirty 

func (s *Subscriber) MarkDirty(name string)

MarkDirty marks a database ID as dirty.

func (*Subscriber) NotifyCh 

func (s *Subscriber) NotifyCh() <-chan struct{}

NotifyCh returns a channel that receives a value when the dirty set has changed.

type WALReader added in v0.3.0 

type WALReader struct {
	// contains filtered or unexported fields
}

WALReader wraps an io.Reader and parses SQLite WAL frames.

This reader verifies the salt & checksum integrity while it reads. It does not enforce transaction boundaries (i.e. it may return uncommitted frames). It is the responsibility of the caller to handle this.

func NewWALReader added in v0.3.0 

func NewWALReader(r io.Reader) *WALReader

NewWALReader returns a new instance of WALReader.

func (*WALReader) Offset added in v0.3.0 

func (r *WALReader) Offset() int64

Offset returns the file offset of the last read frame. Returns zero if no frames have been read.

func (*WALReader) PageSize added in v0.3.0 

func (r *WALReader) PageSize() uint32

PageSize returns the page size from the header. Must call ReadHeader() first.

func (*WALReader) ReadFrame added in v0.3.0 

func (r *WALReader) ReadFrame(data []byte) (pgno, commit uint32, err error)

ReadFrame reads the next frame from the WAL and returns the page number. Returns io.EOF at the end of the valid WAL.

func (*WALReader) ReadHeader added in v0.3.0 

func (r *WALReader) ReadHeader() error

ReadHeader reads the WAL header into the reader. Returns io.EOF if WAL is invalid.

Source Files

View all Source files

Directories

Path Synopsis
cmd
litefs command
go:build linux
 go:build linux
litefs-bench command
testingutil

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.