ltx

package module
v0.3.14 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jan 8, 2025 License: Apache-2.0 Imports: 16 Imported by: 32

README

Lite Transaction File (LTX)

The LTX file format provides a way to store SQLite transactional data in a way that can be encrypted and compacted and is optimized for performance.

File Format

An LTX file is composed of several sections:

  1. Header
  2. Page block
  3. Trailer

The header contains metadata about the file, the page block contains page frames, and the trailer contains checksums of the file and the database end state.

Header

The header provides information about the number of page frames as well as database information such as the page size and database size. LTX files can be compacted together so each file contains the transaction ID (TXID) range that it represents. A timestamp provides users with a rough approximation of the time the transaction occurred and the checksum provides a basic integrity check.

Offset Size Description
0 4 Magic number. Always "LTX1".
4 4 Flags. Reserved. Always 0.
8 4 Page size, in bytes.
12 4 Size of DB after transaction, in pages.
16 4 Database ID.
20 8 Minimum transaction ID.
28 8 Maximum transaction ID.
36 8 Timestamp (Milliseconds since epoch)
44 8 Pre-apply DB checksum (CRC-ISO-64)
52 48 Reserved.
Page block

This block stores a series of page headers and page data.

Offset Size Description
0 4 Page number.
4 N Page data.
Trailer

The trailer provides checksum for the LTX file data, a rolling checksum of the database state after the LTX file is applied, and the checksum of the trailer itself.

Offset Size Description
0 8 Post-apply DB checksum (CRC-ISO-64)
8 8 File checksum (CRC-ISO-64)

Documentation

Overview

Package ltx reads and writes Liteserver Transaction (LTX) files.

Index

Constants

View Source 
const (
	// Magic is the first 4 bytes of every LTX file.
	Magic = "LTX1"

	// Version is the current version of the LTX file format.
	Version = 1
)
View Source 
const (
	HeaderSize     = 100
	PageHeaderSize = 4
	TrailerSize    = 16
)

Size constants.

View Source 
const (
	ChecksumSize          = 8
	TrailerChecksumOffset = TrailerSize - ChecksumSize
)

Checksum size & positions.

View Source 
const (
	HeaderFlagMask = uint32(0x00000001)

	HeaderFlagCompressLZ4 = uint32(0x00000001)
)

Header flags.

View Source 
const MaxPageSize = 65536

MaxPageSize is the maximum allowed page size for SQLite.

View Source 
const PENDING_BYTE = 0x40000000
View Source 
const RFC3339Milli = "2006-01-02T15:04:05.000Z07:00"

RFC3339Milli is the standard time format for LTX timestamps. It uses fixed-width millisecond resolution which makes it sortable.

Variables

View Source 
var (
	ErrInvalidFile   = errors.New("invalid LTX file")
	ErrDecoderClosed = errors.New("ltx decoder closed")
	ErrEncoderClosed = errors.New("ltx encoder closed")

	ErrNoChecksum            = errors.New("no file checksum")
	ErrInvalidChecksumFormat = errors.New("invalid file checksum format")
	ErrChecksumMismatch      = errors.New("file checksum mismatch")
)

Errors

Functions

func ChecksumPages added in v0.3.14 

func ChecksumPages(dbPath string, pageSize, nPages, nWorkers uint32, checksums []Checksum) (uint32, error)

ChecksumPages updates the provided checksums slice with the checksum of each page in the specified file. The first (by page number) error encountered is returned along with the number of the last page successfully checksummed. Checksums for subsequent pages may be updated, regardless of an error being returned.

nWorkers specifies the amount of parallelism to use. A reasonable default will be used if nWorkers is 0.

func FormatFilename 

func FormatFilename(minTXID, maxTXID TXID) string

FormatFilename returns an LTX filename representing a range of transactions.

func FormatTimestamp added in v0.3.12 

func FormatTimestamp(t time.Time) string

FormatTimestamp returns t with a fixed-width, millisecond-resolution UTC format.

func IsValidHeaderFlags 

func IsValidHeaderFlags(flags uint32) bool

IsValidHeaderFlags returns true unless flags outside the valid mask are set.

func IsValidPageSize 

func IsValidPageSize(sz uint32) bool

IsValidPageSize returns true if sz is between 512 and 64K and a power of two.

func LockPgno added in v0.2.11 

func LockPgno(pageSize uint32) uint32

LockPgno returns the page number where the PENDING_BYTE exists.

func NewHasher added in v0.2.10 

func NewHasher() hash.Hash64

NewHasher returns a new CRC64-ISO hasher.

func ParseTimestamp added in v0.3.12 

func ParseTimestamp(value string) (time.Time, error)

ParseTimestamp parses a timestamp as RFC3339Milli (fixed-width) but will fallback to RFC3339Nano if it fails. This is to support timestamps written before the introduction of the standard time format.

Types

type Checksum added in v0.3.13 

type Checksum uint64

Checksum represents an LTX checksum. 

const ChecksumFlag Checksum = 1 << 63

ChecksumFlag is a flag on the checksum to ensure it is non-zero.

func ChecksumPage 

func ChecksumPage(pgno uint32, data []byte) Checksum

ChecksumPage returns a CRC64 checksum that combines the page number & page data.

func ChecksumPageWithHasher added in v0.2.10 

func ChecksumPageWithHasher(h hash.Hash64, pgno uint32, data []byte) Checksum

ChecksumPageWithHasher returns a CRC64 checksum that combines the page number & page data.

func ChecksumReader added in v0.2.1 

func ChecksumReader(r io.Reader, pageSize int) (Checksum, error)

ChecksumReader reads an entire database file from r and computes its rolling checksum.

func ParseChecksum added in v0.3.13 

func ParseChecksum(s string) (Checksum, error)

ParseChecksum parses a 16-character hex string into a checksum.

func (Checksum) MarshalJSON added in v0.3.13 

func (c Checksum) MarshalJSON() ([]byte, error)

func (Checksum) String added in v0.3.13 

func (c Checksum) String() string

String returns c formatted as a fixed-width hex number.

func (*Checksum) UnmarshalJSON added in v0.3.13 

func (c *Checksum) UnmarshalJSON(data []byte) (err error)

type Compactor 

type Compactor struct {

	// These flags will be set when encoding the header.
	HeaderFlags uint32

	// If true, the compactor will not validate that input files have contiguous
	// transaction IDs. This is false by default but can be enabled when
	// rebuilding snapshots with missing transactions.
	AllowNonContiguousTXIDs bool
	// contains filtered or unexported fields
}

Compactor represents a compactor of LTX files.

func NewCompactor 

func NewCompactor(w io.Writer, rdrs []io.Reader) *Compactor

NewCompactor returns a new instance of Compactor with default settings.

func (*Compactor) Compact 

func (c *Compactor) Compact(ctx context.Context) (retErr error)

Compact merges the input readers into a single LTX writer.

func (*Compactor) Header added in v0.3.3 

func (c *Compactor) Header() Header

Header returns the LTX header of the compacted file. Only valid after successful Compact().

func (*Compactor) Trailer added in v0.3.3 

func (c *Compactor) Trailer() Trailer

Trailer returns the LTX trailer of the compacted file. Only valid after successful Compact().

type Decoder added in v0.2.2 

type Decoder struct {
	// contains filtered or unexported fields
}

Decoder represents a decoder of an LTX file.

func NewDecoder added in v0.2.2 

func NewDecoder(r io.Reader) *Decoder

NewDecoder returns a new instance of Decoder.

func (*Decoder) Close added in v0.2.2 

func (dec *Decoder) Close() error

Close verifies the reader is at the end of the file and that the checksum matches.

func (*Decoder) DecodeDatabaseTo added in v0.3.4 

func (dec *Decoder) DecodeDatabaseTo(w io.Writer) error

DecodeDatabaseTo decodes the LTX file as a SQLite database to w. The LTX file MUST be a snapshot file.

func (*Decoder) DecodeHeader added in v0.2.2 

func (dec *Decoder) DecodeHeader() error

DecodeHeader reads the LTX file header frame and stores it internally. Call Header() to retrieve the header after this is successfully called.

func (*Decoder) DecodePage added in v0.2.2 

func (dec *Decoder) DecodePage(hdr *PageHeader, data []byte) error

DecodePage reads the next page header into hdr and associated page data.

func (*Decoder) Header added in v0.2.2 

func (dec *Decoder) Header() Header

Header returns a copy of the header.

func (*Decoder) N added in v0.2.2 

func (dec *Decoder) N() int64

N returns the number of bytes read.

func (*Decoder) PageN added in v0.3.0 

func (dec *Decoder) PageN() int

PageN returns the number of pages read.

func (*Decoder) PostApplyPos added in v0.3.6 

func (dec *Decoder) PostApplyPos() Pos

PostApplyPos returns the replication position after underlying the LTX file is applied. Only valid after successful Close().

func (*Decoder) Trailer added in v0.2.2 

func (dec *Decoder) Trailer() Trailer

Trailer returns a copy of the trailer. File checksum available after Close().

func (*Decoder) Verify added in v0.2.2 

func (dec *Decoder) Verify() error

Verify reads the entire file. Header & trailer can be accessed via methods after the file is successfully verified. All other data is discarded.

type Encoder added in v0.2.2 

type Encoder struct {
	// contains filtered or unexported fields
}

Encoder implements a encoder for an LTX file.

func NewEncoder added in v0.2.2 

func NewEncoder(w io.Writer) *Encoder

NewEncoder returns a new instance of Encoder.

func (*Encoder) Close added in v0.2.2 

func (enc *Encoder) Close() error

Close flushes the checksum to the header.

func (*Encoder) EncodeHeader added in v0.2.2 

func (enc *Encoder) EncodeHeader(hdr Header) error

EncodeHeader writes hdr to the file's header block.

func (*Encoder) EncodePage added in v0.2.2 

func (enc *Encoder) EncodePage(hdr PageHeader, data []byte) (err error)

EncodePage writes hdr & data to the file's page block.

func (*Encoder) Header added in v0.2.2 

func (enc *Encoder) Header() Header

Header returns a copy of the header.

func (*Encoder) N added in v0.2.2 

func (enc *Encoder) N() int64

N returns the number of bytes written.

func (*Encoder) PostApplyPos added in v0.3.6 

func (enc *Encoder) PostApplyPos() Pos

PostApplyPos returns the replication position after underlying the LTX file is applied. Only valid after successful Close().

func (*Encoder) SetPostApplyChecksum added in v0.2.2 

func (enc *Encoder) SetPostApplyChecksum(chksum Checksum)

SetPostApplyChecksum sets the post-apply checksum of the database. Must call before Close().

func (*Encoder) Trailer added in v0.2.2 

func (enc *Encoder) Trailer() Trailer

Trailer returns a copy of the trailer. File checksum available after Close().

type FileSpec 

type FileSpec struct {
	Header  Header
	Pages   []PageSpec
	Trailer Trailer
}

FileSpec is an in-memory representation of an LTX file. Typically used for testing.

func (*FileSpec) GoString added in v0.3.1 

func (s *FileSpec) GoString() string

GoString returns the Go representation of s.

func (*FileSpec) ReadFrom 

func (s *FileSpec) ReadFrom(src io.Reader) (n int, err error)

ReadFromFile encodes a file spec to a file. Always return n of zero.

func (*FileSpec) WriteTo 

func (s *FileSpec) WriteTo(dst io.Writer) (n int64, err error)

Write encodes a file spec to a file. 

type Header struct {
	Version          int      // based on magic
	Flags            uint32   // reserved flags
	PageSize         uint32   // page size, in bytes
	Commit           uint32   // db size after transaction, in pages
	MinTXID          TXID     // minimum transaction ID
	MaxTXID          TXID     // maximum transaction ID
	Timestamp        int64    // milliseconds since unix epoch
	PreApplyChecksum Checksum // rolling checksum of database before applying this LTX file
	WALOffset        int64    // file offset from original WAL; zero if journal
	WALSize          int64    // size of original WAL segment; zero if journal
	WALSalt1         uint32   // header salt-1 from original WAL; zero if journal or compaction
	WALSalt2         uint32   // header salt-2 from original WAL; zero if journal or compaction
	NodeID           uint64   // node id where the LTX file was created, zero if unset
}

Header represents the header frame of an LTX file.

func DecodeHeader added in v0.3.0 

func DecodeHeader(r io.Reader) (hdr Header, data []byte, err error)

DecodeHeader decodes the header from r. Returns the header & read bytes.

func PeekHeader added in v0.3.7 

func PeekHeader(r io.Reader) (Header, io.Reader, error)

PeekHeader reads & unmarshals the header from r. It returns a new io.Reader that prepends the header data back on.

func (*Header) IsSnapshot 

func (h *Header) IsSnapshot() bool

IsSnapshot returns true if header represents a complete database snapshot. This is true if the header includes the initial transaction. Snapshots must include all pages in the database.

func (*Header) LockPgno added in v0.3.11 

func (h *Header) LockPgno() uint32

LockPgno returns the lock page number based on the header's page size.

func (*Header) MarshalBinary 

func (h *Header) MarshalBinary() ([]byte, error)

MarshalBinary encodes h to a byte slice.

func (Header) PreApplyPos added in v0.3.6 

func (h Header) PreApplyPos() Pos

PreApplyPos returns the replication position before the LTX file is applies.

func (*Header) UnmarshalBinary 

func (h *Header) UnmarshalBinary(b []byte) error

UnmarshalBinary decodes h from a byte slice.

func (*Header) Validate 

func (h *Header) Validate() error

Validate returns an error if h is invalid. 

type PageHeader struct {
	Pgno uint32
}

PageHeader represents the header for a single page in an LTX file.

func (*PageHeader) IsZero added in v0.2.0 

func (h *PageHeader) IsZero() bool

IsZero returns true if the header is empty.

func (*PageHeader) MarshalBinary 

func (h *PageHeader) MarshalBinary() ([]byte, error)

MarshalBinary encodes h to a byte slice.

func (*PageHeader) UnmarshalBinary 

func (h *PageHeader) UnmarshalBinary(b []byte) error

UnmarshalBinary decodes h from a byte slice.

func (*PageHeader) Validate 

func (h *PageHeader) Validate() error

Validate returns an error if h is invalid.

type PageSpec added in v0.2.0 

type PageSpec struct {
	Header PageHeader
	Data   []byte
}

PageSpec is an in-memory representation of an LTX page frame. Typically used for testing.

func (*PageSpec) GoString added in v0.3.1 

func (s *PageSpec) GoString() string

GoString returns the Go representation of s.

type Pos added in v0.3.1 

type Pos struct {
	TXID              TXID
	PostApplyChecksum Checksum
}

Pos represents the transactional position of a database.

func NewPos added in v0.3.3 

func NewPos(txID TXID, postApplyChecksum Checksum) Pos

NewPos returns a new instance of Pos.

func ParsePos added in v0.3.10 

func ParsePos(s string) (Pos, error)

ParsePos parses Pos from its string representation.

func (Pos) IsZero added in v0.3.1 

func (p Pos) IsZero() bool

IsZero returns true if the position is empty.

func (Pos) String added in v0.3.1 

func (p Pos) String() string

String returns a string representation of the position.

type PosMismatchError added in v0.3.1 

type PosMismatchError struct {
	Pos Pos `json:"pos"`
}

PosMismatchError is returned when an LTX file is not contiguous with the current position.

func NewPosMismatchError added in v0.3.1 

func NewPosMismatchError(pos Pos) *PosMismatchError

NewPosMismatchError returns a new instance of PosMismatchError.

func (*PosMismatchError) Error added in v0.3.1 

func (e *PosMismatchError) Error() string

Error returns the string representation of the error.

type TXID added in v0.3.2 

type TXID uint64

TXID represents a transaction ID.

func ParseFilename 

func ParseFilename(name string) (minTXID, maxTXID TXID, err error)

ParseFilename parses a transaction range from an LTX file.

func ParseTXID added in v0.2.2 

func ParseTXID(s string) (TXID, error)

ParseTXID parses a 16-character hex string into a transaction ID.

func (TXID) MarshalJSON added in v0.3.2 

func (t TXID) MarshalJSON() ([]byte, error)

func (TXID) String added in v0.3.2 

func (t TXID) String() string

String returns id formatted as a fixed-width hex number.

func (*TXID) UnmarshalJSON added in v0.3.2 

func (t *TXID) UnmarshalJSON(data []byte) (err error)

type Trailer added in v0.2.0 

type Trailer struct {
	PostApplyChecksum Checksum // rolling checksum of database after this LTX file is applied
	FileChecksum      Checksum // crc64 checksum of entire file
}

Trailer represents the ending frame of an LTX file.

func (*Trailer) MarshalBinary added in v0.2.0 

func (t *Trailer) MarshalBinary() ([]byte, error)

MarshalBinary encodes h to a byte slice.

func (*Trailer) UnmarshalBinary added in v0.2.0 

func (t *Trailer) UnmarshalBinary(b []byte) error

UnmarshalBinary decodes h from a byte slice.

func (*Trailer) Validate added in v0.2.0 

func (t *Trailer) Validate() error

Validate returns an error if t is invalid.

Source Files

View all Source files

Directories

Path Synopsis
cmd
ltx command

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.