ltx

package module
v0.2.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Aug 10, 2022 License: Apache-2.0 Imports: 10 Imported by: 32

README

Ltx Transaction File (LTX)

The LTX file format provides a way to store SQLite transactional data in a way that can be encrypted and compacted and is optimized for performance.

File Format

An LTX file is composed of several sections:

  1. Header
  2. Page block
  3. Trailer

The header contains metadata about the file, the page block contains page frames, and the trailer contains checksums of the file and the database end state.

Header

The header provides information about the number of page frames as well as database information such as the page size and database size. LTX files can be compacted together so each file contains the transaction ID (TXID) range that it represents. A timestamp provides users with a rough approximation of the time the transaction occurred and the checksum provides a basic integrity check.

Offset Size Description
0 4 Magic number. Always "LTX1".
4 4 Flags. Reserved. Always 0.
8 4 Page size, in bytes.
12 4 Page count.
16 4 Size of DB after transaction, in pages.
0 8 Minimum transaction ID.
0 8 Maximum transaction ID.
0 8 Timestamp (seconds since Unix epoch)
0 8 Pre-apply DB checksum (CRC-ISO-64)
Page block

This block stores a series of page headers and page data.

Offset Size Description
0 4 Page number.
4 12 AES-GCM nonce.
16 16 AES-GCM authentication tag.
32 ... Page data.
Trailer

The trailer provides checksum for the LTX file data, a rolling checksum of the database state after the LTX file is applied, and the checksum of the trailer itself.

Offset Size Description
0 8 File checksum (CRC-ISO-64)
8 8 Post-apply DB checksum (CRC-ISO-64)
16 8 Trailer checksum (CRC-ISO-64)

Documentation

Overview

Package ltx reads and writes Liteserver Transaction (LTX) files.

Index

Constants

View Source 
const (
	// Magic is the first 4 bytes of every LTX file.
	Magic = "LTX1"

	// Version is the current version of the LTX file format.
	Version = 1
)
View Source 
const (
	HeaderSize     = 52
	PageHeaderSize = 4
	TrailerSize    = 16
)

Size constants.

View Source 
const (
	ChecksumSize          = 8
	TrailerChecksumOffset = TrailerSize - ChecksumSize
)

Checksum size & positions.

View Source 
const (
	// ChecksumFlag is a flag on the checksum to ensure it is non-zero.
	ChecksumFlag uint64 = 1 << 63

	// ChecksumMask is the mask of the bits used for the page checksum.
	ChecksumMask uint64 = (1 << 63) - 1
)

Variables

View Source 
var (
	ErrInvalidFile  = errors.New("invalid LTX file")
	ErrReaderClosed = errors.New("reader closed")
	ErrWriterClosed = errors.New("writer closed")

	ErrNoChecksum            = errors.New("no file checksum")
	ErrInvalidChecksumFormat = errors.New("invalid file checksum format")
	ErrChecksumMismatch      = errors.New("file checksum mismatch")
)

Errors

Functions

func ChecksumPage 

func ChecksumPage(pgno uint32, data []byte) uint64

ChecksumPage returns a CRC64 checksum that combines the page number & page data.

func FormatFilename 

func FormatFilename(minTXID, maxTXID uint64) string

FormatFilename returns an LTX filename representing a range of transactions.

func FormatTXID 

func FormatTXID(id uint64) string

FormatTXID returns id formatted as a fixed-width hex number.

func FormatTXIDRange 

func FormatTXIDRange(min, max uint64) string

FormatTXIDRange returns min & max formatted as a single number if equal or a range if different.

func IsValidHeaderFlags 

func IsValidHeaderFlags(flags uint32) bool

IsValidHeaderFlags returns true if flags are unset. Flags are reserved.

func IsValidPageSize 

func IsValidPageSize(sz uint32) bool

IsValidPageSize returns true if sz is between 512 and 64K and a power of two.

func ParseFilename 

func ParseFilename(name string) (minTXID, maxTXID uint64, err error)

ParseFilename parses a transaction range from an LTX file.

Types

type Compactor 

type Compactor struct {
	// contains filtered or unexported fields
}

Compactor represents a compactor of LTX files.

func NewCompactor 

func NewCompactor(w io.Writer, rdrs []io.Reader) *Compactor

NewCompactor returns a new instance of Compactor with default settings.

func (*Compactor) Compact 

func (c *Compactor) Compact(ctx context.Context) (retErr error)

Compact merges the input readers into a single LTX writer.

type FileSpec 

type FileSpec struct {
	Header  Header
	Pages   []PageSpec
	Trailer Trailer
}

FileSpec is an in-memory representation of an LTX file. Typically used for testing.

func (*FileSpec) ReadFrom 

func (s *FileSpec) ReadFrom(src io.Reader) (n int, err error)

ReadFromFile encodes a file spec to a file. Always return n of zero.

func (*FileSpec) WriteTo 

func (s *FileSpec) WriteTo(dst io.Writer) (n int64, err error)

Write encodes a file spec to a file. 

type Header struct {
	Version          int    // based on magic
	Flags            uint32 // reserved flags
	PageSize         uint32 // page size, in bytes
	Commit           uint32 // db size after transaction, in pages
	DBID             uint32 // database ID
	MinTXID          uint64 // minimum transaction ID
	MaxTXID          uint64 // maximum transaction ID
	Timestamp        uint64 // seconds since unix epoch
	PreApplyChecksum uint64 // rolling checksum of database before applying this LTX file
}

Header represents the header frame of an LTX file.

func (*Header) IsSnapshot 

func (h *Header) IsSnapshot() bool

IsSnapshot returns true if header represents a complete database snapshot. This is true if the header includes the initial transaction. Snapshots must include all pages in the database.

func (*Header) MarshalBinary 

func (h *Header) MarshalBinary() ([]byte, error)

MarshalBinary encodes h to a byte slice.

func (*Header) UnmarshalBinary 

func (h *Header) UnmarshalBinary(b []byte) error

UnmarshalBinary decodes h from a byte slice.

func (*Header) Validate 

func (h *Header) Validate() error

Validate returns an error if h is invalid. 

type PageHeader struct {
	Pgno uint32
}

PageHeader represents the header for a single page in an LTX file.

func (*PageHeader) IsZero added in v0.2.0 

func (h *PageHeader) IsZero() bool

IsZero returns true if the header is empty.

func (*PageHeader) MarshalBinary 

func (h *PageHeader) MarshalBinary() ([]byte, error)

MarshalBinary encodes h to a byte slice.

func (*PageHeader) UnmarshalBinary 

func (h *PageHeader) UnmarshalBinary(b []byte) error

UnmarshalBinary decodes h from a byte slice.

func (*PageHeader) Validate 

func (h *PageHeader) Validate() error

Validate returns an error if h is invalid.

type PageSpec added in v0.2.0 

type PageSpec struct {
	Header PageHeader
	Data   []byte
}

PageSpec is an in-memory representation of an LTX page frame. Typically used for testing.

type Reader added in v0.2.0 

type Reader struct {
	// contains filtered or unexported fields
}

Reader represents a reader of an LTX file.

func NewReader added in v0.2.0 

func NewReader(r io.Reader) *Reader

NewReader returns a new instance of Reader.

func (*Reader) Checksum added in v0.2.0 

func (r *Reader) Checksum() uint64

Checksum returns the checksum of the file. Only valid after close.

func (*Reader) Close added in v0.2.0 

func (r *Reader) Close() error

Close verifies the reader is at the end of the file and that the checksum matches.

func (*Reader) Header added in v0.2.0 

func (r *Reader) Header() Header

Header returns a copy of the header.

func (*Reader) N added in v0.2.0 

func (r *Reader) N() int64

N returns the number of bytes read.

func (*Reader) ReadHeader added in v0.2.0 

func (r *Reader) ReadHeader() error

ReadHeader reads the LTX file header frame and stores it internally. Call Header() to retrieve the header after this is successfully called.

func (*Reader) ReadPage added in v0.2.0 

func (r *Reader) ReadPage(hdr *PageHeader, data []byte) error

ReadPage reads the next page header into hdr and associated page data.

func (*Reader) Trailer added in v0.2.0 

func (r *Reader) Trailer() Trailer

Trailer returns a copy of the trailer. File checksum available after Close().

func (*Reader) Verify added in v0.2.0 

func (r *Reader) Verify() (Header, Trailer, error)

Verify reads the entire file and returns the header & trailer. All page data is discarded.

type Trailer added in v0.2.0 

type Trailer struct {
	PostApplyChecksum uint64 // rolling checksum of database after this LTX file is applied
	FileChecksum      uint64 // crc64 checksum of entire file
}

Trailer represents the ending frame of an LTX file.

func (*Trailer) MarshalBinary added in v0.2.0 

func (t *Trailer) MarshalBinary() ([]byte, error)

MarshalBinary encodes h to a byte slice.

func (*Trailer) UnmarshalBinary added in v0.2.0 

func (t *Trailer) UnmarshalBinary(b []byte) error

UnmarshalBinary decodes h from a byte slice.

func (*Trailer) Validate added in v0.2.0 

func (t *Trailer) Validate() error

Validate returns an error if t is invalid.

type Writer added in v0.2.0 

type Writer struct {
	// contains filtered or unexported fields
}

Writer implements a writer an LTX file.

func NewWriter added in v0.2.0 

func NewWriter(w io.Writer) *Writer

NewWriter returns a new instance of Writer.

func (*Writer) Close added in v0.2.0 

func (w *Writer) Close() error

Close flushes the checksum to the header.

func (*Writer) Header added in v0.2.0 

func (w *Writer) Header() Header

Header returns a copy of the header.

func (*Writer) N added in v0.2.0 

func (w *Writer) N() int64

N returns the number of bytes written.

func (*Writer) SetPostApplyChecksum added in v0.2.0 

func (w *Writer) SetPostApplyChecksum(chksum uint64)

SetPostApplyChecksum sets the post-apply checksum of the database. Must call before Close().

func (*Writer) Trailer added in v0.2.0 

func (w *Writer) Trailer() Trailer

Trailer returns a copy of the trailer. File checksum available after Close().

func (*Writer) WriteHeader added in v0.2.0 

func (w *Writer) WriteHeader(hdr Header) error

WriteHeader writes hdr to the file's header block.

func (*Writer) WritePage added in v0.2.0 

func (w *Writer) WritePage(hdr PageHeader, data []byte) (err error)

WritePage writes hdr & data to the file's page block.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.