ltx

package module
v0.2.5 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Nov 17, 2022 License: Apache-2.0 Imports: 10 Imported by: 32

README

Lite Transaction File (LTX)

The LTX file format provides a way to store SQLite transactional data in a way that can be encrypted and compacted and is optimized for performance.

File Format

An LTX file is composed of several sections:

  1. Header
  2. Page block
  3. Trailer

The header contains metadata about the file, the page block contains page frames, and the trailer contains checksums of the file and the database end state.

Header

The header provides information about the number of page frames as well as database information such as the page size and database size. LTX files can be compacted together so each file contains the transaction ID (TXID) range that it represents. A timestamp provides users with a rough approximation of the time the transaction occurred and the checksum provides a basic integrity check.

Offset Size Description
0 4 Magic number. Always "LTX1".
4 4 Flags. Reserved. Always 0.
8 4 Page size, in bytes.
12 4 Size of DB after transaction, in pages.
16 4 Database ID.
20 8 Minimum transaction ID.
28 8 Maximum transaction ID.
36 8 Timestamp (Milliseconds since epoch)
44 8 Pre-apply DB checksum (CRC-ISO-64)
52 48 Reserved.
Page block

This block stores a series of page headers and page data.

Offset Size Description
0 4 Page number.
4 N Page data.
Trailer

The trailer provides checksum for the LTX file data, a rolling checksum of the database state after the LTX file is applied, and the checksum of the trailer itself.

Offset Size Description
0 8 Post-apply DB checksum (CRC-ISO-64)
8 8 File checksum (CRC-ISO-64)

Documentation

Overview

Package ltx reads and writes Liteserver Transaction (LTX) files.

Index

Constants

View Source 
const (
	// Magic is the first 4 bytes of every LTX file.
	Magic = "LTX1"

	// Version is the current version of the LTX file format.
	Version = 1
)
View Source 
const (
	HeaderSize     = 100
	PageHeaderSize = 4
	TrailerSize    = 16
)

Size constants.

View Source 
const (
	ChecksumSize          = 8
	TrailerChecksumOffset = TrailerSize - ChecksumSize
)

Checksum size & positions.

View Source 
const (
	// ChecksumFlag is a flag on the checksum to ensure it is non-zero.
	ChecksumFlag uint64 = 1 << 63

	// ChecksumMask is the mask of the bits used for the page checksum.
	ChecksumMask uint64 = (1 << 63) - 1
)
View Source 
const MaxPageSize = 65536

MaxPageSize is the maximum allowed page size for SQLite.

Variables

View Source 
var (
	ErrInvalidFile   = errors.New("invalid LTX file")
	ErrDecoderClosed = errors.New("ltx decoder closed")
	ErrEncoderClosed = errors.New("ltx encoder closed")

	ErrNoChecksum            = errors.New("no file checksum")
	ErrInvalidChecksumFormat = errors.New("invalid file checksum format")
	ErrChecksumMismatch      = errors.New("file checksum mismatch")
)

Errors

Functions

func ChecksumPage 

func ChecksumPage(pgno uint32, data []byte) uint64

ChecksumPage returns a CRC64 checksum that combines the page number & page data.

func ChecksumReader added in v0.2.1 

func ChecksumReader(r io.Reader, pageSize int) (uint64, error)

ChecksumReader reads an entire database file from r and computes its rolling checksum.

func FormatFilename 

func FormatFilename(minTXID, maxTXID uint64) string

FormatFilename returns an LTX filename representing a range of transactions.

func FormatTXID 

func FormatTXID(id uint64) string

FormatTXID returns id formatted as a fixed-width hex number.

func IsValidHeaderFlags 

func IsValidHeaderFlags(flags uint32) bool

IsValidHeaderFlags returns true if flags are unset. Flags are reserved.

func IsValidPageSize 

func IsValidPageSize(sz uint32) bool

IsValidPageSize returns true if sz is between 512 and 64K and a power of two.

func ParseFilename 

func ParseFilename(name string) (minTXID, maxTXID uint64, err error)

ParseFilename parses a transaction range from an LTX file.

func ParseTXID added in v0.2.2 

func ParseTXID(s string) (uint64, error)

ParseTXID parses a 16-character hex string into a transaction ID.

Types

type Compactor 

type Compactor struct {
	// contains filtered or unexported fields
}

Compactor represents a compactor of LTX files.

func NewCompactor 

func NewCompactor(w io.Writer, rdrs []io.Reader) *Compactor

NewCompactor returns a new instance of Compactor with default settings.

func (*Compactor) Compact 

func (c *Compactor) Compact(ctx context.Context) (retErr error)

Compact merges the input readers into a single LTX writer.

type Decoder added in v0.2.2 

type Decoder struct {
	// contains filtered or unexported fields
}

Decoder represents a decoder of an LTX file.

func NewDecoder added in v0.2.2 

func NewDecoder(r io.Reader) *Decoder

NewDecoder returns a new instance of Decoder.

func (*Decoder) Close added in v0.2.2 

func (dec *Decoder) Close() error

Close verifies the reader is at the end of the file and that the checksum matches.

func (*Decoder) DecodeHeader added in v0.2.2 

func (dec *Decoder) DecodeHeader() error

DecodeHeader reads the LTX file header frame and stores it internally. Call Header() to retrieve the header after this is successfully called.

func (*Decoder) DecodePage added in v0.2.2 

func (dec *Decoder) DecodePage(hdr *PageHeader, data []byte) error

DecodePage reads the next page header into hdr and associated page data.

func (*Decoder) Header added in v0.2.2 

func (dec *Decoder) Header() Header

Header returns a copy of the header.

func (*Decoder) N added in v0.2.2 

func (dec *Decoder) N() int64

N returns the number of bytes read.

func (*Decoder) Trailer added in v0.2.2 

func (dec *Decoder) Trailer() Trailer

Trailer returns a copy of the trailer. File checksum available after Close().

func (*Decoder) Verify added in v0.2.2 

func (dec *Decoder) Verify() (Header, Trailer, error)

Verify reads the entire file and returns the header & trailer. All page data is discarded.

type Encoder added in v0.2.2 

type Encoder struct {
	// contains filtered or unexported fields
}

Encoder implements a encoder for an LTX file.

func NewEncoder added in v0.2.2 

func NewEncoder(w io.Writer) *Encoder

NewEncoder returns a new instance of Encoder.

func (*Encoder) Close added in v0.2.2 

func (enc *Encoder) Close() error

Close flushes the checksum to the header.

func (*Encoder) EncodeHeader added in v0.2.2 

func (enc *Encoder) EncodeHeader(hdr Header) error

EncodeHeader writes hdr to the file's header block.

func (*Encoder) EncodePage added in v0.2.2 

func (enc *Encoder) EncodePage(hdr PageHeader, data []byte) (err error)

EncodePage writes hdr & data to the file's page block.

func (*Encoder) Header added in v0.2.2 

func (enc *Encoder) Header() Header

Header returns a copy of the header.

func (*Encoder) N added in v0.2.2 

func (enc *Encoder) N() int64

N returns the number of bytes written.

func (*Encoder) SetPostApplyChecksum added in v0.2.2 

func (enc *Encoder) SetPostApplyChecksum(chksum uint64)

SetPostApplyChecksum sets the post-apply checksum of the database. Must call before Close().

func (*Encoder) Trailer added in v0.2.2 

func (enc *Encoder) Trailer() Trailer

Trailer returns a copy of the trailer. File checksum available after Close().

type FileSpec 

type FileSpec struct {
	Header  Header
	Pages   []PageSpec
	Trailer Trailer
}

FileSpec is an in-memory representation of an LTX file. Typically used for testing.

func (*FileSpec) ReadFrom 

func (s *FileSpec) ReadFrom(src io.Reader) (n int, err error)

ReadFromFile encodes a file spec to a file. Always return n of zero.

func (*FileSpec) WriteTo 

func (s *FileSpec) WriteTo(dst io.Writer) (n int64, err error)

Write encodes a file spec to a file. 

type Header struct {
	Version          int    // based on magic
	Flags            uint32 // reserved flags
	PageSize         uint32 // page size, in bytes
	Commit           uint32 // db size after transaction, in pages
	MinTXID          uint64 // minimum transaction ID
	MaxTXID          uint64 // maximum transaction ID
	Timestamp        uint64 // seconds since unix epoch
	PreApplyChecksum uint64 // rolling checksum of database before applying this LTX file
}

Header represents the header frame of an LTX file.

func (*Header) IsSnapshot 

func (h *Header) IsSnapshot() bool

IsSnapshot returns true if header represents a complete database snapshot. This is true if the header includes the initial transaction. Snapshots must include all pages in the database.

func (*Header) MarshalBinary 

func (h *Header) MarshalBinary() ([]byte, error)

MarshalBinary encodes h to a byte slice.

func (*Header) UnmarshalBinary 

func (h *Header) UnmarshalBinary(b []byte) error

UnmarshalBinary decodes h from a byte slice.

func (*Header) Validate 

func (h *Header) Validate() error

Validate returns an error if h is invalid. 

type PageHeader struct {
	Pgno uint32
}

PageHeader represents the header for a single page in an LTX file.

func (*PageHeader) IsZero added in v0.2.0 

func (h *PageHeader) IsZero() bool

IsZero returns true if the header is empty.

func (*PageHeader) MarshalBinary 

func (h *PageHeader) MarshalBinary() ([]byte, error)

MarshalBinary encodes h to a byte slice.

func (*PageHeader) UnmarshalBinary 

func (h *PageHeader) UnmarshalBinary(b []byte) error

UnmarshalBinary decodes h from a byte slice.

func (*PageHeader) Validate 

func (h *PageHeader) Validate() error

Validate returns an error if h is invalid.

type PageSpec added in v0.2.0 

type PageSpec struct {
	Header PageHeader
	Data   []byte
}

PageSpec is an in-memory representation of an LTX page frame. Typically used for testing.

type Reader added in v0.2.0 

type Reader struct {
	// contains filtered or unexported fields
}

Reader is a passthrough that validates the contents of the underlying reader.

func NewReader added in v0.2.0 

func NewReader(r io.Reader) *Reader

NewReader returns a new instance of Reader.

func (*Reader) Header added in v0.2.0 

func (r *Reader) Header() Header

Header returns a copy of the header. Available after DecodeHeader().

func (*Reader) PeekHeader added in v0.2.2 

func (r *Reader) PeekHeader() error

PeekHeader reads the header into a buffer and allows the caller to inspect it.

func (*Reader) Read added in v0.2.2 

func (r *Reader) Read(p []byte) (n int, err error)

Read reads bytes from the underlying reader into p. Returns io.ErrShortBuffer if len(p) is less than the size of the page frame.

func (*Reader) Trailer added in v0.2.0 

func (r *Reader) Trailer() Trailer

Trailer returns a copy of the trailer. Available after Close().

func (*Reader) WriteTo added in v0.2.2 

func (r *Reader) WriteTo(dst io.Writer) (written int64, err error)

WriteTo implements io.WriterTo. It prevents io.Copy() from using a small buffer when copying using io.ReaderFrom.

type Trailer added in v0.2.0 

type Trailer struct {
	PostApplyChecksum uint64 // rolling checksum of database after this LTX file is applied
	FileChecksum      uint64 // crc64 checksum of entire file
}

Trailer represents the ending frame of an LTX file.

func (*Trailer) MarshalBinary added in v0.2.0 

func (t *Trailer) MarshalBinary() ([]byte, error)

MarshalBinary encodes h to a byte slice.

func (*Trailer) UnmarshalBinary added in v0.2.0 

func (t *Trailer) UnmarshalBinary(b []byte) error

UnmarshalBinary decodes h from a byte slice.

func (*Trailer) Validate added in v0.2.0 

func (t *Trailer) Validate() error

Validate returns an error if t is invalid.

Source Files

View all Source files

Directories

Path Synopsis
cmd
ltx command

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.