ltx

package module
v0.1.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jul 1, 2022 License: Apache-2.0 Imports: 12 Imported by: 32

README

Liteserver Transaction File (LTX)

The LTX file format provides a way to store SQLite transactional data in a way that can be encrypted and compacted and is optimized for performance. It also provides built-in support for simple event data associated with each transaction.

File Format

An LTX file is composed of 2 blocks:

  1. Header block
  2. Page data block

The header block contains a header frame, event frames, and page headers. The page data block contains raw page data and is page-aligned. These two blocks are separated to allow them to be read & written separately to reduce memory usage.

Header block

The header block is composed of 3 sections:

  1. Header frame
  2. Event frames
  3. Page headers
Header frame

The header provides information about the number of data and event frames as well as database information such as the page size and database size. LTX files can be compacted together so each file contains the transaction ID (TXID) range that it represents. A timestamp provides users with a rough approximation of the time the transaction occurred and the checksum provides a basic integrity check.

Offset Size Description
0 4 Magic number. Always "LTX1".
4 4 Flags. Reserved. Always 0.
8 4 Page size, in bytes.
12 4 Event count.
16 4 Total event data size.
20 4 Page count.
24 4 Size of DB after transaction, in pages.
28 8 Minimum transaction ID.
36 8 Maximum transaction ID.
44 8 Timestamp (seconds since Unix epoch)
52 8 Header checksum (CRC-ISO-64)
60 8 Page data checksum (CRC-ISO-64)
Event frames

These frames store the size of the event payload, in bytes, as well as the nonce and authentication tag which is used for AES-GCM-256 encryption. The event frame data follows afterward for the number of bytes specified by the size field.

Offset Size Description
0 4 Event data size, in bytes.
4 12 AES-GCM nonce.
16 16 AES-GCM authentication tag.
32 * Event data
Page headers

This header stores the page number associated with the page data as well as the nonce & authentication tag which is used for AES-GCM-256 encryption.

Offset Size Description
0 4 Page number.
4 12 AES-GCM nonce.
16 16 AES-GCM authentication tag.
Page data block

The page data block holds encrypted pages in the same order the page headers. This block is aligned to the page size specified in the header. The size can be calculated as the number of pages specified in the header frame multiplied by the page size.

Documentation

Overview

Package ltx reads and writes Liteserver Transaction (LTX) files.

Index

Constants

View Source 
const (
	// Magic is the first 4 bytes of every LTX file.
	Magic = "LTX1"

	// Version is the current version of the LTX file format.
	Version = 1
)
View Source 
const (
	HeaderSize      = 96
	PageHeaderSize  = 32
	EventHeaderSize = 32
)

Header size constants.

View Source 
const (
	ChecksumSize = 8

	HeaderChecksumOffset      = HeaderSize - ChecksumSize
	PageBlockChecksumOffset   = HeaderChecksumOffset - ChecksumSize
	HeaderBlockChecksumOffset = PageBlockChecksumOffset - ChecksumSize
)

Checksum size & positions.

View Source 
const (
	// ChecksumFlag is a flag on the checksum to ensure it is non-zero.
	ChecksumFlag uint64 = 1 << 63

	// ChecksumMask is the mask of the bits used for the page checksum.
	ChecksumMask uint64 = (1 << 63) - 1
)

Variables

View Source 
var (
	ErrInvalidFile  = errors.New("invalid LTX file")
	ErrReaderClosed = errors.New("reader closed")
	ErrWriterClosed = errors.New("writer closed")

	ErrNoHeaderChecksum            = errors.New("no header checksum")
	ErrInvalidHeaderChecksumFormat = errors.New("invalid header checksum format")
	ErrHeaderChecksumMismatch      = errors.New("header checksum mismatch")

	ErrHeaderBlockChecksumMismatch = errors.New("header block checksum mismatch")
	ErrPageBlockChecksumMismatch   = errors.New("header page checksum mismatch")
)

Errors

Functions

func ChecksumPage 

func ChecksumPage(pgno uint32, data []byte) uint64

ChecksumPage returns a CRC64 checksum that combines the page number & page data.

func FormatFilename 

func FormatFilename(minTXID, maxTXID uint64) string

FormatFilename returns an LTX filename representing a range of transactions.

func FormatTXID 

func FormatTXID(id uint64) string

FormatTXID returns id formatted as a fixed-width hex number.

func FormatTXIDRange 

func FormatTXIDRange(min, max uint64) string

FormatTXIDRange returns min & max formatted as a single number if equal or a range if different.

func IsValidHeaderFlags 

func IsValidHeaderFlags(flags uint32) bool

IsValidHeaderFlags returns true if flags are unset. Flags are reserved.

func IsValidPageSize 

func IsValidPageSize(sz uint32) bool

IsValidPageSize returns true if sz is between 512 and 64K and a power of two.

func PageAlign 

func PageAlign(v int64, pageSize uint32) int64

PageAlign returns v if it a multiple of pageSize. Otherwise returns next multiple of pageSize.

func ParseFilename 

func ParseFilename(name string) (minTXID, maxTXID uint64, err error)

ParseFilename parses a transaction range from an LTX file.

func UniquePageN 

func UniquePageN(filenames []string) (pageN uint32, err error)

UniquePageN returns the unique page number count of a set of LTX files.

Types

type Compactor 

type Compactor struct {
	// If true, the compactor calls fsync() after a successful compaction.
	Sync bool

	// If true, the output file will include all events from input files.
	IncludeEvents bool
}

Compactor represents a compactor of LTX files.

func NewCompactor 

func NewCompactor() *Compactor

NewCompactor returns a new instance of Compactor with default settings.

func (*Compactor) Compact 

func (c *Compactor) Compact(outputFilename string, inputFilenames []string) (retErr error)

Compact merges the LTX files specified in inputFilenames and combines them into a single LTX file which is written to outputFilename.

type EventHeader 

type EventHeader struct {
	Size  uint32
	Nonce [12]byte
	Tag   [16]byte
}

EventHeader represents the header for a single event frame in an LTX file.

func (*EventHeader) MarshalBinary 

func (h *EventHeader) MarshalBinary() ([]byte, error)

MarshalBinary encodes h to a byte slice.

func (*EventHeader) UnmarshalBinary 

func (h *EventHeader) UnmarshalBinary(b []byte) error

UnmarshalBinary decodes h from a byte slice.

func (*EventHeader) Validate 

func (h *EventHeader) Validate() error

Validate returns an error if h is invalid.

type FileSpec 

type FileSpec struct {
	Header       Header
	PageHeaders  []PageHeader
	EventHeaders []EventHeader
	EventData    [][]byte
	PageData     [][]byte
}

FileSpec is an in-memory representation of an LTX file. Typically used for testing.

func (*FileSpec) Bytes 

func (s *FileSpec) Bytes() ([]byte, error)

Bytes returns the encoded file spec.

func (*FileSpec) MustBytes 

func (s *FileSpec) MustBytes() []byte

MustBytes returns the encoded file spec. Panic on error.

func (*FileSpec) ReadFrom 

func (s *FileSpec) ReadFrom(r io.Reader) (n int, err error)

ReadFromFile encodes a file spec to a file. Always return n of zero.

func (*FileSpec) WriteTo 

func (s *FileSpec) WriteTo(w io.Writer) (n int64, err error)

Write encodes a file spec to a file. 

type Header struct {
	Version             int    // based on magic
	Flags               uint32 // reserved flags
	PageSize            uint32 // page size, in bytes
	PageN               uint32 // page count in ltx file
	EventN              uint32 // event count in ltx file
	EventDataSize       uint32 // total size of all event data
	Commit              uint32 // db size after transaction, in pages
	DBID                uint32 // database ID
	MinTXID             uint64 // minimum transaction ID
	MaxTXID             uint64 // maximum transaction ID
	Timestamp           uint64 // seconds since unix epoch
	PreChecksum         uint64 // checksum of database at previous transaction
	PostChecksum        uint64 // checksum of database after this LTX file is applied
	HeaderBlockChecksum uint64 // crc64 checksum of header block, excluding header
	PageBlockChecksum   uint64 // crc64 checksum of page block
	HeaderChecksum      uint64 // crc64 checksum of header
}

Header represents the header frame of an LTX file.

func (*Header) HeaderBlockSize 

func (h *Header) HeaderBlockSize() int64

HeaderBlockSize returns the total size of the header block, in bytes. Must be a valid header frame.

func (*Header) IsSnapshot 

func (h *Header) IsSnapshot() bool

IsSnapshot returns true if header represents a complete database snapshot. This is true if the header includes the initial transaction. Snapshots must include all pages in the database.

func (*Header) MarshalBinary 

func (h *Header) MarshalBinary() ([]byte, error)

MarshalBinary encodes h to a byte slice.

func (*Header) Prevalidate 

func (h *Header) Prevalidate() error

Prevalidate returns an error if h is invalid. This function does not check calculated fields, which includes most of the checksums. It is called when initializing a writer so that we can perform basic validation before writing a lot of data.

func (*Header) UnmarshalBinary 

func (h *Header) UnmarshalBinary(b []byte) error

UnmarshalBinary decodes h from a byte slice.

func (*Header) Validate 

func (h *Header) Validate() error

Validate returns an error if h is invalid. This checks all fields including calculated fields such as checksums.

type HeaderBlockReader 

type HeaderBlockReader struct {
	// contains filtered or unexported fields
}

func NewHeaderBlockReader 

func NewHeaderBlockReader(r io.Reader) *HeaderBlockReader

func (*HeaderBlockReader) Checksum 

func (r *HeaderBlockReader) Checksum() uint64

Checksum returns the checksum of the file. Only valid after close.

func (*HeaderBlockReader) Close 

func (r *HeaderBlockReader) Close() error

Close verifies the reader is at the end of the file and that the checksum matches.

func (*HeaderBlockReader) Read 

func (r *HeaderBlockReader) Read(p []byte) (n int, err error)

Read reads bytes for an event or page data frame. Only valid after a call to ReadEventHeader() or ReadPageHeader().

func (*HeaderBlockReader) ReadEventHeader 

func (r *HeaderBlockReader) ReadEventHeader(hdr *EventHeader) error

ReadEventHeader returns true if more events are available.

func (*HeaderBlockReader) ReadHeader 

func (r *HeaderBlockReader) ReadHeader(hdr *Header) error

ReadHeader returns the LTX file header frame.

func (*HeaderBlockReader) ReadPageHeader 

func (r *HeaderBlockReader) ReadPageHeader(hdr *PageHeader) error

ReadPageHeader reads the next page header into hdr and initializes reader to read associated page data.

type HeaderBlockWriter 

type HeaderBlockWriter struct {
	// contains filtered or unexported fields
}

HeaderBlockWriter implements a writer for the header block of an LTX file.

func NewHeaderBlockWriter 

func NewHeaderBlockWriter(w io.WriteSeeker) *HeaderBlockWriter

NewHeaderBlockWriter returns a new instance of HeaderBlockWriter.

func (*HeaderBlockWriter) Close 

func (w *HeaderBlockWriter) Close() error

Close flushes the checksum to the header.

func (*HeaderBlockWriter) Header 

func (w *HeaderBlockWriter) Header() Header

Header returns the header. Checksums are only applied after Close().

func (*HeaderBlockWriter) SetPageBlockChecksum 

func (w *HeaderBlockWriter) SetPageBlockChecksum(chksum uint64)

SetPageBlockChecksum sets the checksum of the page block. Must call after WriteHeaderFrame() & before Close().

func (*HeaderBlockWriter) SetPostChecksum 

func (w *HeaderBlockWriter) SetPostChecksum(chksum uint64)

SetPostChecksum sets the post-apply checksum of the database. Must call after WriteHeaderFrame() & before Close().

func (*HeaderBlockWriter) Write 

func (w *HeaderBlockWriter) Write(p []byte) (n int, err error)

Write writes data to a single event. Should only be called after a successful call to WriteEventHeader().

func (*HeaderBlockWriter) WriteEventHeader 

func (w *HeaderBlockWriter) WriteEventHeader(hdr EventHeader) (err error)

WriteEventHeader writes hdr to the file's event block.

func (*HeaderBlockWriter) WriteHeader 

func (w *HeaderBlockWriter) WriteHeader(hdr Header) error

WriteHeader writes hdr to the file's header block.

func (*HeaderBlockWriter) WritePageHeader 

func (w *HeaderBlockWriter) WritePageHeader(hdr PageHeader) (err error)

WritePageHeader writes hdr to the file's page header block.

type PageBlockReader 

type PageBlockReader struct {
	// contains filtered or unexported fields
}

func NewPageBlockReader 

func NewPageBlockReader(r io.Reader, pageN, pageSize uint32, checksum uint64) *PageBlockReader

func (*PageBlockReader) Checksum 

func (r *PageBlockReader) Checksum() uint64

Checksum returns the calculated checksum of the page data read.

func (*PageBlockReader) Close 

func (r *PageBlockReader) Close() error

Close verifies the reader is at the end of the file and that the checksum matches.

func (*PageBlockReader) Read 

func (r *PageBlockReader) Read(p []byte) (n int, err error)

func (*PageBlockReader) Remaining 

func (r *PageBlockReader) Remaining() int64

Remaining returns the number of bytes to be read from the reader.

func (*PageBlockReader) Size 

func (r *PageBlockReader) Size() int64

Size returns the expected number of bytes to be read from the reader.

type PageBlockWriter 

type PageBlockWriter struct {
	// contains filtered or unexported fields
}

PageBlockWriter implements a writer for the page data block of the LTX file format.

func NewPageBlockWriter 

func NewPageBlockWriter(w io.Writer, pageN, pageSize uint32) *PageBlockWriter

NewPageBlockWriter returns a new instance of PageBlockWriter.

func (*PageBlockWriter) Checksum 

func (w *PageBlockWriter) Checksum() uint64

Checksum returns the checksum of the header. Only valid after close.

func (*PageBlockWriter) Close 

func (w *PageBlockWriter) Close() error

Close returns an error if all bytes have not been written to the writer.

func (*PageBlockWriter) Remaining 

func (w *PageBlockWriter) Remaining() int64

Remaining returns the number of bytes to be written to the writer.

func (*PageBlockWriter) Size 

func (w *PageBlockWriter) Size() int64

Size returns the expected number of bytes to be written to the writer.

func (*PageBlockWriter) Write 

func (w *PageBlockWriter) Write(p []byte) (n int, err error)

Write writes data to the file's page data block. 

type PageHeader struct {
	Pgno  uint32
	Nonce [12]byte
	Tag   [16]byte
}

PageHeader represents the header for a single page in an LTX file.

func (*PageHeader) MarshalBinary 

func (h *PageHeader) MarshalBinary() ([]byte, error)

MarshalBinary encodes h to a byte slice.

func (*PageHeader) UnmarshalBinary 

func (h *PageHeader) UnmarshalBinary(b []byte) error

UnmarshalBinary decodes h from a byte slice.

func (*PageHeader) Validate 

func (h *PageHeader) Validate() error

Validate returns an error if h is invalid.

Source Files

View all Source files

Directories

Path Synopsis

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.