tinystore

Overview ¶

Package tinystore stores immutable analytical table snapshots in a compact, column-oriented format. The TINY01 magic and .tsnap extension identify version 1 of the format.

Snapshots contain table/column names, TEXT/INT/FLOAT types and values only. Constraints, defaults, declared types, indexes and statistics are not persisted. This is a data interchange format, not a replacement for a database backup or tinySQL's transactional storage. Callers must not mutate a table during Write.

Text dictionaries share repeated strings on load. Columns are encoded and decoded with bounded parallelism; CRC32C detects accidental corruption.

File layout (little endian, all counts and lengths are unsigned varints unless said):

"TINY01"                      magic, includes the format version
block 0, crc32c(block 0)      one block per column, in column order; the checksum is 4 bytes
block 1, crc32c(block 1)
...
footer, crc32c(footer)        table name, column count, row count, and for every column its
                              name, type, encoding, offset and length of its block
footer length (uint32)
"TINY01"                      magic again: a file that does not end with it was cut off

The footer sits at the end so that a table can be written in one streaming pass without knowing the size of the blocks beforehand, and so that a complete file always ends in the magic. Offsets in the footer are checked to be exactly consecutive, which makes every inconsistency (a shifted block, a cut file, appended garbage) a detected error and not a misread table.

Values: a TEXT column holds string or NULL, an INT column int (or int64) or NULL, a FLOAT column float64 or NULL. That is all the copy ever stores; any other value or column type is refused when writing instead of being stored wrongly.

package main

import (
	"context"
	"fmt"
	"os"

	tsql "github.com/SimonWaldherr/tinySQL"
	"github.com/SimonWaldherr/tinySQL/tinystore"
)

func main() {
	dir, err := os.MkdirTemp("", "tinystore-example-")
	if err != nil {
		panic(err)
	}
	defer os.RemoveAll(dir)
	facts := tsql.NewTable("sales", []tsql.Column{{Name: "region", Type: tsql.TextType}, {Name: "amount", Type: tsql.FloatType}}, false)
	facts.Rows = [][]any{{"north", 12.5}, {"north", 7.5}, {"south", 8.0}}
	if _, err := tinystore.Write(dir, facts); err != nil {
		panic(err)
	}
	tables, err := tinystore.ReadDir(dir)
	if err != nil {
		panic(err)
	}
	db := tsql.NewDB()
	defer db.Close()
	for _, table := range tables {
		if err := db.Put("default", table); err != nil {
			panic(err)
		}
	}
	stmt, err := tsql.ParseSQL("SELECT SUM(amount) AS total FROM sales")
	if err != nil {
		panic(err)
	}
	result, err := tsql.Execute(context.Background(), db, "default", stmt)
	if err != nil {
		panic(err)
	}
	fmt.Println(result.Rows[0]["total"])
}

Output:
28