tar

Go Advanced TAR Library (with Ratarmount-compatible Indexing)

This library is a highly optimized and advanced drop-in replacement for the Go standard library archive/tar.

It integrates high-speed compression, parallel extraction, and ratarmount-compatible SQLite indexing to enable instant random access within massive archives.

Design Principles & Requirements

This library is built with strict adherence to the following design constraints:

• 100% Pure Go (Zero CGO): The entire library is written in pure Go. It cross-compiles natively and seamlessly to any target platform (Linux, macOS, Windows, FreeBSD, etc.) and architecture (amd64, arm64, 386, etc.) without requiring external C libraries or a GCC toolchain.
• Maximum Code Reuse: Instead of reinventing decompression and archiving from scratch, the library maximizes the reuse of existing industry-standard code. It extends Go's native archive/tar and klauspost/compress to leverage their proven performance, safety, and stability.
• True O(1) Random Access: Unlike standard sequential readers, this library provides true random access to compressed streams.
  • ZSTD (.tar.zst): Employs an O(1) fast path by jumping directly to byte-aligned, independent ZSTD frame boundaries stored in the index database.
  • GZIP (.tar.gz): Employs an O(1) fast path by parsing binary GZIDX metadata, recovering the 32KB sliding window (dictionary) via flate.NewReaderDict, and using a bit-shifting reader (bitShiftingReader) to align the unaligned DEFLATE bitstream.
• Industrial Cache Compatibility: The indexing architecture and SQLite database schema are binary-compatible with the industrial standards used by ratarmount and zran (GZIDX). Index databases generated by the Python-based ratarmount can be natively opened and used by this Go library in O(1) time, and vice versa.

Key Features

• Drop-in Compatibility: 100% compatible with the archive/tar API. You can seamlessly replace your existing archive/tar imports with github.com/unxed/tar.
• High Performance: Uses klauspost/compress for highly-parallelized ZStandard and optimized Gzip compression streams.
• Broad Compression Support: Built-in automatic format detection (magic bytes) for GZIP, BZIP2, XZ, and ZSTD.
• Parallel Extraction: Reads TAR sequentially but delegates filesystem writes, file creations, and metadata restoration (chmod/chown) to a parallel worker pool.
• In-place Updates (Updater): Truncates existing TAR EOF zero blocks to allow appending new files without full archive rewrite.
• ratarmount-compatible Indexing & Random Access (fs.FS):
  • Indexes .tar or compressed .tar.* archives on the fly into an SQLite database with a schema identical to ratarmount.
  • Exposes the archive as a standard Go fs.FS interface.
  • Facilitates O(1) random-access file reading for uncompressed .tar archives via io.SectionReader.
  • Automatically synthesizes missing parent folders on the fly.
  • Supports F4 Embedded TAR Index Format (F4SS) which bundles the index directly inside .tar, .tar.gz, or .tar.zst archives natively and compliantly. Read f4tar.md for details.
• Unix Properties Preservation: Transparently preserves and restores Symlinks, Hardlinks, Unix permissions, UID/GID (numeric & string), timestamps, and special files (Devices, FIFOs).
• NTFS & Windows Compatibility: Supports reading, writing, and restoring Windows Security Descriptors (NTFS ACLs) via PAX extended headers, alongside physical pre-allocation to prevent file fragmentation on NTFS.
• Safe Extraction and Sanitization: Implements automatic path verification to block symlink directory traversal attacks (such as Tar Slip), sanitizes Zone.Identifier (Mark of the Web) streams, and safely cleans up partial files on errors unless configured otherwise.
• POSIX Extended Attributes (PAX xattrs): Stores and restores complete extended attributes (including SELinux contexts and POSIX ACLs) using standard PAX records with SCHILY.xattr.* and LIBARCHIVE.xattr.* prefixes.
• Windows Security Descriptors (PAX MSWINDOWS.raw_sd): Encodes raw Windows Security Descriptors (NTFS ACLs) as Base64 strings under the standard PAX record MSWINDOWS.raw_sd. This allows cross-platform preservation of Windows security settings.

Usage

1. Standard Drop-in Usage

Simply replace the import path.

import "github.com/unxed/tar"

// Use exactly like the standard library
r, err := tar.OpenReader("archive.tar.gz")
if err != nil {
	log.Fatal(err)
}
defer r.Close()

2. High-Speed Random Access via fs.FS

Generate an SQLite index once and enjoy O(1) random access reads (uncompressed and indexed ZSTD/GZIP) and full io/fs compatibility.

package main

import (
	"io/fs"
	"log"

	"github.com/unxed/tar"
)

func main() {
	// Automatically generates index.sqlite on first run if missing
	tfs, err := tar.NewFS("archive.tar", "index.sqlite")
	if err != nil {
		log.Fatal(err)
	}
	defer tfs.Close()

	// Perform random-access ReadFile (O(1) seek)
	data, err := fs.ReadFile(tfs, "folder/file.txt")
	if err != nil {
		log.Fatal(err)
	}
	log.Println("File contents:", string(data))
}

3. Concurrent Extraction

e, err := tar.NewExtractor("archive.tar.gz", "/path/to/dst", tar.WithExtractorConcurrency(8))
if err != nil {
	log.Fatal(err)
}
defer e.Close()

if err := e.Extract(context.Background()); err != nil {
	log.Fatal(err)
}

4. Append files to TAR

f, err := os.OpenFile("archive.tar", os.O_RDWR, 0644)
if err != nil {
	log.Fatal(err)
}
defer f.Close()

updater, err := tar.NewUpdater(f, tar.APPEND_MODE_OVERWRITE)
if err != nil {
	log.Fatal(err)
}

err = updater.Append("newfile.txt", 4, []byte("data"))
if err != nil {
	log.Fatal(err)
}
updater.Close()

Pluggable Compression Architecture

This library features a modular, pluggable compression registry. If you want to enable true $O(1)$ random-access seeking for custom decompressors (e.g., custom GZIP or BZIP2 decoders), you can register them by implementing the following pure-Go interfaces:

// BlockOffsetImporter allows block-based decompressors (like ZSTD or BZIP2)
// to resume decompression from a specific byte or bit-aligned block boundary.
type BlockOffsetImporter interface {
	ResumeFromBlockOffset(r io.ReaderAt, bo *BlockOffset) (io.ReadCloser, error)
}

// GzipIndexImporter allows GZIP decompressors to resume decompression
// from a specific 32KB window dictionary checkpoint.
type GzipIndexImporter interface {
	ResumeFromGzipIndex(r io.ReaderAt, indexData []byte, targetOffset int64) (reader io.ReadCloser, uncompOffset int64, err error)
}

Once registered via RegisterDecompressor(), TarFS will automatically detect them and use their $O(1)$ fast paths during file reads.

Format Extensions

This library extends the standard TAR format by codifying advanced cross-platform metadata support and random-access indexing into the f4 tar extensions. These extensions provide high-fidelity storage for Windows ACLs, POSIX Xattrs, and $O(1)$ random-access indexes while remaining 100% compliant with standard TAR utilities.

See the technical specification in f4tar.md.

Why "f4"?

The name comes from the f4 file manager, a cross-platform asynchronous clone of Far Manager. This library was built to provide f4 with high-fidelity archive support, ensuring that system-specific metadata like Windows Security Descriptors (ACLs) and Linux Extended Attributes (Xattrs) are preserved seamlessly when moving data across OS boundaries.

License

This project is released under the BSD-3-Clause License. See the LICENSE file for details.

Acknowledgements

Please see CREDITS.md for a detailed list of third-party acknowledgements and inspirations.