Documentation
¶
Overview ¶
Package squashfs writes squashfs 4.0 filesystem images.
This is a write-only implementation. The intended consumer is the Linux kernel's squashfs driver, which is the canonical reader; we do not implement reading. The intended producer is hpcc's image pipeline (OCI layer tar -> rootfs image), where the input is a validated, bounded entry stream and the output is mounted read-only inside a Firecracker microVM.
Scope ¶
What this package implements:
- Squashfs 4.0 (the only format version since 2009 — kernel 2.6.29+).
- Regular files, directories, symlinks, hardlinks, character and block device nodes, FIFOs and sockets.
- One pluggable compressor for data and metadata blocks (see Compressor); the package itself does not import a compression library so callers pick their dependency.
- UID/GID dedup table.
What this package explicitly skips for v1:
- Fragment blocks (small-file tail packing). Costs ~5–15% space on toolchain images, zero correctness. Easy to add later — the data block writer is the same code path with a different placement rule.
- Extended attributes. OCI images occasionally carry file capabilities via xattrs; the v1 writer has no API to attach them. Add a follow-up Create*WithXattrs variant when a real image needs it.
- Export table (NFS file-handle lookups). Irrelevant for our mount-and-exec consumption pattern.
- Reading. The kernel reads; the e2e test mounts and execs.
API shape ¶
The Writer exposes one method per entry type: Writer.CreateFile, Writer.CreateDir, Writer.CreateSymlink, Writer.CreateHardlink, Writer.CreateCharDevice, Writer.CreateBlockDevice, Writer.CreateFIFO, Writer.CreateSocket. This keeps each entry shape statically typed — a symlink without a target or a device without a major number is unrepresentable at the call site, not a runtime validation error.
Writer.CreateFile returns an io.Writer for streaming the file's contents; the returned writer is invalidated by the next Create* call or by Close. Writer.Close flushes the metadata tables and backpatches the superblock and MUST be called for the image to be valid.
Entries may arrive in any order with one exception: Writer.CreateHardlink requires the target file to have been created earlier in the Writer's lifetime so the writer can resolve the target's inode number. Most OCI tars already satisfy this; the validating extractor on top of this package should enforce it explicitly.
Index ¶
- Constants
- Variables
- type Attrs
- type Compressor
- type CompressorID
- type GzipCompressor
- type Option
- type StoredCompressor
- type Writer
- func (w *Writer) Close() error
- func (w *Writer) CreateBlockDevice(attrs Attrs, major, minor uint32) error
- func (w *Writer) CreateCharDevice(attrs Attrs, major, minor uint32) error
- func (w *Writer) CreateDir(attrs Attrs) error
- func (w *Writer) CreateFIFO(attrs Attrs) error
- func (w *Writer) CreateFile(attrs Attrs) (io.Writer, error)
- func (w *Writer) CreateHardlink(linkPath, target string) error
- func (w *Writer) CreateSocket(attrs Attrs) error
- func (w *Writer) CreateSymlink(attrs Attrs, target string) error
Constants ¶
const ( VersionMajor uint16 = 4 VersionMinor uint16 = 0 )
Format version. Squashfs 4.0 has been the only on-disk version since 2009; we do not implement any other.
const ( MinBlockSize uint32 = 4 * 1024 MaxBlockSize uint32 = 1024 * 1024 DefaultBlockSize uint32 = 128 * 1024 )
Block size bounds from the format. Squashfs stores log2(block_size) as a uint16 in the superblock alongside the raw size for validation; both must agree. 128 KiB (log 17) is the mksquashfs default and a reasonable default here too.
const ( FlagUncompressedInodes uint16 = 1 << 0 FlagUncompressedData uint16 = 1 << 1 FlagCheck uint16 = 1 << 2 // unused since 4.0 FlagUncompressedFragments uint16 = 1 << 3 FlagNoFragments uint16 = 1 << 4 FlagAlwaysFragments uint16 = 1 << 5 FlagDuplicates uint16 = 1 << 6 FlagExportable uint16 = 1 << 7 FlagUncompressedXattrs uint16 = 1 << 8 FlagNoXattrs uint16 = 1 << 9 FlagCompressorOptions uint16 = 1 << 10 FlagUncompressedIDs uint16 = 1 << 11 )
Superblock feature flags. The writer sets these as appropriate based on what the image actually contains; callers do not need to touch them.
const ( InodeBasicDir uint16 = 1 InodeBasicFile uint16 = 2 InodeBasicSymlink uint16 = 3 InodeBasicBlock uint16 = 4 InodeBasicChar uint16 = 5 InodeBasicFIFO uint16 = 6 InodeBasicSocket uint16 = 7 InodeExtDir uint16 = 8 InodeExtFile uint16 = 9 InodeExtSymlink uint16 = 10 InodeExtBlock uint16 = 11 InodeExtChar uint16 = 12 InodeExtFIFO uint16 = 13 InodeExtSocket uint16 = 14 )
Inode type tags. "Basic" variants are used when the entry's metadata fits a fixed-size record; "extended" variants carry extra fields (sparse hints, xattr indices, etc.) and a 32-bit nlink. The v1 writer emits basic variants only, plus extended directories when a directory's index would exceed the basic-inode field widths.
const DataBlockUncompressedBit uint32 = 1 << 24
DataBlockUncompressedBit is set in a data-block size word to mark the block as stored uncompressed. Squashfs uses size==0 to mark a sparse hole; an uncompressed block of zero length is encoded as the bit alone.
const InvalidFragment uint32 = 0xFFFFFFFF
InvalidFragment is the value written into a file inode's fragment index when the file has no tail-fragment. The v1 writer always uses this (we skip fragments entirely).
const InvalidXattr uint32 = 0xFFFFFFFF
InvalidXattr is the value written into an inode's xattr index when the inode has no xattrs. The v1 writer always uses this.
const Magic uint32 = 0x73717368
Magic is the squashfs superblock magic ("hsqs" little-endian).
const MetadataBlockSize uint32 = 8 * 1024
Metadata block size. Inode and directory tables are written as a stream of compressed "metadata blocks" of at most 8 KiB each, with a 16-bit length prefix (high bit set = stored uncompressed). This constant is fixed by the format; do not parameterize.
const MetadataUncompressedBit uint16 = 1 << 15
MetadataUncompressedBit is the equivalent flag in the 16-bit metadata block header.
Variables ¶
var ErrClosed = errors.New("squashfs: writer is closed")
ErrClosed is returned by Writer methods called after Close.
var ErrHardlinkTarget = errors.New("squashfs: hardlink target must be a regular file created earlier")
ErrHardlinkTarget is returned by CreateHardlink when target has not been created earlier in this Writer's lifetime, or refers to a non-file entry (directory, symlink, device, FIFO, socket). Squashfs hardlinks are inode references, so the target inode must already exist and be a regular file.
var ErrIncompressible = errors.New("squashfs: block did not compress smaller; store raw")
ErrIncompressible is the sentinel a Compressor returns when the compressed output would not be smaller than the input. The writer then stores the block raw with the squashfs "uncompressed" bit set. Returning this is not an error condition — it is the normal signal for "store raw."
var ErrInvalidPath = errors.New("squashfs: entry path must be absolute and canonical")
ErrInvalidPath is returned for entry paths that aren't an absolute canonical path inside the image: empty string, missing leading /, trailing /, or any "." / ".." component. The writer does not resolve, normalize, or sanitize — that is the validating extractor's job upstream.
var ErrStaleEntry = errors.New("squashfs: write to a CreateFile writer invalidated by a later call")
ErrStaleEntry is returned when the caller writes to an io.Writer previously returned by CreateFile after a subsequent Create* call or Close has invalidated it. Capture and consume the writer before starting the next entry.
Functions ¶
This section is empty.
Types ¶
type Attrs ¶
type Attrs struct {
// Path is the entry's absolute path inside the image, with a
// leading "/" and no trailing "/". Must be canonical: no ".",
// "..", or empty components. The root directory ("/") is
// implicit and must not be written explicitly.
Path string
// Mode carries the Unix permission bits (the low 12 bits:
// rwx for u/g/o plus setuid/setgid/sticky). The type bits of
// fs.FileMode are ignored — the entry type is determined by
// which Create* method was called.
Mode fs.FileMode
// UID, GID are the numeric owner/group. The writer dedups
// distinct values into the squashfs ID table; callers do not
// need to dedup themselves.
UID uint32
GID uint32
// Mtime is the entry's modification time. Squashfs stores
// seconds since the Unix epoch as a uint32; the writer
// truncates sub-second precision and rejects times outside
// [1970-01-01, 2106-02-07]. If WithFixedMtime was supplied at
// construction, that value overrides Mtime.
Mtime time.Time
}
Attrs is the metadata common to every entry that owns an inode of its own: ownership, permission bits, and modification time. Used by every Create* method except CreateHardlink (which inherits all metadata from its target inode).
The writer takes ownership of an Attrs for the duration of the Create* call only; callers may reuse the value afterwards.
type Compressor ¶
type Compressor interface {
// ID returns the squashfs compressor ID for this algorithm. It
// is written into the superblock and must be stable across a
// single Writer's lifetime.
ID() CompressorID
// Options returns the compressor options block embedded after
// the superblock when the image sets FlagCompressorOptions.
// Return nil to omit the options block entirely (most callers).
// The exact layout is compressor-specific; see the format spec.
Options() []byte
// Compress appends the compressed form of src to dst and
// returns the extended slice. If the compressed output would
// not be smaller than src, implementations should return
// (dst, ErrIncompressible) and the writer will store src raw
// with the uncompressed bit set.
//
// Implementations must not retain src or dst after Compress
// returns; the writer reuses both buffers.
Compress(dst, src []byte) ([]byte, error)
}
Compressor is the pluggable compression backend for data blocks and metadata blocks. The package does not import any compression library — the caller picks the dependency (e.g. github.com/klauspost/compress/zstd) and supplies a Compressor implementation.
One Compressor is shared across all blocks in an image; squashfs stores a single CompressorID in the superblock, so mixing algorithms within one image is not allowed.
Implementations must be safe to call concurrently — the writer may compress multiple blocks in parallel once the implementation lands.
type CompressorID ¶
type CompressorID uint16
CompressorID identifies the compression algorithm recorded in the superblock. Values come from the format spec; the kernel uses the same numbering.
const ( CompressorGzip CompressorID = 1 CompressorLZMA CompressorID = 2 // legacy, kernel dropped support CompressorLZO CompressorID = 3 CompressorXZ CompressorID = 4 CompressorLZ4 CompressorID = 5 CompressorZstd CompressorID = 6 )
type GzipCompressor ¶
type GzipCompressor struct {
// Level is the compress/zlib compression level. Zero means
// "stdlib default" (currently 6).
Level int
}
GzipCompressor compresses blocks with the squashfs "gzip" algorithm. Despite the name, squashfs's compressor ID 1 expects payloads in the zlib stream format (RFC 1950 — a 2-byte header plus a deflate stream), NOT the gzip stream format (RFC 1952, which has an extra header with magic/filename/mtime fields). This type uses compress/zlib from the stdlib, which produces the right shape. compress/gzip would produce something the kernel's squashfs driver rejects on the first decompression attempt.
Level matches compress/zlib: 1 (fastest) through 9 (best), or 0 to take the stdlib default (~6). If the compressed output is not strictly smaller than the input, Compress returns ErrIncompressible so the writer stores the block raw with the per-block uncompressed bit set.
func (GzipCompressor) Compress ¶
func (c GzipCompressor) Compress(dst, src []byte) ([]byte, error)
Compress runs a zlib stream over src; if the compressed result is not smaller than the input, returns ErrIncompressible so the writer stores src raw. dst's underlying capacity is reused when large enough; otherwise the returned slice is freshly allocated.
func (GzipCompressor) Options ¶
func (GzipCompressor) Options() []byte
Options omits the optional gzip-options block. The format allows callers to advertise non-default compression-level, window-bits, and strategy via that block, but compress/zlib hardcodes window-bits=15 and a single strategy, so there's nothing useful to negotiate with the kernel decoder.
type Option ¶
type Option func(*writerConfig) error
Option configures a Writer at construction time. Pass options to NewWriter.
func WithBlockSize ¶
WithBlockSize sets the data-block size for the image. Must be a power of two between MinBlockSize and MaxBlockSize inclusive. The default is DefaultBlockSize (128 KiB).
func WithCompressor ¶
func WithCompressor(c Compressor) Option
WithCompressor selects the compression backend. Required — there is no default, so callers must make a deliberate dependency choice.
func WithFixedMtime ¶
WithFixedMtime overrides every entry's Mtime with t. Useful for deterministic builds: when the image is keyed by content digest (as in hpcc's rootfs cache), the source tarball's per-file mtimes otherwise leak into the digest and defeat caching.
type StoredCompressor ¶
type StoredCompressor struct{}
StoredCompressor is a no-op compressor that asks the writer to store every block raw. It declares itself as gzip in the superblock — the kernel never invokes the gzip decoder for blocks whose on-disk header bit marks them uncompressed, so the resulting image is valid and mountable without pulling a gzip dependency into this package.
Useful for tests and for callers who explicitly want an uncompressed image (deterministic byte-for-byte output, no compression CPU cost). For production rootfs caches a real compressor — zstd is the typical choice — produces 2–4× smaller images.
func (StoredCompressor) Compress ¶
func (StoredCompressor) Compress(dst, src []byte) ([]byte, error)
Compress always returns ErrIncompressible so the writer stores every block raw.
func (StoredCompressor) ID ¶
func (StoredCompressor) ID() CompressorID
ID reports gzip. See type doc for why.
func (StoredCompressor) Options ¶
func (StoredCompressor) Options() []byte
Options returns nil; gzip needs no options block, and even if it did the writer never produces a gzip-encoded payload here.
type Writer ¶
type Writer struct {
// contains filtered or unexported fields
}
Writer streams a squashfs image to an underlying io.WriteSeeker. See package doc for the API contract.
func NewWriter ¶
func NewWriter(out io.WriteSeeker, opts ...Option) (*Writer, error)
NewWriter constructs a Writer that emits its image to out and applies the supplied options. WithCompressor is mandatory.
The Writer writes a 96-byte placeholder at file offset 0 so callers cannot mistake an unclosed image for a valid one — a kernel opening the file before Close would fail the magic check.
func (*Writer) Close ¶
Close flushes the inode table, directory table, ID table, and the final superblock. Required.
func (*Writer) CreateBlockDevice ¶
CreateBlockDevice creates a block-device node.
func (*Writer) CreateCharDevice ¶
CreateCharDevice creates a character-device node.
func (*Writer) CreateFIFO ¶
CreateFIFO creates a named-pipe entry.
func (*Writer) CreateFile ¶
CreateFile starts a new regular-file entry and returns an io.Writer for its contents.
func (*Writer) CreateHardlink ¶
CreateHardlink adds an additional directory entry at path referencing the inode of target. target must already exist and be a regular file.
func (*Writer) CreateSocket ¶
CreateSocket creates a unix-socket entry.
Source Files