archive

package
v0.3.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Nov 16, 2026 License: MIT Imports: 11 Imported by: 0

Documentation

Overview

Package archive reads zip, tar and tar.gz archives: the walker the item service explodes and serves archive entries with, and a bounded walker and reader the format handlers (decision 0032) read packages with, such as an npm tarball's package.json and file count, a NuGet package's .nuspec or a jar's POM. Every count an archive's headers give is the uploader's claim, so the bounded walker counts what it actually decompresses.

Role

platform layer: archive reading that no module owns. In the artifacts module, the item service explodes deployed archives and serves archive entries with it, the folder archive download takes its kind, media type and extension from Kind, the format handlers read package archives with it, and the Helm, NuGet, PyPI and Terraform servers read stored and remote package content through it. The target layout also has security (P26) read archives here. It imports the standard library and the kernel, for the item path rule that Clean applies, and no module.

Inputs

  • Archive content as an io.ReaderAt with its size, for Walk, Scan, ReadFile and KindOf, or a tar stream the caller decompressed, as an io.Reader, for ScanTar. ReaderAt gives random access over stored content that can only seek.
  • The archive's Kind: an archiveType parameter (ParseKind), or the content's first SniffLen bytes and its name (Sniff, KindOf).
  • An entry name as the archive spells it, for Clean.
  • Limits, for Scan, ScanTar, ReadEntry and ReadFile: how many entries a walk may pass, how many bytes a tar walk may read once decompressed and how large the file read may be. DefaultLimits suit a package's descriptor.

Outputs

  • Kind, with ParseKind, Sniff and KindOf, and the media type and file extension of each kind (ContentType, Ext).
  • Entry: one entry as the archive lists it, with the size its header claims, Open, which reads it, and Wrap, which reports an error from opening or reading it as the walk would.
  • Walk: every entry, in archive order, until the callback returns an error.
  • Scan and ScanTar: Walk within Limits. ReadEntry: one entry whole, within Limits.File. ReadFile: the first regular file whose cleaned name a predicate accepts, with that name.
  • Clean: an entry name as a relative path, or a refusal.
  • ReaderAt: an io.ReaderAt over an io.ReadSeeker, which reads at the end of the content as bytes.Reader does.
  • The sentinels ErrUnusable ("archive: unusable archive"), which every refusal wraps, and ErrNotFound ("archive: no such entry"), which ReadFile returns when no file matches. Callers test them with errors.Is. A read of the content that fails is no refusal: Walk, Scan, ScanTar, ReadFile, ReadEntry and Wrap return its error as the reader gave it.

Invariants

  • Every count an archive's headers give is the uploader's claim, an entry's Size included, so the bounded walker counts what it decompresses: Scan and ScanTar count the entries they pass and every byte the tar stream yields, the bytes of skipped entries included, and ReadEntry reads at most one byte past Limits.File. A zip walk decompresses only the entries a caller opens, so Inflated bounds tar walks alone.
  • An entry that escapes its folder (an absolute path, a drive letter or a ".." segment), or whose name is not a valid item path (kernel.SplitItemPath), makes the archive unusable: Clean refuses it with ErrUnusable, and an explode refuses the whole archive. ReadFile passes over such an entry, which cannot be the file it looks for.
  • A failed read is not an unusable archive. Each walk keeps the first read of its content that failed, other than at the content's end, and returns that read's error where a parser's would be. Every other error a parser reports is ErrUnusable, whichever error it is, so a corrupt archive is refused even when the parser names it with io.EOF, io.ErrUnexpectedEOF or flate.CorruptInputError. A walk reads no further than the size it is given, so an offset that a corrupt archive gives past its end meets the end of the content, not a store that might fail the read. ScanTar returns the errors of the stream it is given as they are, since the caller, which decompresses that stream, is the one that can tell them apart.
  • A tar entry's reader is valid only until the walk moves on, so a callback reads the entry before it returns.
  • ReaderAt serialises reads, because they share one seek position. It tells the end of the content by the size that one seek to the end gives, never by a read's error, since io.ReadFull and S3's reader both say io.ErrUnexpectedEOF: a read past the end returns the bytes before it with io.EOF, a read that ends at the end returns nil, as bytes.Reader's and *os.File's do, and content that ends before that size is a failed read, the reader's own error or io.ErrUnexpectedEOF, never io.EOF. So content smaller than SniffLen reads the same from every blob store, those without ReadAt included.
  • Error texts reach clients: the item API answers an unusable archive with a 400 whose message is the error's text, so the texts stay byte-identical.

References

Status

Implemented P3: the walker, the archive kinds and their sniffer, the entry name check and the reader over seek-only content, for archive explode, archive entry downloads and folder archive downloads (issue 47). P7 added the bounded reader that the format handlers read packages with (Limits and ReadFile), and Scan and ReadEntry for npm's tarball inspection. P8 added ScanTar for PyPI's tar.bz2 source distributions. Moved here in P9d.

Testing

  • TestUnitClean: names that clean to a relative path, backslashes included, and the absolute, drive letter, ".." and invalid item path names that Clean refuses.
  • TestUnitSniffAndKinds: each signature and each name that tells a kind, ParseKind, and each kind's media type and extension.
  • TestUnitWalk: zip, tar and tar.gz walks list a directory, a file and a link in archive order; a callback's error ends the walk; garbage and a truncated tar are unusable.
  • TestUnitWalkReadFailures and TestUnitReadEntryReadFailure: a read that fails, before a walk lists the entries or after, returns its own error, a store's io.ErrUnexpectedEOF included, and never ErrUnusable.
  • TestUnitWalkCorruptArchives and TestUnitReadEntryCorrupt: corrupt content stays ErrUnusable whichever error the parser names it with, an offset past the content's end never reaches the store, and a refused entry is named in the refusal.
  • TestUnitReaderAt: a reader that already reads at offsets is used as it is, a seek-only reader reads at every offset as bytes.Reader does, at and past the end of the content included, and content whose end a seek cannot find fails to read.
  • TestUnitReaderAtCutShort: content cut short before its size, whether the store says io.ErrUnexpectedEOF or io.EOF, is a failed read from ReadAt, and a walk wherever the cut either returns that failure, never ErrUnusable, or has read every entry; it never takes the cut for the end of the content.
  • TestUnitReadFile and TestUnitReadFileInflationBudget: the first file the predicate accepts, ErrNotFound, each limit, and a tar.gz whose skipped entries inflate past the budget.
  • TestUnitScan, TestUnitScanTar and TestUnitReadEntryFailures: the limits of a scan over a sniffed or a caller-decompressed archive, the errors of a caller's stream returned as they are, and an entry a caller made that cannot be opened or read.

The item service's explode and archive entry tests, and the format handlers' package inspection tests, exercise it too; the Helm, npm, NuGet and PyPI readers' tests read packages smaller than SniffLen through ReaderAt over a seek-only reader.

Index

Constants

View Source
const SniffLen = 512

SniffLen is how much of the content Sniff reads: a tar header's magic ends at byte 262.

Variables

View Source
var DefaultLimits = Limits{Entries: 1000, Inflated: 256 << 20, File: 1 << 20}

DefaultLimits suit the descriptors formats read: a package.json, a go.mod or a POM is kilobytes, and a package whose descriptor lies past a thousand entries or 256 MiB of content is refused rather than read.

View Source
var ErrNotFound = errors.New("archive: no such entry")

ErrNotFound is an archive with no entry the caller asked for.

View Source
var ErrUnusable = errors.New("archive: unusable archive")

ErrUnusable is content that is not a usable archive: an unknown or corrupt format, an entry escaping its folder, or a limit exceeded. A read of the content that fails is none of these.

Functions

func Clean

func Clean(name string) (string, error)

Clean makes an entry name a relative path, refusing names that would leave the folder the archive is read into: absolute paths, drive letters and ".." segments. Backslashes count as separators, as Windows tools write them. A name that is only the folder itself cleans to "".

func ReadEntry

func ReadEntry(e Entry, lim Limits) ([]byte, error)

ReadEntry reads e whole, refusing content past lim.File with ErrUnusable. When e cannot be opened or read, it returns Wrap's error.

func ReadFile

func ReadFile(
	ra io.ReaderAt,
	size int64,
	kind Kind,
	lim Limits,
	want func(name string) bool,
) (string, []byte, error)

ReadFile returns the first regular file of the archive in ra whose cleaned name want accepts, with that name. ErrNotFound when no file matches; ErrUnusable when the content is no archive of kind or a limit is passed first; a read of ra that fails returns its own error.

func ReaderAt

func ReaderAt(rs io.ReadSeeker) io.ReaderAt

ReaderAt gives random access over a reader that can only seek, such as stored content, for zip, which reads its central directory from the end. Reads are serialised because they share one seek position. The first read seeks to the end of the content to learn its size, and reads at the end answer as bytes.Reader's do. A reader that already reads at offsets is returned as it is.

func Scan

func Scan(ra io.ReaderAt, size int64, kind Kind, lim Limits, fn func(Entry) error) error

Scan is Walk within lim's Entries and Inflated: ErrUnusable once the walk passes either, so a format can read a whole package, such as an npm tarball it counts the files of, without trusting its headers. lim.File is ReadEntry's.

func ScanTar

func ScanTar(r io.Reader, lim Limits, fn func(Entry) error) error

ScanTar is Scan over a tar stream the caller decompressed, for a compression the sniffer does not tell, such as a PyPI sdist's tar.bz2: lim's Inflated counts the bytes r yields, so a bomb in that compression stops at the budget too. An error r gives returns as r gave it, its decompressor's included: the caller, which decompresses, tells a corrupt stream from a failed read.

func Walk

func Walk(ra io.ReaderAt, size int64, kind Kind, fn func(Entry) error) error

Walk calls fn for every entry of the archive in ra, in archive order, until fn returns an error, which Walk returns. Content that is no archive of kind is ErrUnusable; a read of ra that fails returns its own error.

Types

type Entry

type Entry struct {
	// Name is the entry's name as the archive spells it; Clean makes it a
	// path.
	Name string
	Dir  bool
	// Regular is a plain file; links and devices are not.
	Regular bool
	// Size is what the archive's header claims.
	Size int64
	// Open reads the entry. A tar entry's reader is valid only until the
	// walk moves on.
	Open func() (io.Reader, error)
	// contains filtered or unexported fields
}

Entry is one entry as an archive lists it.

func (Entry) Wrap

func (e Entry) Wrap(err error) error

Wrap returns err, which opening or reading e gave, as the walk reports errors: a failed read of the archive's content as the read gave it, and anything else, an entry that is corrupt or passes a limit, as ErrUnusable naming e. An entry a caller made reads no archive, so its errors are all ErrUnusable.

type Kind

type Kind int

Kind is an archive container format.

const (
	KindNone Kind = iota
	KindZip
	KindTar
	KindTarGz
)

Archive kinds; KindNone is a file none of them.

func KindOf

func KindOf(ra io.ReaderAt, name string) Kind

KindOf sniffs the kind of the content in ra, named name, from whatever its read returns, ignoring the read's error: a read that fails can leave the name alone to tell the kind.

func ParseKind

func ParseKind(s string) (Kind, bool)

ParseKind reads an archiveType parameter (zip, tar, tar.gz, tgz).

func Sniff

func Sniff(head []byte, name string) Kind

Sniff tells the kind from the content's first bytes, falling back to the name for what has no signature to read.

func (Kind) ContentType

func (k Kind) ContentType() string

ContentType is the media type of an archive of the kind.

func (Kind) Ext

func (k Kind) Ext() string

Ext is the file extension of an archive of the kind.

type Limits

type Limits struct {
	// Entries is how many entries the walk may pass.
	Entries int
	// Inflated bounds the bytes a tar.gz walk decompresses, including the
	// entries it skips on the way, which a zip walk does not read.
	Inflated int64
	// File bounds the content of the file read.
	File int64
}

Limits bound what reading one file from an archive may cost.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL