Documentation
¶
Overview ¶
Package unarchive reads the members of a zip, gzip or gzipped tar archive as streams, so a caller can write each one somewhere without holding it (#1879).
It knows nothing about where members go. It answers three questions: which members an archive holds and whether their names are safe to file, whether the archive is inside the limits an operator set, and what the bytes of each selected member are. Everything it refuses, it refuses before the first member is handed over, wherever the format lets it know in advance: a zip's central directory declares every name, size and flag up front, and a tar is read through once without writing anything before it is read again for its bytes. A gzip holds one member whose size is not known until it has been read, so its limits are enforced as it streams, and a refusal there leaves nothing behind because the one write it feeds is abandoned.
The archive is read through a range function rather than an io.Reader, which is what lets a zip -- whose directory is at its end -- be opened in place in object storage without being downloaded first.
Index ¶
Constants ¶
const ( // DefaultMaxMemberBytes is the largest one member may be uncompressed. DefaultMaxMemberBytes = 2 << 30 // DefaultMaxTotalBytes is the most the selected members may add up to. DefaultMaxTotalBytes = 4 << 30 // DefaultMaxMembers is how many entries an archive may hold, directories // included. A zip's directory is parsed whole, so this is also what bounds // the memory opening one takes. DefaultMaxMembers = 10000 // DefaultMaxRatio is the largest uncompressed-to-compressed ratio a member // may have. A CSV compresses around ten to one and a very repetitive one // a few hundred to one; an archive built to expand runs to millions. DefaultMaxRatio = 500 )
Default limits. They are sized for the case the feature exists for, an upstream delivering one CSV of about a gigabyte in a zip, with room to spare, while still stopping an archive built to expand without bound.
const ( LimitMemberBytes = "max_member_bytes" LimitTotalBytes = "max_total_bytes" LimitMembers = "max_members" LimitRatio = "max_ratio" )
The limit names a LimitError carries. They are the configuration keys an operator raises, so a refusal says which knob it was.
Variables ¶
var ( // ErrUnsafeName is a member name that would leave the destination: an // absolute path, a drive letter, or a ".." segment. ErrUnsafeName = errors.New("unsafe member name") // ErrEncrypted is an encrypted member. Nothing here can decrypt one. ErrEncrypted = errors.New("encrypted member") // ErrUnsupported is a source that is none of the formats, or a zip member // compressed by a method other than stored or deflate. ErrUnsupported = errors.New("unsupported archive") // ErrCorrupt is an archive whose structure or bytes cannot be read. ErrCorrupt = errors.New("corrupt archive") // ErrNoMembers is a selection that matched no file in the archive. ErrNoMembers = errors.New("no member matches") )
The refusals that are not limits. Each is wrapped with the member it is about.
Functions ¶
This section is empty.
Types ¶
type Archive ¶
type Archive struct {
// contains filtered or unexported fields
}
Archive is an opened archive whose selected members passed every check that can be made before their bytes are read.
func Open ¶
Open detects the source's format, reads what it declares, and refuses it when a name is unsafe, a member is encrypted, a limit is passed, or the members pattern matches nothing. The pattern is a glob: one with no slash matches a member's file name wherever it is filed, so "*.csv" finds every CSV, and one with a slash matches the member's whole cleaned path. Empty selects every file.
func (*Archive) Extract ¶
func (a *Archive) Extract(ctx context.Context, fn MemberFunc) error
Extract hands each selected member's bytes to fn in archive order. It stops at the first failure, which is fn's error or the member's own: a limit passed while streaming, a checksum that did not match, a truncated stream.
type LimitError ¶
type LimitError struct {
// Limit is the configuration key that was passed.
Limit string
// Max is that limit's value, and Got what the archive reached.
Max, Got int64
// Member is the member that passed it, empty for a limit on the whole
// archive.
Member string
}
LimitError is an archive past one of its limits.
func (*LimitError) Error ¶
func (e *LimitError) Error() string
type Limits ¶
type Limits struct {
MaxMemberBytes int64 `yaml:"max_member_bytes"`
MaxTotalBytes int64 `yaml:"max_total_bytes"`
MaxMembers int `yaml:"max_members"`
MaxRatio int64 `yaml:"max_ratio"`
}
Limits bound what one extraction may expand to. A non-positive field takes its default. It is the resources.managed.extract configuration section as well, so the key a LimitError names is the key an operator sets.
func (Limits) Normalized ¶
Normalized returns the limits with every non-positive field at its default.
type Member ¶
type Member struct {
// Name is the name as the archive records it.
Name string
// Dirs are the directories the member is filed under, with empty and "."
// segments removed, and Base is its file name.
Dirs []string
Base string
// Size is the declared uncompressed size, or -1 where the format declares
// none ahead of the bytes (gzip).
Size int64
}
Member is one file an archive holds.
type MemberFunc ¶
MemberFunc receives one member's bytes. It must read r to its end: a member's checksum and size are verified at the end of its stream, and a failure there is returned by the read.
Source Files