Documentation
¶
Overview ¶
Package sniff identifies an input's compression and document format from its leading bytes, using the file extension only as a hint for inputs the content can't settle.
Index ¶
- Constants
- Variables
- func Compression(head []byte) options.Compression
- func Detect(head []byte, name string) options.Compression
- func Format(head []byte, hint options.OutputFormat) (options.OutputFormat, error)
- func FormatFromExt(name string) (options.OutputFormat, bool)
- func NeedsExtension(c options.Compression) bool
- func StripCompressionExt(name string) (string, options.Compression)
Constants ¶
const HeadSize = 4096
HeadSize is how many leading bytes Compression and Format need to see.
Variables ¶
var ErrEncryptedParquet = errors.New("streamio: encrypted Parquet footers are not supported")
ErrEncryptedParquet is returned for a Parquet file with an encrypted footer ("PARE" magic).
var ErrUnknownFormat = errors.New("streamio: cannot determine input format")
ErrUnknownFormat is returned when the content matches no supported format and the extension offers no hint.
Functions ¶
func Compression ¶
func Compression(head []byte) options.Compression
Compression returns the compression head begins with, or CompressionNone.
func Detect ¶
func Detect(head []byte, name string) options.Compression
Detect returns the compression of a file: its magic number when it has one, else the compression its name's extension names, but only for codecs with no magic of their own. A name like x.csv.gz over plain text is therefore read as plain text rather than failing to gunzip.
func Format ¶
func Format(head []byte, hint options.OutputFormat) (options.OutputFormat, error)
Format returns the document format of head, a prefix of an already-decompressed input.
Content decides first: container magic for Parquet and Arrow, then a leading '{' for NDJSON, then delimiter scoring for CSV and TSV. hint (a format guessed from the extension, or empty) only settles what the content leaves open: an empty file, or text with no consistent delimiter.
func FormatFromExt ¶
func FormatFromExt(name string) (options.OutputFormat, bool)
FormatFromExt maps a file name's extension to a format, reporting false when it names none.
func NeedsExtension ¶
func NeedsExtension(c options.Compression) bool
NeedsExtension reports whether c has no magic number, so only a file name or an explicit option can say a stream uses it.
func StripCompressionExt ¶
func StripCompressionExt(name string) (string, options.Compression)
StripCompressionExt removes a trailing compression extension (.gz, .zst, .bz2, .xz, .lz4, .sz, .br and their long spellings) from name, returning the remaining name and the compression the extension named.
Types ¶
This section is empty.
Source Files