Documentation
¶
Index ¶
- Constants
- Variables
- func CRC32CombineIEEE(crc1, crc2 uint32, len2 int64) uint32
- type CDFileHeader
- type CentralDirectory
- type Config
- type EndOfCDRecord
- type Error
- type FileEntry
- type LocalFileHeader
- type Option
- type Reader
- type SegmentEntry
- type SegmentMetadata
- func (sm *SegmentMetadata) AddSegment(index int, originalSize uint64, originalCRC32 uint32) error
- func (sm *SegmentMetadata) FinalizeCRC()
- func (sm *SegmentMetadata) GetMissingSegments() []int
- func (sm *SegmentMetadata) GetTotalCRC32() uint32
- func (sm *SegmentMetadata) IsComplete() bool
- func (sm *SegmentMetadata) SetOrder(order []int) error
- type SegmentWriter
- type TDFReader
- type TDFReaderOptions
- type Writer
- type Zip32DataDescriptor
- type Zip64DataDescriptor
- type Zip64EndOfCDRecord
- type Zip64EndOfCDRecordLocator
- type Zip64ExtendedInfoExtraField
- type Zip64ExtendedLocalInfoExtraField
- type Zip64Mode
Constants ¶
const ( // TDFManifestFileName is the manifest entry name this SDK writes. The 0. // prefix is a holdover from an early design that anticipated several // payload/manifest pairs per archive and never shipped. TDFManifestFileName = "0.manifest.json" // TDFManifestFileNameSpec is the manifest entry name given by the OpenTDF // spec: https://opentdf.io/spec#tdf-structure. Readers accept it so that // archives from implementations written against the spec can be opened. // The writer is unchanged and still emits TDFManifestFileName. // See https://github.com/opentdf/platform/issues/3513. TDFManifestFileNameSpec = "manifest.json" TDFPayloadFileName = "0.payload" )
Variables ¶
var ( ErrWriterClosed = errors.New("archive writer closed") ErrInvalidSegment = errors.New("invalid segment index") ErrOutOfOrder = errors.New("segment out of order") ErrDuplicateSegment = errors.New("duplicate segment already written") ErrSegmentMissing = errors.New("segment missing") ErrNoSegmentZero = errors.New("segment 0 missing; it carries the payload local file header") ErrInvalidSize = errors.New("invalid size") ErrZip64Required = errors.New("ZIP64 required but disabled (Zip64Never)") ErrFieldOverflow = errors.New("value too large for zip field") )
Common errors
Functions ¶
func CRC32CombineIEEE ¶
CRC32CombineIEEE combines two CRC-32 (IEEE) checksums as if the data were concatenated. crc1 is the CRC of the first part, crc2 of the second part, and len2 is the byte length of the second part. This uses the standard reflected IEEE polynomial 0xEDB88320 as used by ZIP.
Types ¶
type CDFileHeader ¶
type CDFileHeader struct {
Signature uint32
VersionCreated uint16
VersionNeeded uint16
GeneralPurposeBitFlag uint16
CompressionMethod uint16
LastModifiedTime uint16
LastModifiedDate uint16
Crc32 uint32
CompressedSize uint32
UncompressedSize uint32
FilenameLength uint16
ExtraFieldLength uint16
FileCommentLength uint16
DiskNumberStart uint16
InternalFileAttributes uint16
ExternalFileAttributes uint32
LocalHeaderOffset uint32
}
type CentralDirectory ¶
type CentralDirectory struct {
Entries []FileEntry // File entries in the archive
Offset uint64 // Offset where central directory starts
Size uint64 // Size of central directory
// MaxNonZip64Value is the largest value written into a 32-bit field
// before the entry switches to ZIP64. Zero means maxNonZip64Value.
// Lowering it is a test seam: it exercises the ZIP64 path without
// materializing a multi-gigabyte archive.
MaxNonZip64Value uint64
}
CentralDirectory manages the ZIP central directory structure
func NewCentralDirectory ¶
func NewCentralDirectory() *CentralDirectory
NewCentralDirectory creates a new central directory
func (*CentralDirectory) AddFile ¶
func (cd *CentralDirectory) AddFile(entry FileEntry)
AddFile adds a file entry to the central directory
func (*CentralDirectory) GenerateBytes ¶
func (cd *CentralDirectory) GenerateBytes(isZip64 bool) ([]byte, error)
GenerateBytes generates the central directory bytes
type Config ¶
type Config struct {
Zip64 Zip64Mode
MaxSegments int
EnableLogging bool
// Now returns the current time for header/metadata timestamps.
// Defaults to time.Now; tests inject a pinned clock for
// deterministic ZIP output.
Now func() time.Time
// MaxNonZip64Value is the largest size or offset written into a 32-bit
// zip field before the archive switches to ZIP64. Zero means
// maxNonZip64Value (2 GiB - 1). See WithMaxNonZip64Value.
MaxNonZip64Value uint64
}
Config holds configuration options for writers
type EndOfCDRecord ¶
type Error ¶
type Error struct {
Op string // Operation that failed
Type string // Writer type: "sequential", "streaming", "segment"
Err error // Underlying error
// Mutated reports whether the writer had already changed its own state
// when the operation failed. False means the writer is byte-for-byte as it
// was before the call and the caller may retry; true means the partial
// change is not rolled back and a retry compounds it rather than
// recovering from it.
//
// Only Finalize sets it. WriteSegment validates everything before it
// mutates anything, so every error it returns is Mutated false.
//
// An implementation that wraps a zipstream writer must not pass the
// delegate's error through unchanged if it performed mutations of its own:
// the flag describes the whole operation, not one layer of it.
Mutated bool
}
Error provides detailed error information for archive operations
type FileEntry ¶
type FileEntry struct {
Name string // Filename in the archive
Offset uint64 // Offset of local file header in archive
Size uint64 // Uncompressed size
CompressedSize uint64 // Compressed size (same as Size for no compression)
CRC32 uint32 // CRC32 checksum of uncompressed data
ModTime time.Time // Last modification time
IsStreaming bool // Whether this uses data descriptor pattern
}
FileEntry represents a file in the ZIP archive with metadata
type LocalFileHeader ¶
type Option ¶
type Option func(*Config)
Option is a functional option for configuring writers
func WithClock ¶ added in v0.32.0
WithClock overrides the time source used to stamp ZIP headers and segment metadata. Tests inject a pinned clock to make output byte- for-byte deterministic.
func WithMaxNonZip64Value ¶ added in v0.32.0
WithMaxNonZip64Value lowers the point at which the writer switches to ZIP64. This exists as a test seam -- mirroring java-sdk's injectable MAX_NON_ZIP64_VALUE -- so the ZIP64 path can be exercised without materializing a 2 GiB payload. Production callers should leave it alone; zero or a value above the default is ignored.
func WithMaxSegments ¶
WithMaxSegments sets the maximum number of segments for SegmentWriter
func WithZip64 ¶
func WithZip64() Option
WithZip64 enables ZIP64 format support for large files WithZip64 forces ZIP64 mode; kept for backward compatibility. Equivalent to WithZip64Mode(Zip64Always).
func WithZip64Mode ¶
WithZip64Mode sets the ZIP64 mode (Auto/Always/Never).
type Reader ¶
type Reader struct {
// contains filtered or unexported fields
}
func NewReader ¶
func NewReader(readSeeker io.ReadSeeker) (Reader, error)
NewReader Create archive reader instance.
func (Reader) ReadAllFileData ¶
ReadAllFileData Return all the data of the file if the file is available and below the specified size. NOTE: Use this method for small file sizes.
func (Reader) ReadFileData ¶
ReadFileData Read data from file of given length of size.
type SegmentEntry ¶
type SegmentEntry struct {
Index int // Segment index (0-based)
Size uint64 // Size of stored segment bytes (no compression)
CRC32 uint32 // CRC32 of stored segment bytes
Written time.Time // When this segment was written
}
SegmentEntry represents a single segment in out-of-order writing
type SegmentMetadata ¶
type SegmentMetadata struct {
ExpectedCount int // Total number of expected segments (unused when Order set)
Segments map[int]*SegmentEntry // Map of segments by index
TotalSize uint64 // Cumulative size of all segments
TotalCRC32 uint32 // Final CRC32 when all segments are processed
// Order, when set, defines the exact logical order of segments for
// completeness checks and CRC computation. Indices may be sparse.
Order []int
// contains filtered or unexported fields
}
SegmentMetadata tracks per-segment metadata for out-of-order writing. It stores only plaintext size and CRC for each index and computes the final CRC via CRC32-combine at finalize time (no payload buffering).
func NewSegmentMetadata ¶
func NewSegmentMetadata(expectedCount int, now func() time.Time) *SegmentMetadata
NewSegmentMetadata creates metadata for tracking segments using combine-based CRC. now stamps SegmentEntry.Written; pass time.Now for production or a pinned clock for deterministic tests.
func (*SegmentMetadata) AddSegment ¶
func (sm *SegmentMetadata) AddSegment(index int, originalSize uint64, originalCRC32 uint32) error
AddSegment records metadata for a segment (size + CRC) without retaining payload bytes.
func (*SegmentMetadata) FinalizeCRC ¶
func (sm *SegmentMetadata) FinalizeCRC()
FinalizeCRC computes the total CRC32 by combining per-segment CRCs in index order.
func (*SegmentMetadata) GetMissingSegments ¶
func (sm *SegmentMetadata) GetMissingSegments() []int
GetMissingSegments returns a list of missing segment indices
func (*SegmentMetadata) GetTotalCRC32 ¶
func (sm *SegmentMetadata) GetTotalCRC32() uint32
GetTotalCRC32 returns the final CRC32 value (only valid when IsComplete() is true)
func (*SegmentMetadata) IsComplete ¶
func (sm *SegmentMetadata) IsComplete() bool
IsComplete returns true if all expected segments have been processed
func (*SegmentMetadata) SetOrder ¶
func (sm *SegmentMetadata) SetOrder(order []int) error
SetOrder defines the exact logical order of segments. Duplicates are not allowed. When set, completeness/CRC use this order; ExpectedCount is ignored.
type SegmentWriter ¶
type SegmentWriter interface {
Writer
WriteSegment(ctx context.Context, index int, size uint64, crc32 uint32) ([]byte, error)
// Finalize writes the trailer (data descriptor, manifest, central
// directory) for the segments recorded so far. Segment 0 must be among
// them: it carries the payload local file header that every recorded
// offset is measured from. Finalize returns ErrNoSegmentZero when index 0
// was never written or was cleaned up, and ErrSegmentMissing when no
// segments remain at all -- none were written, or every one was cleaned
// up. Gaps between the remaining indices are accepted; order is inferred
// by sorting whichever indices are present.
Finalize(ctx context.Context, manifest []byte) ([]byte, error)
// CleanupSegment drops a segment index, rolling back both its presence
// marker and the payload size it contributed. A cleaned-up index becomes
// indistinguishable from one that was never written: it falls out of the
// order Finalize infers, exactly as a gap in the write set would, and the
// caller must leave its bytes out of the assembled archive. Index 0 is the
// exception -- Finalize rejects its absence with ErrNoSegmentZero.
// Cleaning up an index that was never written is a no-op.
CleanupSegment(index int) error
}
SegmentWriter handles out-of-order segments with deterministic output
func NewSegmentTDFWriter ¶
func NewSegmentTDFWriter(expectedSegments int, opts ...Option) SegmentWriter
NewSegmentTDFWriter creates a new SegmentWriter for out-of-order segment writing
type TDFReader ¶
type TDFReader struct {
// contains filtered or unexported fields
}
func NewTDFReader ¶
func NewTDFReader(readSeeker io.ReadSeeker, opt ...TDFReaderOptions) (TDFReader, error)
NewTDFReader Create tdf reader instance.
func (TDFReader) Manifest ¶
Manifest Return the manifest of the tdf.
The spec name wins over the name this SDK writes when an archive carries both. Only a missing entry triggers the fallback: a manifest that is present but too large is a size failure, and retrying under the other name would both report the wrong reason and, in an archive holding both, hand back the superseded manifest.
func (TDFReader) PayloadSize ¶
PayloadSize Return the size of the payload.
type TDFReaderOptions ¶ added in v0.13.0
type TDFReaderOptions func(*TDFReader)
func WithTDFManifestMaxSize ¶ added in v0.13.0
func WithTDFManifestMaxSize(size int64) TDFReaderOptions
WithTDFManifestMaxSize caps the manifest size in bytes. A size of zero or less means no cap.