Documentation
¶
Overview ¶
Package compress provides compression utilities for large payloads.
The compress package supports multiple compression algorithms with a focus on ZSTD for optimal compression ratio and speed balance. It also provides payload analysis to determine the best upload strategy.
Supported algorithms:
- ZSTD (Zstandard): Best balance of speed and compression ratio
- Gzip: Maximum compatibility with existing infrastructure
Example usage:
compressor := compress.NewCompressor(compress.AlgorithmZSTD, compress.LevelDefault)
compressed, err := compressor.Compress(jsonData)
if err != nil {
log.Fatal(err)
}
// Later, decompress
original, err := compressor.Decompress(compressed)
Index ¶
- Variables
- func QuickCompress(data []byte) ([]byte, error)
- func QuickDecompress(data []byte) ([]byte, error)
- type Algorithm
- type AnalysisResult
- type Analyzer
- func (a *Analyzer) Analyze(report *ctis.Report) (*AnalysisResult, error)
- func (a *Analyzer) AnalyzeBytes(data []byte) (*AnalysisResult, error)
- func (a *Analyzer) QuickAnalyze(findingsCount, assetsCount int) Strategy
- func (a *Analyzer) ShouldChunk(findingsCount, rawSize int) bool
- func (a *Analyzer) ShouldCompress(findingsCount, rawSize int) bool
- type AnalyzerConfig
- type CompressionStats
- type Compressor
- type Level
- type Strategy
Constants ¶
This section is empty.
Variables ¶
var ( // DefaultZSTD is the default ZSTD compressor. DefaultZSTD = NewCompressor(AlgorithmZSTD, LevelDefault) // DefaultGzip is the default gzip compressor. DefaultGzip = NewCompressor(AlgorithmGzip, LevelDefault) )
Default compressors for convenience.
var DefaultAnalyzer = NewAnalyzer(nil)
DefaultAnalyzer is the default payload analyzer.
Functions ¶
func QuickCompress ¶
QuickCompress compresses data using the default ZSTD compressor.
func QuickDecompress ¶
QuickDecompress decompresses ZSTD data using the default decompressor.
Types ¶
type Algorithm ¶
type Algorithm string
Algorithm represents a compression algorithm.
const ( // AlgorithmZSTD is the Zstandard compression algorithm. // Best balance of compression ratio and speed. AlgorithmZSTD Algorithm = "zstd" // AlgorithmGzip is the gzip compression algorithm. // Maximum compatibility with existing infrastructure. AlgorithmGzip Algorithm = "gzip" // AlgorithmNone indicates no compression. AlgorithmNone Algorithm = "none" )
type AnalysisResult ¶
type AnalysisResult struct {
// Strategy is the recommended upload strategy.
Strategy Strategy `json:"strategy"`
// Metrics about the payload.
FindingsCount int `json:"findings_count"`
AssetsCount int `json:"assets_count"`
RawSize int `json:"raw_size"`
// Compression estimate (only if compression recommended).
EstimatedCompressedSize int `json:"estimated_compressed_size,omitempty"`
EstimatedRatio float64 `json:"estimated_ratio,omitempty"`
// Chunking estimate (only if chunking recommended).
EstimatedChunks int `json:"estimated_chunks,omitempty"`
// Reason explains why this strategy was chosen.
Reason string `json:"reason"`
}
AnalysisResult contains the result of payload analysis.
type Analyzer ¶
type Analyzer struct {
// contains filtered or unexported fields
}
Analyzer analyzes payloads and determines the optimal upload strategy.
func NewAnalyzer ¶
func NewAnalyzer(config *AnalyzerConfig) *Analyzer
NewAnalyzer creates a new payload analyzer.
func (*Analyzer) Analyze ¶
func (a *Analyzer) Analyze(report *ctis.Report) (*AnalysisResult, error)
Analyze analyzes a CTIS report and returns the recommended upload strategy.
func (*Analyzer) AnalyzeBytes ¶
func (a *Analyzer) AnalyzeBytes(data []byte) (*AnalysisResult, error)
AnalyzeBytes analyzes raw JSON bytes and returns the recommended strategy. This is useful when you already have serialized data.
func (*Analyzer) QuickAnalyze ¶
QuickAnalyze performs a quick analysis based only on counts. This is faster but less accurate than full analysis.
func (*Analyzer) ShouldChunk ¶
ShouldChunk returns true if chunking should be used.
func (*Analyzer) ShouldCompress ¶
ShouldCompress returns true if compression should be used.
type AnalyzerConfig ¶
type AnalyzerConfig struct {
// MinFindingsForCompression is the minimum number of findings to trigger compression.
// Default: 500
MinFindingsForCompression int
// MinFindingsForChunking is the minimum number of findings to trigger chunking.
// Default: 2000
MinFindingsForChunking int
// MinSizeForCompression is the minimum payload size (bytes) to trigger compression.
// Default: 1MB (1048576)
MinSizeForCompression int
// MinSizeForChunking is the minimum payload size (bytes) to trigger chunking.
// Default: 5MB (5242880)
MinSizeForChunking int
// MaxCompressedSizeForSingleUpload is the maximum compressed size for single upload.
// If compressed size exceeds this, chunking will be used.
// Default: 5MB (5242880)
MaxCompressedSizeForSingleUpload int
}
AnalyzerConfig configures the payload analyzer thresholds.
func DefaultAnalyzerConfig ¶
func DefaultAnalyzerConfig() *AnalyzerConfig
DefaultAnalyzerConfig returns the default analyzer configuration.
type CompressionStats ¶
type CompressionStats struct {
OriginalSize int `json:"original_size"`
CompressedSize int `json:"compressed_size"`
Ratio float64 `json:"ratio"` // compressed/original
Savings float64 `json:"savings_percent"` // (1 - ratio) * 100
Algorithm string `json:"algorithm"`
}
CompressionStats holds statistics about a compression operation.
type Compressor ¶
type Compressor struct {
// contains filtered or unexported fields
}
Compressor provides compression and decompression functionality.
func NewCompressor ¶
func NewCompressor(algorithm Algorithm, level Level) *Compressor
NewCompressor creates a new compressor with the specified algorithm and level.
func (*Compressor) Algorithm ¶
func (c *Compressor) Algorithm() Algorithm
Algorithm returns the compression algorithm.
func (*Compressor) Compress ¶
func (c *Compressor) Compress(data []byte) ([]byte, error)
Compress compresses the input data.
func (*Compressor) CompressWithStats ¶
func (c *Compressor) CompressWithStats(data []byte) ([]byte, *CompressionStats, error)
CompressWithStats compresses data and returns statistics.
func (*Compressor) ContentEncoding ¶
func (c *Compressor) ContentEncoding() string
ContentEncoding returns the HTTP Content-Encoding header value.
func (*Compressor) Decompress ¶
func (c *Compressor) Decompress(data []byte) ([]byte, error)
Decompress decompresses the input data.
type Level ¶
type Level int
Level represents compression level.
const ( // LevelFastest prioritizes speed over compression ratio. LevelFastest Level = 1 // LevelDefault is the default compression level (good balance). LevelDefault Level = 3 // LevelBetter provides better compression at the cost of speed. LevelBetter Level = 6 // LevelBest provides maximum compression (slowest). LevelBest Level = 9 )
type Strategy ¶
type Strategy string
Strategy represents the upload strategy based on payload analysis.
const ( // StrategyDirect uploads directly without compression or chunking. // Used for small payloads (<500 findings, <1MB). StrategyDirect Strategy = "direct" // StrategyCompressOnly compresses the payload but uploads in a single request. // Used for medium payloads (500-2000 findings, 1-5MB). StrategyCompressOnly Strategy = "compress_only" // StrategyCompressAndChunk compresses then chunks the payload for gradual upload. // Used for large payloads (>2000 findings, >5MB). StrategyCompressAndChunk Strategy = "compress_and_chunk" )
Source Files