chunk

package
v0.12.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Oct 2, 2026 License: GPL-3.0 Imports: 15 Imported by: 0

Documentation

Overview

Package chunk provides chunked upload functionality for large scan reports.

The chunk package splits large reports for upload:

  • SplitSegments: protocol v2 segments (RFC-026 §3.5), each a complete CTIS document with the report's tool, metadata and the assets its findings reference. The v2 client (pkg/client) uses it.
  • Splitter and Manager: protocol v1 chunks (/api/v1/agent/ingest/chunk). The Manager stores chunks in the durable outbox (pkg/outbox), which replaced the SQLite storage of SDKs before v0.8.0.

Key components:

  • Config: Configuration for chunking behavior
  • Splitter: Algorithm to split reports into v1 chunks
  • Manager: Main orchestration of v1 chunking and upload

Example usage:

manager, err := chunk.NewManager(chunk.DefaultConfig())
if err != nil {
    log.Fatal(err)
}
defer manager.Close()

report, err := manager.SubmitReport(ctx, largeReport)
if err != nil {
    log.Fatal(err)
}

manager.Start(ctx) // Start background upload

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func BindFindingAssets added in v0.8.0

func BindFindingAssets(r *ctis.Report) (unresolved int)

BindFindingAssets makes every finding's asset explicit, so a report can be split without changing which asset a finding belongs to:

  • an asset without an id gets one ("asset-<index>"), so it can be referenced;
  • a finding without asset_ref is bound to the report's only asset when there is exactly one (the server applies the same rule to a whole document, RFC-026 §10.1, but a segment may carry several assets).

It returns how many findings still resolve to no asset in the report (empty asset_ref with several or no assets, or a reference to an unknown id). The server rejects those findings; they are never bound to a guess. It modifies r.

func SplitSegments added in v0.8.0

func SplitSegments(r *ctis.Report, lim SegmentLimits) ([]*ctis.Report, error)

SplitSegments splits a report into protocol v2 segments. Every segment is a complete CTIS document: the same version, schema, metadata and tool as the report (the server checks they are identical), the findings of that segment and every asset those findings reference. An asset referenced by findings in several segments is repeated in each (the ingest merge is idempotent). Assets no finding references are packed into segments of their own. Dependencies and report properties go into the first segment.

Findings that resolve to no asset (see BindFindingAssets) are put in segments that carry no asset, so they are rejected by the server rather than bound to whichever single asset a segment happens to hold.

Call BindFindingAssets first. One segment is returned when the report fits the limits; it is r itself.

Types

type Chunk

type Chunk struct {
	ID               string      `json:"id"`
	ReportID         string      `json:"report_id"`
	ChunkIndex       int         `json:"chunk_index"`
	TotalChunks      int         `json:"total_chunks"`
	Data             []byte      `json:"-"` // Compressed data (not in JSON)
	UncompressedSize int         `json:"uncompressed_size"`
	CompressedSize   int         `json:"compressed_size"`
	Status           ChunkStatus `json:"status"`
	RetryCount       int         `json:"retry_count"`
	LastError        string      `json:"last_error,omitempty"`
	CreatedAt        time.Time   `json:"created_at"`
	UploadedAt       *time.Time  `json:"uploaded_at,omitempty"`
}

Chunk represents a single chunk of a report.

func (*Chunk) CanRetry

func (c *Chunk) CanRetry(maxRetries int) bool

CanRetry checks if there are chunks that can be retried.

type ChunkData

type ChunkData struct {
	ReportID    string               `json:"report_id"`
	ChunkIndex  int                  `json:"chunk_index"`
	TotalChunks int                  `json:"total_chunks"`
	Tool        *ctis.Tool           `json:"tool,omitempty"`
	Metadata    *ctis.ReportMetadata `json:"metadata,omitempty"`
	Assets      []ctis.Asset         `json:"assets,omitempty"`
	Findings    []ctis.Finding       `json:"findings,omitempty"`
	IsFinal     bool                 `json:"is_final"`
}

ChunkData represents the actual data in a chunk. This is what gets serialized, compressed, and uploaded.

type ChunkStatus

type ChunkStatus string

ChunkStatus represents the status of a single chunk.

const (
	// ChunkStatusPending indicates the chunk is waiting for upload.
	ChunkStatusPending ChunkStatus = "pending"

	// ChunkStatusUploading indicates the chunk is currently uploading.
	ChunkStatusUploading ChunkStatus = "uploading"

	// ChunkStatusCompleted indicates the chunk was successfully uploaded.
	ChunkStatusCompleted ChunkStatus = "completed"

	// ChunkStatusFailed indicates the chunk failed after all retries.
	ChunkStatusFailed ChunkStatus = "failed"
)

type Config

type Config struct {
	// Chunking thresholds - when to trigger chunking
	MinFindingsForChunking int // Minimum findings to trigger chunking (default: 2000)
	MinAssetsForChunking   int // Minimum assets to trigger chunking (default: 200)
	MinSizeForChunking     int // Minimum raw size in bytes to trigger chunking (default: 5MB)

	// Chunk size limits
	MaxFindingsPerChunk int // Max findings per chunk (default: 500)
	MaxAssetsPerChunk   int // Max assets per chunk (default: 100)
	MaxChunkSizeBytes   int // Max uncompressed size per chunk (default: 2MB)

	// Upload behavior
	UploadDelayMs        int // Delay between chunk uploads in ms (default: 100)
	MaxConcurrentUploads int // Max concurrent upload workers (default: 2)
	UploadTimeoutSeconds int // Timeout per chunk upload (default: 30)

	// Retry configuration
	MaxRetries     int // Max retries per chunk (default: 3)
	RetryBackoffMs int // Initial backoff between retries (default: 1000)

	// Storage configuration. Chunks live in a durable outbox (pkg/outbox).
	OutboxDir string // Outbox directory (default: "chunk-outbox" next to DatabasePath)
	// Deprecated: the SQLite chunk database was replaced by the outbox in
	// v0.8.0. DatabasePath now only locates the default OutboxDir (its
	// directory) and a leftover database, which is reported, not read.
	DatabasePath   string
	RetentionHours int // Deprecated: unused; the outbox caps age itself (7 days).
	MaxStorageMB   int // Max bytes the chunk outbox may hold, in MiB (default: 500)

	// Compression (chunks are always compressed before storage)
	CompressionLevel int // gzip/zstd level 1-9 (default: 3)

	// Auto-cleanup configuration
	AutoCleanupOnUpload     bool // Delete chunk data immediately after successful upload (default: true)
	CleanupOnReportComplete bool // Delete all chunks when report completes (default: true)
	CleanupIntervalMinutes  int  // How often to run cleanup in minutes (default: 15)
	AggressiveCleanup       bool // Enable aggressive cleanup when storage exceeds MaxStorageMB (default: true)
}

Config configures chunking behavior.

func DefaultConfig

func DefaultConfig() *Config

DefaultConfig returns sensible defaults for most environments.

func (*Config) Validate

func (c *Config) Validate() error

Validate validates the configuration.

type Manager

type Manager struct {
	// contains filtered or unexported fields
}

Manager splits large reports into protocol v1 chunks and uploads them from the durable outbox (pkg/outbox): every chunk is on disk before the first upload and removed only when the platform accepted it, so a crash or restart resumes where it stopped. (Before sdk-go v0.8.0 the chunks lived in a SQLite database; see Config.DatabasePath.)

func NewManager

func NewManager(cfg *Config) (*Manager, error)

NewManager creates a chunk manager and opens its outbox.

func (*Manager) Close

func (m *Manager) Close() error

Close stops uploading and releases the outbox. Chunks not uploaded yet stay on disk for the next process.

func (*Manager) GetProgress

func (m *Manager) GetProgress(_ context.Context, reportID string) (*Progress, error)

GetProgress returns upload progress for a report.

func (*Manager) GetStats

func (m *Manager) GetStats(_ context.Context) (*StorageStats, error)

GetStats returns statistics of the reports this manager knows and the bytes its outbox holds.

func (*Manager) IsRunning

func (m *Manager) IsRunning() bool

IsRunning returns whether the manager is running.

func (*Manager) NeedsChunking

func (m *Manager) NeedsChunking(report *ctis.Report) bool

NeedsChunking checks if a report needs to be chunked.

func (*Manager) Outbox added in v0.8.0

func (m *Manager) Outbox() *outbox.Outbox

Outbox returns the manager's outbox (for its Stats and dead letters).

func (*Manager) ProcessPending

func (m *Manager) ProcessPending(ctx context.Context) error

ProcessPending uploads every pending chunk that can be uploaded now and returns when none is left (or only chunks backing off after a failure).

func (*Manager) SetCallbacks

func (m *Manager) SetCallbacks(onProgress func(*Progress), onComplete func(string), onError func(string, error))

SetCallbacks sets the callback functions.

func (*Manager) SetUploader

func (m *Manager) SetUploader(uploader Uploader)

SetUploader configures the uploader.

func (*Manager) SetVerbose

func (m *Manager) SetVerbose(v bool)

SetVerbose enables verbose logging.

func (*Manager) Start

func (m *Manager) Start(ctx context.Context) error

Start begins background upload processing.

func (*Manager) Stop

func (m *Manager) Stop()

Stop gracefully stops the upload process.

func (*Manager) SubmitReport

func (m *Manager) SubmitReport(_ context.Context, report *ctis.Report) (*Report, error)

SubmitReport splits a report into chunks and stores them durably in the outbox. It returns once every chunk is on disk.

type Metadata

type Metadata struct {
	ToolName    string    `json:"tool_name,omitempty"`
	ToolVersion string    `json:"tool_version,omitempty"`
	ScanID      string    `json:"scan_id,omitempty"`
	SensorID    string    `json:"sensor_id,omitempty"`
	StartedAt   time.Time `json:"started_at,omitempty"`
	FinishedAt  time.Time `json:"finished_at,omitempty"`
}

Metadata stores additional report context.

type Progress

type Progress struct {
	ReportID        string  `json:"report_id"`
	TotalChunks     int     `json:"total_chunks"`
	CompletedChunks int     `json:"completed_chunks"`
	FailedChunks    int     `json:"failed_chunks"`
	PendingChunks   int     `json:"pending_chunks"`
	PercentComplete float64 `json:"percent_complete"`
	BytesUploaded   int64   `json:"bytes_uploaded"`
	BytesTotal      int64   `json:"bytes_total"`
	Status          string  `json:"status"`
}

Progress represents upload progress for a report.

type Report

type Report struct {
	ID                    string       `json:"id"`
	OriginalFindingsCount int          `json:"original_findings_count"`
	OriginalAssetsCount   int          `json:"original_assets_count"`
	OriginalSize          int          `json:"original_size"`
	CompressedSize        int          `json:"compressed_size"`
	TotalChunks           int          `json:"total_chunks"`
	CompletedChunks       int          `json:"completed_chunks"`
	FailedChunks          int          `json:"failed_chunks"`
	Status                ReportStatus `json:"status"`
	CompressionAlgo       string       `json:"compression_algo"`
	CreatedAt             time.Time    `json:"created_at"`
	UpdatedAt             time.Time    `json:"updated_at"`
	CompletedAt           *time.Time   `json:"completed_at,omitempty"`
	Metadata              *Metadata    `json:"metadata,omitempty"`
}

Report represents a chunked report in the database.

func (*Report) CalculateProgress

func (r *Report) CalculateProgress() *Progress

CalculateProgress calculates progress from a report.

func (*Report) HasFailed

func (r *Report) HasFailed() bool

HasFailed checks if the report has failed.

func (*Report) IsComplete

func (r *Report) IsComplete() bool

IsComplete checks if all chunks have been uploaded.

type ReportStatus

type ReportStatus string

ReportStatus represents the status of a chunked report.

const (
	// ReportStatusPending indicates the report is queued for upload.
	ReportStatusPending ReportStatus = "pending"

	// ReportStatusUploading indicates chunks are being uploaded.
	ReportStatusUploading ReportStatus = "uploading"

	// ReportStatusCompleted indicates all chunks have been uploaded.
	ReportStatusCompleted ReportStatus = "completed"

	// ReportStatusFailed indicates the upload failed after all retries.
	ReportStatusFailed ReportStatus = "failed"
)

type SegmentLimits added in v0.8.0

type SegmentLimits struct {
	// MaxFindings and MaxAssets per segment.
	MaxFindings int
	MaxAssets   int
	// MaxBytes is the target size of a segment's JSON before compression.
	// A single finding larger than this still gets a segment of its own.
	MaxBytes int
}

SegmentLimits bound one protocol v2 segment (RFC-026 §3.5, §3.6). The v2 client derives them from the server's hello limits.

type Splitter

type Splitter struct {
	// contains filtered or unexported fields
}

Splitter handles report chunking logic.

func NewSplitter

func NewSplitter(cfg *Config) *Splitter

NewSplitter creates a new splitter with the given config.

func (*Splitter) EstimateChunks

func (s *Splitter) EstimateChunks(findingsCount, assetsCount int) int

EstimateChunks estimates the number of chunks without actually splitting.

func (*Splitter) NeedsChunking

func (s *Splitter) NeedsChunking(report *ctis.Report) bool

NeedsChunking determines if a report should be chunked.

func (*Splitter) Split

func (s *Splitter) Split(report *ctis.Report) ([]*ChunkData, error)

Split divides a report into chunks.

Algorithm:

  1. If report is small enough, return single chunk
  2. Group findings by their asset_ref
  3. Create chunks that respect both findings and assets limits
  4. Every chunk is self-describing: it carries the report's tool and metadata, and every asset its findings reference

The platform ingests each chunk as a report of its own. A chunk without the tool was attributed to tool "unknown", and a finding whose asset was sent in an earlier chunk landed on a placeholder "scan:unknown:<report>" asset. So the tool and metadata are repeated in every chunk, and an asset whose findings span several chunks is repeated in each of them.

type StorageStats

type StorageStats struct {
	TotalReports      int   `json:"total_reports"`
	PendingReports    int   `json:"pending_reports"`
	UploadingReports  int   `json:"uploading_reports"`
	CompletedReports  int   `json:"completed_reports"`
	FailedReports     int   `json:"failed_reports"`
	TotalChunks       int   `json:"total_chunks"`
	PendingChunks     int   `json:"pending_chunks"`
	CompletedChunks   int   `json:"completed_chunks"`
	FailedChunks      int   `json:"failed_chunks"`
	TotalStorageBytes int64 `json:"total_storage_bytes"`
}

StorageStats contains chunk statistics (Manager.GetStats).

type Uploader

type Uploader interface {
	// UploadChunk uploads a single chunk.
	UploadChunk(ctx context.Context, data *ChunkData) error
}

Uploader is the interface for uploading chunks.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL