Documentation
¶
Overview ¶
Package chunk provides chunked upload functionality for large scan reports.
The chunk package splits large reports for upload:
- SplitSegments: protocol v2 segments (RFC-026 §3.5), each a complete CTIS document with the report's tool, metadata and the assets its findings reference. The v2 client (pkg/client) uses it.
- Splitter and Manager: protocol v1 chunks (/api/v1/agent/ingest/chunk). The Manager stores chunks in the durable outbox (pkg/outbox), which replaced the SQLite storage of SDKs before v0.8.0.
Key components:
- Config: Configuration for chunking behavior
- Splitter: Algorithm to split reports into v1 chunks
- Manager: Main orchestration of v1 chunking and upload
Example usage:
manager, err := chunk.NewManager(chunk.DefaultConfig())
if err != nil {
log.Fatal(err)
}
defer manager.Close()
report, err := manager.SubmitReport(ctx, largeReport)
if err != nil {
log.Fatal(err)
}
manager.Start(ctx) // Start background upload
Index ¶
- func BindFindingAssets(r *ctis.Report) (unresolved int)
- func SplitSegments(r *ctis.Report, lim SegmentLimits) ([]*ctis.Report, error)
- type Chunk
- type ChunkData
- type ChunkStatus
- type Config
- type Manager
- func (m *Manager) Close() error
- func (m *Manager) GetProgress(_ context.Context, reportID string) (*Progress, error)
- func (m *Manager) GetStats(_ context.Context) (*StorageStats, error)
- func (m *Manager) IsRunning() bool
- func (m *Manager) NeedsChunking(report *ctis.Report) bool
- func (m *Manager) Outbox() *outbox.Outbox
- func (m *Manager) ProcessPending(ctx context.Context) error
- func (m *Manager) SetCallbacks(onProgress func(*Progress), onComplete func(string), ...)
- func (m *Manager) SetUploader(uploader Uploader)
- func (m *Manager) SetVerbose(v bool)
- func (m *Manager) Start(ctx context.Context) error
- func (m *Manager) Stop()
- func (m *Manager) SubmitReport(_ context.Context, report *ctis.Report) (*Report, error)
- type Metadata
- type Progress
- type Report
- type ReportStatus
- type SegmentLimits
- type Splitter
- type StorageStats
- type Uploader
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func BindFindingAssets ¶ added in v0.8.0
BindFindingAssets makes every finding's asset explicit, so a report can be split without changing which asset a finding belongs to:
- an asset without an id gets one ("asset-<index>"), so it can be referenced;
- a finding without asset_ref is bound to the report's only asset when there is exactly one (the server applies the same rule to a whole document, RFC-026 §10.1, but a segment may carry several assets).
It returns how many findings still resolve to no asset in the report (empty asset_ref with several or no assets, or a reference to an unknown id). The server rejects those findings; they are never bound to a guess. It modifies r.
func SplitSegments ¶ added in v0.8.0
SplitSegments splits a report into protocol v2 segments. Every segment is a complete CTIS document: the same version, schema, metadata and tool as the report (the server checks they are identical), the findings of that segment and every asset those findings reference. An asset referenced by findings in several segments is repeated in each (the ingest merge is idempotent). Assets no finding references are packed into segments of their own. Dependencies and report properties go into the first segment.
Findings that resolve to no asset (see BindFindingAssets) are put in segments that carry no asset, so they are rejected by the server rather than bound to whichever single asset a segment happens to hold.
Call BindFindingAssets first. One segment is returned when the report fits the limits; it is r itself.
Types ¶
type Chunk ¶
type Chunk struct {
ID string `json:"id"`
ReportID string `json:"report_id"`
ChunkIndex int `json:"chunk_index"`
TotalChunks int `json:"total_chunks"`
Data []byte `json:"-"` // Compressed data (not in JSON)
UncompressedSize int `json:"uncompressed_size"`
CompressedSize int `json:"compressed_size"`
Status ChunkStatus `json:"status"`
RetryCount int `json:"retry_count"`
LastError string `json:"last_error,omitempty"`
CreatedAt time.Time `json:"created_at"`
UploadedAt *time.Time `json:"uploaded_at,omitempty"`
}
Chunk represents a single chunk of a report.
type ChunkData ¶
type ChunkData struct {
ReportID string `json:"report_id"`
ChunkIndex int `json:"chunk_index"`
TotalChunks int `json:"total_chunks"`
Tool *ctis.Tool `json:"tool,omitempty"`
Metadata *ctis.ReportMetadata `json:"metadata,omitempty"`
Assets []ctis.Asset `json:"assets,omitempty"`
Findings []ctis.Finding `json:"findings,omitempty"`
IsFinal bool `json:"is_final"`
}
ChunkData represents the actual data in a chunk. This is what gets serialized, compressed, and uploaded.
type ChunkStatus ¶
type ChunkStatus string
ChunkStatus represents the status of a single chunk.
const ( // ChunkStatusPending indicates the chunk is waiting for upload. ChunkStatusPending ChunkStatus = "pending" // ChunkStatusUploading indicates the chunk is currently uploading. ChunkStatusUploading ChunkStatus = "uploading" // ChunkStatusCompleted indicates the chunk was successfully uploaded. ChunkStatusCompleted ChunkStatus = "completed" // ChunkStatusFailed indicates the chunk failed after all retries. ChunkStatusFailed ChunkStatus = "failed" )
type Config ¶
type Config struct {
// Chunking thresholds - when to trigger chunking
MinFindingsForChunking int // Minimum findings to trigger chunking (default: 2000)
MinAssetsForChunking int // Minimum assets to trigger chunking (default: 200)
MinSizeForChunking int // Minimum raw size in bytes to trigger chunking (default: 5MB)
// Chunk size limits
MaxFindingsPerChunk int // Max findings per chunk (default: 500)
MaxAssetsPerChunk int // Max assets per chunk (default: 100)
MaxChunkSizeBytes int // Max uncompressed size per chunk (default: 2MB)
// Upload behavior
UploadDelayMs int // Delay between chunk uploads in ms (default: 100)
MaxConcurrentUploads int // Max concurrent upload workers (default: 2)
UploadTimeoutSeconds int // Timeout per chunk upload (default: 30)
// Retry configuration
MaxRetries int // Max retries per chunk (default: 3)
RetryBackoffMs int // Initial backoff between retries (default: 1000)
// Storage configuration. Chunks live in a durable outbox (pkg/outbox).
OutboxDir string // Outbox directory (default: "chunk-outbox" next to DatabasePath)
// Deprecated: the SQLite chunk database was replaced by the outbox in
// v0.8.0. DatabasePath now only locates the default OutboxDir (its
// directory) and a leftover database, which is reported, not read.
DatabasePath string
RetentionHours int // Deprecated: unused; the outbox caps age itself (7 days).
MaxStorageMB int // Max bytes the chunk outbox may hold, in MiB (default: 500)
// Compression (chunks are always compressed before storage)
CompressionLevel int // gzip/zstd level 1-9 (default: 3)
// Auto-cleanup configuration
AutoCleanupOnUpload bool // Delete chunk data immediately after successful upload (default: true)
CleanupOnReportComplete bool // Delete all chunks when report completes (default: true)
CleanupIntervalMinutes int // How often to run cleanup in minutes (default: 15)
AggressiveCleanup bool // Enable aggressive cleanup when storage exceeds MaxStorageMB (default: true)
}
Config configures chunking behavior.
func DefaultConfig ¶
func DefaultConfig() *Config
DefaultConfig returns sensible defaults for most environments.
type Manager ¶
type Manager struct {
// contains filtered or unexported fields
}
Manager splits large reports into protocol v1 chunks and uploads them from the durable outbox (pkg/outbox): every chunk is on disk before the first upload and removed only when the platform accepted it, so a crash or restart resumes where it stopped. (Before sdk-go v0.8.0 the chunks lived in a SQLite database; see Config.DatabasePath.)
func NewManager ¶
NewManager creates a chunk manager and opens its outbox.
func (*Manager) Close ¶
Close stops uploading and releases the outbox. Chunks not uploaded yet stay on disk for the next process.
func (*Manager) GetProgress ¶
GetProgress returns upload progress for a report.
func (*Manager) GetStats ¶
func (m *Manager) GetStats(_ context.Context) (*StorageStats, error)
GetStats returns statistics of the reports this manager knows and the bytes its outbox holds.
func (*Manager) NeedsChunking ¶
NeedsChunking checks if a report needs to be chunked.
func (*Manager) Outbox ¶ added in v0.8.0
Outbox returns the manager's outbox (for its Stats and dead letters).
func (*Manager) ProcessPending ¶
ProcessPending uploads every pending chunk that can be uploaded now and returns when none is left (or only chunks backing off after a failure).
func (*Manager) SetCallbacks ¶
func (m *Manager) SetCallbacks(onProgress func(*Progress), onComplete func(string), onError func(string, error))
SetCallbacks sets the callback functions.
func (*Manager) SetUploader ¶
SetUploader configures the uploader.
func (*Manager) SetVerbose ¶
SetVerbose enables verbose logging.
type Metadata ¶
type Metadata struct {
ToolName string `json:"tool_name,omitempty"`
ToolVersion string `json:"tool_version,omitempty"`
ScanID string `json:"scan_id,omitempty"`
SensorID string `json:"sensor_id,omitempty"`
StartedAt time.Time `json:"started_at,omitempty"`
FinishedAt time.Time `json:"finished_at,omitempty"`
}
Metadata stores additional report context.
type Progress ¶
type Progress struct {
ReportID string `json:"report_id"`
TotalChunks int `json:"total_chunks"`
CompletedChunks int `json:"completed_chunks"`
FailedChunks int `json:"failed_chunks"`
PendingChunks int `json:"pending_chunks"`
PercentComplete float64 `json:"percent_complete"`
BytesUploaded int64 `json:"bytes_uploaded"`
BytesTotal int64 `json:"bytes_total"`
Status string `json:"status"`
}
Progress represents upload progress for a report.
type Report ¶
type Report struct {
ID string `json:"id"`
OriginalFindingsCount int `json:"original_findings_count"`
OriginalAssetsCount int `json:"original_assets_count"`
OriginalSize int `json:"original_size"`
CompressedSize int `json:"compressed_size"`
TotalChunks int `json:"total_chunks"`
CompletedChunks int `json:"completed_chunks"`
FailedChunks int `json:"failed_chunks"`
Status ReportStatus `json:"status"`
CompressionAlgo string `json:"compression_algo"`
CreatedAt time.Time `json:"created_at"`
UpdatedAt time.Time `json:"updated_at"`
CompletedAt *time.Time `json:"completed_at,omitempty"`
Metadata *Metadata `json:"metadata,omitempty"`
}
Report represents a chunked report in the database.
func (*Report) CalculateProgress ¶
CalculateProgress calculates progress from a report.
func (*Report) IsComplete ¶
IsComplete checks if all chunks have been uploaded.
type ReportStatus ¶
type ReportStatus string
ReportStatus represents the status of a chunked report.
const ( // ReportStatusPending indicates the report is queued for upload. ReportStatusPending ReportStatus = "pending" // ReportStatusUploading indicates chunks are being uploaded. ReportStatusUploading ReportStatus = "uploading" // ReportStatusCompleted indicates all chunks have been uploaded. ReportStatusCompleted ReportStatus = "completed" // ReportStatusFailed indicates the upload failed after all retries. ReportStatusFailed ReportStatus = "failed" )
type SegmentLimits ¶ added in v0.8.0
type SegmentLimits struct {
// MaxFindings and MaxAssets per segment.
MaxFindings int
MaxAssets int
// MaxBytes is the target size of a segment's JSON before compression.
// A single finding larger than this still gets a segment of its own.
MaxBytes int
}
SegmentLimits bound one protocol v2 segment (RFC-026 §3.5, §3.6). The v2 client derives them from the server's hello limits.
type Splitter ¶
type Splitter struct {
// contains filtered or unexported fields
}
Splitter handles report chunking logic.
func NewSplitter ¶
NewSplitter creates a new splitter with the given config.
func (*Splitter) EstimateChunks ¶
EstimateChunks estimates the number of chunks without actually splitting.
func (*Splitter) NeedsChunking ¶
NeedsChunking determines if a report should be chunked.
func (*Splitter) Split ¶
Split divides a report into chunks.
Algorithm:
- If report is small enough, return single chunk
- Group findings by their asset_ref
- Create chunks that respect both findings and assets limits
- Every chunk is self-describing: it carries the report's tool and metadata, and every asset its findings reference
The platform ingests each chunk as a report of its own. A chunk without the tool was attributed to tool "unknown", and a finding whose asset was sent in an earlier chunk landed on a placeholder "scan:unknown:<report>" asset. So the tool and metadata are repeated in every chunk, and an asset whose findings span several chunks is repeated in each of them.
type StorageStats ¶
type StorageStats struct {
TotalReports int `json:"total_reports"`
PendingReports int `json:"pending_reports"`
UploadingReports int `json:"uploading_reports"`
CompletedReports int `json:"completed_reports"`
FailedReports int `json:"failed_reports"`
TotalChunks int `json:"total_chunks"`
PendingChunks int `json:"pending_chunks"`
CompletedChunks int `json:"completed_chunks"`
FailedChunks int `json:"failed_chunks"`
TotalStorageBytes int64 `json:"total_storage_bytes"`
}
StorageStats contains chunk statistics (Manager.GetStats).