snapshotter

Overview ¶

Package snapshotter captures comprehensive system configuration snapshots.

Overview ¶

The snapshotter package orchestrates parallel collection of system measurements from multiple sources (Kubernetes, GPU, OS, SystemD) and produces structured snapshots that can be serialized for analysis, auditing, or recommendation generation.

Core Types ¶

NodeSnapshotter: collects from the current node (or, when AgentConfig is set, deploys a Kubernetes Job to capture from a remote GPU node).

type NodeSnapshotter struct {
    Version     string                // Snapshotter version
    Factory     collector.Factory     // Collector factory (optional)
    Serializer  serializer.Serializer // Output serializer (optional)
    AgentConfig *AgentConfig          // Optional remote agent deployment
    RequireGPU  bool                  // Fail snapshot if no GPU detected
}

The exported entry point is the Measure method:

func (n *NodeSnapshotter) Measure(ctx context.Context) error

Snapshot: Captured configuration data

type Snapshot struct {
    Header                            // API version, kind, metadata
    Measurements []*measurement.Measurement // Collected data
}

Usage ¶

Basic snapshot with defaults (stdout YAML):

snapshotter := &snapshotter.NodeSnapshotter{
    Version: "v1.0.0",
}

ctx := context.Background()
if err := snapshotter.Measure(ctx); err != nil {
    log.Fatalf("snapshot failed: %v", err)
}

Custom collector factory:

factory := collector.NewDefaultFactory(
    collector.WithSystemDServices([]string{"containerd.service"}),
)

snapshotter := &snapshotter.NodeSnapshotter{
    Version: "v1.0.0",
    Factory: factory,
}

if err := snapshotter.Measure(context.Background()); err != nil {
    log.Fatal(err)
}

Custom output serializer:

serializer, err := serializer.NewFileSerializer("snapshot.json")
if err != nil {
    log.Fatal(err)
}
defer serializer.Close()

snapshotter := &snapshotter.NodeSnapshotter{
    Version:    "v1.0.0",
    Serializer: serializer,
}

if err := snapshotter.Measure(context.Background()); err != nil {
    log.Fatal(err)
}

With timeout:

ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
defer cancel()

snapshotter := &snapshotter.NodeSnapshotter{Version: "v1.0.0"}
if err := snapshotter.Measure(ctx); err != nil {
    log.Fatal(err)
}

Snapshot Structure ¶

Snapshots contain a header and measurements:

apiVersion: aicr.nvidia.com/v1alpha1
kind: Snapshot
metadata:
  version: v1.0.0
  source: node-1
  timestamp: 2025-01-15T10:30:00Z
measurements:
  - type: K8s
    subtypes:
      - subtype: server
        data:
          version: 1.33.5
          platform: linux/amd64
      - subtype: node
        data:
          provider: eks
          kernel-version: 6.8.0
      - subtype: image
        data:
          kube-apiserver: v1.33.5
      - subtype: policy
        data:
          driver.version: 570.86.16
      - subtype: helm
        data:
          gpu-operator.chart: gpu-operator
          gpu-operator.version: 25.3.0
      - subtype: argocd
        data:
          gpu-operator.source.chart: gpu-operator
          gpu-operator.syncStatus: Synced
  - type: GPU
    subtypes:
      - subtype: device
        data:
          driver: 570.158.01
          model: H100

Parallel Collection ¶

NodeSnapshotter runs all collectors concurrently using errgroup:

1. Metadata collection (node name, version)
2. Kubernetes resources (cluster config, policies)
3. SystemD services (containerd, kubelet)
4. OS configuration (grub, sysctl, modules)
5. GPU hardware (driver, model, settings)

Individual collector failures are logged and skipped — the snapshot contains all measurements that could be successfully collected. The overall Measure call only returns an error for setup, context, or serialization failures (and for missing GPU when RequireGPU is set).

Node Name Detection ¶

Node name is determined with fallback priority:

1. NODE_NAME environment variable
2. KUBERNETES_NODE_NAME environment variable
3. HOSTNAME environment variable

This ensures correct node identification in various deployment scenarios.

Error Handling ¶

Measure() returns an error when:

• Context is canceled or times out
• Serialization fails
• RequireGPU is set and no GPU was detected

Individual collector errors do not fail the snapshot; they are logged and the affected measurement is omitted, so partial snapshots are the expected outcome on heterogeneous hosts.

Observability ¶

The snapshotter exports Prometheus metrics:

• snapshot_collection_duration_seconds: Total time to collect snapshot
• snapshot_collector_duration_seconds{collector}: Per-collector timing

Structured logs are emitted for:

• Snapshot start
• Collector progress
• Errors and failures

Resource Requirements ¶

Collectors may require:

• Kubernetes API access (in-cluster config or kubeconfig)
• NVIDIA GPU and nvidia-smi binary
• systemd and systemctl binary
• Read access to /proc, /sys, /etc

Failures due to missing resources are reported as errors.

Integration ¶

The snapshotter is invoked by:

• pkg/cli - snapshot command
• Kubernetes Job - aicr-agent deployment

It depends on:

• pkg/collector - Data collection implementations
• pkg/serializer - Output formatting
• pkg/measurement - Data structures

Snapshots are consumed by:

• pkg/recipe - Recipe generation from snapshots
• External analysis tools
• Auditing and compliance systems