schema

package

v0.1.12 Latest Latest Go to latest Published: Mar 9, 2026 License: GPL-3.0 Imports: 5 Imported by: 0

Details

Valid go.mod file
Redistributable license
Tagged version
Stable version
Learn more about best practices

Repository

github.com/fulgidus/revoco

Links

Open Source Insights

Documentation ¶

Overview ¶

Package schema defines session configuration schemas and migration functions.

This package provides:

Schema definitions for each config version
Version detection from raw JSON
Migration functions between versions

Version history:

v1: Legacy pipeline-based architecture (service + ingester + processor)
v2: Connector-based architecture (connectors + processors)

Constants ¶

This section is empty.

Variables ¶

This section is empty.

Functions ¶

func Migrate ¶

func Migrate(data []byte, from, to Version) ([]byte, error)

Migrate transforms a session config from one version to the next. Returns the migrated config as JSON bytes.

func NeedsMigration ¶

func NeedsMigration(version Version) bool

NeedsMigration checks if a config needs to be migrated to the current version.

Types ¶

type ConfigV1 ¶

type ConfigV1 struct {
	Name               string          `json:"name"`
	Created            time.Time       `json:"created"`
	Updated            time.Time       `json:"updated"`
	Source             SourceV1        `json:"source"`
	OutputDir          string          `json:"output_dir"`
	UseMove            bool            `json:"use_move"`
	DryRun             bool            `json:"dry_run"`
	Recover            RecoverSettings `json:"recover"`
	Status             string          `json:"status"`
	LastPhaseCompleted int             `json:"last_phase_completed"`
	LastError          string          `json:"last_error,omitempty"`
	Pipeline           PipelineConfig  `json:"pipeline,omitempty"`
}

ConfigV1 is the legacy pipeline-based session configuration. This is used for parsing and migration purposes only.

type ConfigV2 ¶

type ConfigV2 struct {
	Name               string           `json:"name"`
	Created            time.Time        `json:"created"`
	Updated            time.Time        `json:"updated"`
	Source             SourceV2         `json:"source"`
	OutputDir          string           `json:"output_dir"`
	UseMove            bool             `json:"use_move"`
	DryRun             bool             `json:"dry_run"`
	Recover            RecoverSettings  `json:"recover"`
	Status             string           `json:"status"`
	LastPhaseCompleted int              `json:"last_phase_completed"`
	LastError          string           `json:"last_error,omitempty"`
	Version            int              `json:"version"` // Always 2 for this schema
	Connectors         ConnectorsConfig `json:"connectors"`
	Pipeline           *PipelineConfig  `json:"pipeline,omitempty"` // Kept for reference during migration
}

ConfigV2 is the connector-based session configuration. This is the current/target schema version.

type ConnectorConfigV2 ¶

type ConnectorConfigV2 struct {
	ConnectorID  string           `json:"connector_id"`
	InstanceID   string           `json:"instance_id"`
	Name         string           `json:"name"`
	Roles        ConnectorRolesV2 `json:"roles"`
	ImportMode   string           `json:"import_mode,omitempty"`
	Settings     map[string]any   `json:"settings"`
	CredentialID string           `json:"credential_id,omitempty"`
	FallbackFor  []string         `json:"fallback_for,omitempty"`
	Enabled      bool             `json:"enabled"`
}

ConnectorConfigV2 holds the configuration for a connector instance.

type ConnectorRolesV2 ¶

type ConnectorRolesV2 struct {
	IsInput    bool `json:"is_input"`    // Primary data source
	IsOutput   bool `json:"is_output"`   // Primary data destination
	IsFallback bool `json:"is_fallback"` // Used for repair/missing data
}

ConnectorRolesV2 describes how a connector instance is used in a session. A connector can have any combination of roles.

type ConnectorsConfig ¶

type ConnectorsConfig struct {
	Connectors        []ConnectorConfigV2 `json:"connectors"`
	ProcessorConfigs  []ProcessorConfigV2 `json:"processor_configs,omitempty"`
	AutoProcess       bool                `json:"auto_process"`
	ParallelRetrieval bool                `json:"parallel_retrieval"`
	DetectedDataTypes []string            `json:"detected_data_types,omitempty"`
	Stats             *DataStats          `json:"stats,omitempty"`
}

ConnectorsConfig holds the connector-based configuration.

type DataStats ¶

type DataStats struct {
	TotalItems    int            `json:"total_items"`
	ByType        map[string]int `json:"by_type"`
	ByConnector   map[string]int `json:"by_connector"`
	TotalSize     int64          `json:"total_size"`
	Duplicates    int            `json:"duplicates"`
	Missing       int            `json:"missing"`
	Repairable    int            `json:"repairable"`
	ProcessedOK   int            `json:"processed_ok"`
	ProcessedFail int            `json:"processed_fail"`
	Errors        []string       `json:"errors,omitempty"`
}

DataStats holds statistics about imported/processed data.

type OutputSettingV1 ¶

type OutputSettingV1 struct {
	OutputID string         `json:"output_id"`
	Config   map[string]any `json:"config"`
}

OutputSettingV1 is a v1 output configuration.

type PipelineConfig ¶

type PipelineConfig struct {
	ServiceID         string            `json:"service_id"`
	IngesterID        string            `json:"ingester_id"`
	ProcessorSettings map[string]any    `json:"processor_settings"`
	OutputSettings    []OutputSettingV1 `json:"output_settings"`
}

PipelineConfig is the v1 service-based pipeline configuration.

type ProcessorConfigV2 ¶

type ProcessorConfigV2 struct {
	ProcessorID string         `json:"processor_id"`
	WorkDir     string         `json:"work_dir"`
	SessionDir  string         `json:"session_dir"`
	DryRun      bool           `json:"dry_run"`
	Settings    map[string]any `json:"settings"`
}

ProcessorConfigV2 holds configuration for a processor.

type RecoverSettings ¶

type RecoverSettings struct {
	InputJSON   string  `json:"input_json"`
	OutputDir   string  `json:"output_dir"`
	Concurrency int     `json:"concurrency"`
	Delay       float64 `json:"delay"`
	MaxRetry    int     `json:"max_retry"`
	StartFrom   int     `json:"start_from"`
}

RecoverSettings holds recovery-specific configuration (same for both versions).

type SourceV1 ¶

type SourceV1 struct {
	Type         string `json:"type"`          // "folder", "zip", "tgz"
	OriginalPath string `json:"original_path"` // path the user provided
	ImportedPath string `json:"imported_path"` // path inside session (if imported)
}

SourceV1 describes the input data for a v1 session.

type SourceV2 ¶

type SourceV2 struct {
	Type         string `json:"type,omitempty"`
	OriginalPath string `json:"original_path,omitempty"`
	ImportedPath string `json:"imported_path,omitempty"`
}

SourceV2 describes the input data for a v2 session. In v2, source is mostly managed by connectors, but we keep this for backwards compat.

type Version ¶

type Version int

Version represents a session config schema version.

const (
	// VersionUnknown indicates the version could not be determined.
	VersionUnknown Version = 0
	// Version1 is the legacy pipeline-based architecture.
	Version1 Version = 1
	// Version2 is the connector-based architecture.
	Version2 Version = 2
	// CurrentVersion is the latest schema version.
	CurrentVersion = Version2
)

func DetectVersion ¶

func DetectVersion(data []byte) (Version, error)

DetectVersion determines the schema version from raw config JSON. It examines the structure to determine which version it matches.

func MigrateToLatest ¶

func MigrateToLatest(data []byte) ([]byte, Version, error)

MigrateToLatest migrates a config to the current version.

func MigrationPath ¶

func MigrationPath(from Version) []Version

MigrationPath returns the sequence of migrations needed to reach the current version.

func (Version) String ¶

func (v Version) String() string

String returns a human-readable version string.

Source Files ¶

View all Source files

?	: This menu
/	: Search site
f or F	: Jump to
y or Y	: Canonical URL