core

package
v0.1.2 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Mar 1, 2026 License: GPL-3.0 Imports: 4 Imported by: 0

Documentation

Overview

Package core defines the interfaces for revoco's modular connector architecture.

The architecture consists of:

  • Connectors: Unified I/O abstractions that can pull data, push data, or both
  • DataTypes: Categories of data (photos, videos, notes, music, etc.)
  • Processors: Transform data, can be type-specific or source-specific
  • Credentials: Authentication storage (global or per-session)

Flow:

  1. Create empty session
  2. Add connectors (configure role: input/output/fallback, auth, import mode)
  3. Retrieve data from input connectors (parallel by default)
  4. Detect data types, configure processors
  5. Process data (manual or auto)
  6. Analyze statistics
  7. Repair using fallback connectors
  8. Push to output connectors

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func CanRead

func CanRead(conn Connector) bool

CanRead checks if a connector can read/retrieve data.

func CanWrite

func CanWrite(conn Connector) bool

CanWrite checks if a connector can write/push data.

func HasCapability

func HasCapability(conn Connector, cap ConnectorCapability) bool

HasCapability checks if a connector has a specific capability.

func RegisterConnector

func RegisterConnector(factory ConnectorFactory) error

RegisterConnector registers a connector in the global registry.

func RegisterProcessor

func RegisterProcessor(p Processor) error

RegisterProcessor registers a processor in the global registry.

func SupportsDataType

func SupportsDataType(conn Connector, dt DataType) bool

SupportsDataType checks if a connector supports a specific data type.

func ValidateRole

func ValidateRole(conn Connector, role ConnectorRole) error

ValidateRole checks if a connector can fulfill a given legacy role. Deprecated: Use ValidateRoles instead.

func ValidateRoles

func ValidateRoles(conn Connector, roles ConnectorRoles) error

ValidateRoles checks if a connector can fulfill the given roles.

Types

type ConfigOption

type ConfigOption struct {
	ID          string   `json:"id"`
	Name        string   `json:"name"`
	Description string   `json:"description"`
	Type        string   `json:"type"` // "bool", "string", "int", "float", "select", "path", "password"
	Default     any      `json:"default"`
	Options     []string `json:"options,omitempty"` // For "select" type
	Required    bool     `json:"required"`
	Sensitive   bool     `json:"sensitive"` // If true, value should be masked in UI
}

ConfigOption describes a configurable setting.

type Connector

type Connector interface {

	// ID returns the unique identifier for this connector type.
	ID() string
	// Name returns the human-readable name for display.
	Name() string
	// Description returns a brief description of this connector.
	Description() string

	// Capabilities returns what this connector can do.
	Capabilities() []ConnectorCapability
	// SupportedDataTypes returns what types of data this connector handles.
	SupportedDataTypes() []DataType
	// RequiresAuth returns true if this connector needs authentication.
	RequiresAuth() bool
	// AuthType returns the authentication type (e.g., "oauth", "apikey", "cookie", "none").
	AuthType() string

	// ConfigSchema returns the configuration options for this connector.
	ConfigSchema() []ConfigOption
	// ValidateConfig checks if the provided configuration is valid.
	ValidateConfig(cfg ConnectorConfig) error

	// FallbackOptions returns connectors that can be used as fallbacks for this one.
	// Each option includes setup instructions for the user.
	FallbackOptions() []FallbackOption
}

Connector is the unified interface for all data sources and destinations. A connector can act as input, output, or both depending on its capabilities and how it's configured in a session.

func CreateConnector

func CreateConnector(id string) (Connector, error)

CreateConnector creates a connector instance from the global registry.

type ConnectorCapability

type ConnectorCapability string

ConnectorCapability describes what a connector can do.

const (
	CapabilityRead   ConnectorCapability = "read"   // Can pull/retrieve data
	CapabilityWrite  ConnectorCapability = "write"  // Can push/upload data
	CapabilityDelete ConnectorCapability = "delete" // Can delete data
	CapabilityList   ConnectorCapability = "list"   // Can list available data
	CapabilitySearch ConnectorCapability = "search" // Can search for specific items
	CapabilityRepair ConnectorCapability = "repair" // Can fetch missing/corrupted items
)

type ConnectorConfig

type ConnectorConfig struct {
	// ConnectorID is the type of connector (e.g., "google-photos-api", "local-folder")
	ConnectorID string `json:"connector_id"`
	// InstanceID is a unique ID for this instance within the session
	InstanceID string `json:"instance_id"`
	// Name is a user-friendly label (e.g., "My Takeout ZIP", "iCloud Backup")
	Name string `json:"name"`
	// Roles defines how this connector is used (input/output/fallback in any combination)
	Roles ConnectorRoles `json:"roles"`
	// ImportMode defines how data is imported (for input connectors)
	ImportMode ImportMode `json:"import_mode,omitempty"`
	// Settings holds connector-specific configuration
	Settings map[string]any `json:"settings"`
	// CredentialID references stored credentials (if RequiresAuth)
	CredentialID string `json:"credential_id,omitempty"`
	// FallbackFor lists instance IDs this connector serves as fallback for
	FallbackFor []string `json:"fallback_for,omitempty"`
	// Enabled indicates if this connector is active
	Enabled bool `json:"enabled"`
}

ConnectorConfig holds the configuration for a connector instance in a session.

type ConnectorFactory

type ConnectorFactory func() Connector

ConnectorFactory creates a new instance of a connector.

type ConnectorInfo

type ConnectorInfo struct {
	ID           string
	Name         string
	Description  string
	Capabilities []ConnectorCapability
	DataTypes    []DataType
	RequiresAuth bool
	AuthType     string
	Factory      ConnectorFactory
}

ConnectorInfo holds metadata about a registered connector type.

func GetConnectorInfo

func GetConnectorInfo(id string) (*ConnectorInfo, bool)

GetConnectorInfo returns connector info from the global registry.

func ListConnectors

func ListConnectors() []*ConnectorInfo

ListConnectors returns all connectors from the global registry.

type ConnectorReader

type ConnectorReader interface {
	Connector
	// Initialize prepares the connector for reading (auth, connection, etc.)
	Initialize(ctx context.Context, cfg ConnectorConfig) error
	// List returns all available items from this connector.
	List(ctx context.Context, progress ProgressFunc) ([]DataItem, error)
	// Read retrieves a single item's content.
	Read(ctx context.Context, item DataItem) (io.ReadCloser, error)
	// ReadTo retrieves an item and writes it to the specified path.
	ReadTo(ctx context.Context, item DataItem, destPath string, mode ImportMode) error
	// Close releases any resources held by the connector.
	Close() error
}

ConnectorReader is implemented by connectors that can read/retrieve data.

type ConnectorRepairer

type ConnectorRepairer interface {
	ConnectorReader
	// CanRepair checks if this connector can provide the given item.
	CanRepair(ctx context.Context, item DataItem) (bool, error)
	// Repair fetches a missing or corrupted item.
	Repair(ctx context.Context, item DataItem, destPath string) error
}

ConnectorRepairer is implemented by connectors that can fetch missing/corrupted items.

type ConnectorRole

type ConnectorRole string

Legacy role constants for migration compatibility

const (
	RoleInput    ConnectorRole = "input"
	RoleOutput   ConnectorRole = "output"
	RoleFallback ConnectorRole = "fallback"
	RoleBoth     ConnectorRole = "both"
)

func RolesToLegacy

func RolesToLegacy(r ConnectorRoles) ConnectorRole

RolesToLegacy converts ConnectorRoles to the legacy single-value format.

type ConnectorRoles

type ConnectorRoles struct {
	IsInput    bool `json:"is_input"`    // Primary data source
	IsOutput   bool `json:"is_output"`   // Primary data destination
	IsFallback bool `json:"is_fallback"` // Used for repair/missing data
}

ConnectorRoles describes how a connector instance is used in a session. A connector can have any combination of roles.

func RolesFromLegacy

func RolesFromLegacy(role ConnectorRole) ConnectorRoles

RolesFromLegacy converts a legacy role string to ConnectorRoles.

func (ConnectorRoles) CanRead

func (r ConnectorRoles) CanRead() bool

CanRead returns true if this connector should be used for reading data.

func (ConnectorRoles) CanWrite

func (r ConnectorRoles) CanWrite() bool

CanWrite returns true if this connector should be used for writing data.

func (ConnectorRoles) HasAnyRole

func (r ConnectorRoles) HasAnyRole() bool

HasAnyRole returns true if at least one role is set.

func (ConnectorRoles) String

func (r ConnectorRoles) String() string

String returns a human-readable representation of the roles.

type ConnectorTester

type ConnectorTester interface {
	Connector
	// TestConnection verifies the connector configuration is valid and can connect.
	// Returns nil if successful, or an error describing what failed.
	// For OAuth connectors, this may trigger the authorization flow.
	TestConnection(ctx context.Context, cfg ConnectorConfig) error
}

ConnectorTester is an optional interface for connectors that can test their connection. This is used to verify configuration before use and in the dashboard context menu.

type ConnectorWithSetup

type ConnectorWithSetup interface {
	Connector
	// SetupInstructions returns markdown-formatted setup instructions.
	// These are displayed to the user before configuration.
	SetupInstructions() string
}

ConnectorWithSetup is an optional interface for connectors that need setup instructions. This is useful for OAuth-based connectors or those requiring external API keys.

type ConnectorWriter

type ConnectorWriter interface {
	Connector
	// Initialize prepares the connector for writing.
	Initialize(ctx context.Context, cfg ConnectorConfig) error
	// Write sends a single item to the destination.
	Write(ctx context.Context, item DataItem, reader io.Reader) error
	// WriteFrom sends an item from a local path to the destination.
	WriteFrom(ctx context.Context, item DataItem, sourcePath string) error
	// WriteBatch sends multiple items (for connectors that benefit from batching).
	WriteBatch(ctx context.Context, items []DataItem, getReader func(DataItem) (io.Reader, error), progress ProgressFunc) error
	// Delete removes an item from the destination (if supported).
	Delete(ctx context.Context, item DataItem) error
	// Close releases any resources.
	Close() error
}

ConnectorWriter is implemented by connectors that can write/push data.

type Credential

type Credential struct {
	ID          string          `json:"id"`
	Name        string          `json:"name"` // User-friendly label
	ConnectorID string          `json:"connector_id"`
	Scope       CredentialScope `json:"scope"`
	AuthType    string          `json:"auth_type"` // "oauth", "apikey", "cookie", etc.
	// Data is encrypted at rest - contents depend on AuthType
	Data      map[string]any `json:"data"`
	CreatedAt int64          `json:"created_at"`
	UpdatedAt int64          `json:"updated_at"`
	ExpiresAt int64          `json:"expires_at,omitempty"` // 0 = no expiry
}

Credential represents stored authentication data.

type CredentialScope

type CredentialScope string

CredentialScope defines where credentials are stored.

const (
	CredentialScopeGlobal  CredentialScope = "global"  // Shared across sessions
	CredentialScopeSession CredentialScope = "session" // Specific to one session
)

type DataItem

type DataItem struct {
	ID           string         `json:"id"`
	Type         DataType       `json:"type"`
	Path         string         `json:"path"`           // Local path (if available)
	RemoteID     string         `json:"remote_id"`      // ID in remote system (if applicable)
	SourceConnID string         `json:"source_conn_id"` // Which connector provided this
	Metadata     map[string]any `json:"metadata"`
	Size         int64          `json:"size"`
	Checksum     string         `json:"checksum,omitempty"`
}

DataItem represents a single piece of data with its type and metadata.

type DataStats

type DataStats struct {
	TotalItems    int              `json:"total_items"`
	ByType        map[DataType]int `json:"by_type"`
	ByConnector   map[string]int   `json:"by_connector"`
	TotalSize     int64            `json:"total_size"`
	Duplicates    int              `json:"duplicates"`
	Missing       int              `json:"missing"`        // Items that failed to import
	Repairable    int              `json:"repairable"`     // Missing items that can be repaired
	ProcessedOK   int              `json:"processed_ok"`   // Successfully processed
	ProcessedFail int              `json:"processed_fail"` // Failed processing
	Errors        []string         `json:"errors,omitempty"`
}

DataStats holds statistics about imported/processed data.

type DataType

type DataType string

DataType represents a category of data that can be processed.

const (
	DataTypePhoto    DataType = "photo"
	DataTypeVideo    DataType = "video"
	DataTypeAudio    DataType = "audio"
	DataTypeNote     DataType = "note"
	DataTypePlaylist DataType = "playlist"
	DataTypeAlbum    DataType = "album"
	DataTypeContact  DataType = "contact"
	DataTypeDocument DataType = "document"
	DataTypeUnknown  DataType = "unknown"
)

type Event

type Event struct {
	Type        EventType `json:"type"`
	ConnectorID string    `json:"connector_id,omitempty"`
	ProcessorID string    `json:"processor_id,omitempty"`
	Phase       string    `json:"phase"`
	Done        int       `json:"done"`
	Total       int       `json:"total"`
	Message     string    `json:"message"`
	ItemID      string    `json:"item_id,omitempty"`
	Error       error     `json:"-"`
}

Event represents something that happened during an operation.

type EventType

type EventType string

EventType categorizes events during operations.

const (
	EventTypeInfo     EventType = "info"
	EventTypeProgress EventType = "progress"
	EventTypeWarning  EventType = "warning"
	EventTypeError    EventType = "error"
	EventTypeComplete EventType = "complete"
)

type FallbackOption

type FallbackOption struct {
	// ConnectorID is the connector type that can serve as fallback
	ConnectorID string `json:"connector_id"`
	// Name is a human-readable name
	Name string `json:"name"`
	// Description explains what this fallback provides
	Description string `json:"description"`
	// SetupInstructions tells the user how to configure this fallback
	SetupInstructions string `json:"setup_instructions"`
	// RequiredCapabilities lists what the fallback needs to provide
	RequiredCapabilities []ConnectorCapability `json:"required_capabilities"`
}

FallbackOption describes a possible fallback connector with setup instructions.

type ImportMode

type ImportMode string

ImportMode describes how data should be imported from a connector.

const (
	ImportModeCopy      ImportMode = "copy"      // Copy data to session folder
	ImportModeMove      ImportMode = "move"      // Move data to session folder
	ImportModeReference ImportMode = "reference" // Just reference, don't copy
)

type Processor

type Processor interface {
	// ID returns the unique identifier for this processor.
	ID() string
	// Name returns the human-readable name.
	Name() string
	// Description describes what this processor does.
	Description() string

	// Scope returns whether this processor is data-type or connector specific.
	Scope() ProcessorScope
	// SupportedDataTypes returns data types this processor handles (if scope is data_type).
	SupportedDataTypes() []DataType
	// SupportedConnectors returns connector IDs this processor handles (if scope is connector).
	SupportedConnectors() []string

	// ConfigSchema returns configuration options.
	ConfigSchema() []ConfigOption
	// CanProcess checks if this processor can handle the given item.
	CanProcess(item DataItem) bool
	// Process transforms a single item. Returns the processed item (may be same or new).
	Process(ctx context.Context, item DataItem, cfg ProcessorConfig) (*DataItem, error)
	// ProcessBatch transforms multiple items.
	ProcessBatch(ctx context.Context, items []DataItem, cfg ProcessorConfig, progress ProgressFunc) ([]DataItem, error)
}

Processor transforms and processes data items.

func GetProcessor

func GetProcessor(id string) (Processor, bool)

GetProcessor returns a processor from the global registry.

func ListProcessors

func ListProcessors() []Processor

ListProcessors returns all processors from the global registry.

type ProcessorConfig

type ProcessorConfig struct {
	ProcessorID string         `json:"processor_id"`
	WorkDir     string         `json:"work_dir"`
	SessionDir  string         `json:"session_dir"`
	DryRun      bool           `json:"dry_run"`
	Settings    map[string]any `json:"settings"`
}

ProcessorConfig holds configuration for a processor.

type ProcessorScope

type ProcessorScope string

ProcessorScope defines what a processor operates on.

const (
	// ProcessorScopeDataType - processor works on any data of specific types
	ProcessorScopeDataType ProcessorScope = "data_type"
	// ProcessorScopeConnector - processor is specific to data from certain connectors
	ProcessorScopeConnector ProcessorScope = "connector"
)

type ProgressFunc

type ProgressFunc func(done, total int)

ProgressFunc reports progress during operations.

type Registry

type Registry struct {
	// contains filtered or unexported fields
}

Registry manages available connector types.

func Global

func Global() *Registry

Global returns the global registry instance.

func NewRegistry

func NewRegistry() *Registry

NewRegistry creates a new empty registry.

func (*Registry) CreateConnector

func (r *Registry) CreateConnector(id string) (Connector, error)

CreateConnector creates a new instance of a registered connector type.

func (*Registry) GetApplicableProcessors

func (r *Registry) GetApplicableProcessors(item DataItem) []Processor

GetApplicableProcessors returns all processors that can handle a given item.

func (*Registry) GetConnectorInfo

func (r *Registry) GetConnectorInfo(id string) (*ConnectorInfo, bool)

GetConnectorInfo returns info about a registered connector type.

func (*Registry) GetProcessor

func (r *Registry) GetProcessor(id string) (Processor, bool)

GetProcessor returns a registered processor by ID.

func (*Registry) ListConnectors

func (r *Registry) ListConnectors() []*ConnectorInfo

ListConnectors returns info about all registered connectors.

func (*Registry) ListConnectorsByCapability

func (r *Registry) ListConnectorsByCapability(cap ConnectorCapability) []*ConnectorInfo

ListConnectorsByCapability returns connectors that have a specific capability.

func (*Registry) ListConnectorsByDataType

func (r *Registry) ListConnectorsByDataType(dt DataType) []*ConnectorInfo

ListConnectorsByDataType returns connectors that support a specific data type.

func (*Registry) ListInputConnectors

func (r *Registry) ListInputConnectors() []*ConnectorInfo

ListInputConnectors returns all connectors that can read data.

func (*Registry) ListOutputConnectors

func (r *Registry) ListOutputConnectors() []*ConnectorInfo

ListOutputConnectors returns all connectors that can write data.

func (*Registry) ListProcessors

func (r *Registry) ListProcessors() []Processor

ListProcessors returns all registered processors.

func (*Registry) ListProcessorsForConnector

func (r *Registry) ListProcessorsForConnector(connectorID string) []Processor

ListProcessorsForConnector returns processors specific to a connector.

func (*Registry) ListProcessorsForDataType

func (r *Registry) ListProcessorsForDataType(dt DataType) []Processor

ListProcessorsForDataType returns processors that handle a specific data type.

func (*Registry) RegisterConnector

func (r *Registry) RegisterConnector(factory ConnectorFactory) error

RegisterConnector adds a connector type to the registry.

func (*Registry) RegisterProcessor

func (r *Registry) RegisterProcessor(p Processor) error

RegisterProcessor adds a processor to the registry.

func (*Registry) UnregisterConnector

func (r *Registry) UnregisterConnector(id string)

UnregisterConnector removes a connector type from the registry.

func (*Registry) UnregisterProcessor

func (r *Registry) UnregisterProcessor(id string)

UnregisterProcessor removes a processor from the registry.

type TestResult

type TestResult struct {
	Success bool   `json:"success"`
	Message string `json:"message"`
	Details string `json:"details,omitempty"`
}

TestResult contains the result of a connection test.

Directories

Path Synopsis
Package googledrive provides a connector for Google Drive with OAuth2 authentication.
Package googledrive provides a connector for Google Drive with OAuth2 authentication.
Package local provides connectors for local filesystem sources.
Package local provides connectors for local filesystem sources.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL