Documentation
¶
Overview ¶
Package core defines the interfaces for revoco's modular connector architecture.
The architecture consists of:
- Connectors: Unified I/O abstractions that can pull data, push data, or both
- DataTypes: Categories of data (photos, videos, notes, music, etc.)
- Processors: Transform data, can be type-specific or source-specific
- Credentials: Authentication storage (global or per-session)
Flow:
- Create empty session
- Add connectors (configure role: input/output/fallback, auth, import mode)
- Retrieve data from input connectors (parallel by default)
- Detect data types, configure processors
- Process data (manual or auto)
- Analyze statistics
- Repair using fallback connectors
- Push to output connectors
Index ¶
- func CanRead(conn Connector) bool
- func CanWrite(conn Connector) bool
- func HasCapability(conn Connector, cap ConnectorCapability) bool
- func RegisterConnector(factory ConnectorFactory) error
- func RegisterProcessor(p Processor) error
- func SupportsDataType(conn Connector, dt DataType) bool
- func ValidateRole(conn Connector, role ConnectorRole) error
- func ValidateRoles(conn Connector, roles ConnectorRoles) error
- type ConfigOption
- type Connector
- type ConnectorCapability
- type ConnectorConfig
- type ConnectorFactory
- type ConnectorInfo
- type ConnectorReader
- type ConnectorRepairer
- type ConnectorRole
- type ConnectorRoles
- type ConnectorTester
- type ConnectorWithSetup
- type ConnectorWriter
- type Credential
- type CredentialScope
- type DataItem
- type DataStats
- type DataType
- type Event
- type EventType
- type FallbackOption
- type ImportMode
- type Processor
- type ProcessorConfig
- type ProcessorScope
- type ProgressFunc
- type Registry
- func (r *Registry) CreateConnector(id string) (Connector, error)
- func (r *Registry) GetApplicableProcessors(item DataItem) []Processor
- func (r *Registry) GetConnectorInfo(id string) (*ConnectorInfo, bool)
- func (r *Registry) GetProcessor(id string) (Processor, bool)
- func (r *Registry) ListConnectors() []*ConnectorInfo
- func (r *Registry) ListConnectorsByCapability(cap ConnectorCapability) []*ConnectorInfo
- func (r *Registry) ListConnectorsByDataType(dt DataType) []*ConnectorInfo
- func (r *Registry) ListInputConnectors() []*ConnectorInfo
- func (r *Registry) ListOutputConnectors() []*ConnectorInfo
- func (r *Registry) ListProcessors() []Processor
- func (r *Registry) ListProcessorsForConnector(connectorID string) []Processor
- func (r *Registry) ListProcessorsForDataType(dt DataType) []Processor
- func (r *Registry) RegisterConnector(factory ConnectorFactory) error
- func (r *Registry) RegisterProcessor(p Processor) error
- func (r *Registry) UnregisterConnector(id string)
- func (r *Registry) UnregisterProcessor(id string)
- type TestResult
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func HasCapability ¶
func HasCapability(conn Connector, cap ConnectorCapability) bool
HasCapability checks if a connector has a specific capability.
func RegisterConnector ¶
func RegisterConnector(factory ConnectorFactory) error
RegisterConnector registers a connector in the global registry.
func RegisterProcessor ¶
RegisterProcessor registers a processor in the global registry.
func SupportsDataType ¶
SupportsDataType checks if a connector supports a specific data type.
func ValidateRole ¶
func ValidateRole(conn Connector, role ConnectorRole) error
ValidateRole checks if a connector can fulfill a given legacy role. Deprecated: Use ValidateRoles instead.
func ValidateRoles ¶
func ValidateRoles(conn Connector, roles ConnectorRoles) error
ValidateRoles checks if a connector can fulfill the given roles.
Types ¶
type ConfigOption ¶
type ConfigOption struct {
ID string `json:"id"`
Name string `json:"name"`
Description string `json:"description"`
Type string `json:"type"` // "bool", "string", "int", "float", "select", "path", "password"
Default any `json:"default"`
Options []string `json:"options,omitempty"` // For "select" type
Required bool `json:"required"`
Sensitive bool `json:"sensitive"` // If true, value should be masked in UI
}
ConfigOption describes a configurable setting.
type Connector ¶
type Connector interface {
// ID returns the unique identifier for this connector type.
ID() string
// Name returns the human-readable name for display.
Name() string
// Description returns a brief description of this connector.
Description() string
// Capabilities returns what this connector can do.
Capabilities() []ConnectorCapability
// SupportedDataTypes returns what types of data this connector handles.
SupportedDataTypes() []DataType
// RequiresAuth returns true if this connector needs authentication.
RequiresAuth() bool
// AuthType returns the authentication type (e.g., "oauth", "apikey", "cookie", "none").
AuthType() string
// ConfigSchema returns the configuration options for this connector.
ConfigSchema() []ConfigOption
// ValidateConfig checks if the provided configuration is valid.
ValidateConfig(cfg ConnectorConfig) error
// FallbackOptions returns connectors that can be used as fallbacks for this one.
// Each option includes setup instructions for the user.
FallbackOptions() []FallbackOption
}
Connector is the unified interface for all data sources and destinations. A connector can act as input, output, or both depending on its capabilities and how it's configured in a session.
func CreateConnector ¶
CreateConnector creates a connector instance from the global registry.
type ConnectorCapability ¶
type ConnectorCapability string
ConnectorCapability describes what a connector can do.
const ( CapabilityRead ConnectorCapability = "read" // Can pull/retrieve data CapabilityWrite ConnectorCapability = "write" // Can push/upload data CapabilityDelete ConnectorCapability = "delete" // Can delete data CapabilityList ConnectorCapability = "list" // Can list available data CapabilitySearch ConnectorCapability = "search" // Can search for specific items CapabilityRepair ConnectorCapability = "repair" // Can fetch missing/corrupted items )
type ConnectorConfig ¶
type ConnectorConfig struct {
// ConnectorID is the type of connector (e.g., "google-photos-api", "local-folder")
ConnectorID string `json:"connector_id"`
// InstanceID is a unique ID for this instance within the session
InstanceID string `json:"instance_id"`
// Name is a user-friendly label (e.g., "My Takeout ZIP", "iCloud Backup")
Name string `json:"name"`
// Roles defines how this connector is used (input/output/fallback in any combination)
Roles ConnectorRoles `json:"roles"`
// ImportMode defines how data is imported (for input connectors)
ImportMode ImportMode `json:"import_mode,omitempty"`
// Settings holds connector-specific configuration
Settings map[string]any `json:"settings"`
// CredentialID references stored credentials (if RequiresAuth)
CredentialID string `json:"credential_id,omitempty"`
// FallbackFor lists instance IDs this connector serves as fallback for
FallbackFor []string `json:"fallback_for,omitempty"`
// Enabled indicates if this connector is active
Enabled bool `json:"enabled"`
}
ConnectorConfig holds the configuration for a connector instance in a session.
type ConnectorFactory ¶
type ConnectorFactory func() Connector
ConnectorFactory creates a new instance of a connector.
type ConnectorInfo ¶
type ConnectorInfo struct {
ID string
Name string
Description string
Capabilities []ConnectorCapability
DataTypes []DataType
RequiresAuth bool
AuthType string
Factory ConnectorFactory
}
ConnectorInfo holds metadata about a registered connector type.
func GetConnectorInfo ¶
func GetConnectorInfo(id string) (*ConnectorInfo, bool)
GetConnectorInfo returns connector info from the global registry.
func ListConnectors ¶
func ListConnectors() []*ConnectorInfo
ListConnectors returns all connectors from the global registry.
type ConnectorReader ¶
type ConnectorReader interface {
Connector
// Initialize prepares the connector for reading (auth, connection, etc.)
Initialize(ctx context.Context, cfg ConnectorConfig) error
// List returns all available items from this connector.
List(ctx context.Context, progress ProgressFunc) ([]DataItem, error)
// Read retrieves a single item's content.
Read(ctx context.Context, item DataItem) (io.ReadCloser, error)
// ReadTo retrieves an item and writes it to the specified path.
ReadTo(ctx context.Context, item DataItem, destPath string, mode ImportMode) error
// Close releases any resources held by the connector.
Close() error
}
ConnectorReader is implemented by connectors that can read/retrieve data.
type ConnectorRepairer ¶
type ConnectorRepairer interface {
ConnectorReader
// CanRepair checks if this connector can provide the given item.
CanRepair(ctx context.Context, item DataItem) (bool, error)
// Repair fetches a missing or corrupted item.
Repair(ctx context.Context, item DataItem, destPath string) error
}
ConnectorRepairer is implemented by connectors that can fetch missing/corrupted items.
type ConnectorRole ¶
type ConnectorRole string
Legacy role constants for migration compatibility
const ( RoleInput ConnectorRole = "input" RoleOutput ConnectorRole = "output" RoleFallback ConnectorRole = "fallback" RoleBoth ConnectorRole = "both" )
func RolesToLegacy ¶
func RolesToLegacy(r ConnectorRoles) ConnectorRole
RolesToLegacy converts ConnectorRoles to the legacy single-value format.
type ConnectorRoles ¶
type ConnectorRoles struct {
IsInput bool `json:"is_input"` // Primary data source
IsOutput bool `json:"is_output"` // Primary data destination
IsFallback bool `json:"is_fallback"` // Used for repair/missing data
}
ConnectorRoles describes how a connector instance is used in a session. A connector can have any combination of roles.
func RolesFromLegacy ¶
func RolesFromLegacy(role ConnectorRole) ConnectorRoles
RolesFromLegacy converts a legacy role string to ConnectorRoles.
func (ConnectorRoles) CanRead ¶
func (r ConnectorRoles) CanRead() bool
CanRead returns true if this connector should be used for reading data.
func (ConnectorRoles) CanWrite ¶
func (r ConnectorRoles) CanWrite() bool
CanWrite returns true if this connector should be used for writing data.
func (ConnectorRoles) HasAnyRole ¶
func (r ConnectorRoles) HasAnyRole() bool
HasAnyRole returns true if at least one role is set.
func (ConnectorRoles) String ¶
func (r ConnectorRoles) String() string
String returns a human-readable representation of the roles.
type ConnectorTester ¶
type ConnectorTester interface {
Connector
// TestConnection verifies the connector configuration is valid and can connect.
// Returns nil if successful, or an error describing what failed.
// For OAuth connectors, this may trigger the authorization flow.
TestConnection(ctx context.Context, cfg ConnectorConfig) error
}
ConnectorTester is an optional interface for connectors that can test their connection. This is used to verify configuration before use and in the dashboard context menu.
type ConnectorWithSetup ¶
type ConnectorWithSetup interface {
Connector
// SetupInstructions returns markdown-formatted setup instructions.
// These are displayed to the user before configuration.
SetupInstructions() string
}
ConnectorWithSetup is an optional interface for connectors that need setup instructions. This is useful for OAuth-based connectors or those requiring external API keys.
type ConnectorWriter ¶
type ConnectorWriter interface {
Connector
// Initialize prepares the connector for writing.
Initialize(ctx context.Context, cfg ConnectorConfig) error
// Write sends a single item to the destination.
Write(ctx context.Context, item DataItem, reader io.Reader) error
// WriteFrom sends an item from a local path to the destination.
WriteFrom(ctx context.Context, item DataItem, sourcePath string) error
// WriteBatch sends multiple items (for connectors that benefit from batching).
WriteBatch(ctx context.Context, items []DataItem, getReader func(DataItem) (io.Reader, error), progress ProgressFunc) error
// Delete removes an item from the destination (if supported).
Delete(ctx context.Context, item DataItem) error
// Close releases any resources.
Close() error
}
ConnectorWriter is implemented by connectors that can write/push data.
type Credential ¶
type Credential struct {
ID string `json:"id"`
Name string `json:"name"` // User-friendly label
ConnectorID string `json:"connector_id"`
Scope CredentialScope `json:"scope"`
AuthType string `json:"auth_type"` // "oauth", "apikey", "cookie", etc.
// Data is encrypted at rest - contents depend on AuthType
Data map[string]any `json:"data"`
CreatedAt int64 `json:"created_at"`
UpdatedAt int64 `json:"updated_at"`
ExpiresAt int64 `json:"expires_at,omitempty"` // 0 = no expiry
}
Credential represents stored authentication data.
type CredentialScope ¶
type CredentialScope string
CredentialScope defines where credentials are stored.
const ( CredentialScopeGlobal CredentialScope = "global" // Shared across sessions CredentialScopeSession CredentialScope = "session" // Specific to one session )
type DataItem ¶
type DataItem struct {
ID string `json:"id"`
Type DataType `json:"type"`
Path string `json:"path"` // Local path (if available)
RemoteID string `json:"remote_id"` // ID in remote system (if applicable)
SourceConnID string `json:"source_conn_id"` // Which connector provided this
Metadata map[string]any `json:"metadata"`
Size int64 `json:"size"`
Checksum string `json:"checksum,omitempty"`
}
DataItem represents a single piece of data with its type and metadata.
type DataStats ¶
type DataStats struct {
TotalItems int `json:"total_items"`
ByType map[DataType]int `json:"by_type"`
ByConnector map[string]int `json:"by_connector"`
TotalSize int64 `json:"total_size"`
Duplicates int `json:"duplicates"`
Missing int `json:"missing"` // Items that failed to import
Repairable int `json:"repairable"` // Missing items that can be repaired
ProcessedOK int `json:"processed_ok"` // Successfully processed
ProcessedFail int `json:"processed_fail"` // Failed processing
Errors []string `json:"errors,omitempty"`
}
DataStats holds statistics about imported/processed data.
type DataType ¶
type DataType string
DataType represents a category of data that can be processed.
const ( DataTypePhoto DataType = "photo" DataTypeVideo DataType = "video" DataTypeAudio DataType = "audio" DataTypeNote DataType = "note" DataTypePlaylist DataType = "playlist" DataTypeAlbum DataType = "album" DataTypeContact DataType = "contact" DataTypeDocument DataType = "document" DataTypeUnknown DataType = "unknown" )
type Event ¶
type Event struct {
Type EventType `json:"type"`
ConnectorID string `json:"connector_id,omitempty"`
ProcessorID string `json:"processor_id,omitempty"`
Phase string `json:"phase"`
Done int `json:"done"`
Total int `json:"total"`
Message string `json:"message"`
ItemID string `json:"item_id,omitempty"`
Error error `json:"-"`
}
Event represents something that happened during an operation.
type FallbackOption ¶
type FallbackOption struct {
// ConnectorID is the connector type that can serve as fallback
ConnectorID string `json:"connector_id"`
// Name is a human-readable name
Name string `json:"name"`
// Description explains what this fallback provides
Description string `json:"description"`
// SetupInstructions tells the user how to configure this fallback
SetupInstructions string `json:"setup_instructions"`
// RequiredCapabilities lists what the fallback needs to provide
RequiredCapabilities []ConnectorCapability `json:"required_capabilities"`
}
FallbackOption describes a possible fallback connector with setup instructions.
type ImportMode ¶
type ImportMode string
ImportMode describes how data should be imported from a connector.
const ( ImportModeCopy ImportMode = "copy" // Copy data to session folder ImportModeMove ImportMode = "move" // Move data to session folder ImportModeReference ImportMode = "reference" // Just reference, don't copy )
type Processor ¶
type Processor interface {
// ID returns the unique identifier for this processor.
ID() string
// Name returns the human-readable name.
Name() string
// Description describes what this processor does.
Description() string
// Scope returns whether this processor is data-type or connector specific.
Scope() ProcessorScope
// SupportedDataTypes returns data types this processor handles (if scope is data_type).
SupportedDataTypes() []DataType
// SupportedConnectors returns connector IDs this processor handles (if scope is connector).
SupportedConnectors() []string
// ConfigSchema returns configuration options.
ConfigSchema() []ConfigOption
// CanProcess checks if this processor can handle the given item.
CanProcess(item DataItem) bool
// Process transforms a single item. Returns the processed item (may be same or new).
Process(ctx context.Context, item DataItem, cfg ProcessorConfig) (*DataItem, error)
// ProcessBatch transforms multiple items.
ProcessBatch(ctx context.Context, items []DataItem, cfg ProcessorConfig, progress ProgressFunc) ([]DataItem, error)
}
Processor transforms and processes data items.
func GetProcessor ¶
GetProcessor returns a processor from the global registry.
func ListProcessors ¶
func ListProcessors() []Processor
ListProcessors returns all processors from the global registry.
type ProcessorConfig ¶
type ProcessorConfig struct {
ProcessorID string `json:"processor_id"`
WorkDir string `json:"work_dir"`
SessionDir string `json:"session_dir"`
DryRun bool `json:"dry_run"`
Settings map[string]any `json:"settings"`
}
ProcessorConfig holds configuration for a processor.
type ProcessorScope ¶
type ProcessorScope string
ProcessorScope defines what a processor operates on.
const ( // ProcessorScopeDataType - processor works on any data of specific types ProcessorScopeDataType ProcessorScope = "data_type" // ProcessorScopeConnector - processor is specific to data from certain connectors ProcessorScopeConnector ProcessorScope = "connector" )
type ProgressFunc ¶
type ProgressFunc func(done, total int)
ProgressFunc reports progress during operations.
type Registry ¶
type Registry struct {
// contains filtered or unexported fields
}
Registry manages available connector types.
func (*Registry) CreateConnector ¶
CreateConnector creates a new instance of a registered connector type.
func (*Registry) GetApplicableProcessors ¶
GetApplicableProcessors returns all processors that can handle a given item.
func (*Registry) GetConnectorInfo ¶
func (r *Registry) GetConnectorInfo(id string) (*ConnectorInfo, bool)
GetConnectorInfo returns info about a registered connector type.
func (*Registry) GetProcessor ¶
GetProcessor returns a registered processor by ID.
func (*Registry) ListConnectors ¶
func (r *Registry) ListConnectors() []*ConnectorInfo
ListConnectors returns info about all registered connectors.
func (*Registry) ListConnectorsByCapability ¶
func (r *Registry) ListConnectorsByCapability(cap ConnectorCapability) []*ConnectorInfo
ListConnectorsByCapability returns connectors that have a specific capability.
func (*Registry) ListConnectorsByDataType ¶
func (r *Registry) ListConnectorsByDataType(dt DataType) []*ConnectorInfo
ListConnectorsByDataType returns connectors that support a specific data type.
func (*Registry) ListInputConnectors ¶
func (r *Registry) ListInputConnectors() []*ConnectorInfo
ListInputConnectors returns all connectors that can read data.
func (*Registry) ListOutputConnectors ¶
func (r *Registry) ListOutputConnectors() []*ConnectorInfo
ListOutputConnectors returns all connectors that can write data.
func (*Registry) ListProcessors ¶
ListProcessors returns all registered processors.
func (*Registry) ListProcessorsForConnector ¶
ListProcessorsForConnector returns processors specific to a connector.
func (*Registry) ListProcessorsForDataType ¶
ListProcessorsForDataType returns processors that handle a specific data type.
func (*Registry) RegisterConnector ¶
func (r *Registry) RegisterConnector(factory ConnectorFactory) error
RegisterConnector adds a connector type to the registry.
func (*Registry) RegisterProcessor ¶
RegisterProcessor adds a processor to the registry.
func (*Registry) UnregisterConnector ¶
UnregisterConnector removes a connector type from the registry.
func (*Registry) UnregisterProcessor ¶
UnregisterProcessor removes a processor from the registry.
type TestResult ¶
type TestResult struct {
Success bool `json:"success"`
Message string `json:"message"`
Details string `json:"details,omitempty"`
}
TestResult contains the result of a connection test.
Directories
¶
| Path | Synopsis |
|---|---|
|
Package googledrive provides a connector for Google Drive with OAuth2 authentication.
|
Package googledrive provides a connector for Google Drive with OAuth2 authentication. |
|
Package local provides connectors for local filesystem sources.
|
Package local provides connectors for local filesystem sources. |