Documentation
¶
Overview ¶
Package storage provides abstractions for storage providers.
Index ¶
- type AccessExample
- type DatasetAvailability
- type DatasetIdentifier
- type NoopProvider
- func (p *NoopProvider) Close() error
- func (p *NoopProvider) GetAccessExamples(_ context.Context, _ string) ([]AccessExample, error)
- func (p *NoopProvider) GetDatasetAvailability(_ context.Context, _ string) (*DatasetAvailability, error)
- func (p *NoopProvider) ListObjects(_ context.Context, _ DatasetIdentifier, _ int) ([]ObjectInfo, error)
- func (p *NoopProvider) Name() string
- func (p *NoopProvider) ResolveDataset(_ context.Context, _ string) (*DatasetIdentifier, error)
- type ObjectInfo
- type Provider
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type AccessExample ¶
type AccessExample struct {
Description string `json:"description"`
Command string `json:"command"`
SDK string `json:"sdk,omitempty"`
}
AccessExample provides an example of how to access a dataset.
type DatasetAvailability ¶
type DatasetAvailability struct {
Available bool `json:"available"`
Bucket string `json:"bucket,omitempty"`
Prefix string `json:"prefix,omitempty"`
Connection string `json:"connection,omitempty"`
ObjectCount int64 `json:"object_count,omitempty"`
TotalSize int64 `json:"total_size,omitempty"`
LastUpdated *time.Time `json:"last_updated,omitempty"`
Error string `json:"error,omitempty"`
}
DatasetAvailability indicates if a dataset is available in storage.
type DatasetIdentifier ¶
type DatasetIdentifier struct {
Bucket string `json:"bucket"`
Prefix string `json:"prefix,omitempty"`
Connection string `json:"connection,omitempty"`
}
DatasetIdentifier uniquely identifies a dataset in storage.
func (DatasetIdentifier) String ¶
func (d DatasetIdentifier) String() string
String returns a string representation.
type NoopProvider ¶
type NoopProvider struct{}
NoopProvider is a no-op implementation for testing.
func NewNoopProvider ¶
func NewNoopProvider() *NoopProvider
NewNoopProvider creates a new no-op provider.
func (*NoopProvider) GetAccessExamples ¶
func (p *NoopProvider) GetAccessExamples(_ context.Context, _ string) ([]AccessExample, error)
GetAccessExamples returns empty for no-op.
func (*NoopProvider) GetDatasetAvailability ¶
func (p *NoopProvider) GetDatasetAvailability(_ context.Context, _ string) (*DatasetAvailability, error)
GetDatasetAvailability returns unavailable for no-op.
func (*NoopProvider) ListObjects ¶
func (p *NoopProvider) ListObjects(_ context.Context, _ DatasetIdentifier, _ int) ([]ObjectInfo, error)
ListObjects returns empty for no-op.
func (*NoopProvider) ResolveDataset ¶
func (p *NoopProvider) ResolveDataset(_ context.Context, _ string) (*DatasetIdentifier, error)
ResolveDataset returns nil for no-op.
type ObjectInfo ¶
type ObjectInfo struct {
Key string `json:"key"`
Bucket string `json:"bucket"`
Size int64 `json:"size"`
LastModified *time.Time `json:"last_modified,omitempty"`
ContentType string `json:"content_type,omitempty"`
Metadata map[string]string `json:"metadata,omitempty"`
}
ObjectInfo provides information about a storage object.
type Provider ¶
type Provider interface {
// Name returns the provider name.
Name() string
// ResolveDataset converts a URN to a storage dataset identifier.
ResolveDataset(ctx context.Context, urn string) (*DatasetIdentifier, error)
// GetDatasetAvailability checks if a dataset is available in storage.
GetDatasetAvailability(ctx context.Context, urn string) (*DatasetAvailability, error)
// GetAccessExamples returns examples for accessing a dataset.
GetAccessExamples(ctx context.Context, urn string) ([]AccessExample, error)
// ListObjects lists objects in a dataset prefix.
ListObjects(ctx context.Context, dataset DatasetIdentifier, limit int) ([]ObjectInfo, error)
// Close releases resources.
Close() error
}
Provider provides storage availability context for metadata entities. S3 implements this. Future storage systems (GCS, Azure Blob) can too.
Source Files
Directories