db

package
v0.3.1 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jun 8, 2026 License: MIT Imports: 5 Imported by: 0

README

Database Architecture

This directory contains all persistence code for syfon.

Layout

  • core/
    • shared interfaces, models, authz/resource helpers, typed DB errors
  • sqlite/
    • SQLite implementation of core.DatabaseInterface
  • postgres/
    • PostgreSQL implementation of core.DatabaseInterface
  • testing.go
    • in-memory DB helper for tests

Primary Tables

The server stores object metadata in a normalized schema:

drs_object

Core record row (one row per object).

  • id (PK, text): canonical object id (current implementation uses sha256 in many flows)
  • size (bigint/int)
  • created_time (timestamp)
  • updated_time (timestamp)
  • name (text)
  • version (text)
  • description (text)
drs_object_access_method

One-to-many scoped access locations for each object.

  • object_id (FK -> drs_object.id)
  • url (text), e.g. s3://bucket/key
  • type (text), e.g. s3
  • org (text), e.g. my-program
  • project (text), e.g. my-project
drs_object_checksum

One-to-many checksum values for each object.

  • object_id (FK -> drs_object.id)
  • type (text), e.g. sha256, md5
  • checksum (text)
s3_credential

Bucket-level signing credentials.

  • bucket (PK, text)
  • region (text)
  • access_key (text)
  • secret_key (text)
  • endpoint (text, optional)

RBAC Model

  • Local mode:
    • no gen3 RBAC enforcement
    • optional basic auth at middleware level
    • recommended local development database: SQLite
  • Gen3 mode:
    • request middleware fetches privileges from Fence/Arborist context
    • DB/API/service checks evaluate object authorizations against user privileges
    • method-aware checks use read/create/update/delete/file_upload

Developer Notes

Add a new query or operation
  1. Add method to core.DatabaseInterface in core/interface.go.
  2. Implement method in both:
    • sqlite/sqlite.go
    • postgres/postgres.go
  3. Update testutils/mocks.go if unit tests rely on the new method.
  4. Add tests in service or API package.
Local schema behavior
  • SQLite schema is initialized in sqlite.initSchema().
  • PostgreSQL object schema is initialized in postgres.NewPostgresDB().
  • Helm init jobs remain optional for deployments that want to pre-create schema outside the app.
  • SQLite helper scripts are provided in db/scripts/:
    • init_sqlite.sql
    • init_sqlite_db.sh
Resource path abstraction

Use helpers in core/resource_scope.go:

  • ResourcePathForScope(org, project)
  • ParseResourcePath(path)

This lets API/business layers work with organization/project while still storing Arborist-compatible paths.

Documentation

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

This section is empty.

Types

type BucketVisibilityLister added in v0.2.8 

type BucketVisibilityLister interface {
	ListBucketVisibilityRows(ctx context.Context, resources []string, includeUnscoped, restrictToResources bool) ([]models.BucketVisibilityRow, error)
}

type CredentialStore 

type CredentialStore interface {
	GetS3Credential(ctx context.Context, bucket string) (*models.S3Credential, error)
	ListS3Credentials(ctx context.Context) ([]models.S3Credential, error)
	SaveS3Credential(ctx context.Context, cred *models.S3Credential) error
	DeleteS3Credential(ctx context.Context, bucket string) error
	CreateBucketScope(ctx context.Context, scope *models.BucketScope) error
	DeleteBucketScope(ctx context.Context, organization, projectID, credentialID string) error
	GetBucketScope(ctx context.Context, organization, projectID string) (*models.BucketScope, error)
	ListBucketScopes(ctx context.Context) ([]models.BucketScope, error)
}

CredentialStore groups bucket credential and scope management.

type DatabaseInterface 

type DatabaseInterface interface {
	ServiceInfoStore
	ObjectStore
	CredentialStore
	PendingLFSMetaStore
	UsageStore
}

DatabaseInterface defines the full database backend contract.

func NewInMemoryDB 

func NewInMemoryDB() DatabaseInterface

NewInMemoryDB returns a new database interface backed by an in-memory SQLite database. This is used primarily for testing.

type FileUsageScopedLister added in v0.2.8 

type FileUsageScopedLister interface {
	ListFileUsagePageByScope(ctx context.Context, organization, project string, limit, offset int, inactiveSince *time.Time) ([]models.FileUsage, error)
	ListFileUsagePageByResources(ctx context.Context, resources []string, includeUnscoped bool, limit, offset int, inactiveSince *time.Time) ([]models.FileUsage, error)
	GetFileUsageSummaryByScope(ctx context.Context, organization, project string, inactiveSince *time.Time) (models.FileUsageSummary, error)
	GetFileUsageSummaryByResources(ctx context.Context, resources []string, includeUnscoped bool, inactiveSince *time.Time) (models.FileUsageSummary, error)
}

type LFSStore 

type LFSStore interface {
	ObjectStore
	CredentialStore
	PendingLFSMetaStore
	UsageStore
}

LFSStore is the minimum storage surface needed by the LFS API.

type MetricsStore 

type MetricsStore interface {
	ListObjectIDsByScope(ctx context.Context, organization, project string) ([]string, error)
	GetObject(ctx context.Context, id string) (*models.InternalObject, error)
	RecordTransferAttributionEvents(ctx context.Context, events []models.TransferAttributionEvent) error
	RecordProviderTransferEvents(ctx context.Context, events []models.ProviderTransferEvent) error
	ListS3Credentials(ctx context.Context) ([]models.S3Credential, error)
	GetTransferAttributionSummary(ctx context.Context, filter models.TransferAttributionFilter) (models.TransferAttributionSummary, error)
	GetTransferAttributionBreakdown(ctx context.Context, filter models.TransferAttributionFilter, groupBy string) ([]models.TransferAttributionBreakdown, error)
	GetFileUsage(ctx context.Context, objectID string) (*models.FileUsage, error)
	ListFileUsageByObjectIDs(ctx context.Context, ids []string) ([]models.FileUsage, error)
	ListFileUsage(ctx context.Context, limit, offset int, inactiveSince *time.Time) ([]models.FileUsage, error)
	GetFileUsageSummary(ctx context.Context, inactiveSince *time.Time) (models.FileUsageSummary, error)
}

MetricsStore is the minimum storage surface needed by the metrics API.

type ObjectAuthorizedLister added in v0.2.8 

type ObjectAuthorizedLister interface {
	ListObjectIDsByScopeAndResources(ctx context.Context, organization, project string, resources []string, restrictToResources bool) ([]string, error)
	ListObjectIDsByChecksumsAndResources(ctx context.Context, checksums []string, resources []string, includeUnscoped, restrictToResources bool) (map[string][]string, error)
}

type ObjectChecksumPageLister added in v0.2.8 

type ObjectChecksumPageLister interface {
	ListObjectIDsPageByChecksum(ctx context.Context, checksum, checksumType, organization, project, startAfter string, limit, offset int, resources []string, includeUnscoped, restrictToResources bool) ([]string, error)
}

type ObjectIDPageLister added in v0.2.8 

type ObjectIDPageLister interface {
	ListObjectIDsPageByScope(ctx context.Context, organization, project, startAfter string, limit, offset int) ([]string, error)
	ListObjectIDsPageByResources(ctx context.Context, resources []string, includeUnscoped bool, startAfter string, limit, offset int) ([]string, error)
}

type ObjectIDResourceLister added in v0.2.8 

type ObjectIDResourceLister interface {
	ListObjectIDsByResources(ctx context.Context, resources []string, includeUnscoped bool) ([]string, error)
}

type ObjectPathPageLister added in v0.3.1 

type ObjectPathPageLister interface {
	ListObjectIDsPageByPath(ctx context.Context, organization, project, path, startAfter string, limit, offset int) ([]string, []models.BrowseDirectory, error)
}

type ObjectStore 

type ObjectStore interface {
	GetObject(ctx context.Context, id string) (*models.InternalObject, error)
	DeleteObject(ctx context.Context, id string) error
	DeleteObjectAlias(ctx context.Context, aliasID string) error
	CreateObject(ctx context.Context, obj *models.InternalObject) error
	GetObjectsByChecksum(ctx context.Context, checksum string) ([]models.InternalObject, error)
	GetObjectsByChecksums(ctx context.Context, checksums []string) (map[string][]models.InternalObject, error)
	ListObjectIDsByScope(ctx context.Context, organization, project string) ([]string, error)
	CreateObjectAlias(ctx context.Context, aliasID, canonicalObjectID string) error
	ResolveObjectAlias(ctx context.Context, aliasID string) (string, error)
	GetBulkObjects(ctx context.Context, ids []string) ([]models.InternalObject, error)
	BulkDeleteObjects(ctx context.Context, ids []string) error
	RegisterObjects(ctx context.Context, objects []models.InternalObject) error
	UpdateObjectAccessMethods(ctx context.Context, objectID string, accessMethods []drs.AccessMethod) error
	BulkUpdateAccessMethods(ctx context.Context, updates map[string][]drs.AccessMethod) error
	RemoveObjectControlledAccess(ctx context.Context, objectID, resource string) error
}

ObjectStore groups the object lifecycle and lookup capabilities used by the API layers.

type ObjectURLPageLister added in v0.2.8 

type ObjectURLPageLister interface {
	ListObjectIDsPageByURL(ctx context.Context, objectURL, organization, project, startAfter string, limit, offset int, resources []string, includeUnscoped, restrictToResources bool) ([]string, error)
}

type ObjectsAPIServiceDatabase 

type ObjectsAPIServiceDatabase interface {
	ServiceInfoStore
	ObjectStore
	CredentialStore
	UsageStore
}

ObjectsAPIServiceDatabase is the storage surface used by the object service package.

type PendingLFSMetaStore 

type PendingLFSMetaStore interface {
	SavePendingLFSMeta(ctx context.Context, entries []models.PendingLFSMeta) error
	GetPendingLFSMeta(ctx context.Context, oid string) (*models.PendingLFSMeta, error)
	PopPendingLFSMeta(ctx context.Context, oid string) (*models.PendingLFSMeta, error)
}

PendingLFSMetaStore manages pending LFS metadata.

type SHA256ValidityStore 

type SHA256ValidityStore interface {
	GetObjectsByChecksums(ctx context.Context, checksums []string) (map[string][]models.InternalObject, error)
	ListS3Credentials(ctx context.Context) ([]models.S3Credential, error)
}

SHA256ValidityStore is the minimum storage surface needed by the SHA256 validity endpoint.

type ServiceInfoStore 

type ServiceInfoStore interface {
	GetServiceInfo(ctx context.Context) (*drs.Service, error)
}

ServiceInfoStore exposes service metadata reads.

type TransferAttributionScopedStore added in v0.2.8 

type TransferAttributionScopedStore interface {
	GetTransferAttributionSummaryByResources(ctx context.Context, filter models.TransferAttributionFilter, resources []string) (models.TransferAttributionSummary, error)
	GetTransferAttributionBreakdownByResources(ctx context.Context, filter models.TransferAttributionFilter, groupBy string, resources []string) ([]models.TransferAttributionBreakdown, error)
}

type UsageStore 

type UsageStore interface {
	RecordFileUpload(ctx context.Context, objectID string) error
	RecordFileDownload(ctx context.Context, objectID string) error
	RecordTransferAttributionEvents(ctx context.Context, events []models.TransferAttributionEvent) error
	RecordProviderTransferEvents(ctx context.Context, events []models.ProviderTransferEvent) error
	GetTransferAttributionSummary(ctx context.Context, filter models.TransferAttributionFilter) (models.TransferAttributionSummary, error)
	GetTransferAttributionBreakdown(ctx context.Context, filter models.TransferAttributionFilter, groupBy string) ([]models.TransferAttributionBreakdown, error)
	GetFileUsage(ctx context.Context, objectID string) (*models.FileUsage, error)
	ListFileUsageByObjectIDs(ctx context.Context, ids []string) ([]models.FileUsage, error)
	ListFileUsage(ctx context.Context, limit, offset int, inactiveSince *time.Time) ([]models.FileUsage, error)
	GetFileUsageSummary(ctx context.Context, inactiveSince *time.Time) (models.FileUsageSummary, error)
}

UsageStore manages file usage counters and summaries.

Source Files

View all Source files

Directories

Path Synopsis

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.