storage

package
v0.3.7 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Aug 4, 2026 License: MIT Imports: 12 Imported by: 0

README

Storage

Storage 用 StorageKey{provider}://{identifier})定位文件。Provider 只负责存储能力, 不知道 HTTP API 的挂载路径。

文件访问

GET /storage/data/:provider/*fileid 始终由 Scene 代理传输文件。

GET /storage/url/:provider/*fileid 用于获取可访问 URL:

  • 不传 modemode=proxy:返回对应的 /storage/data/... 路径。返回值会保留 Gin 容器的实际挂载前缀,例如 /api/storage/url/... 会得到 /api/storage/data/...
  • mode=direct:调用 IStorageService.GetDirectURL,由 provider 返回直连 URL。 不支持直连的 provider 返回 ErrDirectURLUnsupported;生成失败返回 ErrGetDirectURLFailed

Local provider 不支持 direct URL。S3 provider 的 direct URL 是有时效的预签名 URL, 有效期由 storage.s3.presigned_url_ttl_seconds 配置。

S3 上传遇到不可 Seek 的 io.Reader 时,会先写入临时文件,以便 AWS SDK 计算签名和 重试。storage.s3.temp_dir 可指定临时文件目录;为空时使用操作系统临时目录。指定的 目录必须已存在且可写,Scene 不会自动创建。

Provider 不需要配置 URL prefix;HTTP 路径只由 delivery 层生成。

访问权限

当前 FileMeta 没有 Public/Private 或 ACL 字段,因此所有文件都按 Private 处理; storage 暂不提供匿名公开文件。

/data 的实际代理下载和 /url 的 URL 获取都要求 storage:file:downloadPermFileDownload)权限:

  • mode=proxy 返回的 /data 路径只表示文件位置,不代表授权;客户端请求该路径时会 再次鉴权。
  • mode=direct 只在鉴权通过后生成预签名 URL。预签名 URL 在有效期内是 bearer credential,后续由 provider 直接校验签名,不再经过 Scene 权限中间件。
Proxy 传输

/data 使用请求级 io.ReadSeekCloser 适配 http.ServeContent。第一次 Read 才调用 provider 的 ranged Load;后续顺序读取复用同一个 reader,并在请求结束时统一关闭。 Seek 只更新逻辑位置,下一次 Read 才按新位置重新打开 reader。

因此普通 GET 和单 Range 请求各只会 open/close 一次;HEAD 不会打开数据流。只有客户端 显式请求多个 Range 时,才会按 Range 分别打开 reader。delivery 会在调用 ServeContent 前设置元数据中的 Content-Type(为空时使用 application/octet-stream),避免 MIME sniff 触发一次额外的读取和回退。

Context

Service、provider、metadata repository 和 upload session tracker 的 I/O 方法都以 context.Context 为第一个参数。HTTP 调用使用 Request.Context(),并一直传递到 GORM、S3 和 Redis。Provider 和 repository 可以返回标准 context error;service 边界会将其映射为当前操作对应的 storage errcode,保持 service 只返回 errcode 的约定。

Context 只表达取消和超时,不负责并发一致性,也不替代锁。Local filesystem 的 syscall 本身不接收 context,因此只能在操作开始前、流式复制的分块之间和提交点前观察取消, 无法中断一个已经阻塞的 filesystem syscall。

远端写入、删除或 multipart 完成已经成功后,元数据和 session 的必要收尾会使用保留 request value、脱离 request cancellation 且最多持续 5 秒的 context。正常前向 I/O 始终服从请求取消。

todo

  • local
  • Aliyun oss
  • Tencent cos
  • Qiniu
  • aws s3 (S3-compatible, e.g. RustFS)

Documentation

Index

Constants

View Source
const Lens scene.ModuleName = "storage"

Variables

View Source
var (
	ErrStorageFailed             = _eg.CreateError(1, "storage failed")
	ErrFileNotFound              = _eg.CreateError(2, "file not found")
	ErrStorageNotFound           = _eg.CreateError(3, "storage not found")
	ErrFailToLoad                = _eg.CreateError(4, "fail to load")
	ErrFailToStore               = _eg.CreateError(5, "fail to store")
	ErrFailToDelete              = _eg.CreateError(6, "fail to delete")
	ErrLoadingMeta               = _eg.CreateError(7, "loading meta")
	ErrInvalidStorageKey         = _eg.CreateError(8, "invalid storage key")
	ErrStorageError              = _eg.CreateError(9, "storage error")
	ErrInvalidOffset             = _eg.CreateError(10, "invalid offset")
	ErrInvalidLength             = _eg.CreateError(11, "invalid length")
	ErrUnknownError              = _eg.CreateError(12, "unknown error")
	ErrInitPartUploadFailed      = _eg.CreateError(13, "init part upload failed")
	ErrUploadSessionNotFound     = _eg.CreateError(14, "upload session not found")
	ErrStorePartFailed           = _eg.CreateError(15, "store part upload failed")
	ErrFailToAbortMultipartStore = _eg.CreateError(16, "fail to abort")
	ErrMetaNotFound              = _eg.CreateError(17, "meta not found")
	ErrFailToListMeta            = _eg.CreateError(18, "fail to list meta")
	ErrStorageKeyExists          = _eg.CreateError(19, "storage key exists")
	ErrDirectURLUnsupported      = _eg.CreateError(20, "direct URL unsupported")
	ErrGetDirectURLFailed        = _eg.CreateError(21, "fail to get direct URL")
)
View Source
var (
	PermFileUpload   = permission.Create("storage:file:upload")
	PermFileDelete   = permission.Create("storage:file:delete")
	PermFileDownload = permission.Create("storage:file:download")
	PermFileList     = permission.Create("storage:file:list")
)

Functions

func IsStorageKey added in v0.3.7

func IsStorageKey(storageKey string) bool

func NormalizeIdentifier added in v0.3.7

func NormalizeIdentifier(identifier string) string

func ValidateStorageKey added in v0.3.7

func ValidateStorageKey(storageKey StorageKey) error

Types

type FileMeta

type FileMeta struct {
	StorageKey       StorageKey `json:"storage_key"`
	Provider         string     `json:"provider"`
	Identifier       string     `json:"identifier"`
	OriginalFilename string     `json:"original_filename"`
	ContentType      string     `json:"content_type"`
	ContentLength    int64      `json:"content_length"`
	Md5Checksum      string     `json:"md5_checksum"`
	Finished         bool       `json:"finished"`
	CreatedAt        time.Time  `json:"created_at"`
	UpdatedAt        time.Time  `json:"updated_at"`
}

func OpenContent added in v0.3.7

func OpenContent(ctx context.Context, srv IStorageService, storageKey StorageKey) (io.ReadSeekCloser, FileMeta, error)

OpenContent opens storage content as a request-scoped seekable stream. Sequential reads share one provider reader. Seeking closes that reader and opens a new ranged reader lazily on the next Read call.

func (*FileMeta) FillMissing

func (f *FileMeta) FillMissing() FileMeta

type IFileMetaRepository

type IFileMetaRepository interface {
	scene.Named
	// Store will store the metadata in the repository
	// will overwrite the old metadata if exists
	Store(ctx context.Context, meta FileMeta) error
	Load(ctx context.Context, storageKey StorageKey) (meta FileMeta, err error)
	Delete(ctx context.Context, storageKey StorageKey) error
	List(ctx context.Context, provider string, offset, limit int64) (model.PaginationResult[FileMeta], error)
}

type IStorageProvider

type IStorageProvider interface {
	scene.Named
	// ProviderName return provider name in storageKey
	ProviderName() string
	// HealthCheck will check if this Storage provider is accessible
	HealthCheck(ctx context.Context) error
	// Meta will return any possible metadata can be found, as a fallback option if IFileMetaRepository failed
	Meta(ctx context.Context, storageKey StorageKey) (meta FileMeta, err error)
	Load(ctx context.Context, storageKey StorageKey, offset, length int64) (reader io.ReadCloser, err error)
	LoadAll(ctx context.Context, storageKey StorageKey) (reader io.ReadCloser, err error)
	// Store will store the data in the storage at path and return the storageKey,
	// if path not exists, it will create the path.
	Store(ctx context.Context, storageKey StorageKey, data io.Reader) (err error)
	Delete(ctx context.Context, storageKey StorageKey) error
	// Multipart related
	InitMultipartStore(ctx context.Context, storageKey StorageKey) (uploadId string, err error)
	StoreMultipart(ctx context.Context, uploadId string, partNumber int, data io.Reader) error
	CompleteMultipart(ctx context.Context, uploadId string) error
	AbortMultipart(ctx context.Context, uploadId string) error
	// GetDirectURL returns a URL that accesses the provider directly.
	// Providers that cannot expose direct URLs return ErrDirectURLUnsupported.
	GetDirectURL(ctx context.Context, storageKey StorageKey) (url string, err error)
}

type IStorageService

type IStorageService interface {
	scene.Service
	ListProviders() []string
	// ListMeta will list meta from specific provider.
	ListMeta(ctx context.Context, provider string, offset, limit int64) (model.PaginationResult[FileMeta], error)
	// Meta return the meta given a storageKey
	Meta(ctx context.Context, storageKey StorageKey) (meta FileMeta, err error)
	// Load will load file stream at offset with length.
	// Caller must close the returned reader.
	Load(ctx context.Context, storageKey StorageKey, offset, length int64) (reader io.ReadCloser, err error)
	// LoadAll will load full file stream.
	// Caller must close the returned reader.
	LoadAll(ctx context.Context, storageKey StorageKey) (reader io.ReadCloser, err error)
	Delete(ctx context.Context, storageKey StorageKey) error
	// Store will store data at default provider
	// it calls StoreAt internally
	// Store consumes data from reader until EOF.
	Store(ctx context.Context, data io.Reader, meta FileMeta) (storageKey StorageKey, err error)
	// StoreAt will store data using the given provider and identifier.
	// If provider is empty, the service default provider will be used.
	// If identifier is empty, the service will generate one internally.
	StoreAt(ctx context.Context, provider, identifier string, data io.Reader, meta FileMeta) (storageKey StorageKey, err error)
	// Multipart related
	// InitMultipartStore will initialize a multipart upload using the given provider and identifier.
	// If provider is empty, the service default provider will be used.
	// If identifier is empty, the service will generate one internally.
	// It returns both the resolved storage key and the upload id.
	InitMultipartStore(ctx context.Context, provider, identifier string, meta FileMeta) (storageKey StorageKey, uploadId string, err error)
	StoreMultipart(ctx context.Context, uploadId string, partNumber int, data io.Reader) error
	// CompleteMultipart completes an upload and returns its finalized metadata.
	CompleteMultipart(ctx context.Context, uploadId string) (FileMeta, error)
	AbortMultipart(ctx context.Context, uploadId string) error
	// GetDirectURL returns a URL that accesses the storage provider directly.
	GetDirectURL(ctx context.Context, storageKey StorageKey) (url string, err error)
}

IStorageService is the storage module's service boundary. Every non-nil error returned directly by a service method must be a storage errcode; provider and repository errors, including context errors, are mapped at this boundary.

type IUploadSessionTracker

type IUploadSessionTracker interface {
	scene.Named
	Save(ctx context.Context, uploadId string, session UploadSession) error
	Get(ctx context.Context, uploadId string) (UploadSession, error)
	Delete(ctx context.Context, uploadId string) error
}

type StorageKey added in v0.3.7

type StorageKey string

StorageKey is the unique identifier of a file in storage. it is composed with {Provider}://{ID} example: tos.buketName://objectName example: local.name://objectName

func NewStorageKey added in v0.3.7

func NewStorageKey(provider string, path ...string) StorageKey

func NewStorageKeyWithUUID added in v0.3.7

func NewStorageKeyWithUUID(provider string) StorageKey

func ParseStorageKey added in v0.3.7

func ParseStorageKey(storageKey string) (StorageKey, bool)

func (StorageKey) FileID added in v0.3.7

func (f StorageKey) FileID() string

func (StorageKey) Provider added in v0.3.7

func (f StorageKey) Provider() string

type UploadSession

type UploadSession struct {
	StorageKey StorageKey `json:"storage_key"`
	Created    time.Time  `json:"created"`
}

UploadSession contains info to resume/complete uploads

Directories

Path Synopsis
repository

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL