storage

package
v2.3.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Aug 28, 2026 License: Apache-2.0 Imports: 17 Imported by: 0

Documentation

Index

Constants

View Source
const (
	S3Provider       = storagetypes.S3Provider
	R2Provider       = storagetypes.R2Provider
	DiskProvider     = storagetypes.DiskProvider
	DatabaseProvider = storagetypes.DatabaseProvider
	// Presign mode constants
	PresignModeProvider = storagetypes.PresignModeProvider
	PresignModeProxy    = storagetypes.PresignModeProxy
)

Provider type constants so we can range, switch, etc

View Source
const (
	DefaultMaxFileSize   = 32 << 20 // 32MB
	DefaultMaxMemory     = 32 << 20 // 32MB
	DefaultUploadFileKey = "uploadFile"
)

Configuration constants

View Source
const DispositionAttachment = "attachment"

DispositionAttachment is the Content-Disposition value used to force the browser to download the file instead of rendering it.

View Source
const DispositionInline = "inline"

DispositionInline is the Content-Disposition value used when a file's MIME type is safe for the browser to render inline (PDFs, raster images).

View Source
const (
	// ErrNoFilesUploaded is returned when no files were uploaded in the request
	ErrNoFilesUploaded = errorMsg("objects: no uploadable files found in request")
)
View Source
const (
	// MIMEDetectionBufferSize defines the buffer size for MIME type detection
	MIMEDetectionBufferSize = 512
)

Variables

View Source
var (
	// ErrJSONParseFailed is returned when JSON parsing fails
	ErrJSONParseFailed = errors.New("failed to parse JSON")
	// ErrYAMLParseFailed is returned when YAML parsing fails
	ErrYAMLParseFailed = errors.New("failed to parse YAML")
	// ErrDOCXParseFailed is returned when DOCX parsing fails
	ErrDOCXParseFailed = errors.New("failed to parse DOCX")
	// ErrProviderResolutionRequired is returned when provider resolution is required but not available
	ErrProviderResolutionRequired = errors.New("provider resolution required - use external orchestration layer")
)
View Source
var (
	DefaultValidationFunc ValidationFunc = func(_ File) error {
		return nil
	}

	DefaultNameGeneratorFunc = func(originalName string) string {
		return originalName
	}

	DefaultSkipper = func(_ *http.Request) bool {
		return false
	}

	DefaultErrorResponseHandler = func(err error, statusCode int) http.HandlerFunc {
		return func(w http.ResponseWriter, _ *http.Request) {
			http.Error(w, err.Error(), statusCode)
		}
	}
)

Default function implementations

Functions

func DetectContentType

func DetectContentType(reader io.ReadSeeker) (string, error)

DetectContentType detects the MIME type of the provided reader using gabriel-vasile/mimetype library

func DispositionFor

func DispositionFor(contentType string) string

DispositionFor returns the Content-Disposition value that is appropriate for serving a file with the given content type. Files whose MIME type is in the inline-safe allowlist receive "inline" so they render in-place when requested from an <iframe> or <img>; everything else receives "attachment" so the browser triggers a download rather than guessing what to do with an unknown payload.

Parameters (e.g. "; charset=utf-8") and casing are normalized via mime.ParseMediaType. Malformed media types fall through to "attachment".

Types

type DownloadOptions

type DownloadOptions = storagetypes.DownloadFileOptions

Alias types from common/storagetypes to maintain clean imports having a bunch of smaller subpackages seemed to just complicate things

type DownloadedMetadata

type DownloadedMetadata = storagetypes.DownloadedFileMetadata

Alias types from common/storagetypes to maintain clean imports having a bunch of smaller subpackages seemed to just complicate things

type ErrResponseHandler

type ErrResponseHandler func(err error, statusCode int) http.HandlerFunc

ErrResponseHandler is a custom error handler for upload failures

type File

type File = storagetypes.File

Alias types from common/storagetypes to maintain clean imports having a bunch of smaller subpackages seemed to just complicate things

func NewUploadFile

func NewUploadFile(path string) (*File, error)

NewUploadFile creates a new File from a file path

type FileMetadata

type FileMetadata = storagetypes.FileMetadata

Alias types from common/storagetypes to maintain clean imports having a bunch of smaller subpackages seemed to just complicate things

type Files

type Files map[string][]File

Files is a map of file uploads organized by key

type Frontmatter

type Frontmatter struct {
	// OpenlaneID is the unique identifier for the document in Openlane platform, used for syncing
	OpenlaneID string `yaml:"openlane_id"`

	// Title of the document
	Title string `yaml:"title"`
	// Status of the document
	Status string `yaml:"status"`
	// Tags associated with the document
	Tags []string `yaml:"tags"`
	// Revision of the document
	Revision string `yaml:"revision"`
	// Satisfies lists the standards or requirements that this document satisfies
	Satisfies map[string][]string `yaml:"satisfies"`
}

Frontmatter represents the front matter metadata in a markdown file only Title is supported for now, but will be extended in the future

func ParseFrontmatter

func ParseFrontmatter(input []byte) (*Frontmatter, []byte, error)

ParseFrontmatter extracts YAML frontmatter and returns (metadata, content, error)

type NameGeneratorFunc

type NameGeneratorFunc func(originalName string) string

NameGeneratorFunc generates names for uploaded files

type ObjectService

type ObjectService struct {
	// contains filtered or unexported fields
}

ObjectService provides pure object management functionality without provider resolution

func NewObjectService

func NewObjectService() *ObjectService

NewObjectService creates a new object service instance with default configuration

func (*ObjectService) Delete

Delete deletes a file using a specific storage provider client

func (*ObjectService) Download

func (s *ObjectService) Download(ctx context.Context, provider Provider, file *storagetypes.File, opts *DownloadOptions) (*DownloadedMetadata, error)

Download downloads a file using a specific storage provider client

func (*ObjectService) ErrorResponseHandler

func (s *ObjectService) ErrorResponseHandler() ErrResponseHandler

ErrorResponseHandler returns the configured error response handler

func (*ObjectService) GetPresignedURL

func (s *ObjectService) GetPresignedURL(ctx context.Context, provider Provider, file *storagetypes.File, opts *storagetypes.PresignedURLOptions) (string, error)

GetPresignedURL gets a presigned URL for a file using a specific storage provider client

func (*ObjectService) IgnoreNonExistentKeys

func (s *ObjectService) IgnoreNonExistentKeys() bool

IgnoreNonExistentKeys returns whether to ignore non-existent form keys

func (*ObjectService) Keys

func (s *ObjectService) Keys() []string

Keys returns the configured form keys

func (*ObjectService) MaxMemory

func (s *ObjectService) MaxMemory() int64

MaxMemory returns the configured maximum memory for multipart forms

func (*ObjectService) MaxSize

func (s *ObjectService) MaxSize() int64

MaxSize returns the configured maximum file size

func (*ObjectService) Skipper

func (s *ObjectService) Skipper() SkipperFunc

Skipper returns the configured skipper function

func (*ObjectService) Upload

func (s *ObjectService) Upload(ctx context.Context, provider Provider, reader io.Reader, opts *UploadOptions) (*File, error)

Upload uploads a file using a specific storage provider client

func (*ObjectService) WithUploader

func (s *ObjectService) WithUploader(uploader UploaderFunc) *ObjectService

WithUploader returns a new ObjectService with the specified uploader function

func (*ObjectService) WithValidation

func (s *ObjectService) WithValidation(validationFunc ValidationFunc) *ObjectService

WithValidation returns a new ObjectService with the specified validation function

type ParentObject

type ParentObject = storagetypes.ParentObject

Alias types from common/storagetypes to maintain clean imports having a bunch of smaller subpackages seemed to just complicate things

type ParsedDocument

type ParsedDocument struct {
	// Frontmatter contains metadata extracted from the document, only for markdown files
	Frontmatter *Frontmatter
	// Data contains the parsed content of the document
	Data any
}

ParsedDocument represents a document parsed with its frontmatter and data

func ParseDocument

func ParseDocument(reader io.Reader, mimeType string) (*ParsedDocument, error)

ParseDocument parses a document based on its MIME type

type PresignMode

type PresignMode = storagetypes.PresignMode

Alias types from common/storagetypes to maintain clean imports having a bunch of smaller subpackages seemed to just complicate things

type Provider

type Provider = storagetypes.Provider

Alias types from common/storagetypes to maintain clean imports having a bunch of smaller subpackages seemed to just complicate things

type ProviderConfig

type ProviderConfig struct {
	// Enabled indicates if object storage is enabled
	Enabled bool `json:"enabled" koanf:"enabled" default:"true"`
	// Keys are the form field keys that will be processed for uploads
	Keys []string `json:"keys" koanf:"keys" default:"[uploadFile]"`
	// MaxSizeMB is the maximum file size allowed in MB
	MaxSizeMB int64 `json:"maxsizemb" koanf:"maxsizemb"`
	// MaxMemoryMB is the maximum memory to use for file uploads in MB
	MaxMemoryMB int64 `json:"maxmemorymb" koanf:"maxmemorymb"`
	// DevMode automatically configures a local disk storage provider (and ensures directories exist) and ignores other provider configs
	DevMode bool `json:"devmode" koanf:"devmode" default:"false"`
	// Providers contains configuration for each storage provider
	Providers Providers `json:"providers" koanf:"providers"`
}

ProviderConfig contains configuration for object storage providers

type ProviderConfigs

type ProviderConfigs struct {
	// Enabled indicates if this provider is enabled
	Enabled bool `json:"enabled" koanf:"enabled" default:"false"`
	// EnsureAvailable enforces provider availability before completing server startup
	EnsureAvailable bool `json:"ensureavailable" koanf:"ensureavailable" default:"false"`
	// Region for cloud providers
	Region string `json:"region" koanf:"region"`
	// Bucket name for cloud providers
	Bucket string `json:"bucket" koanf:"bucket"`
	// Endpoint for custom endpoints
	Endpoint string `json:"endpoint" koanf:"endpoint"`
	// ProxyPresignEnabled toggles proxy-signed download URL generation
	ProxyPresignEnabled bool `json:"proxypresignenabled" koanf:"proxypresignenabled" default:"false"`
	// BaseURL is the prefix for proxy download URLs (e.g., http://localhost:17608/v1/files).
	BaseURL string `json:"baseurl" koanf:"baseurl" default:"http://localhost:17608/v1/files"`
	// Credentials contains the credentials for accessing the provider
	Credentials ProviderCredentials `json:"credentials" koanf:"credentials"`
}

ProviderConfigs contains configuration for all storage providers This is structured to allow easy extension for additional providers in the future

type ProviderCredentials

type ProviderCredentials struct {
	// AccessKeyID for cloud providers
	AccessKeyID string `json:"accesskeyid" koanf:"accesskeyid" sensitive:"true"`
	// SecretAccessKey for cloud providers
	SecretAccessKey string `json:"secretaccesskey" koanf:"secretaccesskey" sensitive:"true"`
	// ProjectID for GCS
	ProjectID string `json:"projectid" koanf:"projectid" sensitive:"true"`
	// AccountID for Cloudflare R2
	AccountID string `json:"accountid" koanf:"accountid" sensitive:"true"`
	// APIToken for Cloudflare R2
	APIToken string `json:"apitoken" koanf:"apitoken" sensitive:"true"`
}

ProviderCredentials contains credentials for a storage provider

type ProviderHints

type ProviderHints = storagetypes.ProviderHints

Alias types from common/storagetypes to maintain clean imports having a bunch of smaller subpackages seemed to just complicate things

type ProviderOption

type ProviderOption func(*ProviderOptions)

ProviderOption configures runtime provider options

func WithBasePath

func WithBasePath(path string) ProviderOption

WithBasePath sets the local base path for disk providers

func WithBucket

func WithBucket(bucket string) ProviderOption

WithBucket sets the bucket/path value

func WithCredentials

func WithCredentials(creds ProviderCredentials) ProviderOption

WithCredentials sets provider credentials

func WithEndpoint

func WithEndpoint(endpoint string) ProviderOption

WithEndpoint sets the custom endpoint

func WithExtra

func WithExtra(key string, value any) ProviderOption

WithExtra attaches provider specific metadata

func WithLocalURL

func WithLocalURL(url string) ProviderOption

WithLocalURL sets the local URL used for presigned links

func WithProxyPresignConfig

func WithProxyPresignConfig(cfg *ProxyPresignConfig) ProviderOption

WithProxyPresignConfig sets proxy presign runtime dependencies.

func WithProxyPresignEnabled

func WithProxyPresignEnabled(enabled bool) ProviderOption

WithProxyPresignEnabled toggles proxy URL generation.

func WithRegion

func WithRegion(region string) ProviderOption

WithRegion sets the region value

type ProviderOptions

type ProviderOptions struct {
	Credentials         ProviderCredentials
	Bucket              string
	Region              string
	Endpoint            string
	BasePath            string
	LocalURL            string
	ProxyPresignEnabled bool
	ProxyPresignConfig  *ProxyPresignConfig
	// contains filtered or unexported fields
}

ProviderOptions captures runtime configuration shared across storage providers

func NewProviderOptions

func NewProviderOptions(opts ...ProviderOption) *ProviderOptions

NewProviderOptions constructs ProviderOptions applying the supplied options

func (*ProviderOptions) Apply

func (p *ProviderOptions) Apply(opts ...ProviderOption)

Apply applies option functions to ProviderOptions

func (*ProviderOptions) Clone

func (p *ProviderOptions) Clone() *ProviderOptions

Clone returns a deep copy of ProviderOptions

func (*ProviderOptions) Extra

func (p *ProviderOptions) Extra(key string) (any, bool)

Extra returns provider specific metadata

type ProviderType

type ProviderType = storagetypes.ProviderType

Alias types from common/storagetypes to maintain clean imports having a bunch of smaller subpackages seemed to just complicate things

type Providers

type Providers struct {
	// S3 provider configuration
	S3 ProviderConfigs `json:"s3" koanf:"s3"`
	// R2 provider configuration
	R2 ProviderConfigs `json:"r2" koanf:"r2"`
	// Disk provider configuration
	Disk ProviderConfigs `json:"disk" koanf:"disk"`
	// Database provider configuration
	Database ProviderConfigs `json:"database" koanf:"database"`
}

type ProxyPresignConfig

type ProxyPresignConfig struct {
	TokenManager  *tokens.TokenManager
	TokenIssuer   string
	TokenAudience string
	BaseURL       string
}

ProxyPresignConfig carries runtime dependencies for proxy download URL generation.

func ApplyProxyPresignOptions

func ApplyProxyPresignOptions(cfg *ProxyPresignConfig, opts ...ProxyPresignOption) *ProxyPresignConfig

ApplyProxyPresignOptions applies options to the provided config, allocating one if needed.

func NewProxyPresignConfig

func NewProxyPresignConfig(opts ...ProxyPresignOption) *ProxyPresignConfig

NewProxyPresignConfig builds a ProxyPresignConfig applying the supplied options.

func (*ProxyPresignConfig) Apply

Apply applies the supplied options to the existing ProxyPresignConfig.

type ProxyPresignOption

type ProxyPresignOption func(*ProxyPresignConfig)

ProxyPresignOption configures a ProxyPresignConfig.

func WithProxyPresignBaseURL

func WithProxyPresignBaseURL(baseURL string) ProxyPresignOption

WithProxyPresignBaseURL sets the base URL for generated download links.

func WithProxyPresignTokenAudience

func WithProxyPresignTokenAudience(audience string) ProxyPresignOption

WithProxyPresignTokenAudience sets the token audience when provided.

func WithProxyPresignTokenIssuer

func WithProxyPresignTokenIssuer(issuer string) ProxyPresignOption

WithProxyPresignTokenIssuer sets the token issuer when provided.

func WithProxyPresignTokenManager

func WithProxyPresignTokenManager(tm *tokens.TokenManager) ProxyPresignOption

WithProxyPresignTokenManager sets the token manager when provided.

type SkipperFunc

type SkipperFunc func(r *http.Request) bool

SkipperFunc defines a function to skip middleware processing

type UploadOptions

type UploadOptions = storagetypes.UploadFileOptions

Alias types from common/storagetypes to maintain clean imports having a bunch of smaller subpackages seemed to just complicate things

type UploadedMetadata

type UploadedMetadata = storagetypes.UploadedFileMetadata

Alias types from common/storagetypes to maintain clean imports having a bunch of smaller subpackages seemed to just complicate things

type UploaderFunc

type UploaderFunc func(ctx context.Context, service *ObjectService, files []File) ([]File, error)

UploaderFunc handles the file upload process and returns uploaded files

type ValidationFunc

type ValidationFunc func(f File) error

ValidationFunc is a type that can be used to dynamically validate a file

Directories

Path Synopsis
providers
disk
Package disk is the local disk storage provider for objects service
Package disk is the local disk storage provider for objects service
r2
Package r2 is the Cloudflare R2 storage provider for objects service
Package r2 is the Cloudflare R2 storage provider for objects service
s3
Package s3 is the AWS S3 storage provider for objects service
Package s3 is the AWS S3 storage provider for objects service
Package proxy implements a storage proxy that provides presigned URL generation
Package proxy implements a storage proxy that provides presigned URL generation

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL