README
¶
Storage Location
The default storage driver url is file:///tmp/mediorum/blobs.
This is the directory on your node where files will stored post transcoding.
To update the local storage location set the AUDIUS_STORAGE_DRIVER_URL environment variable
AUDIUS_STORAGE_DRIVER_URL="file:///my/directory"
The driver is determined based on the storage_url (eg s3://your-bucket uses the s3 driver)
💡 Note: URLs beginning with https:// will default to s3
Cloud Storage
To use cloud object storage instead of local storage, you will need to:
- create either an s3 bucket, gcs bucket or azure storage container
- your bucket access policy should NOT BE PUBLIC
- provision credentials that allow object storage access
AWS_ACCESS_KEY_ID="<my-access-key-id>"
AWS_SECRET_ACCESS_KEY="<my-secret-access-key>"
AWS_REGION="<my-aws-region>"
AUDIUS_STORAGE_DRIVER_URL="s3://my-bucket?region=us-west-1"
GOOGLE_APPLICATION_CREDENTIALS="/path/to/creds.json"
AUDIUS_STORAGE_DRIVER_URL="gs://my-bucket"
AZURE_STORAGE_ACCOUNT="<storage-account-name>"
AZURE_STORAGE_KEY="<storage-key>"
AUDIUS_STORAGE_DRIVER_URL="azblob://my-container"
Documentation
¶
Overview ¶
Package persistence handles taking content from the temp store and storing it in the persistence store.
Index ¶
- Constants
- func FileDirFromDSN(blobDriverURL string) (string, bool)
- func MoveAllFiles(from, to *blob.Bucket) error
- func Open(blobDriverURL string) (*blob.Bucket, error)
- func SweepStaleTempFiles(ctx context.Context, blobDriverURL string, minAge time.Duration, dryRun bool, ...) (int, int, error)
- type Prefix
Constants ¶
const ( AWS_ACCESS_KEY_ID = "AWS_ACCESS_KEY_ID" AWS_SECRET_ACCESS_KEY = "AWS_SECRET_ACCESS_KEY" AWS_REGION = "AWS_REGION" GOOGLE_APPLICATION_CREDENTIALS = "GOOGLE_APPLICATION_CREDENTIALS" AZURE_STORAGE_ACCOUNT = "AZURE_STORAGE_ACCOUNT" AZURE_STORAGE_KEY = "AZURE_STORAGE_KEY" )
const DefaultStaleTempFileAge = time.Hour
DefaultStaleTempFileAge is how old a fileblob ".tmp" file must be before the sweeper will delete it.
fileblob writes to "<path>.<nanos>.tmp" and renames on a successful Close, so a surviving ".tmp" is an interrupted write — but a write *in progress* looks identical by name. mtime advances as bytes are written, and outbound peer requests are capped at 3 minutes by peerHTTPClient, so an hour is a wide margin against deleting a live write. The cost of guessing too high is only that an orphan survives until the next sweep.
Variables ¶
This section is empty.
Functions ¶
func FileDirFromDSN ¶ added in v1.9.0
FileDirFromDSN returns the filesystem directory backing a file:// blob DSN, stripping any query parameters (e.g. "?no_tmp_dir=true"). It reports false for cloud backends, which have no local tree to sweep.
func MoveAllFiles ¶
func Open ¶
New creates a struct that listens to streamToStoreFrom and downloads content from the temp store to the persistent store.
func SweepStaleTempFiles ¶ added in v1.9.0
func SweepStaleTempFiles(ctx context.Context, blobDriverURL string, minAge time.Duration, dryRun bool, onProgress func(scanned, matched int)) (int, int, error)
SweepStaleTempFiles removes orphaned ".tmp" files left behind by the fileblob driver when a write is interrupted before Close renames it into place. See https://github.com/google/go-cloud/issues/3286 for the original issue and https://github.com/google/go-cloud/issues/3294 for why it isn't truly fixed.
Only files last modified more than minAge ago are removed, so the sweep is safe to run concurrently with live writes. Cloud backends are a no-op.
With dryRun set nothing is deleted and the return value is the number of files that *would* have been -- the prune job defaults to that, so an operator can see the cost before authorising it.
Returns a count that is meaningful even alongside an error, since the walk continues past unreadable entries.
Source Files