store

package
v0.1.14 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Nov 17, 2025 License: Apache-2.0 Imports: 22 Imported by: 68

Documentation

Overview

Copyright (c) 2016-2019 Uber Technologies, Inc.

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at 

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Copyright (c) 2016-2019 Uber Technologies, Inc.

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at 

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Copyright (c) 2016-2019 Uber Technologies, Inc.

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at 

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Copyright (c) 2016-2019 Uber Technologies, Inc.

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at 

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Copyright (c) 2016-2019 Uber Technologies, Inc.

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at 

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Copyright (c) 2016-2019 Uber Technologies, Inc.

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at 

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Copyright (c) 2016-2019 Uber Technologies, Inc.

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at 

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Copyright (c) 2016-2019 Uber Technologies, Inc.

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at 

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Copyright (c) 2016-2019 Uber Technologies, Inc.

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at 

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Copyright (c) 2016-2019 Uber Technologies, Inc.

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at 

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Copyright (c) 2016-2019 Uber Technologies, Inc.

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at 

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func RunDownload 

func RunDownload(cads *CADownloadStore, d core.Digest, content []byte) error

RunDownload downloads content to cads.

Types

type CADownloadStore 

type CADownloadStore struct {
	// contains filtered or unexported fields
}

CADownloadStore allows simultaneously downloading and uploading content-adddressable files.

func CADownloadStoreFixture 

func CADownloadStoreFixture() (*CADownloadStore, func())

CADownloadStoreFixture returns a CADownloadStore for testing purposes.

func NewCADownloadStore 

func NewCADownloadStore(config CADownloadStoreConfig, stats tally.Scope) (*CADownloadStore, error)

NewCADownloadStore creates a new CADownloadStore.

func (*CADownloadStore) Any 

func (s *CADownloadStore) Any() *CADownloadStoreScope

Any scopes the store to files in any state.

func (*CADownloadStore) Cache 

func (s *CADownloadStore) Cache() *CADownloadStoreScope

Cache scopes the store to files in the cache state.

func (*CADownloadStore) Close 

func (s *CADownloadStore) Close()

Close terminates all goroutines started by s.

func (*CADownloadStore) CreateDownloadFile 

func (s *CADownloadStore) CreateDownloadFile(name string, length int64) error

CreateDownloadFile creates an empty download file initialized with length.

func (*CADownloadStore) Download 

func (s *CADownloadStore) Download() *CADownloadStoreScope

Download scopes the store to files in the download state.

func (*CADownloadStore) GetCacheFileReader 

func (s *CADownloadStore) GetCacheFileReader(name string) (FileReader, error)

GetCacheFileReader gets a cache file reader. Implemented for compatibility with other stores.

func (*CADownloadStore) GetCacheFileStat 

func (s *CADownloadStore) GetCacheFileStat(name string) (os.FileInfo, error)

GetCacheFileStat stats a cache file. Implemented for compatibility with other stores.

func (*CADownloadStore) GetDownloadFileReadWriter 

func (s *CADownloadStore) GetDownloadFileReadWriter(name string) (FileReadWriter, error)

GetDownloadFileReadWriter returns a FileReadWriter for name.

func (*CADownloadStore) InCacheError 

func (s *CADownloadStore) InCacheError(err error) bool

InCacheError returns true for errors originating from file store operations which do not accept files in cache state.

func (*CADownloadStore) InDownloadError 

func (s *CADownloadStore) InDownloadError(err error) bool

InDownloadError returns true for errors originating from file store operations which do not accept files in download state.

func (*CADownloadStore) MoveDownloadFileToCache 

func (s *CADownloadStore) MoveDownloadFileToCache(name string) error

MoveDownloadFileToCache moves a download file to the cache.

type CADownloadStoreConfig 

type CADownloadStoreConfig struct {
	DownloadDir     string        `yaml:"download_dir"`
	CacheDir        string        `yaml:"cache_dir"`
	DownloadCleanup CleanupConfig `yaml:"download_cleanup"`
	CacheCleanup    CleanupConfig `yaml:"cache_cleanup"`
	// Part size limit for each file read. 0 means no limit.
	ReadPartSize int `yaml:"read_part_size"`
	// Part size limit for each file write. 0 means no limit.
	WritePartSize int `yaml:"write_part_size"`
}

CADownloadStoreConfig defines CADownloadStore configuration. TODO(evelynl94): rename

type CADownloadStoreScope 

type CADownloadStoreScope struct {
	// contains filtered or unexported fields
}

CADownloadStoreScope scopes what states an operation may be accepted within. Should only be used for read / write operations which are acceptable in any state.

func (*CADownloadStoreScope) DeleteFile 

func (a *CADownloadStoreScope) DeleteFile(name string) error

DeleteFile deletes name.

func (*CADownloadStoreScope) GetFileReader 

func (a *CADownloadStoreScope) GetFileReader(name string) (FileReader, error)

GetFileReader returns a reader for name.

func (*CADownloadStoreScope) GetFileStat 

func (a *CADownloadStoreScope) GetFileStat(name string) (os.FileInfo, error)

GetFileStat returns file info for name.

func (*CADownloadStoreScope) GetMetadata 

func (a *CADownloadStoreScope) GetMetadata(name string, md metadata.Metadata) error

GetMetadata returns the metadata content of md for name.

func (*CADownloadStoreScope) GetOrSetMetadata 

func (a *CADownloadStoreScope) GetOrSetMetadata(name string, md metadata.Metadata) error

GetOrSetMetadata returns the metadata content of md for name, or initializes the metadata content to b if not set.

func (*CADownloadStoreScope) SetMetadata 

func (a *CADownloadStoreScope) SetMetadata(
	name string, md metadata.Metadata) (updated bool, err error)

SetMetadata writes b to metadata content of md for name.

func (*CADownloadStoreScope) SetMetadataAt 

func (a *CADownloadStoreScope) SetMetadataAt(
	name string, md metadata.Metadata, b []byte, offset int64) (updated bool, err error)

SetMetadataAt writes b to metadata content of md starting at index i for name.

type CAStore 

type CAStore struct {
	// contains filtered or unexported fields
}

CAStore allows uploading / caching content-addressable files.

func CAStoreFixture 

func CAStoreFixture() (*CAStore, func())

CAStoreFixture returns a CAStore for testing purposes.

func CAStoreFixtureWithClock added in v0.1.13 

func CAStoreFixtureWithClock(config CAStoreConfig, clk clock.Clock) (*CAStore, func())

CAStoreFixtureWithClock returns a CAStore with a custom clock for testing purposes. This is useful for tests that need to control time, such as preventing automatic drain operations in memory cache tests.

func NewCAStore 

func NewCAStore(config CAStoreConfig, stats tally.Scope) (*CAStore, error)

NewCAStore creates a new CAStore.

func (*CAStore) CheckInMemCache added in v0.1.13 

func (s *CAStore) CheckInMemCache(name string) bool

CheckInMemCache returns true if the blob is present in memcache Used in tests

func (*CAStore) Close 

func (s *CAStore) Close()

Close terminates any goroutines started by s.

func (*CAStore) CreateCacheFile 

func (s *CAStore) CreateCacheFile(name string, r io.Reader) error

CreateCacheFile initializes a cache file for name from r. name should be a raw hex sha256 digest, and the contents of r must hash to name.

func (CAStore) CreateUploadFile 

func (s CAStore) CreateUploadFile(name string, length int64) error

func (CAStore) DeleteCacheFile 

func (s CAStore) DeleteCacheFile(name string) error

func (CAStore) DeleteCacheFileMetadata 

func (s CAStore) DeleteCacheFileMetadata(name string, md metadata.Metadata) error

func (CAStore) DeleteUploadFile 

func (s CAStore) DeleteUploadFile(name string) error

func (*CAStore) GetCacheFileMetadata 

func (s *CAStore) GetCacheFileMetadata(name string, md metadata.Metadata) error

GetCacheFileMetadata overrides cacheStore.GetCacheFileMetadata to serve TorrentMeta from memory cache when available.

func (*CAStore) GetCacheFileReader 

func (s *CAStore) GetCacheFileReader(name string) (FileReader, error)

GetCacheFileReader overrides cacheStore.GetCacheFileReader to check memory cache first before reading from disk.

func (*CAStore) GetCacheFileStat 

func (s *CAStore) GetCacheFileStat(name string) (os.FileInfo, error)

GetCacheFileStat overrides cacheStore.GetCacheFileStat to check memory cache first.

func (CAStore) GetOrSetCacheFileMetadata 

func (s CAStore) GetOrSetCacheFileMetadata(name string, md metadata.Metadata) error

func (CAStore) GetUploadFileMetadata 

func (s CAStore) GetUploadFileMetadata(name string, md metadata.Metadata) error

func (CAStore) GetUploadFileReadWriter 

func (s CAStore) GetUploadFileReadWriter(name string) (FileReadWriter, error)

func (CAStore) GetUploadFileReader 

func (s CAStore) GetUploadFileReader(name string) (FileReader, error)

func (CAStore) GetUploadFileStat 

func (s CAStore) GetUploadFileStat(name string) (os.FileInfo, error)

func (*CAStore) ListCacheFiles 

func (s *CAStore) ListCacheFiles() ([]string, error)

ListCacheFiles overrides cacheStore.ListCacheFiles to include memory cache entries.

func (*CAStore) MoveUploadFileToCache 

func (s *CAStore) MoveUploadFileToCache(uploadName, cacheName string) error

MoveUploadFileToCache commits uploadName as cacheName. Clients are expected to validate the content of the upload file matches the cacheName digest.

func (CAStore) RangeUploadMetadata 

func (s CAStore) RangeUploadMetadata(name string, f func(metadata.Metadata) error) error

func (CAStore) SetCacheFileMetadata 

func (s CAStore) SetCacheFileMetadata(name string, md metadata.Metadata) (bool, error)

func (CAStore) SetUploadFileMetadata 

func (s CAStore) SetUploadFileMetadata(name string, md metadata.Metadata) error

func (*CAStore) WriteBlobToCacheWithMetaInfo added in v0.1.13 

func (s *CAStore) WriteBlobToCacheWithMetaInfo(
	name string,
	size uint64,
	write func(w FileReadWriter) error,
	pieceLength int64) error

WriteBlobToCacheWithMetaInfo writes a blob and its metadata to disk, potentially going through a write-through memory cache, if memory is available.

func (*CAStore) WriteCacheFile 

func (s *CAStore) WriteCacheFile(name string, write func(w FileReadWriter) error) error

WriteCacheFile initializes a cache file for name by passing a temporary upload file writer to the write function.

type CAStoreConfig 

type CAStoreConfig struct {
	UploadDir     string        `yaml:"upload_dir"`
	CacheDir      string        `yaml:"cache_dir"`
	Volumes       []Volume      `yaml:"volumes"`
	Capacity      int           `yaml:"capacity"`
	UploadCleanup CleanupConfig `yaml:"upload_cleanup"`
	CacheCleanup  CleanupConfig `yaml:"cache_cleanup"`
	// Part size limit for each file read. 0 means no limit.
	ReadPartSize int `yaml:"read_part_size"`
	// Part size limit for each file write. 0 means no limit.
	WritePartSize int `yaml:"write_part_size"`

	SkipHashVerification bool `yaml:"skip_hash_verification"`

	MemoryCache MemoryCacheConfig `yaml:"memory_cache"`
}

CAStoreConfig defines CAStore configuration.

func CAStoreConfigFixture 

func CAStoreConfigFixture() (CAStoreConfig, func())

CAStoreConfigFixture returns config for CAStore for testing purposes.

type CleanupConfig 

type CleanupConfig struct {
	Disabled                 bool          `yaml:"disabled"`
	Interval                 time.Duration `yaml:"interval"`                   // How often cleanup runs.
	TTI                      time.Duration `yaml:"tti"`                        // Time to idle based on last access time.
	TTL                      time.Duration `yaml:"ttl"`                        // Time to live regardless of access. If 0, disables TTL.
	AggressiveThreshold      int           `yaml:"aggressive_threshold"`       // The disk util threshold to trigger aggressive cleanup. If 0, disables aggressive cleanup.
	AggressiveTTL            time.Duration `yaml:"aggressive_ttL"`             // Time to live regardless of access if aggressive cleanup is triggered.
	AggressiveLowerThreshold int           `yaml:"aggressive_lower_threshold"` // The lower disk util threshold in percent, below which aggressive cleanup will stop. If 0, no lower threshold.
}

CleanupConfig defines configuration for periodically cleaning up idle files.

type FileReadWriter 

type FileReadWriter = base.FileReadWriter

FileReadWriter is a readable, writable file.

type FileReader 

type FileReader = base.FileReader

FileReader is a read-only file.

func NewBufferFileReader 

func NewBufferFileReader(b []byte) FileReader

NewBufferFileReader returns an in-memory FileReader backed by b.

type MemoryCacheConfig added in v0.1.13 

type MemoryCacheConfig struct {
	Enabled         bool          `yaml:"enabled"`
	MaxSize         uint64        `yaml:"max_size"`
	DrainWorkers    int           `yaml:"drain_workers"`
	DrainMaxRetries int           `yaml:"drain_max_retries"`
	TTL             time.Duration `yaml:"ttl"`
	TTLInterval     time.Duration `yaml:"ttl_interval"`
}

MemoryCacheConfig defines memory cache configuration.

type MockFileReadWriter 

type MockFileReadWriter struct {
	*os.File
	Committed bool
}

MockFileReadWriter is a mock FileReadWriter that is backed by a physical file. This is preferred to a gomock struct because read/write operations are greatly simplified.

func NewMockFileReadWriter 

func NewMockFileReadWriter(content []byte) (*MockFileReadWriter, func())

NewMockFileReadWriter returns a new MockFileReadWriter and a cleanup function.

func (*MockFileReadWriter) Cancel 

func (f *MockFileReadWriter) Cancel() error

Cancel implements FileReadWriter.Cancel

func (*MockFileReadWriter) Commit 

func (f *MockFileReadWriter) Commit() error

Commit implements FileReadWriter.Commit

func (*MockFileReadWriter) Size 

func (f *MockFileReadWriter) Size() int64

Size implements FileReadWriter.Size

type SimpleStore 

type SimpleStore struct {
	// contains filtered or unexported fields
}

SimpleStore allows uploading / caching raw files of any format.

func NewSimpleStore 

func NewSimpleStore(config SimpleStoreConfig, stats tally.Scope) (*SimpleStore, error)

NewSimpleStore creates a new SimpleStore.

func SimpleStoreFixture 

func SimpleStoreFixture() (*SimpleStore, func())

SimpleStoreFixture returns a SimpleStore for testing purposes.

func (*SimpleStore) Close 

func (s *SimpleStore) Close()

Close terminates goroutines started by s.

func (*SimpleStore) CreateCacheFile 

func (s *SimpleStore) CreateCacheFile(name string, r io.Reader) error

CreateCacheFile initializes a cache file for name from r.

func (SimpleStore) CreateUploadFile 

func (s SimpleStore) CreateUploadFile(name string, length int64) error

func (SimpleStore) DeleteCacheFile 

func (s SimpleStore) DeleteCacheFile(name string) error

func (SimpleStore) DeleteCacheFileMetadata 

func (s SimpleStore) DeleteCacheFileMetadata(name string, md metadata.Metadata) error

func (SimpleStore) DeleteUploadFile 

func (s SimpleStore) DeleteUploadFile(name string) error

func (SimpleStore) GetCacheFileMetadata 

func (s SimpleStore) GetCacheFileMetadata(name string, md metadata.Metadata) error

func (SimpleStore) GetCacheFileReader 

func (s SimpleStore) GetCacheFileReader(name string) (FileReader, error)

func (SimpleStore) GetCacheFileStat 

func (s SimpleStore) GetCacheFileStat(name string) (os.FileInfo, error)

func (SimpleStore) GetOrSetCacheFileMetadata 

func (s SimpleStore) GetOrSetCacheFileMetadata(name string, md metadata.Metadata) error

func (SimpleStore) GetUploadFileMetadata 

func (s SimpleStore) GetUploadFileMetadata(name string, md metadata.Metadata) error

func (SimpleStore) GetUploadFileReadWriter 

func (s SimpleStore) GetUploadFileReadWriter(name string) (FileReadWriter, error)

func (SimpleStore) GetUploadFileReader 

func (s SimpleStore) GetUploadFileReader(name string) (FileReader, error)

func (SimpleStore) GetUploadFileStat 

func (s SimpleStore) GetUploadFileStat(name string) (os.FileInfo, error)

func (SimpleStore) ListCacheFiles 

func (s SimpleStore) ListCacheFiles() ([]string, error)

func (*SimpleStore) MoveUploadFileToCache 

func (s *SimpleStore) MoveUploadFileToCache(uploadName, cacheName string) error

MoveUploadFileToCache commits uploadName as cacheName.

func (SimpleStore) RangeUploadMetadata 

func (s SimpleStore) RangeUploadMetadata(name string, f func(metadata.Metadata) error) error

func (SimpleStore) SetCacheFileMetadata 

func (s SimpleStore) SetCacheFileMetadata(name string, md metadata.Metadata) (bool, error)

func (SimpleStore) SetUploadFileMetadata 

func (s SimpleStore) SetUploadFileMetadata(name string, md metadata.Metadata) error

type SimpleStoreConfig 

type SimpleStoreConfig struct {
	UploadDir     string        `yaml:"upload_dir"`
	CacheDir      string        `yaml:"cache_dir"`
	UploadCleanup CleanupConfig `yaml:"upload_cleanup"`
	CacheCleanup  CleanupConfig `yaml:"cache_cleanup"`
	// Part size limit for each file read. 0 means no limit.
	ReadPartSize int `yaml:"read_part_size"`
	// Part size limit for each file write. 0 means no limit.
	WritePartSize int `yaml:"write_part_size"`
}

SimpleStoreConfig defines SimpleStore configuration.

type Volume 

type Volume struct {
	Location string
	Weight   int
}

Volume - if provided, volumes are used to store the actual files. Symlinks will be created under state directories. This configuration is needed on hosts with multiple disks.

Source Files

View all Source files

Directories

Path Synopsis

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.