base

package
v0.1.17 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Dec 16, 2025 License: Apache-2.0 Imports: 20 Imported by: 12

Documentation

Overview

Copyright (c) 2016-2019 Uber Technologies, Inc.

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at 

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Copyright (c) 2016-2019 Uber Technologies, Inc.

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at 

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Copyright (c) 2016-2019 Uber Technologies, Inc.

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at 

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Copyright (c) 2016-2019 Uber Technologies, Inc.

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at 

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Copyright (c) 2016-2019 Uber Technologies, Inc.

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at 

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Copyright (c) 2016-2019 Uber Technologies, Inc.

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at 

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Copyright (c) 2016-2019 Uber Technologies, Inc.

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at 

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Copyright (c) 2016-2019 Uber Technologies, Inc.

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at 

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Index

Constants

View Source 
const DefaultDataFileName = "data"

DefaultDataFileName is the name of the actual blob data file.

View Source 
const DefaultDirPermission = 0775

DefaultDirPermission is the default permission for new directories.

View Source 
const DefaultShardIDLength = 2

DefaultShardIDLength is the number of bytes of file digest to be used for shard ID. For every byte (2 HEX char), one more level of directories will be created.

Variables

View Source 
var (
	ErrFilePersisted = errors.New("file is persisted")
	ErrInvalidName   = errors.New("invalid name")
)

FileEntry errors.

Functions

func IsFileStateError 

func IsFileStateError(err error) bool

IsFileStateError returns true if the param is of FileStateError type.

Types

type BufferReadWriter added in v0.1.13 

type BufferReadWriter struct {
	// contains filtered or unexported fields
}

BufferReadWriter implements FileReadWriter interface for in-memory buffering.

func NewBufferReadWriter added in v0.1.13 

func NewBufferReadWriter(size uint64) *BufferReadWriter

NewBufferReadWriter creates a new BufferReadWriter with an initial capacity of size bytes.

func (*BufferReadWriter) Bytes added in v0.1.13 

func (b *BufferReadWriter) Bytes() []byte

Bytes returns the full buffer

func (*BufferReadWriter) Cancel added in v0.1.13 

func (b *BufferReadWriter) Cancel() error

Cancel is no-op

func (*BufferReadWriter) Close added in v0.1.13 

func (b *BufferReadWriter) Close() error

Close is no-op

func (*BufferReadWriter) Commit added in v0.1.13 

func (b *BufferReadWriter) Commit() error

Commit is no-op

func (*BufferReadWriter) Read added in v0.1.13 

func (b *BufferReadWriter) Read(p []byte) (n int, err error)

Read implements io.Reader for sequential reads.

func (*BufferReadWriter) ReadAt added in v0.1.13 

func (b *BufferReadWriter) ReadAt(p []byte, off int64) (n int, err error)

ReadAt implements io.ReaderAt.

func (*BufferReadWriter) Seek added in v0.1.13 

func (b *BufferReadWriter) Seek(offset int64, whence int) (int64, error)

Seek implements io.Seeker.

func (*BufferReadWriter) Size added in v0.1.13 

func (b *BufferReadWriter) Size() int64

Size returns the size of the buffer

func (*BufferReadWriter) Write added in v0.1.13 

func (b *BufferReadWriter) Write(p []byte) (n int, err error)

Write implements io.Writer by using WriteAt with current write offset.

func (*BufferReadWriter) WriteAt added in v0.1.13 

func (b *BufferReadWriter) WriteAt(p []byte, off int64) (n int, err error)

WriteAt implements io.WriterAt for parallel writes.

type FileEntry 

type FileEntry interface {
	GetState() FileState
	GetName() string
	GetPath() string
	GetStat() (os.FileInfo, error)

	Create(targetState FileState, len int64) error
	Reload() error
	MoveFrom(targetState FileState, sourcePath string) error
	Move(targetState FileState) error
	LinkTo(targetPath string) error
	Delete() error

	GetReader(readPartSize int) (FileReader, error)
	GetReadWriter(readPartSize, writePartSize int) (FileReadWriter, error)

	AddMetadata(md metadata.Metadata) error

	GetMetadata(md metadata.Metadata) error
	SetMetadata(md metadata.Metadata) (bool, error)
	SetMetadataAt(md metadata.Metadata, b []byte, offset int64) (updated bool, err error)
	GetOrSetMetadata(md metadata.Metadata) error
	DeleteMetadata(md metadata.Metadata) error

	RangeMetadata(f func(md metadata.Metadata) error) error
}

FileEntry manages one file and its metadata. It doesn't guarantee thread-safety; That should be handled by FileMap.

type FileEntryFactory 

type FileEntryFactory interface {
	// Create creates a file entry given a state directory and a name.
	// It calls GetRelativePath to generate the actual file path under given directory,
	Create(name string, state FileState) (FileEntry, error)

	// GetRelativePath returns the relative path for a file entry.
	// The path is relative to the state directory that file entry belongs to.
	// i.e. a file entry can have a relative path of 00/0e/filename under directory /var/cache/
	GetRelativePath(name string) string

	// ListNames lists all file entry names in state.
	ListNames(state FileState) ([]string, error)
}

FileEntryFactory initializes FileEntry obj.

func NewCASFileEntryFactory 

func NewCASFileEntryFactory() FileEntryFactory

NewCASFileEntryFactory is the constructor for casFileEntryFactory.

func NewLocalFileEntryFactory 

func NewLocalFileEntryFactory() FileEntryFactory

NewLocalFileEntryFactory is the constructor for localFileEntryFactory.

type FileMap 

type FileMap interface {
	Contains(name string) bool
	TryStore(name string, entry FileEntry, f func(string, FileEntry) bool) bool
	LoadForWrite(name string, f func(string, FileEntry)) bool
	LoadForRead(name string, f func(string, FileEntry)) bool
	LoadForPeek(name string, f func(string, FileEntry)) bool
	Delete(name string, f func(string, FileEntry) bool) bool
}

FileMap is a thread-safe name -> FileEntry map.

func NewLATFileMap 

func NewLATFileMap(clk clock.Clock) FileMap

NewLATFileMap creates a new file map that tracks last access time, but no auto-eviction.

func NewLRUFileMap 

func NewLRUFileMap(size int, clk clock.Clock) FileMap

NewLRUFileMap creates a new LRU map given capacity.

type FileOp 

type FileOp interface {
	AcceptState(state FileState) FileOp
	GetAcceptableStates() map[FileState]interface{}

	CreateFile(name string, createState FileState, len int64) error
	MoveFileFrom(name string, createState FileState, sourcePath string) error
	MoveFile(name string, goalState FileState) error
	LinkFileTo(name string, targetPath string) error
	DeleteFile(name string) error

	GetFilePath(name string) (string, error)
	GetFileStat(name string) (os.FileInfo, error)

	GetFileReader(name string, readPartSize int) (FileReader, error)
	GetFileReadWriter(name string, readPartSize, writePartSize int) (FileReadWriter, error)

	GetFileMetadata(name string, md metadata.Metadata) error
	SetFileMetadata(name string, md metadata.Metadata) (bool, error)
	SetFileMetadataAt(name string, md metadata.Metadata, b []byte, offset int64) (bool, error)
	GetOrSetFileMetadata(name string, md metadata.Metadata) error
	DeleteFileMetadata(name string, md metadata.Metadata) error

	RangeFileMetadata(name string, f func(metadata.Metadata) error) error

	ListNames() ([]string, error)

	String() string
}

FileOp performs one file or metadata operation on FileStore, given a list of acceptable states.

func NewLocalFileOp 

func NewLocalFileOp(s *localFileStore) FileOp

NewLocalFileOp inits a new FileOp obj.

type FileReadWriter 

type FileReadWriter interface {
	FileReader
	io.Writer
	io.WriterAt

	Cancel() error // required by docker registry.
	Commit() error // required by docker registry.
}

FileReadWriter provides read/write operation on a file.

type FileReader 

type FileReader interface {
	io.Reader
	io.ReaderAt
	io.Seeker
	io.Closer
	Size() int64
}

FileReader provides read operation on a file.

type FileState 

type FileState struct {
	// contains filtered or unexported fields
}

FileState decides what directory a file is in. A file can only be in one state at any given time.

func NewFileState 

func NewFileState(directory string) FileState

NewFileState creates a new FileState for directory.

func (FileState) GetDirectory 

func (s FileState) GetDirectory() string

GetDirectory returns the FileState's directory.

type FileStateError 

type FileStateError struct {
	Op    string
	Name  string
	State FileState
	Msg   string
}

FileStateError represents errors related to file state. It's used when a file is not in the state it was supposed to be in.

func (*FileStateError) Error 

func (e *FileStateError) Error() string

type FileStore 

type FileStore interface {
	NewFileOp() FileOp
}

FileStore manages files and their metadata. Actual operations are done through FileOp.

func NewCASFileStore 

func NewCASFileStore(clk clock.Clock) FileStore

NewCASFileStore initializes and returns a new Content-Addressable FileStore. It uses the first few bytes of file digest (which is also used as file name) as shard ID. For every byte, one more level of directories will be created.

func NewCASFileStoreWithLRUMap 

func NewCASFileStoreWithLRUMap(size int, clk clock.Clock) FileStore

NewCASFileStoreWithLRUMap initializes and returns a new Content-Addressable FileStore. It uses the first few bytes of file digest (which is also used as file name) as shard ID. For every byte, one more level of directories will be created. It also stores objects in a LRU FileStore. When size exceeds limit, the least recently accessed entry will be removed.

func NewLRUFileStore 

func NewLRUFileStore(size int, clk clock.Clock) FileStore

NewLRUFileStore initializes and returns a new LRU FileStore. When size exceeds limit, the least recently accessed entry will be removed.

func NewLocalFileStore 

func NewLocalFileStore(clk clock.Clock) FileStore

NewLocalFileStore initializes and returns a new FileStore.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.