gotfs

package
v0.4.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: May 7, 2026 License: GPL-3.0 Imports: 27 Imported by: 0

README

GotFS

GotFS is a filesystem implemented on top of a key-value store. The key value store is GotKV, which is an immutable key-value store, sitting on top of a content-addressed data store. Every operation on a GotKV store returns a new reference to a new store with the change. GotFS inherits this property. All operations return a reference to a new filesystem. A GotFS filesystem is just a GotKV store with a specific key structure used to represent files and directories.

Objects

Ref

A reference to data in the content-addressed store. This type is provided by GotKV.

Extent

A part of a file. It includes a Ref, an offset, and a length. It is basically an instruction to:

  1. Get the referenced data from content-addressed storage
  2. Only include the data from offset to offset+length.

This enables packing small files into the same blob.

Info

Information about a file or directory. Most importantly the permissions and type of file.

Key Layout

All objects are represented by an Info entry at a specific key, and content stored under keys prefixed with the metadata key.

File data is stored in a content-addressed store, and references to the data are stored in GotKV.

In the following examples <0> will be used to represent a single 0 (NULL) byte.

Example: 1 File

For example: The file "test.txt" with 10B of data in it would produce the following key value pairs.

<0>< 64 bit: 0  >            -> Info (dir)
<0>test.txt<0>< 64 bit: 0  >   -> Info (file)
<0>test.txt<0>< 64 bit: 10 >   -> Extent
Example: 2 File + 1 Directory

A directory is stored as a metadata object.

<0>< 64 bit: 0  >                     -> Info (dir)
<0>mydir<0>< 64 bit: 0 >                -> Info (dir)
<0>mydir<0>myfile.txt<0>< 64 bit: 0     >  -> Info (file)
<0>mydir<0>myfile.txt<0>< 64 bit offset >  -> Part
Example 3: File at the Root

It is possible for a file to be at the root

<0>< 64 bit: 0      >       -> Info (file)
<0>< 64 bits extent >       -> Extent
<0>< next offset    >       -> Extent

Reading A File

To read from a file in GotFS you first lookup the Info entry for the path of the file. If there is not an entry at the path or the entry is not for a regular file, then return an error.

Then convert the offset to read from to a key. Seek to the first entry after that key. The extent referenced by that entry will end at the offset in the key, and will contain data overlapping the target range.

Documentation

Index

Constants

View Source 
const (
	DefaultMaxBlobSize          = 1 << 21
	DefaultMinBlobSizeData      = 1 << 12
	DefaultMeanBlobSizeData     = 1 << 20
	DefaultMeanBlobSizeMetadata = 1 << 13

	DefaultContentCacheSize  = 16
	DefaultMetadataCacheSize = 16
)
View Source 
const MaxPathLen = gotkv.MaxKeySize - 9
View Source 
const RootSize = gdat.RefSize + 1
View Source 
const Sep = '/'

Variables

This section is empty.

Functions

func Dump 

func Dump(ctx context.Context, ms stores.RO, root Root, w io.Writer) error

func Equal 

func Equal(a, b Root) bool

Equal returns true if a and b contain equivalent data.

func SpanForPath 

func SpanForPath(p string) gotkv.Span

func SplitPath 

func SplitPath(p string) []string

Types

type Builder 

type Builder struct {
	// contains filtered or unexported fields
}

Builder manages building a filesystem.

func (*Builder) BeginFile 

func (b *Builder) BeginFile(p string, mode os.FileMode) error

BeginFile creates a metadata entry for a regular file at p and directs Write calls to the content of that file.

func (*Builder) Finish 

func (b *Builder) Finish() (*Root, error)

Finish closes the builder and returns the Root to the filesystem. Finish is idempotent, and is safe to call multiple times. Not calling finish is not an error, the builder does not allocate resources other than memory.

func (*Builder) IsFinished 

func (b *Builder) IsFinished() bool

func (*Builder) Mkdir 

func (b *Builder) Mkdir(p string, mode os.FileMode) error

Mkdir creates a directory for p.

func (*Builder) Write 

func (b *Builder) Write(data []byte) (int, error)

type DiffEntry added in v0.4.0 

type DiffEntry struct {
	Key Key

	Left  maybe.Maybe[Value]
	Right maybe.Maybe[Value]
}

DiffEntry is a single element of the difference between 2 filesystems.

type Differ 

type Differ struct {
	// contains filtered or unexported fields
}

Differ iterates over the difference between 2 filesystems.

func (*Differ) Next 

func (d *Differ) Next(ctx context.Context, dsts []DiffEntry) (int, error)

type DirEnt 

type DirEnt struct {
	Name string
	Mode os.FileMode
}

type Entry added in v0.4.0 

type Entry struct {
	Key
	Value
}

type Extent 

type Extent = gotlob.Extent

type Info 

type Info struct {
	Mode  fs.FileMode
	Attrs map[string][]byte
}

func (*Info) Marshal added in v0.4.0 

func (info *Info) Marshal(out []byte) []byte

func (*Info) Unmarshal added in v0.4.0 

func (info *Info) Unmarshal(data []byte) error

type InfoDiff added in v0.4.0 

type InfoDiff struct {
	Path  string
	Left  maybe.Maybe[Info]
	Right maybe.Maybe[Info]
}

type InfoDiffer added in v0.4.0 

type InfoDiffer struct {
	// contains filtered or unexported fields
}

func (*InfoDiffer) Next added in v0.4.0 

func (it *InfoDiffer) Next(ctx context.Context, dst []InfoDiff) (int, error)

type InfoEntry added in v0.3.0 

type InfoEntry struct {
	Path string
	Info Info
}

type InfoIterator added in v0.3.0 

type InfoIterator struct {
	// contains filtered or unexported fields
}

func (*InfoIterator) Next added in v0.3.0 

func (it *InfoIterator) Next(ctx context.Context, dst []InfoEntry) (int, error)

type Iterator added in v0.4.0 

type Iterator struct {
	// contains filtered or unexported fields
}

Iterate iterates over the metadata in a gotfs filesystem.

func (*Iterator) Next added in v0.4.0 

func (it *Iterator) Next(ctx context.Context, dst []Entry) (int, error)

type Key 

type Key struct {
	// contains filtered or unexported fields
}

Key is a key that GotFS stores in GotKV. It contains a path and either an Extent end offset or an Info sentinel value.

func NewInfoKey added in v0.4.0 

func NewInfoKey(p string) (Key, error)

func (Key) ChildrenSpan 

func (k Key) ChildrenSpan() gotkv.Span

ChildrenSpan returns a span that contains all children or the path if it was a directory

func (*Key) EndAt 

func (k *Key) EndAt() uint64

EndAt returns the ending offset for an Extent

func (*Key) IsInfo 

func (k *Key) IsInfo() bool

IsInfo returns true if the key is for an Info object

func (Key) Marshal 

func (k Key) Marshal(out []byte) []byte

func (*Key) Path 

func (k *Key) Path() string

Path returns the path associated with the key.

func (Key) Prefix 

func (k Key) Prefix(out []byte) []byte

Prefix returns a prefix which all keys for this path, including Infos and Extents will have. The prefix will also include any children of the object.

func (*Key) Unmarshal 

func (k *Key) Unmarshal(data []byte) error

type Machine 

type Machine struct {
	// contains filtered or unexported fields
}

func NewMachine 

func NewMachine(par Params) Machine

func (*Machine) Check 

func (mach *Machine) Check(ctx context.Context, ms stores.RO, root Root, checkData func(ref gdat.Ref) error) error

func (*Machine) Concat added in v0.4.0 

func (mach *Machine) Concat(ctx context.Context, ss RW, segs iter.Seq[Segment]) (Segment, error)

func (*Machine) CreateFile 

func (mach *Machine) CreateFile(ctx context.Context, ss RW, x Root, p string, r io.Reader) (*Root, error)

CreateFile creates a file at p with data from r If there is an entry at p CreateFile returns an error ms is the store used for metadata ds is the store used for data.

func (*Machine) Exists 

func (mach *Machine) Exists(ctx context.Context, ms stores.RO, root Root, p string) (bool, error)

func (*Machine) ExistsDir added in v0.3.0 

func (mach *Machine) ExistsDir(ctx context.Context, ms stores.RO, root Root, p string) (bool, error)

ExistsDir returns true if there is an INFO object at p which is a directory.

func (*Machine) FileFromReader 

func (mach *Machine) FileFromReader(ctx context.Context, ss RW, mode posixfs.FileMode, r io.Reader) (*Root, error)

func (*Machine) FileFromReaders 

func (mach *Machine) FileFromReaders(ctx context.Context, ss RW, mode posixfs.FileMode, rs []io.Reader) (*Root, error)

ImportReaders creates a single file at the root from concatenating the data in rs. Each reader will be imported from in parallel.

func (*Machine) ForEach 

func (mach *Machine) ForEach(ctx context.Context, s stores.RO, root Root, p string, fn func(p string, md *Info) error) error

func (*Machine) ForEachLeaf 

func (mach *Machine) ForEachLeaf(ctx context.Context, s stores.RO, root Root, p string, fn func(p string, md *Info) error) error

ForEachLeaf calls fn with each regular file in root, beneath p.

func (*Machine) GetDirInfo 

func (mach *Machine) GetDirInfo(ctx context.Context, s stores.RO, x Root, p string) (*Info, error)

GetDirInfo returns directory metadata at p if it exists, and errors otherwise

func (*Machine) GetFileInfo 

func (mach *Machine) GetFileInfo(ctx context.Context, s stores.RO, x Root, p string) (*Info, error)

GetFileInfo returns the file metadata at p if it exists, and errors otherwise

func (*Machine) GetInfo 

func (mach *Machine) GetInfo(ctx context.Context, s stores.RO, x Root, p string) (*Info, error)

GetInfo retrieves the metadata at p if it exists and errors otherwise

func (*Machine) Graft 

func (mach *Machine) Graft(ctx context.Context, ss RW, root Root, p string, branch Root) (*Root, error)

Graft places branch at p in root. If p == "" then branch is returned unaltered.

func (*Machine) MaxInfo 

func (mach *Machine) MaxInfo(ctx context.Context, ms stores.RO, root Root, span Span) (string, *Info, error)

MaxInfo returns the maximum path and the corresponding Info for the path. If no Info entry can be found MaxInfo returns ("", nil, nil)

func (*Machine) MeanBlobSizeData 

func (mach *Machine) MeanBlobSizeData() int

func (*Machine) MeanBlobSizeMetadata 

func (mach *Machine) MeanBlobSizeMetadata() int

func (*Machine) MetadataKV added in v0.4.0 

func (mach *Machine) MetadataKV() *gotkv.Machine

MetadataKV returns the gotkv.Machine used for metadata

func (*Machine) Mkdir 

func (mach *Machine) Mkdir(ctx context.Context, s stores.RW, x Root, p string) (*Root, error)

Mkdir creates a directory at path p

func (*Machine) MkdirAll 

func (mach *Machine) MkdirAll(ctx context.Context, s stores.RW, x Root, p string) (*Root, error)

MkdirAll creates the directory p and any of p's ancestors if necessary.

func (*Machine) NewBuilder 

func (mach *Machine) NewBuilder(ctx context.Context, ss RW) *Builder

func (*Machine) NewDiffer 

func (mach *Machine) NewDiffer(ms stores.RO, left, right Root) *Differ

func (*Machine) NewEmpty 

func (mach *Machine) NewEmpty(ctx context.Context, ms stores.RW, mode fs.FileMode) (*Root, error)

NewEmpty creates a new filesystem with an empty root directory

func (*Machine) NewInfoDiffer added in v0.4.0 

func (mach *Machine) NewInfoDiffer(s stores.RO, left, right Root) InfoDiffer

func (*Machine) NewInfoIterator added in v0.3.0 

func (mach *Machine) NewInfoIterator(s stores.RW, root Root) *InfoIterator

func (*Machine) NewIterator added in v0.4.0 

func (m *Machine) NewIterator(s stores.RO, root Root, span Span) Iterator

func (*Machine) NewReader 

func (mach *Machine) NewReader(ctx context.Context, ss RO, x Root, p string) (*Reader, error)

NewReader returns an io.Reader | io.Seeker | io.ReaderAt

func (*Machine) Pick added in v0.4.0 

func (mach *Machine) Pick(ctx context.Context, s stores.RW, root Root, p string) (*Root, error)

Pick returns a new root containing everything under p, shifted to the root.

func (*Machine) Populate 

func (mach *Machine) Populate(ctx context.Context, s stores.RO, root Root, mdSet, dataSet stores.Set) error

Populate adds the ID for all the metadata blobs to mdSet and all the data blobs to dataSet

func (*Machine) PutFile 

func (mach *Machine) PutFile(ctx context.Context, ss RW, x Root, p string, r io.Reader) (*Root, error)

PutFile creates or replaces the file at path using data from r

func (*Machine) PutInfo 

func (mach *Machine) PutInfo(ctx context.Context, s stores.RW, x Root, p string, info *Info) (*Root, error)

PutInfo assigns metadata to p

func (*Machine) ReadDir 

func (mach *Machine) ReadDir(ctx context.Context, s stores.RO, x Root, p string, fn func(e DirEnt) error) error

ReadDir calls fn for every child of the directory at p.

func (*Machine) ReadFile 

func (mach *Machine) ReadFile(ctx context.Context, ss RO, x Root, p string, max int) ([]byte, error)

func (*Machine) ReadFileAt 

func (mach *Machine) ReadFileAt(ctx context.Context, ss RO, x Root, p string, start int64, buf []byte) (int, error)

ReadFileAt fills `buf` with data in the file at `p` starting at offset `start`

func (*Machine) RemoveAll 

func (mach *Machine) RemoveAll(ctx context.Context, s stores.RW, x Root, p string) (*Root, error)

func (*Machine) ShiftOut added in v0.4.0 

func (mach *Machine) ShiftOut(x Segment, p string) Segment

ShiftOut shifts all the entries in a segment out by path. A path at a/b/ in x will be at p + a/b/ in the returned segment.

func (*Machine) SizeOfFile 

func (mach *Machine) SizeOfFile(ctx context.Context, s stores.RO, x Root, p string) (uint64, error)

SizeOfFile returns the size of the file at p in bytes.

func (*Machine) Splice 

func (mach *Machine) Splice(ctx context.Context, ss RW, segs []Segment) (*Root, error)

Splice is equivalent to Concat followed by Promote

func (*Machine) Sync 

func (mach *Machine) Sync(ctx context.Context, src RO, dst WO, root Root) error

Sync ensures dst has all the data reachable from root dst and src should both be metadata stores. copyData will be called to sync metadata

type Params added in v0.4.0 

type Params struct {
	Salt [32]byte

	// MaxBlobSize is the maximum size of any blob posted by GotFS
	MaxBlobSize *int
	// MinSizeData is the minimum size of any content blob.
	MinBlobSizeData *int
	// MeanSizeData is the target mean size of all the content blobs.
	MeanBlobSizeData *int
	// MeanSizeMeta is the target meansize of all metadata blobs.
	MeanBlobSizeMetadata *int

	// ContentCacheSize is the number of blobs to keep in the content cache.
	ContentCacheSize *int
	// MetaCacheSize is the number of blobs to keep in the metadata cache.
	MetaCacheSize *int
}

func (Params) GetContentCacheSize added in v0.4.0 

func (p Params) GetContentCacheSize() int

func (Params) GetMaxBlobSize added in v0.4.0 

func (p Params) GetMaxBlobSize() int

func (Params) GetMeanBlobSizeData added in v0.4.0 

func (p Params) GetMeanBlobSizeData() int

func (Params) GetMeanBlobSizeMetadata added in v0.4.0 

func (p Params) GetMeanBlobSizeMetadata() int

func (Params) GetMetaCacheSize added in v0.4.0 

func (p Params) GetMetaCacheSize() int

func (Params) GetMinSizeData added in v0.4.0 

func (p Params) GetMinSizeData() int

type RO added in v0.4.0 

type RO struct {
	Data     stores.RO
	Metadata stores.RO
}

RO are read only stores used by gotfs

type RW added in v0.4.0 

type RW struct {
	Data     stores.RW
	Metadata stores.RW
}

RW are read-write stores used by gotfs

func (RW) RO added in v0.4.0 

func (rw RW) RO() RO

func (RW) WO added in v0.4.0 

func (rw RW) WO() WO

type Reader 

type Reader = gotlob.Reader

type Ref 

type Ref = gotkv.Ref

type Root 

type Root struct {
	Ref   Ref   `json:"ref"`
	Depth uint8 `json:"depth"`
}

func ParseRoot 

func ParseRoot(data []byte) (*Root, error)

func Promote added in v0.4.0 

func Promote(ctx context.Context, seg Segment) (*Root, error)

Promote promotes a segment to a Root if the segment has the correct first key.

func (Root) Marshal 

func (r Root) Marshal(out []byte) []byte

Marshal appends the root data to out and returns the new slice.

func (Root) Segment added in v0.4.0 

func (r Root) Segment() Segment

Segment returns the root as a single segment.

func (Root) ToGotKV 

func (r Root) ToGotKV() gotkv.Root

func (*Root) Unmarshal 

func (r *Root) Unmarshal(data []byte) error

Unmarshal parses the root data from data and returns an error if the data is invalid.

type Segment 

type Segment struct {
	// Contents is the gotkv instance representing the segment.
	// If it contains entries outside of Span, they will not be used.
	// If Contents is the zero value, then it will be interpretted as empty
	Contents gotkv.Root
	// Span is the span in the final Splice operation
	Span gotkv.Span
}

Segment is a contiguous subset of a GotFS instance. It may not be a valid Root.

func (*Segment) Marshal added in v0.4.0 

func (s *Segment) Marshal(out []byte) []byte

func (Segment) String 

func (s Segment) String() string

func (*Segment) Unmarshal added in v0.4.0 

func (s *Segment) Unmarshal(data []byte) error

type Span 

type Span = gotkv.Span

type Store 

type Store = gotkv.Store

type Value added in v0.4.0 

type Value struct {
	Info   Info
	Extent Extent
}

Value is either an Info or an Extent

type WO added in v0.4.0 

type WO struct {
	Data     stores.WO
	Metadata stores.WO
}

WO are read only stores used by gotfs

Source Files

View all Source files

Directories

Path Synopsis

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.