Documentation
¶
Overview ¶
Package mfs implements an in-memory model of a mutable IPFS filesystem.
The filesystem is rooted at a Root which contains a tree of Directory and File nodes. Changes to files and directories are accumulated in memory and flushed to the underlying DAG service.
Structure ¶
- Root: Top-level entry point, created via NewRoot, with optional republishing of the root CID on changes
- Directory: A mutable directory that maps names to child nodes
- File: A mutable file backed by a UnixFS DAG
Filesystem Operations ¶
Top-level functions Mv, Lookup, and FlushPath operate on paths within the MFS tree. Directories support adding, removing, and listing entries. Files support reading and writing through FileDescriptor.
Index ¶
- Constants
- Variables
- func Chmod(rt *Root, pth string, mode os.FileMode) error
- func FlushPath(ctx context.Context, rt *Root, pth string) (ipld.Node, error)
- func IsDir(fsn FSNode) bool
- func IsFile(fsn FSNode) bool
- func Mkdir(r *Root, pth string, opts MkdirOpts, dirOpts ...Option) error
- func Mv(r *Root, src, dst string) error
- func PutNode(r *Root, path string, nd ipld.Node) error
- func Touch(rt *Root, pth string, ts time.Time) error
- type Directory
- func (d *Directory) AddChild(name string, nd ipld.Node) error
- func (d *Directory) Child(name string) (FSNode, error)
- func (d *Directory) Flush() error
- func (d *Directory) ForEachEntry(ctx context.Context, f func(NodeListing) error) error
- func (d *Directory) GetCidBuilder() cid.Builder
- func (d *Directory) GetNode() (ipld.Node, error)
- func (d *Directory) List(ctx context.Context) ([]NodeListing, error)
- func (d *Directory) ListNames(ctx context.Context) ([]string, error)
- func (d *Directory) Mkdir(name string) (*Directory, error)
- func (d *Directory) MkdirWithOpts(name string, opts ...Option) (*Directory, error)
- func (d *Directory) ModTime() (time.Time, error)
- func (d *Directory) Mode() (os.FileMode, error)
- func (d *Directory) Path() string
- func (d *Directory) SetCidBuilder(b cid.Builder)
- func (d *Directory) SetModTime(ts time.Time) error
- func (d *Directory) SetMode(mode os.FileMode) error
- func (d *Directory) Type() NodeType
- func (d *Directory) Uncache(name string)
- func (d *Directory) Unlink(name string) error
- type FSNode
- type File
- func (fi *File) Flush() error
- func (fi *File) GetNode() (ipld.Node, error)
- func (fi *File) ModTime() (time.Time, error)
- func (fi *File) Mode() (os.FileMode, error)
- func (fi *File) Open(ctx context.Context, flags Flags) (_ FileDescriptor, _retErr error)
- func (fi *File) SetModTime(ts time.Time) error
- func (fi *File) SetMode(mode os.FileMode) error
- func (fi *File) Size() (int64, error)
- func (fi *File) Sync() error
- func (fi *File) Type() NodeType
- type FileDescriptor
- type Flags
- type MkdirOpts
- type NodeListing
- type NodeType
- type Option
- func WithChunker(c chunker.SplitterGen) Option
- func WithCidBuilder(b cid.Builder) Option
- func WithFetchTimeout(d time.Duration) Option
- func WithHAMTShardingSize(size int) Option
- func WithMaxHAMTFanout(n int) Option
- func WithMaxLinks(n int) Option
- func WithModTime(t time.Time) Option
- func WithMode(mode os.FileMode) Option
- func WithSizeEstimationMode(mode uio.SizeEstimationMode) Option
- type PubFunc
- type Republisher
- type Root
Constants ¶
const DefaultFetchTimeout = 5 * time.Minute
DefaultFetchTimeout is the fetch timeout applied to an MFS root that does not set its own with WithFetchTimeout. It bounds a single under-lock DAG read (a child lookup or HAMT-shard walk) so a missing, unreachable block fails with a deadline error instead of blocking MFS forever.
Five minutes is deliberately generous. These reads fetch small directory and shard nodes: one resolves in well under a second from a connected peer, and usually within tens of seconds even when its provider must be found through the DHT, so a live-but-slow fetch never reaches the bound. It only takes effect against genuinely unreachable data, trading a permanent wedge for a recoverable error. On teardown the root context cancels the read at once, so the bound never delays shutdown.
Pass WithFetchTimeout(0) to disable it and restore unbounded reads.
Variables ¶
var ( ErrNotYetImplemented = errors.New("not yet implemented") ErrInvalidChild = errors.New("invalid child node") ErrDirExists = errors.New("directory already has entry by that name") )
var ( ErrNotExist = errors.New("no such rootfs") ErrClosed = errors.New("file closed") )
TODO: Remove if not used.
var ErrIsDirectory = errors.New("error: is a directory")
TODO: Remove if not used.
Functions ¶
func Mkdir ¶
Mkdir creates a directory at 'path' under the root. Any Option values not explicitly provided are inherited from the root directory's current settings.
func Mv ¶
Mv moves the file or directory at 'src' to 'dst' TODO: Document what the strings 'src' and 'dst' represent.
func PutNode ¶
PutNode inserts 'nd' at 'path' in the given mfs TODO: Rename or clearly document that this is not about nodes but actually MFS files/directories (that in the underlying representation can be considered as just nodes). TODO: Document why are we handling IPLD nodes in the first place when we are actually referring to files/directories (that is, it can't be any node, it has to have a specific format). TODO: Can this function add directories or just files? What would be the difference between adding a directory with this method and creating it with `Mkdir`.
Types ¶
type Directory ¶
type Directory struct {
// contains filtered or unexported fields
}
TODO: There's too much functionality associated with this structure, let's organize it (and if possible extract part of it elsewhere) and document the main features of `Directory` here.
func NewDirectory ¶
func NewDirectory(ctx context.Context, name string, node ipld.Node, parent parent, dserv ipld.DAGService, prov provider.MultihashProvider) (*Directory, error)
NewDirectory constructs a new MFS directory.
You probably don't want to call this directly. Instead, construct a new root using NewRoot.
func NewEmptyDirectory ¶ added in v0.30.0
func NewEmptyDirectory(ctx context.Context, name string, p parent, dserv ipld.DAGService, prov provider.MultihashProvider, opts ...Option) (*Directory, error)
NewEmptyDirectory creates an empty MFS directory with the given [Option]s. The directory is added to the DAGService. To create a new MFS root use NewEmptyRoot instead.
func (*Directory) AddChild ¶
AddChild adds the node 'nd' under this directory giving it the name 'name'
func (*Directory) ForEachEntry ¶
func (*Directory) GetCidBuilder ¶
GetCidBuilder gets the CID builder of the root node
func (*Directory) Mkdir ¶
Mkdir creates a child directory that inherits settings from this directory.
func (*Directory) MkdirWithOpts ¶ added in v0.23.0
MkdirWithOpts creates a child directory with explicit [Option]s.
func (*Directory) ModTime ¶ added in v0.38.0
ModTime returns the directory's last modification time from UnixFS metadata. Returns zero time when no mtime is stored.
func (*Directory) Mode ¶ added in v0.38.0
Mode returns the directory's POSIX permission bits from UnixFS metadata. Returns 0 when no mode is stored.
func (*Directory) SetCidBuilder ¶
SetCidBuilder sets the CID builder
type FSNode ¶
type FSNode interface {
GetNode() (ipld.Node, error)
Flush() error
Type() NodeType
SetModTime(ts time.Time) error
SetMode(mode os.FileMode) error
}
FSNode abstracts the `Directory` and `File` structures, it represents any child node in the MFS (i.e., all the nodes besides the `Root`). It is the counterpart of the `parent` interface which represents any parent node in the MFS (`Root` and `Directory`). (Not to be confused with the `unixfs.FSNode`.)
type File ¶
type File struct {
RawLeaves bool
// contains filtered or unexported fields
}
File represents a file in the MFS, its logic its mainly targeted to coordinating (potentially many) `FileDescriptor`s pointing to it.
func NewFile ¶
func NewFile(name string, node ipld.Node, parent parent, dserv ipld.DAGService, providing provider.MultihashProvider) (*File, error)
NewFile returns a NewFile object with the given parameters. If the CID version is non-zero RawLeaves will be enabled. A nil providing parameter means that MFS will not provide content to the routing system.
func (*File) Flush ¶
TODO: Tight coupling with the `FileDescriptor`, at the very least this should be an independent function that takes a `File` argument and automates the open/flush/close operations.
TODO: Why do we need to flush a file that isn't opened? (the `OpenWriteOnly` seems to implicitly be targeting a closed file, a file we forgot to flush? can we close a file without flushing?)
func (*File) GetNode ¶
GetNode returns the dag node associated with this file
TODO: Use this method and do not access the `nodeLock` directly anywhere else.
func (*File) Open ¶
Open returns a FileDescriptor for reading or writing the file. The context controls how long the descriptor waits for blocks it has to fetch from the network while writing (and while reading through the plain Read method): if a block is missing, the operation stops when ctx is cancelled instead of waiting forever. Use CtxReadFull to give a single read its own context.
func (*File) SetModTime ¶ added in v0.23.0
SetModTime sets the files' last modification time.
func (*File) Size ¶
Size returns the size of this file
TODO: Should we be providing this API?
TODO: There's already a `FileDescriptor.Size()` that through the `DagModifier`'s `fileSize` function is doing pretty much the same thing as here, we should at least call that function and wrap the `ErrNotUnixfs` with an MFS text.
type FileDescriptor ¶
type FileDescriptor interface {
io.Reader
CtxReadFull(context.Context, []byte) (int, error)
io.Writer
io.WriterAt
io.Closer
io.Seeker
Truncate(int64) error
Size() (int64, error)
Flush() error
}
One `File` can have many `FileDescriptor`s associated to it (only one if it's RW, many if they are RO, see `File.desclock`). A `FileDescriptor` contains the "view" of the file (through an instance of a `DagModifier`), that's why it (and not the `File`) has the responsibility to `Flush` (which crystallizes that view in the `File`'s `Node`).
type MkdirOpts ¶
type MkdirOpts struct {
Mkparents bool // create intermediate directories as needed
Flush bool // flush the final directory after creation
}
MkdirOpts holds operation flags for Mkdir.
type Option ¶ added in v0.38.0
type Option func(*options)
Option configures MFS root and directory creation.
Options follow the functional options pattern used by the underlying uio.DirectoryOption in ipld/unixfs/io. Most settings are inherited from the parent directory when not explicitly set.
Chunker is the only setting that uses a different inheritance path: it is stored per-directory and accessed by files via the [parent] interface's getChunker method, rather than being copied through options. This is because chunker is consumed by File.Open (not by the unixfs directory layer), and a single interface method is a cleaner abstraction than explicit copying for a value that never changes after root creation.
func WithChunker ¶ added in v0.37.0
func WithChunker(c chunker.SplitterGen) Option
WithChunker sets the chunker factory for files created under this MFS root. If not set, chunker.DefaultSplitter is used.
Unlike other options, the chunker is not propagated through [options] but through the [parent] interface (see Option for details). This option only takes effect when passed to NewRoot or NewEmptyRoot.
func WithCidBuilder ¶ added in v0.38.0
WithCidBuilder sets the CID builder (version, codec, hash function) for new directories. If not set, the parent directory's builder is used.
func WithFetchTimeout ¶ added in v0.42.0
WithFetchTimeout bounds each DAG read that MFS performs while holding a directory lock: resolving a child link, or walking a HAMT shard. When unset, DefaultFetchTimeout applies; pass zero to disable the bound and restore unbounded reads.
MFS routinely holds a DAG that is only lazily linked: the root node is present locally while the rest of the tree is fetched from the network on demand, the first time a path is traversed. This is what backs a reference like the one "ipfs files cp /ipfs/<cid> /dst" creates, where only the referenced root is stored and its children are pulled in as they are accessed. Those reads must stay online, so this is a timeout, not a switch to local-only lookups: a reachable remote node is still fetched. Directory and shard nodes are small, so a present-or-reachable one resolves well within a generous bound.
The bound only bites when a node is neither local nor reachable: a lazily-linked child whose providers are gone, a block removed out of band, or a store left inconsistent by a crash. Without a bound that read blocks on the network forever while the directory lock is held, wedging every operation queued behind it, and graceful shutdown (which flushes MFS) with it. With one the read fails with a deadline error and releases the lock, turning a permanent wedge into an ordinary, recoverable error. See DefaultFetchTimeout for the default and its rationale.
This option only takes effect when passed to NewRoot or NewEmptyRoot; it is inherited by every directory under the root.
func WithHAMTShardingSize ¶ added in v0.37.0
WithHAMTShardingSize sets the per-directory serialized block size threshold in bytes for converting to a HAMT shard. If not set, the global uio.HAMTShardingSize is used.
func WithMaxHAMTFanout ¶ added in v0.37.0
WithMaxHAMTFanout sets the maximum fanout (bucket width) for HAMT sharded directories. Must be a power of 2 and a multiple of 8.
func WithMaxLinks ¶ added in v0.37.0
WithMaxLinks sets the maximum number of directory entries before the directory is converted to a HAMT shard.
func WithModTime ¶ added in v0.38.0
WithModTime sets the modification time on the created directory.
func WithSizeEstimationMode ¶ added in v0.37.0
func WithSizeEstimationMode(mode uio.SizeEstimationMode) Option
WithSizeEstimationMode sets the method used to estimate directory size for HAMT sharding threshold decisions.
type PubFunc ¶
PubFunc is the user-defined function that determines exactly what logic entails "publishing" a `Cid` value.
type Republisher ¶
type Republisher struct {
// contains filtered or unexported fields
}
Republisher manages when to publish a given entry.
func NewRepublisher ¶
NewRepublisher creates a new Republisher object to republish the given root using the given short and long time intervals.
func (*Republisher) Close ¶
func (rp *Republisher) Close() error
Close tells the republisher to stop and waits for it to stop.
func (*Republisher) Update ¶
func (rp *Republisher) Update(c cid.Cid)
Update the current value. The value will be published after a delay but each consecutive call to Update may extend this delay up to TimeoutLong.
type Root ¶
type Root struct {
// contains filtered or unexported fields
}
Root represents the root of a filesystem tree.
func NewEmptyRoot ¶ added in v0.30.0
func NewEmptyRoot(ctx context.Context, ds ipld.DAGService, pf PubFunc, prov provider.MultihashProvider, opts ...Option) (*Root, error)
NewEmptyRoot creates an empty Root directory with the given [Option]s. A republisher is created if PubFunc is not nil.
func NewRoot ¶
func NewRoot(ctx context.Context, ds ipld.DAGService, node *dag.ProtoNode, pf PubFunc, prov provider.MultihashProvider, opts ...Option) (*Root, error)
NewRoot creates a new Root from an existing DAG node and starts a republisher routine for it. The provided [Option]s configure the root directory's DAG-shape settings (CidBuilder, MaxLinks, etc.).
func (*Root) Flush ¶
Flush signals that an update has occurred since the last publish, and updates the Root republisher. TODO: We are definitely abusing the "flush" terminology here.
func (*Root) FlushMemFree ¶
FlushMemFree flushes the root directory and then uncaches all of its links. This has the effect of clearing out potentially stale references and allows them to be garbage collected. CAUTION: Take care not to ever call this while holding a reference to any child directories. Those directories will be bad references and using them may have unintended racy side effects. A better implemented mfs system (one that does smarter internal caching and refcounting) shouldnt need this method. TODO: Review the motivation behind this method once the cache system is refactored.
func (*Root) GetChunker ¶ added in v0.37.0
func (kr *Root) GetChunker() chunker.SplitterGen
GetChunker returns the chunker factory, or nil if using default.
func (*Root) GetDirectory ¶
GetDirectory returns the root directory.
Source Files