Documentation
¶
Overview ¶
Package filecache keeps a local copy of a big file that lives on a slow backend, so that the request which pays for the transfer is honest about it ("preparing… 40 %") and every request after it is served at local-disk speed.
The requirement, in the owner's words (translated from Turkish):
"if the fs is slow and the file is big, we need to do something like telling the user we're building a cache just for them and starting the download once the cache is ready."
A 4 GB file on a NAS or a distant bucket used to dribble at the backend's speed with no explanation, no seeking and no resume; a retry paid for the whole object again. This package is the other half of the answer (the first half is storage.RangeReader): fetch it once, locally, say so while it happens, and then serve it like a local file.
This is internal/sharezip, generalised ¶
sharezip already caches folder-share ZIPs on disk behind a generation registry with Percent/Wait/StartOrGet and a "preparing…" page. The shape is proven, so it is reused rather than reinvented — with the one thing it does NOT have, and the reason this package exists as its own thing:
⚠⚠ A GLOBAL CAP. sharezip prunes per node (pruneOld), which bounds nothing: N nodes cost N archives. A per-node file cache with no ceiling is a disk incident, and this repo had one this month — leaked multipart temp files filled 29 GB of the host in two hours. So MaxBytes is enforced here, on every fill, including the bytes of fills that are still being written.
What it is keyed on ¶
nodeID + a content signature (the driver's ETag when it has one, size+mtime when it does not — local has no ETag, measured). A changed file therefore gets a different key and invalidates itself, exactly like the ZIP cache's content signature. Nothing is ever overwritten in place.
What it deliberately does NOT do ¶
- It does not decide WHO may read a file. ACL, tenancy and share caps are checked by the caller before a cache key is ever computed; this package never sees a request. A cached entry is bytes plus a key, nothing more.
- It does not persist across restarts as state. The directory is re-read at startup (the files are still valid — the key carries the signature), but an interrupted fill leaves only a .tmp- file, which is swept.
- It is not internal/ops. A fill is not a user operation: nobody submitted it, nobody may cancel it, and it must not survive into a tray as a row that outlives the download that triggered it. It is an in-memory generation registry, like sharezip's, and dying with the process is correct — the next request starts it again.
Index ¶
- Constants
- Variables
- func Key(nodeID int64, sig string) string
- func Signature(etag string, size int64, mtime time.Time) string
- type Cache
- func (c *Cache) Acquire(key string) (*Entry, bool)
- func (c *Cache) Bytes() int64
- func (c *Cache) Drop(key string) bool
- func (c *Cache) Enabled() bool
- func (c *Cache) Filling(key string) (*Fill, bool)
- func (c *Cache) Len() int
- func (c *Cache) MaxBytes() int64
- func (c *Cache) MinSize() int64
- func (c *Cache) Qualifies(storageID int64, size int64, operatorSlow bool) bool
- func (c *Cache) Ready(key string) bool
- func (c *Cache) Slow(storageID int64, operatorSlow bool) bool
- func (c *Cache) StartOrGet(ctx context.Context, key string, size int64, open OpenFunc) (*Fill, error)
- type Config
- type Entry
- type Fill
- type OpenFunc
Constants ¶
const ( // DefaultMinSize is the smallest file worth preparing. Below it the // preparation costs more than the transfer it saves. DefaultMinSize = 64 << 20 // 64 MiB // DefaultMaxBytes is the global ceiling on the whole cache directory. DefaultMaxBytes = 20 << 30 // 20 GiB // DefaultPinTTL is how long a "this file is cached" answer stays true // for eviction purposes. It exists so that a caller which asked // "can I range this?" and then ranged it cannot have the entry pulled // out from under it in between. DefaultPinTTL = time.Minute )
Defaults. Every one of them is overridable from the environment; see config.CacheConfig.
const DefaultSlowBytesPerSec = 10 << 20
DefaultSlowBytesPerSec is the measured throughput below which a storage is treated as slow. 10 MiB/s is roughly a 100 Mbit link, and at that rate the smallest qualifying file (64 MiB) already takes seven seconds — long enough that "preparing…" is information rather than noise, and short enough that a healthy S3 or a local disk (both an order of magnitude faster) never trips it.
Variables ¶
var ErrNoRoom = errors.New("filecache: no room under the global cap")
ErrNoRoom means the global cap cannot accommodate this file without evicting something that is in use. The caller must fall back to streaming from the driver — never to exceeding the cap.
var ErrSizeChanged = errors.New("filecache: object changed size during fetch")
ErrSizeChanged means the object changed length while it was being fetched. The partial copy is discarded: the key promises a specific signature, and serving a body of a different length through http.ServeContent (which trusts the size it was given) would hand out a truncated or padded file.
Functions ¶
func Key ¶
Key combines a node id with a content signature. An empty result means "not cacheable" — no node row (a path outside the catalogue) or no signature at all — and every entry point treats it as a miss.
func Signature ¶
Signature is the content half of a cache key: the driver's ETag when there is one, and size+mtime when there is not.
⚠ The fallback is not a nicety. The local driver reports no ETag at all (measured: drivers/local never sets Object.Etag), and it is the driver an NFS/SMB mount uses — precisely the slow case this package exists for. A key that required an ETag would therefore never cache the most important case.
Types ¶
type Cache ¶
type Cache struct {
// contains filtered or unexported fields
}
Cache is a global-capped, LRU-evicted local cache of file bodies. Which storages are slow enough to be worth caching is decided in meter.go, off the rate internal/throughput measures — this type keeps no samples of its own.
func New ¶
New returns a Cache. It reads the directory once so that entries survive a restart, and sweeps the .tmp- files an interrupted fill leaves behind.
func (*Cache) Acquire ¶
Acquire pins and returns a ready entry, or reports false. A fill in flight is not a hit — its file does not exist yet.
func (*Cache) Bytes ¶
Bytes is what the cache currently occupies, counting fills in flight at their declared size. Exported for tests and for the metrics chunk.
func (*Cache) Drop ¶
Drop removes an entry by key if it is not in use. Reports whether it went. Used by tests and by the invalidation paths that know a node's bytes have been replaced out from under a key.
func (*Cache) Filling ¶
Filling returns the in-flight fetch for key, if there is one. It starts nothing: a progress poll must be able to ask "how is it going" without being able to make it go.
func (*Cache) Qualifies ¶
Qualifies is the whole gate in one place: caching is on, the file is big enough to be worth it, and the storage is slow enough to need it.
Small files never qualify, whatever the operator says about the storage — a 40 KiB PNG on the slowest NAS in the world is one round trip, and answering it with "preparing… 0 %" would be a worse product than answering it with the file.
func (*Cache) Ready ¶
Ready reports whether key is cached AND promises, for PinTTL, that it will still be there. Callers use it to answer "can this be served ranged?" before they commit to serving it that way; the soft pin is what stops the answer from going stale between the question and the read.
func (*Cache) Slow ¶
Slow reports whether a storage is slow enough that preparing a local copy beats streaming from it.
The policy, in order, and the reason for each step:
- Measured fast overrides the operator. A storage flagged `slow: true` that measures at twice the threshold is NOT cached. This is the "must not make things worse" rule: on a fast backend, preparing turns an instant stream into a wait for a full prefetch, and the operator's flag is a guess about the network while the meter is a measurement of it. The margin (2x) means we only override when the evidence is not close.
- The operator's flag. A NAS that has never been read is the case the meter cannot know about yet, and the operator can.
- Measured slow. Once there are enough samples, the meter can promote a storage nobody flagged.
- Otherwise no. An unmeasured, unflagged storage keeps today's behaviour. Defaulting to "cache it" would put every fresh install behind a preparing screen on its first big download.
func (*Cache) StartOrGet ¶
func (c *Cache) StartOrGet(ctx context.Context, key string, size int64, open OpenFunc) (*Fill, error)
StartOrGet makes sure a fetch for key is running and returns it. A key that is already cached returns a finished Fill (Done, no error), so the common path is one map lookup.
⚠ The fetch runs on context.WithoutCancel(ctx), taken here rather than left to callers. The whole point of the cache is that several requests wait on one fetch: if the first client hangs up and takes the fetch with it, every other waiter is punished for someone else's disconnect. (Post-response work dying with the request context is how the share download counter lost writes in v0.18.0 — same trap, different surface.)
It returns ErrNoRoom when the global cap cannot fit `size` without evicting something in use. That is not an error to show a user: it means "serve this the way you always did".
type Config ¶
type Config struct {
// Dir is the cache directory, normally <data_dir>/cache. Empty disables
// the cache entirely and every method degrades to "not cached".
Dir string
// MinSize is the smallest file that qualifies. <=0 uses DefaultMinSize.
MinSize int64
// MaxBytes is the global ceiling. <=0 uses DefaultMaxBytes; it is never
// unlimited, on purpose.
MaxBytes int64
// SlowBytesPerSec is the measured throughput below which a storage counts
// as slow. <=0 uses DefaultSlowBytesPerSec.
SlowBytesPerSec int64
// PinTTL overrides DefaultPinTTL. Tests use a short one.
PinTTL time.Duration
}
Config configures a Cache. The zero value disables caching (Dir == "").
type Entry ¶
type Entry struct {
// contains filtered or unexported fields
}
Entry is a pinned, ready cache entry. It holds a hard pin: eviction cannot remove it while the handle is alive, which is what makes "never evict an entry a request is streaming from" a property rather than a hope.
Exactly one of Reader, RangeReader or Release must be called. Reader and RangeReader hand the pin to the returned ReadCloser, so closing the body releases it.
func (*Entry) Path ¶
Path is the file on disk. Callers outside this package should prefer Reader/RangeReader; Path exists for tests and for surfaces that need to hand a filename to something else.
func (*Entry) RangeReader ¶
func (e *Entry) RangeReader(off, length int64) (io.ReadCloser, error)
RangeReader returns a window, with exactly storage.RangeReader's contract: off >= 0; length < 0 means "to the end"; length == 0 is an immediate EOF; an offset at or past EOF is EOF, not an error; a length past EOF is clamped. The pin is released when the returned body is closed.
func (*Entry) Reader ¶
func (e *Entry) Reader() (io.ReadCloser, error)
Reader returns the whole body. The pin is released when it is closed.
type Fill ¶
type Fill struct {
// contains filtered or unexported fields
}
Fill tracks one in-flight (or just-finished) fetch so concurrent callers deduplicate onto it and can watch its progress.
func (*Fill) Percent ¶
Percent is 0..99 while fetching. 100 is deliberately not reachable here: it belongs to a finished file on disk, so a poller only ever sees 100 once the bytes are actually servable. (Same rule as sharezip.Gen.)
Source Files