Documentation
¶
Overview ¶
Package sharezip caches the ZIP archive produced when a shared FOLDER is downloaded ("download all"). Without a cache, every download re-walks the folder and re-reads + re-compresses every file from object storage — slow for large folders (e.g. a receipt month with hundreds of images). The cache is keyed by node id + a content signature (file set + sizes + mtimes), so any change to the folder invalidates it and the next request (or the background warmer) regenerates.
A small in-memory generation registry deduplicates concurrent builds: if a zip for a given signature is already being generated (by another download or by the warmer), other callers attach to the same job and can watch its progress instead of kicking off a second build.
Nothing outlives its share ¶
A cached archive exists to serve one folder share. When that share expires, is revoked or runs out of downloads, the archive is garbage — regenerable garbage — and Sweep deletes it on the warmer's next pass. A build in flight for a share that dies mid-build stops and takes its partial file with it (shareGone). Both read the same view of active shares, ActiveShares, so "active" means one thing in this package.
Index ¶
- Constants
- Variables
- type ActiveShares
- type Cache
- func (c *Cache) Cached(cachePath string) (os.FileInfo, bool)
- func (c *Cache) Enabled() bool
- func (c *Cache) Plan(ctx context.Context, drv storage.Driver, root string, nodeID int64) (string, []File, error)
- func (c *Cache) StartOrGet(cachePath string, files []File, nodeID int64, drv storage.Driver) *Gen
- func (c *Cache) Sweep() (removed int, freed int64)
- func (c *Cache) Track(a *ActiveShares)
- func (c *Cache) Warm(ctx context.Context, drv storage.Driver, root string, nodeID int64) (bool, error)
- type DirShare
- type File
- type Gen
- type Warmer
Constants ¶
const ( // DefaultWarmMaxBytes bounds what the background warmer pre-builds // (2 GiB). On-demand builds are not bounded by it. DefaultWarmMaxBytes = int64(2) << 30 // DefaultMaxAge is how long a cached archive may sit on disk before the // sweeper removes it regardless of whether its share is still live. DefaultMaxAge = 7 * 24 * time.Hour )
DefaultShareRefreshInterval is how often the view of active folder shares is refreshed.
⚠ It is refreshed by its own loop, not by the warm pass, and this is the whole point: a warm pass can sit inside a single multi-hour build, and a running build asks this view whether to give up. Refreshing only between passes would mean never refreshing during the one build that most needs it — exactly the build that read a 16.7 GB folder for three hours for a share that had already expired.
const DefaultWarmInterval = 5 * time.Minute
DefaultWarmInterval is how often the background warmer re-checks folder shares for changes.
Variables ¶
ErrShareGone ends a build whose folder share stopped existing while it ran. Callers treat it as "no cached archive" — there is nobody left to serve.
var ErrTooLargeToWarm = errors.New("sharezip: folder is over the warm ceiling, left for on-demand build")
ErrTooLargeToWarm is Warm's answer for a folder over WarmMaxBytes: nothing was built and nothing will be until somebody actually asks for the archive.
Functions ¶
This section is empty.
Types ¶
type ActiveShares ¶
type ActiveShares struct {
// contains filtered or unexported fields
}
ActiveShares is the single answer to "which nodes have a live folder share right now". Everything that needs that answer reads it here: the warmer (what to pre-build), the sweeper (what may be deleted) and a running build (whether to abandon itself). One list function, one definition of active.
func NewActiveShares ¶
func NewActiveShares(list func(context.Context) ([]DirShare, error)) *ActiveShares
NewActiveShares returns a view backed by list. A nil list yields nil, and every reader treats that as "I know nothing" rather than "nothing is active".
func (*ActiveShares) Refresh ¶
func (a *ActiveShares) Refresh(ctx context.Context) ([]DirShare, error)
Refresh re-lists the active folder shares and replaces the snapshot. On error the previous snapshot is kept: a database hiccup must not read as "every share is gone", which would abandon live builds and sweep live archives.
type Cache ¶
type Cache struct {
// WarmMaxBytes is the largest folder (sum of file sizes) Warm will
// pre-build; 0 = no ceiling. It bounds SPECULATIVE work only — the
// on-demand path (StartOrGet, what a download click takes) ignores it,
// so no folder is ever refused, it just is not built before anyone asks.
// Set by the server from FILEX_SHAREZIP_WARM_MAX_BYTES; New defaults it
// to DefaultWarmMaxBytes.
WarmMaxBytes int64
// MaxAge is how old a published archive may get before Sweep removes it
// even though its share is still live; 0 = never. A regenerable file
// nobody has asked for in a week is disk, not cache. Set by the server
// from FILEX_SHAREZIP_MAX_AGE; New defaults it to DefaultMaxAge.
MaxAge time.Duration
// contains filtered or unexported fields
}
Cache is a local-disk cache of folder-share ZIPs plus a generation registry. The zero value is unusable; construct with New.
func New ¶
New returns a Cache rooted at dir. An empty dir disables caching (Enabled reports false and callers fall back to streaming).
func (*Cache) Plan ¶
func (c *Cache) Plan(ctx context.Context, drv storage.Driver, root string, nodeID int64) (string, []File, error)
Plan walks the folder and returns the cache path it would occupy plus the file list (no generation, no bytes read). The cache path is <dir>/<nodeID>-<sig>.zip.
func (*Cache) StartOrGet ¶
StartOrGet returns a generation for cachePath, starting one if none is running. If the archive already exists on disk it returns an instantly-finished generation (so Wait returns immediately).
func (*Cache) Sweep ¶
Sweep deletes cached archives that no live folder share can serve, and the wreckage of builds that never finished. It is the answer to a cache that only ever grew: pruneOld drops OLD SIGNATURES OF A NODE THAT IS STILL SHARED, so when a share expires its archive was simply immortal — one such file was 15 GB, was never downloaded once, and rode into three backups.
It deletes exactly three things:
- <nodeID>-<sig>.zip whose node has no active folder share. "Active" is not re-defined here: it is whatever the shared ActiveShares view says, which is the same query the warmer builds from. Two definitions of "active" in two places is how a sweeper eventually deletes something that was still being served.
- <nodeID>-<sig>.zip older than MaxAge, live share or not (MaxAge > 0). It is regenerable — the warmer's next pass rebuilds it if the folder is under the warm ceiling, a download click rebuilds it otherwise.
- .tmp-*.zip older than tmpMaxAge that no running build claims.
It refuses to touch: any name that is not one of those two shapes (a cache directory sharing a disk with something else loses nothing), directories, the archive of any node that still has a share (whatever its signature — that is pruneOld's job), any file a build is writing or about to publish, and everything at all until a share listing has actually succeeded once. A file it cannot delete (Windows will not unlink a file a download is streaming) is left for the next pass.
func (*Cache) Track ¶
func (c *Cache) Track(a *ActiveShares)
Track wires the cache to the view of active folder shares. Both things that need to know whether a share is alive read it from here: the sweeper (what may be deleted) and a running build (whether to give up). NewWarmer calls this; a cache with no warmer keeps its pre-sweeper behaviour.
func (*Cache) Warm ¶
func (c *Cache) Warm(ctx context.Context, drv storage.Driver, root string, nodeID int64) (bool, error)
Warm ensures a fresh cached archive exists for a folder, generating it if missing. Blocks until the (possibly already-running) build completes. The bool reports whether a (re)generation was needed (false = cache already fresh). Used by the background warmer. A no-op when caching is disabled.
A folder whose files add up to more than WarmMaxBytes is NOT built here: Warm returns ErrTooLargeToWarm and leaves the archive to the on-demand path (StartOrGet), which a visitor's download click still drives to completion with the progress page. The ceiling is about not spending hours of object storage reads on a link nobody may ever open — the 16.7 GB incident — not about refusing anything.
type DirShare ¶
type DirShare struct {
}
DirShare is the minimal view of an active folder share the warmer needs. The caller supplies these via a list function so this package stays decoupled from the db/model layers.
type File ¶
type File struct {
Path string // driver path (for Read)
Rel string // path inside the zip
Size int64
Mtime time.Time
}
File is one file collected from a folder walk (metadata only — no bytes read yet). Used to compute the cache signature and, on a miss, to build the zip.
func CollectFiles ¶
CollectFiles walks root and returns every file under it (metadata only). Internal dirs (trash, thumbnails, keepdir) are skipped so the archive matches what the streaming path would produce.
Exported because the UNCACHED folder-share download (api/handlers/share.go) needs the same walk, and had its own copy of it. Two walks is how one of them quietly starts including `.filex-trash` after somebody adds a fourth internal directory to the other.
type Gen ¶
type Gen struct {
Total int
// contains filtered or unexported fields
}
Gen tracks one in-flight (or just-finished) zip build so concurrent callers can dedup and watch progress.
type Warmer ¶
type Warmer struct {
// contains filtered or unexported fields
}
Warmer periodically pre-generates (or refreshes) the cached ZIP for every active folder share, so a downloader almost always hits a warm cache instead of waiting for an on-demand build. A changed folder is detected cheaply via the content signature — an unchanged folder costs one metadata listing and no re-compression.
It is also where the cache is garbage-collected: each pass sweeps the archives of shares that no longer exist (see Cache.Sweep).
func NewWarmer ¶
func NewWarmer(cache *Cache, list func(ctx context.Context) ([]DirShare, error), resolver func(int64) (storage.Driver, error), interval time.Duration, logf func(string, ...any)) *Warmer
NewWarmer builds a Warmer. interval<=0 uses DefaultWarmInterval; logf==nil discards logs. It also wires the cache to the share view it builds, so the sweeper and any running build judge "active" by this same list function.
Source Files