Documentation
¶
Overview ¶
Package gc implements the mark-and-sweep collector of architecture/mark-sweep-gc.md, a port of Mic92's bitmap GC (draganm/amber-store#9): a cycle marks every key reachable from the references' roots into a packstore.MarkSet — one bit per sealed record, slotted by the packs' own footer indexes — and sweeps by rewriting the packs whose dead ratio crosses the line (packstore.Compact). Nothing is persisted between cycles: no closure files, no union, no refcounts. A write barrier (packstore.BeginBarrier) keeps ingests running while the mark walks; reference publication and the sweep serialize on the collector's reference lock. Wire protocol and pack formats are unchanged.
Index ¶
- Constants
- Variables
- type Collector
- func (c *Collector) BeginSpan() (*Span, error)
- func (c *Collector) BeginWrite() (done func())
- func (c *Collector) Close() error
- func (c *Collector) PrepareRef(root key.Key) (commit, abort func(), err error)
- func (c *Collector) ReleaseRef(root key.Key) error
- func (c *Collector) Run(ctx context.Context, garbage float64) (CycleStats, error)
- func (c *Collector) RunWithCopyBudget(ctx context.Context, garbage float64, maxCopyBytes uint64) (CycleStats, error)
- func (c *Collector) Status(ctx context.Context) (Status, error)
- func (c *Collector) UnreachableFrom(ctx context.Context, roots, candidates []key.Key) ([]key.Key, error)
- func (c *Collector) Why(k key.Key) ([]string, error)
- func (c *Collector) Wipe(reset func() error) error
- type CycleStats
- type Options
- type PackStatus
- type Span
- type Status
Constants ¶
const ( // DefaultGrace is the minimum age of a sealed pack before it can be // reaped. DefaultGrace = time.Hour // DefaultGarbage is the selection line: an eligible pack with at // least this fraction of garbage is reaped. DefaultGarbage = 0.5 )
Variables ¶
var ErrCycleRunning = errors.New("gc: a cycle is already running")
ErrCycleRunning reports an overlapping Run; cycles never overlap.
Functions ¶
This section is empty.
Types ¶
type Collector ¶
type Collector struct {
// contains filtered or unexported fields
}
Collector implements the cycle and the reference hooks over an open packstore and refstore pair. A process has one per pair; other processes may run their own on the same store (architecture/mark-sweep-gc.md, "Across processes"). Close it before the stores.
func Open ¶
func Open(dir string, objects *packstore.Store, refs *refstore.Store, opts Options) (*Collector, error)
Open opens the collector next to an already-open packstore and refstore. dir is the layout slot the simple-gc collector kept closure files in (<store>/closures): it is created empty and any leftover closure state from a previous collector is swept — closures were derived data. With opts.Interval > 0 a goroutine runs a cycle per interval until Close.
func (*Collector) BeginSpan ¶ added in v0.0.10
BeginSpan opens a write span. End it when its writes are durable and its references are stored.
func (*Collector) BeginWrite ¶ added in v0.0.6
func (c *Collector) BeginWrite() (done func())
BeginWrite gates one object-write span (an ingest, a pull, an inbox drain) against the sweep. The span holds the reference lock shared: the sweep waits out in-flight writes, and a write stalls while a sweep runs — never during the mark, which writers pass behind the write barrier. Without the gate a dedup hit against a record in a condemned pack could report success and then lose the record to the pack's removal (packstore.Compact must not overlap ingests). Returns the release, idempotent; call it when the span's writes are durable.
func (*Collector) Close ¶
Close stops the background loop and waits out a running cycle. Close the collector before the stores under it.
func (*Collector) PrepareRef ¶
PrepareRef readies a reference PUT naming root under the reference lock, held shared until commit or abort: the tree is walked for completeness (a missing object aborts with the error naming it — the caller's 404) and the walked closure is handed to the write barrier, so a PUT landing while a mark runs cannot lose its objects to the sweep. Exactly one of commit (after the reference record is stored) or abort (it was not) must be called. Release of an old root needs no bookkeeping; ReleaseRef exists for symmetry.
func (*Collector) ReleaseRef ¶
ReleaseRef records that one reference naming root was deleted or overwritten. The mark-and-sweep collector keeps no per-root state, so this is a no-op: the next cycle simply no longer marks from the root.
func (*Collector) Run ¶
Run marks from every reference root and sweeps the eligible packs above the line: garbage >= 0 forces that line, garbage < 0 uses policy — 0.5, or 0.1 under min-free pressure. Ingests and reads run through the mark (the write barrier keeps them); reference publication and the sweep stall each other on the reference lock.
func (*Collector) RunWithCopyBudget ¶ added in v0.0.12
func (c *Collector) RunWithCopyBudget(ctx context.Context, garbage float64, maxCopyBytes uint64) (CycleStats, error)
RunWithCopyBudget is Run with a cap on the live record bytes the sweep copies, for a collector that has only so much room to copy into. It sets packstore.CompactOpts.MaxCopyBytes, which says what the cap counts. A pack that needs more than is left of the cap stays for a later cycle. At 0 the cycle copies nothing, seals nothing, and still reaps the packs with nothing live in them.
func (*Collector) Status ¶
Status marks from the current references and scores every sealed pack — a full mark walk, the cost of keeping no persistent liveness state.
func (*Collector) UnreachableFrom ¶ added in v0.0.12
func (c *Collector) UnreachableFrom(ctx context.Context, roots, candidates []key.Key) ([]key.Key, error)
UnreachableFrom returns the candidates that roots does not reach, in the order they were given. The mark starts from roots alone: a published reference is not a root here unless the caller passes its key. Advisory: the walk takes no barrier and no reference lock, so the caller must keep roots, and anything published since, alive while it acts on the answer. Deletes nothing.
type CycleStats ¶
type CycleStats struct {
Start time.Time
Duration time.Duration
MarkDuration time.Duration // roots snapshot + mark walk
SweepDuration time.Duration // Compact: select, copy, delete
Threshold float64
Marked int // distinct live objects marked
Scored int // sealed packs considered
Reaped []uint64 // victim ids, ascending
CopiedRecords int
CopiedBytes int64
FreedBytes int64 // victim file bytes freed on disk, footers included
}
CycleStats describes one cycle.
type Options ¶
type Options struct {
Grace time.Duration // pack eligibility age
Garbage float64 // reap packs with at least this fraction of garbage
MinFree uint64 // free-space floor in bytes; 0 = 5 % of the filesystem
Rate int64 // copier bandwidth cap in bytes/s; 0 = unlimited
Interval time.Duration // time between background cycles; 0 = none
Jobs int // PrepareRef walk parallelism; <= 0 = GOMAXPROCS
}
Options configure a Collector. The zero value means: 1 h grace, 0.5 garbage line, min-free at 5 % of the filesystem, unlimited copy rate, no background cycles, GOMAXPROCS walk parallelism.
type PackStatus ¶
type PackStatus struct {
ID uint64
Sealed time.Time
Body int64 // record bytes
Keys uint64
Live int64 // Σ (46 + slen) over marked entries
Garbage float64
Eligible bool // sealed before the horizon (now − grace)
}
PackStatus is one sealed pack's score against a mark.
type Span ¶ added in v0.0.10
type Span struct {
// contains filtered or unexported fields
}
Span is an open write span: object writes and the reference puts that name them, as one unit against the sweep, in this process and in every other. A sweep here or elsewhere waits for it to end, and it waits for one. Inside it references are prepared with Span.PrepareRef, which takes no lock again: Collector.PrepareRef inside a span would take the reference lock a second time, and wait for ever behind a cycle that is waiting for the span.
BeginSpan takes the reference lock and then the store's gate, the order a cycle takes them in. A span opened on the store itself (packstore.Store.BeginWrite) and held across Collector.PrepareRef takes them the other way round; in a process that runs cycles the two can wait for each other.
func (*Span) End ¶ added in v0.0.10
func (sp *Span) End()
End closes the span. It may be called more than once.
func (*Span) PrepareRef ¶ added in v0.0.10
PrepareRef readies a reference PUT naming root, as Collector.PrepareRef does, under the locks the span already holds. The reference may be stored any time before End; commit and abort are there for the shape and do nothing.
type Status ¶
type Status struct {
Packs []PackStatus
LiveBytes int64
GarbageBytes int64
Refs int // reference names
Marked int // distinct live objects marked
Last *CycleStats
LastError string
}
Status is the gc report: per-pack scores against a fresh advisory mark, totals, the last cycle. The mark runs without quiescing writers, so concurrent churn can skew the numbers; a cycle's own mark is exact.
Source Files