remotecache

package
v1.0.0-beta.27 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Nov 10, 2026 License: Apache-2.0 Imports: 26 Imported by: 0

Documentation

Overview

Package remotecache connects an engine to the remote cache service. Run keeps one WebSocket open to the service, answers its requests through the engine's Adapter, and asks it for fresh download addresses when a read of an offered part needs one. The engine serves from its first second whether or not the connection is up, and Run never waits on the service to return.

Index

Constants

View Source
const (
	// ExportsInFlight is how many exports that select outputs the engine
	// works on at once.
	ExportsInFlight = 4
	// RecordsExportsInFlight is how many more exports the engine works on at
	// once, beside ExportsInFlight, that select no outputs. Such an export
	// opens no chain and diffs no layer, so it never waits behind the
	// others' layer builds, and it may also take a slot of ExportsInFlight
	// that none of those holds.
	RecordsExportsInFlight = 1
	// UploadConcurrency is how many blobs the engine uploads at once, across
	// all its exports. Most blobs are small, so a PUT's time is mostly the
	// store's per-request latency, not bytes. 16 is measured: on test-base,
	// fresh engines' runs were faster with 16 than with 8 in all four blocks,
	// by 41 to 198 s over 8 runs, with no measured cost to the seeding
	// engine's check time, memory peak or exit time.
	UploadConcurrency = 16
	// LookupsInFlight is how many lookups the engine has unanswered at once.
	LookupsInFlight = 4
)

The engine side's limits.

Variables

This section is empty.

Functions

func Run

func Run(ctx context.Context, adapter Adapter, cfg Config) error

Run keeps the engine connected to the service until ctx is cancelled, and then returns nil promptly: it closes the connection without waiting for the service, cancels uploads, and releases the chains held for exports.

Types

type Adapter

type Adapter interface {
	// CacheIdentity returns the engine cache's identity and generation.
	CacheIdentity() (cacheID string, generation uint64)
	// Export captures the requested roots and calls consume exactly once,
	// while the captured chains stay open. The export's Bundle is nil when no
	// root survived. The chains are released when consume returns. known are
	// the rows the service merged over this connection: an export sends
	// those as held rows.
	Export(ctx context.Context, req protocol.Export, known *dagql.KnownRows, consume func(context.Context, Export) error) error
	// Merge merges a bundle of the service's values into the cache. id is
	// the service's request ID, for logs. known are the rows the service
	// merged over this connection: the rows a committed merge lands on join
	// them, since the service holds the records it sent.
	Merge(ctx context.Context, id uint64, req protocol.Merge, known *dagql.KnownRows) (protocol.Merged, error)
	// OfferParts places offered parts on the cache's entries by number.
	OfferParts(ctx context.Context, req protocol.Offer) (protocol.Offered, error)
	// TakeRenewalRequest waits for the next read that needs fresh download
	// addresses; ReplyRenewal answers it.
	TakeRenewalRequest(ctx context.Context) (*dagql.RenewalRequest, error)
	ReplyRenewal(dagql.RenewalReply) dagql.RenewalReplyDisposition
	// StartLookups starts queuing the cache's lookup misses for this
	// connection, with nothing asked yet, and StopLookups stops it.
	StartLookups()
	StopLookups()
	// TakeLookupMisses waits for the cache's next misses, at most limit,
	// lingering up to linger after the first for more. dropped counts the
	// misses a full queue refused since the last take.
	TakeLookupMisses(ctx context.Context, limit int, linger time.Duration) (misses []dagql.RemoteMiss, dropped uint64, err error)
	// SettleState returns the engine's side of settling with the service,
	// and a channel closed when it next changes.
	SettleState() (SettleState, <-chan struct{})
	// Settled records the service's Settled for a settle of period that
	// listed sessions.
	Settled(period uint64, sessions []protocol.SettleSession)
}

Adapter is the engine's side of the service's requests: its cache operations by entry number, and the renewal mailbox.

type BlobSource

type BlobSource interface {
	ReadBlob(ctx context.Context, dgst digest.Digest) (io.ReadCloser, int64, error)
}

BlobSource reads a layer blob of an export's chains.

type Config

type Config struct {
	// URL is the service's base URL; Run dials protocol.EnginePath under it.
	URL string
	// Token is the engine token, sent as the Basic username.
	Token string
	// EngineVersion and EngineName are sent in hello.
	EngineVersion string
	EngineName    string
	// Logger receives the connection's warnings and errors. Nil discards.
	Logger *slog.Logger
	// Lookups asks the service for the calls the engine's cache misses. The
	// engine always sets it; a test that serves no lookups leaves it off.
	Lookups bool
}

Config configures Run.

type Export

type Export struct {
	// Bundle is nil when no root survived.
	Bundle  *dagql.ValueBundle
	Skipped []protocol.SkippedRoot
	// Blobs reads the bundle's layer blobs for upload.
	Blobs BlobSource
	// Rows are the rows the bundle carries in full and as held rows, for
	// known once the service says whether it merged them.
	Rows dagql.ExportedRows
}

Export is one capture, as Adapter.Export hands it to its consumer.

type SettleState

type SettleState struct {
	// Period is the engine's current idle period (protocol.Open).
	Period uint64
	// Live is set while a session is live.
	Live bool
	// Sessions are the ended sessions no Settled has covered.
	Sessions []protocol.SettleSession
	// Settled is set once a Settled covered Period.
	Settled bool
	// TimedOut is set once the engine gave up waiting to settle after its
	// last session ended, until a session starts again.
	TimedOut bool
	// Claims are the sessions that started under a Dagger Cloud claim,
	// sorted by session ID (protocol.Claims): those live, and those ended
	// that no Settled has covered.
	Claims []protocol.SessionClaim
}

SettleState is where the engine stands in settling with the service.

Directories

Path Synopsis
Package pack is the format of a remote-cache pack: one stored object that holds several small layer blobs, so that they upload, download and are addressed as one.
Package pack is the format of a remote-cache pack: one stored object that holds several small layer blobs, so that they upload, download and are addressed as one.
Package protocol defines the messages an engine and the remote cache service exchange over one WebSocket.
Package protocol defines the messages an engine and the remote cache service exchange over one WebSocket.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL