Documentation
¶
Overview ¶
Package dispatcher holds the dispatcher's application state: the registry of the executors that currently hold a control session, admission of submitted runs against reserved capacity, and the results those runs report.
An executor is in the registry as an entry keyed by its ID, holding the metadata of its session and the transport owner that session belongs to. The owner is the authority: an entry is exposed, and work is admitted for it, only while its owner is active, registered and inside its lease, never because an executor reported a heartbeat. The transport publishes an owner and calls registration under the setup operation it admitted for it; publication into the registry is committed under the owner's own guard, so a retirement racing it wins. Package transport/rpc defines owners, mutations and their lifetime.
Callbacks from the control transport arrive with the mutation they were admitted under, and that mutation's owner is compared against the executor's current owner before any effect of it is applied. Work this package starts itself, uploading or aborting a run on an executor, admits its own mutation on that executor's owner, and a continuation that has to outlive the request which started it is forked from the live mutation rather than detached from it.
Close closes admission first and then cancels and joins what this package owns: the registrations in flight, the lease expiry goroutine and the control transport. Cancelling is not completing — a canceled call is still waited for.
Index ¶
- Constants
- Variables
- func AdmissionPaused() error
- func PublicTerminalError(message string) string
- func ReceiptKeyID(public ed25519.PublicKey) string
- type AccountQuotaError
- type CapabilityError
- type ControlMetrics
- type DestinationPolicy
- type DestinationPolicyChange
- type Dispatcher
- func (d *Dispatcher) AbortDebuglet(ctx context.Context, executorID string, debugletID uuid.UUID, reason string) error
- func (d *Dispatcher) AllowAdmissionRequest(ctx context.Context, owner *uuid.UUID) error
- func (d *Dispatcher) Cancellation(ctx context.Context, id uuid.UUID) (wire.Cancellation, error)
- func (d *Dispatcher) CheckExecutorCapabilities(executorID string, orderID int64, policy models.DebugletPolicy) error
- func (d *Dispatcher) Close()
- func (d *Dispatcher) CollectMetrics(ctx context.Context) ControlMetrics
- func (d *Dispatcher) ConfigureAttribution(cfg config.AttributionConfig) error
- func (d *Dispatcher) ConfigureDataLimits(admission config.AdmissionConfig, retention config.RetentionConfig) error
- func (d *Dispatcher) ConfigureEgress(ctx context.Context, cfg config.EgressConfig) error
- func (d *Dispatcher) ConfigureExecutorDisplay(executors map[string]config.ExecutorDisplay) error
- func (d *Dispatcher) ConfigureIPMetadata(databases *ipmetadata.Databases) error
- func (d *Dispatcher) ConfigureOutputLimits(limits config.OutputConfig) error
- func (d *Dispatcher) ConfiguredDisplay(id string) wire.ExecutorDisplay
- func (d *Dispatcher) ControlIncarnation() string
- func (d *Dispatcher) ControlLeaseDuration() time.Duration
- func (d *Dispatcher) DeleteCompletedPayload(ctx context.Context, id uuid.UUID, expired bool) error
- func (d *Dispatcher) DestinationLimit(destination string) bitrate.Bitrate
- func (d *Dispatcher) ExecutorEligibility() (eligible, registered int)
- func (d *Dispatcher) GetExecutor(id string) (*RegisteredExecutor, bool)
- func (d *Dispatcher) GetExecutorByIPFull(ip string) (RegisteredExecutor, bool)
- func (d *Dispatcher) GetKeyStore() *tag.KeyStore
- func (d *Dispatcher) GetVersion() string
- func (d *Dispatcher) ListDestinationPolicies(ctx context.Context) ([]DestinationPolicy, error)
- func (d *Dispatcher) ListExecutors() []RegisteredExecutor
- func (d *Dispatcher) LiveProbeBook() *ProbeBook
- func (d *Dispatcher) OnDebugletAllocate(ctx context.Context, mutation *rpc.Mutation, req *pb.DebugletAllocateRequest) (*pb.DebugletAllocateResponse, error)
- func (d *Dispatcher) OnDebugletExit(ctx context.Context, mutation *rpc.Mutation, req *pb.DebugletExitRequest) (*pb.DebugletExitResponse, error)
- func (d *Dispatcher) OnDebugletState(ctx context.Context, mutation *rpc.Mutation, req *pb.DebugletStateRequest) (*pb.DebugletStateResponse, error)
- func (d *Dispatcher) OnDebugletStream(owner *rpc.SessionOwner, ...) error
- func (d *Dispatcher) OnExecutorConnected(ctx context.Context, owner *rpc.SessionOwner, hello *pb.HelloResponse, ...) error
- func (d *Dispatcher) OnExecutorDisconnected(owner *rpc.SessionOwner)
- func (d *Dispatcher) OnExperimentReady(ctx context.Context, mutation *rpc.Mutation, req *pb.ExperimentReadyRequest) (*pb.ExperimentReadyResponse, error)
- func (d *Dispatcher) OnHeartbeat(ctx context.Context, mutation *rpc.Mutation, req *pb.HeartbeatRequest) (*pb.HeartbeatResponse, error)
- func (d *Dispatcher) OnReflectAddress(ctx context.Context, mutation *rpc.Mutation, req *pb.ReflectAddressRequest, ...) (*pb.ReflectAddressResponse, error)
- func (d *Dispatcher) OnResources(ctx context.Context, mutation *rpc.Mutation, req *pb.ResourcesRequest) (*pb.ResourcesResponse, error)
- func (d *Dispatcher) OpenReceiptSigner(ctx context.Context, stateDirectory string) (*ReceiptSigner, error)
- func (d *Dispatcher) ProbeBook(ctx context.Context) (*ProbeBook, error)
- func (d *Dispatcher) PruneAttribution(ctx context.Context, now time.Time) error
- func (d *Dispatcher) ReceiptKeys(ctx context.Context) ([]ReceiptKey, error)
- func (d *Dispatcher) Recovery(ctx context.Context, id uuid.UUID) (wire.Recovery, error)
- func (d *Dispatcher) RegisterExecutor(ctx context.Context, owner *rpc.SessionOwner, hello *pb.HelloResponse, ...) error
- func (d *Dispatcher) RestoreScheduler(ctx context.Context) error
- func (d *Dispatcher) SetDestinationLimit(destination string, limit bitrate.Bitrate) error
- func (d *Dispatcher) SetDestinationPolicy(ctx context.Context, actor, destination string, change DestinationPolicyChange) error
- func (d *Dispatcher) SubmitDebuglets(ctx context.Context, specs []models.DebugletSpec, userID *uuid.UUID) (uuid.UUIDs, error)
- func (d *Dispatcher) UpdateOwnerDisplay(ctx context.Context, params database.UpdateOwnedExecutorDisplayParams) error
- func (d *Dispatcher) VerifyAttribution(ctx context.Context, packets []VerifyPacket) ([]VerifyGroupResult, error)
- type ExecutorHealthMetrics
- type ExecutorResourceMetric
- type MaintenanceSwitch
- type ProbeBook
- type ReceiptKey
- type ReceiptSigner
- type RegisteredExecutor
- func (e *RegisteredExecutor) Addressing(private bool) wire.ProbeAddressing
- func (e *RegisteredExecutor) Admission(paused bool) string
- func (e *RegisteredExecutor) AppendDebugletID(id uuid.UUID)
- func (e *RegisteredExecutor) CapabilityObservation() *wire.CapabilityObservation
- func (e *RegisteredExecutor) Clock() wire.ObservedClock
- func (e *RegisteredExecutor) Connectivity(private bool) *wire.Connectivity
- func (e *RegisteredExecutor) Display() wire.ExecutorDisplay
- func (e *RegisteredExecutor) IPMetadata() *wire.IPMetadata
- func (e *RegisteredExecutor) PublicHost() string
- func (e *RegisteredExecutor) RecentDebugletIDs(n int) []uuid.UUID
- func (e *RegisteredExecutor) Vantage() (wire.ObservedString, wire.ObservedList)
- type RetainedRunMetrics
- type VerifyBudgetState
- type VerifyGroupResult
- type VerifyPacket
Constants ¶
const ( // VerifyBudget is R, the candidate trials per executor, chain and epoch: // a query charges one trial for every candidate run the executor tests, // since each tests a guessed tag against another derived key. VerifyBudget = 16 // MaxVerifyPackets bounds the packets of one request. MaxVerifyPackets = 256 // MaxVerifyGroups bounds both the source/epoch groups and result subsets. MaxVerifyGroups = 16 )
Limits of server-assisted verification (docs/verification.md#limits).
const ( VerifyVerified = "verified" VerifyInvalid = "invalid" VerifyPending = "pending" VerifyMissing = "missing" VerifyUnsupported = "unsupported" VerifyMethodOffline = "offline" VerifyMethodServer = "server" // VerifyBudgetExhausted: the trials left for the epoch are fewer than // the group's candidates. VerifyBudgetExhausted = "budget_exhausted" // VerifyUnmatched: the executor found no single candidate that // reproduces every tag. Two runs from one source in one epoch answer so // as well as forged traffic, so the offline check after disclosure // decides, per run. VerifyUnmatched = "unmatched" VerifyExecutorUnavailable = "executor_unavailable" )
Group verdicts, methods and the reasons this file adds to the client's.
const ( DestinationPolicyLimit = "limit" DestinationPolicyDeny = "deny" DestinationPolicyAllow = "allow" )
Kinds of destination policy events.
const ( // SystemActor is recorded for changes the dispatcher makes itself, such // as restoring the default when a policy expires. SystemActor = "system" // MaxDestinationPolicyReason bounds the recorded reason, in bytes. MaxDestinationPolicyReason = 500 // MaxDestinationLength bounds a destination a policy is recorded for. MaxDestinationLength = 255 )
const MaintenanceFileEnv = "DEBUGLET_MAINTENANCE_FILE"
MaintenanceFileEnv names the environment variable that points the dispatcher at its maintenance switch. A dispatcher started without it has no switch and performs no file access per submission at all.
Variables ¶
var ( // ErrInvalidDestinationPolicy refuses a change before anything is // recorded or applied. ErrInvalidDestinationPolicy = errors.New("invalid destination policy") // ErrDestinationPolicyNotRecorded is a change whose event was not // committed; nothing of it was applied or sent, and the previous // policy stays authoritative. ErrDestinationPolicyNotRecorded = errors.New("destination policy was not recorded") // ErrDenialUnsupported is a recorded deny that an executor holding an // allocation acknowledged without being able to revoke its traffic. ErrDenialUnsupported = errors.New("executor upgrade required to revoke denied destinations") )
var ( ErrSessionRetired = errors.New("executor session retired") ErrDispatcherClosed = errors.New("dispatcher closed") )
var ErrAbortRefused = errors.New("executor refused the cancellation")
ErrAbortRefused marks a cancellation its executor answered with a refusal, as distinct from a cancellation that did not reach it.
var ErrAccountQuota = errors.New("account admission quota exceeded")
var ErrAdmissionCommitted = errors.New("new admission committed")
ErrAdmissionCommitted identifies an upload failure from the call that committed the new runs. Receipt lookup and pre-admission refusals cannot authorize the admitted-batch refund path.
var ErrCancellationNotRecorded = errors.New("cancellation acknowledged but its result was not recorded")
ErrCancellationNotRecorded marks a cancellation its executor acknowledged but whose terminal result the dispatcher failed to record. It is no refusal: the run may already be stopping, while its stored state does not show the cancellation.
var ErrEgressBudget = fmt.Errorf("aggregate egress budget unavailable: %w", resource.ErrCapacityFull)
var ErrInvalidPolicy = errors.New("invalid policy")
ErrInvalidPolicy marks a submission the dispatcher refuses because of the numbers it carries, rather than because of anything the dispatcher is or does. It is a rejection of the request, so a caller can answer it as one: the HTTP boundary reports it as an invalid policy and not as a failure of the server.
var ErrMaintenanceMode = errors.New("dispatcher is in maintenance: submission admission is stopped")
ErrMaintenanceMode reports a dispatcher whose operator has stopped submission admission. It is a refusal to accept new work, not a failure: nothing was reserved, persisted or charged.
var ErrNoReceiptKey = errors.New("no receipt key: attribution.receipt_key_path is not set and the database directory is unknown")
ErrNoReceiptKey reports a dispatcher that has no place for its receipt key: neither attribution.receipt_key_path nor a database directory is known.
var ErrOrderedBandwidthUnsupported = errors.New("executor upgrade required for ordered destination updates")
var ErrOutputCapacity = errors.New("output storage capacity exhausted")
var ErrPayloadNotDeletable = errors.New("payload requires completed output and confirmed executor retirement")
var ErrPaymentInUse = errors.New("payment order already has admitted runs")
ErrPaymentInUse marks a submission refused where the transaction's orders already carry runs, or where the dispatcher could not tell whether they do. Those runs may be executing or already credited to their executor, so the caller must not refund the transaction on such a refusal.
var ErrTooManyVerifyGroups = errors.New("the packets form more than 16 groups or run subsets")
ErrTooManyVerifyGroups reports a request whose packets form more than MaxVerifyGroups groups.
var ErrUnknownExecutor = errors.New("executor is not registered")
ErrUnknownExecutor marks a submission for an executor that is not registered, or no longer available, when the batch is admitted. It is a rejection of the request, not a failure of the dispatcher.
Functions ¶
func AdmissionPaused ¶
func AdmissionPaused() error
AdmissionPaused reports whether this dispatcher may admit new submissions. A transport checks it before it looks up or charges anything, so a refusal costs a submitter nothing; the submission path checks it again, because that is where the decision has to hold.
func PublicTerminalError ¶ added in v0.3.0
PublicTerminalError is the public projection of a retained terminal string, including historical reports. The database retains the original private diagnostic until payload deletion. Empty remains success; every nonempty report remains a failure. Arbitrary executor or runtime text is never an authority to publish a stack, credential, address or module fragment.
func ReceiptKeyID ¶ added in v0.3.0
ReceiptKeyID is the key ID of an Ed25519 public key.
Types ¶
type AccountQuotaError ¶ added in v0.3.0
AccountQuotaError identifies a finite admission limit. RetryAfter is supplied only for the replenishing request budget; work capacity requires retirement, so a clock-based promise would be misleading.
func (*AccountQuotaError) Error ¶ added in v0.3.0
func (e *AccountQuotaError) Error() string
func (*AccountQuotaError) Is ¶ added in v0.3.0
func (e *AccountQuotaError) Is(target error) bool
type CapabilityError ¶ added in v0.3.0
CapabilityError identifies the requested policy field that current discovery evidence cannot serve. It remains an invalid-policy error for older clients.
func (*CapabilityError) Error ¶ added in v0.3.0
func (e *CapabilityError) Error() string
func (*CapabilityError) Unwrap ¶ added in v0.3.0
func (e *CapabilityError) Unwrap() error
type ControlMetrics ¶ added in v0.3.0
type ControlMetrics struct {
ObservedAt time.Time
Registered, Ready int
ReadyCapacityBitsPerSecond float64
Health ExecutorHealthMetrics
Runs RetainedRunMetrics
Settlement payments.SettlementMetrics
}
ControlMetrics is an observation, not a lifetime event counter. Runs describes retained durable rows; pruning changes its counts. Registry and storage are read separately, so a transition concurrent with collection may appear on the next scrape. No run, executor, account or destination identifiers are exported.
type DestinationPolicy ¶ added in v0.3.0
type DestinationPolicy struct {
Destination string
Kind string
// Limit is nil when the destination has the default capacity.
Limit *bitrate.Bitrate
Reason string
Actor string
SetAt time.Time
ExpiresAt *time.Time
Revision int64
// Recipients is the number of executors that held an allocation on the
// destination when the policy was applied in this dispatcher lifetime,
// and Unconfirmed the number of them that did not acknowledge it.
Recipients int
Unconfirmed int
}
DestinationPolicy is the current policy of a destination: its latest recorded event, and how far the latest application of it was delivered.
type DestinationPolicyChange ¶ added in v0.3.0
type DestinationPolicyChange struct {
Limit bitrate.Bitrate
Denied bool
Reason string
ExpiresAt time.Time
}
DestinationPolicyChange is the complete policy an operator sets for a destination. A denied destination admits nothing, so its limit is not recorded; lifting the deny sets the limit of that change. A zero ExpiresAt never expires.
type Dispatcher ¶
type Dispatcher struct {
Bidi *rpc.BidiServer
Payment *payments.PaymentHandler
// contains filtered or unexported fields
}
func (*Dispatcher) AbortDebuglet ¶
func (d *Dispatcher) AbortDebuglet(ctx context.Context, executorID string, debugletID uuid.UUID, reason string) error
AbortDebuglet asks the run's executor to cancel it and records the cancellation as the run's terminal result. A nil error means that result is recorded, or that the run was already terminal and keeps its own. An error wrapping ErrCancellationNotRecorded means the executor acknowledged the cancellation but its result is not recorded. Other errors do not confirm that the cancellation was recorded.
A run whose control session has ended, because the dispatcher restarted or the executor registered again in a new session, has no executor to ask: its cancellation is recorded locally, see cancelUnbound.
func (*Dispatcher) AllowAdmissionRequest ¶ added in v0.3.0
AllowAdmissionRequest uses the authenticated account, never a request body or address. Only intent/submission routes call it; cancellation, reads, logout and executor control remain available at exhaustion. Restart replenishes this short-lived token bucket, but never the durable accepted-work reservations.
func (*Dispatcher) Cancellation ¶ added in v0.3.0
func (d *Dispatcher) Cancellation(ctx context.Context, id uuid.UUID) (wire.Cancellation, error)
Cancellation is a read-only snapshot. It never retries delivery, repairs a terminal result, or infers remote success from a locally cancelled run.
func (*Dispatcher) CheckExecutorCapabilities ¶ added in v0.3.0
func (d *Dispatcher) CheckExecutorCapabilities(executorID string, orderID int64, policy models.DebugletPolicy) error
CheckExecutorCapabilities applies the capability check of admission to an order that is not being admitted: it reports whether the available executor can serve what policy requires of it now, with the refusal admission would return. It reads the registry only. An executor that is not registered or not available is ErrUnknownExecutor.
func (*Dispatcher) Close ¶
func (d *Dispatcher) Close()
Close closes admission before it cancels and joins owned work. Resource Close and cancellation hooks never run under either shared map lock.
func (*Dispatcher) CollectMetrics ¶ added in v0.3.0
func (d *Dispatcher) CollectMetrics(ctx context.Context) ControlMetrics
CollectMetrics reads at most 10,001 run rows and retains no per-run data. An oversized or unreadable history is unavailable, never a partial count. The two-second query context bounds storage work; registry locks are released before waiting for the database connection or reading any rows.
func (*Dispatcher) ConfigureAttribution ¶ added in v0.3.0
func (d *Dispatcher) ConfigureAttribution(cfg config.AttributionConfig) error
ConfigureAttribution fixes the attribution history retention before startup.
func (*Dispatcher) ConfigureDataLimits ¶ added in v0.3.0
func (d *Dispatcher) ConfigureDataLimits(admission config.AdmissionConfig, retention config.RetentionConfig) error
ConfigureDataLimits is called before startup, as ConfigureOutputLimits is.
func (*Dispatcher) ConfigureEgress ¶ added in v0.3.0
func (d *Dispatcher) ConfigureEgress(ctx context.Context, cfg config.EgressConfig) error
ConfigureEgress refuses policy changes while previously issued grants remain usable. Restarting cannot advertise a lower bound than outstanding authority.
func (*Dispatcher) ConfigureExecutorDisplay ¶ added in v0.3.0
func (d *Dispatcher) ConfigureExecutorDisplay(executors map[string]config.ExecutorDisplay) error
ConfigureExecutorDisplay fixes operator display metadata before registration starts; each registration copies its executor's entry.
func (*Dispatcher) ConfigureIPMetadata ¶ added in v0.3.0
func (d *Dispatcher) ConfigureIPMetadata(databases *ipmetadata.Databases) error
ConfigureIPMetadata sets the offline databases before registration. They may switch to a replaced file while the dispatcher runs (Databases.Reload); each registration's lookups are a snapshot of the database in use then. The caller owns the databases and closes them only after dispatcher shutdown.
func (*Dispatcher) ConfigureOutputLimits ¶ added in v0.3.0
func (d *Dispatcher) ConfigureOutputLimits(limits config.OutputConfig) error
ConfigureOutputLimits fixes output storage limits before registration starts.
func (*Dispatcher) ConfiguredDisplay ¶ added in v0.3.0
func (d *Dispatcher) ConfiguredDisplay(id string) wire.ExecutorDisplay
ConfiguredDisplay is the operator's metadata for an executor ID, registered or not.
func (*Dispatcher) ControlIncarnation ¶
func (d *Dispatcher) ControlIncarnation() string
ControlIncarnation is the nonsecret identity of this dispatcher lifetime. A transport installed in place of the one New built carries it unchanged.
func (*Dispatcher) ControlLeaseDuration ¶
func (d *Dispatcher) ControlLeaseDuration() time.Duration
ControlLeaseDuration is the lease policy of this dispatcher lifetime, retained with the incarnation by any such replacement.
func (*Dispatcher) DeleteCompletedPayload ¶ added in v0.3.0
DeleteCompletedPayload preserves durable ownership/accounting/verification references. Authorization belongs to the HTTP boundary. Repeated deletion succeeds without releasing storage twice.
func (*Dispatcher) DestinationLimit ¶ added in v0.3.0
func (d *Dispatcher) DestinationLimit(destination string) bitrate.Bitrate
DestinationLimit is the capacity admission currently uses for a destination.
func (*Dispatcher) ExecutorEligibility ¶
func (d *Dispatcher) ExecutorEligibility() (eligible, registered int)
ExecutorEligibility counts the registry and the subset of it that may be given work now, which is the entries whose owner is available. It copies no record and reads no database.
func (*Dispatcher) GetExecutor ¶
func (d *Dispatcher) GetExecutor(id string) (*RegisteredExecutor, bool)
func (*Dispatcher) GetExecutorByIPFull ¶
func (d *Dispatcher) GetExecutorByIPFull(ip string) (RegisteredExecutor, bool)
GetExecutorByIPFull selects the latest local observation, with a stable ID tie-break, and returns metadata independent of subsequent registry changes.
func (*Dispatcher) GetKeyStore ¶
func (d *Dispatcher) GetKeyStore() *tag.KeyStore
func (*Dispatcher) GetVersion ¶
func (d *Dispatcher) GetVersion() string
func (*Dispatcher) ListDestinationPolicies ¶ added in v0.3.0
func (d *Dispatcher) ListDestinationPolicies(ctx context.Context) ([]DestinationPolicy, error)
ListDestinationPolicies returns the current policy of every destination that has one recorded, ordered by destination.
func (*Dispatcher) ListExecutors ¶
func (d *Dispatcher) ListExecutors() []RegisteredExecutor
func (*Dispatcher) LiveProbeBook ¶ added in v0.3.0
func (d *Dispatcher) LiveProbeBook() *ProbeBook
LiveProbeBook is a book without the history: every listed executor is connected since its registration and nothing else is known.
func (*Dispatcher) OnDebugletAllocate ¶
func (d *Dispatcher) OnDebugletAllocate(ctx context.Context, mutation *rpc.Mutation, req *pb.DebugletAllocateRequest) (*pb.DebugletAllocateResponse, error)
OnDebugletAllocate records the admitted run's idempotent allocation, then waits for delivery to every affected recipient. A failed delivery is returned to the executor and retried on the recipient's next heartbeat. The caller owns run failure/terminal reporting; a repeated allocation never charges twice.
func (*Dispatcher) OnDebugletExit ¶
func (d *Dispatcher) OnDebugletExit(ctx context.Context, mutation *rpc.Mutation, req *pb.DebugletExitRequest) (*pb.DebugletExitResponse, error)
OnDebugletExit selects one immutable terminal result. Only its winner handles payment and fairshare notification. Winner and duplicates release the run's resources by run identity, so a duplicate completes a release an earlier delivery did not reach without repeating payment or subtracting another run's reservation.
func (*Dispatcher) OnDebugletState ¶
func (d *Dispatcher) OnDebugletState(ctx context.Context, mutation *rpc.Mutation, req *pb.DebugletStateRequest) (*pb.DebugletStateResponse, error)
func (*Dispatcher) OnDebugletStream ¶
func (d *Dispatcher) OnDebugletStream(owner *rpc.SessionOwner, stream grpc.BidiStreamingServer[pb.DebugletStreamRequest, pb.DebugletStreamResponse]) error
func (*Dispatcher) OnExecutorConnected ¶
func (d *Dispatcher) OnExecutorConnected(ctx context.Context, owner *rpc.SessionOwner, hello *pb.HelloResponse, sourceIP string) error
func (*Dispatcher) OnExecutorDisconnected ¶
func (d *Dispatcher) OnExecutorDisconnected(owner *rpc.SessionOwner)
func (*Dispatcher) OnExperimentReady ¶ added in v0.3.0
func (d *Dispatcher) OnExperimentReady(ctx context.Context, mutation *rpc.Mutation, req *pb.ExperimentReadyRequest) (*pb.ExperimentReadyResponse, error)
OnExperimentReady performs one bounded poll. SQLite serializes readiness, release and cancellation; a committed release is immutable across restarts.
func (*Dispatcher) OnHeartbeat ¶
func (d *Dispatcher) OnHeartbeat(ctx context.Context, mutation *rpc.Mutation, req *pb.HeartbeatRequest) (*pb.HeartbeatResponse, error)
func (*Dispatcher) OnReflectAddress ¶ added in v0.3.0
func (d *Dispatcher) OnReflectAddress(ctx context.Context, mutation *rpc.Mutation, req *pb.ReflectAddressRequest, address string) (*pb.ReflectAddressResponse, error)
OnReflectAddress records only the authenticated transport's actual peer IP. Two slots, one per family, bound storage and tie proof to the current owner.
func (*Dispatcher) OnResources ¶
func (d *Dispatcher) OnResources(ctx context.Context, mutation *rpc.Mutation, req *pb.ResourcesRequest) (*pb.ResourcesResponse, error)
func (*Dispatcher) OpenReceiptSigner ¶ added in v0.3.0
func (d *Dispatcher) OpenReceiptSigner(ctx context.Context, stateDirectory string) (*ReceiptSigner, error)
OpenReceiptSigner loads the receipt key, creating it when the file is absent, and records it as the current key: a key seen for the first time becomes valid now, and every other recorded key stops being valid now. stateDirectory is the database directory, where the key is kept when attribution.receipt_key_path is empty.
func (*Dispatcher) ProbeBook ¶ added in v0.3.0
func (d *Dispatcher) ProbeBook(ctx context.Context) (*ProbeBook, error)
ProbeBook reads the status history. Without a database every listed executor is connected since its registration and nothing else is known.
func (*Dispatcher) PruneAttribution ¶ added in v0.3.0
PruneAttribution deletes the attribution history older than the configured retention before now and advances retained_from to the cutoff: runs whose interval ended before it, keys whose epoch ended before it and chains with neither left that were last seen before it.
func (*Dispatcher) ReceiptKeys ¶ added in v0.3.0
func (d *Dispatcher) ReceiptKeys(ctx context.Context) ([]ReceiptKey, error)
ReceiptKeys lists every recorded receipt key, oldest first.
func (*Dispatcher) Recovery ¶ added in v0.3.0
Recovery reads a known run and makes at most one retained-run lookup against the captured current executor. It changes neither that run nor its charges.
func (*Dispatcher) RegisterExecutor ¶
func (d *Dispatcher) RegisterExecutor(ctx context.Context, owner *rpc.SessionOwner, hello *pb.HelloResponse, sourceIP string) error
RegisterExecutor stages metadata for an already-published transport owner. ctx belongs to the setup mutation the transport admitted on that owner, so retirement cancels it and a replacement session waits for this call to return. Registration admits no mutation of its own.
func (*Dispatcher) RestoreScheduler ¶
func (d *Dispatcher) RestoreScheduler(ctx context.Context) error
RestoreScheduler reserves again, when the dispatcher starts, the floors of the stored runs whose window has not ended, so that admission counts them as the previous dispatcher did. A run whose stored state is exited reserves nothing: every resource a terminal release frees is held in memory only, so a restart leaves nothing of it to release. Every other run, pending or of uncertain outcome, keeps its reservation until its window ends. A run whose window ended less than expiredWindowGrace ago is logged and not reserved. Release uses run identity, so a skipped reservation cannot subtract another run's floor in overlapping rounded buckets.
A restored run bound to a previous dispatcher lifetime is logged as a warning with its ID: its control session ended with that lifetime, so it cannot be replayed, and its reservation lasts until the run is cancelled or its local allocation is reclaimed after its window ends. A run without a complete stored binding cannot be cancelled or reclaimed; its reservation lasts until its window ends.
Reservations are restored once per dispatcher lifetime: after a restore has succeeded, a further call reserves nothing and returns an error, so no run is counted twice. A call that fails has reserved nothing and may be repeated. A successful restore also starts the expiry loop if no executor registration has started it yet, so a restarted dispatcher reclaims ended windows before any executor connects.
func (*Dispatcher) SetDestinationLimit ¶
func (d *Dispatcher) SetDestinationLimit(destination string, limit bitrate.Bitrate) error
SetDestinationLimit sets the limit of a destination, lifting a deny, as a change of the dispatcher itself. See SetDestinationPolicy.
func (*Dispatcher) SetDestinationPolicy ¶ added in v0.3.0
func (d *Dispatcher) SetDestinationPolicy(ctx context.Context, actor, destination string, change DestinationPolicyChange) error
SetDestinationPolicy records change as the policy of destination, applies it to admission and sends the recomputed shares to every executor holding an allocation there, waiting up to five seconds for those deliveries.
The event is recorded before anything is applied: a refused or failed write changes nothing. A limit below the floors already charged or reserved on the destination is refused with resource.ErrCapacityFull; a deny is never refused for floors, since it admits nothing new and the admitted runs keep their allocations until they end. Once recorded, the policy governs admission whether or not every executor acknowledged; the error joins the failed deliveries, and the delivery outcome is kept for ListDestinationPolicies.
func (*Dispatcher) SubmitDebuglets ¶
func (d *Dispatcher) SubmitDebuglets(ctx context.Context, specs []models.DebugletSpec, userID *uuid.UUID) (uuid.UUIDs, error)
func (*Dispatcher) UpdateOwnerDisplay ¶ added in v0.3.0
func (d *Dispatcher) UpdateOwnerDisplay(ctx context.Context, params database.UpdateOwnedExecutorDisplayParams) error
UpdateOwnerDisplay serializes edits without holding the registry lock during storage I/O. The update returns its committed fields, so cancellation after storage returns cannot prevent publishing them to discovery. Existing result snapshots stay fixed.
func (*Dispatcher) VerifyAttribution ¶ added in v0.3.0
func (d *Dispatcher) VerifyAttribution(ctx context.Context, packets []VerifyPacket) ([]VerifyGroupResult, error)
VerifyAttribution forms the groups of packets and decides each of them.
type ExecutorHealthMetrics ¶ added in v0.3.0
type ExecutorHealthMetrics struct {
EBPF, Fallback, EnforcementUnknown int
AttachmentPresent, AttachmentMissing, AttachmentUnknown, AttachmentNotRequired int
RSS, FDs, StateAvailable, StateCapacity, StateAvailableRatio ExecutorResourceMetric
AttributionAvailable int
EpochZero int
ChainExhausted int
RefreshFailing int
DisclosureHeld int
ClockUnready int
ClockDrift int
Untagged int
AttributionUnknown int
DisclosureHeldSeconds float64
ClockReady, ClockDegraded, ClockUnknown int
ClockEstimateUnknown int
ClockEstimatedErrorSeconds *float64
ScheduleUnknown, ScheduleExpired int
ScheduleRemainingSeconds *float64
// DisclosureLag is the maximum disclosure delivery lag in seconds.
DisclosureLag ExecutorResourceMetric
DisclosureCompletion ExecutorResourceMetric
RefusedAdmissions, RevokedSockets ExecutorResourceMetric
DropVerdicts, DropSKBBytes [2]ExecutorResourceMetric // Ingress, egress; current counter instance.
}
ExecutorHealthMetrics aggregates the registry's validated reports. Missing, expired or disconnected observations count as unknown, not healthy. Counter mode is the executor's selection; it does not prove continuous enforcement.
type ExecutorResourceMetric ¶ added in v0.3.0
A numeric aggregate is publishable only when all registered observations are known. Unknown counts distinguish incomplete observations from a real zero.
type MaintenanceSwitch ¶
type MaintenanceSwitch struct {
SchemaVersion int `json:"schema_version"`
Paused bool `json:"paused"`
Reason string `json:"reason,omitempty"`
Since string `json:"since,omitempty"`
}
MaintenanceSwitch is the record the switch file carries.
type ProbeBook ¶ added in v0.3.0
type ProbeBook struct {
// contains filtered or unexported fields
}
ProbeBook is one read of the durable status history, for one listing.
func (*ProbeBook) Live ¶ added in v0.3.0
func (b *ProbeBook) Live(e *RegisteredExecutor) wire.ProbeStatus
Live is the status and tags of a registered executor.
func (*ProbeBook) Offline ¶ added in v0.3.0
func (b *ProbeBook) Offline(live map[string]bool, statuses []string, display func(string) wire.ExecutorDisplay, private bool) []wire.Executor
Offline lists the known executors that are not registered and whose status is in statuses, sorted by ID. live names the registered ones; display is the operator's metadata; private reveals a private executor's addresses.
type ReceiptKey ¶ added in v0.3.0
type ReceiptKey struct {
KeyID string
PublicKey ed25519.PublicKey
ValidFrom time.Time
ValidTo *time.Time
}
ReceiptKey is one recorded receipt key; ValidTo is nil while it is current.
type ReceiptSigner ¶ added in v0.3.0
type ReceiptSigner struct {
// contains filtered or unexported fields
}
ReceiptSigner signs verification receipts. It never exposes the private key.
func (*ReceiptSigner) KeyID ¶ added in v0.3.0
func (s *ReceiptSigner) KeyID() string
KeyID names the signing key: the hex of the first 16 bytes of the SHA-256 of its public key.
func (*ReceiptSigner) PublicKey ¶ added in v0.3.0
func (s *ReceiptSigner) PublicKey() ed25519.PublicKey
PublicKey is the key that verifies the signatures.
func (*ReceiptSigner) Sign ¶ added in v0.3.0
func (s *ReceiptSigner) Sign(payload []byte) []byte
Sign returns the detached signature of payload.
type RegisteredExecutor ¶
type RegisteredExecutor struct {
ID string
Version string
Ready bool
LastSeen time.Time
TeslaDelay time.Duration // the epoch length I (historical name)
TeslaAnchorTimestamp time.Time
TeslaAnchorKey []byte // k_0, the public chain anchor
// TeslaDisclosureDelay is d in epochs: the key of epoch i is disclosed
// from the start of epoch i+d. Zero means the executor predates it and
// disclosed after one epoch.
TeslaDisclosureDelay int64
// TeslaChainLength is L, the number of epochs the chain serves; zero is
// an executor that predates reporting it.
TeslaChainLength int64
// TeslaTagSpec is the tag specification version of the chain, from the
// executor's hello capability report: tag.TagSpecV1 when it reports
// debuglet-tag-v1, tag.TagSpecLegacy otherwise, including executors that
// predate the report.
TeslaTagSpec int64
ICMPEnabled bool
Capabilities *wire.ExecutorCapabilities
PricePerBwS int64
Currency string
SuiWallet string
// contains filtered or unexported fields
}
RegisteredExecutor represents a registered executor and its metadata.
func (*RegisteredExecutor) Addressing ¶ added in v0.3.0
func (e *RegisteredExecutor) Addressing(private bool) wire.ProbeAddressing
Addressing is the executor's RIPE Atlas-style addressing for the listing. Public callers see only globally routable addresses of public executors. Established operators also see non-global and opted-out observations.
func (*RegisteredExecutor) Admission ¶ added in v0.3.0
func (e *RegisteredExecutor) Admission(paused bool) string
Admission summarises a listed executor for selection: offline until its current session has sent a heartbeat, then maintenance while the operator's switch stops admission. Normal admission still decides each submission.
func (*RegisteredExecutor) AppendDebugletID ¶
func (e *RegisteredExecutor) AppendDebugletID(id uuid.UUID)
AppendDebugletID adds id to the executor's history, trimming old entries so the total length stays within 2× the maximum to bound memory usage.
func (*RegisteredExecutor) CapabilityObservation ¶ added in v0.3.0
func (e *RegisteredExecutor) CapabilityObservation() *wire.CapabilityObservation
func (*RegisteredExecutor) Clock ¶ added in v0.3.0
func (e *RegisteredExecutor) Clock() wire.ObservedClock
Clock returns the live clock observation; unknown on an expired snapshot.
func (*RegisteredExecutor) Connectivity ¶ added in v0.3.0
func (e *RegisteredExecutor) Connectivity(private bool) *wire.Connectivity
Connectivity returns detached, explicitly stale observations. The API uses private=false except for an established operator; results retain full facts.
func (*RegisteredExecutor) Display ¶ added in v0.3.0
func (e *RegisteredExecutor) Display() wire.ExecutorDisplay
Display labels each field with its source. Dispatcher operator labels take precedence over owner labels. Neither source is verified against the host.
func (*RegisteredExecutor) IPMetadata ¶ added in v0.3.0
func (e *RegisteredExecutor) IPMetadata() *wire.IPMetadata
IPMetadata is independent of the registry's immutable observation. Raw addresses remain private; the result's existing source_ip/public_host carry those lookup keys for its owner.
func (*RegisteredExecutor) PublicHost ¶
func (e *RegisteredExecutor) PublicHost() string
PublicHost returns the executor's public host (IP or domain) at which debuglet listeners can be contacted, or "" if none is configured.
func (*RegisteredExecutor) RecentDebugletIDs ¶
func (e *RegisteredExecutor) RecentDebugletIDs(n int) []uuid.UUID
RecentDebugletIDs returns up to n recent debuglet IDs for this executor, newest first. If n ≤ 0 the default (lastDebugletHistory) is used.
func (*RegisteredExecutor) Vantage ¶ added in v0.3.0
func (e *RegisteredExecutor) Vantage() (wire.ObservedString, wire.ObservedList)
Vantage returns the live SCION ISD-AS and listener observations. Called on a snapshot, whose expired report is already cleared, both are then unknown.
type RetainedRunMetrics ¶ added in v0.3.0
type VerifyBudgetState ¶ added in v0.3.0
VerifyBudgetState is the budget of one executor, chain and epoch, in candidate trials.
type VerifyGroupResult ¶ added in v0.3.0
type VerifyGroupResult struct {
Source, ChainID, ExecutorID, RunID string
Epoch int64
Verdict, Reason, Method string
Packets []int
Budget *VerifyBudgetState
PendingUntil *time.Time
}
VerifyGroupResult is the verdict of one group. Packets are indices into the request. Budget is set for a group that has a chain, PendingUntil for a pending group that the disclosure of its key decides.
type VerifyPacket ¶ added in v0.3.0
VerifyPacket is one packet of a verification request.
Source Files
¶
- account_limits.go
- attribution.go
- attribution_verify.go
- cancellation.go
- capabilities.go
- capability_admission.go
- connectivity.go
- debuglet_service.go
- destination_policy.go
- dispatcher.go
- egress.go
- executor_registry.go
- experiments.go
- ip_metadata.go
- maintenance.go
- measurement_execution.go
- measurement_request.go
- metrics.go
- metrics_health.go
- output_store.go
- payload_retention.go
- probe_addresses.go
- probe_status.go
- public_terminal.go
- receipt_keys.go
- recovery.go
- resources.go
- result_provenance.go
- rpc_handlers.go
- settlement_sweep.go
- terminal_cleanup.go
- vantage.go
- window_expiry.go
Directories
¶
| Path | Synopsis |
|---|---|
|
Package enrollment binds an executor ID to the node credential its machine holds: the SHA-256 fingerprint of a verified client certificate.
|
Package enrollment binds an executor ID to the node credential its machine holds: the SHA-256 fingerprint of a verified client certificate. |
|
transport
|
|
|
rpc
Package rpc serves the dispatcher's side of the executor control plane and owns one session per connected executor.
|
Package rpc serves the dispatcher's side of the executor control plane and owns one session per connected executor. |