Documentation
¶
Overview ¶
Package pool is the surface's provider seam: the model-switchable client a slot holds, the per-model clients a pinned job is served by, and the billing that makes every structuring call land on the same rail its leaves land on.
It sits beside internal/provider rather than inside it because the thing it switches is a configured client: it reads config.Config to build one, hands back router.Client so a panel keeps its rungs, and journals store.NodeUsage so the day's total is honest. All three of those packages already depend on internal/provider, so the seam that composes them cannot live there — a sub-package is the same shelf without the cycle.
Index ¶
- Constants
- Variables
- func CauseInWords(err error) string
- func CloseReplaced(client router.Client)
- func SpendNode(ctx context.Context) string
- func TurnEnd(ctx context.Context, response *ai.Response) *store.EndedPart
- func WithSpendNode(ctx context.Context, nodeID string) context.Context
- type Client
- func (l *Client) CallWall() time.Duration
- func (l *Client) Close()
- func (l *Client) CompleteWithMessages(ctx context.Context, messages []ai.Message, options ...ai.Option) (*ai.Response, error)
- func (l *Client) Escalatable() bool
- func (l *Client) Model() string
- func (l *Client) Routed() bool
- func (l *Client) SetModel(model string) error
- func (l *Client) Snapshot() (string, router.Client)
- func (l *Client) WithCallWall(wall time.Duration) *Client
- func (l *Client) WithUsageJournal(journal func(store.NodeUsage)) *Client
- type Pool
Constants ¶
const DefaultCallWall = 4 * time.Minute
DefaultCallWall is how long one structuring completion may take before the system stops waiting on it.
FOUR MINUTES IS A MEASURED FIGURE NOW, AND IT IS SHOWN TO THE MODEL. It used to be argued as "deliberately far past honest", on production rows where the slowest structuring call was under a minute. Issue #927 measured otherwise on a reasoning model with a long request: an honest compile of 225 seconds and spine passes of 53 to 165, beside a grounding pass that thought for the whole four minutes on one machine and answered in two seconds on another. So the wall is no longer only a line past which a completion is gone. It is the time the completion is TOLD it has: every walled request carries it to the effort ladder, which derives the thinking budget the model is given from it (provider's provider.WithThinkingWall), and a completion that still reaches it with thought on the wire is asked for its answer rather than thrown away ([walled.CompleteWithMessages]).
The unit matters. This bounds ONE completion, not one command and not one agent loop: a head turn that makes nine tool calls gets nine fresh walls, and a leaf that thinks for an hour across forty round-trips is never touched by it. Bounding the call rather than the caller is what lets the number be small enough to catch a hang without ever cutting honest work in half.
The incident it exists for: a contract call on a chat splice never returned. There was no deadline anywhere on the path — not on the context, not on the client, not on the transport for a streamed request — so the reconciler's command queue stopped forever behind one open socket while the process went on looking alive.
const LongestCall = completionsPerCall * DefaultCallWall
LongestCall is the longest one call through a slot walled at DefaultCallWall can honestly take: the completion it was asked for, cut at its wall, and the one ask for the answer that completion's thought had reached, under a wall of its own. It is exported for the reconciler's command rail, which is counted in these (internal/resident's commandWall) so that the two figures cannot drift.
const RanOutOfTime = "the model thought past its time"
RanOutOfTime is the cause, spelled ONCE for every sentence that carries it.
Issue #927's receipt said "the model stopped answering", which was false — every first token had arrived in under a second. The same falsehood has two other spellings on the structuring road, and they are worse because they are machinery: a planning stage that ends on a clock reaches the person as `context deadline exceeded`, inside "the plan for task-2 was drawn with faults (size stage 1: context deadline exceeded)". A person cannot act on that sentence and it does not say what happened. CauseInWords is the one door every such sentence goes through, and this is the one phrase it uses.
Variables ¶
var ErrCallWall = fmt.Errorf("%s: %w", RanOutOfTime, context.DeadlineExceeded)
ErrCallWall is what a call that outlived its wall returns. It is an ordinary provider failure by design: every caller on the structuring path already has an error branch, and this arrives on it rather than inventing a new one. The layer above owns the retry — this layer refuses to wait forever and keeps what was thought ([walled.CompleteWithMessages]), and a call that returns this is one whose answer ask ran out of time as well.
THE SENTENCE IS THE CAUSE. Silence never reaches this wall: a stream that stops writing is cut by the stream guard's own silence bounds long before four minutes, with a sentence of its own. What reaches it is a model still WORKING — in every row issue #927 measured, still thinking — so "stopped answering", which this said until then, was the one account of the event that was false.
It wraps context.DeadlineExceeded deliberately. A stall has to be legible to the reconciler's watchdog, which decides between striking a command and failing it, and the alternative was for internal/resident to import this package for one sentinel. Wrapping the standard error instead means the watchdog asks the only question it actually has — "did this die of time?" — with errors.Is.
Functions ¶
func CauseInWords ¶
CauseInWords is an error as a person should read it: whatever it says about where it happened, with a clock's own vocabulary replaced by the cause.
IT KEEPS THE PLACE AND REPLACES THE JARGON. "size stage 1: context deadline exceeded" becomes "size stage 1: the model thought past its time", because which pass ran out is information the person's next decision uses, while `context deadline exceeded` is this program's internals leaking. An error that did not die of time is handed back exactly as it came: this composer renames one cause and invents nothing.
func CloseReplaced ¶
CloseReplaced releases a client that has just been swapped out.
A router is not a value: it owns the append handle on router-events.jsonl and a queue of graded observations the run has already paid for. Every model switch used to drop one on the floor, which leaks the handle for the life of the process and loses whatever had not reached the ledger file yet. Closing is best-effort and idempotent; a plain adapter has nothing to close and is left alone.
func TurnEnd ¶
TurnEnd reads how one completion ended, or nil when it ended on its own terms. It is the truncation law's read side for every pool caller — the compiler, the delivery gate, the revision sentinel, the narrator, the head — and it is a plain function over the value they already hold rather than state on the client, because a client is shared by concurrent callers and "the last finish reason" on a shared object is a race wearing a field name.
A non-nil result belongs on the message that call produced, as store.EndedMark(*end). Dropping it is how a 600-token cap became a diagram that stopped mid-path and a journal that said nothing about it.
Types ¶
type Client ¶
type Client struct {
// contains filtered or unexported fields
}
Client is a model-switchable completion client. Long-running leaves take a snapshot so one measurement has one model; structuring consumers hold this handle directly, so a swap takes effect on their next call.
It is also the one seam every structuring call in this surface passes through — the head's routing loop, the compiler, the delivery gate, the revision sentinel, the narrator, the distiller. None of that spend reached the journal: the daily rail is summed from the usage table and nowhere else, so it systematically understated the bill by the entire cost of thinking about the work. The headless path has journaled its own preparation spend since it existed; chat is what forgot. Billing here rather than at each of a dozen call sites is what makes it hard to forget again.
func Adopt ¶
Adopt wraps a client somebody else already built. It is the same object New returns and the door for a caller that holds its own transport — a pinned pool entry, a scripted provider in a test — so there is exactly one shape of switchable client in the system rather than two.
func New ¶
New builds the ordinary client: one slot, one model, resolved through the settings the process was started with.
func (*Client) Close ¶
func (l *Client) Close()
Close releases the underlying router client so its ledger flushes and its events handle is returned before the process exits.
func (*Client) CompleteWithMessages ¶
func (l *Client) CompleteWithMessages(ctx context.Context, messages []ai.Message, options ...ai.Option) (*ai.Response, error)
CompleteWithMessages is the one seam every structuring call passes through, which makes it the one place two facts about a turn are both in hand: what it cost, and how it ended. The cost is journaled here because a dozen call sites would otherwise each have to remember to. How it ended is NOT journaled here, and the asymmetry is deliberate: spend belongs to the day's rail no matter who spent it, while an unfinished turn belongs to the message that turn produced — and this seam does not know which message that is, or whether there will be one. So the end mark rides back out on the response, and TurnEnd below is how a caller reads it in one line at the moment it posts.
func (*Client) Escalatable ¶
Escalatable reports whether a failed leaf has somewhere stronger to go — the same condition the headless runner uses to grant one escalation.
func (*Client) Routed ¶
Routed reports whether a panel is behind this slot, which is what decides whether a structured-output schema has a second rung to unlock.
func (*Client) Snapshot ¶
Snapshot returns a model and client from the same instant, which keeps the profile key and the executor it describes inseparable.
The client it hands back carries this slot's call wall. That is deliberate and it is the whole reason the wall works: the expensive structuring callers — plan.Build, plan.Contracts, the JIT expander — do not hold this handle, they take a snapshot once and call it for the rest of the pass. A wall that lived only on the method below would have bounded every call except the ones that hung.
func (*Client) WithCallWall ¶
WithCallWall sets the per-completion wall for every call served through this slot, including the ones a caller makes on a snapshot it took earlier.
It is opt-in per slot, and the split is the whole safety argument. The structuring slots — the one that talks and the one that plans — are single completions whose only honest duration is short, so they are walled. The work slot is not: a leaf is an agent loop bounded by the executor's own deadline, and a wall here would be a second, dumber governor over work that is legitimately allowed to take hours.
func (*Client) WithUsageJournal ¶
WithUsageJournal wires the durable rail. It is set after the graph opens rather than at construction because the clients exist first; until it is set a client simply does not bill, which is the old behaviour and the right one for a client that has no store to bill to.
type Pool ¶
type Pool struct {
// contains filtered or unexported fields
}
Pool keeps per-message chat overrides pinned to the exact model recorded on the durable user message. A later work-model change therefore cannot race an armed submission that the head has not tailed yet.
func (*Pool) Adopt ¶
Adopt pins a client somebody else built under an exact model slug. The pool builds its own on a miss; this is the door for a caller that already holds the transport it wants that slug served by.
func (*Pool) Close ¶
func (p *Pool) Close()
Close releases every pinned per-model client the pool has handed out.
Source Files