Documentation
¶
Index ¶
Constants ¶
const DefaultProviderPropagationWait = 15 * time.Second
DefaultProviderPropagationWait is how long to wait after creating or updating the mock LLM provider before starting chats. Provider config is cached per coderd replica with a 10 second TTL (see coderd/x/chatd/configcache.go), and a change is only guaranteed to be visible everywhere once every replica's cached entry has expired. 15 seconds comfortably exceeds that TTL.
Variables ¶
This section is empty.
Functions ¶
func EnsureScaletestChatModel ¶ added in v2.37.0
func EnsureScaletestChatModel(ctx context.Context, client *codersdk.Client, logger slog.Logger, llmMockURL string, propagationWait time.Duration) (func(organizationID uuid.UUID) (uuid.UUID, error), error)
EnsureScaletestChatModel bootstraps the shared AI provider used by chat scaletests. It returns a function that creates or reuses the ChatModel for each organization. When the provider changes, it waits for cached provider configurations to expire before chats start.
Types ¶
type Config ¶
type Config struct {
// OrganizationID is the organization that owns the target workspace.
OrganizationID uuid.UUID `json:"organization_id"`
// WorkspaceID is the pre-existing workspace to use for this chat run.
// When empty, the chat runs without workspace context.
WorkspaceID uuid.UUID `json:"workspace_id"`
// Prompt is the text content sent on every turn.
Prompt string `json:"prompt"`
// ModelConfigID is the scaletest mock LLM model config.
ModelConfigID uuid.UUID `json:"model_config_id"`
// Turns is the total number of user to assistant exchanges per chat.
// Must be at least 1.
Turns int `json:"turns"`
// TurnStartDelay is the shared delay between every runner completing
// its initial turn and the release of the follow-up turns. Set
// to 0 to send all turns without an inter-phase pause.
TurnStartDelay time.Duration `json:"turn_start_delay"`
// TurnStartReadyWaitGroup coordinates the gap between the initial turn
// finishing and the follow-up turns. Each runner signals exactly
// once after its first turn reaches a terminal status, or when it
// knows it will never reach that point.
TurnStartReadyWaitGroup *sync.WaitGroup `json:"-"`
// StartTurnsChan blocks follow-up turns until the CLI layer releases them.
StartTurnsChan chan struct{} `json:"-"`
Metrics *Metrics `json:"-"`
}
Config describes a single chat runner within a scaletest invocation.
type Metrics ¶
type Metrics struct {
ChatCreateLatencySeconds prometheus.Histogram
ChatMessageLatencySeconds *prometheus.HistogramVec
ChatConversationDurationSeconds prometheus.Histogram
ChatTimeToRunningSeconds *prometheus.HistogramVec
ChatTimeToFirstOutputSeconds *prometheus.HistogramVec
ChatTimeToTerminalStatusSeconds *prometheus.HistogramVec
ChatStageFailuresTotal *prometheus.CounterVec
ChatTerminalStatusTotal *prometheus.CounterVec
ChatTurnsCompletedTotal prometheus.Counter
ChatRetryEventsTotal prometheus.Counter
ActiveChatStreams prometheus.Gauge
}
Metrics holds the Prometheus metrics emitted by the chat scaletest.
func NewMetrics ¶
func NewMetrics(reg prometheus.Registerer) *Metrics