Documentation
¶
Index ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type RateLimiter ¶
type RateLimiter struct {
Limiter *rate.Limiter
Semaphore *semaphore.Weighted
// contains filtered or unexported fields
}
RateLimiter holds the rate limiter and semaphore
func NewRateLimiter ¶
func NewRateLimiter(config RateLimiterConfig) *RateLimiter
NewRateLimiter creates a new RateLimiter based on the config
func (*RateLimiter) StartInstanceCountUpdater ¶
func (rl *RateLimiter) StartInstanceCountUpdater(appLabel string, ticker *time.Ticker, cfg *RateLimiterConfig)
StartInstanceCountUpdater starts a goroutine to update the rate limiter's limits based on the instance count
func (*RateLimiter) UpdateLimits ¶
func (rl *RateLimiter) UpdateLimits(cfg *RateLimiterConfig, instanceCount int)
type RateLimiterConfig ¶
type RateLimiterConfig struct {
TotalRPM int // Total Requests Per Minute allowed by OpenAI
MaxConcurrency int // Maximum concurrent requests allowed
InstanceCount int // Number of instances running
}
RateLimiterConfig holds configuration for rate limiting
Source Files
Click to show internal directories.
Click to hide internal directories.