README
¶
Eviction Priority-Then-Time Ordering Plugin
Type: eviction-priority-then-time-ordering
An eviction ordering policy that selects which queued request to evict when the system is overloaded. It prioritizes evicting the lowest-priority request first. When two requests share the same priority, the most recently dispatched one is evicted first, minimizing wasted KV-cache investment.
Eviction ordering:
- Lowest priority first — requests with more negative priority are evicted before those with less negative or zero priority.
- Newest dispatch time first (tie-breaker) — among equal-priority requests, the one dispatched most recently is evicted, as it has the least sunk cost in KV-cache memory.
Parameters: None.
Related Documentation
Documentation
¶
Index ¶
Constants ¶
View Source
const PriorityThenTimeOrderingType = "priority-then-time-eviction-order-policy"
PriorityThenTimeOrderingType evicts the lowest priority request first, breaking ties by newest dispatch time (least KV-cache investment).
Variables ¶
This section is empty.
Functions ¶
Types ¶
type PriorityThenTimeOrdering ¶
type PriorityThenTimeOrdering struct {
// contains filtered or unexported fields
}
PriorityThenTimeOrdering evicts the lowest priority request first. When priorities are equal, the newest dispatched request is evicted first to minimize wasted KV-cache investment.
func (*PriorityThenTimeOrdering) Less ¶
func (p *PriorityThenTimeOrdering) Less(a, b *flowcontrol.EvictionItem) bool
func (*PriorityThenTimeOrdering) TypedName ¶
func (p *PriorityThenTimeOrdering) TypedName() plugin.TypedName
Source Files
Click to show internal directories.
Click to hide internal directories.