README
¶
pkg/httpstore
Pure HTTP resource store with two-version (pending/accepted) caching for safe content updates.
Overview
HTTPStore fetches arbitrary HTTP content (IP blocklists, JSON config, anything templates need from outside the cluster) and caches it under a two-version pattern: refreshed content lands in pending until the controller validates it, then it's promoted to accepted. Production renders only ever see the accepted version, so an upstream serving garbage can't break the live HAProxy config.
This is the pure half of the design. Periodic refresh timers, validation event handling, eviction, and the template-callable wrapper all live in pkg/controller/httpstore. Reach for the pure store directly only from tests; in production the event adapter is what you want.
Quick Start
import "gitlab.com/haproxy-haptic/haptic/pkg/httpstore"
store := httpstore.New(logger, 2*time.Minute) // maxAge for eviction; 0 disables
// Initial fetch (synchronous)
content, err := store.Fetch(ctx, "https://example.com/blocklist.txt",
httpstore.FetchOptions{
Timeout: 30 * time.Second,
Retries: 3,
Critical: true, // false → return "" + warn on failure
Delay: 5 * time.Minute, // refresh interval (the event adapter drives the timer)
},
&httpstore.AuthConfig{Type: "bearer", Token: "secret"},
)
// Later: refresh into pending (returns true if content changed)
changed, err := store.RefreshURL(ctx, "https://example.com/blocklist.txt")
// Production render reads accepted only
got, ok := store.Get(url)
// Validation render reads pending if available, otherwise accepted
got, ok = store.GetForValidation(url)
// After the validation pipeline returns:
store.PromotePending(url) // success → pending becomes accepted
store.RejectPending(url) // failure → pending discarded, accepted preserved
// Test fixtures bypass HTTP entirely
store.LoadFixture(url, "mock content")
Authentication
Three modes via AuthConfig.Type:
"basic"—Username+Password"bearer"—Token"header"—Headersmap (e.g.X-API-Key: ...)
Conditional Requests
Refreshes use ETag and Last-Modified automatically; a 304 returns changed=false so the existing accepted/pending state is preserved. Repeated refreshes against an unchanged upstream cost one round-trip and zero cache churn.
Size Limit
Responses are capped at MaxContentSize (10 MiB, declared in types.go and enforced in fetcher.go via io.LimitReader). Larger payloads fail with an explicit response body exceeds maximum size of N bytes error rather than being silently truncated, and the limited reader means the store never buffers more than ~10 MiB even for a runaway upstream. There is no per-call override; if you legitimately need larger blobs, change the constant.
Eviction
Entries that haven't been read in maxAge are evicted on the next EvictUnused() call. Entries with pending content are never evicted (pending must always have a place to land or be rejected). The event adapter calls EvictUnused periodically; for tests, pass maxAge: 0 to disable eviction entirely.
See Also
pkg/controller/httpstore— event adapter that drives refresh timers, handles validation lifecycle, and exposes the wrapper to templatespkg/controller/testrunner— usesLoadFixtureto mock HTTP content during validation testspkg/httpstore/CLAUDE.md— design notes (two-version pattern, eviction tuning, conditional-request mechanics)
License
Apache-2.0 — see root LICENSE.
Documentation
¶
Overview ¶
Package httpstore provides HTTP resource fetching with caching and validation.
This package implements a store that fetches resources from HTTP(S) URLs, caches them, and supports periodic refresh with a two-version cache for safe validation before accepting new content.
Index ¶
- Constants
- type AuthConfig
- type CacheEntry
- type FetchOptions
- type HTTPOverlay
- type HTTPStore
- func (s *HTTPStore) EvictUnused() []string
- func (s *HTTPStore) Fetch(ctx context.Context, url string, opts FetchOptions, auth *AuthConfig) (string, error)
- func (s *HTTPStore) Get(url string) (string, bool)
- func (s *HTTPStore) GetDelay(url string) time.Duration
- func (s *HTTPStore) GetEntry(url string) *CacheEntry
- func (s *HTTPStore) GetForValidation(url string) (string, bool)
- func (s *HTTPStore) GetPendingURLs() []string
- func (s *HTTPStore) LoadFixture(url, content string)
- func (s *HTTPStore) PromotePending(url string) bool
- func (s *HTTPStore) RefreshURL(ctx context.Context, url string) (changed bool, err error)
- func (s *HTTPStore) RejectPending(url string) bool
- type ValidationState
Constants ¶
const ( // AuthTypeBasic selects HTTP basic authentication (Username/Password). AuthTypeBasic = "basic" // AuthTypeBearer selects bearer-token authentication (Token). AuthTypeBearer = "bearer" // AuthTypeHeader selects custom-header authentication (Headers). AuthTypeHeader = "header" )
Supported AuthConfig.Type values.
const DefaultRetries = 2
DefaultRetries is the default number of retry attempts. Combined with DefaultRetryDelay's exponential backoff (0.5s + 1s = 1.5s of waits across 3 total attempts), this caps the worst-case budget for an unreachable URL well under the 5-second envelope a non-critical HTTP fetch should respect on a template render hot path. Raising either value here pushes per-render latency past the conformance test convergence windows that gate CI.
const DefaultRetryDelay = 500 * time.Millisecond
DefaultRetryDelay is the base delay between retry attempts. The backoff is exponential — attempt N waits `RetryDelay * 2^(N-1)` — so with Retries=2 the per-call wait budget is 0.5s + 1s = 1.5s. Each fetch attempt that fails fast on connection refused adds at most ~1 RTT, so total worst case is ~2s on a healthy network with an unreachable target.
const DefaultTimeout = 30 * time.Second
DefaultTimeout is the default HTTP request timeout.
const MaxContentSize = 10 * 1024 * 1024
MaxContentSize is the maximum allowed content size (10MB).
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type AuthConfig ¶
type AuthConfig struct {
// Type is the authentication type: AuthTypeBasic, AuthTypeBearer, or AuthTypeHeader.
Type string
// Username for basic auth.
Username string
// Password for basic auth.
Password string
// Token for bearer auth.
Token string
// Headers for custom header auth (e.g., API keys).
// These headers are added to every request.
Headers map[string]string
}
AuthConfig configures HTTP authentication.
type CacheEntry ¶
type CacheEntry struct {
// URL is the source URL for this entry.
URL string
// Accepted version (validated, in production use)
AcceptedContent string
AcceptedChecksum string
AcceptedTime time.Time
// LastAccessTime tracks when this entry was last accessed via Get/Fetch.
// Used for cache eviction of unused entries.
LastAccessTime time.Time
// Pending version (fetched, awaiting validation)
PendingContent string
PendingChecksum string
HasPending bool
// ValidationState tracks the current state of this entry.
ValidationState ValidationState
// HTTP caching headers for conditional requests
ETag string
LastModified string
// Configuration for this URL
Options FetchOptions
Auth *AuthConfig
}
CacheEntry holds cached content with two-version support for safe validation.
The two-version design ensures that new content is only accepted after successful validation. This is critical for resources like IP blocklists where we must not discard the old blocklist before knowing the new one is valid.
type FetchOptions ¶
type FetchOptions struct {
// Delay is the refresh interval (how often to re-fetch).
// Zero means no automatic refresh (fetch once).
Delay time.Duration
// Timeout is the HTTP request timeout.
// Default: 30s
Timeout time.Duration
// Retries is the number of retry attempts on failure.
// Default: 3
Retries int
// RetryDelay is the wait time between retries.
// Default: 1s
RetryDelay time.Duration
// Critical indicates whether fetch failure should fail the template render.
// If true, a failed fetch returns an error.
// If false, a failed fetch returns empty string and logs a warning.
Critical bool
}
FetchOptions configures HTTP fetching behavior.
func (FetchOptions) WithDefaults ¶
func (o FetchOptions) WithDefaults() FetchOptions
WithDefaults returns a copy of the options with default values applied.
type HTTPOverlay ¶
type HTTPOverlay struct {
// contains filtered or unexported fields
}
HTTPOverlay represents pending HTTP content changes awaiting validation.
Unlike K8s overlays which are constructed explicitly with additions/modifications/deletions, HTTP overlays are derived from the HTTPStore's pending content state. When content changes during refresh, it's stored as pending in the HTTPStore. The overlay provides access to this pending content during validation rendering.
This type implements the stores.ContentOverlay interface, enabling unified handling of both K8s and HTTP overlays in the validation pipeline.
func NewHTTPOverlay ¶
func NewHTTPOverlay(store *HTTPStore) *HTTPOverlay
NewHTTPOverlay creates an overlay from the store's current pending state.
The overlay captures a snapshot of which URLs have pending content at creation time. This ensures consistent behavior even if the store's state changes during validation.
Parameters:
- store: The HTTPStore to derive pending state from
Returns:
- An HTTPOverlay with the current pending URLs snapshot
func (*HTTPOverlay) GetContent ¶
func (o *HTTPOverlay) GetContent(url string) (string, bool)
GetContent returns content for the given URL.
If the URL has pending content, returns the pending content. Otherwise, returns the accepted content if available. This behavior matches what templates should see during validation rendering.
Parameters:
- url: The URL to get content for
Returns:
- content: The content string (pending preferred, otherwise accepted)
- ok: True if content was found
func (*HTTPOverlay) HasPendingURL ¶
func (o *HTTPOverlay) HasPendingURL(url string) bool
HasPendingURL returns true if the given URL has pending content.
func (*HTTPOverlay) IsEmpty ¶
func (o *HTTPOverlay) IsEmpty() bool
IsEmpty returns true if the overlay contains no pending content. Implements the stores.ContentOverlay interface.
func (*HTTPOverlay) PendingURLs ¶
func (o *HTTPOverlay) PendingURLs() []string
PendingURLs returns the list of URLs with pending content. This is the snapshot captured at overlay creation time.
type HTTPStore ¶
type HTTPStore struct {
// contains filtered or unexported fields
}
HTTPStore provides HTTP resource fetching with caching and two-version validation.
The store supports:
- Synchronous initial fetch (blocks until complete)
- Cached access to previously fetched content
- Two-version cache for safe validation (pending vs accepted)
- Conditional requests using ETag/If-Modified-Since
- Automatic eviction of unused entries based on last access time
Thread-safe for concurrent access.
func New ¶
New creates a new HTTPStore with the given logger and maximum cache age.
maxAge is the maximum time an entry can remain unused before becoming eligible for eviction. If maxAge is 0, entries are never evicted based on access time.
func (*HTTPStore) EvictUnused ¶
EvictUnused removes cache entries that haven't been accessed within maxAge. Entries with pending validation are never evicted to protect the two-version cache. Returns the list of evicted URLs (empty if none evicted or eviction disabled).
This method is called periodically by the event adapter to prevent unbounded memory growth when templates change and old URLs are no longer used.
func (*HTTPStore) Fetch ¶
func (s *HTTPStore) Fetch(ctx context.Context, url string, opts FetchOptions, auth *AuthConfig) (string, error)
Fetch retrieves content from a URL, using cache if available.
On first call for a URL, this performs a synchronous HTTP fetch and caches the result. Subsequent calls return cached content immediately.
If the URL has a Delay > 0 in options, the caller is responsible for scheduling refreshes (typically done by the event adapter component).
Parameters:
- ctx: Context for cancellation and timeout
- url: The HTTP(S) URL to fetch
- opts: Fetch options (timeout, retries, critical flag, delay for refresh)
- auth: Optional authentication configuration
Returns:
- Content string (empty if fetch failed and not critical)
- Error if critical fetch fails
func (*HTTPStore) Get ¶
Get returns the accepted content for a URL if it exists in cache. Returns empty string and false if not cached. Updates LastAccessTime to track usage for cache eviction.
func (*HTTPStore) GetDelay ¶
GetDelay returns the configured delay for a URL. Returns 0 if URL not in cache or no delay configured.
func (*HTTPStore) GetEntry ¶
func (s *HTTPStore) GetEntry(url string) *CacheEntry
GetEntry returns a copy of the cache entry for a URL. Returns nil if not cached.
func (*HTTPStore) GetForValidation ¶
GetForValidation returns content for validation rendering. If pending content exists, returns pending; otherwise returns accepted. Updates LastAccessTime to track usage for cache eviction.
func (*HTTPStore) GetPendingURLs ¶
GetPendingURLs returns all URLs with pending content awaiting validation.
func (*HTTPStore) LoadFixture ¶
LoadFixture loads a single HTTP fixture directly into the store as accepted content. This is used by validation tests to provide mock HTTP responses without making actual HTTP requests.
The fixture is stored directly as accepted content, bypassing the normal fetch and validation workflow.
func (*HTTPStore) PromotePending ¶
PromotePending promotes pending content to accepted for a URL. This should be called after successful validation.
func (*HTTPStore) RefreshURL ¶
RefreshURL fetches fresh content for a URL and stores it as pending.
This does NOT replace accepted content immediately. The caller must: 1. Trigger re-render with pending content (using GetForValidation) 2. On successful validation, call PromotePending 3. On failed validation, call RejectPending
Returns:
- changed: true if content changed from accepted version
- err: fetch error (nil if successful or 304 Not Modified)
func (*HTTPStore) RejectPending ¶
RejectPending discards pending content for a URL. This should be called when validation fails.
type ValidationState ¶
type ValidationState int
ValidationState represents the current validation state of a cached entry.
const ( // StateAccepted means the accepted content is in use, no pending content. StateAccepted ValidationState = iota // StateValidating means pending content exists and is being validated. StateValidating // StateRejected means the last pending content was rejected, using accepted. StateRejected )
func (ValidationState) String ¶
func (s ValidationState) String() string
String returns a string representation of the validation state.
Source Files