Documentation
¶
Overview ¶
Package httpstore provides HTTP resource fetching with caching and validation.
This package implements a store that fetches resources from HTTP(S) URLs, caches them, and supports periodic refresh with a two-version cache for safe validation before accepting new content.
Index ¶
- Constants
- func Checksum(content string) string
- type AuthConfig
- type CacheEntry
- type FetchOptions
- type HTTPOverlay
- type HTTPStore
- func (s *HTTPStore) Clear()
- func (s *HTTPStore) EvictUnused() []string
- func (s *HTTPStore) Fetch(ctx context.Context, url string, opts FetchOptions, auth *AuthConfig) (string, error)
- func (s *HTTPStore) Get(url string) (string, bool)
- func (s *HTTPStore) GetDelay(url string) time.Duration
- func (s *HTTPStore) GetEntry(url string) *CacheEntry
- func (s *HTTPStore) GetForValidation(url string) (string, bool)
- func (s *HTTPStore) GetPending(url string) (string, bool)
- func (s *HTTPStore) GetPendingURLs() []string
- func (s *HTTPStore) GetURLsWithDelay() []string
- func (s *HTTPStore) HasPendingValidation() bool
- func (s *HTTPStore) LoadFixture(url, content string)
- func (s *HTTPStore) PromoteAllPending() int
- func (s *HTTPStore) PromotePending(url string) bool
- func (s *HTTPStore) RefreshURL(ctx context.Context, url string) (changed bool, err error)
- func (s *HTTPStore) RejectAllPending() int
- func (s *HTTPStore) RejectPending(url string) bool
- func (s *HTTPStore) Size() int
- type ValidationState
Constants ¶
const DefaultRetries = 3
DefaultRetries is the default number of retry attempts.
const DefaultRetryDelay = time.Second
DefaultRetryDelay is the default delay between retry attempts.
const DefaultTimeout = 30 * time.Second
DefaultTimeout is the default HTTP request timeout.
const MaxContentSize = 10 * 1024 * 1024
MaxContentSize is the maximum allowed content size (10MB).
Variables ¶
This section is empty.
Functions ¶
Types ¶
type AuthConfig ¶
type AuthConfig struct {
// Type is the authentication type: "basic", "bearer", or "header".
Type string
// Username for basic auth.
Username string
// Password for basic auth.
Password string
// Token for bearer auth.
Token string
// Headers for custom header auth (e.g., API keys).
// These headers are added to every request.
Headers map[string]string
}
AuthConfig configures HTTP authentication.
type CacheEntry ¶
type CacheEntry struct {
// URL is the source URL for this entry.
URL string
// Accepted version (validated, in production use)
AcceptedContent string
AcceptedChecksum string
AcceptedTime time.Time
// LastAccessTime tracks when this entry was last accessed via Get/Fetch.
// Used for cache eviction of unused entries.
LastAccessTime time.Time
// Pending version (fetched, awaiting validation)
PendingContent string
PendingChecksum string
HasPending bool
// ValidationState tracks the current state of this entry.
ValidationState ValidationState
// HTTP caching headers for conditional requests
ETag string
LastModified string
// Configuration for this URL
Options FetchOptions
Auth *AuthConfig
}
CacheEntry holds cached content with two-version support for safe validation.
The two-version design ensures that new content is only accepted after successful validation. This is critical for resources like IP blocklists where we must not discard the old blocklist before knowing the new one is valid.
type FetchOptions ¶
type FetchOptions struct {
// Delay is the refresh interval (how often to re-fetch).
// Zero means no automatic refresh (fetch once).
Delay time.Duration
// Timeout is the HTTP request timeout.
// Default: 30s
Timeout time.Duration
// Retries is the number of retry attempts on failure.
// Default: 3
Retries int
// RetryDelay is the wait time between retries.
// Default: 1s
RetryDelay time.Duration
// Critical indicates whether fetch failure should fail the template render.
// If true, a failed fetch returns an error.
// If false, a failed fetch returns empty string and logs a warning.
Critical bool
}
FetchOptions configures HTTP fetching behavior.
func (FetchOptions) WithDefaults ¶
func (o FetchOptions) WithDefaults() FetchOptions
WithDefaults returns a copy of the options with default values applied.
type HTTPOverlay ¶
type HTTPOverlay struct {
// contains filtered or unexported fields
}
HTTPOverlay represents pending HTTP content changes awaiting validation.
Unlike K8s overlays which are constructed explicitly with additions/modifications/deletions, HTTP overlays are derived from the HTTPStore's pending content state. When content changes during refresh, it's stored as pending in the HTTPStore. The overlay provides access to this pending content during validation rendering.
This type implements the stores.ContentOverlay interface, enabling unified handling of both K8s and HTTP overlays in the validation pipeline.
func NewHTTPOverlay ¶
func NewHTTPOverlay(store *HTTPStore) *HTTPOverlay
NewHTTPOverlay creates an overlay from the store's current pending state.
The overlay captures a snapshot of which URLs have pending content at creation time. This ensures consistent behavior even if the store's state changes during validation.
Parameters:
- store: The HTTPStore to derive pending state from
Returns:
- An HTTPOverlay with the current pending URLs snapshot
func (*HTTPOverlay) GetContent ¶
func (o *HTTPOverlay) GetContent(url string) (string, bool)
GetContent returns content for the given URL.
If the URL has pending content, returns the pending content. Otherwise, returns the accepted content if available. This behavior matches what templates should see during validation rendering.
Parameters:
- url: The URL to get content for
Returns:
- content: The content string (pending preferred, otherwise accepted)
- ok: True if content was found
func (*HTTPOverlay) HasPendingURL ¶
func (o *HTTPOverlay) HasPendingURL(url string) bool
HasPendingURL returns true if the given URL has pending content.
func (*HTTPOverlay) IsEmpty ¶
func (o *HTTPOverlay) IsEmpty() bool
IsEmpty returns true if the overlay contains no pending content. Implements the stores.ContentOverlay interface.
func (*HTTPOverlay) PendingURLs ¶
func (o *HTTPOverlay) PendingURLs() []string
PendingURLs returns the list of URLs with pending content. This is the snapshot captured at overlay creation time.
type HTTPStore ¶
type HTTPStore struct {
// contains filtered or unexported fields
}
HTTPStore provides HTTP resource fetching with caching and two-version validation.
The store supports:
- Synchronous initial fetch (blocks until complete)
- Cached access to previously fetched content
- Two-version cache for safe validation (pending vs accepted)
- Conditional requests using ETag/If-Modified-Since
- Automatic eviction of unused entries based on last access time
Thread-safe for concurrent access.
func New ¶
New creates a new HTTPStore with the given logger and maximum cache age.
maxAge is the maximum time an entry can remain unused before becoming eligible for eviction. If maxAge is 0, entries are never evicted based on access time.
func (*HTTPStore) EvictUnused ¶
EvictUnused removes cache entries that haven't been accessed within maxAge. Entries with pending validation are never evicted to protect the two-version cache. Returns the list of evicted URLs (empty if none evicted or eviction disabled).
This method is called periodically by the event adapter to prevent unbounded memory growth when templates change and old URLs are no longer used.
func (*HTTPStore) Fetch ¶
func (s *HTTPStore) Fetch(ctx context.Context, url string, opts FetchOptions, auth *AuthConfig) (string, error)
Fetch retrieves content from a URL, using cache if available.
On first call for a URL, this performs a synchronous HTTP fetch and caches the result. Subsequent calls return cached content immediately.
If the URL has a Delay > 0 in options, the caller is responsible for scheduling refreshes (typically done by the event adapter component).
Parameters:
- ctx: Context for cancellation and timeout
- url: The HTTP(S) URL to fetch
- opts: Fetch options (timeout, retries, critical flag, delay for refresh)
- auth: Optional authentication configuration
Returns:
- Content string (empty if fetch failed and not critical)
- Error if critical fetch fails
func (*HTTPStore) Get ¶
Get returns the accepted content for a URL if it exists in cache. Returns empty string and false if not cached. Updates LastAccessTime to track usage for cache eviction.
func (*HTTPStore) GetDelay ¶
GetDelay returns the configured delay for a URL. Returns 0 if URL not in cache or no delay configured.
func (*HTTPStore) GetEntry ¶
func (s *HTTPStore) GetEntry(url string) *CacheEntry
GetEntry returns a copy of the cache entry for a URL. Returns nil if not cached.
func (*HTTPStore) GetForValidation ¶
GetForValidation returns content for validation rendering. If pending content exists, returns pending; otherwise returns accepted. Updates LastAccessTime to track usage for cache eviction.
func (*HTTPStore) GetPending ¶
GetPending returns the pending content for a URL if it exists. This is used during validation to render with pending content. Returns empty string and false if no pending content. Updates LastAccessTime to track usage for cache eviction.
func (*HTTPStore) GetPendingURLs ¶
GetPendingURLs returns all URLs with pending content awaiting validation.
func (*HTTPStore) GetURLsWithDelay ¶
GetURLsWithDelay returns all cached URLs that have a refresh delay configured. This is used by the event adapter to schedule refresh timers.
func (*HTTPStore) HasPendingValidation ¶
HasPendingValidation returns true if any URL has pending content awaiting validation.
func (*HTTPStore) LoadFixture ¶
LoadFixture loads a single HTTP fixture directly into the store as accepted content. This is used by validation tests to provide mock HTTP responses without making actual HTTP requests.
The fixture is stored directly as accepted content, bypassing the normal fetch and validation workflow.
func (*HTTPStore) PromoteAllPending ¶
PromoteAllPending promotes all pending content to accepted. This is used when validation succeeds for the entire config.
func (*HTTPStore) PromotePending ¶
PromotePending promotes pending content to accepted for a URL. This should be called after successful validation.
func (*HTTPStore) RefreshURL ¶
RefreshURL fetches fresh content for a URL and stores it as pending.
This does NOT replace accepted content immediately. The caller must: 1. Trigger re-render with pending content (using GetForValidation) 2. On successful validation, call PromotePending 3. On failed validation, call RejectPending
Returns:
- changed: true if content changed from accepted version
- err: fetch error (nil if successful or 304 Not Modified)
func (*HTTPStore) RejectAllPending ¶
RejectAllPending rejects all pending content. This is used when validation fails for the entire config.
func (*HTTPStore) RejectPending ¶
RejectPending discards pending content for a URL. This should be called when validation fails.
type ValidationState ¶
type ValidationState int
ValidationState represents the current validation state of a cached entry.
const ( // StateAccepted means the accepted content is in use, no pending content. StateAccepted ValidationState = iota // StateValidating means pending content exists and is being validated. StateValidating // StateRejected means the last pending content was rejected, using accepted. StateRejected )
func (ValidationState) String ¶
func (s ValidationState) String() string
String returns a string representation of the validation state.
Source Files