Documentation
¶
Overview ¶
Package httpstore provides HTTP resource fetching with caching and validation.
This package implements a store that fetches resources from HTTP(S) URLs, caches them, and supports periodic refresh with a two-version cache for safe validation before accepting new content.
Index ¶
- Constants
- func Checksum(content string) string
- type AuthConfig
- type CacheEntry
- type FetchOptions
- type HTTPStore
- func (s *HTTPStore) Clear()
- func (s *HTTPStore) EvictUnused() []string
- func (s *HTTPStore) Fetch(ctx context.Context, url string, opts FetchOptions, auth *AuthConfig) (string, error)
- func (s *HTTPStore) Get(url string) (string, bool)
- func (s *HTTPStore) GetDelay(url string) time.Duration
- func (s *HTTPStore) GetEntry(url string) *CacheEntry
- func (s *HTTPStore) GetForValidation(url string) (string, bool)
- func (s *HTTPStore) GetPending(url string) (string, bool)
- func (s *HTTPStore) GetPendingURLs() []string
- func (s *HTTPStore) GetURLsWithDelay() []string
- func (s *HTTPStore) HasPendingValidation() bool
- func (s *HTTPStore) LoadFixture(url, content string)
- func (s *HTTPStore) PromoteAllPending() int
- func (s *HTTPStore) PromotePending(url string) bool
- func (s *HTTPStore) RefreshURL(ctx context.Context, url string) (changed bool, err error)
- func (s *HTTPStore) RejectAllPending() int
- func (s *HTTPStore) RejectPending(url string) bool
- func (s *HTTPStore) Size() int
- type ValidationState
Constants ¶
const DefaultRetries = 3
DefaultRetries is the default number of retry attempts.
const DefaultRetryDelay = time.Second
DefaultRetryDelay is the default delay between retry attempts.
const DefaultTimeout = 30 * time.Second
DefaultTimeout is the default HTTP request timeout.
const MaxContentSize = 10 * 1024 * 1024
MaxContentSize is the maximum allowed content size (10MB).
Variables ¶
This section is empty.
Functions ¶
Types ¶
type AuthConfig ¶
type AuthConfig struct {
// Type is the authentication type: "basic", "bearer", or "header".
Type string
// Username for basic auth.
Username string
// Password for basic auth.
Password string
// Token for bearer auth.
Token string
// Headers for custom header auth (e.g., API keys).
// These headers are added to every request.
Headers map[string]string
}
AuthConfig configures HTTP authentication.
type CacheEntry ¶
type CacheEntry struct {
// URL is the source URL for this entry.
URL string
// Accepted version (validated, in production use)
AcceptedContent string
AcceptedChecksum string
AcceptedTime time.Time
// LastAccessTime tracks when this entry was last accessed via Get/Fetch.
// Used for cache eviction of unused entries.
LastAccessTime time.Time
// Pending version (fetched, awaiting validation)
PendingContent string
PendingChecksum string
HasPending bool
// ValidationState tracks the current state of this entry.
ValidationState ValidationState
// HTTP caching headers for conditional requests
ETag string
LastModified string
// Configuration for this URL
Options FetchOptions
Auth *AuthConfig
}
CacheEntry holds cached content with two-version support for safe validation.
The two-version design ensures that new content is only accepted after successful validation. This is critical for resources like IP blocklists where we must not discard the old blocklist before knowing the new one is valid.
type FetchOptions ¶
type FetchOptions struct {
// Delay is the refresh interval (how often to re-fetch).
// Zero means no automatic refresh (fetch once).
Delay time.Duration
// Timeout is the HTTP request timeout.
// Default: 30s
Timeout time.Duration
// Retries is the number of retry attempts on failure.
// Default: 3
Retries int
// RetryDelay is the wait time between retries.
// Default: 1s
RetryDelay time.Duration
// Critical indicates whether fetch failure should fail the template render.
// If true, a failed fetch returns an error.
// If false, a failed fetch returns empty string and logs a warning.
Critical bool
}
FetchOptions configures HTTP fetching behavior.
func (FetchOptions) WithDefaults ¶
func (o FetchOptions) WithDefaults() FetchOptions
WithDefaults returns a copy of the options with default values applied.
type HTTPStore ¶
type HTTPStore struct {
// contains filtered or unexported fields
}
HTTPStore provides HTTP resource fetching with caching and two-version validation.
The store supports:
- Synchronous initial fetch (blocks until complete)
- Cached access to previously fetched content
- Two-version cache for safe validation (pending vs accepted)
- Conditional requests using ETag/If-Modified-Since
- Automatic eviction of unused entries based on last access time
Thread-safe for concurrent access.
func New ¶
New creates a new HTTPStore with the given logger and maximum cache age.
maxAge is the maximum time an entry can remain unused before becoming eligible for eviction. If maxAge is 0, entries are never evicted based on access time.
func (*HTTPStore) EvictUnused ¶
EvictUnused removes cache entries that haven't been accessed within maxAge. Entries with pending validation are never evicted to protect the two-version cache. Returns the list of evicted URLs (empty if none evicted or eviction disabled).
This method is called periodically by the event adapter to prevent unbounded memory growth when templates change and old URLs are no longer used.
func (*HTTPStore) Fetch ¶
func (s *HTTPStore) Fetch(ctx context.Context, url string, opts FetchOptions, auth *AuthConfig) (string, error)
Fetch retrieves content from a URL, using cache if available.
On first call for a URL, this performs a synchronous HTTP fetch and caches the result. Subsequent calls return cached content immediately.
If the URL has a Delay > 0 in options, the caller is responsible for scheduling refreshes (typically done by the event adapter component).
Parameters:
- ctx: Context for cancellation and timeout
- url: The HTTP(S) URL to fetch
- opts: Fetch options (timeout, retries, critical flag, delay for refresh)
- auth: Optional authentication configuration
Returns:
- Content string (empty if fetch failed and not critical)
- Error if critical fetch fails
func (*HTTPStore) Get ¶
Get returns the accepted content for a URL if it exists in cache. Returns empty string and false if not cached. Updates LastAccessTime to track usage for cache eviction.
func (*HTTPStore) GetDelay ¶
GetDelay returns the configured delay for a URL. Returns 0 if URL not in cache or no delay configured.
func (*HTTPStore) GetEntry ¶
func (s *HTTPStore) GetEntry(url string) *CacheEntry
GetEntry returns a copy of the cache entry for a URL. Returns nil if not cached.
func (*HTTPStore) GetForValidation ¶
GetForValidation returns content for validation rendering. If pending content exists, returns pending; otherwise returns accepted. Updates LastAccessTime to track usage for cache eviction.
func (*HTTPStore) GetPending ¶
GetPending returns the pending content for a URL if it exists. This is used during validation to render with pending content. Returns empty string and false if no pending content. Updates LastAccessTime to track usage for cache eviction.
func (*HTTPStore) GetPendingURLs ¶
GetPendingURLs returns all URLs with pending content awaiting validation.
func (*HTTPStore) GetURLsWithDelay ¶
GetURLsWithDelay returns all cached URLs that have a refresh delay configured. This is used by the event adapter to schedule refresh timers.
func (*HTTPStore) HasPendingValidation ¶
HasPendingValidation returns true if any URL has pending content awaiting validation.
func (*HTTPStore) LoadFixture ¶
LoadFixture loads a single HTTP fixture directly into the store as accepted content. This is used by validation tests to provide mock HTTP responses without making actual HTTP requests.
The fixture is stored directly as accepted content, bypassing the normal fetch and validation workflow.
func (*HTTPStore) PromoteAllPending ¶
PromoteAllPending promotes all pending content to accepted. This is used when validation succeeds for the entire config.
func (*HTTPStore) PromotePending ¶
PromotePending promotes pending content to accepted for a URL. This should be called after successful validation.
func (*HTTPStore) RefreshURL ¶
RefreshURL fetches fresh content for a URL and stores it as pending.
This does NOT replace accepted content immediately. The caller must: 1. Trigger re-render with pending content (using GetForValidation) 2. On successful validation, call PromotePending 3. On failed validation, call RejectPending
Returns:
- changed: true if content changed from accepted version
- err: fetch error (nil if successful or 304 Not Modified)
func (*HTTPStore) RejectAllPending ¶
RejectAllPending rejects all pending content. This is used when validation fails for the entire config.
func (*HTTPStore) RejectPending ¶
RejectPending discards pending content for a URL. This should be called when validation fails.
type ValidationState ¶
type ValidationState int
ValidationState represents the current validation state of a cached entry.
const ( // StateAccepted means the accepted content is in use, no pending content. StateAccepted ValidationState = iota // StateValidating means pending content exists and is being validated. StateValidating // StateRejected means the last pending content was rejected, using accepted. StateRejected )
func (ValidationState) String ¶
func (s ValidationState) String() string
String returns a string representation of the validation state.
Source Files