Documentation
¶
Overview ¶
internal/httpclient/cache.go
This file implements a simple in-memory cache for HTTP GET responses. It is intentionally small and conservative, but provides a real, effective caching layer that reduces repeated network requests.
internal/httpclient/limiter.go
This file implements simple concurrency limiting for outbound HTTP requests, both globally and per-host. It helps ensure that Aether behaves politely when accessing remote servers.
internal/httpclient/response.go
This file defines the internal HTTP response type used by Aether's HTTP client. It is deliberately small and immutable from the point of view of callers.
internal/httpclient/robots.go
This file implements robots.txt fetching and caching for the HTTP client. Robots files are fetched once per host and reused.
Index ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type Client ¶
type Client struct {
// contains filtered or unexported fields
}
Client is Aether's internal HTTP client.
NOTE: - Consumers do NOT use this directly. - The public aether.Client wraps this and provides Fetch().
func New ¶
New constructs a new internal HTTP client.
This client: - uses a single http.Client for pooling - respects timeouts - sets up robots.txt cache - sets concurrency limits - initializes the composite cache (memory/file/redis)
func (*Client) Fetch ¶
func (c *Client) Fetch( ctx context.Context, rawURL string, headers http.Header, ) (*Response, error)
Fetch performs a robots.txt compliant HTTP GET with: - concurrency limiting - composite caching - retry logic - transparent User-Agent injection
headers: optional additional request headers.
Source Files