Documentation
¶
Overview ¶
Package fetch provides an SSRF-hardened HTTP client for safely retrieving content from external URLs. It enforces HTTPS-only access, domain allowlists, DNS pre-resolution with IP validation, and DNS-rebinding protection via transport-level IP pinning.
Index ¶
- Variables
- func AppendFetchAudit(logPath string, entry FetchAuditEntry) error
- func CacheNamedSymlink(treePath, skillName string) (string, error)
- func CachePath(workspaceRoot, hash string) (string, error)
- func CachePut(workspaceRoot, url string, content []byte) error
- func CachePutDir(workspaceRoot, url string, files map[string][]byte, opts ...DirCachePutOpts) (string, error)
- func ComputeSHA256(data []byte) string
- func ComputeTreeHash(files map[string][]byte) string
- func FetchURL(ctx context.Context, rawURL string, policy FetchPolicy) ([]byte, error)
- type CacheEntry
- type DirCacheEntry
- type DirCachePutOpts
- type DirFileEntry
- type FetchAuditEntry
- type FetchPolicy
Constants ¶
This section is empty.
Variables ¶
var DefaultPolicy = FetchPolicy{ AllowedDomains: []string{"github.com", "raw.githubusercontent.com"}, MaxSizeBytes: 10 * 1024 * 1024, Timeout: 30 * time.Second, }
DefaultPolicy is a sensible default policy allowing GitHub content hosts.
Functions ¶
func AppendFetchAudit ¶
func AppendFetchAudit(logPath string, entry FetchAuditEntry) error
AppendFetchAudit writes a fetch audit entry as a JSON line to the given log path.
func CacheNamedSymlink ¶ added in v0.31.0
CacheNamedSymlink creates a symlink in a cache directory so that the directory can be referenced by a human-readable skill name instead of the cache-internal "tree" directory name. It sanitizes the skill name, handles race conditions (concurrent symlink creation), and returns the symlinked path. If the skill name is invalid or matches a reserved cache file, the original treePath is returned unchanged.
func CachePath ¶ added in v0.12.0
CachePath returns the filesystem path for the cache directory keyed by the given content hash. The layout is:
<workspaceRoot>/.fullsend-cache/resources/sha256/<hash>/
The hash must be exactly 64 lowercase hex characters (SHA-256 digest).
func CachePut ¶ added in v0.12.0
CachePut stores content in the content-addressed cache. The content is keyed by its SHA-256 hash, so identical content from different URLs shares a single cache entry (the last URL wins in metadata — provenance of all source URLs is tracked via fetch audit logging, not cache metadata). Both the content and metadata files are written atomically using a temp-file-then-rename pattern with fsync for durability.
func CachePutDir ¶ added in v0.17.0
func CachePutDir(workspaceRoot, url string, files map[string][]byte, opts ...DirCachePutOpts) (string, error)
CachePutDir stores a directory tree in the content-addressed cache. files maps relative paths to their content bytes. Returns the computed tree hash. The directory is stored under:
<workspaceRoot>/.fullsend-cache/resources/sha256/<treeHash>/tree/<path>
Uses atomic file writes within the tree directory.
func ComputeSHA256 ¶
ComputeSHA256 returns the lowercase hex-encoded SHA-256 digest of data.
func ComputeTreeHash ¶ added in v0.17.0
ComputeTreeHash computes a deterministic SHA256 hash for a directory tree. The hash is SHA256 of the sorted concatenation of "path:sha256(content)\n" for all files. This is forge-agnostic and deterministic — any implementation can reproduce it from the same file set.
Types ¶
type CacheEntry ¶ added in v0.12.0
type CacheEntry struct {
URL string `json:"url"`
FetchTime time.Time `json:"fetch_time"`
SHA256 string `json:"sha256"`
}
CacheEntry is metadata for a cached remote resource.
func CacheGet ¶ added in v0.12.0
func CacheGet(workspaceRoot, hash string) ([]byte, *CacheEntry, error)
CacheGet retrieves a previously cached resource by its content hash. It returns (nil, nil, nil) on a cache miss (directory or files missing). If the cached content fails integrity re-verification, it returns an error.
type DirCacheEntry ¶ added in v0.17.0
type DirCacheEntry struct {
URL string `json:"url"`
FetchTime time.Time `json:"fetch_time"`
SHA256 string `json:"sha256"` // tree hash
Type string `json:"type"` // always "directory"
Files []DirFileEntry `json:"files"`
FullListing bool `json:"full_listing,omitempty"` // true when fetched via ForgeClient directory listing
}
DirCacheEntry is metadata for a cached directory resource (e.g., a skill).
func CacheGetDir ¶ added in v0.17.0
func CacheGetDir(workspaceRoot, hash string) (string, *DirCacheEntry, error)
CacheGetDir retrieves a previously cached directory resource by its tree hash. Returns ("", nil, nil) on a cache miss. On a hit, returns the path to the tree/ subdirectory and the cache metadata. Re-verifies integrity by recomputing the tree hash from the cached files.
type DirCachePutOpts ¶ added in v0.22.1
type DirCachePutOpts struct {
FullListing bool // mark entry as fetched via directory listing (not single-file fallback)
}
DirCachePutOpts controls optional behavior for CachePutDir.
type DirFileEntry ¶ added in v0.17.0
DirFileEntry records one file within a cached directory tree.
type FetchAuditEntry ¶
type FetchAuditEntry struct {
TraceID string `json:"trace_id"`
FetchTime time.Time `json:"fetch_time"`
URL string `json:"url"`
SHA256 string `json:"sha256"`
FetchType string `json:"fetch_type"`
AllowedBy string `json:"allowed_by"`
CacheHit bool `json:"cache_hit"`
}
FetchAuditEntry is a JSONL audit record for a remote resource fetch.
type FetchPolicy ¶
type FetchPolicy struct {
// AllowedDomains is the list of permitted hostnames. Entries may use a
// wildcard prefix (e.g. "*.example.com") to match any subdomain,
// including multi-level subdomains (e.g. "deep.sub.example.com").
AllowedDomains []string
// AllowedPorts restricts which ports may be used. When empty, only
// port 443 is permitted.
AllowedPorts []string
// MaxSizeBytes is the maximum response body size in bytes.
MaxSizeBytes int64
// Timeout is the overall HTTP request timeout.
Timeout time.Duration
// Offline, when true, causes FetchURL to reject all requests immediately.
Offline bool
// contains filtered or unexported fields
}
FetchPolicy controls the security constraints applied when fetching a URL.
func NewTestPolicy ¶ added in v0.13.0
func NewTestPolicy(tlsConfig *tls.Config, allowedDomains, allowedPorts []string) FetchPolicy
NewTestPolicy returns a FetchPolicy configured for use with httptest.NewTLSServer. Testing only — do not use in production code. It enables the TLS config and skips internal-IP validation (necessary because test servers listen on 127.0.0.1). These overrides are unexported to prevent direct bypass of SSRF protections.
Source Files