fetch

package
v0.15.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Jun 8, 2026 License: Apache-2.0 Imports: 16 Imported by: 0

Documentation

Overview

Package fetch provides an SSRF-hardened HTTP client for safely retrieving content from external URLs. It enforces HTTPS-only access, domain allowlists, DNS pre-resolution with IP validation, and DNS-rebinding protection via transport-level IP pinning.

Index

Constants

This section is empty.

Variables

View Source
var DefaultPolicy = FetchPolicy{
	AllowedDomains: []string{"github.com", "raw.githubusercontent.com"},
	MaxSizeBytes:   10 * 1024 * 1024,
	Timeout:        30 * time.Second,
}

DefaultPolicy is a sensible default policy allowing GitHub content hosts.

Functions

func AppendFetchAudit

func AppendFetchAudit(logPath string, entry FetchAuditEntry) error

AppendFetchAudit writes a fetch audit entry as a JSON line to the given log path.

func CachePath added in v0.12.0

func CachePath(workspaceRoot, hash string) (string, error)

CachePath returns the filesystem path for the cache directory keyed by the given content hash. The layout is:

<workspaceRoot>/.fullsend-cache/resources/sha256/<hash>/

The hash must be exactly 64 lowercase hex characters (SHA-256 digest).

func CachePut added in v0.12.0

func CachePut(workspaceRoot, url string, content []byte) error

CachePut stores content in the content-addressed cache. The content is keyed by its SHA-256 hash, so identical content from different URLs shares a single cache entry (the last URL wins in metadata — provenance of all source URLs is tracked via fetch audit logging, not cache metadata). Both the content and metadata files are written atomically using a temp-file-then-rename pattern with fsync for durability.

func ComputeSHA256

func ComputeSHA256(data []byte) string

ComputeSHA256 returns the lowercase hex-encoded SHA-256 digest of data.

func FetchURL

func FetchURL(ctx context.Context, rawURL string, policy FetchPolicy) ([]byte, error)

FetchURL retrieves the content at rawURL subject to the given policy. It returns the response body bytes or an error describing why the fetch was rejected or failed.

Types

type CacheEntry added in v0.12.0

type CacheEntry struct {
	URL       string    `json:"url"`
	FetchTime time.Time `json:"fetch_time"`
	SHA256    string    `json:"sha256"`
}

CacheEntry is metadata for a cached remote resource.

func CacheGet added in v0.12.0

func CacheGet(workspaceRoot, hash string) ([]byte, *CacheEntry, error)

CacheGet retrieves a previously cached resource by its content hash. It returns (nil, nil, nil) on a cache miss (directory or files missing). If the cached content fails integrity re-verification, it returns an error.

type FetchAuditEntry

type FetchAuditEntry struct {
	TraceID   string    `json:"trace_id"`
	FetchTime time.Time `json:"fetch_time"`
	URL       string    `json:"url"`
	SHA256    string    `json:"sha256"`
	FetchType string    `json:"fetch_type"`
	AllowedBy string    `json:"allowed_by"`
	CacheHit  bool      `json:"cache_hit"`
}

FetchAuditEntry is a JSONL audit record for a remote resource fetch.

type FetchPolicy

type FetchPolicy struct {
	// AllowedDomains is the list of permitted hostnames. Entries may use a
	// wildcard prefix (e.g. "*.example.com") to match any subdomain,
	// including multi-level subdomains (e.g. "deep.sub.example.com").
	AllowedDomains []string

	// AllowedPorts restricts which ports may be used. When empty, only
	// port 443 is permitted.
	AllowedPorts []string

	// MaxSizeBytes is the maximum response body size in bytes.
	MaxSizeBytes int64

	// Timeout is the overall HTTP request timeout.
	Timeout time.Duration

	// Offline, when true, causes FetchURL to reject all requests immediately.
	Offline bool
	// contains filtered or unexported fields
}

FetchPolicy controls the security constraints applied when fetching a URL.

func NewTestPolicy added in v0.13.0

func NewTestPolicy(tlsConfig *tls.Config, allowedDomains, allowedPorts []string) FetchPolicy

NewTestPolicy returns a FetchPolicy configured for use with httptest.NewTLSServer. Testing only — do not use in production code. It enables the TLS config and skips internal-IP validation (necessary because test servers listen on 127.0.0.1). These overrides are unexported to prevent direct bypass of SSRF protections.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL