README
¶
ESI (Edge Side Includes)
Package github.com/indragunawan/titip/esi provides an Edge Side Includes (ESI) processor for Go.
Features
- Buffer Pooling: Uses
sync.Poolfor buffer reuse during document splicing. - Concurrent Fetching: Fetches fragment targets concurrently up to a configured limit.
- SSRF Protection: Blocks private, loopback, and link-local IP ranges by default for outbound HTTP includes.
- In-Process Fetching: Resolves local paths via
http.Handlerbefore falling back to outbound HTTP. - Recursion Limits & Cycle Detection: Prevents infinite loops and limits nesting depth.
- Prometheus Metrics: Exports fragment counts and latency distributions.
Supported Directives
| Directive | Syntax | Description |
|---|---|---|
| Include | <esi:include src="..." /> |
Fetches and inlines fragment content. |
| Include with Fallback | <esi:include src="..." alt="..." /> |
Fetches alt URL if src fails. |
| Paired Include | <esi:include src="...">Fallback Content</esi:include> |
Renders enclosed content if src and alt fail. |
| Continue on Error | <esi:include src="..." onerror="continue" /> |
Suppresses errors and omits fragment if fetch fails. |
| Remove | <esi:remove>...</esi:remove> |
Strips enclosed content when ESI processing is active. |
| Comment | <esi:comment text="..." /> |
Stripped from output. |
| Comment Wrapper | <!--esi ... --> |
Unescapes enclosed content during ESI processing. |
Tag Attributes (<esi:include>)
| Attribute | Required | Supported Formats | Interaction with Global Config |
|---|---|---|---|
src |
Yes | Relative path (/api/user) or absolute URL (https://...) |
Primary fragment target. Handled in-process when matched by WithInternalFetcher, otherwise outbound HTTP. |
alt |
No | Relative path or absolute URL | Secondary fallback target attempted if src returns an error or non-2xx status. |
timeout |
No | 500ms, 2s, 0.5 (seconds), 500 (ms) |
Total SLA budget for the include slot (src + alt). Bounded by WithMaxTimeout and the parent branch's remaining Tree Budget. |
max-depth |
No | Integer (e.g. 2) |
Maximum recursion depth for nested includes within this fragment. Bounded by WithMaxDepth. |
onerror |
No | "continue" |
When "continue", suppresses fetch failure and renders paired fallback content or empty string. If omitted, unhandled errors render WithIncludeErrorMarker. |
Tag Attributes (<esi:comment>)
| Attribute | Required | Description |
|---|---|---|
text |
No | Descriptive comment text. The entire tag is stripped from the rendered output. |
Quickstart
Standalone Document Processing
package main
import (
"context"
"fmt"
"net/http"
"github.com/indragunawan/titip/esi"
)
func main() {
proc := esi.NewProcessor()
html := []byte(`
<html>
<body>
<header><esi:include src="https://example.com/header" /></header>
<main>Main Content</main>
<!--esi <footer>Rendered by ESI</footer> -->
</body>
</html>
`)
req, _ := http.NewRequestWithContext(context.Background(), http.MethodGet, "https://example.com/", nil)
fragments := esi.Scan(html)
if len(fragments) == 0 {
fmt.Println(string(html))
return
}
result, err := proc.ProcessFragments(context.Background(), req, html, fragments)
if err != nil {
panic(err)
}
defer result.Release()
fmt.Println(string(result.Body()))
}
In-Process Subrequests (HandlerFetcher)
Resolve local paths directly through an http.Handler instead of outbound HTTP:
mux := http.NewServeMux()
mux.HandleFunc("/fragments/user", func(w http.ResponseWriter, r *http.Request) {
w.Write([]byte(`<span>Logged in as Alice</span>`))
})
proc := esi.NewProcessor(
esi.WithInternalFetcher(esi.HandlerFetcher(mux)),
)
If the handler returns 404, the processor falls back to outbound HTTP.
Configuration Options
| Option | Default | Description |
|---|---|---|
WithInternalFetcher(fn) |
nil |
In-process subrequest handler. Use esi.HandlerFetcher(router) to adapt an http.Handler. |
WithHeaderRequired() |
(disabled) | Documents are only processed if origin response Surrogate-Control contains ESI/1.0. |
WithMaxDepth(uint32) |
3 |
Maximum nesting depth for recursive includes. |
WithMaxTimeout(time.Duration) |
30s |
Timeout per fragment request. |
WithMaxConcurrentRequests(int) |
8 |
Maximum concurrent fragment requests per document. |
WithMaxResponseSize(int64) |
10MB |
Maximum fragment response size in bytes. |
WithAllowPrivateIPs() |
(disabled) | Allows requests to private, loopback, and link-local IP addresses. |
WithAllowedHosts(...string) |
[] |
Allowed hostnames for outbound requests (empty allows any public host). |
WithAllowPrivateIPsForAllowedHosts() |
(disabled) | Allows private IPs specifically for explicitly allowed hosts. |
WithoutForwardCookies() |
(forwarding enabled) | Disables forwarding Set-Cookie headers from fragment responses. |
WithIncludeErrorMarker(string) |
"" |
HTML placeholder rendered when an include fails without fallback. |
WithPreserveETag() |
(disabled) | Preserves downstream ETag (weakened) and Last-Modified headers. |
WithMetrics(reg) |
nil |
Prometheus registerer for fragment metrics. |
WithLogger(logger) |
slog.Default() |
Logger instance (*slog.Logger). |
WithHTTPClient(client) |
SSRF-safe client | Custom *http.Client for outbound requests. |
Protocol Helpers
Helpers for upstream capability negotiation and downstream response reconciliation defined in the Edge Side Includes (ESI) specification and RFC 9110:
proc.AddSurrogateCapability(header http.Header, deviceToken string): AdvertisesSurrogate-Capability: <deviceToken>="ESI/1.0"to upstream origin servers (nil-safe no-op).proc.CanProcess(header http.Header) bool: Reports whether response headers meet ESI processing requirements based onWithHeaderRequired.proc.ShouldPreserveETag() bool: Reports whether downstream ETag (weakened) and Last-Modified headers are preserved based onWithPreserveETag.proc.ReconcileHeaders(header http.Header, result *Result): Modifies response headers in-place according to ESI specifications (removesSurrogate-Control, adjustsETagandLast-ModifiedperWithPreserveETag, updatesContent-Length, and appends fragmentSet-Cookieheaders).
Memory Management
Call result.Release() after reading result.Body() to return the buffer to the pool:
fragments := esi.Scan(body)
if len(fragments) > 0 {
result, err := proc.ProcessFragments(ctx, req, body, fragments)
if err != nil {
return err
}
defer result.Release()
proc.ReconcileHeaders(w.Header(), result)
w.Write(result.Body())
}
Documentation
¶
Index ¶
- Variables
- func Scan(b []byte) []*proto.EsiFragment
- type InternalFetcherFunc
- type Option
- func WithAllowPrivateIPs() Option
- func WithAllowPrivateIPsForAllowedHosts() Option
- func WithAllowedHosts(hosts ...string) Option
- func WithHTTPClient(client *http.Client) Option
- func WithHeaderRequired() Option
- func WithIncludeErrorMarker(marker string) Option
- func WithInternalFetcher(fetcher InternalFetcherFunc) Option
- func WithLogger(logger *slog.Logger) Option
- func WithMaxConcurrentRequests(limit int) Option
- func WithMaxDepth(depth uint32) Option
- func WithMaxResponseSize(size int64) Option
- func WithMaxTimeout(timeout time.Duration) Option
- func WithMetrics(reg prometheus.Registerer) Option
- func WithPreserveETag() Option
- func WithoutForwardCookies() Option
- type Processor
- func (p *Processor) AddSurrogateCapability(h http.Header, deviceToken string)
- func (p *Processor) CanProcess(h http.Header) bool
- func (p *Processor) ProcessFragments(ctx context.Context, parentReq *http.Request, parentBody []byte, ...) (*Result, error)
- func (p *Processor) ReconcileHeaders(h http.Header, res *Result)
- func (p *Processor) ShouldPreserveETag() bool
- type Result
Constants ¶
This section is empty.
Variables ¶
var ( // ErrInvalidOption is returned when an invalid configuration option is provided to NewProcessor. ErrInvalidOption = errors.New("esi: invalid option") // ErrFallbackToHTTP is returned by an InternalFetcher to signal that the processor should resolve this include via outbound HTTP. ErrFallbackToHTTP = errors.New("esi: fallback to outbound http") )
Functions ¶
func Scan ¶
func Scan(b []byte) []*proto.EsiFragment
Scan inspects the HTML byte slice for ESI tags and extracts pre-compiled fragment metadata. It returns a slice of EsiFragment descriptors if any ESI directives exist, or nil if none are found.
Types ¶
type InternalFetcherFunc ¶
type InternalFetcherFunc func(ctx context.Context, targetPath string, r *http.Request) ([]byte, http.Header, error)
InternalFetcherFunc defines the signature for resolving in-process ESI includes.
func HandlerFetcher ¶
func HandlerFetcher(router http.Handler) InternalFetcherFunc
HandlerFetcher adapts any standard http.Handler into an InternalFetcherFunc for in-process subrequests.
type Option ¶
type Option func(*config) error
Option configures ESI processor parameters.
func WithAllowPrivateIPs ¶
func WithAllowPrivateIPs() Option
WithAllowPrivateIPs permits external include fetches to dial private, loopback, and link-local CIDRs. By default, SSRF guards block all private network addresses.
func WithAllowPrivateIPsForAllowedHosts ¶
func WithAllowPrivateIPsForAllowedHosts() Option
WithAllowPrivateIPsForAllowedHosts permits private IPs specifically for explicitly allowed hosts.
func WithAllowedHosts ¶
WithAllowedHosts restricts external HTTP includes to matching domain patterns (default: empty = all public).
func WithHTTPClient ¶ added in v0.2.0
WithHTTPClient configures a custom http.Client for outbound fragment fetching.
func WithHeaderRequired ¶
func WithHeaderRequired() Option
WithHeaderRequired requires origin responses to explicitly advertise ESI capability via Surrogate-Control: content="ESI/1.0" before fragment tags are processed. By default, ESI tags are processed unconditionally.
func WithIncludeErrorMarker ¶
WithIncludeErrorMarker configures an HTML placeholder rendered on unhandled fetch errors.
func WithInternalFetcher ¶
func WithInternalFetcher(fetcher InternalFetcherFunc) Option
WithInternalFetcher configures a custom in-process handler for resolving local fragment subrequests.
func WithLogger ¶ added in v0.2.0
WithLogger configures a custom slog.Logger for ESI processing.
func WithMaxConcurrentRequests ¶
WithMaxConcurrentRequests caps concurrent fragment fetch goroutines per document (default: 8).
func WithMaxDepth ¶
WithMaxDepth configures the maximum global recursion depth for nested includes (default: 3).
func WithMaxResponseSize ¶
WithMaxResponseSize caps the maximum allowed fragment body size in bytes (default: 10MB, 0 = unlimited).
func WithMaxTimeout ¶
WithMaxTimeout configures the maximum fetch timeout for resolving an include fragment (default: 30s).
func WithMetrics ¶ added in v0.2.0
func WithMetrics(reg prometheus.Registerer) Option
WithMetrics registers ESI telemetry collectors with the provided Prometheus Registerer.
func WithPreserveETag ¶ added in v0.2.0
func WithPreserveETag() Option
WithPreserveETag preserves downstream ETag (weakened to W/"...") and Last-Modified on ESI documents. By default, ETag and Last-Modified are stripped to guarantee fresh dynamic fragment evaluation.
func WithoutForwardCookies ¶ added in v0.3.0
func WithoutForwardCookies() Option
WithoutForwardCookies disables forwarding Set-Cookie headers from fragment subrequests to downstream clients. By default, fragment cookies are forwarded.
type Processor ¶ added in v0.2.0
type Processor struct {
// contains filtered or unexported fields
}
Processor orchestrates ESI fragment fetching, recursion handling, error fallbacks, and document splicing.
func NewProcessor ¶ added in v0.2.0
NewProcessor constructs a new ESI Processor from the provided options.
func (*Processor) AddSurrogateCapability ¶ added in v0.2.0
AddSurrogateCapability appends an ESI/1.0 capability token for the specified deviceToken to the request's Surrogate-Capability header (e.g. `deviceToken="ESI/1.0"`). If p is nil or h is nil, this is a no-op. If deviceToken is empty, "esi" is used.
func (*Processor) CanProcess ¶ added in v0.2.0
CanProcess reports whether the response headers satisfy ESI processing criteria. By default, it returns true; when WithHeaderRequired() is configured, it verifies that Surrogate-Control contains "ESI/1.0".
func (*Processor) ProcessFragments ¶ added in v0.2.0
func (p *Processor) ProcessFragments( ctx context.Context, parentReq *http.Request, parentBody []byte, fragments []*proto.EsiFragment, ) (*Result, error)
ProcessFragments resolves all ESI fragments in parentBody, fetches includes concurrently, splices the results, and returns an assembled Result.
func (*Processor) ReconcileHeaders ¶ added in v0.2.0
ReconcileHeaders updates h in-place per ESI (§3.2) and RFC 9110 specifications: - Removes Surrogate-Control header - Weakens or removes ETag and Last-Modified according to PreserveETag - Updates Content-Length if present to match the spliced body length - Appends fragment Set-Cookie headers from res
If the original headers must be preserved, call h.Clone() before passing.
func (*Processor) ShouldPreserveETag ¶ added in v0.2.0
ShouldPreserveETag reports whether downstream ETag (weakened) and Last-Modified headers are preserved.
type Result ¶ added in v0.2.0
type Result struct {
// contains filtered or unexported fields
}
Result contains the assembled document and metadata resulting from ESI processing. Callers must invoke Release when finished to return the underlying buffer to the memory pool.
func (*Result) Body ¶ added in v0.2.0
Body returns the assembled document byte slice. The slice is valid only until Release is called.
Source Files