engines

package
v2.0.4 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jul 3, 2026 License: Apache-2.0 Imports: 50 Imported by: 0

Documentation

Overview

Package engines provides the Reverse Proxy, Reverse Proxy Cache, and Time Series Delta Proxy Cache features for use by handlers.

Index

Constants

View Source 
const (
	// RevalStatusNone indicates the object will not undergo revalidation against the origin
	RevalStatusNone = RevalidationStatus(iota)
	// RevalStatusInProgress indicates the object is currently being revalidated against the origin
	RevalStatusInProgress
	// RevalStatusLocal is used during a cache Range Miss, and indicates that while the user-requested ranges
	// for an object are uncached and must be fetched from the origin, other ranges for the object are cached
	// but require revalidation. When a request is in this state, Trickster will use the response headers from
	// the range miss to locally revalidate the cached content instead of making a separate request to the
	// origin for revalidating the cached ranges.
	RevalStatusLocal
	// RevalStatusOK indicates the object was successfully revalidated against the origin and is still fresh
	RevalStatusOK
	// RevalStatusFailed indicates the origin returned a new object for the URL to replace the cached version
	RevalStatusFailed
)
View Source 
const ClockOffsetWarning = "clock offset between trickster host and origin is high and may cause data anomalies"

ClockOffsetWarning is the warning provided to users when the origin's clock offset is suspect

View Source 
const HTTPBlockSize = 32 * 1024

HTTPBlockSize represents 32K of bytes

Variables

This section is empty.

Functions

func CheckIfNoneMatch 

func CheckIfNoneMatch(etag string, headerValue string, ls status.LookupStatus) bool

CheckIfNoneMatch determines if the provided match value satisfies an "If-None-Match" condition against the cached object. As Trickster is a cache, matching is always weak.

func ComposeCacheKey added in v2.0.2 

func ComposeCacheKey(name, prefix, engine, suffix string) string

ComposeCacheKey assembles a per-backend cache key. The name prefix isolates keys when cache_name is shared across backends. engine is "" for plain HTTP proxy, "opc" for object cache, "dpc" for delta proxy cache.

func DeltaProxyCacheRequest 

func DeltaProxyCacheRequest(w http.ResponseWriter, r *http.Request, modeler *timeseries.Modeler)

DeltaProxyCacheRequest identifies the gaps between the cache and a new timeseries request, requests the gaps from the origin server and returns the reconstituted dataset to the downstream request while caching the results for subsequent requests of the same data

func DoProxy 

func DoProxy(w io.Writer, r *http.Request, closeResponse bool) *http.Response

DoProxy proxies an inbound request to its corresponding upstream origin with no caching features

func FetchViaObjectProxyCache 

func FetchViaObjectProxyCache(r *http.Request) ([]byte, *http.Response, bool)

FetchViaObjectProxyCache Fetches an object from Cache or Origin (on miss), writes the object to the cache, and returns the object to the caller

func ObjectProxyCacheRequest 

func ObjectProxyCacheRequest(w http.ResponseWriter, r *http.Request)

ObjectProxyCacheRequest provides a Basic HTTP Reverse Proxy/Cache

func PrepareFetchReader 

func PrepareFetchReader(r *http.Request) (io.ReadCloser, *http.Response, int64)

PrepareFetchReader prepares an http response and returns io.ReadCloser to provide the response data, the response object and the content length. Used in Fetch.

func PrepareResponseWriter 

func PrepareResponseWriter(w io.Writer, code int, header http.Header) io.Writer

PrepareResponseWriter prepares a response and returns a destination io.Writer for the payload Used in Respond.

Hop-by-hop headers named in the upstream's Connection header (and the static HopHeaders set) are stripped from the downstream response before write, per RFC 7230 6.1. Without this, an upstream that emitted `Connection: X-Internal-Auth` plus `X-Internal-Auth: ...` would leak the hop-only header to the client. Mirrors the request-side strip applied by PrepareFetchReader via headers.StripClientHeaders.

func Respond 

func Respond(w io.Writer, code int, header http.Header, body io.Reader)

Respond sends an HTTP Response down to the requesting client

func WriteCache 

func WriteCache(ctx context.Context, c cache.Cache, key string, d *HTTPDocument,
	ttl time.Duration, compressTypes sets.Set[string], marshal timeseries.MarshalerFunc,
) error

WriteCache writes an HTTPDocument to the cache

Types

type ByterangeChunkQueryIterator 

type ByterangeChunkQueryIterator struct {
	// contains filtered or unexported fields
}

ByterangeChunkQueryIterator implements ChunkQueryIterator for byterange chunks (reading)

func (*ByterangeChunkQueryIterator) IterateChunks 

func (bci *ByterangeChunkQueryIterator) IterateChunks(fn func(int, string) bool)

type ByterangeChunkQueryProcessor 

type ByterangeChunkQueryProcessor struct {
	// contains filtered or unexported fields
}

ByterangeChunkQueryProcessor implements ChunkQueryProcessor for byterange chunks (reading)

func (*ByterangeChunkQueryProcessor) Finalize 

func (bcp *ByterangeChunkQueryProcessor) Finalize() error

func (*ByterangeChunkQueryProcessor) ProcessChunk 

func (bcp *ByterangeChunkQueryProcessor) ProcessChunk(index int, subkey string, qr *queryResult, c cache.Cache) error

type ByterangeChunkWriter 

type ByterangeChunkWriter struct {
	// contains filtered or unexported fields
}

ByterangeChunkWriter handles byterange chunking operations when writing to cache

func NewByterangeChunkWriter 

func NewByterangeChunkWriter(c cache.Cache, key string, d CacheableDocument) *ByterangeChunkWriter

NewByterangeChunkWriter creates a new byterange chunk writer

func (*ByterangeChunkWriter) ChunkCount 

func (bc *ByterangeChunkWriter) ChunkCount() int

func (*ByterangeChunkWriter) GetMeta 

func (bc *ByterangeChunkWriter) GetMeta(d CacheableDocument) any

func (*ByterangeChunkWriter) IterateChunks 

func (bc *ByterangeChunkWriter) IterateChunks(
	d CacheableDocument,
	writeFunc func(int, string, any) error,
) error

type CacheableDocument 

type CacheableDocument interface {
	// GetMeta returns the metadata document that acts as a manifest for all chunks.
	GetMeta() *HTTPDocument

	// GetTimeseriesChunk retrieves the specific chunk of data for a given time extent.
	GetTimeseriesChunk(extent timeseries.Extent) *HTTPDocument

	// GetByterangeChunk retrieves the specific chunk of data for a given byte range.
	GetByterangeChunk(dataRange byterange.Range, size int64) *HTTPDocument
	// contains filtered or unexported methods
}

CacheableDocument abstracts the source data for chunking operations when writing to cache.

type CachingPolicy 

type CachingPolicy struct {
	IsFresh              bool `msg:"is_fresh"`
	NoCache              bool `msg:"nocache"`
	NoTransform          bool `msg:"notransform"`
	CanRevalidate        bool `msg:"can_revalidate"`
	MustRevalidate       bool `msg:"must_revalidate"`
	IsNegativeCache      bool `msg:"is_negative_cache"`
	IsClientConditional  bool `msg:"-"`
	IsClientFresh        bool `msg:"-"`
	HasIfModifiedSince   bool `msg:"-"`
	HasIfUnmodifiedSince bool `msg:"-"`
	HasIfNoneMatch       bool `msg:"-"`
	IfNoneMatchResult    bool `msg:"-"`

	FreshnessLifetime int `msg:"freshness_lifetime"`

	LastModified time.Time `msg:"last_modified"`
	Expires      time.Time `msg:"expires"`
	Date         time.Time `msg:"date"`
	LocalDate    time.Time `msg:"local_date"`

	ETag string `msg:"etag"`

	IfNoneMatchValue      string    `msg:"-"`
	IfModifiedSinceTime   time.Time `msg:"-"`
	IfUnmodifiedSinceTime time.Time `msg:"-"`
}

CachingPolicy defines the attributes for determining the cachability of an HTTP object

func GetRequestCachingPolicy 

func GetRequestCachingPolicy(h http.Header) *CachingPolicy

GetRequestCachingPolicy examines HTTP request headers for caching headers and true if the corresponding response is OK to cache

func GetResponseCachingPolicy 

func GetResponseCachingPolicy(code int, negativeCache map[int]time.Duration, h http.Header) *CachingPolicy

GetResponseCachingPolicy examines HTTP response headers for caching headers a returns a CachingPolicy reference

func (*CachingPolicy) Clone 

func (cp *CachingPolicy) Clone() *CachingPolicy

Clone returns an exact copy of the Caching Policy

func (*CachingPolicy) DecodeMsg 

func (z *CachingPolicy) DecodeMsg(dc *msgp.Reader) (err error)

DecodeMsg implements msgp.Decodable

func (*CachingPolicy) EncodeMsg 

func (z *CachingPolicy) EncodeMsg(en *msgp.Writer) (err error)

EncodeMsg implements msgp.Encodable

func (*CachingPolicy) MarshalMsg 

func (z *CachingPolicy) MarshalMsg(b []byte) (o []byte, err error)

MarshalMsg implements msgp.Marshaler

func (*CachingPolicy) Merge 

func (cp *CachingPolicy) Merge(src *CachingPolicy)

Merge merges the source CachingPolicy into the subject CachingPolicy

func (*CachingPolicy) Msgsize 

func (z *CachingPolicy) Msgsize() (s int)

Msgsize returns an upper bound estimate of the number of bytes occupied by the serialized message

func (*CachingPolicy) ParseClientConditionals 

func (cp *CachingPolicy) ParseClientConditionals()

ParseClientConditionals inspects the client http request to determine if it includes any conditions

func (*CachingPolicy) ResetClientConditionals 

func (cp *CachingPolicy) ResetClientConditionals()

ResetClientConditionals sets the request-specific conditional values of the subject caching policy to false, so as to facilitate reuse of the policy with subsequent requests for the same cache object

func (*CachingPolicy) ResolveClientConditionals 

func (cp *CachingPolicy) ResolveClientConditionals(ls status.LookupStatus)

ResolveClientConditionals ensures any client conditionals are handled before responding to the client request

func (*CachingPolicy) String 

func (cp *CachingPolicy) String() string

func (*CachingPolicy) TTL 

func (cp *CachingPolicy) TTL(multiplier float64, maxDur time.Duration) time.Duration

TTL returns a TTL based on the subject caching policy and the provided multiplier and max values

func (*CachingPolicy) UnmarshalMsg 

func (z *CachingPolicy) UnmarshalMsg(bts []byte) (o []byte, err error)

UnmarshalMsg implements msgp.Unmarshaler

type ChunkQueryIterator 

type ChunkQueryIterator interface {
	// IterateChunks calls the provided function for each chunk
	// The function receives (index, subkey) and should return whether to continue
	IterateChunks(func(int, string) bool)
}

ChunkQueryIterator provides iteration over chunks for cache queries (reading)

type ChunkQueryProcessor 

type ChunkQueryProcessor interface {
	// ProcessChunk processes a successful query result for a chunk
	ProcessChunk(index int, subkey string, qr *queryResult, c cache.Cache) error

	// Finalize performs any final processing after all chunks are processed
	Finalize() error
}

ChunkQueryProcessor handles the result of querying a single chunk from cache (reading)

type ChunkWriter 

type ChunkWriter interface {
	// Determine the number of chunks plus one for the metadata document.
	ChunkCount() int

	// Iterate over chunks and execute the provided write function for each.
	// The write function takes (index, subkey, chunkData).
	IterateChunks(
		d CacheableDocument,
		writeFunc func(int, string, any) error,
	) error

	// GetMeta returns the metadata document to be stored separately.
	GetMeta(d CacheableDocument) any
}

ChunkWriter abstracts the logic for different types of chunking when writing to cache.

type HTTPDocument 

type HTTPDocument struct {
	IsMeta        bool                `msg:"is_meta"`
	IsChunk       bool                `msg:"is_chunk"`
	StatusCode    int                 `msg:"status_code"`
	Status        string              `msg:"status"`
	Headers       map[string][]string `msg:"headers"`
	Body          []byte              `msg:"body"`
	ContentLength int64               `msg:"content_length"`
	ContentType   string              `msg:"content_type"`
	CachingPolicy *CachingPolicy      `msg:"caching_policy"`
	// Ranges is the list of Byte Ranges contained in the body of this document
	Ranges     byterange.Ranges              `msg:"ranges"`
	RangeParts byterange.MultipartByteRanges `msg:"-"`
	// StoredRangeParts is a version of RangeParts that can be exported to MessagePack
	StoredRangeParts map[string]*byterange.MultipartByteRange `msg:"range_parts"`
	// contains filtered or unexported fields
}

HTTPDocument represents a full HTTP Response/Cache Document with unbuffered body

func DocumentFromHTTPResponse 

func DocumentFromHTTPResponse(resp *http.Response, body []byte,
	cp *CachingPolicy,
) *HTTPDocument

DocumentFromHTTPResponse returns an HTTPDocument from the provided HTTP Response and Body

func QueryCache 

func QueryCache(ctx context.Context, c cache.Cache, key string,
	ranges byterange.Ranges, unmarshal timeseries.UnmarshalerFunc,
) (*HTTPDocument, status.LookupStatus, byterange.Ranges, error)

QueryCache queries the cache for an HTTPDocument and returns it

func (*HTTPDocument) DecodeMsg 

func (z *HTTPDocument) DecodeMsg(dc *msgp.Reader) (err error)

DecodeMsg implements msgp.Decodable

func (*HTTPDocument) EncodeMsg 

func (z *HTTPDocument) EncodeMsg(en *msgp.Writer) (err error)

EncodeMsg implements msgp.Encodable

func (*HTTPDocument) FulfillContentBody 

func (d *HTTPDocument) FulfillContentBody() error

FulfillContentBody will concatenate the document's Range parts into a single, full content body the caller must know that document's multipart ranges include full content length before calling this method

func (*HTTPDocument) GetByterangeChunk 

func (d *HTTPDocument) GetByterangeChunk(chunkRange byterange.Range, _ int64) *HTTPDocument

func (*HTTPDocument) GetMeta 

func (d *HTTPDocument) GetMeta() *HTTPDocument

func (*HTTPDocument) GetTimeseriesChunk 

func (d *HTTPDocument) GetTimeseriesChunk(chunkExtent timeseries.Extent) *HTTPDocument

func (*HTTPDocument) LoadRangeParts 

func (d *HTTPDocument) LoadRangeParts()

LoadRangeParts convert a StoredRangeParts into a RangeParts

func (*HTTPDocument) MarshalMsg 

func (z *HTTPDocument) MarshalMsg(b []byte) (o []byte, err error)

MarshalMsg implements msgp.Marshaler

func (*HTTPDocument) Msgsize 

func (z *HTTPDocument) Msgsize() (s int)

Msgsize returns an upper bound estimate of the number of bytes occupied by the serialized message

func (*HTTPDocument) ParsePartialContentBody 

func (d *HTTPDocument) ParsePartialContentBody(resp *http.Response,
	body []byte,
)

ParsePartialContentBody parses a Partial Content response body into 0 or more discrete parts

func (*HTTPDocument) SafeHeaderClone 

func (d *HTTPDocument) SafeHeaderClone() http.Header

SafeHeaderClone returns a threadsafe copy of the Document Header

func (*HTTPDocument) SetBody 

func (d *HTTPDocument) SetBody(body []byte)

SetBody sets the Document Body as well as the Content Length, based on the length of body. This assumes that the caller has checked that the request is not a Range request

func (*HTTPDocument) ShallowCopy added in v2.0.2 

func (d *HTTPDocument) ShallowCopy() *HTTPDocument

ShallowCopy returns a shallow copy of the HTTPDocument with a fresh headerLock.

func (*HTTPDocument) Size 

func (d *HTTPDocument) Size() int

Size returns the size of the HTTPDocument's headers, CachingPolicy, RangeParts, Body and timeseries data

func (*HTTPDocument) UnmarshalMsg 

func (z *HTTPDocument) UnmarshalMsg(bts []byte) (o []byte, err error)

UnmarshalMsg implements msgp.Unmarshaler

type IndexReader 

type IndexReader func(index uint64, b []byte) (int, error)

IndexReader implements a reader to read data at a specific index into slice b

type ProgressiveCollapseForwarder 

type ProgressiveCollapseForwarder interface {
	AddClient(io.Writer) error
	Write([]byte) (int, error)
	Close()
	IndexRead(uint64, []byte) (int, error)
	WaitServerComplete()
	WaitAllComplete()
	GetBody() ([]byte, error)
	GetResp() *http.Response
}

ProgressiveCollapseForwarder accepts data written through the io.Writer interface, caches it and makes all the data written available to n readers. The readers can request data at index i, to which the PCF may block or return the data immediately.

func NewPCF 

func NewPCF(resp *http.Response, contentLength int64) ProgressiveCollapseForwarder

NewPCF returns a new instance of a ProgressiveCollapseForwarder

type RevalidationStatus 

type RevalidationStatus int

RevalidationStatus enumerates the possible statuses for cache revalidation

func (RevalidationStatus) String 

func (s RevalidationStatus) String() string

type TimeseriesChunkQueryIterator 

type TimeseriesChunkQueryIterator struct {
	// contains filtered or unexported fields
}

TimeseriesChunkQueryIterator implements ChunkQueryIterator for timeseries chunks (reading)

func (*TimeseriesChunkQueryIterator) IterateChunks 

func (tci *TimeseriesChunkQueryIterator) IterateChunks(fn func(int, string) bool)

type TimeseriesChunkQueryProcessor 

type TimeseriesChunkQueryProcessor struct {
	// contains filtered or unexported fields
}

TimeseriesChunkQueryProcessor implements ChunkQueryProcessor for timeseries chunks (reading)

func (*TimeseriesChunkQueryProcessor) Finalize 

func (tcp *TimeseriesChunkQueryProcessor) Finalize() error

func (*TimeseriesChunkQueryProcessor) ProcessChunk 

func (tcp *TimeseriesChunkQueryProcessor) ProcessChunk(index int, subkey string, qr *queryResult, c cache.Cache) error

type TimeseriesChunkWriter 

type TimeseriesChunkWriter struct {
	// contains filtered or unexported fields
}

TimeseriesChunkWriter handles timeseries chunking operations when writing to cache

func NewTimeseriesChunkWriter 

func NewTimeseriesChunkWriter(c cache.Cache, key string, trq *timeseries.TimeRangeQuery, marshal timeseries.MarshalerFunc) *TimeseriesChunkWriter

NewTimeseriesChunkWriter creates a new TimeseriesChunkWriter

func (*TimeseriesChunkWriter) ChunkCount 

func (tc *TimeseriesChunkWriter) ChunkCount() int

func (*TimeseriesChunkWriter) GetMeta 

func (tc *TimeseriesChunkWriter) GetMeta(d CacheableDocument) any

func (*TimeseriesChunkWriter) IterateChunks 

func (tc *TimeseriesChunkWriter) IterateChunks(
	d CacheableDocument,
	writeFunc func(int, string, any) error,
) error

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.