diskcache

package module
v0.1.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Dec 14, 2020 License: MIT Imports: 30 Imported by: 8

README

diskcache GoDoc

Package diskcache provides a standard Go HTTP transport (http.RoundTripper) implementation able to cache, minify, compress, and transform HTTP responses on disk. Allows definition of caching policies on a per-method, per-host, or per-path basis.

Provides a number of content transformers that alter cached HTTP response headers and bodies (prior to storage on disk) including rewriting or white/black-listing headers, stripping XSS prefixes, Base64 encoding/decoding, webpage content minification, Gzip/Zlib compression, and more.

Package diskcache does not aim to work as a on-disk HTTP proxy. See github.com/gregjones/httpcache for a HTTP transport implementation that provides a RFC 7234 compliant cache.

Example

A basic Go example:

// _example/example.go
package main

import (
	"fmt"
	"log"
	"net/http"
	"net/http/httputil"
	"os"
	"time"

	// "github.com/spf13/afero"
	"github.com/kenshaw/diskcache"
)

func main() {
	d, err := diskcache.New(
		// diskcache.WithFs(afero.New*(...)),
		diskcache.WithTTL(365*24*time.Hour),
		diskcache.WithHeaderWhitelist("Date", "Set-Cookie", "Content-Type"),
		diskcache.WithHeaderTransform(
			`Date:\s+(.+?)`, `Date: Not "$1"`,
		),
		diskcache.WithMinifier(),
		diskcache.WithErrorTruncator(),
		diskcache.WithGzipCompression(),
	)
	if err != nil {
		log.Fatal(err)
	}

	// create client using diskcache as the transport
	cl := &http.Client{
		Transport: d,
	}

	for i, urlstr := range []string{
		"https://github.com/kenshaw/diskcache",      // a path that exists
		"https://github.com/kenshaw/does-not-exist", // a path that doesn't
		// repeat
		"https://github.com/kenshaw/diskcache",
		"https://github.com/kenshaw/does-not-exist",
	} {
		if err := grab(cl, "GET", urlstr, i); err != nil {
			log.Fatal(err)
		}
	}
}

func grab(cl *http.Client, method, urlstr string, id int) error {
	req, err := http.NewRequest(method, urlstr, nil)
	if err != nil {
		return err
	}

	// execute request
	res, err := cl.Do(req)
	if err != nil {
		return err
	}

	fmt.Fprintf(os.Stdout, "------------------- %s %s (%d) -------------------\n", method, urlstr, id)
	buf, err := httputil.DumpResponse(res, true)
	if err != nil {
		return err
	}
	if _, err = os.Stdout.Write(buf); err != nil {
		return err
	}
	fmt.Fprintf(os.Stdout, "\n------------------- END %s %s (%d) -------------------\n\n", method, urlstr, id)
	return nil
}

See the GoDoc listing for more examples.

Notes

Prior to writing diskcache, a number of the available HTTP transports were were investigated to see if they could met the specific needs that diskcache was designed for. In fact, it was found that many (notably httpcache), it was possible to accomplish what diskcache was designed for. However, it required additional dependencies on non-standard (or uncommon) packages, layering disk storage, or overcoming additional technical hurdles or obstacles such as non-idiomatic code/design.

In short, no other packages were deemed to work "out-of-the-box", and thus diskcache was born. diskcache has been made public in case it proves useful to others.

diskcache SHOULD NOT be considered ready for general use in production by others at this time. If you need a tried, tested, and true caching transport layer for Go, please use httpcache.

Documentation

Overview

Package diskcache provides a http.RoundTripper implementation that can minify, compress, and cache HTTP responses retrieved using a standard http.Client on disk. Provides ability to define custom retention and storage policies depending on the host, path, or other URL components.

Package diskcache does not aim to work as a on-disk HTTP proxy -- see github.com/gregjones/httpcache for a HTTP transport (http.RoundTripper) implementation that provides a RFC 7234 compliant cache.

See _example/example.go for a more complete example.

Index

Examples

Constants

This section is empty.

Variables

This section is empty.

Functions

This section is empty.

Types

type Base64Decoder 

type Base64Decoder struct {
	Priority     TransformPriority
	ContentTypes []string
	Encoding     *base64.Encoding
}

Base64Decoder is a body transformer that base64 decodes the body.

func (Base64Decoder) BodyTransform 

func (t Base64Decoder) BodyTransform(w io.Writer, r io.Reader, urlstr string, code int, contentType string) (bool, error)

BodyTransform satisfies the BodyTransformer interface.

func (Base64Decoder) TransformPriority 

func (t Base64Decoder) TransformPriority() TransformPriority

TransformPriority satisfies the BodyTransformer interface.

type BodyTransformer 

type BodyTransformer interface {
	// TransformPriority returns the order for the transformer.
	TransformPriority() TransformPriority

	// BodyTransform mangles data from r to w for the provided URL, status
	// code, and content type. A return of false prevents further passing the
	// stream to lower priority body transformers.
	BodyTransform(w io.Writer, r io.Reader, urlstr string, code int, contentType string) (bool, error)
}

BodyTransformer is the shared interface for mangling body content prior to storage in the fs.

type Cache 

type Cache struct {
	// contains filtered or unexported fields
}

Cache is a http.RoundTripper compatible disk cache.

func New 

func New(opts ...Option) (*Cache, error)

New creates a new disk cache.

By default, the cache path will be <working directory>/cache. Change location using options.

Example

ExampleNew demonstrates setting up a simple diskcache for use with a http.Client. 

package main

import (
	"bytes"
	"fmt"
	"net/http"
	"net/http/httptest"
	"net/http/httputil"
	"time"

	"github.com/kenshaw/diskcache"
)

func main() {
	// set up simple test server for demonstration
	s := httptest.NewServer(http.HandlerFunc(func(res http.ResponseWriter, req *http.Request) {
		res.Header().Set("content-type", "text/html")
		res.Header().Set("X-Header", "test")
		fmt.Fprintf(res, `<!doctype html>
		<html lang="en">
			<head>
			</head>
			<body attribute="value">
				<p> hello %s! </p>
				<div> something </div>
				<a href="http://example.com/full/path">a link!</a>
			</body>
		</html>
`, req.URL.Query().Get("name"))
	}))
	defer s.Close()

	// create disk cache
	d, err := diskcache.New(
		// diskcache.WithBasePathFs("/path/to/cacheDir"),
		diskcache.WithHeaderBlacklist("Set-Cookie", "Date"),
		diskcache.WithMinifier(),
		diskcache.WithErrorTruncator(),
		diskcache.WithGzipCompression(),
		diskcache.WithTTL(365*24*time.Hour),
	)
	if err != nil {
		panic(err)
	}

	// build and execute request
	cl := &http.Client{Transport: d}
	req, err := http.NewRequest("GET", s.URL+"/hello?name=ken", nil)
	if err != nil {
		panic(err)
	}
	res, err := cl.Do(req)
	if err != nil {
		panic(err)
	}
	defer res.Body.Close()

	// dump
	buf, err := httputil.DumpResponse(res, true)
	if err != nil {
		panic(err)
	}
	fmt.Println(string(bytes.Replace(buf, []byte("\r\n"), []byte("\n"), -1)))

}

Output:

HTTP/1.1 200 OK
Connection: close
Content-Type: text/html
X-Header: test

<!doctype html><html lang=en><body attribute=value><p>hello ken!<div>something</div><a href=//example.com/full/path>a link!</a>

func (*Cache) Cached 

func (c *Cache) Cached(req *http.Request) (bool, error)

Cached returns whether or not the request is cached or not. Wraps Match, Stale.

func (*Cache) Evict 

func (c *Cache) Evict(req *http.Request) error

Evict forces a cache eviction (deletion) for the key matching the request.

func (*Cache) EvictKey 

func (c *Cache) EvictKey(key string) error

Evict forces a cache eviction (deletion) of the specified key.

func (*Cache) Exec 

func (c *Cache) Exec(key string, p Policy, req *http.Request) (*http.Response, error)

Exec executes the request storing the response using the provided key and cache policy. Applies header and body transformers, before marshaling and the response.

func (*Cache) Load 

func (c *Cache) Load(key string, p Policy, req *http.Request) (*http.Response, error)

Load unmarshals and loads the cached response for the provided key and cache policy.

func (*Cache) Match 

func (c *Cache) Match(req *http.Request) (string, Policy, error)

Match finds the first matching cache policy for the request.

func (*Cache) RoundTrip 

func (c *Cache) RoundTrip(req *http.Request) (*http.Response, error)

RoundTrip satisfies the http.RoundTripper interface.

func (*Cache) Stale 

func (c *Cache) Stale(key string, ttl time.Duration) (bool, error)

Stale indicates whether or not the key is stale, based on the passed ttl.

type FlatMarshalUnmarshaler 

type FlatMarshalUnmarshaler struct {
	// Chain is an additional MarshalUnmarshaler that the data can be sent to
	// prior to storage on disk, but after the header has been stripped.
	Chain MarshalUnmarshaler
}

FlatMarhsalUnmarshaler is a flat file marshaler/unmarshaler, dropping original response header when marshaling.

func (FlatMarshalUnmarshaler) Marshal 

func (z FlatMarshalUnmarshaler) Marshal(w io.Writer, r io.Reader) error

Marshal satisfies the MarshalUnmarshaler interface.

func (FlatMarshalUnmarshaler) Unmarshal 

func (z FlatMarshalUnmarshaler) Unmarshal(w io.Writer, r io.Reader) error

Unmarshal satisfies the MarshalUnmarshaler interface.

type GzipMarshalUnmarshaler 

type GzipMarshalUnmarshaler struct {
	// Level is the compression level.
	Level int
}

GzipMarshalUnmarshaler is a gzip mashaler/unmarshaler.

func (GzipMarshalUnmarshaler) Marshal 

func (z GzipMarshalUnmarshaler) Marshal(w io.Writer, r io.Reader) error

Marshal satisfies the MarshalUnmarshaler interface.

func (GzipMarshalUnmarshaler) Unmarshal 

func (z GzipMarshalUnmarshaler) Unmarshal(w io.Writer, r io.Reader) error

Unmarshal satisfies the MarshalUnmarshaler interface.

type HeaderTransformer 

type HeaderTransformer interface {
	HeaderTransform([]byte) []byte
}

HeaderTransformer is the shared interface for modifying/altering headers prior to storage on disk.

type HeaderTransformerFunc 

type HeaderTransformerFunc func([]byte) []byte

HeaderTransformerFunc is a header rewriter func.

func (HeaderTransformerFunc) HeaderTransform 

func (f HeaderTransformerFunc) HeaderTransform(buf []byte) []byte

HeaderTransformer satisfies the HeaderTransformer interface.

type MarshalUnmarshaler 

type MarshalUnmarshaler interface {
	Marshal(w io.Writer, r io.Reader) error
	Unmarshal(w io.Writer, r io.Reader) error
}

MarshalUnmarshaler is the shared interface for marshaling/unmarshaling.

type Matcher 

type Matcher interface {
	// Match matches the passed request, returning the key and ttl.
	Match(*http.Request) (string, Policy, error)
}

Matcher is the shared interface for retrivieving a disk cache policy for requests.

type Minifier 

type Minifier struct {
	Priority TransformPriority
}

Minifier is a body transformer that performs that minifies HTML, XML, SVG, JavaScript, JSON, and CSS content.

See: github.com/tdewolff/minify

func (Minifier) BodyTransform 

func (t Minifier) BodyTransform(w io.Writer, r io.Reader, urlstr string, code int, contentType string) (bool, error)

BodyTransform satisfies the BodyTransformer interface.

func (Minifier) TransformPriority 

func (t Minifier) TransformPriority() TransformPriority

TransformPriority satisfies the Transformer interface.

type Option 

type Option interface {
	// contains filtered or unexported methods
}

Option is a disk cache option.

func WithBase64Decoder 

func WithBase64Decoder(contentTypes ...string) Option

WithBase64Decoder is a disk cache option to add a body transformer that does base64 decoding of responses for specific content types.

func WithBasePathFs 

func WithBasePathFs(basePath string) Option

WithBasePathFs is a disk cache option to set the Afero filesystem as an Afero BasePathFs.

func WithBodyTransformers 

func WithBodyTransformers(bodyTransformers ...BodyTransformer) Option

WithBodyTransformers is a disk cache option to set the body transformers.

func WithDefaultMatcher 

func WithDefaultMatcher(method, host, path, key string, opts ...Option) Option

WithDefaultMatcher is a disk cache option to set the default matcher.

func WithErrorTruncator 

func WithErrorTruncator() Option

WithErrorTruncator is a disk cache option to add a body transformer that truncates responses when the HTTP status code != OK (200).

func WithFlatChain 

func WithFlatChain(marshalUnmarshaler MarshalUnmarshaler) Option

WithFlatChain is a disk cache option that marshals/unmarshals responses, removing headers from responses, and chaining marshaling/unmarshaling to a provided marshaler/unmarshaler.

Note: cached responses will not have original headers.

func WithFlatGzipCompression 

func WithFlatGzipCompression() Option

WithFlatGzipCompression is a disk cache option that marshals/unmarshals responses, with headers removed from responses, and with gzip compression.

Note: cached responses will not have original headers.

func WithFlatStorage 

func WithFlatStorage() Option

WithFlatStorage is a disk cache option to set a flat marshaler/unmarshaler removing headers from responses.

Note: cached responses will not have original headers.

func WithFlatZlibCompression 

func WithFlatZlibCompression() Option

WithFlatZlibCompression is a disk cache option that marshals/unmarshals responses, with headers removed from responses, and with zlib compression.

Note: cached responses will not have original headers.

func WithFs 

func WithFs(fs afero.Fs) Option

WithFs is a disk cache option to set the Afero filesystem used.

See: github.com/spf13/afero

func WithGzipCompression 

func WithGzipCompression() Option

WithGzipCompression is a disk cache option to set a gzip marshaler/unmarshaler.

func WithHeaderBlacklist 

func WithHeaderBlacklist(blacklist ...string) Option

WithHeaderBlacklist is a disk cache option to add a header transformer that removes any header in the blacklist.

func WithHeaderTransform 

func WithHeaderTransform(pairs ...string) Option

WithHeaderTransform is a disk cache option to add a header transformer that transforms headers matching the provided regexp pairs and replacements.

func WithHeaderTransformers 

func WithHeaderTransformers(headerTransformers ...HeaderTransformer) Option

WithHeaderTransformers is a disk cache option to set the header transformers.

func WithHeaderWhitelist 

func WithHeaderWhitelist(whitelist ...string) Option

WithHeaderWhitelist is a disk cache option to add a header transformer that removes any header not in the whitelist.

func WithIndexPath 

func WithIndexPath(indexPath string) Option

WithIndexPath is a disk cache option to set the index path name.

func WithLongPathHandler 

func WithLongPathHandler(longPathHandler func(string) string) Option

WithLongPathHandler is a disk cache option to set a long path handler.

func WithMarshalUnmarshaler 

func WithMarshalUnmarshaler(marshalUnmarshaler MarshalUnmarshaler) Option

WithMarshalUnmarshaler is a disk cache option to set a marshaler/unmarshaler.

func WithMatchers 

func WithMatchers(matchers ...Matcher) Option

WithMatchers is a disk cache option to set matchers.

func WithMinifier 

func WithMinifier() Option

WithMinifier is a disk cache option to add a body transformer that does content minification of HTML, XML, SVG, JavaScript, JSON, and CSS data. Useful for reducing disk storage sizes.

See: github.com/tdewolff/minify

func WithMode 

func WithMode(dirMode, fileMode os.FileMode) Option

WithMode is a disk cache option to set the file mode used when creating files and directories on disk.

func WithNoDefault 

func WithNoDefault() Option

WithNoDefault is a disk cache option to disable default matcher.

Prevents propagating settings from default matcher.

func WithPrefixStripper 

func WithPrefixStripper(prefix []byte, contentTypes ...string) Option

WithPrefixStripper is a disk cache option to add a body transformer that strips a specific XSS prefix for a specified content type.

Useful for removing XSS prefixes added to JavaScript or JSON content.

func WithQueryEncoder 

func WithQueryEncoder(queryEncoder func(url.Values) string) Option

WithQueryEncoder is a disk cache option to set the query encoder.

func WithQueryPrefix 

func WithQueryPrefix(prefix string, fields ...string) Option

WithQueryPrefix is a disk cache option that sets a query encoder, that adds the supplied prefix to non-empty and canonical encoding

The query string encoder can be limited to only the passed fields.

func WithStatusCodeTruncator 

func WithStatusCodeTruncator(codes ...int) Option

WithStatusCodeTrunactor is a disk cache option to add a body transformer that truncates responses when the status code is not in the provided list.

func WithTTL 

func WithTTL(ttl time.Duration) Option

WithTTL is a disk cache option to set the cache policy TTL.

func WithTransport 

func WithTransport(transport http.RoundTripper) Option

WithTransport is a disk cache option to set the underlying HTTP transport.

func WithTruncator 

func WithTruncator(priority TransformPriority, match func(string, int, string) bool) Option

WithTruncator is a disk cache option to add a body transformer that truncates responses based on match criteria.

func WithZlibCompression 

func WithZlibCompression() Option

WithZlibCompression is a disk cache option to set a zlib marshaler/unmarshaler.

type Policy 

type Policy struct {
	// TTL is the time-to-live.
	TTL time.Duration

	// HeaderTransformers are the set of header transformers.
	HeaderTransformers []HeaderTransformer

	// BodyTransformers are the set of body tranformers.
	BodyTransformers []BodyTransformer

	// MarshalUnmarshaler is the marshal/unmarshaler responsible for storage on
	// disk.
	MarshalUnmarshaler MarshalUnmarshaler
}

Policy is a disk cache policy.

type PrefixStripper 

type PrefixStripper struct {
	Priority     TransformPriority
	ContentTypes []string
	Prefix       []byte
}

PrefixStripper is a body transformer that strips a prefix.

Useful for munging content that may have had a preventative XSS prefix attached to it, such as certan JavaScript or JSON content.

func (PrefixStripper) BodyTransform 

func (t PrefixStripper) BodyTransform(w io.Writer, r io.Reader, urlstr string, code int, contentType string) (bool, error)

BodyTransform satisfies the BodyTransformer interface.

func (PrefixStripper) TransformPriority 

func (t PrefixStripper) TransformPriority() TransformPriority

TransformPriority satisfies the BodyTransformer interface.

type RegexpHeaderTransformer 

type RegexpHeaderTransformer struct {
	Regexps []*regexp.Regexp
	Repls   [][]byte
}

RegexpHeaderTransformer mangles headers matching regexps and replacements.

func NewHeaderTransformer 

func NewHeaderTransformer(pairs ...string) (*RegexpHeaderTransformer, error)

NewHeaderTransformer creates a new header transformer from the passed matching regexp and replacement pairs.

func (*RegexpHeaderTransformer) HeaderTransform 

func (t *RegexpHeaderTransformer) HeaderTransform(buf []byte) []byte

HeaderTransform satisfies the HeaderTransformer interface.

type SimpleMatcher 

type SimpleMatcher struct {
	// contains filtered or unexported fields
}

SimpleMatcher handles matching caching policies to requests.

func Match 

func Match(method, host, path, key string, opts ...Option) *SimpleMatcher

Match creates a new simple match for the provided method, host, path, and substitution key string.

func (*SimpleMatcher) Match 

func (m *SimpleMatcher) Match(req *http.Request) (string, Policy, error)

Match satisifies the Matcher interface.

type TransformPriority 

type TransformPriority int

TransformPriority is the body transform priority. 

const (
	TransformPriorityFirst  TransformPriority = 10
	TransformPriorityDecode TransformPriority = 50
	TransformPriorityModify TransformPriority = 60
	TransformPriorityMinify TransformPriority = 80
	TransformPriorityLast   TransformPriority = 90
)

Body mangle priorities.

type Truncator 

type Truncator struct {
	Priority TransformPriority
	Match    func(string, int, string) bool
}

Truncator is a body transformer that truncates responses based on match criteria.

func (Truncator) BodyTransform 

func (t Truncator) BodyTransform(w io.Writer, r io.Reader, urlstr string, code int, contentType string) (bool, error)

BodyTransform satisfies the BodyTransformer interface.

func (Truncator) TransformPriority 

func (t Truncator) TransformPriority() TransformPriority

TransformPriority satisfies the BodyTransformer interface.

type ZlibMarshalUnmarshaler 

type ZlibMarshalUnmarshaler struct {
	// Level is the compression level.
	Level int

	// Dict is the compression dictionary.
	Dict []byte
}

ZlibMarshalUnmarshaler is a zlib mashaler/unmarshaler.

func (ZlibMarshalUnmarshaler) Marshal 

func (z ZlibMarshalUnmarshaler) Marshal(w io.Writer, r io.Reader) error

Marshal satisfies the MarshalUnmarshaler interface.

func (ZlibMarshalUnmarshaler) Unmarshal 

func (z ZlibMarshalUnmarshaler) Unmarshal(w io.Writer, r io.Reader) error

Unmarshal satisfies the MarshalUnmarshaler interface.

Source Files

View all Source files

Directories

Path Synopsis
_example/example.go
 _example/example.go

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.