compresshttp

func DefaultBrotliWriterOpts ¶ added in v0.4.0

func DefaultBrotliWriterOpts() *BrotliWriterOpts

DefaultBrotliWriterPool is the default pool for brotli writers used by ClientEncodeBrotliMiddleware and ServerMiddleware.

• Quality: brotli.DefaultCompression (6)
• CompressThreshold: 512 bytes
• Pool: DefaultBrotliWriterPool

package main

import (
	"fmt"

	_ "embed"
	"gitlab.com/efronlicht/compresshttp"
)

func main() {
	opts := compresshttp.DefaultBrotliWriterOpts()
	fmt.Printf(`quality=%d lgwin=%d compressthreshold=%d`, opts.Quality, opts.LGWin, opts.CompressThreshold)
}

Output:
quality=6 lgwin=0 compressthreshold=512

func DefaultServerBrotliOpts ¶ added in v0.4.0

func DefaultServerBrotliOpts() *BrotliWriterOpts

DefaultServerBrotliOpts returns a ServerBrotliCfg with default settings.

• Level: brotli.DefaultCompression (6)
• Pool: DefaultBrotliWriterPool. If you change Level, either set Pool to nil or create a new WriterPool with the same level.

func DefaultClientAcceptConfig ¶ added in v0.3.0

func DefaultClientAcceptConfig() *ClientAcceptOpts

DefaultClientAcceptConfig returns a ClientAcceptConfig with default settings:

• Quality: DefaultAcceptQuality() (all ones)

func DefaultFlateWriterOpts ¶ added in v0.4.0

func DefaultFlateWriterOpts() *FlateWriterOpts

FlateWriterOption with default settings.

• Level: flate.DefaultCompression (-1)
• Pool: DefaultFlatePool. If you change the level, either set Pool to nil or create a new WriterPool with the same level.
• CompressThreshold: 512 bytes

package main

import (
	"fmt"

	_ "embed"
	"gitlab.com/efronlicht/compresshttp"
)

func main() {
	opts := compresshttp.DefaultFlateWriterOpts()
	fmt.Printf(`level=%d compressthreshold=%d`, opts.Level, opts.CompressThreshold)
}

Output:
level=-1 compressthreshold=512

func DefaultGZIPWriterOpts ¶ added in v0.4.0

func DefaultGZIPWriterOpts() *GZIPWriterOpts

DefaultGZIPWriterOpts returns a GZIPWriterOpts with default settings.

• Lvl: gzip.DefaultCompression (-1)
• Pool: DefaultGZIPWriterPool
• CompressThreshold: 512 bytes
• Skip: NeverSkip

package main

import (
	"fmt"

	_ "embed"
	"gitlab.com/efronlicht/compresshttp"
)

func main() {
	opts := compresshttp.DefaultGZIPWriterOpts()
	fmt.Printf(`level=%d compressthreshold=%d`, opts.Lvl, opts.CompressThreshold)
}

Output:
level=-1 compressthreshold=512

func DefaultAcceptQuality ¶ added in v0.3.0

func DefaultAcceptQuality() *Quality

DefaultAcceptQuality returns an AcceptQuality with all quality values set to 1.

package main

import (
	"fmt"

	_ "embed"
	"gitlab.com/efronlicht/compresshttp"
)

func main() {
	fmt.Printf("%+v\n", compresshttp.DefaultAcceptQuality())
}

Output:
&{GZIP:1 Flate:1 ZSTD:1 Brotli:1 Snappy:1}

func DefaultSnappyWriterOpts ¶ added in v0.4.0

func DefaultSnappyWriterOpts() *SnappyWriterOpts

DefaultSnappyWriterOpts returns a SnappyWriterConfig with default settings.

• CompressThreshold: 512 bytes

package main

import (
	"fmt"

	_ "embed"
	"gitlab.com/efronlicht/compresshttp"
)

func main() {
	fmt.Printf("%+v\n", compresshttp.DefaultSnappyWriterOpts())
}

Output:
&{CompressThreshold:512 Skip:<nil>}

func ServerMiddleware ¶

func ServerMiddleware(h http.Handler, cfg *ServerConfig) UnwrappingHandler

ServerMiddleware transparently compresses and decompresses HTTP request and response bodies encoded with GZIP, ZSTD, or Deflate. All configuration is optional, both overall and per subconfig; if nil, defaults are used.

Basic Usage ¶

	var echo http.Handler = http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
	    io.Copy(w, r.Body)
	})
	h = compresshttp.ServerMiddleware(h, nil) // use default config
 // h will now automatically decompress request bodies and compress response bodies as needed.

Request Decompression ¶

ServerMiddleware looks at the 'Content-Encoding' header of incoming requests and repeatedly decodes the body according to the encodings specified there. That is, it will remove any number of 'flate', 'gzip', or 'zstd' layers from the request body, stopping as soon as it finds an unrecognized encoding. It supports the following encodings:

• 'zstd', 'x-zstd', or 'zstandard': decompresses with zstd (taken from the stdlib/internal/zstd, vendored here)
• 'gzip' or 'x-gzip': decompresses with stdlib gzip or klauspost/pgzip, depending on the PGZipDecompress and PGDecompressThreshold settings.
• 'deflate', 'flat': decompresses with stdlib flate
• 'br' or 'brotli': decompresses with andybalholm/brotli
• 'snappy': decompresses with golang/snappy

any other 'Content-Encoding' value is ignored and the request body is passed through unmodified.

Response Compression ¶

ServerMiddleware looks at the 'Accept-Encoding' header of incoming requests to determine how to encode responses. It respects the quality values specified by the client, and picks the encoding with the highest quality value that it supports. In the case of unspecified or tied quality values, it prefers encodings in this order (highest first):

• zstd > brotli > deflate > gzip > snappy > none

See https://developer.mozilla.org/en-US/docs/Glossary/Quality_values for details on quality values.

If none of these encodings are accepted by the client, this passes the request to the underlying handler without modification.

func ClientAcceptAllMiddleware ¶ added in v0.3.0

func ClientAcceptAllMiddleware(rt http.RoundTripper, cfg *ClientAcceptOpts) UnwrappingRoundTripper

ClientAcceptAllMiddleware is an HTTP RoundTripper that adds Accept-Encoding headers. It transparently decompresses response bodies encoded with GZIP, ZSTD, or Deflate. Wrap your existing RoundTripper with this to get automatic decompression support. If rt is nil, http.DefaultTransport is used.

Performance: ¶

All decoders use pooled readers for efficiency: this assumes that if you get one request with a given encoding, you'll likely get more soon after. This uses significantly less memory under steady-state load but can be slower or inefficient for one-offs or bursty loads.

func ClientEncodeBrotliMiddleware ¶ added in v0.3.0

func ClientEncodeBrotliMiddleware(transport http.RoundTripper, opts *BrotliWriterOpts) UnwrappingRoundTripper

ClientEncodeBrotliMiddleware is an HTTP RoundTripper that compresses request bodies with Brotli. You can pass nil for any argument to use default settings (transport: http.DefaultTransport, opts: DefaultBrotliWriterOptions).

Performance ¶

This spawns a new goroutine per request that does compression as the request body is read. This is necessary to avoid buffering the entire compressed body in memory before sending it to the server. For small request bodies, this may be less efficient than buffering the entire compressed body in memory first, but avoids indefinitely blocking on requests that lie about their Content-Length. This uses a pool of brotli.Writers for efficiency.

Pools must match the BrotliWriterOptions used to create them; using mismatched options will panic before returning the RoundTripper.

package main

import (
	"fmt"
	"io"
	"net/http"
	"net/http/httptest"
	"strings"

	_ "embed"
	"github.com/andybalholm/brotli"
	"gitlab.com/efronlicht/compresshttp"
)

//go:embed warandpeace.txt
var warandpeace string

const gettysburgSentence = "Four score and seven years ago our fathers brought forth on this continent, a new nation, conceived in Liberty, and dedicated to the proposition that all men are created equal."

func main() {
	const format = "%s: encoding: %q, content-length: %d\n"
	var h http.Handler = http.HandlerFunc(func(w http.ResponseWriter, serverReq *http.Request) {
		encoded, _ := io.ReadAll(serverReq.Body)
		if serverReq.Header.Get("Content-Encoding") == "br" {
			decoded := brotli.NewReader(strings.NewReader(string(encoded)))
			got, _ := io.ReadAll(decoded)
			if string(got) != gettysburgSentence && string(got) != warandpeace {
				panic("decoded body mismatch")
			}

		}

		fmt.Printf(format, "server", serverReq.Header.Get("Content-Encoding"), len(encoded))
		w.WriteHeader(200)
	})

	srv := httptest.NewServer(h)
	c := srv.Client()
	c.Transport = compresshttp.ClientEncodeBrotliMiddleware(c.Transport, nil)

	sendReq := func(s string) {
		clientReq, _ := http.NewRequest("POST", srv.URL, strings.NewReader(s))
		fmt.Printf(format, "client", clientReq.Header.Get("Content-Encoding"), clientReq.ContentLength)
		_, _ = c.Do(clientReq)
	}

	sendReq(gettysburgSentence)
	sendReq(warandpeace)

}

Output:
client: encoding: "", content-length: 176
server: encoding: "", content-length: 176
client: encoding: "", content-length: 3288446
server: encoding: "br", content-length: 1009174

func ClientEncodeFlateMiddleware ¶ added in v0.3.0

func ClientEncodeFlateMiddleware(transport http.RoundTripper, cfg *FlateWriterOpts) UnwrappingRoundTripper

package main

import (
	"fmt"
	"io"
	"net/http"
	"net/http/httptest"
	"strings"

	_ "embed"
	"gitlab.com/efronlicht/compresshttp"
)

//go:embed warandpeace.txt
var warandpeace string

const gettysburgSentence = "Four score and seven years ago our fathers brought forth on this continent, a new nation, conceived in Liberty, and dedicated to the proposition that all men are created equal."

func main() {
	const format = "%s: encoding: %q, content-length: %d\n"
	var h http.Handler = http.HandlerFunc(func(w http.ResponseWriter, serverReq *http.Request) {
		body := io.Reader(serverReq.Body)
		got, _ := io.ReadAll(body)
		fmt.Printf(format, "server", serverReq.Header.Get("Content-Encoding"), len(got))
		w.WriteHeader(200)
	})

	srv := httptest.NewServer(h)
	c := srv.Client()
	c.Transport = compresshttp.ClientEncodeFlateMiddleware(c.Transport, nil)

	sendReq := func(s string) {
		clientReq, _ := http.NewRequest("POST", srv.URL, strings.NewReader(s))
		fmt.Printf(format, "client", clientReq.Header.Get("Content-Encoding"), clientReq.ContentLength)
		_, _ = c.Do(clientReq)
	}

	sendReq(gettysburgSentence) // small bodies are not compressed by default to save on overhead

	sendReq(warandpeace) // but large bodies are.

}

Output:
client: encoding: "", content-length: 176
server: encoding: "", content-length: 176
client: encoding: "", content-length: 3288446
server: encoding: "deflate", content-length: 1169922

func ClientEncodeGZIPMiddleware ¶

func ClientEncodeGZIPMiddleware(rt http.RoundTripper, cfg *GZIPWriterOpts) UnwrappingRoundTripper

ClientEncodeGZIPMiddleware is an HTTP RoundTripper that compresses request bodies or GZIP or PGZIP if they're large enough.

GZIP is widely-supported but somewhat slow and inefficient for large bodies.

Performance ¶

This spawns a new goroutine per request that does compression as the request body is read. This is necessary to avoid buffering the entire compressed body in memory before sending it to the server. For small request bodies, this may be less efficient than buffering the entire compressed body in memory first, but avoids indefinitely blocking on requests that lie about their Content-Length. You can adjust the CompressThreshold field in ClientGZipConfig to control when compression is applied based on the request's Content-Length. In the default (auto), mode, pgzip is used for large bodies (>=2MiB), the stdlib for medium bodies (>=2KiB and <2MiB), and no compression for small bodies (<2KiB). See ClientGZipConfig for details.

Warnings ¶

• not all servers support compressed request bodies: make sure the server you're talking to can handle this.
• level 0 (gzip.NoCompression) has almost no use case: it adds gzip headers but does no compression: this function will warn you about this unless you set the COMPRESSHTTP_WARN_GZIP_NO_COMPRESSION=0 environment variable.

This function panics if the provided configuration is invalid. Call cfg.Validate() to check for errors ahead of time.v

package main

import (
	"fmt"
	"io"
	"net/http"
	"net/http/httptest"
	"os"
	"strings"

	_ "embed"
	"gitlab.com/efronlicht/compresshttp"
)

//go:embed warandpeace.txt
var warandpeace string

const gettysburgSentence = "Four score and seven years ago our fathers brought forth on this continent, a new nation, conceived in Liberty, and dedicated to the proposition that all men are created equal."

func main() {
	const format = "%s: encoding: %q, content-length: %v\n"
	var h http.Handler = http.HandlerFunc(func(w http.ResponseWriter, serverReq *http.Request) {
		got, _ := io.ReadAll(serverReq.Body)

		fmt.Fprintf(os.Stdout, format, "server", serverReq.Header.Get("Content-Encoding"), len(got))

		w.WriteHeader(200)
	})

	srv := httptest.NewServer(h)
	c := srv.Client()
	c.Transport = compresshttp.ClientEncodeGZIPMiddleware(c.Transport, nil)

	sendReq := func(s string) {
		clientReq, _ := http.NewRequest("POST", srv.URL, strings.NewReader(s)) // create request
		fmt.Printf(format, "client", clientReq.Header.Get("Content-Encoding"), clientReq.ContentLength)
		_, _ = c.Do(clientReq) // send request
	}

	// small bodies are not compressed by default to save on overhead:
	sendReq(gettysburgSentence)

	// but larger bodies are...
	sendReq(warandpeace)

}

Output:
client: encoding: "", content-length: 176
server: encoding: "", content-length: 176
client: encoding: "", content-length: 3288446
server: encoding: "gzip", content-length: 1169940

func ClientEncodeSnappyMiddleware ¶ added in v0.3.0

func ClientEncodeSnappyMiddleware(transport http.RoundTripper, cfg *SnappyWriterOpts) UnwrappingRoundTripper

ClientEncodeSnappyMiddleware is an HTTP RoundTripper that compresses request bodies with Snappy.

Snappy is a fast but not very efficient compression algorithm; it's largely fallen out of favor as ZSTD and Brotli have become more widespread and CPUs have become faster.

Performance ¶

This uses a buffered writer and spawns a goroutine to do compression as the request body is read: for small bodies, it may be more efficient to do in-memory compression first or skip compression entirely. Use the CompressThreshold field in SnappyWriterConfig to control this behavior.

Writers are pooled for efficiency.

package main

import (
	"fmt"
	"io"
	"net/http"
	"net/http/httptest"
	"strings"

	_ "embed"
	"gitlab.com/efronlicht/compresshttp"
)

//go:embed warandpeace.txt
var warandpeace string

const gettysburgSentence = "Four score and seven years ago our fathers brought forth on this continent, a new nation, conceived in Liberty, and dedicated to the proposition that all men are created equal."

func main() {
	const format = "%s: encoding: %q, content-length: %d\n"
	var h http.Handler = http.HandlerFunc(func(w http.ResponseWriter, serverReq *http.Request) {
		got, _ := io.ReadAll(serverReq.Body)

		fmt.Printf(format, "server", serverReq.Header.Get("Content-Encoding"), len(got))
	})

	srv := httptest.NewServer(h)
	c := srv.Client()
	c.Transport = compresshttp.ClientEncodeSnappyMiddleware(c.Transport, nil)

	sendReq := func(s string) {
		clientReq, _ := http.NewRequest("POST", srv.URL, strings.NewReader(s))
		fmt.Printf(format, "client", clientReq.Header.Get("Content-Encoding"), clientReq.ContentLength)
		_, _ = c.Do(clientReq)
	}

	sendReq(gettysburgSentence) // small bodies are not compressed by default
	sendReq(warandpeace)        // but large ones are.

}

Output:
client: encoding: "", content-length: 176
server: encoding: "", content-length: 176
client: encoding: "", content-length: 3288446
server: encoding: "snappy", content-length: 1913059

func ClientEncodeZSTDMiddleware ¶

func ClientEncodeZSTDMiddleware(transport http.RoundTripper, cfg *ZstdWriteOpts) UnwrappingRoundTripper

ClientEncodeZSTDMiddleware is an HTTP RoundTripper that compresses request bodies with ZSTD if they're large enough. You can pass nil for any argument to use default settings (transport: http.DefaultTransport, cfg: DefaultClientZSTDEncodeConfig()).

Performance ¶

• This spawns a new goroutine per request that does compression as the request body is read. This is necessary to avoid buffering the entire compressed body in memory before sending it to the server.
• For small request bodies, this may be less efficient than buffering the entire compressed body in memory first, but avoids indefinitely blocking on requests that lie about their Content-Length.
• This uses a new zstd.Encoder per request: zstd.Encoder.Reset() has some issues that make pooling difficult. TODO: revisit this in the future.

Warning ¶

ZSTD compression is not transparently supported by most servers: don't expect this to work unless someone has a mirroring middleware on the server side.

Bounds: ¶

This function panics if the provided configuration is invalid. Invalid configuration includes:

• negative CompressThreshold
• invalid EncoderOptions

Caveats/Improvements: ¶

• we create new zstd.Writers for every request, which leads to memory presure via garbage collection. ideally, we would reuse these via pooling, but klauspost/compress/zstd's Reset() method has some issues that make pooling difficult.
• TODO: when the stdlib supports zstd WRITING natively, switch to that and pool those writers instead.

package main

import (
	"fmt"
	"io"
	"net/http"
	"net/http/httptest"
	"strings"

	_ "embed"
	"gitlab.com/efronlicht/compresshttp"
)

//go:embed warandpeace.txt
var warandpeace string

const gettysburgSentence = "Four score and seven years ago our fathers brought forth on this continent, a new nation, conceived in Liberty, and dedicated to the proposition that all men are created equal."

func main() {
	const format = "%s: encoding: %q, content-length: %d\n"
	var h http.Handler = http.HandlerFunc(func(w http.ResponseWriter, serverReq *http.Request) {
		got, _ := io.ReadAll(serverReq.Body)

		fmt.Printf(format, "server", serverReq.Header.Get("Content-Encoding"), len(got))
		w.WriteHeader(200)
	})

	srv := httptest.NewServer(h)
	c := srv.Client()
	c.Transport = compresshttp.ClientEncodeZSTDMiddleware(c.Transport, nil)

	sendReq := func(s string) {
		clientReq, _ := http.NewRequest("POST", srv.URL, strings.NewReader(s))
		fmt.Printf(format, "client", clientReq.Header.Get("Content-Encoding"), clientReq.ContentLength)
		_, _ = c.Do(clientReq)
	}

	// small bodies are not compressed by default...
	sendReq(gettysburgSentence)
	// ... but large ones are:
	sendReq(warandpeace)

}

Output:
client: encoding: "", content-length: 176
server: encoding: "", content-length: 176
client: encoding: "", content-length: 3288446
server: encoding: "zstd", content-length: 1137550

func DefaultClientZSTDEncodeConfig ¶

func DefaultClientZSTDEncodeConfig() *ZstdWriteOpts

DefaultClientZSTDEncodeConfig returns a ZSTDEncodeConfig with default settings:

• CompressThreshold: 2KiB,
• EncoderOptions: nil