Documentation
¶
Overview ¶
Package beta contains a prototype HTTP client middleware implementation that handles compresshttp compression and decompression for gzip, deflate (flate), and zstd.
Overview and usage ¶
- Use ClientAcceptGZIPMiddleware, ClientAcceptFlateMiddleware, or ClientAcceptZstdMiddleware to accept compressed http responses.
- Use ServerAcceptGZIPMiddleware, ServerAcceptFlateMiddleware, or ServerAcceptZstdMiddleware to accept compressed http requests.
- Use up to one of ClientWriteGZIPMiddleware, ClientWriteFlateMiddleware, or ClientWriteZstdMiddleware to automatically compress request bodies with the specified method.
- Use one or more of ServerWriteGZIPMiddleware, ServerWriteFlateMiddleware, or ServerWriteZstdMiddleware to automatically compress responses using the specified method if the client indicates support via the 'Accept-Encoding' header.
Commentary ¶
This is a more complete implementation than the alpha package, covering multiple compression methods and with more robust error-checking and options. This is generally suited for use in real applications, although it still has some performance and API limitations compared to the final compresshttp package.
Improvements over alpha package ¶
- supports multiple compression methods: gzip, deflate, and zstd
- more robust error-checking
- middlewares are 'composable' and properly forward io.Closers through the chain and handle http headers (i.e, multiple layers of content-encoding)//
- shared implementation functions reduce duplication and make a smaller surface for bugs
Implementation flaws: ¶
- synchronous compression will break for large or indefinite-length bodies (i.e, streaming log files); final design uses streaming compression/decompression
Performance limitations: ¶
- (minor): chaining handlers has some overhead compared to a single monolithic handler: fixed in final design via a single ClientAcceptMiddleware and ServerWriteMiddleware (this is more of a matter of taste, but I really care about performance).
- (major): wasted memory while compressing requests and responses: fixed in final design via pooling and streaming
- (major): synchronous compression/decompression: for large bodies, this wastes memory and CPU time; fixed in final design via streaming compression/decompression
- (major): gzip and flate compression/decompression is single-threaded and slow for large bodies; should use pgzip for large bodies: fixed in final design via pgzip support
- (medium): ALWAYS compresses request bodies, even ones that are empty or very small: fixed in final design via configurable size thresholds
API flaws: ¶
- ease of use: middleware must be manually selected for each compression method; no single 'ClientAcceptMiddleware' or 'ServerWriteMiddleware' that handles all supported methods//
- order-sensitive: allowing multiple compression methods means that the order of middleware matters: prioritizing zstd over flate over gzip requires careful ordering of middleware that is best left to the library to handle
Index ¶
- func ServerAccept(encoding string, makeReader func(io.Reader) io.Reader, h http.Handler) http.HandlerFunc
- func ServerAcceptFlateMiddleware(h http.Handler) http.HandlerFunc
- func ServerAcceptGZIPMiddleware(h http.Handler) http.HandlerFunc
- func ServerAcceptZstdMiddleware(h http.Handler) http.HandlerFunc
- func ServerCompressIfAccepts(encoding string, h http.Handler, makeWriter func(w io.Writer) io.WriteCloser) http.HandlerFunc
- func ServerWriteFlateMiddleware(h http.Handler) http.HandlerFunc
- func ServerWriteGZIPMiddleware(h http.Handler, level int) http.HandlerFunc
- func ServerWriteZstdMiddleware(h http.Handler, enc ...zstd.EOption) http.HandlerFunc
- type RoundTripFunc
- func ClientAccept(encoding string, rt http.RoundTripper, makeReader func(io.Reader) io.Reader) RoundTripFunc
- func ClientAcceptFlateMiddleware(rt http.RoundTripper) RoundTripFunc
- func ClientAcceptGZIPMiddleware(rt http.RoundTripper) RoundTripFunc
- func ClientAcceptZstdMiddleware(rt http.RoundTripper) RoundTripFunc
- func ClientWriteCompression(encoding string, rt http.RoundTripper, ...) RoundTripFunc
- func ClientWriteFlateMiddleware(rt http.RoundTripper) RoundTripFunc
- func ClientWriteGZIPMiddleware(rt http.RoundTripper, level int) RoundTripFunc
- func ClientWriteZstdMiddleware(rt http.RoundTripper) RoundTripFunc
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func ServerAccept ¶
func ServerAccept(encoding string, makeReader func(io.Reader) io.Reader, h http.Handler) http.HandlerFunc
ServerAccept accepts requests with the specified encoding by decompressing them using makeReader before passing them to the inner handler.
This is a low-level building block that serves as the implementation behind ServerAcceptGZIPMiddleware, ServerAcceptFlateMiddleware, and ServerAcceptZstdMiddleware.
API limitations ¶
- only one level of content-encoding supported: overwrites existing Content-Encoding header
Possible footguns ¶
- no bounds checks for makeReader prior to returning the handler - this will explode while handling requests: we'd rather catch these at setup time
func ServerAcceptFlateMiddleware ¶
func ServerAcceptFlateMiddleware(h http.Handler) http.HandlerFunc
accept deflate compressed requests
API limitations: ¶
- only understands "deflate" content-encoding: doesn't understand "flate", "x-flate", "x-deflate"
Performance limitations: ¶
- we create new flate.Readers for every request, which leads to memory presure via garbage collection. we can reuse these via pooling.
func ServerAcceptGZIPMiddleware ¶ added in v0.4.0
func ServerAcceptGZIPMiddleware(h http.Handler) http.HandlerFunc
accept gzip compressed requests
API limitations: ¶
- only one level of content-encoding supported: overwrites existing Content-Encoding header
- forces gzip default compression level: no way to tune for performance vs compression ratio
- only understands "gzip" content-encoding: doesn't understand "x-gzip", etc.
Performance limitations ¶
- for large bodies, regular gzip is slow: should use pgzip
- performance: we create new gzip.Writers for every request, which leads to memory presure via garbage collection. we can reuse these via pooling.
func ServerAcceptZstdMiddleware ¶
func ServerAcceptZstdMiddleware(h http.Handler) http.HandlerFunc
See ServerAcceptGZIPMiddleware for commentary
func ServerCompressIfAccepts ¶
func ServerCompressIfAccepts(encoding string, h http.Handler, makeWriter func(w io.Writer) io.WriteCloser) http.HandlerFunc
ServerCompressIfAccepts checks for the specified encoding in the request's Accept-Encoding header and, if present, compresses the response using the specified encoding via the provided writer. if no matching encoding is found, the inner handler is called directly.
This is a low-level building block that serves as the implementation of ServerWriteGZIPMiddleware, ServerWriteFlateMiddleware, and ServerWriteZstdMiddleware.
func ServerWriteFlateMiddleware ¶
func ServerWriteFlateMiddleware(h http.Handler) http.HandlerFunc
write deflate compressed request bodies
API limitations: ¶
- only one level of content-encoding supported: overwrites existing Content-Encoding header
- forces flate default compression level: no way to tune for performance vs compression ratio
- only understands "deflate" content-encoding: doesn't understand "flate", "x-flate", "x-deflate"
Performance limitations ¶
- we create new flate.Writers for every request, which leads to memory presure via garbage collection. we can reuse these via pooling.
func ServerWriteGZIPMiddleware ¶ added in v0.4.0
func ServerWriteGZIPMiddleware(h http.Handler, level int) http.HandlerFunc
respond to 'Accept-Encoding: gzip' by writing gzip-compressed HTTP response bodies.
Why not this design? ¶
- for large bodies, regular gzip is slow: should use pgzip
- performance: we create new gzip.Writers for every request, which leads to memory presure via garbage collection. we can reuse these via pooling.
func ServerWriteZstdMiddleware ¶
respond to 'Accept-Encoding: zstd' by writing zstd-compressed HTTP response bodies. See ServerCompressIfAccepts for implementation details.
Performance limitations: ¶
- we create new zstd.Writers for every request, which leads to memory presure via garbage collection. we can reuse these via pooling.
Types ¶
type RoundTripFunc ¶
RoundTripFunc is to http.RoundTripper what http.HandlerFunc is to http.Handler. That is, it's a helper to let you use ordinary functions as HTTP round trippers.
func ClientAccept ¶
func ClientAccept(encoding string, rt http.RoundTripper, makeReader func(io.Reader) io.Reader) RoundTripFunc
ClientAccept adds an Accept-Encoding header and decompresses responses using the specified encoding if and only if the response has a matching Content-Encoding header.
This is a low-level building block that serves as the implementation of ClientAcceptGZIPMiddleware, ClientAcceptFlateMiddleware, and ClientAcceptZstdMiddleware.
func ClientAcceptFlateMiddleware ¶
func ClientAcceptFlateMiddleware(rt http.RoundTripper) RoundTripFunc
ClientAcceptFlateMiddleware accepts deflate-compressed responses
API limitations: ¶
- only understands "deflate" content-encoding: doesn't understand "flate", "x-flate", "x-deflate"
func ClientAcceptGZIPMiddleware ¶ added in v0.4.0
func ClientAcceptGZIPMiddleware(rt http.RoundTripper) RoundTripFunc
accept gzip compressed responses
non-shared API limitations: ¶
- only understands "gzip" content-encoding: doesn't understand "x-gzip"
- for large bodies, regular gzip is slow: should use pgzip
func ClientAcceptZstdMiddleware ¶
func ClientAcceptZstdMiddleware(rt http.RoundTripper) RoundTripFunc
ClientAcceptZstdMiddleware accepts zstd-compressed responses, decompressing them transparently.
See ClientAccept for shared implementation details and commentary.
func ClientWriteCompression ¶
func ClientWriteCompression(encoding string, rt http.RoundTripper, makeWriter func(io.Writer) (io.WriteCloser, error)) RoundTripFunc
ClientWriteCompression ALWAYS compresses the request body using the specified encoding. This is a low-level building block that serves as the implementation of ClientWriteGZIPMiddleware, ClientWriteFlateMiddleware, and ClientWriteZstdMiddleware.
API limitations ¶
- only one level of content-encoding supported: overwrites existing Content-Encoding header
- synchronous: compresses entire body into memory before sending request: wasteful for large bodies and breaks entirely for indefinite or slow streams (e.g, sending log files as they are written)
Possible footguns ¶
- no bounds checks for MakeWriter prior to returning the handler - this will explode while handling requests: we'd rather catch these at setup time
Performance limitations ¶
- we create new compressors for every request, which leads to memory presure via garbage collection. we can reuse these via pooling.
- we allocate a buffer to hold the entire compressed body in memory before sending the request; for large bodies, this is wasteful.
it is the implementation behind ClientWriteGZIPMiddleware, ClientWriteFlateMiddleware, and ClientWriteZstdMiddleware.
func ClientWriteFlateMiddleware ¶
func ClientWriteFlateMiddleware(rt http.RoundTripper) RoundTripFunc
ClientWriteFlateMiddleware compresses outgoing client requests using deflate compression.
See ClientWriteCompression for shared implementation details and commentary.
func ClientWriteGZIPMiddleware ¶ added in v0.4.0
func ClientWriteGZIPMiddleware(rt http.RoundTripper, level int) RoundTripFunc
gzip all outgoing client requests using stdlib gzip. see ClientWriteCompression for shared implementation details and commentary.
weaknesses compared to final design: ¶
- always uses stdlib gzip: for large bodies: pgzip may be a better choice for large bodies (i.e, files, log streams, etc)
func ClientWriteZstdMiddleware ¶
func ClientWriteZstdMiddleware(rt http.RoundTripper) RoundTripFunc
ClientWriteZstdMiddleware compresses outgoing client requests using zstd compression.
See ClientWriteCompression for shared implementation details and commentary.
Source Files