Documentation
¶
Overview ¶
Package casenc encodes tool bodies for the content-addressed store. The CAS key is the sha256 of the STORED bytes, not the raw body: a body large enough to be worth it is stored zstd-compressed (content type application/zstd) and keyed by the hash of that compressed form; a smaller body is stored verbatim (application/octet-stream) and keyed by the hash of its raw bytes.
Compression lives entirely on the client. The server stores and serves these bytes opaquely, verifying only that they hash to the declared key; it never compresses or decompresses, so it stays off the CPU-bound path and links no compression code. The browser decompresses transparently via Content-Encoding.
The client deliberately spends redundant CPU to keep the server that simple. Because the CAS key is the hash of the stored (compressed) bytes, the only way to learn a body's key is to encode the body: so a body that turns out to be absent is encoded a second time for the upload itself (HashStream then StreamAs), and cold-cache prefix verification re-encodes bodies to recompute their keys. Each redundant pass is a fixed constant factor over one body, never superlinear, and it buys a server that does no compression and a protocol with no server round-trip to hand the client a key. We accept wasting some client CPU here precisely so the server never pays it.
Determinism is the contract that makes content addressing and resync dedup work: the same raw body must always produce the same stored bytes and the same key, whether it is encoded from memory (EncodeBody) or streamed from disk (HashStream / StreamAs). Two things guarantee that: a single fixed zstd configuration, and single-threaded encoding (klauspost's parallel encoder splits blocks nondeterministically, which would change the bytes and thus the key). Both are pinned in newZstd. The compression threshold is checked against the raw body length identically in both paths, so a body never gets two different keys just because its transcript line happened to be buffered in one session and streamed in another.
Index ¶
Constants ¶
This section is empty.
Variables ¶
var Level = zstd.SpeedDefault
Level is the zstd compression level. SpeedDefault (level 3) matches the courier CAS this is modeled on and balances ratio against the client CPU we deliberately spend here. It is a var so a test or operator can retune it; changing it changes the stored bytes (and so the key) of future uploads, which at worst re-stores some bodies under new keys, never corrupts anything.
var Threshold = 1024
Threshold is the raw-body size at or above which a body is zstd-compressed. Below it the zstd frame overhead is not worth paying (and can even expand a tiny body), so the body is stored verbatim. It must stay far below the client's big-line threshold (1 MiB) so a body's compression decision never depends on whether its transcript line was buffered or streamed: any body large enough to be streamed is far above this and is compressed either way. It is a var so tests can shrink it to exercise the compressed path on small inputs.
Functions ¶
This section is empty.
Types ¶
type Encoder ¶
type Encoder struct {
// contains filtered or unexported fields
}
Encoder encodes bodies with one fixed configuration. It is safe for concurrent use; a Client holds a single shared instance, so many files lifting bodies at once share one Encoder.
compSem, when set, bounds how many zstd compression passes run at once across every goroutine sharing the Encoder, so building CAS keys for many bodies in parallel (a fleet of files syncing, or batched uploads) cannot oversubscribe the CPUs. It guards only the compression branches that produce a key (EncodeBody and HashStream); the cheap raw-hash path needs no bound. The upload re-compression in StreamAs is left to the caller's upload-concurrency limiter instead, so a slow network upload never holds a CPU permit while it waits on the wire. A nil compSem means unbounded, the default for direct use and tests.
func NewLimited ¶
NewLimited builds an Encoder that lets at most maxConcurrency zstd compression passes run at once across all goroutines that share it. A non-positive bound is treated as unbounded, matching New.
func (Encoder) EncodeBody ¶
EncodeBody encodes a body whose raw bytes are already in hand (the buffered small-line path). It returns the CAS key (sha256 of the stored bytes), the stored bytes, and their storage content type. It satisfies parser.BodyEncoder.
A body shorter than Threshold is stored verbatim and keyed by its raw hash. The returned Stored slice for that case aliases raw (no copy): RewriteLine has already copied the line out of the scan buffer, so the bytes are stable for the upload.
func (Encoder) HashStream ¶
func (e Encoder) HashStream(ctx context.Context, r io.Reader) (sha, contentType string, rawLen int, err error)
HashStream encodes a body streamed from r (the big-line path and the cold-cache digest path) without buffering it, returning the CAS key, the storage content type, and the RAW (uncompressed) body length the sentinel records. It peeks the first Threshold bytes to make the very same size decision EncodeBody makes, so a streamed body and an in-hand body of identical content produce identical keys.
It hashes the STORED bytes: for a small body that is the raw bytes; for a large one it is the zstd output, hashed as it is produced so the compressed form is never resident. The caller then calls StreamAs to (re)produce the same stored bytes for the actual upload.
func (Encoder) StreamAs ¶
StreamAs returns a reader of the stored bytes for a body streamed from r, given the content type HashStream already decided. For a raw body it is r unchanged; for a zstd body it streams the compression through a pipe so the upload moves in O(window) memory and the compressed form is never fully resident. The bytes it yields are byte-identical to what HashStream hashed, so the server's verification against the key holds.
Source Files