Documentation
¶
Index ¶
- Constants
- func AsyncReadCloser(proxy ProxyCallback) io.ReadCloser
- func BreakInBitmap(start, end int64, partSize int64) bitmap.Bitmap
- func BufBlock(blocks []uint32) (offset, limit int64)
- func ChunkPart(blocks []uint32, partSize uint32) (offset, limit int64)
- func ChunkWriterCloser(file io.ReadWriteCloser, closer func() error) io.WriteCloser
- func CopyWithRateLimit(dst io.Writer, src io.Reader, limiter *rate.Limiter) error
- func FullHit(first, last uint32, fs bitmap.Bitmap) bool
- func LimitReadCloser(readCloser io.ReadCloser, max int64) io.ReadCloser
- func NewRateLimitReader(r io.ReadCloser, Kbps int) io.ReadCloser
- func NopCloser() io.Closer
- func PartHit(first, last uint32, fs bitmap.Bitmap) bool
- func PartsReadCloser(optionalCloser io.Closer, readers ...io.ReadCloser) io.ReadCloser
- func RangeReader(r io.ReadCloser, newStart int, newEnd int, rawStart int, rawEnd int) io.ReadCloser
- func SavepartAsyncReader(r io.ReadCloser, blockSize uint64, startAt uint, flushBuffer EventSuccess, ...) io.ReadCloser
- func SavepartReader(r io.ReadCloser, blockSize uint64, startAt uint, flushBuffer EventSuccess, ...) io.ReadCloser
- func SkipReadCloser(R io.ReadCloser, skip int64) io.ReadCloser
- type AllCloser
- type Block
- type EventClose
- type EventError
- type EventSuccess
- type ProxyCallback
- type RateLimitedWriter
Constants ¶
const BitBlock = 1 << 15
BitBlock is the default block size (32 KB) used by BreakInBitmap and BufBlock when breaking a byte range into logical blocks for bitmap indexing.
Variables ¶
This section is empty.
Functions ¶
func AsyncReadCloser ¶
func AsyncReadCloser(proxy ProxyCallback) io.ReadCloser
AsyncReadCloser returns an io.ReadCloser that invokes proxy in a background goroutine and pipes the response body back to the caller through an io.Pipe.
The proxy callback is expected to fetch an upstream HTTP response. On success the response body is copied to the pipe writer; on failure the pipe is closed with the error so the caller's next Read returns it.
This is used in the caching middleware to issue a sub-request to origin without blocking the caller's goroutine — the HTTP round-trip happens asynchronously while the caller can start reading the body as soon as bytes arrive.
func BreakInBitmap ¶
BreakInBitmap converts a byte range [start, end] into a bitmap where each set bit represents one block-sized chunk that the range spans. partSize is the chunk size (typically 1 MB in tavern's cache storage).
The resulting bitmap is used to record which chunks of a file have been written to disk — set bits mean "this chunk is stored on disk".
func BufBlock ¶
BufBlock converts a run of block indices (as returned by BlockGroup) to a byte offset and total byte length using the default BitBlock size.
func ChunkPart ¶
ChunkPart converts a run of block indices to a byte offset and total byte length using a caller-supplied partSize (typically the cache slice_size).
func ChunkWriterCloser ¶
func ChunkWriterCloser(file io.ReadWriteCloser, closer func() error) io.WriteCloser
ChunkWriterCloser returns an io.WriteCloser that wraps a disk file (typically an opened chunk file). When Close is called the underlying file is closed first, then the closer callback is invoked — this is where the storage layer updates the LSM-tree index to record the new chunk.
func CopyWithRateLimit ¶
CopyWithRateLimit copies from src to dst using a 32 KB buffer, waiting for tokens before each write. Returns nil on io.EOF from src.
Unlike RateLimitedWriter which wraps a single writer, this function can be used with any io.Reader/io.Writer pair and a shared limiter.
func FullHit ¶
FullHit returns true when every block index in [first, last] (inclusive) is present in the bitmap — meaning the entire requested range is cached locally.
Used by the caching middleware to decide whether a Range request can be served entirely from disk without an upstream fetch.
func LimitReadCloser ¶
func LimitReadCloser(readCloser io.ReadCloser, max int64) io.ReadCloser
LimitReadCloser returns an io.ReadCloser that reads at most max bytes from the underlying reader before returning io.EOF. It is used in the caching middleware to cap the response body to the advertised Content-Length (or a range length), preventing over-read.
func NewRateLimitReader ¶
func NewRateLimitReader(r io.ReadCloser, Kbps int) io.ReadCloser
NewRateLimitReader returns an io.ReadCloser that limits the read throughput to Kbps kilobits per second. The burst size equals one second of tokens so the reader can handle short spikes.
Used primarily in tests (mockserver, e2e) to simulate bandwidth-constrained upstream servers.
func PartHit ¶
PartHit returns true when at least one block index in [first, last] is present in the bitmap — meaning a partial (possibly fragmented) cache hit.
Used alongside FullHit to decide whether the request needs a range-fill (some blocks cached, some must be fetched from origin).
func PartsReadCloser ¶
func PartsReadCloser(optionalCloser io.Closer, readers ...io.ReadCloser) io.ReadCloser
PartsReadCloser stitches multiple io.ReadCloser instances into a single io.ReadCloser. Readers are consumed sequentially; each is closed when fully read. An optionalCloser (may be nil) is called once during the final Close.
Returns nil when readers is empty.
This is the central combinator used by the caching middleware to join cached chunk files and a live upstream reader into one continuous body for the HTTP client.
func RangeReader ¶
func RangeReader(r io.ReadCloser, newStart int, newEnd int, rawStart int, rawEnd int) io.ReadCloser
RangeReader returns an io.ReadCloser that extracts the byte range [rawStart, rawEnd] from r. The caller typically sets newStart=0, newEnd=(fileSize-1) and passes the HTTP Range's start/end as rawStart/rawEnd.
func SavepartAsyncReader ¶
func SavepartAsyncReader(r io.ReadCloser, blockSize uint64, startAt uint, flushBuffer EventSuccess, flushFailed EventError, cleanup EventClose, writeQueueSize int) io.ReadCloser
SavepartAsyncReader is the asynchronous variant of SavepartReader. It reads from r in fixed blockSize chunks and sends complete blocks to a background goroutine via a buffered channel. The onSuccess callback runs on the writer goroutine.
startAt is the byte offset to skip (e.g. from a Range request). writeQueueSize controls channel buffer depth (defaults to 16 when <= 0).
Used in the caching middleware's production path where disk write latency must not add backpressure to the upstream body stream.
func SavepartReader ¶
func SavepartReader(r io.ReadCloser, blockSize uint64, startAt uint, flushBuffer EventSuccess, flushFailed EventError, cleanup EventClose) io.ReadCloser
SavepartReader wraps an io.ReadCloser and splits the incoming byte stream into fixed-size blocks (blockSize). Each time a full block is accumulated the onSuccess callback is invoked synchronously — the caller typically writes the block to a cache file on disk.
startAt is an initial byte offset; when > 0 the reader enters skip mode, discarding data until it aligns to a block boundary, then begins buffering.
onError is called on read or write errors. onClose is called once when Close is invoked, receiving the eof flag.
This is the synchronous version; see SavepartAsyncReader for the async variant used in the production caching path.
func SkipReadCloser ¶
func SkipReadCloser(R io.ReadCloser, skip int64) io.ReadCloser
SkipReadCloser returns an io.ReadCloser that skips skip bytes before the first byte is returned to the caller. If the underlying reader supports io.Seeker, a single Seek call is used; otherwise bytes are discarded via io.CopyN to io.Discard (which may happen across multiple Read calls if the discard is interrupted).
Used by the caching middleware to skip past leading chunk data that sits before the requested Range offset.
Types ¶
type AllCloser ¶
type AllCloser []io.ReadCloser
AllCloser is a slice of io.ReadCloser that closes every element in order when its Close method is called. Individual close errors are silently discarded — this is a best-effort cleanup helper suitable for defer blocks where partial failure should not prevent the remaining resources from being released.
type Block ¶
type Block struct {
Match bool // true: hit, false: miss
BlockRange []uint32 // [first, ..., last]
}
Block describes a contiguous range of chunk indices and whether they are cached (Match=true) or missing (Match=false). A sorted list of Blocks is produced by BlockGroup to drive range-fill logic.
func BlockGroup ¶
BlockGroup compares the desired chunk set (want) against the locally-available set (hitter), producing a sorted list of Block entries grouped by consecutive hit/miss runs. The result drives the caching middleware's logic for stitching together cached file chunks and upstream range requests.
Example: if want = {0,1,2,3,4} and hitter = {1,2,4}, the result is:
[miss:0] [hit:1,2] [miss:3] [hit:4]
type EventClose ¶
type EventClose func(eof bool)
EventSuccess defines a callback for successful events with data buffer, bit index, position, and EOF flag as parameters. EventError defines a callback to handle errors passed as an argument. EventClose defines a callback invoked when the process is closed, providing an EOF flag.
type EventError ¶
type EventError func(err error)
EventSuccess defines a callback for successful events with data buffer, bit index, position, and EOF flag as parameters. EventError defines a callback to handle errors passed as an argument. EventClose defines a callback invoked when the process is closed, providing an EOF flag.
type EventSuccess ¶
EventSuccess defines a callback for successful events with data buffer, bit index, position, and EOF flag as parameters. EventError defines a callback to handle errors passed as an argument. EventClose defines a callback invoked when the process is closed, providing an EOF flag.
type ProxyCallback ¶
ProxyCallback defines a function type that returns an HTTP response and an error when called. It is used by AsyncReadCloser to perform the upstream fetch in a background goroutine.
type RateLimitedWriter ¶
type RateLimitedWriter struct {
// contains filtered or unexported fields
}
RateLimitedWriter throttles writes through a token-bucket limiter. Each Write call waits for enough tokens before writing to the underlying writer.
func NewRateLimitedWriter ¶
func NewRateLimitedWriter(w io.Writer, bytesPerSec int) *RateLimitedWriter
NewRateLimitedWriter returns a *RateLimitedWriter that limits write throughput to bytesPerSec bytes per second. The burst is set equal to one second of tokens.
Source Files
Directories