Documentation
¶
Overview ¶
Package whspool is a webhook receiver's local write-ahead spool (#2121): the events of an acknowledged request are appended to a file on this replica's disk and synced before the request is answered, then drained to object storage in the background.
Without it a request is answered only once its events are in object storage, so a sender's wait is object storage's latency at that moment, and a slow object store fails deliveries at the sender while the platform is healthy. With it the wait is a local fsync.
The spool holds one directory per source. Each source has at most one active file, which requests append to, and any number of sealed files waiting to be drained, oldest first. A file is named for the process that created it and a sequence, which is also the name its segments are written under in object storage: draining a file a second time -- after a crash between its write and its removal -- overwrites the same objects instead of adding to them, so every acknowledged event lands once.
A record is its length, its CRC-32C and its JSON. A record whose sync did not finish was never acknowledged, so the torn tail a crash leaves is cut off when the spool is opened.
Index ¶
- Constants
- Variables
- type Options
- type Record
- type Segment
- type Spool
- func (s *Spool) Active(source string) (Stats, bool)
- func (s *Spool) Append(source string, rec Record) (time.Duration, error)
- func (s *Spool) Close() error
- func (s *Spool) Dir() string
- func (*Spool) Read(seg Segment) ([]Record, error)
- func (s *Spool) Remove(seg Segment) error
- func (s *Spool) Seal(source string) error
- func (s *Spool) Sealed(source string) []Segment
- func (s *Spool) Sources() []string
- func (s *Spool) Stats() Stats
- type Stats
- type Volume
Constants ¶
const ( DefaultMaxBytes int64 = 1 << 30 DefaultMaxAge = time.Hour )
Defaults for the bounds past which a spool refuses new requests.
Variables ¶
var ErrClosed = errors.New("the receiver's spool is closed")
ErrClosed is an append after the spool was closed.
var ErrFull = errors.New("the receiver's spool is full")
ErrFull is an append refused because the spool holds as much as it may: its files' bytes, or the age of its oldest event, are at their bound. The request is answered 503 so its sender retries.
Functions ¶
This section is empty.
Types ¶
type Options ¶
type Options struct {
// Dir is the spool's directory, created when it does not exist.
Dir string
// MaxBytes bounds the bytes of every file in the spool. Zero takes
// DefaultMaxBytes.
MaxBytes int64
// MaxAge bounds the age of the oldest event not yet drained. Zero takes
// DefaultMaxAge.
MaxAge time.Duration
// Replica is this process's identity, which the files it creates are
// named for.
Replica string
Now func() time.Time
}
Options open a spool.
type Segment ¶
type Segment struct {
Source string
// Writer and Seq name the file, and the objects it is drained to.
Writer string
Seq uint64
Bytes int64
Events int
// Oldest is when the file's first event was received.
Oldest time.Time
// contains filtered or unexported fields
}
Segment is one sealed spool file waiting to be drained.
type Spool ¶
type Spool struct {
// contains filtered or unexported fields
}
Spool is one replica's spool directory.
func Open ¶
Open opens the spool at o.Dir, taking this process's lock on it, and reads what an earlier process left there: every file is sealed, a torn tail is cut off, and a file with no whole record is removed.
func (*Spool) Active ¶
Active reports what the source's active file holds, as the Stats of that one file, and false when it has none.
func (*Spool) Append ¶
Append writes rec to the source's active file and syncs it. Once it returns nil the record survives a crash, and the request may be answered. It returns ErrFull when the spool is at a bound and ErrClosed once closed, and how long the write and the sync took.
func (*Spool) Close ¶
Close seals every active file and releases the lock. Nothing is lost: the next process to open the spool drains what it holds.
func (*Spool) Seal ¶
Seal closes the source's active file, synced, and queues it to be drained. The next append opens a new one.
type Stats ¶
type Stats struct {
Bytes int64
Events int64
Files int
// Oldest is when the oldest event not yet drained was received; zero
// when the spool is empty.
Oldest time.Time
}
Stats is what a spool holds: every file, active and sealed.
Source Files