cachekey

package
v1.2.2 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: May 5, 2025 License: Apache-2.0 Imports: 7 Imported by: 0

Documentation

Overview

Package cachekey defines several helper functions for building up and decoding keys for use by clients of galaxycache.

The encoding scheme has three key design criteria: reasonable performance, simplicity and robustness

  • All fields are separated by null bytes
  • Strings are prefixed with a varint length (as handled by encoding/binary)
  • Integers are encoded as a varint.

This design is inspired by [OrderedCodes](https://github.com/google/orderedcode), but, since keys do not need to preserve ordering, we can use a slightly simpler design.

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func AppendEpoch

func AppendEpoch(buf []byte, cfg EpochConfig, now time.Time, jitter Jitter) []byte

AppendEpoch appends the jittered epoch suffix. Jitter is taken mod the epoch config's EpochLength (in nanoseconds) The epoch number is encoded with AppendUint See the comment on EpochConfig for information about how to use this tool.

func AppendStrID

func AppendStrID[I ~string](buf []byte, id I) []byte

AppendStrID encodes a string ID into a form that can be concatenated into a V2 galaxycache key

func AppendUint

func AppendUint[I ~uint64](buf []byte, id I) []byte

AppendUint encodes an unsigned integer into a form that can be concatenated into a V2 galaxycache key

func ConsumeInt

func ConsumeInt(buf []byte) (uint64, []byte, error)

ConsumeInt returns a consumed int and the remaining buffer or an error if the input is malformed

func ConsumeString

func ConsumeString(buf []byte) (string, []byte, error)

ConsumeString returns a consumed string and the remaining buffer or an error if the input is malformed

Types

type EpochConfig

type EpochConfig struct {
	// Base timestamp from which epochs are measured. By making a
	// relatively recent timestamp that still guarantees that the epoch is
	// positive, we can save a couple bytes in our keys.
	EpochZero time.Time

	// A prime number of nanoseconds indicating the length of an epoch.
	// For example, one can use 396_999_999_953 for a ~6.5-ish minute epoch
	// that's about 23s less than 7 minutes.
	// note that 23, 7 and 397 are all primes, so the epochs will wrap around nicely over time.
	EpochLength time.Duration

	// A large prime to multiply integer IDs by to ensure that adjacent IDs get spread out.
	// For example, one might use 330_999_999_991 to work with the 7 minute-ish value above
	// it would be chosen to be about 5 minutes 31 seconds in nanoseconds.
	// 331 is a prime number of seconds, as are 5 minutes and 31 seconds.
	// Due to constraints on the bits.Div64 function, we can't have too
	// large a value in hi, so this multiplier is required to be strictly
	// less than epochLength if we want to support the full uint64 space for integer IDs.
	// This should give a good spread of jitter values from epochMode.
	IntIDJitterPrimeMultiplier time.Duration
}

EpochConfig defines the tunables behind how epoch-numbers are calculated. In particular, it encapsulates the reference time, epoch length, and an integer ID jitter coefficient (so small integer IDs get spread out sufficiently).

This implements the scheme described in the [Vimeo Engineering Blog post entitled "Vimeo metadata caching at Vimeo"](https://medium.com/vimeo-engineering-blog/video-metadata-caching-at-vimeo-a54b25f0b304). In particular, it provides a way to implement TTLs without explicit support from Galaxycache by rotating the cache-key every [EpochConfig.EpochLength], and offsetting that by a bit based on the input ID, we prevent individual keys from expiring/rotating at exactly the same time.

Note: for this to be useful during rollouts, these parameters should be stable over time, so we recommend defining constants in your code for the two time.Duration fields.

type Jitter

type Jitter struct {
	// contains filtered or unexported fields
}

Jitter is an int128 opaque value used with AppendEpoch, and EpochConfig to track the epoch boundary offset for a specific ID.

func JitterIntID

func JitterIntID[I ~uint64](e *EpochConfig, id I) Jitter

JitterIntID returns a jittered offset within the ID

func JitterStrID

func JitterStrID[I ~string](id I) Jitter

JitterStrID hashes the ID and computes a jittered value using a truncated sha256.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL