Documentation
¶
Overview ¶
Package hdrhistogram provides an implementation of Gil Tene's HDR Histogram data structure. The HDR Histogram allows for fast and accurate analysis of the extreme ranges of data with non-normal distributions, like latency.
Package hdrhistogram provides facilities for encoding and decoding high dynamic range histograms using the HdrHistogram V2 format.
The V2 format uses a modified ZigZag LEB128 encoding scheme optimized for compactness:
- Consecutive zero counters are represented as a negative integer indicating the run length of zeros.
- Non-zero counters are represented as positive integers.
This encoding allows a typical histogram (with 2 digits of precision covering a range from 1 microsecond to 1 day) to be represented in less than a single MTU-sized packet (~1500 bytes).
Histograms can be serialized to or deserialized from compressed Base64-encoded binary representations for efficient transmission or archival. The implementation supports only the V2 compressed encoding format.
The log format encodes into a single file, multiple histograms with optional shared meta data.
Index ¶
- Constants
- type Bar
- type Bracket
- type Histogram
- func (h *Histogram) ByteSize() int
- func (h *Histogram) Clone() *Histogram
- func (h *Histogram) CumulativeDistribution() []Bracket
- func (h *Histogram) CumulativeDistributionWithTicks(ticksPerHalfDistance int32) []Bracket
- func (h *Histogram) Distribution() (result []Bar)
- func (h *Histogram) Encode(version int32) (buffer []byte, err error)
- func (h *Histogram) EncodeV2() ([]byte, error)
- func (h *Histogram) EndTimeMs() int64
- func (h *Histogram) Equals(other *Histogram) bool
- func (h *Histogram) Export() *Snapshot
- func (h *Histogram) HighestTrackableValue() int64
- func (h *Histogram) LowestTrackableValue() int64
- func (h *Histogram) Max() int64
- func (h *Histogram) Mean() float64
- func (h *Histogram) Merge(from *Histogram) (dropped int64)
- func (h *Histogram) Min() int64
- func (h *Histogram) PercentilesPrint(writer io.Writer, ticksPerHalfDistance int32, valueScale float64) (outputWriter io.Writer, err error)
- func (h *Histogram) RecordCorrectedValue(v, expectedInterval int64) error
- func (h *Histogram) RecordValue(v int64) error
- func (h *Histogram) RecordValues(v, n int64) error
- func (h *Histogram) Reset()
- func (h *Histogram) SetEndTimeMs(endTimeMs int64)
- func (h *Histogram) SetStartTimeMs(startTimeMs int64)
- func (h *Histogram) SetTag(tag string)
- func (h *Histogram) SignificantFigures() int64
- func (h *Histogram) StartTimeMs() int64
- func (h *Histogram) StdDev() float64
- func (h *Histogram) Tag() string
- func (h *Histogram) TotalCount() int64
- func (h *Histogram) ValueAtPercentile(percentile float64) int64
- func (h *Histogram) ValueAtPercentiles(percentiles []float64) (values map[float64]int64)
- func (h *Histogram) ValueAtPercentilesSlice(percentiles []float64) []int64
- func (h *Histogram) ValueAtQuantile(q float64) int64
- func (h *Histogram) ValuesAreEquivalent(value1, value2 int64) (result bool)
- type HistogramLogOptions
- type HistogramLogReader
- func (hlr *HistogramLogReader) NextIntervalHistogram() (histogram *Histogram, err error)
- func (hlr *HistogramLogReader) NextIntervalHistogramWithRange(rangeStartTimeSec, rangeEndTimeSec float64, absolute bool) (histogram *Histogram, err error)
- func (hlr *HistogramLogReader) ObservedMax() bool
- func (hlr *HistogramLogReader) ObservedMin() bool
- func (hlr *HistogramLogReader) RangeObservedMax() int64
- func (hlr *HistogramLogReader) RangeObservedMin() int64
- type HistogramLogWriter
- func (lw *HistogramLogWriter) BaseTime() int64
- func (lw *HistogramLogWriter) OutputBaseTime(msec int64) (err error)
- func (lw *HistogramLogWriter) OutputComment(comment string) (err error)
- func (lw *HistogramLogWriter) OutputIntervalHistogram(histogram *Histogram) (err error)
- func (lw *HistogramLogWriter) OutputIntervalHistogramWithLogOptions(histogram *Histogram, logOptions *HistogramLogOptions) (err error)
- func (lw *HistogramLogWriter) OutputIntervalPackedHistogram(histogram *PackedHistogram) error
- func (lw *HistogramLogWriter) OutputIntervalPackedHistogramWithLogOptions(histogram *PackedHistogram, logOptions *HistogramLogOptions) error
- func (lw *HistogramLogWriter) OutputLegend() (err error)
- func (lw *HistogramLogWriter) OutputLogFormatVersion() (err error)
- func (lw *HistogramLogWriter) OutputStartTime(msec int64) (err error)
- func (lw *HistogramLogWriter) SetBaseTime(baseTime int64)
- type PackedHistogram
- func (p *PackedHistogram) Clone() *PackedHistogram
- func (p *PackedHistogram) Compact()
- func (p *PackedHistogram) CountAtValue(v int64) int64
- func (p *PackedHistogram) CountWidth() int
- func (p *PackedHistogram) Encode() ([]byte, error)
- func (p *PackedHistogram) EncodeV2() ([]byte, error)
- func (p *PackedHistogram) EndTimeMs() int64
- func (p *PackedHistogram) ForEachBucket(fn func(value, count int64) bool)
- func (p *PackedHistogram) GetMemorySize() int
- func (p *PackedHistogram) HighestTrackableValue() int64
- func (p *PackedHistogram) LowestTrackableValue() int64
- func (p *PackedHistogram) Max() int64
- func (p *PackedHistogram) Mean() float64
- func (p *PackedHistogram) Merge(from *PackedHistogram) (dropped int64)
- func (p *PackedHistogram) MergeFrom(src *Histogram) (dropped int64)
- func (p *PackedHistogram) MergeInto(dst *Histogram) (dropped int64)
- func (p *PackedHistogram) Min() int64
- func (p *PackedHistogram) Populated() int32
- func (p *PackedHistogram) RecordCorrectedValue(v, expectedInterval int64) error
- func (p *PackedHistogram) RecordValue(v int64) error
- func (p *PackedHistogram) RecordValues(v, n int64) error
- func (p *PackedHistogram) Reset()
- func (p *PackedHistogram) SetEndTimeMs(endTimeMs int64)
- func (p *PackedHistogram) SetStartTimeMs(startTimeMs int64)
- func (p *PackedHistogram) SetTag(tag string)
- func (p *PackedHistogram) SignificantFigures() int64
- func (p *PackedHistogram) StartTimeMs() int64
- func (p *PackedHistogram) StdDev() float64
- func (p *PackedHistogram) Tag() string
- func (p *PackedHistogram) TotalCount() int64
- func (p *PackedHistogram) ValueAtPercentile(percentile float64) int64
- func (p *PackedHistogram) ValueAtPercentilesSlice(percentiles []float64) []int64
- type Snapshot
- type WindowedHistogram
Examples ¶
Constants ¶
const ( V2EncodingCookieBase int32 = 0x1c849303 V2CompressedEncodingCookieBase int32 = 0x1c849304 ENCODING_HEADER_SIZE = 40 )
const HISTOGRAM_LOG_FORMAT_VERSION = "1.3"
const MsToNsRatio float64 = 1000000.0
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type Histogram ¶
type Histogram struct {
// contains filtered or unexported fields
}
A Histogram is a lossy data structure used to record the distribution of non-normally distributed data (like latency) with a high degree of accuracy and a bounded degree of precision.
A Histogram must not be copied by value after it is created: a struct copy shares the counts array but keeps its own total, so recording into or resetting one makes the other inconsistent. Use Clone for an independent copy (Import(h.Export()) also gives independent storage, but keeps only the geometry, counts and conversion ratio, not the tag or start/end times). A Histogram provides no internal synchronization: concurrent reads are safe only while nothing mutates it, and callers must synchronize recording, Reset and Merge against all other use.
func Decode ¶ added in v1.0.0
Decode returns a new Histogram by decoding it from a String containing a base64 encoded compressed histogram representation. It uses the serialized geometry exactly, including zero significant digits; precision outside 0-5 and unrepresentable geometry are rejected. Lowest values below 1 are read as 1 for compatibility with streams written by hdrhistogram-go v1.0.0. Any range Go's constructors accept decodes. The conversion ratio is kept as metadata and written back by Encode.
The normalizingIndexOffset header field is ignored: writers serialize the payload in logical index order (Java's encoder reads through its in-memory rotation), so the field only describes the writer's internal layout. This covers shifted Java histograms and the offset 1 that hdrhistogram-go v1.2.0 and earlier wrote. A payload whose bucket counts sum past MaxInt64 is rejected, since a dense histogram cannot represent that total (DecodePacked keeps such buckets and saturates its total).
func Import ¶
Import returns a new Histogram populated from the Snapshot data. It returns no error, so call Snapshot.Validate first for snapshots that may be invalid. For a snapshot that passes Validate, the histogram's total equals the sum of its counts. Otherwise: significant figures outside 0-5 are clamped as New clamps them, geometry New cannot represent panics as New does, negative counts are imported as zero (the total has always excluded them), counts beyond the geometry's range are dropped, and counts summing past math.MaxInt64 wrap the total.
func New ¶
func New(lowestDiscernibleValue, highestTrackableValue int64, numberOfSignificantValueDigits int) *Histogram
Construct a Histogram given the Lowest and Highest values to be tracked and a number of significant decimal digits.
Providing a lowestDiscernibleValue is useful in situations where the units used for the histogram's values are much smaller that the minimal accuracy required. E.g. when tracking time values stated in nanosecond units, where the minimal accuracy required is a microsecond, the proper value for lowestDiscernibleValue would be 1000.
Note: the numberOfSignificantValueDigits must be [1,5]. If lower than 1 the numberOfSignificantValueDigits will be forced to 1, and if higher than 5 the numberOfSignificantValueDigits will be forced to 5.
Example ¶
This latency Histogram could be used to track and analyze the counts of observed integer values between 1 us and 30000000 us ( 30 secs ) while maintaining a value precision of 4 significant digits across that range, translating to a value resolution of :
- 1 microsecond up to 10 milliseconds,
- 100 microsecond (or better) from 10 milliseconds up to 10 seconds,
- 300 microsecond (or better) from 10 seconds up to 30 seconds,
nolint
package main
import (
"fmt"
"github.com/HdrHistogram/hdrhistogram-go"
)
func main() {
lH := hdrhistogram.New(1, 30000000, 4)
input := []int64{
459876, 669187, 711612, 816326, 931423, 1033197, 1131895, 2477317,
3964974, 12718782,
}
for _, sample := range input {
lH.RecordValue(sample)
}
fmt.Printf("Percentile 50: %d\n", lH.ValueAtQuantile(50.0))
}
Output: Percentile 50: 931423
func (*Histogram) ByteSize ¶
ByteSize returns an estimate of the amount of memory allocated to the histogram in bytes.
N.B.: This does not take into account the overhead for slices, which are small, constant, and specific to the compiler version.
func (*Histogram) Clone ¶ added in v1.4.0
Clone returns an independent deep copy of h: the same geometry, counts, total, conversion ratio, tag and start/end times, with its own storage. Unlike a struct copy, later changes to either histogram do not affect the other.
func (*Histogram) CumulativeDistribution ¶
CumulativeDistribution returns an ordered list of brackets of the distribution of recorded values.
func (*Histogram) CumulativeDistributionWithTicks ¶ added in v1.0.0
CumulativeDistribution returns an ordered list of brackets of the distribution of recorded values.
func (*Histogram) Distribution ¶
Distribution returns an ordered list of bars of the distribution of recorded values, counts can be normalized to a probability
func (*Histogram) Encode ¶ added in v1.0.0
Encode returns a snapshot view of the Histogram. The snapshot is compact binary representations of the state of the histogram. They are intended to be used for archival or transmission to other systems for further analysis.
func (*Histogram) EncodeV2 ¶ added in v1.4.0
EncodeV2 returns the standard V2 compressed, base64-encoded representation. Both histogram types expose this method for use through a common interface.
func (*Histogram) Export ¶
Export returns a snapshot view of the Histogram. This can be later passed to Import to construct a new Histogram with the same state.
func (*Histogram) HighestTrackableValue ¶
HighestTrackableValue returns the upper bound on values that will be added to the histogram
func (*Histogram) LowestTrackableValue ¶
LowestTrackableValue returns the lower bound on values that will be added to the histogram
func (*Histogram) Merge ¶
Merge merges the data stored in the given histogram with the receiver, returning the number of recorded values which had to be dropped: values outside the receiver's range, and counts that would overflow its total count (RecordValues rejects those). The dropped total saturates at math.MaxInt64 rather than wrapping.
func (*Histogram) PercentilesPrint ¶ added in v1.0.0
func (h *Histogram) PercentilesPrint(writer io.Writer, ticksPerHalfDistance int32, valueScale float64) (outputWriter io.Writer, err error)
Output the percentiles distribution in a text format
Example ¶
The following example details the creation of an histogram used to track and analyze the counts of observed integer values between 0 us and 30000000 us ( 30 secs ) and the printing of the percentile output format nolint
package main
import (
"os"
"github.com/HdrHistogram/hdrhistogram-go"
)
func main() {
lH := hdrhistogram.New(1, 30000000, 3)
input := []int64{
459876, 669187, 711612, 816326, 931423, 1033197, 1131895, 2477317,
3964974, 12718782,
}
for _, sample := range input {
lH.RecordValue(sample)
}
lH.PercentilesPrint(os.Stdout, 1, 1.0)
}
Output: Value Percentile TotalCount 1/(1-Percentile) 460031.000 0.000000 1 1.00 931839.000 0.500000 5 2.00 2478079.000 0.750000 8 4.00 3966975.000 0.875000 9 8.00 12722175.000 0.937500 10 16.00 12722175.000 1.000000 10 inf #[Mean = 2491481.600, StdDeviation = 3557920.109] #[Max = 12722175.000, Total count = 10] #[Buckets = 15, SubBuckets = 2048]
func (*Histogram) RecordCorrectedValue ¶
RecordCorrectedValue records the given value, correcting for stalls in the recording process. This only works for processes which are recording values at an expected interval (e.g., doing jitter analysis). Processes which are recording ad-hoc values (e.g., latency for incoming requests) can't take advantage of this. As with PackedHistogram, it stops at the first value that cannot be recorded and returns that error; values already recorded remain.
func (*Histogram) RecordValue ¶
RecordValue records the given value, returning an error if the value is out of range.
Example ¶
This latency Histogram could be used to track and analyze the counts of observed integer values between 0 us and 30000000 us ( 30 secs ) while maintaining a value precision of 3 significant digits across that range, translating to a value resolution of :
- 1 microsecond up to 1 millisecond,
- 1 millisecond (or better) up to one second,
- 1 second (or better) up to it's maximum tracked value ( 30 seconds ).
nolint
package main
import (
"fmt"
"github.com/HdrHistogram/hdrhistogram-go"
)
func main() {
lH := hdrhistogram.New(1, 30000000, 3)
input := []int64{
459876, 669187, 711612, 816326, 931423, 1033197, 1131895, 2477317,
3964974, 12718782,
}
for _, sample := range input {
lH.RecordValue(sample)
}
fmt.Printf("Percentile 50: %d\n", lH.ValueAtQuantile(50.0))
}
Output: Percentile 50: 931839
func (*Histogram) RecordValues ¶
RecordValues records n occurrences of the given value, returning an error if the value is out of range, n is negative, or n would push the total count past math.MaxInt64. On error the histogram is unchanged.
func (*Histogram) Reset ¶
func (h *Histogram) Reset()
Reset deletes all recorded values and restores the histogram to its original state.
func (*Histogram) SetEndTimeMs ¶ added in v1.0.0
func (*Histogram) SetStartTimeMs ¶ added in v1.0.0
func (*Histogram) SignificantFigures ¶
SignificantFigures returns the significant figures used to create the histogram
func (*Histogram) StartTimeMs ¶ added in v1.0.0
func (*Histogram) StdDev ¶
StdDev returns the approximate standard deviation of the recorded values.
func (*Histogram) TotalCount ¶
TotalCount returns total number of values recorded.
func (*Histogram) ValueAtPercentile ¶ added in v1.1.0
ValueAtPercentile returns the largest value that (100% - percentile) of the overall recorded value entries in the histogram are either larger than or equivalent to.
The passed percentile must be a float64 value in [0.0 .. 100.0] Note that two values are "equivalent" if `ValuesAreEquivalent(value1,value2)` would return true.
Returns 0 if no recorded values exist.
func (*Histogram) ValueAtPercentiles ¶ added in v1.1.0
ValueAtPercentiles, given an slice of percentiles returns a map containing for each passed percentile, the largest value that (100% - percentile) of the overall recorded value entries in the histogram are either larger than or equivalent to.
Each element in the given an slice of percentiles must be a float64 value in [0.0 .. 100.0] Note that two values are "equivalent" if `ValuesAreEquivalent(value1,value2)` would return true.
Returns a map of 0's if no recorded values exist.
Example ¶
When doing an percentile analysis we normally require more than one percentile to be calculated for the given histogram.
When that is the case ValueAtPercentiles() will deeply optimize the total time to retrieve the percentiles vs the other option which is multiple calls to ValueAtQuantile().
nolint
package main
import (
"fmt"
"github.com/HdrHistogram/hdrhistogram-go"
)
func main() {
histogram := hdrhistogram.New(1, 30000000, 3)
for i := 0; i < 1000000; i++ {
histogram.RecordValue(int64(i))
}
percentileValuesMap := histogram.ValueAtPercentiles([]float64{50.0, 95.0, 99.0, 99.9})
fmt.Printf("Percentile 50: %d\n", percentileValuesMap[50.0])
fmt.Printf("Percentile 95: %d\n", percentileValuesMap[95.0])
fmt.Printf("Percentile 99: %d\n", percentileValuesMap[99.0])
fmt.Printf("Percentile 99.9: %d\n", percentileValuesMap[99.9])
}
Output: Percentile 50: 500223 Percentile 95: 950271 Percentile 99: 990207 Percentile 99.9: 999423
func (*Histogram) ValueAtPercentilesSlice ¶ added in v1.3.0
ValueAtPercentilesSlice returns, for each requested percentile, the largest value that (100% - percentile) of the recorded entries are larger than or equivalent to — the same values as ValueAtPercentiles, but written to a []int64 in the SAME ORDER as percentiles (input order is preserved even if percentiles is unsorted or has duplicates).
It avoids the per-call map allocation and float64-key hashing of ValueAtPercentiles, and does NOT mutate the input slice. Each percentile is clamped to [0, 100]; an empty histogram yields all 0's; percentile 0.0 uses the lowest equivalent value.
func (*Histogram) ValueAtQuantile ¶
ValueAtQuantile returns the largest value that (100% - percentile) of the overall recorded value entries in the histogram are either larger than or equivalent to.
The passed quantile must be a float64 value in [0.0 .. 100.0] Note that two values are "equivalent" if `ValuesAreEquivalent(value1,value2)` would return true.
Returns 0 if no recorded values exist.
func (*Histogram) ValuesAreEquivalent ¶ added in v1.0.1
Determine if two values are equivalent with the histogram's resolution. Where "equivalent" means that value samples recorded for any two equivalent values are counted in a common total count.
type HistogramLogOptions ¶ added in v1.0.0
type HistogramLogOptions struct {
// contains filtered or unexported fields
}
func DefaultHistogramLogOptions ¶ added in v1.0.0
func DefaultHistogramLogOptions() *HistogramLogOptions
type HistogramLogReader ¶ added in v1.0.0
type HistogramLogReader struct {
// contains filtered or unexported fields
}
func NewHistogramLogReader ¶ added in v1.0.0
func NewHistogramLogReader(log io.Reader) *HistogramLogReader
Example ¶
The log format encodes into a single file, multiple histograms with optional shared meta data. The following example showcases reading a log file into a slice of histograms nolint
package main
import (
"bytes"
"fmt"
"os"
hdrhistogram "github.com/HdrHistogram/hdrhistogram-go"
)
func main() {
dat, _ := os.ReadFile("./test/tagged-Log.logV2.hlog")
r := bytes.NewReader(dat)
// Create a histogram log reader
reader := hdrhistogram.NewHistogramLogReader(r)
var histograms []*hdrhistogram.Histogram = make([]*hdrhistogram.Histogram, 0)
// Read all histograms in the file
for hist, err := reader.NextIntervalHistogram(); hist != nil && err == nil; hist, err = reader.NextIntervalHistogram() {
histograms = append(histograms, hist)
}
fmt.Printf("Read a total of %d histograms\n", len(histograms))
min := reader.RangeObservedMin()
max := reader.RangeObservedMax()
sigdigits := 3
overallHistogram := hdrhistogram.New(min, max, sigdigits)
//// We can then merge all histograms into one and retrieve overall metrics
for _, hist := range histograms {
overallHistogram.Merge(hist)
}
fmt.Printf("Overall count: %d samples\n", overallHistogram.TotalCount())
fmt.Printf("Overall Percentile 50: %d\n", overallHistogram.ValueAtQuantile(50.0))
}
Output: Read a total of 42 histograms Overall count: 32290 samples Overall Percentile 50: 344319
func (*HistogramLogReader) NextIntervalHistogram ¶ added in v1.0.0
func (hlr *HistogramLogReader) NextIntervalHistogram() (histogram *Histogram, err error)
func (*HistogramLogReader) NextIntervalHistogramWithRange ¶ added in v1.0.0
func (hlr *HistogramLogReader) NextIntervalHistogramWithRange(rangeStartTimeSec, rangeEndTimeSec float64, absolute bool) (histogram *Histogram, err error)
func (*HistogramLogReader) ObservedMax ¶ added in v1.0.0
func (hlr *HistogramLogReader) ObservedMax() bool
func (*HistogramLogReader) ObservedMin ¶ added in v1.0.0
func (hlr *HistogramLogReader) ObservedMin() bool
func (*HistogramLogReader) RangeObservedMax ¶ added in v1.0.0
func (hlr *HistogramLogReader) RangeObservedMax() int64
RangeObservedMax returns the overall observed max limit ( up to the current point ) of the read histograms
func (*HistogramLogReader) RangeObservedMin ¶ added in v1.0.0
func (hlr *HistogramLogReader) RangeObservedMin() int64
RangeObservedMin returns the overall observed min limit ( up to the current point ) of the read histograms
type HistogramLogWriter ¶ added in v1.0.0
type HistogramLogWriter struct {
// contains filtered or unexported fields
}
func NewHistogramLogWriter ¶ added in v1.0.0
func NewHistogramLogWriter(log io.Writer) *HistogramLogWriter
Example ¶
The log format encodes into a single file, multiple histograms with optional shared meta data. The following example showcases writing multiple histograms into a log file and then processing them again to confirm a proper encode-decode flow nolint
package main
import (
"bytes"
"fmt"
"os"
hdrhistogram "github.com/HdrHistogram/hdrhistogram-go"
)
func main() {
var buff bytes.Buffer
// Create a histogram log writer to write to a bytes.Buffer
writer := hdrhistogram.NewHistogramLogWriter(&buff)
writer.OutputLogFormatVersion()
writer.OutputStartTime(0)
writer.OutputLegend()
// Lets create 3 distinct histograms to exemply the logwriter features
// each one with a time-frame of 60 secs ( 60000 ms )
hist1 := hdrhistogram.New(1, 30000000, 3)
hist1.SetStartTimeMs(0)
hist1.SetEndTimeMs(60000)
for _, sample := range []int64{10, 20, 30, 40} {
hist1.RecordValue(sample)
}
hist2 := hdrhistogram.New(1, 3000, 3)
hist1.SetStartTimeMs(60001)
hist1.SetEndTimeMs(120000)
for _, sample := range []int64{50, 70, 80, 60} {
hist2.RecordValue(sample)
}
hist3 := hdrhistogram.New(1, 30000, 3)
hist1.SetStartTimeMs(120001)
hist1.SetEndTimeMs(180000)
for _, sample := range []int64{90, 100} {
hist3.RecordValue(sample)
}
writer.OutputIntervalHistogram(hist1)
writer.OutputIntervalHistogram(hist2)
writer.OutputIntervalHistogram(hist3)
err := os.WriteFile("example.logV2.hlog", buff.Bytes(), 0644)
if err != nil {
panic(err)
}
// read check
// Lets read all again and confirm that the total sample count is 10
dat, _ := os.ReadFile("example.logV2.hlog")
r := bytes.NewReader(dat)
// Create a histogram log reader
reader := hdrhistogram.NewHistogramLogReader(r)
var histograms []*hdrhistogram.Histogram = make([]*hdrhistogram.Histogram, 0)
// Read all histograms in the file
for hist, err := reader.NextIntervalHistogram(); hist != nil && err == nil; hist, err = reader.NextIntervalHistogram() {
histograms = append(histograms, hist)
}
fmt.Printf("Read a total of %d histograms\n", len(histograms))
min := reader.RangeObservedMin()
max := reader.RangeObservedMax()
sigdigits := 3
overallHistogram := hdrhistogram.New(min, max, sigdigits)
//// We can then merge all histograms into one and retrieve overall metrics
for _, hist := range histograms {
overallHistogram.Merge(hist)
}
fmt.Printf("Overall count: %d samples\n", overallHistogram.TotalCount())
}
Output: Read a total of 3 histograms Overall count: 10 samples
func (*HistogramLogWriter) BaseTime ¶ added in v1.0.0
func (lw *HistogramLogWriter) BaseTime() int64
BaseTime returns the current base time offset
func (*HistogramLogWriter) OutputBaseTime ¶ added in v1.0.0
func (lw *HistogramLogWriter) OutputBaseTime(msec int64) (err error)
OutputBaseTime logs a base time in the log. Base time is represented as seconds since epoch with up to 3 decimal places. Line starts with the leading text '#[BaseTime:'
func (*HistogramLogWriter) OutputComment ¶ added in v1.0.0
func (lw *HistogramLogWriter) OutputComment(comment string) (err error)
OutputComment logs a comment to the log. A comment is any line that leads with '#' that is not matched by the BaseTime or StartTime formats. Comments are ignored when parsed.
func (*HistogramLogWriter) OutputIntervalHistogram ¶ added in v1.0.0
func (lw *HistogramLogWriter) OutputIntervalHistogram(histogram *Histogram) (err error)
OutputIntervalHistogram outputs an interval histogram, using the start/end timestamp indicated in the histogram, and the [optional] tag associated with the histogram. The histogram start and end timestamps are assumed to be in msec units
By convention, histogram start/end time are generally stamped with absolute times in msec since the epoch. For logging with absolute time stamps, the base time would remain zero ( default ). For logging with relative time stamps (time since a start point), the base time should be set with SetBaseTime(baseTime int64)
The max value in the histogram will be reported scaled down by a default maxValueUnitRatio of 1000000.0 (which is the msec : nsec ratio). If you need to specify a different start/end timestamp or a different maxValueUnitRatio you should use OutputIntervalHistogramWithLogOptions(histogram *Histogram, logOptions *HistogramLogOptions)
func (*HistogramLogWriter) OutputIntervalHistogramWithLogOptions ¶ added in v1.0.0
func (lw *HistogramLogWriter) OutputIntervalHistogramWithLogOptions(histogram *Histogram, logOptions *HistogramLogOptions) (err error)
OutputIntervalHistogramWithLogOptions outputs an interval histogram, with the given timestamp information and the [optional] tag associated with the histogram
If you specify non-nil logOptions, and non-zero start timestamp, the specified timestamp information will be used, and the start timestamp information in the actual histogram will be ignored. If you specify non-nil logOptions, and non-zero start timestamp, the specified timestamp information will be used, and the end timestamp information in the actual histogram will be ignored. If you specify non-nil logOptions, The max value reported with the interval line will be scaled by the given maxValueUnitRatio, otherwise a default maxValueUnitRatio of 1,000,000 (which is the msec : nsec ratio) will be used.
By convention, histogram start/end time are generally stamped with absolute times in msec since the epoch. For logging with absolute time stamps, the base time would remain zero ( default ). For logging with relative time stamps (time since a start point), the base time should be set with SetBaseTime(baseTime int64)
func (*HistogramLogWriter) OutputIntervalPackedHistogram ¶ added in v1.4.0
func (lw *HistogramLogWriter) OutputIntervalPackedHistogram(histogram *PackedHistogram) error
OutputIntervalPackedHistogram outputs a packed interval histogram using its attached timestamps and optional tag. Timestamps are in milliseconds; the writer subtracts BaseTime and scales the reported maximum by MsToNsRatio, just as OutputIntervalHistogram does for dense histograms.
func (*HistogramLogWriter) OutputIntervalPackedHistogramWithLogOptions ¶ added in v1.4.0
func (lw *HistogramLogWriter) OutputIntervalPackedHistogramWithLogOptions(histogram *PackedHistogram, logOptions *HistogramLogOptions) error
OutputIntervalPackedHistogramWithLogOptions outputs a packed interval histogram using the same options as OutputIntervalHistogramWithLogOptions. Encoding operates directly on sparse storage.
func (*HistogramLogWriter) OutputLegend ¶ added in v1.0.0
func (lw *HistogramLogWriter) OutputLegend() (err error)
OutputLegend outputs a legend line to the log. Human-readable column headers. Ignored when parsed.
func (*HistogramLogWriter) OutputLogFormatVersion ¶ added in v1.0.0
func (lw *HistogramLogWriter) OutputLogFormatVersion() (err error)
OutputLogFormatVersion outputs a log format version to the log.
func (*HistogramLogWriter) OutputStartTime ¶ added in v1.0.0
func (lw *HistogramLogWriter) OutputStartTime(msec int64) (err error)
OutputStartTime logs a start time in the log. Start time is represented as seconds since epoch with up to 3 decimal places. Line starts with the leading text '#[StartTime:'
func (*HistogramLogWriter) SetBaseTime ¶ added in v1.0.0
func (lw *HistogramLogWriter) SetBaseTime(baseTime int64)
SetBaseTime sets a base time to subtract from supplied histogram start/end timestamps when logging based on histogram timestamps. baseTime is expected to be in msec since the epoch, as histogram start/end times are typically stamped with absolute times in msec since the epoch.
type PackedHistogram ¶ added in v1.4.0
type PackedHistogram struct {
// contains filtered or unexported fields
}
PackedHistogram is a memory-optimised sparse variant of Histogram. Create it with NewPacked; the zero value is not ready for use. It keeps the same bucket geometry as Histogram without allocating a dense counts array. Its storage grows with the number of populated buckets, using sorted indexes and counts of a uniform adaptive width (1, 2, 4 or 8 bytes).
Recording into an existing bucket uses a binary search. Populating a new bucket also shifts the following entries and costs O(populated buckets). Filling many distinct buckets can therefore take quadratic time; prefer Histogram for hot recording paths or when most buckets will be populated.
A PackedHistogram must not be copied after initialization: a struct copy shares backing arrays but has separate totals and other bookkeeping. Use Clone for an independent copy preserving exact geometry, bucket counts (including decoded distributions with saturated totals), and log metadata. Clone followed by Reset makes an empty sibling with exactly the same geometry. Reconstructing from geometry getters with NewPacked applies constructor normalization, which cannot preserve decoded zero-digit precision. Encode followed by DecodePacked also creates independent storage and keeps the conversion ratio, but requires valid wire geometry and does not retain log metadata.
PackedHistogram provides no internal synchronization. Concurrent read-only calls are safe only while no goroutine mutates the histogram. Callers must synchronize all mutation, including recording, Reset, Compact, merges and metadata setters, against reads and other mutations. A merge also requires its source to remain unchanged for the duration of the call.
func DecodePacked ¶ added in v1.4.0
func DecodePacked(encoded []byte) (*PackedHistogram, error)
DecodePacked decodes a standard V2 compressed (base64) stream into a new PackedHistogram. Like Decode, it ignores the normalizingIndexOffset header field, which only describes the writer's in-memory layout (the payload is in logical index order), so shifted Java histograms and streams from hdrhistogram-go v1.2.0 and earlier (offset 1) decode. It uses the serialized geometry exactly, including zero significant digits; precision outside 0-5 and unrepresentable geometry are rejected. Lowest values below 1 are read as 1 for compatibility with hdrhistogram-go v1.0.0 streams, and any range Go's constructors accept decodes. The conversion ratio is kept as metadata (counts remain integer bucket counts) and written back by Encode.
func NewPacked ¶ added in v1.4.0
func NewPacked(lowestDiscernibleValue, highestTrackableValue int64, numberOfSignificantValueDigits int) *PackedHistogram
NewPacked creates a sparse histogram with the same arguments as New. Its per-instance geometry oracle never allocates a counts array.
func (*PackedHistogram) Clone ¶ added in v1.4.0
func (p *PackedHistogram) Clone() *PackedHistogram
Clone returns an independent sparse copy, preserving the exact geometry, bucket counts, count width, and metadata. This includes decoded distributions whose counts exceed MaxInt64, with their saturated TotalCount. It allocates storage proportional to the populated buckets, without a dense counts array or spare capacity retained by Reset.
To create an empty histogram with exactly the same geometry, call Clone followed by Reset. This also preserves decoded geometry that NewPacked would normalize if reconstructed using the geometry accessors.
func (*PackedHistogram) Compact ¶ added in v1.4.0
func (p *PackedHistogram) Compact()
Compact shrinks the histogram's storage to fit its contents: the count width becomes the narrowest that holds the largest current count, and spare capacity is released. After Reset it returns the histogram to the footprint of a new one, so a ring slot widened by a burst can be shrunk in place. It allocates exactly-sized arrays (none when empty).
func (*PackedHistogram) CountAtValue ¶ added in v1.4.0
func (p *PackedHistogram) CountAtValue(v int64) int64
CountAtValue returns the recorded count at v's bucket (0 if out of range). The range is the same one RecordValues accepts, which extends past highestTrackableValue up to the top of the last bucket.
func (*PackedHistogram) CountWidth ¶ added in v1.4.0
func (p *PackedHistogram) CountWidth() int
CountWidth returns the current per-count byte width (1/2/4/8).
func (*PackedHistogram) Encode ¶ added in v1.4.0
func (p *PackedHistogram) Encode() ([]byte, error)
Encode returns the standard V2 compressed (base64) representation, byte-for-byte identical to the dense encoder on equivalent data.
func (*PackedHistogram) EncodeV2 ¶ added in v1.4.0
func (p *PackedHistogram) EncodeV2() ([]byte, error)
EncodeV2 returns the standard V2 compressed, base64-encoded representation directly from sparse storage, without constructing a dense histogram.
func (*PackedHistogram) EndTimeMs ¶ added in v1.4.0
func (p *PackedHistogram) EndTimeMs() int64
EndTimeMs returns the interval end time in milliseconds.
func (*PackedHistogram) ForEachBucket ¶ added in v1.4.0
func (p *PackedHistogram) ForEachBucket(fn func(value, count int64) bool)
ForEachBucket calls fn once per populated bucket, in ascending value order, with the bucket's value (the lowest value equivalent to it, as dense iteration reports it) and its count. It stops early if fn returns false. fn must not modify p. The work is proportional to the number of populated buckets and it does not allocate.
Its signature is an iter.Seq2, so it also works with range:
for value, count := range p.ForEachBucket { ... }
func (*PackedHistogram) GetMemorySize ¶ added in v1.4.0
func (p *PackedHistogram) GetMemorySize() int
GetMemorySize returns the approximate bytes held by this histogram: the struct, its per-instance geometry oracle, and the idx + cnt capacities.
func (*PackedHistogram) HighestTrackableValue ¶ added in v1.4.0
func (p *PackedHistogram) HighestTrackableValue() int64
HighestTrackableValue returns the configured upper bound, consistently with Histogram.HighestTrackableValue. The allocated bucket geometry may also accept larger values in the final bucket.
func (*PackedHistogram) LowestTrackableValue ¶ added in v1.4.0
func (p *PackedHistogram) LowestTrackableValue() int64
LowestTrackableValue returns the configured lower bound, after constructor normalization, consistently with Histogram.LowestTrackableValue.
func (*PackedHistogram) Max ¶ added in v1.4.0
func (p *PackedHistogram) Max() int64
Max mirrors dense: the highest-equivalent of the last populated bucket (overflow-safe), matching Histogram.Max bit-for-bit.
func (*PackedHistogram) Mean ¶ added in v1.4.0
func (p *PackedHistogram) Mean() float64
Mean returns the approximate arithmetic mean of the recorded values, using the median of each equivalent-value range. It returns zero when empty. It visits only populated buckets and does not allocate.
If a decoded histogram's counts exceed MaxInt64, TotalCount is saturated but Mean uses the sum of the bucket counts in float64 to normalize the distribution.
func (*PackedHistogram) Merge ¶ added in v1.4.0
func (p *PackedHistogram) Merge(from *PackedHistogram) (dropped int64)
Merge adds the counts of another packed histogram to p and returns the total count that p could not hold. The result is that of recording every populated bucket of from into p in value order: buckets whose value is out of p's range are dropped, and so are counts that would overflow p's total. from must not share backing storage with p (a struct copy of p does); p.Merge(p) itself is allowed and doubles every count.
When both index values the same way (see MergeInto) and the totals cannot overflow, the two sorted bucket lists are merged in place from the end in a single pass, linear in both sizes; it allocates only when p needs more capacity. Otherwise each bucket is re-recorded at its value.
The returned dropped total saturates at math.MaxInt64 rather than wrapping (a decoded source can hold buckets whose sum exceeds int64).
func (*PackedHistogram) MergeFrom ¶ added in v1.4.0
func (p *PackedHistogram) MergeFrom(src *Histogram) (dropped int64)
MergeFrom adds the counts of a dense histogram to p and returns the total count that p could not hold: buckets whose value is out of p's range, and counts that would overflow p's total (PackedHistogram rejects those rather than wrapping). Non-positive dense counts, which only a corrupted dense histogram can hold, are skipped and not counted as dropped. It scans src's counts once and builds no temporary; with p empty, as after Reset, each bucket is appended in order, and into a slot with enough capacity it does not allocate.
To move a completed dense slice into a reused slot:
slot.Reset() slot.MergeFrom(active) active.Reset()
The returned dropped total saturates at math.MaxInt64 rather than wrapping (a decoded source can hold buckets whose sum exceeds int64).
func (*PackedHistogram) MergeInto ¶ added in v1.4.0
func (p *PackedHistogram) MergeInto(dst *Histogram) (dropped int64)
MergeInto adds this histogram's counts to dst and returns the total count that dst could not hold. For a source whose bucket-count sum fits in int64, the result matches dst.Merge applied to its dense equivalent: each populated bucket is recorded at its value, and buckets outside dst's range are dropped. As with Histogram.Merge, a bucket whose count would push dst's total past math.MaxInt64 is dropped rather than wrapping the total.
DecodePacked can preserve positive buckets whose sum exceeds MaxInt64 while saturating TotalCount. MergeInto visits all of these buckets and drops the counts dst cannot hold.
When dst indexes values the same way (the same significant digits and the same unit magnitude, floor(log2(lowestDiscernibleValue))), counts are added index to index, with work proportional to the number of populated buckets. Otherwise each bucket is re-recorded at its value. Neither path allocates.
The returned dropped total saturates at math.MaxInt64 rather than wrapping (a decoded source can hold buckets whose sum exceeds int64).
Example ¶
A rolling window: record into one dense histogram, move each completed one-second slice into a reused packed slot of a ring, and merge the retained slices into a dense aggregate when percentiles are needed.
package main
import (
"fmt"
"github.com/HdrHistogram/hdrhistogram-go"
)
func main() {
const slots = 3
active := hdrhistogram.New(1, 30000000, 3)
ring := make([]*hdrhistogram.PackedHistogram, slots)
for i := range ring {
ring[i] = hdrhistogram.NewPacked(1, 30000000, 3)
}
for second := 0; second < 5; second++ {
for v := int64(1); v <= 100; v++ {
_ = active.RecordValue(v * int64(second+1) * 100)
}
// The second is complete: reuse the oldest slot for it.
slot := ring[second%slots]
slot.Reset()
slot.MergeFrom(active)
active.Reset()
}
// Aggregate the last three seconds (seconds 2, 3 and 4).
window := hdrhistogram.New(1, 30000000, 3)
for _, slot := range ring {
slot.MergeInto(window)
}
fmt.Println("count:", window.TotalCount())
fmt.Println("p50:", window.ValueAtQuantile(50))
fmt.Println("max:", window.Max())
}
Output: count: 300 p50: 19215 max: 50015
func (*PackedHistogram) Min ¶ added in v1.4.0
func (p *PackedHistogram) Min() int64
Min mirrors dense: the lowest-equivalent of the first populated bucket.
func (*PackedHistogram) Populated ¶ added in v1.4.0
func (p *PackedHistogram) Populated() int32
Populated returns the number of populated buckets.
func (*PackedHistogram) RecordCorrectedValue ¶ added in v1.4.0
func (p *PackedHistogram) RecordCorrectedValue(v, expectedInterval int64) error
RecordCorrectedValue records v and corrects for stalls in a process that records values at expectedInterval. It also records v-expectedInterval, v-2*expectedInterval, and so on, while those values are at least expectedInterval. A nonpositive interval records only v, matching Histogram.RecordCorrectedValue.
It uses sparse storage. If recording fails, it returns the error immediately; the original observation and any synthetic observations already recorded are retained. In particular, reaching the total-count limit can leave a partial correction.
func (*PackedHistogram) RecordValue ¶ added in v1.4.0
func (p *PackedHistogram) RecordValue(v int64) error
RecordValue records a single occurrence of v.
func (*PackedHistogram) RecordValues ¶ added in v1.4.0
func (p *PackedHistogram) RecordValues(v, n int64) error
RecordValues records n occurrences of v, returning an error if v is out of range, n is negative, or the total would overflow int64.
func (*PackedHistogram) Reset ¶ added in v1.4.0
func (p *PackedHistogram) Reset()
Reset empties the histogram for reuse. Like the dense Histogram.Reset and HdrHistogram_c's hdr_packed_reset, it keeps the allocated backing arrays, and it also keeps the current count width, so a record/merge/reset cycle does not reallocate. The width only grows; to give the memory back (say, after a burst widened a ring slot to 8-byte counts), call Compact.
func (*PackedHistogram) SetEndTimeMs ¶ added in v1.4.0
func (p *PackedHistogram) SetEndTimeMs(endTimeMs int64)
SetEndTimeMs sets the interval end time in milliseconds.
func (*PackedHistogram) SetStartTimeMs ¶ added in v1.4.0
func (p *PackedHistogram) SetStartTimeMs(startTimeMs int64)
SetStartTimeMs sets the interval start time in milliseconds.
func (*PackedHistogram) SetTag ¶ added in v1.4.0
func (p *PackedHistogram) SetTag(tag string)
SetTag sets the optional interval log tag.
func (*PackedHistogram) SignificantFigures ¶ added in v1.4.0
func (p *PackedHistogram) SignificantFigures() int64
SignificantFigures returns the significant figures used to create the histogram.
func (*PackedHistogram) StartTimeMs ¶ added in v1.4.0
func (p *PackedHistogram) StartTimeMs() int64
StartTimeMs returns the interval start time in milliseconds.
func (*PackedHistogram) StdDev ¶ added in v1.4.0
func (p *PackedHistogram) StdDev() float64
StdDev returns the approximate population standard deviation of the recorded values, using the median of each equivalent-value range. It returns zero when empty. Like Mean, it visits only populated buckets without allocating and normalizes overflowing decoded counts using their sum in float64.
func (*PackedHistogram) Tag ¶ added in v1.4.0
func (p *PackedHistogram) Tag() string
Tag returns the optional interval log tag.
func (*PackedHistogram) TotalCount ¶ added in v1.4.0
func (p *PackedHistogram) TotalCount() int64
TotalCount returns the number of recorded values.
func (*PackedHistogram) ValueAtPercentile ¶ added in v1.4.0
func (p *PackedHistogram) ValueAtPercentile(percentile float64) int64
ValueAtPercentile returns the largest value that (100%-percentile) of recorded entries are >= to. Returns 0 for an empty histogram.
func (*PackedHistogram) ValueAtPercentilesSlice ¶ added in v1.4.0
func (p *PackedHistogram) ValueAtPercentilesSlice(percentiles []float64) []int64
ValueAtPercentilesSlice returns ValueAtPercentile for each percentile.
Instead of calling ValueAtPercentile once per percentile (each re-walking the populated buckets from the start), it resolves every percentile to a cumulative-count target, orders those targets ascending, and walks the populated buckets a single time, emitting each value as the running count crosses its target. Results are bit-identical to the per-percentile calls (including the negative->0 clamp and the p==0 lowest-equivalent vs p>0 highest-equivalent distinction) and order-agnostic (it sorts internally and emits into the caller's original slot order).
type Snapshot ¶
type Snapshot struct {
LowestTrackableValue int64
HighestTrackableValue int64
SignificantFigures int64
Counts []int64
// IntegerToDoubleConversionRatio is the V2 integer-to-double conversion
// ratio carried as metadata; 0 means the default of 1.
IntegerToDoubleConversionRatio float64
}
A Snapshot is an exported view of a Histogram, useful for serializing them. A Histogram can be constructed from it by passing it to Import.
func (*Snapshot) Validate ¶ added in v1.4.0
Validate reports whether the snapshot can be imported without losing or misinterpreting data: its geometry must be representable (0-5 significant digits, as on the wire), every count must be non-negative, counts beyond the geometry's range must be zero (Import would drop them), and the counts must sum to at most math.MaxInt64. Validate untrusted snapshots, for example ones decoded from JSON, before passing them to Import.
Like Decode, it accepts every range the constructors accept (including a highest value below twice the lowest), reads a lowest value below 1 as 1, and does not check the conversion ratio, which is metadata.
type WindowedHistogram ¶
type WindowedHistogram struct {
// Current is the window's histogram that receives new values. It is one of
// the window's own histograms, so a later Rotate resets it once it becomes
// the oldest; copy it with Clone to keep its values.
Current *Histogram
// contains filtered or unexported fields
}
A WindowedHistogram combines histograms to provide windowed statistics.
It provides no internal synchronization: callers must synchronize recording into Current, Rotate and Merge against each other and against reads of the histograms they return.
func NewWindowed ¶
func NewWindowed(n int, minValue, maxValue int64, sigfigs int) *WindowedHistogram
NewWindowed creates a new WindowedHistogram with N underlying histograms with the given parameters. n must be at least 1.
func (*WindowedHistogram) Merge ¶
func (w *WindowedHistogram) Merge() *Histogram
Merge returns a histogram which includes the recorded values from all the sections of the window. Counts that would push its total past math.MaxInt64 are dropped; unlike Histogram.Merge, it does not report how many. The returned histogram is reused: every call to Merge resets and refills the same histogram, so a result is only valid until the next Merge. Use Clone to keep an independent snapshot:
saved := w.Merge().Clone()
func (*WindowedHistogram) Rotate ¶
func (w *WindowedHistogram) Rotate()
Rotate resets the oldest histogram and rotates it to be used as the current histogram.