README
¶
Sampler
This package does two things:
- Produces strats. That is, it produces a table of row counts for each stratum of an input source. The strata are defined by the user.
- Creates an output table from the input source that is balanced along the desired strata.
Sampling Procedure
The goal is to generate a sample with an equal number of rows for each stratum. A sample rate for each stratum is determined to achieve this. If each stratum has sufficient rows, then we're done. To the extent some strata have insufficient rows, then the total sample will be short. Also, the sample won't be exactly balanced. To achieve balance one would have to do one of these:
- Target a sample per strata that is equal to the size of the smallest stratum;
- Resample strata with insufficient rows
Practically, the first is not an option as often there will be a stratum with very few rows. The second is tantamount to up-weighting the small strata. In that case, these observations may become influential observations.
Instead of these, this package adopts the philosophy that an approximate balance goes a long way to reducing the leverage of huge strata which is typical in data. The sampling algorithm used by sampler is:
- At the start
- desired sample = total sample desired / # of strata
- stratum sample rate = min(desired sample / stratum size, sample rate cap)
- stratum captured sample = stratum sample rate * stratum size
- stratum free observations = stratum size - stratum captured sample
- Update
- total sample desired = total sample desired - total captured
- if this number is "small" stop.
- stratum desired sample = total sample desired / # of strata with free observations
- stratum sample rate = min(stratum desired sample + stratum previously captured sample / stratum size, sample rate cap)
- stratum captured sample = stratum sample rate * stratum size
- stratum free observations = stratum size - stratum captured sample
Step 2 is repeated (iterations capped at 5) until the target sample size is achieved (within tolerance) or no strata have free observations.
Documentation
¶
Overview ¶
Package sampler produces strats and stratified samples. The package works with ClickHouse tables.
This package does two things:
- Produces strats. That is, it produces a table of row counts for each stratum of an input source. The strata are defined by the user.
- Creates an output table from the input source that is balanced along the desired strata.
Sampling Procedure ¶
The goal is to generate a sample with an equal number of rows for each stratum. A sample rate for each stratum is determined to achieve this. If each stratum has sufficient rows, then we're done. To the extent some strata have insufficient rows, then the total sample will be short. Also, the sample won't be exactly balanced. To achieve balance one would have to do one of these:
- Target a sample per strata that is equal to the size of the smallest stratum;
- Resample strata with insufficient rows
Practically, the first is not an option as often there will be a stratum with very few rows. The second is tantamount to up-weighting the small strata. In that case, these observations may become influential observations.
Instead of these, this package adopts the philosophy that an approximate balance goes a long way to reducing the leverage of huge strata which is typical in data. The sampling algorithm used by sampler is:
1. At the start
- desired sample = total sample desired / # of strata
- stratum sample rate = min(desired sample / stratum size, sample rate cap)
- stratum captured sample = stratum sample rate * stratum size
- stratum free observations = stratum size - stratum captured sample
2. Update
- total sample desired = total sample desired - total captured
- if this number is "small" stop.
- stratum desired sample = total sample desired / # of strata with free observations
- stratum sample rate = min(stratum desired sample + stratum previously captured sample / stratum size, sample rate cap)
- stratum captured sample = stratum sample rate * stratum size
- stratum free observations = stratum size - stratum captured sample
Step 2 is repeated (iterations capped at 5) until the target sample size is achieved (within tolerance) or no strata have free observations.
Index ¶
- func Max(a, b int) int
- func Min(a, b int) int
- type Generator
- func (gn *Generator) CalcRates(fields ...string) error
- func (gn *Generator) MakeQuery() string
- func (gn *Generator) MakeTable(timeOut int64) error
- func (gn *Generator) Marginals() ([]*Strat, string, error)
- func (gn *Generator) MinCount(mc int) uint64
- func (gn *Generator) SampleCap(sCap float64) float64
- func (gn *Generator) SampleRates() []float64
- func (gn *Generator) SampleStrats() *Strat
- func (gn *Generator) Save() error
- func (gn *Generator) Strats() *Strat
- func (gn *Generator) String() string
- type Strat
- func (strt *Strat) Fields() []string
- func (strt *Strat) Make(fields ...string) error
- func (strt *Strat) MinCount(mc int) int
- func (strt *Strat) N() uint64
- func (strt *Strat) Plot(outDir, outFile string, imageTypes []utilities.PlotlyImage, show bool) error
- func (strt *Strat) String() string
- func (strt *Strat) Table() (keys [][]any, counts []uint64)
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
Types ¶
type Generator ¶
type Generator struct {
// inputs
Query string // Query to fetch data to sample
// contains filtered or unexported fields
}
Generator is used to produce stratified samples.
func NewGenerator ¶
func NewGenerator(query, sampleTable, stratTable string, targetTotal int, sortByCount bool, conn *chutils.Connect) *Generator
NewGenerator returns a *Generator. Query is the CH Query to fetch the input data. sampleTable is the output table of the sampled input data. stratTable is the output table of strats & sampling rates of the input data. targetTotal is the target size of sampleTable sortByCount sorts strats by descending count, if true. If false, the table is sorted by the strat fields.
func (*Generator) CalcRates ¶
CalcRates calculates the sampling rate for each strat to achieve a balanced sample with a total size of TargetTotal. fields is the set of fields to stratify on.
func (*Generator) MinCount ¶
MinCount returns (and optionally sets) the minimum # of obs for a strat to be sampled. The value is not updated if mc < 0.
func (*Generator) SampleCap ¶
SampleCap returns (and optional sets) the maximum sampling rate for strats. The value is not updated if cap <= 0.0 or cap > 1.0
func (*Generator) SampleRates ¶
SampleRates returns the calculated sample rates. The slice is in the same order as Strats.
func (*Generator) SampleStrats ¶
SampleStrats returns the strats of sampleTable.
type Strat ¶
type Strat struct {
Query string // Query to pull data for strats
// contains filtered or unexported fields
}
Strat produces stratifications.
func (*Strat) MinCount ¶
MinCount returns (and optionally sets) the minimum # of obs for a strat to be included
func (*Strat) N ¶
N returns the total number of observations in the strats. This does not include strats dropped if MinCount > 0.
Source Files