Documentation
¶
Index ¶
- Constants
- func FloatEntry(prec int) *entry
- func Marshal(data interface{}, id string) ([]byte, error)
- type Cachable
- type CacheScope
- type Config
- type Dictionary
- type Host
- type Message
- func (m *Message) BatchSize() int
- func (m *Message) Bytes() []byte
- func (m *Message) CacheHit(index int) bool
- func (m *Message) CacheKey() string
- func (m *Message) CacheKeyAt(index int) string
- func (m *Message) FlagCacheHit(index int)
- func (m *Message) FloatKey(key string, value float32)
- func (m *Message) FloatsKey(key string, values []float32)
- func (m *Message) IntKey(key string, value int)
- func (m *Message) IntsKey(key string, values []int)
- func (m *Message) Release()
- func (m *Message) SetBatchSize(batchSize int)
- func (m *Message) Size() int
- func (m *Message) StringKey(key, value string)
- func (m *Message) Strings() []string
- func (m *Message) StringsKey(key string, values []string)
- type Messages
- type Option
- func WithCacheScope(scope CacheScope) Option
- func WithCacheSize(sizeMB int) Option
- func WithClientOptions(clientOptions ...dscli.Option) Option
- func WithConnectionSharing(connections map[string]*dscli.Service) Option
- func WithDataStorer(storer datastore.Storer) Option
- func WithDebug(enable bool) Option
- func WithDictionary(dictionary *Dictionary) Option
- func WithGmetrics(gmetrics *gmetric.Service) Option
- func WithHashValidation(enable bool) Option
- func WithLatencyBreaker(latest, rolling, window time.Duration, k int, fraction float64) Option
- func WithRemoteConfig(config *cconfig.Remote) Option
- type Releaser
- type Response
- type Service
Constants ¶
const ( CacheScopeLocal = CacheScope(1) CacheScopeL1 = CacheScope(2) CacheScopeL2 = CacheScope(4) )
Variables ¶
This section is empty.
Functions ¶
func FloatEntry ¶ added in v0.8.0
func FloatEntry(prec int) *entry
FloatEntry creates entry with prec digits after the decimal. A prec of less than or equal to 0 will be ignored.
Types ¶
type Cachable ¶
type Cachable interface {
CacheKey() string
CacheKeyAt(index int) string
BatchSize() int
FlagCacheHit(index int)
CacheHit(index int) bool
}
Cachable returns a cacheable key
type CacheScope ¶
type CacheScope int
func (CacheScope) IsL1 ¶
func (c CacheScope) IsL1() bool
func (CacheScope) IsL2 ¶
func (c CacheScope) IsL2() bool
func (CacheScope) IsLocal ¶
func (c CacheScope) IsLocal() bool
type Config ¶
type Config struct {
Hosts []*Host
Model string
CacheSizeMb int
// CacheScope limits which caches are available to this client.
CacheScope *CacheScope
Datastore *config.Remote
// MaxRetry defines the maximum number of HTTP requests that should be sent
// during a shared/client.(*Service).Run()
MaxRetry int
Debug bool
DictHashValidation bool
// LatencyBreaker fields are passed through to circut.LatencyBreaker
// at Service init time. When both LatencyBreakerLatestThreshold and
// LatencyBreakerRollingThreshold are zero, the LatencyBreaker is not
// constructed and the host's IsUp() reflects only the connection
// breaker -- backward-compatible default.
//
// LatestThreshold is the per-attempt latency above which a single
// observation is enough to trip into the shedding state. The caller
// is expected to size this near (or just below) its own request
// timeout so the breaker fires before requests would have failed.
LatencyBreakerLatestThreshold time.Duration
// RollingThreshold is the rolling-average latency above which the
// breaker trips. Detects sustained slow-creep that no single
// observation crosses LatestThreshold for.
LatencyBreakerRollingThreshold time.Duration
// RollingWindow is the duration over which the rolling average is
// computed. Default 1s if zero.
LatencyBreakerRollingWindow time.Duration
// KConsecutive is the number of consecutive observations satisfying
// (latest < LatestThreshold AND rolling < RollingThreshold) needed
// to transition from ON back to OFF. Higher = more conservative
// recovery, prevents flap on outliers. Default 3 if zero.
LatencyBreakerKConsecutive int
// PassThroughFraction is the probability that a request is allowed
// through while the breaker is ON, to drive recovery sensing.
// Default 0.01 (1%). Set higher for low-QPS models that need more
// observations to recover, or 0 to fully shed without recovery.
LatencyBreakerPassThroughFraction float64
}
Config represents a client config
type Dictionary ¶
type Dictionary struct {
// contains filtered or unexported fields
}
Dictionary helps identify any out-of-vocabulary input values for reducing the cache space - this enables us to leverage any dimensionality reduction within the model to optimize wall-clock performance. This is primarily useful for categorical inputs as well as any continous inputs with an acceptable quantization.
func NewDictionary ¶
func NewDictionary(dict *common.Dictionary, inputs []*shared.Field) *Dictionary
NewDictionary creates new Dictionary
func (*Dictionary) Fields ¶ added in v0.2.2
func (d *Dictionary) Fields() map[string]*shared.Field
TODO refactor, this has a singular use case
func (*Dictionary) KeysLen ¶
func (d *Dictionary) KeysLen() int
type Host ¶
type Host struct {
// used as both a check to see if a host is down
// and to pause requests to prevent downstream overload
RequestTimeout time.Duration
*circut.Breaker
// LatencyBreaker is an optional latency-driven shed mechanism that
// runs in parallel to the connection-failure-based Breaker. When
// configured (non-nil), getHost() requires both Breaker.IsUp() and
// LatencyBreaker.IsUp() to return true before letting a request
// through. nil = disabled (acts as permanently up).
LatencyBreaker *circut.LatencyBreaker
// contains filtered or unexported fields
}
Host represents endpoint host
func (*Host) IsSecurePort ¶
IsSecurePort() returns true if secure port
type Message ¶
type Message struct {
// contains filtered or unexported fields
}
Message represents the client-side perspective of the ML prediction. The JSON payload is built along the method calls; be sure to call (*Message).start() to set up the opening "{". TODO document how cache management is built into this type. There are 2 "modes" for building the message: single and batch modes. For single mode, the JSON object contents are written to Message.buf per method call. Single mode functions include:
(*Message).StringKey(string, string) (*Message).IntKey(string, int) (*Message).FloatKey(string, float32)
Batch mode is initiated by called (*Message).SetBatchSize() to a value greater than 0. For batch mode, the JSON payload is generated when (*Message).end() is called. Batch mode functions include (the type name is plural):
(*Message).StringsKey(string, []string) (*Message).IntsKey(string, []int) (*Message).FloatsKey(string, []float32)
There is no strict struct for request payload since some of the keys of the request are dynamically generated based on the model inputs. The resulting JSON will have property keys that are set based on the model, and two optional keys, "batch_size" and "cache_key". Depending on if single or batch mode, the property values will be scalars or arrays. See service.Request for server-side perspective. TODO separate out single and batch sized request to their respective calls endpoints; the abstracted polymorphism currently is more painful than convenient.
func (*Message) CacheKeyAt ¶
CacheKeyAt returns cache key for supplied index
func (*Message) FlagCacheHit ¶
func (*Message) Release ¶
func (m *Message) Release()
Release releases message to the grpcPool TODO this should not be public, Service should be 100% responsible for reuse OR TODO caller should be 100% responsible for reuse
func (*Message) SetBatchSize ¶ added in v0.1.3
func (*Message) StringsKey ¶
StringsKey sets key/values pair
type Messages ¶
type Messages interface {
Borrow() *Message
}
Messages represent a message
func NewMessages ¶
func NewMessages(newDict func() *Dictionary) Messages
NewMessages creates a new message grpcPool
type Option ¶
type Option interface {
// Apply applies settings
Apply(c *Service)
}
Option is a pattern to apply a client option.
func WithCacheScope ¶
func WithCacheScope(scope CacheScope) Option
WithCacheScope creates cache scope option
func WithCacheSize ¶
WithCacheSize overrides the cache size provided by the server.
func WithClientOptions ¶ added in v0.16.0
WithClientOptions adds shared/datastore/client.Option options.
func WithConnectionSharing ¶ added in v0.15.0
WithConnectionSharing enables Aerospike connection sharing. Note that connections are shared via ID, not by Hostnames.
func WithDataStorer ¶
WithDataStorer will provide a coded instane of datastore instead of determining it based off configuration values.
func WithDictionary ¶
func WithDictionary(dictionary *Dictionary) Option
WithDictionary overwrites the initial dictionary.
func WithGmetrics ¶
WithGmetrics binds the *gmetric.Service to the client.
func WithHashValidation ¶
WithHashValidation overrides DictHashValidation.
func WithLatencyBreaker ¶ added in v0.20.0
WithLatencyBreaker enables the latency-aware breaker on each host constructed for this Service. Pass-through fraction defaults to 0.01 when fraction <= 0; rolling window defaults to 1s; KConsecutive defaults to 3.
Setting both latest and rolling to zero leaves the breaker disabled (backward compatible). Both thresholds are taken as raw durations; the caller is responsible for sizing them appropriately for the model's traffic profile and the caller's request timeout.
func WithRemoteConfig ¶
WithRemoteConfig will provide a hard-coded configuration instead of sending a request to the mly server to fetch the configuration.
type Response ¶
type Response struct {
Status string `json:"status"`
Error string `json:"error,omitempty"`
ServiceTime time.Duration `json:"serviceTime"`
DictHash int `json:"dictHash"`
Data interface{} `json:"data"`
}
Response represents a response
type Service ¶
type Service struct {
Config
// Lock for dictionary.
sync.RWMutex
ErrorHistory tracker.Tracker
// contains filtered or unexported fields
}
Service represent mly client
func (*Service) NewMessage ¶
NewMessage returns a new message