fus

package module
v0.2.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Sep 9, 2026 License: Apache-2.0 Imports: 25 Imported by: 0

README

fus-reporting-api-go

[!WARNING] This library is maintained for JetBrains-owned products and services only. No public support, compatibility guarantees, or semantic versioning commitment is provided. APIs and behavior may change without notice.

Go client for JetBrains Feature Usage Statistics (FUS). Wire-compatible with the JVM implementation.

Installation

go get github.com/JetBrains/fus-reporting-api-go

Quick start

This example uses the low-level runtime metadata API. For new integrations and registration schema generation, use the event declarations below.

import fus "github.com/JetBrains/fus-reporting-api-go"

// 1. Define your scheme — used at runtime for validation and marshalable
//    to schema.json for AP metadata registration.
scheme := &fus.Scheme{
    Version: "1",
    Rules: &fus.SchemeRules{
        Enums:   map[string][]string{"boolean": {"true", "false"}},
        Regexps: map[string]string{"version": `\d+\.\d+\.\d+`},
    },
    Groups: []fus.GroupSchema{
        {
            ID: "cli.command",
            Rules: &fus.SchemeRules{
                EventID: []string{fus.EnumExpr("executed")},
                EventData: map[string][]string{
                    "command":     {fus.EnumExpr("build", "run", "test")},
                    "duration_ms": {fus.RegexpExpr(`\d+`)},
                    "has_error":   {fus.EnumRefExpr("boolean")},
                },
            },
        },
    },
}
validator, err := fus.NewValidator(scheme)
if err != nil { /* ... */ }

// 2. Create a logger — validator is required, config fetch must succeed.
ctx := context.Background()
logger, err := fus.NewLogger(ctx, fus.RecorderConfig{
    RecorderID:      "FUS",       // assigned by Data Office
    RecorderVersion: 1,
    ProductCode:     "TCC",       // registered in FUS
    BuildVersion:    "0.5.0",
    DataDir:         "~/.config/myproduct/",
}, fus.WithValidator(validator))
if err != nil { /* ... */ }

// 3. Track events — appends to disk, sub-millisecond, never blocks.
logger.Track(
    fus.EventGroup{ID: "cli.command", Version: 1, State: false},
    "executed",
    map[string]any{"command": "build", "duration_ms": 1234, "has_error": false},
)

// 4. Flush on shutdown.
logger.Close(ctx)

See example_test.go for a runnable version that prints the full JSON payload.

How it works

Track()  ──▶  validate  ──▶  escape  ──▶  buffer (JSONL on disk)  ──▶  Flush()  ──▶  FUS
  • Track appends one JSON line to a local buffer file. No network call.
  • Validate rewrites events so only scheme-approved keys and values reach the wire. Unrecognized keys/values are replaced with sentinels; out-of-range events are dropped.
  • Escape normalizes strings for wire safety.
  • Buffer survives process restarts and network failures.
  • Flush sends buffered events over HTTP, re-queuing unsent batches on failure.

The SDK is fail-closed: NewLogger refuses to start without a valid anonymization salt (fetched from the config server or loaded from a local cache) and a configured validator.

Validation rules

The validator is required. It supports the same rule vocabulary as the JVM SensitiveDataValidator:

Rule Helper
Inline enum fus.EnumExpr("a", "b", "c")
Enum reference fus.EnumRefExpr("name")
Inline regexp fus.RegexpExpr(pattern)
Regexp reference fus.RegexpRefExpr("name")
Inline int range fus.RangeExpr(from, to)
Range reference fus.RangeRefExpr("name")

References resolve against the scheme's top-level Rules block.

For tests, fus.NewPermissiveValidator() passes every event unchanged. Do not use in production.

Generating schema.json

Scheme is JSON-marshalable in the AP metadata format. Ship a generator alongside your product:

package main

import (
    "os"
    fus "github.com/JetBrains/fus-reporting-api-go"
    "your.product/internal/telemetry"
)

func main() {
    f, _ := os.Create("schema.json")
    defer f.Close()
    if err := fus.WriteSchemeJSON(telemetry.Scheme, f); err != nil {
        panic(err)
    }
}

The same Scheme value goes to both NewValidator and WriteSchemeJSON, so the serialized runtime metadata matches the local validator. For event registration, generate events-scheme.json from declarations as described below.

Declaring events and generating registration metadata

Use Definition for event-specific fields, like Kotlin's EventLogGroup. Scheme remains the group-level AP/CDN validation format.

auth := fus.GroupDefinition{
    ID: "cli.auth", Version: 1, Description: "Authentication",
    Events: []fus.EventDefinition{
        {
            ID: "login.completed", Description: "Login result",
            Fields: []fus.FieldDefinition{
                {Path: "method", Rules: []string{fus.EnumExpr("token", "guest")}},
            },
        },
        {
            ID: "token.loaded", Description: "Token source",
            Fields: []fus.FieldDefinition{
                {Path: "source", Rules: []string{fus.EnumExpr("env", "keyring")}},
            },
        },
    },
}
definition := &fus.Definition{Groups: []fus.GroupDefinition{auth}}

registration, err := definition.BuildEventsScheme(cfg, buildNumber)
if err != nil { /* handle error */ }
err = fus.WriteEventsSchemeJSON(registration, output)
if err != nil { /* handle error */ }

fallback, err := definition.BuildValidationScheme()
if err != nil { /* handle error */ }
validator, err := fus.NewValidator(fallback)
if err != nil { /* handle error */ }

Registration preserves each event's fields, descriptions, types and rule references. Definition.Rules supplies fallback definitions; referenced rules must also exist in AP metadata. Use inline rules when a shared AP rule is not intended. Anonymized: true emits {regexp#hash} and per-event fallback anonymization. Configure NewAnonymizer from the selected runtime scheme as usual.

Fallback rules are unioned per group and restricted to its declared version. Use CDN metadata when available; fallback generation does not reproduce Data Office history/build ranges. Arrays and nested fields work in registration output but are rejected by fallback generation because the runtime validator does not support them yet. Track and runtime validation are unchanged.

The old fus.BuildEventsScheme(*Scheme, ...) is deprecated: flattened metadata cannot recover field ownership. Migrate using actual event call sites.

For JetBrains teams

Coordinate recorder setup, product registration, and event metadata with the Data Office. See config.go for endpoint configuration and regional options.

Documentation

Index

Constants

View Source
const AnonymizedValueRegex = `[0-9a-f]{64}(#[SC])+`

AnonymizedValueRegex matches the post-anonymization wire format: 64 hex chars + one-or-more #C/#S markers (mirrors the JVM HASH_PATTERN; coordinate any change with the metadata repo).

View Source
const HashRuleRef = "hash"

HashRuleRef is the metadata-global regexp rule name for anonymized (hashed) values. The events scheme references it as {regexp#hash} so the metadata pipeline applies its maintained global rule instead of an inline regexp (which reads as a false change on every diff).

View Source
const MaxDataFields = 10

MaxDataFields is the per-event data-field cap enforced by the FUS analytics UI. Track drops events that exceed this limit.

Variables

This section is empty.

Functions

func Anonymize

func Anonymize(salt []byte, value string) string

Anonymize returns SHA-256(salt||value) as hex with a "#C" (client) suffix. Blank/whitespace values pass through unchanged.

func ComputeBucket

func ComputeBucket(deviceID string) int

ComputeBucket returns a value in [0, 256) derived from the device ID via SHA-256.

func EnumExpr

func EnumExpr(values ...string) string

EnumExpr returns an inline enum rule: "enum:a|b|c".

func EnumRefExpr

func EnumRefExpr(ref string) string

EnumRefExpr returns an enum reference to the scheme-global Rules.Enums[ref].

func GetOrCreateDeviceID

func GetOrCreateDeviceID(dataDir string) (string, error)

GetOrCreateDeviceID returns a persistent device ID from dataDir, generating one in ddMMyy+OSChar+UUID format if absent.

func RangeExpr

func RangeExpr(from, to int) string

RangeExpr returns an inline integer range rule: "range:<from>..<to>".

func RangeRefExpr

func RangeRefExpr(ref string) string

RangeRefExpr returns a range reference to the scheme-global Rules.Ranges[ref].

func RegexpExpr

func RegexpExpr(pattern string) string

RegexpExpr returns an inline regexp rule: "regexp:<pattern>". The pattern is anchored automatically by the validator.

func RegexpRefExpr

func RegexpRefExpr(ref string) string

RegexpRefExpr returns a regexp reference to the scheme-global Rules.Regexps[ref].

func WriteEventsSchemeJSON

func WriteEventsSchemeJSON(es *EventsScheme, w io.Writer) error

WriteEventsSchemeJSON serializes an EventsScheme to w as indented JSON.

func WriteSchemeJSON

func WriteSchemeJSON(s *Scheme, w io.Writer) error

WriteSchemeJSON serializes a scheme to w in the AP metadata repo format (indented, 2-space). A product typically wires this into a `go run`-based code generator whose output (schema.json) is what Data Office registers.

Types

type AnonymizedField

type AnonymizedField struct {
	Event  string   `json:"event"`
	Fields []string `json:"fields"`
}

AnonymizedField declares which event_data fields should be hashed for a specific event ID within its group.

type Anonymizer

type Anonymizer struct {
	// contains filtered or unexported fields
}

Anonymizer hashes event_data fields declared in the scheme's anonymized_fields using Anonymize(salt, value).

func NewAnonymizer

func NewAnonymizer(s *Scheme, salt []byte) *Anonymizer

NewAnonymizer builds an Anonymizer from the scheme's anonymized_fields.

func (*Anonymizer) AnonymizeEvent

func (a *Anonymizer) AnonymizeEvent(e *LogEvent)

AnonymizeEvent hashes declared fields in e.Event.Data in place.

type Client

type Client struct {
	// contains filtered or unexported fields
}

func NewClient

func NewClient(endpoint string, timeout time.Duration, userAgent string) *Client

func (*Client) Send

func (c *Client) Send(ctx context.Context, report Report) error

Send posts a report with retry on transient HTTP errors.

func (*Client) SendBatched

func (c *Client) SendBatched(ctx context.Context, events []LogEvent) (int, error)

SendBatched sends events in batches of up to 500. Returns the number of events successfully sent before the first error.

type Definition

type Definition struct {
	Version string
	Rules   *SchemeRules // shared enum, regexp and range definitions
	Groups  []GroupDefinition
}

Definition declares groups, events and their fields, like Kotlin EventLogGroup.

func (*Definition) BuildEventsScheme

func (d *Definition) BuildEventsScheme(cfg RecorderConfig, buildNumber string) (*EventsScheme, error)

BuildEventsScheme generates registration metadata from each event's own fields.

func (*Definition) BuildValidationScheme

func (d *Definition) BuildValidationScheme() (*Scheme, error)

BuildValidationScheme merges event rules per group for an embedded fallback.

func (*Definition) Validate

func (d *Definition) Validate() error

Validate checks declaration structure before either output is generated.

type DropReason

type DropReason string

DropReason describes why an event was silently dropped by Track.

const (
	DropTooManyFields DropReason = "too_many_data_fields"
	DropValidation    DropReason = "validation_rejected"
	DropBufferError   DropReason = "buffer_write_failed"
)

type ESEventDescriptor

type ESEventDescriptor struct {
	Event       string              `json:"event"`
	Fields      []ESFieldDescriptor `json:"fields"`
	Description string              `json:"description,omitempty"`
}

type ESFieldDescriptor

type ESFieldDescriptor struct {
	Path               string   `json:"path"`
	Value              []string `json:"value"`
	DataType           string   `json:"dataType"`
	ShouldBeAnonymized bool     `json:"shouldBeAnonymized"`
	Description        string   `json:"description,omitempty"`
}

type ESGroupDescriptor

type ESGroupDescriptor struct {
	ID          string              `json:"id"`
	Type        GroupType           `json:"type"`
	Version     int                 `json:"version"`
	Schema      []ESEventDescriptor `json:"schema"`
	Description string              `json:"description,omitempty"`
	Recorder    string              `json:"recorder,omitempty"`
}

type ESRecorderDesc

type ESRecorderDesc struct {
	RecorderID      string                `json:"recorderId"`
	RecorderVersion int                   `json:"recorderVersion"`
	IDs             []ESRecorderFieldDesc `json:"ids"`
	ClientData      []ESRecorderFieldDesc `json:"clientData"`
	SystemData      []ESRecorderFieldDesc `json:"systemData"`
}

type ESRecorderFieldDesc

type ESRecorderFieldDesc struct {
	Path     string   `json:"path"`
	Required bool     `json:"required"`
	Values   []string `json:"values,omitempty"`
}

type EventAction

type EventAction struct {
	ID    string         `json:"id"`
	Data  map[string]any `json:"data,omitempty"`
	Count int            `json:"count"`
}

type EventDefinition

type EventDefinition struct {
	ID          string
	Description string
	Fields      []FieldDefinition
}

type EventGroup

type EventGroup struct {
	ID      string `json:"id"`
	Version int    `json:"version"`
	State   bool   `json:"state"`
}

func Group

func Group(id string, version int) EventGroup

Group returns a counter EventGroup (State=false).

func StateGroup

func StateGroup(id string, version int) EventGroup

StateGroup returns a state EventGroup (State=true).

type EventsScheme

type EventsScheme struct {
	BuildNumber    string              `json:"buildNumber,omitempty"`
	Scheme         []ESGroupDescriptor `json:"scheme"`
	CommitHash     string              `json:"commitHash,omitempty"`
	RecorderScheme []ESRecorderDesc    `json:"recorderScheme,omitempty"`
}

EventsScheme is the product schema JSON that the FUS metadata team uses to generate metadata entries. Attach this to YT issues in the FUS project when requesting metadata changes.

func BuildEventsScheme deprecated

func BuildEventsScheme(s *Scheme, cfg RecorderConfig, buildNumber string) (*EventsScheme, error)

Deprecated: use Definition.BuildEventsScheme; Scheme cannot preserve event-specific fields.

type FUSConfig

type FUSConfig struct {
	SendEndpoint     string `json:"send_endpoint"`
	MetadataEndpoint string `json:"metadata_endpoint,omitempty"`
	Salt             string `json:"salt"`
	SaltRevision     int    `json:"salt_revision"`
	FetchedAt        int64  `json:"fetched_at"`
}

func FetchTestConfig

func FetchTestConfig(recorderID, productCode string) (*FUSConfig, error)

func LoadOrFetchConfig

func LoadOrFetchConfig(recorderID, productCode, productVersion, dataDir string, region RegionCode) (*FUSConfig, error)

LoadOrFetchConfig returns the public FUS config (send endpoint, options); salt may be empty — supply it via RecorderConfig.AnonymizationSalt.

func (*FUSConfig) SchemeURL

func (c *FUSConfig) SchemeURL(productCode string) string

SchemeURL returns the full metadata URL for the given product code, mirroring JVM ConfigurationVersion.provideMetadataProductUrl.

type FieldDefinition

type FieldDefinition struct {
	Path        string
	Rules       []string
	DataType    string // PRIMITIVE (default) or ARRAY
	Anonymized  bool
	Description string
}

FieldDefinition describes a leaf; dotted paths represent nested objects.

type GroupDefinition

type GroupDefinition struct {
	ID          string
	Version     int
	Type        GroupType
	Description string
	Events      []EventDefinition
}

type GroupSchema

type GroupSchema struct {
	ID               string            `json:"id"`
	Type             GroupType         `json:"type,omitempty"`
	Builds           []SchemeRange     `json:"builds,omitempty"`
	Versions         []SchemeRange     `json:"versions,omitempty"`
	Rules            *SchemeRules      `json:"rules,omitempty"`
	AnonymizedFields []AnonymizedField `json:"anonymized_fields,omitempty"`
}

GroupSchema is one EventGroupRemoteDescriptor: the per-group rules plus the build/version ranges that gate whether an event for this group is accepted.

type GroupType

type GroupType string

GroupType distinguishes state collectors (per-device snapshots) from counter collectors (occurrence counts).

const (
	GroupTypeCounter GroupType = "counter"
	GroupTypeState   GroupType = "state"
)

type IntRange

type IntRange struct {
	From int `json:"from"`
	To   int `json:"to"`
}

IntRange is an inclusive [from, to] integer range used by range rules.

type LogEvent

type LogEvent struct {
	Recorder   Recorder          `json:"recorder"`
	Product    string            `json:"product"`
	IDs        map[string]string `json:"ids"`
	Internal   bool              `json:"internal"`
	Time       int64             `json:"time"`
	Build      string            `json:"build"`
	Session    string            `json:"session"`
	Group      EventGroup        `json:"group"`
	Bucket     int               `json:"bucket"`
	Event      EventAction       `json:"event"`
	SystemData map[string]any    `json:"system_data,omitempty"`
	ClientData map[string]any    `json:"client_data,omitempty"`
}

LogEvent is a single LION v4 wire-format event.

type Logger

type Logger struct {
	// contains filtered or unexported fields
}

Logger buffers FUS events to disk and sends them on Flush.

func NewLogger

func NewLogger(ctx context.Context, cfg RecorderConfig, opts ...LoggerOption) (*Logger, error)

func (*Logger) Close

func (l *Logger) Close(ctx context.Context) error

Close flushes all buffered events. Pass a context with a deadline to bound the flush duration; use context.Background() for an unbounded flush.

func (*Logger) Flush

func (l *Logger) Flush(ctx context.Context) error

Flush sends all buffered events, merging consecutive duplicates. The mutex is held only during buffer I/O, not during the HTTP send, so Track calls are not blocked by network latency.

func (*Logger) Track

func (l *Logger) Track(group EventGroup, eventID string, data map[string]any)

Track buffers an event to disk. Failures are silent unless an OnDrop callback is registered. Events with more than MaxDataFields entries are dropped.

Pipeline: raw event -> validator -> anonymizer -> escaper -> disk buffer.

type LoggerOption

type LoggerOption func(*Logger)

func WithAnonymizer

func WithAnonymizer(a *Anonymizer) LoggerOption

WithAnonymizer installs the field anonymizer. When set, event_data fields declared in the scheme's anonymized_fields are hashed before buffering.

func WithClient

func WithClient(c *Client) LoggerOption

WithClient overrides the default HTTP client.

func WithFUSConfig

func WithFUSConfig(cfg *FUSConfig) LoggerOption

WithFUSConfig overrides automatic config fetching.

func WithOnDrop

func WithOnDrop(fn OnDropFunc) LoggerOption

WithOnDrop registers a callback invoked when Track silently drops an event.

func WithValidator

func WithValidator(v *Validator) LoggerOption

WithValidator installs the client-side scheme validator. Required by NewLogger — every event is rewritten so that only scheme-approved keys and values reach the wire, and events whose group is registered but out of build/version range are dropped.

type OnDropFunc

type OnDropFunc func(group, event string, reason DropReason)

OnDropFunc is called when Track silently drops an event.

type Recorder

type Recorder struct {
	ID      string `json:"id"`
	Version int    `json:"version"`
}

type RecorderConfig

type RecorderConfig struct {
	// FUS recorder code (e.g., "TC").
	RecorderID string

	// Recorder protocol version.
	RecorderVersion int

	// Product identifier (e.g., "TCC" for TeamCity CLI).
	ProductCode string

	// Product version string (e.g., "0.5.0").
	BuildVersion string

	// Whether this is a JetBrains internal user.
	Internal bool

	// Directory for persisting device ID, buffer, and config cache.
	DataDir string

	// Overrides automatic device ID generation. If empty, one is generated.
	DeviceID string

	// CDN region for config endpoints. Defaults to RegionAll.
	Region RegionCode

	// AnonymizationSalt is the product-supplied salt used for SHA-256 device/session anonymization. Mirrors JVM FusClientConfig.anonymizationSalt. Takes priority over any salt in the public config.
	AnonymizationSalt string
}

type RegionCode

type RegionCode string
const (
	RegionAll RegionCode = "ALL"
	RegionCN  RegionCode = "CN"
)

type Report

type Report struct {
	Events []LogEvent `json:"events"`
}

type Scheme

type Scheme struct {
	Version string        `json:"version,omitempty"`
	Rules   *SchemeRules  `json:"rules,omitempty"`
	Groups  []GroupSchema `json:"groups"`
}

Scheme mirrors the AP metadata JSON (EventGroupRemoteDescriptors on the JVM side). Each recorder publishes one of these in the Data Office metadata repo.

func FetchScheme

func FetchScheme(url string) (*Scheme, error)

FetchScheme downloads event group metadata from url and decodes it into a Scheme. The response format matches the JVM EventGroupRemoteDescriptors JSON published to the AP metadata CDN.

func LoadOrFetchScheme

func LoadOrFetchScheme(schemeURL, dataDir string) (*Scheme, error)

LoadOrFetchScheme returns the event group metadata for schemeURL, using a disk cache in dataDir with TTL-based freshness. When the remote is unreachable it falls back to the cached copy. This mirrors the JVM MetadataFileUpdater.getNewestFile flow.

schemeURL is the full metadata URL (use FUSConfig.SchemeURL to build it). Returns an error only when both remote and cache are unavailable.

func LoadSchemeFromFile

func LoadSchemeFromFile(path string) (*Scheme, error)

LoadSchemeFromFile parses a scheme JSON file from disk.

type SchemeRange

type SchemeRange struct {
	From string `json:"from,omitempty"`
	To   string `json:"to,omitempty"`
}

SchemeRange carries a build or version range used to gate a group. Empty From or To is treated as open-ended.

type SchemeRules

type SchemeRules struct {
	EventID   []string            `json:"event_id,omitempty"`
	EventData map[string][]string `json:"event_data,omitempty"`
	Enums     map[string][]string `json:"enums,omitempty"`
	Regexps   map[string]string   `json:"regexps,omitempty"`
	Ranges    map[string]IntRange `json:"ranges,omitempty"`
}

SchemeRules carries either group-local or global (scheme-level) rule bundles.

At group scope, EventID and EventData drive per-event validation. At scheme scope, Enums/Regexps/Ranges are global reference tables looked up by {enum#ref}/{regexp#ref}/{range#ref} expressions in group rules.

type Validator

type Validator struct {
	// contains filtered or unexported fields
}

Validator implements the client-side event validation contract from the JVM SensitiveDataValidator. It rewrites an event so that only scheme-approved keys and values reach the wire, replacing unknown / non-matching pieces with validator sentinels. If the event's group is registered but its build/version falls outside the declared ranges, the event is dropped.

Scope ported from JVM:

  • event ID whitelist per group
  • per-field rule set: enum, enum#ref, regexp, regexp#ref, range, range#ref
  • build / version range filter per group
  • sentinel replacement (third.party / validation.unmatched_rule / validation.undefined_rule / validation.incorrect_rule)
  • pass-through for FUS system events (registered, invoked, …)

Not ported (unused by the TeamCity CLI spec):

  • util#, expression, required:, default_value: rules
  • dictionary rules beyond plain enum
  • recursive validation of nested Map/List values in event_data
  • system_data / client_data / ids validation pipelines

func NewValidator

func NewValidator(s *Scheme) (*Validator, error)

NewValidator compiles a Scheme into a ready-to-use validator. Rule expressions are parsed eagerly; a broken expression turns into an incorrectRuleStub so events still produce deterministic sentinels instead of crashing later.

func (*Validator) Validate

func (v *Validator) Validate(e LogEvent) (LogEvent, bool)

Validate mirrors SensitiveDataValidator.validateEvent for LION v4. Returns (validated, false) when the event may be kept (possibly with sentinel replacement) and (zero, true) when the event must be dropped entirely — which only happens when the group is registered but its build/version filter rejects the event.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL