statsstore

package module
v1.20.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Aug 5, 2026 License: AGPL-3.0 Imports: 24 Imported by: 0

README

Stats Store

Open in Gitpod

Tests Status Go Report Card PkgGoDev

Stats Store - a visitor stats storage implementation for Go.

License

This project is licensed under the GNU Affero General Public License v3.0 (AGPL-3.0). You can find a copy of the license at https://www.gnu.org/licenses/agpl-3.0.en.html

For commercial use, please use my contact page to obtain a commercial license.

Installation

go get -u github.com/dracory/statsstore

Setup

store, err := NewStore(NewStoreOptions{
	VisitorTableName:     "stats_visitor",
	DB:                 databaseInstance,
	AutomigrateEnabled: true,
})

Geo-IP Enrichment

Visitor records are saved with an empty country field by default. To populate country codes (ISO 3166-1 alpha-2), configure a GeoIPResolver and call VisitorEnhance from a background task on your preferred schedule (e.g. every 5 minutes).

Setup
store, err := NewStore(NewStoreOptions{
	VisitorTableName:   "stats_visitor",
	DB:                 databaseInstance,
	AutomigrateEnabled: true,
	GeoIPResolver:      NewDefaultGeoIPResolver(), // uses ip2c.org
	EnhanceBatchSize:   10,                        // records per call (default: 10)
})
Running Enrichment

Call VisitorEnhance from a cron job, task scheduler, or goroutine ticker:

processed, err := store.VisitorEnhance(context.Background())
// processed = number of records that were successfully enriched

VisitorEnhance will:

  1. Fetch up to EnhanceBatchSize visitor records where country is empty
  2. For each record, parse the user agent to fill in browser, OS, device, and device type (if those fields are empty)
  3. Look up the country via the configured GeoIPResolver
  4. Update the record with the enriched data
  5. Return the count of fully processed records (country + UA)

UA fields are updated even if the geo-IP lookup fails, but the country stays empty so the record gets retried on the next call. This makes VisitorEnhance a complete replacement for any custom post-processing task — it handles both UA parsing and country enrichment.

Default Resolver (ip2c.org)

DefaultGeoIPResolver uses the free ip2c.org service. It includes:

  • In-memory cache with 24h TTL — avoids duplicate lookups for the same IP
  • Localhost/private IP detection — returns "UN" (unknown) without making an HTTP call
  • Configurable timeout (default: 5s) and HTTP client (for testing)
resolver := &DefaultGeoIPResolver{
	Endpoint:   "https://ip2c.org/",     // default
	Timeout:    5 * time.Second,         // default
	CacheTTL:   24 * time.Hour,          // default; set to 0 to disable caching
	HTTPClient: myCustomClient,          // optional; nil uses default
}
Custom Resolver

Implement the GeoIPResolver interface to use any geo-IP provider (MaxMind, ipinfo, etc.):

type GeoIPResolver interface {
	Resolve(ctx context.Context, ip string) (string, error)
}
  • Return an ISO2 country code (e.g. "US", "GB") on success
  • Return "" + error on failure (record stays empty for retry)
  • Return "UN" + nil error for unresolvable IPs (localhost, private ranges, etc.)

Example with a local MaxMind database:

type maxmindResolver struct {
	db *geoip2.Reader
}

func (r *maxmindResolver) Resolve(ctx context.Context, ip string) (string, error) {
	addr := net.ParseIP(ip)
	if addr == nil {
		return statsstore.CountryUnknown, nil
	}
	country, err := r.db.Country(addr)
	if err != nil {
		return "", err
	}
	return country.Country.IsoCode, nil
}

store, _ := NewStore(NewStoreOptions{
	// ...
	GeoIPResolver: &maxmindResolver{db: myGeoIPDB},
})

Screenshots

Dashboard

Dashboard

Visitor Activity

Visitor Activity

Visitor Paths

Visitor Paths

Documentation

Index

Constants

View Source
const (
	COLUMN_ID                   = "id"
	COLUMN_COUNTRY              = "country"
	COLUMN_CREATED_AT           = "created_at"
	COLUMN_SOFT_DELETED_AT      = "soft_deleted_at"
	COLUMN_IP_ADDRESS           = "ip_address"
	COLUMN_PATH                 = "path"
	COLUMN_UPDATED_AT           = "updated_at"
	COLUMN_FINGERPRINT          = "fingerprint"
	COLUMN_USER_AGENT           = "user_agent"
	COLUMN_USER_ACCEPT_LANGUAGE = "user_accept_language"
	COLUMN_USER_ACCEPT_ENCODING = "user_accept_encoding"
	COLUMN_USER_BROWSER         = "user_browser"
	COLUMN_USER_BROWSER_VERSION = "user_browser_version"
	COLUMN_USER_DEVICE          = "user_device"
	COLUMN_USER_DEVICE_TYPE     = "user_device_type"
	COLUMN_USER_OS              = "user_os"
	COLUMN_USER_OS_VERSION      = "user_os_version"
	COLUMN_USER_REFERRER        = "user_referrer"
)
View Source
const (
	COLUMN_KEY           = "key"
	COLUMN_VALUE         = "value"
	SETTING_EXCLUDED_IPS = "excluded_ips"
)

Settings table column names.

View Source
const (
	// GeoIPEndpointDefault is the default IP geolocation endpoint (ip2c.org).
	GeoIPEndpointDefault = "https://ip2c.org/"

	// GeoIPTimeoutDefault is the default timeout for IP geolocation lookups.
	GeoIPTimeoutDefault = 5 * time.Second

	// CountryUnknown is used for IPs that cannot be resolved (e.g. localhost).
	CountryUnknown = "UN"

	// GeoIPCacheTTLDefault is the default TTL for the in-memory IP cache.
	GeoIPCacheTTLDefault = 24 * time.Hour
)
View Source
const DEFAULT_SETTINGS_TABLE = "statsstore_settings"

Default table name for key-value settings.

View Source
const MAX_DATETIME = "9999-12-31 23:59:59"

MAX_DATETIME is a far-future datetime used as the default soft-delete sentinel.

Variables

This section is empty.

Functions

func IsBot added in v1.8.0

func IsBot(userAgent string) bool

IsBot checks whether a user-agent string matches known bot/crawler patterns. Broad patterns (bot, crawler, spider, scraper, slurp) are matched with a word-boundary check to avoid false positives (e.g. "bot" inside a non-bot word). Specific patterns (semrush, curl, googlebot, etc.) are matched as plain case-insensitive substrings.

func IsDataCenterIP added in v1.8.0

func IsDataCenterIP(ip string) bool

IsDataCenterIP checks whether an IP address falls within known data center CIDR ranges (AWS, GCP, Azure, DigitalOcean, Oracle Cloud).

func IsReferrerSpam added in v1.8.0

func IsReferrerSpam(referrer string) bool

IsReferrerSpam checks whether a referrer URL host matches a known spam domain. The referrer can be a full URL or just a domain. Matching is case-insensitive.

Types

type DefaultGeoIPResolver added in v1.11.0

type DefaultGeoIPResolver struct {
	Endpoint   string        // default: GeoIPEndpointDefault
	Timeout    time.Duration // default: GeoIPTimeoutDefault
	HTTPClient *http.Client  // injectable for testing; if nil, a default client is used
	CacheTTL   time.Duration // default: GeoIPCacheTTLDefault; set to 0 to disable caching
	// contains filtered or unexported fields
}

DefaultGeoIPResolver implements GeoIPResolver using the ip2c.org service. It includes an in-memory cache with TTL to avoid duplicate lookups for the same IP within the cache window.

func NewDefaultGeoIPResolver added in v1.11.0

func NewDefaultGeoIPResolver() *DefaultGeoIPResolver

NewDefaultGeoIPResolver creates a DefaultGeoIPResolver with sensible defaults.

func (*DefaultGeoIPResolver) Resolve added in v1.11.0

func (r *DefaultGeoIPResolver) Resolve(ctx context.Context, ip string) (string, error)

Resolve looks up the country code for the given IP via ip2c.org.

type GeoIPResolver added in v1.11.0

type GeoIPResolver interface {
	// Resolve returns an ISO2 country code (e.g. "US", "GB") for the given IP.
	// If the IP cannot be resolved, it returns CountryUnknown and nil error.
	// If the lookup itself fails (network error, bad response), it returns
	// an empty string and the error so the caller can leave the field empty
	// for a retry.
	Resolve(ctx context.Context, ip string) (string, error)
}

GeoIPResolver resolves an IP address to an ISO 3166-1 alpha-2 country code. Implementations may use an external API, a local MaxMind database, or any other source. A nil/empty result with no error means the IP could not be resolved and the country should be left empty for a later retry.

type NewStoreOptions

type NewStoreOptions struct {
	VisitorTableName     string
	SettingsTableName    string
	DB                   *sql.DB
	AutomigrateEnabled   bool
	DebugEnabled         bool
	BotFilterEnabled     bool
	ExcludedPathPrefixes []string
	ExcludedIPs          []string
	GeoIPResolver        GeoIPResolver // optional; enables VisitorEnhance for batch country enrichment
	EnhanceBatchSize     int           // number of records per VisitorEnhance call; default 10
}

NewStoreOptions defines the options for creating a new stats store.

type StoreInterface

type StoreInterface interface {
	// MigrateDown drops the stats store tables
	MigrateDown(ctx context.Context, tx ...*sql.Tx) error

	// MigrateUp creates the stats store tables
	MigrateUp(ctx context.Context, tx ...*sql.Tx) error

	EnableDebug(debug bool)
	GetDB() *sql.DB

	SetBotFilterEnabled(enabled bool)
	IsBotFilterEnabled() bool

	SetExcludedPathPrefixes(prefixes []string)
	GetExcludedPathPrefixes() []string

	SetExcludedIPs(ips []string)
	GetExcludedIPs() []string

	ExcludedIPList(ctx context.Context) ([]string, error)
	ExcludedIPAdd(ctx context.Context, ip string) error
	ExcludedIPRemove(ctx context.Context, ip string) error

	// SettingGet retrieves a setting value by key. Returns empty string and
	// nil error if the key does not exist.
	SettingGet(ctx context.Context, key string) (string, error)
	// SettingSet stores a setting value by key, using upsert semantics.
	SettingSet(ctx context.Context, key, value string) error
	// SettingDelete removes a setting by key. No error if the key is absent.
	SettingDelete(ctx context.Context, key string) error
	// SettingHas reports whether a setting key exists.
	SettingHas(ctx context.Context, key string) (bool, error)
	// SettingList returns all settings as a map of key to value.
	// Returns an empty map if the settings table does not exist or is empty.
	SettingList(ctx context.Context) (map[string]string, error)

	VisitorCount(ctx context.Context, query VisitorQueryInterface) (int64, error)
	VisitorCreate(ctx context.Context, user VisitorInterface) error
	VisitorDelete(ctx context.Context, user VisitorInterface) error
	VisitorDeleteByID(ctx context.Context, id string) error
	VisitorDeleteByIP(ctx context.Context, ip string) (int64, error)
	VisitorFindByID(ctx context.Context, userID string) (VisitorInterface, error)
	VisitorList(ctx context.Context, query VisitorQueryInterface) ([]VisitorInterface, error)
	VisitorRegister(ctx context.Context, r *http.Request) error
	VisitorSoftDelete(ctx context.Context, user VisitorInterface) error
	VisitorSoftDeleteByID(ctx context.Context, id string) error
	VisitorUpdate(ctx context.Context, user VisitorInterface) error

	// VisitorEnhance enriches visitor records that have an empty country field.
	// It parses the user agent (browser, OS, device type) and looks up the
	// country via the configured GeoIPResolver. UA fields are updated even if
	// the geo-IP lookup fails; the country stays empty for retry.
	// Returns the number of records fully processed (country + UA).
	// Call this from a background task/cron on whatever schedule suits your
	// traffic (e.g. every 5 minutes).
	//
	// If no GeoIPResolver was configured, it returns 0 and an error.
	VisitorEnhance(ctx context.Context) (int, error)
}

StoreInterface defines the interface for a stats store.

func NewStore

func NewStore(opts NewStoreOptions) (StoreInterface, error)

NewStore creates a new stats store.

type UserAgentInfo added in v1.16.0

type UserAgentInfo struct {
	Browser        string
	BrowserVersion string
	Os             string
	OsVersion      string
	Device         string
	DeviceType     string
}

UserAgentInfo holds parsed user-agent data.

func ParseUserAgent added in v1.16.0

func ParseUserAgent(ua string) UserAgentInfo

ParseUserAgent extracts browser, OS, device, and device-type from a user-agent string using the uasurfer library. Unknown values are left as empty strings.

type VisitorInterface

type VisitorInterface interface {
	// Methods
	FingerprintCalculate() string
	IsSoftDeleted() bool

	GetID() string
	SetID(id string) VisitorInterface

	GetPath() string
	SetPath(path string) VisitorInterface

	GetCountry() string
	SetCountry(country string) VisitorInterface

	GetCreatedAt() string
	GetCreatedAtCarbon() *carbon.Carbon
	SetCreatedAt(createdAt string) VisitorInterface

	GetSoftDeletedAt() string
	GetSoftDeletedAtCarbon() *carbon.Carbon
	SetSoftDeletedAt(deletedAt string) VisitorInterface

	GetFingerprint() string
	SetFingerprint(fingerprint string) VisitorInterface

	GetIpAddress() string
	SetIpAddress(ipAddress string) VisitorInterface

	GetUpdatedAt() string
	GetUpdatedAtCarbon() *carbon.Carbon
	SetUpdatedAt(updatedAt string) VisitorInterface

	GetUserAcceptLanguage() string
	SetUserAcceptLanguage(userAcceptLanguage string) VisitorInterface

	GetUserAcceptEncoding() string
	SetUserAcceptEncoding(userAcceptEncoding string) VisitorInterface

	GetUserAgent() string
	SetUserAgent(userAgent string) VisitorInterface

	GetUserBrowser() string
	SetUserBrowser(userBrowser string) VisitorInterface

	GetUserBrowserVersion() string
	SetUserBrowserVersion(userBrowserVersion string) VisitorInterface

	GetUserDevice() string
	SetUserDevice(userDevice string) VisitorInterface

	GetUserDeviceType() string
	SetUserDeviceType(userDeviceType string) VisitorInterface

	GetUserOs() string
	SetUserOs(userOs string) VisitorInterface

	GetUserOsVersion() string
	SetUserOsVersion(userOsVersion string) VisitorInterface

	GetUserReferrer() string
	SetUserReferrer(userReferrer string) VisitorInterface
}

VisitorInterface defines the interface for a visitor record.

func NewVisitor

func NewVisitor() VisitorInterface

NewVisitor creates a new visitor.

func NewVisitorFromExistingData

func NewVisitorFromExistingData(data map[string]string) VisitorInterface

NewVisitorFromExistingData creates a new visitor from a raw column map.

type VisitorQueryInterface added in v1.1.0

type VisitorQueryInterface interface {
	Validate() error

	HasCountry() bool
	Country() string
	SetCountry(country string) VisitorQueryInterface

	HasCreatedAtGte() bool
	CreatedAtGte() string
	SetCreatedAtGte(createdAtGte string) VisitorQueryInterface

	HasCreatedAtLte() bool
	CreatedAtLte() string
	SetCreatedAtLte(createdAtLte string) VisitorQueryInterface

	HasDeviceType() bool
	DeviceType() string
	SetDeviceType(deviceType string) VisitorQueryInterface

	HasDistinct() bool
	Distinct() string
	SetDistinct(distinct string) VisitorQueryInterface

	HasID() bool
	ID() string
	SetID(id string) VisitorQueryInterface

	HasIDIn() bool
	IDIn() []string
	SetIDIn(idIn []string) VisitorQueryInterface

	HasLimit() bool
	Limit() int
	SetLimit(limit int) VisitorQueryInterface

	HasOffset() bool
	Offset() int
	SetOffset(offset int) VisitorQueryInterface

	HasOrderBy() bool
	OrderBy() string
	SetOrderBy(orderBy string) VisitorQueryInterface

	HasPathContains() bool
	PathContains() string
	SetPathContains(pathContains string) VisitorQueryInterface

	HasPathExact() bool
	PathExact() string
	SetPathExact(pathExact string) VisitorQueryInterface

	HasSortOrder() bool
	SortOrder() string
	SetSortOrder(sortOrder string) VisitorQueryInterface

	HasSoftDeletedIncluded() bool
	SoftDeletedIncluded() bool
	SetSoftDeletedIncluded(withSoftDeleted bool) VisitorQueryInterface
}

VisitorQueryInterface defines the interface for visitor query operations.

func NewVisitorQuery added in v1.1.0

func NewVisitorQuery() VisitorQueryInterface

NewVisitorQuery creates a new visitor query.

func VisitorQuery added in v1.1.0

func VisitorQuery() VisitorQueryInterface

VisitorQuery is a shortcut for NewVisitorQuery.

Directories

Path Synopsis
examples
admin-demo command

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL