store

package
v0.1.6 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jul 24, 2026 License: GPL-3.0 Imports: 13 Imported by: 0

Documentation

Overview

Package store implements the persistence layer with redis.

Index

Constants

View Source 
const (
	// APIKeyPattern pattern that represents a api key.
	APIKeyPattern = "api-key:%s" //nolint: gosec

	// ExpInfoKeyPattern is a pattern to a key that holds
	// information on experiments.
	ExpInfoKeyPattern = "info:exp:%s"

	// ExpKeyPattern is a pattern to an experiment key
	// that holds index of all runs linked to that exp.
	ExpKeyPattern = "exp:%s"

	// RunKeyPattern pattern of a Run's key
	// Example:
	// run:linear-regression
	RunKeyPattern = "run:%s"

	// MetricKeyPattern pattern of a metric metric:<metric_name>:<run_id>
	// Example
	// metric:mse:linear-regression
	MetricKeyPattern = "metric:%s:%s"

	// ArtifactKeyPattern pattern of a artifact's key
	// Example
	// artifact:logs:linear-regression
	ArtifactKeyPattern = "artifact:%s:%s"

	// ModelRegistryInfoKeyPattern pattern of model registry's data info value.
	ModelRegistryInfoKeyPattern = "info:registry:%s"

	// ModelRegistryKeyPattern pattern of a model registry key
	// Example
	// registry:yolov12 maps to a list of model entries
	ModelRegistryKeyPattern = "registry:%s"

	// ModelRegistriesKey is a Sorted Set index of all known registry names, scored by
	// insertion order (see RegistriesCounterKey). Populated on registry creation
	// regardless of whether it has any model entries yet, since a registry's
	// "registry:<name>" list key (ModelRegistryKeyPattern) only comes into existence
	// once the first model entry is pushed to it.
	ModelRegistriesKey = "index:registries"

	// RegistriesCounterKey is an auto-incrementing counter used to assign strictly
	// increasing scores to new entries in ModelRegistriesKey.
	RegistriesCounterKey = "counter:registries"

	// ModelRegistryTagsKeyPattern pattern a model registry's tags
	// Example
	// tag:registry:yolov12 -> [prod, latest, ...]
	ModelRegistryTagsKeyPattern = "tag:registry:%s"

	// ModelRegistryTagKeyPattern pattern a model registry tag
	// Example
	// tag:registry:yolov12:prod
	ModelRegistryTagKeyPattern = "tag:registry:%s:%s"

	// ModelRegistryBenchmarksIndexPattern (Set holding all linked benchmarks)
	// form: index:registry:<registry-name>:benchs
	ModelRegistryBenchmarksIndexPattern = "index:registry:%s:benchs"

	// BenchmarksKey is a Sorted Set index of all known benchmark ids, scored by
	// insertion order (see BenchmarksCounterKey).
	BenchmarksKey = "index:benchs"

	// BenchmarksCounterKey is an auto-incrementing counter used to assign strictly
	// increasing scores to new entries in BenchmarksKey.
	BenchmarksCounterKey = "counter:benchs"

	// ExpsIndexKey is a Sorted Set index of all known experiment ids, scored by
	// insertion order (see ExpsCounterKey). Populated the first time a run is
	// recorded under a given experiment id.
	ExpsIndexKey = "index:exps"

	// ExpsCounterKey is an auto-incrementing counter used to assign strictly
	// increasing scores to new entries in ExpsIndexKey.
	ExpsCounterKey = "counter:exps"

	// BenchmarkKeyPattern represents the key for a benchmark.
	BenchmarkKeyPattern = "bench:%s"

	// BenchmarkMetricsKeyPattern represents all metrics linked to a benchmark.
	BenchmarkMetricsKeyPattern = "bench:%s:metrics"

	// BenchmarkRegistriesKeyPattern represents all registries linked to a benchmark.
	BenchmarkRegistriesKeyPattern = "bench:%s:registries"

	// BenchmarkRunsKeyPattern index used to pull all benchmark runs
	// It follows this form: index:bench:<bench-id>:runs.
	BenchmarkRunsKeyPattern = "index:bench:%s:runs"

	// BenchmarkRunKeyPattern key pattern used to save a benchmark run
	// It follows this order: bench:<bench-id>:run:<registry-name>:<version>.
	BenchmarkRunKeyPattern = "bench:%s:run:%s:%d"
)

Variables

This section is empty.

Functions

This section is empty.

Types

type RedisStore 

type RedisStore struct {
	Client redis.Client
	Logger zerolog.Logger
}

RedisStore implements the store for mlsolid.

func (*RedisStore) AddBenchmarkMetrics added in v0.1.3 

func (r *RedisStore) AddBenchmarkMetrics(ctx context.Context, benchID string, metrics []types.BenchMetric) error

AddBenchmarkMetrics adds metrics to a benchmark.

func (*RedisStore) AddBenchmarkRegistries added in v0.1.3 

func (r *RedisStore) AddBenchmarkRegistries(ctx context.Context, benchID string, registries []string) error

AddBenchmarkRegistries adds registries to a benchmark.

func (*RedisStore) AddModel 

func (r *RedisStore) AddModel(ctx context.Context, name string, m types.ModelEntry, tags ...string) error

AddModel adds a new model entry to a registry with the specified tags.

func (*RedisStore) Artifact 

func (r *RedisStore) Artifact(ctx context.Context, runID string, id string) (types.SavedArtifact, error)

Artifact pulls a saved artifact from the store.

func (*RedisStore) ArtifactExist 

func (r *RedisStore) ArtifactExist(ctx context.Context, runID string, id string) error

func (*RedisStore) Artifacts 

func (r *RedisStore) Artifacts(ctx context.Context, runID string) (map[string]types.SavedArtifact, error)

func (*RedisStore) ArtifactsExist 

func (r *RedisStore) ArtifactsExist(ctx context.Context, runID string, ids []string) map[string]error

func (*RedisStore) BackfillBenchmarksIndex added in v0.1.6 

func (r *RedisStore) BackfillBenchmarksIndex(ctx context.Context) error

BackfillBenchmarksIndex migrates BenchmarksKey from its legacy Set representation to a Sorted Set, if it hasn't been already. It is idempotent and safe to run on every startup.

func (*RedisStore) BackfillExpsIndex added in v0.1.6 

func (r *RedisStore) BackfillExpsIndex(ctx context.Context) error

BackfillExpsIndex adds every experiment that already exists in the store (identified by its "exp:<id>" key, which held the run-membership Set before ExpsIndexKey existed) to ExpsIndexKey. It is idempotent and safe to run on every startup.

func (*RedisStore) BackfillModelRegistriesIndex added in v0.1.6 

func (r *RedisStore) BackfillModelRegistriesIndex(ctx context.Context) error

BackfillModelRegistriesIndex ensures ModelRegistriesKey reflects every registry that already exists in the store. It first migrates the index itself if it is still the legacy Set representation, then adds any registry that predates the index entirely (identified by its "info:registry:<name>" key, which is written unconditionally on creation). It is idempotent and safe to run on every startup.

func (*RedisStore) Benchmark added in v0.1.3 

func (r *RedisStore) Benchmark(ctx context.Context, benchID string) (*types.Bench, error)

Benchmark pulls a benchmark by its name from the redis store.

func (*RedisStore) BenchmarkExists added in v0.1.3 

func (r *RedisStore) BenchmarkExists(ctx context.Context, benchID string) (bool, error)

BenchmarkExists checks if a benchmark with BenchName exists.

func (*RedisStore) BenchmarkMetrics added in v0.1.3 

func (r *RedisStore) BenchmarkMetrics(ctx context.Context, benchID string) ([]types.BenchMetric, error)

BenchmarkMetrics pulls metrics linked to a benchmark.

func (*RedisStore) BenchmarkNames added in v0.1.6 

func (r *RedisStore) BenchmarkNames(ctx context.Context, ids []string) (map[string]string, error)

BenchmarkNames returns a map of benchmark id to benchmark name for the given ids.

func (*RedisStore) BenchmarkRegistries added in v0.1.3 

func (r *RedisStore) BenchmarkRegistries(ctx context.Context, benchID string) ([]string, error)

BenchmarkRegistries pulls model registries linked to a benchmark.

func (*RedisStore) BenchmarkRuns added in v0.1.3 

func (r *RedisStore) BenchmarkRuns(ctx context.Context, benchID string) ([]*types.BenchRun, error)

BenchmarkRuns returns all benchmark runs recorded.

func (*RedisStore) Benchmarks added in v0.1.3 

func (r *RedisStore) Benchmarks(ctx context.Context) ([]string, error)

Benchmarks returns all known benchmarks.

func (*RedisStore) BenchmarksPage added in v0.1.6 

func (r *RedisStore) BenchmarksPage(ctx context.Context, cursor uint64, count int64) ([]string, uint64, error)

BenchmarksPage returns a single page of benchmark ids starting at cursor. A returned cursor of 0 means there are no more pages.

func (*RedisStore) BenchmarksWithIDs added in v0.1.4 

func (r *RedisStore) BenchmarksWithIDs(ctx context.Context, ids []string) ([]*types.Bench, error)

BenchmarksWithIDs returns all benchmarks with specified ids without checking if they exist in the store.

func (*RedisStore) CreateBenchmark added in v0.1.3 

func (r *RedisStore) CreateBenchmark(ctx context.Context, b types.Bench) (bool, error)

CreateBenchmark creates a new benchmark.

func (*RedisStore) CreateModelRegistry 

func (r *RedisStore) CreateModelRegistry(ctx context.Context, m types.ModelRegistry) error

CreateModelRegistry creates and saved a new model registry to the store.

func (*RedisStore) Exp 

func (r *RedisStore) Exp(ctx context.Context, expID string) (*types.Experiment, error)

Exp returns an experiment by its id.

func (*RedisStore) ExpExists 

func (r *RedisStore) ExpExists(ctx context.Context, expID string) error

ExpExists returns whether experiment exists.

func (*RedisStore) ExpInfo 

func (r *RedisStore) ExpInfo(ctx context.Context, expID string) (types.ExperimentInfo, error)

ExpInfo pulls an experiment's info data.

func (*RedisStore) ExpRunIDs 

func (r *RedisStore) ExpRunIDs(ctx context.Context, expID string) ([]string, error)

ExpRunIDs returns run ids of an experiment.

func (*RedisStore) Exps 

func (r *RedisStore) Exps(ctx context.Context) ([]string, error)

Exps returns all known experiment ids.

func (*RedisStore) ExpsPage added in v0.1.6 

func (r *RedisStore) ExpsPage(ctx context.Context, cursor uint64, count int64) ([]string, uint64, error)

ExpsPage returns a single page of experiment ids starting at cursor. A returned cursor of 0 means there are no more pages.

func (*RedisStore) GenerateKey added in v0.1.3 

func (r *RedisStore) GenerateKey(ctx context.Context, label string) (string, error)

GenerateKey generates a new key and saves it to the store.

func (*RedisStore) IsValidAPIKey added in v0.1.3 

func (r *RedisStore) IsValidAPIKey(ctx context.Context, key string) (bool, error)

IsValidAPIKey returns true if the API key is valid.

func (*RedisStore) KeyLabels added in v0.1.5 

func (r *RedisStore) KeyLabels(ctx context.Context) (map[string]time.Time, error)

KeyLabels returns all api key labels still active.

func (*RedisStore) LastModel 

func (r *RedisStore) LastModel(ctx context.Context, name string) (types.ModelEntry, error)

LastModel returns the last model entry of a registry.

func (*RedisStore) Metrics 

func (r *RedisStore) Metrics(ctx context.Context, runID string) (map[string]types.Metric, error)

Metrics returns all metrics of a run.

func (*RedisStore) ModelByTag 

func (r *RedisStore) ModelByTag(ctx context.Context, name string, tag string) (types.ModelEntry, error)

ModelByTag returns the model entry of a registry by its tag.

func (*RedisStore) ModelByVersion 

func (r *RedisStore) ModelByVersion(ctx context.Context, name string, version int) (types.ModelEntry, error)

ModelByVersion returns the model entry of a registry by its version number.

func (*RedisStore) ModelRegistries 

func (r *RedisStore) ModelRegistries(ctx context.Context) ([]*types.ModelRegistry, error)

ModelRegistries returns a slice of all known registries.

func (*RedisStore) ModelRegistriesID 

func (r *RedisStore) ModelRegistriesID(ctx context.Context) ([]string, error)

ModelRegistriesID returns a slice of all known registry ids.

func (*RedisStore) ModelRegistriesIDPage added in v0.1.6 

func (r *RedisStore) ModelRegistriesIDPage(ctx context.Context, cursor uint64, count int64) ([]string, uint64, error)

ModelRegistriesIDPage returns a single page of registry ids starting at cursor. A returned cursor of 0 means there are no more pages.

func (*RedisStore) ModelRegistry 

func (r *RedisStore) ModelRegistry(ctx context.Context, name string) (*types.ModelRegistry, error)

ModelRegistry pulls a saved model registry from the store.

func (*RedisStore) ModelRegistryExists 

func (r *RedisStore) ModelRegistryExists(ctx context.Context, name string) error

ModelRegistryExists returns non-nil error if registry was not found.

func (*RedisStore) RecordRuns added in v0.1.3 

func (r *RedisStore) RecordRuns(ctx context.Context, benchID string, runs []types.BenchRun) error

RecordRuns records new benchmark runs to the store.

func (*RedisStore) RegistryBenchmarks added in v0.1.3 

func (r *RedisStore) RegistryBenchmarks(ctx context.Context, registry string) ([]string, error)

RegistryBenchmarks returns all benchmarks linked with a registry.

func (*RedisStore) RemActiveBenchRun added in v0.1.6 

func (r *RedisStore) RemActiveBenchRun(ctx context.Context, benchID string) error

RemActiveBenchRun clears the "ActiveBenchRun" field set by SetActiveBenchRun, e.g. once a run has finished and been recorded. It is a no-op, not an error, when no active run is set.

func (*RedisStore) RemBenchmarkMetrics added in v0.1.3 

func (r *RedisStore) RemBenchmarkMetrics(ctx context.Context, benchID string, metrics []string) error

RemBenchmarkMetrics removes metrics from a benchmark.

func (*RedisStore) RemBenchmarkRegistries added in v0.1.3 

func (r *RedisStore) RemBenchmarkRegistries(ctx context.Context, benchID string, registries []string) error

RemBenchmarkRegistries removes model registries from a benchmark.

func (*RedisStore) Run 

func (r *RedisStore) Run(ctx context.Context, id string) (*types.Run, error)

Run returns a run from the store by its id.

func (*RedisStore) RunExists 

func (r *RedisStore) RunExists(ctx context.Context, runID string) (bool, error)

RunExists checks if a run exists in the redis store.

func (*RedisStore) RunMetrics 

func (r *RedisStore) RunMetrics(ctx context.Context, runID string) ([]string, error)

RunMetrics returns the ids of all metrics of a run.

func (*RedisStore) Runs 

func (r *RedisStore) Runs(ctx context.Context, ids []string) ([]*types.Run, error)

Runs returns all runs with the specified ids.

func (*RedisStore) SelectBenchmarkMetrics added in v0.1.3 

func (r *RedisStore) SelectBenchmarkMetrics(ctx context.Context, benchID string,
	metrics []string,
) ([]types.BenchMetric, error)

SelectBenchmarkMetrics searches for metrics linked to a benchmark.

func (*RedisStore) SetActiveBenchRun added in v0.1.6 

func (r *RedisStore) SetActiveBenchRun(ctx context.Context, benchID string, run types.BenchRun) error

SetActiveBenchRun records run as the benchmark run currently in progress for benchID, so callers can tell which run is executing before it finishes and is written via RecordRuns. It is stored in the "ActiveBenchRun" field of the benchmark's hash as a JSON blob and overwrites whatever active run was previously set, so only one run is ever considered active per benchmark. Callers are responsible for clearing it with RemActiveBenchRun once the run completes.

func (*RedisStore) SetArtifact 

func (r *RedisStore) SetArtifact(ctx context.Context, runID string, a types.SavedArtifact) error

func (*RedisStore) SetArtifacts 

func (r *RedisStore) SetArtifacts(ctx context.Context, runID string, as []types.SavedArtifact) error

func (*RedisStore) SetExpInfo 

func (r *RedisStore) SetExpInfo(ctx context.Context, expID string, info types.ExperimentInfo) error

SetExpInfo sets an experiment's info data. returns non-nil error if operation was not successful.

func (*RedisStore) SetMetric 

func (r *RedisStore) SetMetric(ctx context.Context, runID string, m types.Metric) error

SetMetric sets the metric for a run.

func (*RedisStore) SetMetrics 

func (r *RedisStore) SetMetrics(ctx context.Context, runID string, ms map[string]types.Metric) error

SetMetrics sets the metrics of a run by its id.

func (*RedisStore) SetRun 

func (r *RedisStore) SetRun(ctx context.Context, run types.Run) error

SetRun saves a run to the redis store. Saving a run involves: 1 - runKey (run:runId) is where the run data is stored 2 - expKey (exp:expId) key index for all runs of an experiment 3 - metrics (metric:metricName:runId) where each metric of the run is stored.

func (*RedisStore) ToggleBenchmark added in v0.1.3 

func (r *RedisStore) ToggleBenchmark(ctx context.Context, benchID string, paused bool) error

ToggleBenchmark toggles a benchmark state between paused and unpaused.

func (*RedisStore) UpdateBenchmark added in v0.1.3 

func (r *RedisStore) UpdateBenchmark(ctx context.Context, benchID string, update types.UpdateBench) error

UpdateBenchmark updates the settings fields of a benchmark.

func (*RedisStore) UpdateModelRegistry 

func (r *RedisStore) UpdateModelRegistry(ctx context.Context, m *types.ModelRegistry) error

UpdateModelRegistry updates the model registry in the store.

func (*RedisStore) UpdateRegistryBenchmarkGpuPassthrough added in v0.1.5 

func (r *RedisStore) UpdateRegistryBenchmarkGpuPassthrough(ctx context.Context, registry string, pass bool) error

UpdateRegistryBenchmarkGpuPassthrough updates a registry's BenchmarkGpuPassthrough option.

func (*RedisStore) UpdateRegistryDockerImage added in v0.1.3 

func (r *RedisStore) UpdateRegistryDockerImage(ctx context.Context, registry, dockerImage string) error

UpdateRegistryDockerImage updates the benchmark image of a registry.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.