OomStore

A Fast Feature Store Powered by Go.

Quickstart · Architecture · Benchmark

Overview

oomstore allows you to:

• Define features with YAML
• Store features in databases of choice (see supported databases)
• Retrieve features for both online and offline use, fast (see benchmark)

Please see our docs for more details.

Architecture

You can interact with oomstore with CLI, Go API or Python API. See Architecture for more details.

Features

Compared to other feature store implementations, oomstore has its edges:

• Fast. Benchmark shows oomstore performs QPS > 50k and latency < 0.3 ms using Redis as the Online Store. For more details, see benchmark.
• Pluggable. We support a wide range of databases already (see below), and there are more to come.
  • Online Store: DynamoDB, Redis, TiKV, Cassandra, TiDB, PostgreSQL, MySQL, SQLite.
  • Offline Store: Snowflake, Redshift, BigQuery, TiDB, PostgreSQL, MySQL, SQLite.
  • Metadata Store: TiDB, PostgreSQL, MySQL, SQLite.
• Simple. In the minimal, oomstore can run aganist a single MySQL/PostgreSQL database. This helps get started quickly, and you can always switch to a different database later without having to rewrite your code.

Quickstart

1. Install oomcli following the guide.

2. oomcli init to initialize oomstore. Make sure there is a ~/.config/oomstore/config.yaml as below.

online-store:
  sqlite:
    db-file: /tmp/oomstore.db

offline-store:
  sqlite:
    db-file: /tmp/oomstore.db

metadata-store:
  sqlite:
    db-file: /tmp/oomstore.db

1. oomcli apply -f config.yaml to register metadata. See config.yaml below.

kind: Entity
name: user
length: 8
description: 'user ID'
groups:
- name: account
  category: batch
  description: 'user account info'
  features:
  - name: state
    value-type: string
  - name: credit_score
    value-type: int64
  - name: account_age_days
    value-type: int64
  - name: has_2fa_installed
    value-type: bool
- name: transaction_stats
  category: batch
  description: 'user transaction statistics'
  features:
  - name: transaction_count_7d
    value-type: int64
  - name: transaction_count_30d
    value-type: int64

1. Import CSV data to Offline Store.

oomcli import \
  --group account \
  --input-file account.csv \
  --description 'sample account data'
oomcli import \
  --group transaction_stats \
  --input-file transaction_stats.csv \
  --description 'sample transaction stat data'

1. Sync data from Offline Store to Online Store.

oomcli sync --revision-id 1
oomcli sync --revision-id 2

1. Fetch features by key.

oomcli get online \
  --entity-key 1006 \
  --feature account.state,account.credit_score,account.account_age_days,account.has_2fa_installed,transaction_stats.transaction_count_7d,transaction_stats.transaction_count_30d

+------+---------------+----------------------+--------------------------+---------------------------+----------------------------------------+-----------------------------------------+
| user | account.state | account.credit_score | account.account_age_days | account.has_2fa_installed | transaction_stats.transaction_count_7d | transaction_stats.transaction_count_30d |
+------+---------------+----------------------+--------------------------+---------------------------+----------------------------------------+-----------------------------------------+
| 1006 | Louisiana     |                  710 |                       32 | false                     |                                      8 |                                      22 |
+------+---------------+----------------------+--------------------------+---------------------------+----------------------------------------+-----------------------------------------+

1. Generate training datasets via point-in-time join.

oomcli join \
	--feature account.state,account.credit_score,account.account_age_days,account.has_2fa_installed,transaction_stats.transaction_count_7d,transaction_stats.transaction_count_30d \
	--input-file label.csv

+------------+------------+---------------+----------------------+--------------------------+---------------------------+----------------------------------------+-----------------------------------------+
| entity_key | unix_milli | account.state | account.credit_score | account.account_age_days | account.has_2fa_installed | transaction_stats.transaction_count_7d | transaction_stats.transaction_count_30d |
+------------+------------+---------------+----------------------+--------------------------+---------------------------+----------------------------------------+-----------------------------------------+
|       1002 | 1950236233 | Hawaii        |                  625 |                      861 | true                      |                                     11 |                                      36 |
|       1003 | 1950411318 | Arkansas      |                  730 |                      958 | false                     |                                      0 |                                      16 |
|       1004 | 1950653614 | Louisiana     |                  610 |                     1570 | false                     |                                     12 |                                      26 |
|       1005 | 1950166137 | South Dakota  |                  635 |                     1953 | false                     |                                      7 |                                      30 |
|       1006 | 1950403162 | Louisiana     |                  710 |                       32 | false                     |                                      8 |                                      22 |
|       1007 | 1950160030 | New Mexico    |                  645 |                       37 | true                      |                                      5 |                                      40 |
|       1008 | 1950274859 | Nevada        |                  735 |                     1627 | false                     |                                     12 |                                      51 |
|       1009 | 1949958846 | Kentucky      |                  650 |                       88 | true                      |                                     11 |                                      23 |
|       1010 | 1949920686 | Delaware      |                  680 |                     1687 | false                     |                                      2 |                                      39 |
+------------+------------+---------------+----------------------+--------------------------+---------------------------+----------------------------------------+-----------------------------------------+

See Quickstart for more complete details.

Roadmap

See Roadmap for more details.

Community

Feel free to join the community for questions and suggestions!