aistore

module
v1.4.1 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Dec 5, 2025 License: MIT

README

AIStore: High-Performance, Scalable Storage for AI Workloads

License Version Go Report Card

AIStore (AIS) is a lightweight distributed storage stack tailored for AI applications. It's an elastic cluster that can grow and shrink at runtime and can be ad-hoc deployed, with or without Kubernetes, anywhere from a single Linux machine to a bare-metal cluster of any size. Built from scratch, AIS provides linear scale-out, consistent performance, and a flexible deployment model.

AIS consistently shows balanced I/O distribution and linear scalability across an arbitrary number of clustered nodes. The system supports fast data access, reliability, and rich customization for data transformation workloads.

Features

  • ✅ Multi-Cloud Access: Seamlessly access and manage content across multiple cloud backends (including AWS S3, GCS, Azure, and OCI), with the additional benefit of fast-tier performance and configurable data redundancy.
  • ✅ Deploy Anywhere: AIS runs on any Linux machine, virtual or physical. Deployment options range from a single Docker container and Google Colab to petascale Kubernetes clusters. There are no built-in limitations on deployment size or functionality.
  • ✅ High Availability: Redundant control and data planes. Self-healing, end-to-end protection, n-way mirroring, and erasure coding. Arbitrary number of lightweight access points.
  • ✅ HTTP-based API: A feature-rich, native API (with user-friendly SDKs for Go and Python), and compliant Amazon S3 API for running unmodified S3 clients.
  • ✅ Monitoring: Comprehensive observability with integrated Prometheus metrics, Grafana dashboards, detailed logs with configurable verbosity, and CLI-based performance tracking for complete cluster visibility and troubleshooting. See AIStore Observability for details.
  • ✅ Chunked Objects: High-performance chunked object representation, with independently retrievable chunks, metadata v2, and checksum-protected manifests. Supports rechunking, parallel reads, and seamless integration with Get-Batch, blob-downloader, and multipart uploads to supported cloud backends.
  • ✅ JWT Authentication and Authorization: Validates request JWTs to provide cluster- and bucket-level access control via static keys or dynamic OIDC issuer JWKS lookup.
  • ✅ Secure Redirects: Configurable cryptographic signing of redirect URLs using HMAC-SHA256 with a versioned cluster key (stored in memory only).
  • ✅ Load-Aware Throttling: Dynamic request throttling based on a five-dimensional load vector (CPU, memory, disk, FDs, goroutines) to protect AIS clusters under stress.
  • ✅ Unified Namespace: Attach AIS clusters together to provide fast, unified access to the entirety of hosted datasets, allowing users to reference shared buckets with cluster-specific identifiers.
  • ✅ Turn-key Cache: In addition to robust data protection features, AIS offers a per-bucket configurable LRU-based cache with eviction thresholds and storage capacity watermarks.
  • ✅ ETL Offload: Execute I/O intensive data transformations close to the data, either inline (on-the-fly as part of each read request) or offline (batch processing, with the destination bucket populated with transformed results).
  • ✅ Get-Batch: Retrieve multiple objects and/or archived files with a single call. Designed for ML/AI pipelines, Get-Batch fetches an entire training batch in one operation, assembling a TAR (or other supported serialization format) that contains all requested items in the exact user-specified order.
  • ✅ Data Consistency: Guaranteed consistency across all gateways, with write-through semantics in presence of remote backends.
  • ✅ Serialization & Sharding: Native support for TAR, TGZ, TAR.LZ4, and ZIP archives for efficient storage and processing of small-file datasets. Features include seamless integration with existing unmodified workflows across all APIs and subsystems.
  • ✅ Kubernetes: For production, AIS runs natively on Kubernetes. The dedicated ais-k8s repository includes the AIS K8s Operator, Ansible playbooks, Helm charts, and deployment guidance.
  • ✅ Batch Jobs: More than 30 cluster-wide batch operations that you can start, monitor, and control otherwise - the list currently includes:
$ ais show job --help

NAME:
    archive        blob-download  cleanup       copy-bucket    copy-objects      delete-objects
    download       dsort          ec-bucket     ec-get         ec-put            ec-resp
    elect-primary  etl-bucket     etl-inline    etl-objects    evict-objects     evict-remote-bucket
    get-batch      list           lru-eviction  mirror         prefetch-objects  promote-files
    put-copies     rebalance      rechunk       rename-bucket  resilver          summary
    warm-up-metadata

The feature set continues to grow and also includes: blob-downloader; lightweight AuthN Service (Beta), to manage users and roles and generate JWTs; runtime management of TLS certificates; full support for adding/removing nodes at runtime; listing, copying, prefetching, and transforming virtual directories; executing presigned S3 requests; adaptive rate limiting; and more.

For the original white paper and design philosophy, please see AIStore Overview, which also includes high-level block diagram, terminology, APIs, CLI, and more. For our 2024 KubeCon presentation, please see AIStore: Enhancing petascale Deep Learning across Cloud backends.

CLI

AIS includes an integrated, scriptable CLI for managing clusters, buckets, and objects, running and monitoring batch jobs, viewing and downloading logs, generating performance reports, and more:

$ ais <TAB-TAB>

advanced         config           get              object           scrub            tls
alias            cp               help             performance      search           wait
archive          create           job              prefetch         show
auth             download         log              put              space-cleanup
blob-download    dsort            ls               remote-cluster   start
bucket           etl              ml               rmb              stop
cluster          evict            mpu              rmo              storage

Developer Tools

AIS runs natively on Kubernetes and features open format - thus, the freedom to copy or move your data from AIS at any time using the familiar Linux tar(1), scp(1), rsync(1) and similar.

For developers and data scientists, there's also:

Quick Start

  1. Read the Getting Started Guide for a 5-minute local install, or
  2. Run a minimal AIS cluster consisting of a single gateway and a single storage node, or
  3. Clone the repo and run make kill cli aisloader deploy followed by ais show cluster

Deployment options

AIS deployment options, as well as intended (development vs. production vs. first-time) usages, are all summarized here.

Since the prerequisites essentially boil down to having Linux with a disk the deployment options range from all-in-one container to a petascale bare-metal cluster of any size, and from a single VM to multiple racks of high-end servers. Practical use cases require, of course, further consideration.

Some of the most popular deployment options include:

Option Use Case
Local playground AIS developers or first-time users, Linux or Mac OS. Run make kill cli aisloader deploy <<< $'N\nM', where N is a number of targets, M is a number of gateways
Minimal production-ready deployment This option utilizes preinstalled docker image and is targeting first-time users or researchers (who could immediately start training their models on smaller datasets)
Docker container Quick testing and evaluation; single-node setup
GCP/GKE automated install Developers, first-time users, AI researchers
Large-scale production deployment Requires Kubernetes; provided via ais-k8s

For performance tuning, see performance and AIS K8s Playbooks.

Existing Datasets

AIS supports multiple ingestion modes:

  • ✅ On Demand: Transparent cloud access during workloads.
  • ✅ PUT: Locally accessible files and directories.
  • ✅ Promote: Import local target directories and/or NFS/SMB shares mounted on AIS targets.
  • ✅ Copy: Full buckets, virtual subdirectories (recursively or non-recursively), lists or ranges (via Bash expansion).
  • ✅ Download: HTTP(S)-accessible datasets and objects.
  • ✅ Prefetch: Remote buckets or selected objects (from remote buckets), including subdirectories, lists, and/or ranges.
  • ✅ Archive: Group and store related small files from an original dataset.

Install from Release Binaries

You can install the CLI and benchmarking tools using:

./scripts/install_from_binaries.sh --help

The script installs aisloader and CLI from the latest or previous GitHub release and enables CLI auto-completions.

PyTorch integration

PyTorch integration is a growing set of datasets (both iterable and map-style), samplers, and dataloaders:

AIStore Badge

Let others know your project is powered by high-performance AI storage:

aistore

[![aistore](https://img.shields.io/badge/powered%20by-AIStore-76B900?style=flat&labelColor=000000)](https://github.com/NVIDIA/aistore)

More Docs & Guides

How to find information

  • See Extended Index
  • Use CLI search command, e.g.: ais search copy
  • Clone the repository and run git grep, e.g.: git grep -n out-of-band -- "*.md"

License

MIT

Author

Alex Aizman (NVIDIA)

Directories

Path Synopsis
3rdparty
golang/mux
ais
Package ais provides AIStore's proxy and target nodes.
 Package ais provides AIStore's proxy and target nodes.
backend
Package backend contains core/backend interface implementations for supported backend providers.
 Package backend contains core/backend interface implementations for supported backend providers.
s3
Package s3 provides Amazon S3 compatibility layer
 Package s3 provides Amazon S3 compatibility layer
api
Package api provides native Go-based API/SDK over HTTP(S).
 Package api provides native Go-based API/SDK over HTTP(S).
apc
Package apc: API control messages and constants
 Package apc: API control messages and constants
authn
Package authn provides AuthN API over HTTP(S)
 Package authn provides AuthN API over HTTP(S)
env
Package env contains environment variables
 Package env contains environment variables
bench
micro/disk/compare command
Package main
 Package main
micro/hrw
Package hrw provides a way to benchmark different HRW variants.
 Package hrw provides a way to benchmark different HRW variants.
micro/map command
Package main
 Package main
micro/nstlvl
Package nstlvl is intended to measure impact (or lack of thereof) of POSIX directory nesting on random read performance.
 Package nstlvl is intended to measure impact (or lack of thereof) of POSIX directory nesting on random read performance.
tools/aisloader
Package aisloader
 Package aisloader
tools/aisloader/namegetter
Package namegetter is a utility to provide random object and archived file names to aisloader
 Package namegetter is a utility to provide random object and archived file names to aisloader
tools/aisloader/stats
Package stats provides various structs for collecting stats
 Package stats provides various structs for collecting stats
tools/aisloader/stats/statsd
Package statsd provides a client to send basic statd metrics (timer, counter and gauge) to listening UDP StatsD server.
 Package statsd provides a client to send basic statd metrics (timer, counter and gauge) to listening UDP StatsD server.
tools/frandread command
Package frandread is a file-reading benchmark that makes a special effort to visit the files randomly and equally.
 Package frandread is a file-reading benchmark that makes a special effort to visit the files randomly and equally.
cmd
aisinit command
Package main contains logic for the aisinit container
 Package main contains logic for the aisinit container
aisloader command
Package main for the `aisloader` executable.
 Package main for the `aisloader` executable.
aisnode command
Package main for the AIS node executable.
 Package main for the AIS node executable.
aisnodeprofile command
Package main for the AIS node executable.
 Package main for the AIS node executable.
authn command
Package authn is authentication server for AIStore.
 Package authn is authentication server for AIStore.
authn/tok
Package tok provides AuthN token (structure and methods) for validation by AIS gateways
 Package tok provides AuthN token (structure and methods) for validation by AIS gateways
xmeta command
Package xmeta provides low-level tools to format or extract into plain text some of the AIS control structures.
 Package xmeta provides low-level tools to format or extract into plain text some of the AIS control structures.
aisfs module
cli module
ishard module
cmn
Package cmn provides common constants, types, and utilities for AIS clients and AIStore.
 Package cmn provides common constants, types, and utilities for AIS clients and AIStore.
archive
Package archive: write, read, copy, append, list primitives across all supported formats
 Package archive: write, read, copy, append, list primitives across all supported formats
atomic
Package atomic provides simple wrappers around numerics to enforce atomic access.
 Package atomic provides simple wrappers around numerics to enforce atomic access.
certloader
Package certloader loads and reloads X.509 certs.
 Package certloader loads and reloads X.509 certs.
cos
Package cos provides common low-level types and utilities for all aistore projects.
 Package cos provides common low-level types and utilities for all aistore projects.
debug
Package memsys provides memory management and slab/SGL allocation with io.Reader and io.Writer interfaces on top of scatter-gather lists of reusable buffers.
 Package memsys provides memory management and slab/SGL allocation with io.Reader and io.Writer interfaces on top of scatter-gather lists of reusable buffers.
feat
Package feat: global runtime-configurable feature flags
 Package feat: global runtime-configurable feature flags
fname
Package fname contains filename constants and common system directories
 Package fname contains filename constants and common system directories
jsp
Package jsp (JSON persistence) provides utilities to store and load arbitrary JSON-encoded structures with optional checksumming and compression.
 Package jsp (JSON persistence) provides utilities to store and load arbitrary JSON-encoded structures with optional checksumming and compression.
k8s
Package k8s: initialization, client, and misc.
 Package k8s: initialization, client, and misc.
kvdb
Package kvdb provides a local key/value database server for AIS.
 Package kvdb provides a local key/value database server for AIS.
load
Package load provides 5-dimensional node-pressure readings and per-dimension grading.
 Package load provides 5-dimensional node-pressure readings and per-dimension grading.
mono
Package mono provides low-level monotonic time
 Package mono provides low-level monotonic time
nlog
Package nlog - aistore logger, provides buffering, timestamping, writing, and flushing/syncing/rotating
 Package nlog - aistore logger, provides buffering, timestamping, writing, and flushing/syncing/rotating
oom
Package oom: serialized goroutine to go ahead and run GC _now_
 Package oom: serialized goroutine to go ahead and run GC _now_
prob
Package prob implements fully features dynamic probabilistic filter.
 Package prob implements fully features dynamic probabilistic filter.
work
Package work provides a bounded worker pool utility for concurrent processing of any kind.
 Package work provides a bounded worker pool utility for concurrent processing of any kind.
xoshiro256
Package xoshiro256 implements the xoshiro256** RNG no-copyright
 Package xoshiro256 implements the xoshiro256** RNG no-copyright
Package core provides core metadata and in-cluster API
 Package core provides core metadata and in-cluster API
meta
Package meta: cluster-level metadata
 Package meta: cluster-level metadata
mock
Package mock provides a variety of mock implementations used for testing.
 Package mock provides a variety of mock implementations used for testing.
Package ec provides erasure coding (EC) based data protection for AIStore.
 Package ec provides erasure coding (EC) based data protection for AIStore.
ext
dload
Package dload implements functionality to download resources into AIS cluster from external source.
 Package dload implements functionality to download resources into AIS cluster from external source.
dsort
Package dsort provides distributed massively parallel resharding for very large datasets.
 Package dsort provides distributed massively parallel resharding for very large datasets.
dsort/shard
Package shard provides Extract(shard), Create(shard), and associated methods across all supported archival formats (see cmn/archive/mime.go)
 Package shard provides Extract(shard), Create(shard), and associated methods across all supported archival formats (see cmn/archive/mime.go)
etl
Package etl provides utilities to initialize and use transformation pods.
 Package etl provides utilities to initialize and use transformation pods.
etl/webserver
Package webserver provides a framework to impelemnt etl transformation webserver in golang.
 Package webserver provides a framework to impelemnt etl transformation webserver in golang.
fs
Package fs provides mountpath and FQN abstractions and methods to resolve/map stored content
 Package fs provides mountpath and FQN abstractions and methods to resolve/map stored content
health
Package health is a basic mountpath health monitor.
 Package health is a basic mountpath health monitor.
lpi
Package lpi: local page iterator
 Package lpi: local page iterator
mpather
Package mpather provides per-mountpath concepts.
 Package mpather provides per-mountpath concepts.
Package hk provides mechanism for registering cleanup functions which are invoked at specified intervals.
 Package hk provides mechanism for registering cleanup functions which are invoked at specified intervals.
Package ios is a collection of interfaces to the local storage subsystem; the package includes OS-dependent implementations for those interfaces.
 Package ios is a collection of interfaces to the local storage subsystem; the package includes OS-dependent implementations for those interfaces.
Package memsys provides memory management and slab/SGL allocation with io.Reader and io.Writer interfaces on top of scatter-gather lists of reusable buffers.
 Package memsys provides memory management and slab/SGL allocation with io.Reader and io.Writer interfaces on top of scatter-gather lists of reusable buffers.
Package mirror provides local mirroring and replica management
 Package mirror provides local mirroring and replica management
Package nl provides interfaces for AIStore notifications
 Package nl provides interfaces for AIStore notifications
python
tests/unit/sdk/hrw command
Package reb provides global cluster-wide rebalance upon adding/removing storage nodes.
 Package reb provides global cluster-wide rebalance upon adding/removing storage nodes.
Package res provides local volume resilvering upon mountpath-attach and similar
 Package res provides local volume resilvering upon mountpath-attach and similar
Package space provides storage cleanup and eviction functionality (the latter based on the least recently used cache replacement).
 Package space provides storage cleanup and eviction functionality (the latter based on the least recently used cache replacement).
Package stats provides methods and functionality to register, track, log, and export metrics that, for the most part, include "counter" and "latency" kinds.
 Package stats provides methods and functionality to register, track, log, and export metrics that, for the most part, include "counter" and "latency" kinds.
Package sys provides methods to read system information
 Package sys provides methods to read system information
Package tools provides common tools and utilities for all unit and integration tests
 Package tools provides common tools and utilities for all unit and integration tests
docker
Packager docker provides common utilities for managing containerized AIS deployments
 Packager docker provides common utilities for managing containerized AIS deployments
gendocs command
Package main contains constants used throughout the documentation generator.
 Package main contains constants used throughout the documentation generator.
readers
Package readers provides implementation for common reader types
 Package readers provides implementation for common reader types
tarch
Package archive provides common low-level utilities for testing archives
 Package archive provides common low-level utilities for testing archives
tassert
Package tassert provides common asserts for tests
 Package tassert provides common asserts for tests
tetl
Package tetl provides helpers for ETL.
 Package tetl provides helpers for ETL.
tlog
Package tlog provides common logf and logln primitives for dev tools
 Package tlog provides common logf and logln primitives for dev tools
trand
Package trand provides random string for dev tools and tests
 Package trand provides random string for dev tools and tests
Package tracing offers support for distributed tracing utilizing OpenTelemetry (OTEL).
 Package tracing offers support for distributed tracing utilizing OpenTelemetry (OTEL).
Package transport provides long-lived http/tcp connections for intra-cluster communications
 Package transport provides long-lived http/tcp connections for intra-cluster communications
bundle
Package bundle provides multi-streaming transport with the functionality to dynamically (un)register receive endpoints, establish long-lived flows, and more.
 Package bundle provides multi-streaming transport with the functionality to dynamically (un)register receive endpoints, establish long-lived flows, and more.
Package volume provides volume (a.k.a.
 Package volume provides volume (a.k.a.
Package xact provides core functionality for the AIStore eXtended Actions (xactions).
 Package xact provides core functionality for the AIStore eXtended Actions (xactions).
xreg
Package xreg provides registry and (renew, find) functions for AIS eXtended Actions (xactions).
 Package xreg provides registry and (renew, find) functions for AIS eXtended Actions (xactions).
xs
Package xs is a collection of eXtended actions (xactions), including multi-object operations, list-objects, (cluster) rebalance and (target) resilver, ETL, and more.
 Package xs is a collection of eXtended actions (xactions), including multi-object operations, list-objects, (cluster) rebalance and (target) resilver, ETL, and more.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.