cephmount

package
v3.6.2 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Mar 29, 2026 License: Apache-2.0 Imports: 36 Imported by: 0

README

CephMount Storage Driver

CephMount provides a CephFS-compatible filesystem interface for Reva using fstab-based configuration.

Requirements

1. Ceph Development Libraries

Install Ceph development libraries on your system:

# Fedora 42
sudo dnf install libcephfs-devel librados-devel

# CentOS/RHEL
sudo yum install libcephfs-devel librados-devel
2. CephFS Kernel Mount

You need a valid persistent CephFS kernel mount accessible by the user running Reva (typically root). Refer to Ceph fstab configuration.

ceph-mon.example.com:6789:/volumes/shared /mnt/cephfs ceph defaults,name=admin,secretfile=/etc/ceph/ceph.client.admin.key,conf=/etc/ceph/ceph.conf 0 2

This entry must be configured in your Reva config as the fstabentry parameter.

Quick Start

Configuration

Configure in your config.toml:

[grpc.services.storageprovider]
driver = "cephmount"

[grpc.services.storageprovider.drivers.cephmount]
fstabentry = "ceph-mon.example.com:6789:/volumes/shared /mnt/cephfs ceph defaults,name=admin,secretfile=/etc/ceph/ceph.client.admin.key,conf=/etc/ceph/ceph.conf 0 2"
root = "/"

[http.services.dataprovider]
driver = "cephmount"

[http.services.dataprovider.drivers.cephmount]
fstabentry = "ceph-mon.example.com:6789:/volumes/shared /mnt/cephfs ceph defaults,name=admin,secretfile=/etc/ceph/ceph.client.admin.key,conf=/etc/ceph/ceph.conf 0 2"
root = "/"

This example will convert Reva paths like /projects/alabasta/planning.md, to /mnt/cephfs/projects/alabasta/planning.md (when using localfs functions) and to /volumes/shared/projects/alabasta/planning.md when connecting directly to MDS.

Testing

There are 2 major testing scenarios:

# Unit tests (no CephFS required)
go test ./pkg/storage/fs/cephmount -v

# Integration tests (requires valid CephFS mount)
export CEPHMOUNT_FSTAB_ENTRY="your-fstab-entry-here"
go test -tags ceph ./pkg/storage/fs/cephmount -v

Depending if you run the tests as root or as another user (typically in your dev machines), some tests like the privilege verification ones may be skipped due to lack of permsissions to switch uid/guid for the forked processes.

Benchmarks

go test ./pkg/storage/fs/cephmount -bench "Benchmark.*" -v
go test -tags ceph ./pkg/storage/fs/cephmount -bench "BenchmarkCeph.*" -v

Use benchstat to compare if needed.

Development Mode

For testing without CephFS:

[grpc.services.storageprovider.drivers.cephmount]
testing_allow_local_mode = true  # Testing only!
root = "/tmp/testing"

Never use testing_allow_local_mode in production!

Functions that require conversion between file ID (inode) and and path will always require access to the MDS and therefore having a valid Ceph configuration. The tests handle this transparently, skipping any tests necessary.

Documentation

Overview

Package cephmount provides a local filesystem implementation that mimics ceph operations

Package cephmount provides a local filesystem implementation for Reva that emulates CephFS operations using the local filesystem.

This driver implements the storage.FS interface using standard Go os package operations instead of libcephfs, making it suitable for development, testing, or single-node deployments.

GetPathByID implementation without ceph support

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func ContextWithTestLogger

func ContextWithTestLogger(t *testing.T) context.Context

ContextWithTestLogger creates a context with a configured logger for testing. The log level can be controlled with CEPHMOUNT_TEST_LOG_LEVEL environment variable. Valid values: debug, info, warn, error, fatal, panic, disabled Default: error (quiet tests - only errors and above are shown)

Usage:

ctx := ContextWithTestLogger(t)
fs, err := newCephAdminConn(ctx, config)

To see verbose logs during testing:

CEPHMOUNT_TEST_LOG_LEVEL=debug go test -tags ceph -v ./pkg/storage/fs/cephmount/

func CreateCephMountFSForIntegration

func CreateCephMountFSForIntegration(t *testing.T, ctx context.Context, config map[string]any) *cephmountfs

CreateCephMountFSForIntegration creates an cephmountfs instance for integration tests. Integration tests use the real fstab entry from CEPHMOUNT_FSTAB_ENTRY environment variable.

func CreateCephMountFSForTesting

func CreateCephMountFSForTesting(t *testing.T, ctx context.Context, config map[string]any, cephVolumePath string, localMountPoint string) *cephmountfs

CreateCephMountFSForTesting creates an cephmountfs instance for unit tests. Unit tests use synthetic configuration and temporary directories. The CEPHMOUNT_FSTAB_ENTRY environment variable is ignored for unit tests.

func GetCephConfig

func GetCephConfig() map[string]any

GetCephConfig returns the Ceph configuration to use for tests. It only uses the CEPHMOUNT_FSTAB_ENTRY environment variable now.

Environment variables:

  • CEPHMOUNT_FSTAB_ENTRY: Complete fstab entry (required for integration tests)
  • CEPHMOUNT_TEST_CHROOT_DIR: Override chroot directory for testing (optional)

Usage:

export CEPHMOUNT_FSTAB_ENTRY="cephfs.cephfs /mnt/cephfs ceph defaults,name=admin,secretfile=/etc/ceph/ceph.client.admin.keyring,conf=/etc/ceph/ceph.conf 0 2"
go test -tags ceph -ceph-integration -v

func GetCurrentTestUser

func GetCurrentTestUser(t *testing.T) *userv1beta1.User

GetCurrentTestUser returns the current user information for use in tests

func GetTestDir

func GetTestDir(t *testing.T, prefix string) (string, func())

GetTestDir returns a directory for testing. It checks for the CEPHMOUNT_TEST_DIR environment variable. If set, it creates a subdirectory within that path. If not set, it falls back to creating a temporary directory.

The returned cleanup function should be called to remove the test directory unless CEPHMOUNT_TEST_DIR is set and CEPHMOUNT_TEST_PRESERVE is also set to "true".

Usage:

testDir, cleanup := GetTestDir(t, "test-prefix")
defer cleanup()

Environment variables:

CEPHMOUNT_TEST_DIR: Base directory for tests (e.g., "/mnt/ceph/test")
CEPHMOUNT_TEST_PRESERVE: If "true", preserves test directories when CEPHMOUNT_TEST_DIR is set

func GetTestSubDir

func GetTestSubDir(t *testing.T, baseDir, subDirName string) string

GetTestSubDir creates a subdirectory within an existing test directory. This is useful when you need multiple directories for a single test.

func New

func New(ctx context.Context, m map[string]any) (storage.FS, error)

New returns an implementation of the storage.FS interface that talks to the local filesystem using os.Root operations instead of libcephfs.

func NewForTesting

func NewForTesting(t *testing.T, ctx context.Context, config map[string]any, cephVolumePath string, localMountPoint string) *cephmountfs

func NewUserThreadPool

NewUserThreadPool creates a new thread pool for managing per-user threads

func RequireCephIntegration

func RequireCephIntegration(t *testing.T)

RequireCephIntegration checks if Ceph integration configuration is valid and Ceph is accessible. Integration tests run automatically when the ceph build tag is used and CEPHMOUNT_FSTAB_ENTRY is set. If CEPHMOUNT_FSTAB_ENTRY is not set or Ceph is not accessible, the test is skipped.

Usage:

func TestCephFeature(t *testing.T) {
    RequireCephIntegration(t)
    // ... rest of test
}

Run with: go test -tags ceph -v (requires CEPHMOUNT_FSTAB_ENTRY to be set and Ceph accessible)

func SetupTestDir

func SetupTestDir(t *testing.T, prefix string, uid, gid int) (string, func())

SetupTestDir is a convenience function that calls GetTestDir and sets up appropriate permissions for the returned directory. It also handles the common pattern of changing ownership to allow test users to write.

The uid and gid parameters specify the desired ownership. If uid is 0, no ownership change is attempted.

func ValidateCephConfig

func ValidateCephConfig(t *testing.T) bool

ValidateCephConfig checks if the required Ceph configuration is available. It returns true if configuration appears to be valid, false otherwise.

Types

type CephAdminConn

type CephAdminConn struct{}

CephAdminConn represents the admin connection to ceph for GetPathByID operations This is a stub when ceph support is disabled

func (*CephAdminConn) Close

func (c *CephAdminConn) Close()

Close is a no-op when ceph support is disabled

type CephMountInfo

type CephMountInfo struct {
	MonitorHost     string // e.g., "cephminiflax.cern.ch:6789"
	CephVolumePath  string // e.g., "/volumes/_nogroup/rasmus"
	LocalMountPoint string // e.g., "/mnt/miniflax"
	ClientName      string // e.g., "mds-admin"
}

CephMountInfo represents information discovered from system configuration

func DiscoverCephMountInfo

func DiscoverCephMountInfo(ctx context.Context, cephConfigFile string) (*CephMountInfo, error)

DiscoverCephMountInfo attempts to auto-discover Ceph mount configuration from system files

type FstabMountInfo

type FstabMountInfo struct {
	MonitorHost     string // e.g., "cephminiflax.cern.ch:6789"
	CephVolumePath  string // e.g., "/volumes/_nogroup/rasmus"
	LocalMountPoint string // e.g., "/mnt/miniflax"
	ClientName      string // e.g., "mds-admin"
	SecretFile      string // e.g., "/etc/ceph/miniflax.mds-admin.secret"
	ConfigFile      string // e.g., "/etc/ceph/miniflax.conf"
	KeyringFile     string // e.g., "/etc/ceph/miniflax.client.mds-admin.keyring"
}

FstabMountInfo contains all information extracted from a Ceph fstab entry

func ParseFstabEntry

func ParseFstabEntry(ctx context.Context, fstabLine string) (*FstabMountInfo, error)

ParseFstabEntry parses a Ceph fstab entry and extracts all configuration information

Example fstab entry: cephminiflax.cern.ch:6789:/volumes/_nogroup/rasmus /mnt/miniflax ceph name=mds-admin,secretfile=/etc/ceph/miniflax.mds-admin.secret,x-systemd.device-timeout=30,x-systemd.mount-timeout=30,noatime,_netdev,wsync 0 2

type Options

type Options struct {
	UploadFolder   string `mapstructure:"uploads"`
	DirPerms       uint32 `mapstructure:"dir_perms"`
	FilePerms      uint32 `mapstructure:"file_perms"`
	UserQuotaBytes uint64 `mapstructure:"user_quota_bytes"`

	// Nobody user/group for fallback operations (instead of root)
	NobodyUID int `mapstructure:"nobody_uid"`
	NobodyGID int `mapstructure:"nobody_gid"`

	// Simplified Ceph configuration - just paste the fstab entry
	FstabEntry string `mapstructure:"fstabentry"` // Complete fstab line for Ceph mount
	RootDir    string `mapstructure:"root_dir"`   // Root directory for the Ceph mount
	FsName     string `mapstructure:"fs_name"`    // Ceph filesystem name (e.g., "cephfs")

	// Testing-only option - allows running without Ceph configuration for local filesystem tests
	TestingAllowLocalMode bool `mapstructure:"testing_allow_local_mode"` // Bypass fstab parsing requirement for tests only

	HiddenDirs map[string]bool
}

Options for the cephmount module

func (*Options) ApplyDefaults

func (c *Options) ApplyDefaults()

type PrivilegeVerificationResult

type PrivilegeVerificationResult struct {
	CanChangeUID    bool
	CanChangeGID    bool
	CurrentUID      int
	CurrentGID      int
	CurrentFsUID    int
	CurrentFsGID    int
	TestedUIDs      []int
	TestedGIDs      []int
	ErrorMessages   []string
	Recommendations []string
}

PrivilegeVerificationResult contains the results of privilege verification

func VerifyPrivileges

func VerifyPrivileges(nobodyUID, nobodyGID int) *PrivilegeVerificationResult

VerifyPrivileges checks if the current process has sufficient privileges to use setfsuid/setfsgid

func (*PrivilegeVerificationResult) HasPartialPrivileges

func (r *PrivilegeVerificationResult) HasPartialPrivileges() bool

HasPartialPrivileges returns true if the process can change at least one of UID or GID

func (*PrivilegeVerificationResult) HasSufficientPrivileges

func (r *PrivilegeVerificationResult) HasSufficientPrivileges() bool

HasSufficientPrivileges returns true if the process can change both UID and GID

func (*PrivilegeVerificationResult) String

func (r *PrivilegeVerificationResult) String() string

String returns a human-readable summary of the verification results

type ThreadRequest

type ThreadRequest struct {
	ID       string
	Function func() (any, error)
	Context  context.Context
}

ThreadRequest represents a request to execute on a user thread

type ThreadResponse

type ThreadResponse struct {
	ID     string
	Result any
	Error  error
}

ThreadResponse represents the response from a user thread

type UserThread

type UserThread struct {
	// contains filtered or unexported fields
}

UserThread represents a dedicated thread for a user with persistent UID

func (*UserThread) Execute

func (ut *UserThread) Execute(ctx context.Context, fn func() (any, error)) (any, error)

Execute executes a function on this user thread

func (*UserThread) GetStats

func (ut *UserThread) GetStats() map[string]any

GetStats returns statistics about the thread

type UserThreadPool

type UserThreadPool struct {
	// contains filtered or unexported fields
}

UserThreadPool manages a pool of threads, each dedicated to a specific user UID

func (*UserThreadPool) ExecuteOnUserThread

func (p *UserThreadPool) ExecuteOnUserThread(ctx context.Context, user *userv1beta1.User, fn func() (any, error)) (any, error)

ExecuteOnUserThread executes a function on the appropriate user thread

func (*UserThreadPool) GetOrCreateUserThread

func (p *UserThreadPool) GetOrCreateUserThread(ctx context.Context, user *userv1beta1.User) (*UserThread, error)

GetOrCreateUserThread gets an existing user thread or creates a new one

func (*UserThreadPool) Shutdown

func (p *UserThreadPool) Shutdown()

Shutdown gracefully shuts down all user threads

type UserThreadPoolConfig

type UserThreadPoolConfig struct {
	ThreadTTL     time.Duration // How long to keep idle threads
	CleanupPeriod time.Duration // How often to check for expired threads
	NobodyUID     int           // UID for nobody user (fallback)
	NobodyGID     int           // GID for nobody group (fallback)
}

UserThreadPoolConfig holds configuration for the thread pool

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL