README
¶
Util Package
The util package provides a collection of utility functions and types for common operations in the ClusterCockpit library.
Overview
This package contains utilities for:
- Array operations - Generic helper functions for slices
- File compression - Gzip compression and decompression
- File/directory operations - Copying files and directories
- Disk usage - Calculating directory size
- Custom types - Selector types
- File system watcher - Event-based file system monitoring
- Statistics - Basic statistical functions (mean, median, min, max)
- Secrets - Resolving secrets from environment variables or secret files
The Float type with JSON NaN support is in the schema package,
not here.
Key Features
Secrets from the Environment
A secret should not have to live in a configuration file. SecretFromEnv
resolves one from three sources, in this order of precedence:
- the environment variable
$VAR, if set and non-empty - the contents of the file named by
$VAR_FILE - the value read from the configuration file
// Resolves $NATS_PASSWORD, else the file named by $NATS_PASSWORD_FILE,
// else the value from the config file.
password, err := util.SecretFromEnv("CC_NATS_PASSWORD", cfg.Password)
if err != nil {
return err
}
The second step is what makes the standard secret-mounting mechanisms usable:
# Docker / docker compose
docker run -e CC_NATS_PASSWORD_FILE=/run/secrets/nats-pw ...
# Kubernetes: mount the secret, then name the path
# env:
# - name: CC_NATS_PASSWORD_FILE
# value: /etc/secrets/nats-pw
# systemd
[Service]
LoadCredential=nats-pw:/etc/cc/nats-pw
Environment=CC_NATS_PASSWORD_FILE=%d/nats-pw
For a repeated configuration section — one sink, one receiver — no fixed
variable name can address a single instance, so the instance names its own
sources through sibling configuration keys and resolves them with
SecretFromConfig:
// {"password": "...", "password_env": "...", "password_file": "..."}
password, err := util.SecretFromConfig(cfg.Password, cfg.PasswordEnv, cfg.PasswordFile)
Rules that apply to both functions:
- An empty environment variable counts as unset.
- File contents are trimmed of surrounding whitespace, so a secret file may end in a newline. A secret with significant leading or trailing whitespace cannot be supplied through a file.
- A secret file that cannot be read, or that holds no non-whitespace characters, is an error rather than a silent fallback to the configured value. An operator who names a file intends it to win, so falling back would quietly start the process with a stale credential.
- Neither function ever logs a resolved secret. Errors carry only the path.
Environment variables that cc-lib itself reads are prefixed CC_, so they
cannot collide with a variable already present in the environment of the
application linking cc-lib.
File Operations
// Compress a file
err := util.CompressFile("input.txt", "output.txt.gz")
// Decompress a file
err = util.UncompressFile("input.txt.gz", "output.txt")
// Copy a file
err = util.CopyFile("source.txt", "destination.txt")
// Copy a directory recursively
err = util.CopyDir("/path/to/source", "/path/to/dest")
Disk Usage
// Get disk usage in megabytes for a directory
usage := util.DiskUsage("/path/to/directory")
fmt.Printf("Directory uses %.2f MB\n", usage)
Array Utilities
// Check if a slice contains an element (works with any comparable type)
numbers := []int{1, 2, 3, 4, 5}
contains := util.Contains(numbers, 3) // true
strs := []string{"apple", "banana", "orange"}
contains = util.Contains(strs, "grape") // false
Statistics
data := []float64{1.0, 2.0, 3.0, 4.0, 5.0}
// Calculate mean
mean, err := util.Mean(data) // 3.0
// Calculate median
median, err := util.Median(data) // 3.0
// Min/Max (works with any ordered type)
minVal := util.Min(5, 3) // 3
maxVal := util.Max(5, 3) // 5
File System Watcher
// Implement the Listener interface
type MyListener struct{}
func (l *MyListener) EventCallback() {
fmt.Println("File changed!")
}
func (l *MyListener) EventMatch(event string) bool {
return strings.Contains(event, "myfile.txt")
}
// Add a listener
listener := &MyListener{}
util.AddListener("/path/to/watch", listener)
// Don't forget to shutdown when done
defer util.FsWatcherShutdown()
Selector Types
The SelectorElement and Selector types support flexible JSON marshaling for configuration:
// Can be a single string
var sel util.SelectorElement
json.Unmarshal([]byte(`"value"`), &sel)
// Can be an array of strings
json.Unmarshal([]byte(`["val1", "val2"]`), &sel)
// Can be a wildcard
json.Unmarshal([]byte(`"*"`), &sel)
Documentation
For complete API documentation, see the godoc.
Testing
The package includes comprehensive unit tests. Run them with:
go test ./util/...
For coverage information:
go test -cover ./util/...
Documentation
¶
Overview ¶
Package util provides common utility functions and types for ClusterCockpit applications. This file implements a file system watcher for monitoring file changes.
Index ¶
- Constants
- func AddListener(path string, l Listener)
- func CheckFileExists(filePath string) bool
- func CompressFile(fileIn string, fileOut string) error
- func Contains[T comparable](items []T, item T) bool
- func CopyDir(src string, dst string) (err error)
- func CopyFile(src, dst string) (err error)
- func DiskUsage(dirpath string) float64
- func FsWatcherShutdown()
- func GetFilecount(path string) int
- func GetFilesize(filePath string) int64
- func Max[T cmp.Ordered](a, b T) T
- func Mean(input []float64) (float64, error)
- func Median(input []float64) (median float64, err error)
- func Min[T cmp.Ordered](a, b T) T
- func SecretFromConfig(value, envName, filePath string) (string, error)
- func SecretFromEnv(envVar, configValue string) (string, error)
- func UncompressFile(fileIn string, fileOut string) error
- type Listener
- type Selector
- type SelectorElement
Constants ¶
const EnvFileSuffix = "_FILE"
EnvFileSuffix is appended to an environment variable name to obtain the name of the variable holding the path of a file that contains the secret.
Variables ¶
This section is empty.
Functions ¶
func AddListener ¶
AddListener registers a new file system watcher for the specified path. The watcher is initialized on the first call to AddListener. The listener will be notified of file system events matching its EventMatch criteria.
func CheckFileExists ¶
CheckFileExists checks if a file or directory exists at the given path. Returns true if the file exists, false otherwise.
func CompressFile ¶
CompressFile compresses a file using gzip compression. It reads the input file (fileIn), creates a gzip-compressed output file (fileOut), and removes the original file upon successful compression. Returns an error if any operation fails.
func Contains ¶
func Contains[T comparable](items []T, item T) bool
Contains checks if an item exists in a slice of comparable items. It is a generic wrapper around slices.Contains that works with any comparable type T.
func CopyDir ¶
CopyDir recursively copies a directory from src to dst, preserving file modes. It creates the destination directory with the same permissions as the source, and recursively copies all files and subdirectories. Symbolic links are skipped during the copy operation. Returns an error if the source is not a directory, if the destination already exists, or if any file operation fails.
func CopyFile ¶
CopyFile copies a file from src to dst, preserving the file mode. It creates the destination file, copies the content, syncs to disk, and sets the same permissions as the source file. Returns an error if any operation fails.
func DiskUsage ¶
DiskUsage calculates the total disk usage of a directory in megabytes (MB). It sums up the sizes of all files in the specified directory (non-recursive) and returns the result in MB (multiplied by 1e-6). Returns 0 if the directory cannot be opened or read.
func FsWatcherShutdown ¶
func FsWatcherShutdown()
FsWatcherShutdown closes the file system watcher. This should be called during application shutdown to clean up resources.
func GetFilecount ¶
GetFilecount returns the number of entries (files and directories) in a directory. Returns 0 if the directory cannot be read.
func GetFilesize ¶
GetFilesize returns the size of a file in bytes. Returns 0 if the file cannot be accessed or does not exist.
func Mean ¶
Mean calculates the arithmetic mean (average) of a float64 slice. Returns NaN and an error if the input slice is empty.
func Median ¶
Median calculates the median value of a float64 slice. For even-length slices, it returns the mean of the two middle values. For odd-length slices, it returns the middle value. Returns NaN and an error if the input slice is empty.
func SecretFromConfig ¶ added in v2.14.0
SecretFromConfig resolves a secret for one instance of a repeated configuration section (one sink, one receiver), where no fixed environment variable name exists. The instance names the sources itself, via sibling configuration keys. Precedence:
- the environment variable envName, if envName is non-empty and that variable is set and non-empty
- the contents of filePath, if filePath is non-empty
- value, the secret configured inline
An empty envName or filePath means that source is not configured. As in SecretFromEnv, file contents are whitespace-trimmed, and an unreadable or effectively empty file is an error rather than a silent fallback.
func SecretFromEnv ¶ added in v2.14.0
SecretFromEnv resolves a secret from the environment or the configuration, in this order of precedence:
- the environment variable envVar, if set and non-empty
- the contents of the file named by the environment variable envVar+EnvFileSuffix, if that variable is set and non-empty
- configValue, the value read from the configuration file
Step 2 lets a deployment inject a secret from a Docker or Kubernetes secret mount, or from systemd LoadCredential, without ever placing it in the process environment or in a configuration file.
An empty environment variable counts as unset. File contents are trimmed of leading and trailing whitespace so that secret files may end in a newline; a secret with significant leading or trailing whitespace can therefore not be supplied through a file.
An error is returned only when envVar+EnvFileSuffix is set but the file cannot be read or holds no non-whitespace characters. An operator who sets that variable intends the file to win, so falling back to configValue there would silently start the process with a stale credential.
envVar must not itself end in EnvFileSuffix, as it would then shadow the file variable of another secret. An empty envVar disables both env sources and returns configValue unchanged.
func UncompressFile ¶
UncompressFile decompresses a gzip-compressed file. It reads the gzip-compressed input file (fileIn), creates an uncompressed output file (fileOut), and removes the compressed file upon successful decompression. Returns an error if any operation fails.
Types ¶
type Listener ¶
Listener is an interface for file system event callbacks. Implementations should define what events to match and how to respond to them.
type Selector ¶
type Selector []SelectorElement
Selector is a slice of SelectorElements used for matching and filtering.
type SelectorElement ¶
SelectorElement represents a selector that can be either a single string, an array of strings, or a wildcard. It supports special JSON marshaling/unmarshaling behavior: - A single string (e.g., "value") is stored in the String field - An array of strings (e.g., ["val1", "val2"]) is stored in the Group field - The wildcard "*" is represented by setting Any to true
func (*SelectorElement) MarshalJSON ¶
func (se *SelectorElement) MarshalJSON() ([]byte, error)
MarshalJSON implements json.Marshaler for SelectorElement. It converts the selector back to JSON: - Any=true becomes "*" - String field becomes a JSON string - Group field becomes a JSON array
func (*SelectorElement) UnmarshalJSON ¶
func (se *SelectorElement) UnmarshalJSON(input []byte) error
UnmarshalJSON implements json.Unmarshaler for SelectorElement. It handles three formats: - A JSON string (converted to String field, or Any if "*") - A JSON array (converted to Group field) - Any other format returns an error
Source Files