gcs/

GCS Client

GCS (Google Cloud Storage) client implementation for the unified storage-cli tool. This module provides Google Cloud Storage operations through the main storage-cli binary.

Note: This is not a standalone CLI. Use the main storage-cli binary with -s gcs flag to access GCS functionality.

For general usage and build instructions, see the main README.

This is not an official Google Product.

GCS-Specific Configuration

The GCS client requires a JSON configuration file.

{
  "bucket_name":            "<string> (required)",
  "credentials_source":     "<string> ['static'|'none'|""]",
  "json_key":               "<string> (required if credentials_source = 'static')",
  "storage_class":          "<string> (optional - default: 'STANDARD', check for more options=https://docs.cloud.google.com/storage/docs/storage-classes)",
  "encryption_key":         "<string> (optional)",
  "uniform_bucket_level_access": "<boolean> (optional)"
}

Credentials Source Types

• "": specifies that credentials should be detected. Application Default Credentials will be used if avaliable. A read-only client will be used otherwise.
• "none": specifies that credentials are explicitly empty and that the client should be restricted to a read-only scope.
• "static:" specifies that a service account file included in json_key should be used for authentication.

Bucket Creation

The ensure-storage-exists command creates a bucket if it does not already exist. The uniform_bucket_level_access configuration option controls the access control model:

• true: Creates a bucket with uniform bucket-level access (IAM-only, ACLs disabled)
• false or omitted (default): Creates a bucket with fine-grained access control (ACLs enabled)

Authentication Methods (credentials_source)

• static: A service account key will be provided via the json_key field.
• none: No credentials are provided. The client is reading from a public bucket.
• <empty>: Application Default Credentials will be used if they exist (either through gcloud auth application-default login or a service account). If they don't exist the client will fall back to none behavior.

Usage examples:

# Upload an object
storage-cli -s gcs -c gcs-config.json put local-file.txt remote-blob

# Fetch an object
storage-cli -s gcs -c gcs-config.json get remote-blob local-file.txt

# Delete an object
storage-cli -s gcs -c gcs-config.json delete remote-blob

# Check if an object exists
storage-cli -s gcs -c gcs-config.json exists remote-blob

# Generate a signed URL (e.g., GET for 1 hour)
storage-cli -s gcs -c gcs-config.json sign remote-blob get 60s

Testing

Unit Tests

Run unit tests from the repository root:

ginkgo --skip-package=integration --cover -v -r ./gcs/...

Integration Tests

1. Create a service account with the Storage Admin role.
2. Create a new key for your service account and download credential as JSON file.
3. Export json content with export google_json_key_data="$(cat <path-to-json-file.json>)".
4. Export export SKIP_LONG_TESTS=yes if you want to run only the fast running tests.
5. Navigate to project's root folder.
6. Run environment setup script to create buckets /.github/scripts/gcs/setup.sh.
7. Run tests /.github/scripts/gcs/run-int.sh.
8. Run environment teardown script to delete resources /.github/scripts/gcs/teardown.sh.