README
¶
s3bytes
s3bytes is a CLI tool that lists the size and number of objects in all buckets in one shot.
Features
- High speed with concurrent processing
- Various output formats
- Simple processing using CloudWatch Metrics API
- Simple command line flags
Options
List of possible values for flags as follows:
| Option | Description | Allowed values | Default value | Environment Variable |
|---|---|---|---|---|
--profile value -p value |
set aws profile | - | - | AWS_PROFILE |
--log-level value -l value |
set log level | debug info warn error |
info |
S3BYTES_LOG_LEVEL |
--region value1,value2... -r value1,value2... |
set target regions | af-south-1 ap-east-1 ap-northeast-1 ap-northeast-2 ap-northeast-3 ap-south-1 ap-south-2 ap-southeast-1 ap-southeast-2 ap-southeast-3 ap-southeast-4 ap-southeast-5 ap-southeast-7 ca-central-1 ca-west-1 eu-central-1 eu-central-2 eu-north-1 eu-south-1 eu-south-2 eu-west-1 eu-west-2 eu-west-3 il-central-1 me-central-1 me-south-1 mx-central-1 sa-east-1 us-east-1 us-east-2 us-west-1 us-west-2 |
All regions with no opt-in | - |
--prefix value -P value |
set bucket name prefix | - | - | - |
--filter value -f value |
set filter expression for metric values | Key: bytes Bytes value ValueExamples: bytes > 2 Bytes >= 4 value < 8 Value <= 16 bytes == 32 Bytes != 64 |
- | - |
--metric-name value -m value |
set metric name of cloudwatch metrics | BucketSizeBytes NumberOfObjects |
BucketSizeBytes |
- |
--storage-type value -s value |
set storage type of s3 objects | StandardStorage IntelligentTieringFAStorage IntelligentTieringIAStorage IntelligentTieringAAStorage IntelligentTieringAIAStorage IntelligentTieringDAAStorage StandardIAStorage StandardIASizeOverhead StandardIAObjectOverhead OneZoneIAStorage OneZoneIASizeOverhead ReducedRedundancyStorage GlacierIRSizeOverhead GlacierInstantRetrievalStorage GlacierStorage GlacierStagingStorage GlacierObjectOverhead GlacierS3ObjectOverhead DeepArchiveStorage DeepArchiveObjectOverhead DeepArchiveS3ObjectOverhead DeepArchiveStagingStorage AllStorageTypes |
StandardStorage |
- |
--output value -o value |
set output type | json prettyjson text compressedtext markdown backlog tsv chart |
text |
S3BYTES_OUTPUT_TYPE |
--help -h |
show help | - | - | - |
--version -v |
print the version | - | - | - |
Output type
Pretty JSON format
[
{
"BucketName": "bucket0",
"Region": "ap-northeast-1",
"MetricName": "BucketSizeBytes",
"StorageType": "StandardStorage",
"Value": 23373655,
},
{
"BucketName": "bucket1",
"Region": "ap-northeast-2",
"MetricName": "BucketSizeBytes",
"StorageType": "StandardStorage",
"Value": 134614,
},
{
"BucketName": "bucket2",
"Region": "us-east-1",
"MetricName": "BucketSizeBytes",
"StorageType": "StandardStorage",
"Value": 0,
}
]
Text table format
$ s3bytes -o text
+------------+----------------+-----------------+-----------------+----------+
| BucketName | Region | MetricName | StorageType | Value |
+------------+----------------+-----------------+-----------------+----------+
| bucket0 | ap-northeast-1 | BucketSizeBytes | StandardStorage | 23373655 |
+------------+----------------+-----------------+-----------------+----------+
| bucket1 | ap-northeast-2 | BucketSizeBytes | StandardStorage | 134614 |
+------------+----------------+-----------------+-----------------+----------+
| bucket2 | us-east-1 | BucketSizeBytes | StandardStorage | 0 |
+------------+----------------+-----------------+-----------------+----------+
Compressed text table format
$ s3bytes -o compressedtext
+------------+----------------+-----------------+-----------------+----------+
| BucketName | Region | MetricName | StorageType | Value |
+------------+----------------+-----------------+-----------------+----------+
| bucket0 | ap-northeast-1 | BucketSizeBytes | StandardStorage | 23373655 |
| bucket1 | ap-northeast-2 | BucketSizeBytes | StandardStorage | 134614 |
| bucket2 | us-east-1 | BucketSizeBytes | StandardStorage | 0 |
+------------+----------------+-----------------+-----------------+----------+
Markdown table format
$ s3bytes -o markdown
| BucketName | Region | MetricName | StorageType | Value |
| ---------- | -------------- | --------------- | --------------- | -------- |
| bucket0 | ap-northeast-1 | BucketSizeBytes | StandardStorage | 23373655 |
| bucket1 | ap-northeast-2 | BucketSizeBytes | StandardStorage | 134614 |
| bucket2 | us-east-1 | BucketSizeBytes | StandardStorage | 0 |
Backlog table format
$ s3bytes -o backlog
| BucketName | Region | MetricName | StorageType | Value |h
| bucket0 | ap-northeast-1 | BucketSizeBytes | StandardStorage | 23373655 |
| bucket1 | ap-northeast-2 | BucketSizeBytes | StandardStorage | 134614 |
| bucket2 | us-east-1 | BucketSizeBytes | StandardStorage | 0 |
And visualization is also possible. Displays a pie chart in your browser in one shot!!
Installation
Install with homebrew
brew install nekrassov01/tap/s3bytes
Install with go
go install github.com/nekrassov01/s3bytes
Or download binary from releases
Shell completion
Supported Shells are as follows:
- bash
- zsh
- fish
- pwsh
s3bytes completion bash|zsh|fish|pwsh
Todo
- Add code comment
- Implement visualization
- Support directory buckets if possible
- Support table buckets if possible
- Support vector buckets if possible
- Support streaming output (in github.com/nekrassov01/mintab)
Author
License
Documentation
¶
Index ¶
- Variables
- func LoadConfig(ctx context.Context, profile string) (aws.Config, error)
- func SortMetrics(data *MetricData)
- func Version() string
- type Client
- type CloudWatch
- type CloudWatchAPI
- type Manager
- func (man *Manager) List(ctx context.Context) (*MetricData, error)
- func (man *Manager) SetFilter(raw string) error
- func (man *Manager) SetMetric(metricName MetricName, storageType StorageType) error
- func (man *Manager) SetPrefix(prefix string) error
- func (man *Manager) SetRegion(regions []string) error
- func (man *Manager) String() string
- type Metric
- type MetricData
- type MetricName
- type OutputType
- type Renderer
- type S3
- type S3API
- type StorageType
Constants ¶
This section is empty.
Variables ¶
View Source
var ( // NumWorker is the number of workers for concurrent processing. NumWorker = int64(runtime.NumCPU()*2 + 1) // MaxQueries is the maximum number of queries for GetMetricData. // See: https://docs.aws.amazon.com/AmazonCloudWatch/latest/APIReference/API_MetricDataQuery.html MaxQueries = 500 // MaxChartItems is the maximum number of items in a chart. MaxChartItems = 11 // DefaultRegion is the region speficied by default. DefaultRegion = "us-east-1" // DefaultRegions is the default target regions. DefaultRegions = []string{ "us-east-1", "us-east-2", "us-west-1", "us-west-2", "ap-south-1", "ap-northeast-3", "ap-northeast-2", "ap-southeast-1", "ap-southeast-2", "ap-northeast-1", "ca-central-1", "eu-central-1", "eu-west-1", "eu-west-2", "eu-west-3", "eu-north-1", "sa-east-1", } )
Functions ¶
func LoadConfig ¶ added in v0.0.8
LoadConfig loads the aws config.
func SortMetrics ¶ added in v0.0.8
func SortMetrics(data *MetricData)
SortMetrics sorts the metrics by value and bucket name.
Types ¶
type Client ¶
type Client struct {
S3API
CloudWatchAPI
}
Client is a wrapper for the s3 and cloudwatch clients.
type CloudWatch ¶ added in v0.0.8
type CloudWatch struct {
*cloudwatch.Client
}
CloudWatch is a wrapper for the cloudwatch client.
type CloudWatchAPI ¶ added in v0.0.8
type CloudWatchAPI interface {
GetMetricData(ctx context.Context, params *cloudwatch.GetMetricDataInput, optFns ...func(*cloudwatch.Options)) (*cloudwatch.GetMetricDataOutput, error)
}
CloudWatchAPI is an interface for the cloudwatch client.
type Manager ¶
type Manager struct {
// contains filtered or unexported fields
}
Manager is a manager struct for the s3bytes package.
func (*Manager) List ¶ added in v0.0.8
func (man *Manager) List(ctx context.Context) (*MetricData, error)
List retrieves the metrics data for all regions and returns it as a MetricData struct. It uses concurrency to fetch the data from multiple regions simultaneously, and handles errors gracefully by canceling the context if any error occurs.
func (*Manager) SetMetric ¶ added in v0.0.8
func (man *Manager) SetMetric(metricName MetricName, storageType StorageType) error
SetMetric sets the metric name and storage type.
type Metric ¶
type Metric struct {
BucketName string
Region string
MetricName MetricName
StorageType StorageType
Value float64
}
Metric represents the metrics data for a single bucket.
type MetricData ¶ added in v0.0.8
MetricData represents the metrics data for all regions, including the header and the list of metrics.
type MetricName ¶
type MetricName int
MetricName represents the metric name.
const ( // MetricNameNone is the metric name that means none. MetricNameNone MetricName = iota //MetricNameBucketSizeBytes is the metric name that means bucket size in bytes. MetricNameBucketSizeBytes // MetricNameNumberOfObjects is the metric name that means number of objects. MetricNameNumberOfObjects )
func ParseMetricName ¶
func ParseMetricName(s string) (MetricName, error)
ParseMetricName parses the metric name from the string representation.
func (MetricName) MarshalJSON ¶
func (t MetricName) MarshalJSON() ([]byte, error)
MarshalJSON returns the JSON representation of the metric name.
func (MetricName) String ¶
func (t MetricName) String() string
String returns the string representation of the metric name.
type OutputType ¶
type OutputType int
OutputType represents the output type of the renderer.
const ( // OutputTypeNone is the output type that means none. OutputTypeNone OutputType = iota // OutputTypeJSON is the output type that means JSON format. OutputTypeJSON // OutputTypePrettyJSON is the output type that means pretty JSON format. OutputTypePrettyJSON // OutputTypeText is the output type that means text format. OutputTypeText // OutputTypeCompressedText is the output type that means compressed text format. OutputTypeCompressedText // OutputTypeMarkdown is the output type that means markdown format. OutputTypeMarkdown // OutputTypeBacklog is the output type that means backlog format. OutputTypeBacklog // OutputTypeTSV is the output type that means TSV format. OutputTypeTSV // OutputTypeChart is the output type that means pie chart. OutputTypeChart )
func ParseOutputType ¶
func ParseOutputType(s string) (OutputType, error)
ParseOutputType parses the output type from the string representation.
func (OutputType) MarshalJSON ¶
func (t OutputType) MarshalJSON() ([]byte, error)
MarshalJSON returns the JSON representation of the output type.
func (OutputType) String ¶
func (t OutputType) String() string
String returns the string representation of the output type.
type Renderer ¶
type Renderer struct {
Data *MetricData
OutputType OutputType
// contains filtered or unexported fields
}
Renderer is a renderer struct for the s3bytes package. OutputType represents the type of the output.
func NewRenderer ¶
func NewRenderer(w io.Writer, data *MetricData, outputType OutputType) *Renderer
NewRenderer creates a new renderer with the specified parameters.
type S3API ¶ added in v0.0.8
type S3API interface {
ListBuckets(ctx context.Context, params *s3.ListBucketsInput, optFns ...func(*s3.Options)) (*s3.ListBucketsOutput, error)
}
S3API is an interface for the s3 client.
type StorageType ¶
type StorageType int
StorageType represents the storage type. See: https://docs.aws.amazon.com/AmazonS3/latest/userguide/metrics-dimensions.html#s3-cloudwatch-metrics
const ( // StorageTypeNone is the storage type that means none. StorageTypeNone StorageType = iota // StorageTypeStandardStorage is the storage type that means standard storage. StorageTypeStandardStorage // StorageTypeIntelligentTieringFAStorage is the storage type that means intelligent tiering FA storage. StorageTypeIntelligentTieringFAStorage // StorageTypeIntelligentTieringIAStorage is the storage type that means intelligent tiering IA storage. StorageTypeIntelligentTieringIAStorage // StorageTypeIntelligentTieringAAStorage is the storage type that means intelligent tiering AA storage. StorageTypeIntelligentTieringAAStorage // StorageTypeIntelligentTieringAIAStorage is the storage type that means intelligent tiering AIA storage. StorageTypeIntelligentTieringAIAStorage // StorageTypeIntelligentTieringDAAStorage is the storage type that means intelligent tiering DAA storage. StorageTypeIntelligentTieringDAAStorage // StorageTypeStandardIAStorage is the storage type that means standard IA storage. StorageTypeStandardIAStorage // StorageTypeStandardIASizeOverhead is the storage type that means standard IA size overhead. StorageTypeStandardIASizeOverhead // StorageTypeStandardIAObjectOverhead is the storage type that means standard IA object overhead. StorageTypeStandardIAObjectOverhead // StorageTypeOneZoneIAStorage is the storage type that means one zone IA storage. StorageTypeOneZoneIAStorage // StorageTypeOneZoneIASizeOverhead is the storage type that means one zone IA size overhead. StorageTypeOneZoneIASizeOverhead // StorageTypeOneZoneIAObjectOverhead is the storage type that means one zone IA object overhead. StorageTypeOneZoneIAObjectOverhead // StorageTypeReducedRedundancyStorage is the storage type that means reduced redundancy storage. StorageTypeReducedRedundancyStorage // StorageTypeGlacierIRSizeOverhead is the storage type that means glacier IR size overhead. StorageTypeGlacierIRSizeOverhead // StorageTypeGlacierInstantRetrievalStorage is the storage type that means glacier instant retrieval storage. StorageTypeGlacierInstantRetrievalStorage // StorageTypeGlacierStorage is the storage type that means glacier storage. StorageTypeGlacierStorage // StorageTypeGlacierStagingStorage is the storage type that means glacier staging storage. StorageTypeGlacierStagingStorage // StorageTypeGlacierObjectOverhead is the storage type that means glacier object overhead. StorageTypeGlacierObjectOverhead // StorageTypeGlacierS3ObjectOverhead is the storage type that means glacier S3 object overhead. StorageTypeGlacierS3ObjectOverhead // StorageTypeDeepArchiveStorage is the storage type that means deep archive storage. StorageTypeDeepArchiveStorage // StorageTypeDeepArchiveObjectOverhead is the storage type that means deep archive object overhead. StorageTypeDeepArchiveObjectOverhead // StorageTypeDeepArchiveS3ObjectOverhead is the storage type that means deep archive S3 object overhead. StorageTypeDeepArchiveS3ObjectOverhead // StorageTypeDeepArchiveStagingStorage is the storage type that means deep archive staging storage. StorageTypeDeepArchiveStagingStorage // StorageTypeAllStorageTypes is the storage type that means all storage types. StorageTypeAllStorageTypes )
func ParseStorageType ¶
func ParseStorageType(s string) (StorageType, error)
ParseStorageType parses the storage type from the string representation.
func (StorageType) MarshalJSON ¶
func (t StorageType) MarshalJSON() ([]byte, error)
MarshalJSON returns the JSON representation of the storage type.
func (StorageType) String ¶
func (t StorageType) String() string
String returns the string representation of the storage type.
Source Files
¶
Directories
Click to show internal directories.
Click to hide internal directories.