gcs

package

v0.9.2 Latest Latest Go to latest Published: Sep 24, 2023 License: Apache-2.0 Imports: 17 Imported by: 0

Details

Valid go.mod file
Redistributable license
Tagged version
Stable version
Learn more about best practices

Repository

github.com/raystack/meteor

Links

Open Source Insights

README ¶

google cloud storage

Usage

source:
  name: gcs
  config:
    project_id: google-project-id
    extract_blob: true
     # Only one of service_account_base64 / service_account_json is needed. 
    # If both are present, service_account_base64 takes precedence
    service_account_base64: _________BASE64_ENCODED_SERVICE_ACCOUNT_________________
    service_account_json:
      {
        "type": "service_account",
        "private_key_id": "xxxxxxx",
        "private_key": "xxxxxxx",
        "client_email": "xxxxxxx",
        "client_id": "xxxxxxx",
        "auth_uri": "https://accounts.google.com/o/oauth2/auth",
        "token_uri": "https://oauth2.googleapis.com/token",
        "auth_provider_x509_cert_url": "xxxxxxx",
        "client_x509_cert_url": "xxxxxxx",
      }
    exclude: [bucket_a, bucket_b]

Inputs

Key	Value	Example	Description
`project_id`	`string`	`my-project`	Cloud Storage Project ID	required
`extract_blob`	`boolean`	`true`	Extract blob metadata inside a bucket	optional
`service_account_base64`	`string`	`____BASE64_ENCODED_SERVICE_ACCOUNT____`	Service Account in base64 encoded string. Takes precedence over `service_account_json` value	optional
`service_account_json`	`string`	`{"private_key": .., "private_id": ...}`	Service Account in JSON string	optional
`exclude`	`[]string`	`["bucket_a","bucket_b"]`	Array of bucket names to excluded from crawling	optional

Notes

Leaving service_account_base64 and service_account_json blank will default to Google's default authentication. It is recommended if Meteor instance runs inside the same Google Cloud environment as the Google Cloud Storage project.

Outputs

Field	Sample Value
`resource.urn`	`project_id/bucket_name`
`resource.name`	`bucket_name`
`resource.service`	`gcs`
`location`	`ASIA`
`storage_type`	`STANDARD`
`labels`	[]{`key`:`value`}
`timestamps.created_at.seconds`	`1551082913`
`timestamps.created_at.nanos`	`1551082913`

Blob

Field	Sample Value
`urn`	`project_id/bucket_name/blob_path`
`name`	`blob_path`
`size`	`311`
`deleted_at.seconds`	`1551082913`
`expired_at.seconds`	`1551082913`
`labels`	[]{`key`:`value`}
`ownership.owners`	[]{`name`:`serviceaccountname@project.gserviceaccount.com`}
`timestamps.created_at.seconds`	`1551082913`
`timestamps.created_at.nanos`	`1551082913`
`timestamps.updated_at.seconds`	`1551082913`
`timestamps.updated_at.nanos`	`1551082913`

Contributing

Refer to the contribution guidelines for information on contributing to this module.

Documentation ¶

Index ¶

type Config
type Extractor
- func New(logger log.Logger, newClient NewClientFunc) *Extractor
- func (e *Extractor) Extract(ctx context.Context, emit plugins.Emit) (err error)
- func (e *Extractor) Init(ctx context.Context, config plugins.Config) error
type NewClientFunc

Constants ¶

This section is empty.

Variables ¶

This section is empty.

Functions ¶

This section is empty.

Types ¶

type Config ¶

type Config struct {
	ProjectID string `json:"project_id" yaml:"project_id" mapstructure:"project_id" validate:"required"`
	// ServiceAccountBase64 takes precedence over ServiceAccountJSON field
	ServiceAccountBase64 string   `json:"service_account_base64" yaml:"service_account_base64" mapstructure:"service_account_base64"`
	ServiceAccountJSON   string   `json:"service_account_json" yaml:"service_account_json" mapstructure:"service_account_json"`
	ExtractBlob          bool     `json:"extract_blob" yaml:"extract_blob" mapstructure:"extract_blob"`
	Exclude              []string `json:"exclude" yaml:"exclude" mapstructure:"exclude"`
}

Config holds the set of configuration for the extractor

type Extractor ¶

type Extractor struct {
	plugins.BaseExtractor
	// contains filtered or unexported fields
}

Extractor manages the extraction of data from the google cloud storage

func New ¶

func New(logger log.Logger, newClient NewClientFunc) *Extractor

New returns a pointer to an initialized Extractor Object

func (*Extractor) Extract ¶

func (e *Extractor) Extract(ctx context.Context, emit plugins.Emit) (err error)

func (*Extractor) Init ¶

func (e *Extractor) Init(ctx context.Context, config plugins.Config) error

Init initializes the extractor

type NewClientFunc ¶ added in v0.9.1

type NewClientFunc func(ctx context.Context, logger log.Logger, config Config) (*storage.Client, error)

Source Files ¶

View all Source files

gcs.go

?	: This menu
/	: Search site
f or F	: Jump to
y or Y	: Canonical URL