README
¶
GCS
Write metadata records as NDJSON to a Google Cloud Storage bucket.
Usage
sinks:
- name: gcs
config:
project_id: google-project-id
url: gcs://bucket_name/target_folder
object_prefix: github-users
service_account_base64: <base64 encoded service account key>
Configuration
| Key | Type | Example | Description | |
|---|---|---|---|---|
project_id |
string |
google-project-id |
Google Cloud project ID | required |
url |
string |
gcs://bucket_name/target_folder |
GCS URL in the format gcs://<bucket>/<optional_path> |
required |
object_prefix |
string |
github-users |
Prefix for the output object name. The final object is named {prefix}-{timestamp}.ndjson. If omitted, the name is {timestamp}.ndjson. |
optional |
service_account_base64 |
string |
ewog....fQo= |
Service account key as a base64-encoded string. Takes precedence over service_account_json. |
optional |
service_account_json |
string |
{"private_key": ...} |
Service account key as a JSON string. Either this or service_account_base64 must be provided. |
optional |
Behavior
Each Record (Entity + Edges) is serialized as JSON and written as one line in an NDJSON object in the configured GCS bucket. A new object with a timestamped name is created on each run.
Contributing
Refer to the contribution guidelines for information on contributing to this module.
Documentation
¶
Index ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
Types ¶
type Config ¶
type Config struct {
ProjectID string `mapstructure:"project_id" validate:"required"`
URL string `mapstructure:"url" validate:"required"`
ObjectPrefix string `mapstructure:"object_prefix"`
ServiceAccountJSON string `mapstructure:"service_account_json"`
ServiceAccountBase64 string `mapstructure:"service_account_base64"`
}
type Sink ¶
type Sink struct {
plugins.BasePlugin
// contains filtered or unexported fields
}
Source Files
Directories
Click to show internal directories.
Click to hide internal directories.