s3

package

v0.14.0 Latest Latest Go to latest Published: Apr 20, 2026 License: Apache-2.0 Imports: 11 Imported by: 0

Details

Valid go.mod file
Redistributable license
Tagged version
Stable version
Learn more about best practices

Repository

github.com/raystack/meteor

Links

Open Source Insights

README ¶

S3

Write metadata records as NDJSON to an Amazon S3 or S3-compatible storage bucket.

Usage

sinks:
  - name: s3
    config:
      bucket_url: s3://bucket-name/optional-prefix
      region: us-east-1
      object_prefix: github-users
      access_key_id: <your-access-key>
      secret_access_key: <your-secret-key>

Configuration

Key	Type	Example	Description
`bucket_url`	`string`	`s3://bucket-name/prefix`	S3 URL in the format `s3://<bucket>/<optional_path>`	required
`region`	`string`	`us-east-1`	AWS region for the S3 bucket	required
`object_prefix`	`string`	`github-users`	Prefix for the output object name. The final object is named `{prefix}-{timestamp}.ndjson`. If omitted, the name is `{timestamp}.ndjson`.	optional
`access_key_id`	`string`	`AKIAIOSFODNN7EXAMPLE`	AWS access key ID. If omitted, the default AWS credential chain is used.	optional
`secret_access_key`	`string`	`wJalrXUtnFEMI/K7MDENG/...`	AWS secret access key. If omitted, the default AWS credential chain is used.	optional
`endpoint`	`string`	`http://localhost:9000`	Custom endpoint URL for S3-compatible stores such as MinIO.	optional

Behavior

Each Record (Entity + Edges) is serialized as JSON and written as one line in an NDJSON object in the configured S3 bucket. A new object with a timestamped name is created on each run. If no explicit credentials are provided, the sink falls back to the default AWS credential chain (environment variables, shared config, instance profile, etc.).

Contributing

Refer to the contribution guidelines for information on contributing to this module.

Documentation ¶

Index ¶

func New(logger log.Logger) plugins.Syncer
type Config
type Sink

Constants ¶

This section is empty.

Variables ¶

This section is empty.

Functions ¶

func New ¶

func New(logger log.Logger) plugins.Syncer

Types ¶

type Config ¶

type Config struct {
	BucketURL       string `mapstructure:"bucket_url" validate:"required"`
	Region          string `mapstructure:"region" validate:"required"`
	ObjectPrefix    string `mapstructure:"object_prefix"`
	AccessKeyID     string `mapstructure:"access_key_id"`
	SecretAccessKey string `mapstructure:"secret_access_key"`
	Endpoint        string `mapstructure:"endpoint"`
}

type Sink ¶

type Sink struct {
	plugins.BasePlugin
	// contains filtered or unexported fields
}

func (*Sink) Close ¶

func (s *Sink) Close() error

func (*Sink) Init ¶

func (s *Sink) Init(ctx context.Context, config plugins.Config) (err error)

func (*Sink) Sink ¶

func (s *Sink) Sink(ctx context.Context, batch []models.Record) (err error)

Source Files ¶

View all Source files

s3.go

Directories ¶

Path	Synopsis
client

?	: This menu
/	: Search site
f or F	: Jump to
y or Y	: Canonical URL