databrew

package
v1.2.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Aug 3, 2026 License: MIT Imports: 21 Imported by: 0

README

Glue DataBrew

Parity grade: A · SDK aws-sdk-go-v2/service/databrew@v1.40.0 · last audited 2026-07-23 (782e2a93)

Coverage

Metric Value
Operations audited 45 (45 ok)
Feature families 4 (4 ok)
Known gaps 2
Deferred items 1
Resource leaks clean
Known gaps
  • CreateProfileJob/UpdateProfileJob's Configuration (ProfileConfiguration) and JobSample are stored as map[string]any pass-through rather than typed structs -- wire-compatible (arbitrary nested JSON round-trips byte-for-byte) but not validated; same for CreateRecipeJob/UpdateRecipeJob's DataCatalogOutputs/DatabaseOutputs. This mirrors the FormatOptions sub-fields deferral below and was a deliberate scope choice this pass (the fields are now at least threaded through and stored/returned, closing the actual data-loss gap; typed validation is a separate, lower-priority refinement -- bd: TODO file if prioritized).
  • StartProjectSession/SendProjectSessionAction (the interactive project-editor session flow) remain near-total no-ops beyond echoing Name -- acceptable since these model an interactive editing session tests don't poll for correctness, but flagged for completeness (unchanged from prior audit).
Deferred
  • CSV/Excel/Json FormatOptions sub-fields (e.g. Delimiter, HeaderRow, SheetNames) are passed through as map[string]any rather than typed structs -- wire-compatible (arbitrary nested JSON round-trips byte-for-byte) but not validated (unchanged from prior audit).

More

Documentation

Overview

Package databrew implements an in-memory AWS Glue DataBrew service backend.

Index

Constants

This section is empty.

Variables

View Source 
var (
	// ErrNotFound is returned when a requested resource does not exist.
	ErrNotFound = awserr.New(errCodeResourceNotFound, awserr.ErrNotFound)
	// ErrAlreadyExists is returned when a resource already exists.
	ErrAlreadyExists = awserr.New("ConflictException", awserr.ErrAlreadyExists)
	// ErrValidation is returned when input validation fails.
	ErrValidation = awserr.New(errCodeValidation, awserr.ErrInvalidParameter)
)
View Source 
var ErrNilAppContext = errors.New("nil AppContext passed to DataBrew Provider.Init")

ErrNilAppContext is returned by Init when a nil AppContext is passed.

Functions

This section is empty.

Types

type DataCatalogInput 

type DataCatalogInput struct {
	DatabaseName string `json:"DatabaseName"`
	TableName    string `json:"TableName"`
}

DataCatalogInput references a Glue Data Catalog table.

type DatabaseInput 

type DatabaseInput struct {
	GlueConnectionName string `json:"GlueConnectionName"`
	DatabaseTableName  string `json:"DatabaseTableName"`
}

DatabaseInput references a database table.

type Dataset 

type Dataset struct {
	PathOptions      *PathOptions         `json:"PathOptions,omitempty"`
	FormatOptions    DatasetFormatOptions `json:"FormatOptions,omitzero"`
	Input            DatasetInput         `json:"Input,omitzero"`
	Tags             map[string]string    `json:"Tags,omitempty"`
	Name             string               `json:"Name"`
	Arn              string               `json:"ResourceArn"`
	Format           string               `json:"Format,omitempty"`
	Source           string               `json:"Source,omitempty"`
	CreatedBy        string               `json:"CreatedBy,omitempty"`
	LastModifiedBy   string               `json:"LastModifiedBy,omitempty"`
	AccountID        string               `json:"AccountId,omitempty"`
	CreateDate       float64              `json:"CreateDate,omitempty"`
	LastModifiedDate float64              `json:"LastModifiedDate,omitempty"`
}

Dataset represents a DataBrew dataset. AccountID mirrors aws-sdk-go-v2/service/databrew/types.Dataset's AccountId member -- present on ListDatasets items (and harmlessly ignored by the real SDK's DescribeDataset deserializer, which has no AccountId case).

type DatasetFormatOptions 

type DatasetFormatOptions struct {
	Csv   map[string]any `json:"Csv,omitempty"`
	Excel map[string]any `json:"Excel,omitempty"`
	JSON  map[string]any `json:"Json,omitempty"`
}

DatasetFormatOptions holds format-specific options for a dataset.

The JSON field's wire key is "Json" (mixed case), NOT "JSON" -- confirmed against aws-sdk-go-v2/service/databrew's deserializer (awsRestjson1_deserializeDocumentFormatOptions switches on the exact, case-sensitive key "Json"). A response emitting the Go-idiomatic "JSON" falls through that switch's default case and the client silently drops the field, so a dataset created with JSON format options would appear to have none on describe/list.

type DatasetInput 

type DatasetInput struct {
	S3InputDefinition          *S3Location       `json:"S3InputDefinition,omitempty"`
	DataCatalogInputDefinition *DataCatalogInput `json:"DataCatalogInputDefinition,omitempty"`
	DatabaseInputDefinition    *DatabaseInput    `json:"DatabaseInputDefinition,omitempty"`
}

DatasetInput holds the data source for a dataset.

type DatasetParameter added in v1.2.0 

type DatasetParameter struct {
	DatetimeOptions *DatetimeOptions  `json:"DatetimeOptions,omitempty"`
	Filter          *FilterExpression `json:"Filter,omitempty"`
	Name            string            `json:"Name"`
	Type            string            `json:"Type"`
	CreateColumn    bool              `json:"CreateColumn,omitempty"`
}

DatasetParameter maps a name used in a dataset's Amazon S3 path to its definition.

type DatetimeOptions added in v1.2.0 

type DatetimeOptions struct {
	Format         string `json:"Format"`
	LocaleCode     string `json:"LocaleCode,omitempty"`
	TimezoneOffset string `json:"TimezoneOffset,omitempty"`
}

DatetimeOptions holds additional options for interpreting datetime parameters used in a dataset's Amazon S3 path.

type FilesLimit added in v1.2.0 

type FilesLimit struct {
	Order     string `json:"Order,omitempty"`
	OrderedBy string `json:"OrderedBy,omitempty"`
	MaxFiles  int    `json:"MaxFiles"`
}

FilesLimit imposes a limit on the number of Amazon S3 files selected for a dataset from a connected Amazon S3 path.

type FilterExpression added in v1.2.0 

type FilterExpression struct {
	ValuesMap  map[string]string `json:"ValuesMap"`
	Expression string            `json:"Expression"`
}

FilterExpression defines parameter-matching conditions (e.g. for dynamic dataset paths or datetime parameter filters).

type Handler 

type Handler struct {
	Backend   StorageBackend
	AccountID string
	Region    string
}

Handler is the HTTP handler for AWS Glue DataBrew operations.

func NewHandler 

func NewHandler(backend StorageBackend) *Handler

NewHandler creates a new DataBrew handler.

func (*Handler) ExtractOperation 

func (h *Handler) ExtractOperation(c *echo.Context) string

func (*Handler) ExtractResource 

func (h *Handler) ExtractResource(c *echo.Context) string

func (*Handler) GetSupportedOperations 

func (h *Handler) GetSupportedOperations() []string

func (*Handler) Handler 

func (h *Handler) Handler() echo.HandlerFunc

func (*Handler) MatchPriority 

func (h *Handler) MatchPriority() int

MatchPriority returns PriorityPathVersioned+1 so DataBrew is evaluated before IoT Analytics, which also claims /datasets at PriorityPathVersioned.

func (*Handler) Name 

func (h *Handler) Name() string

func (*Handler) Reset 

func (h *Handler) Reset()

func (*Handler) Restore 

func (h *Handler) Restore(ctx context.Context, data []byte) error

Restore implements persistence.Persistable by delegating to the backend.

func (*Handler) RouteMatcher 

func (h *Handler) RouteMatcher() service.Matcher

func (*Handler) Shutdown 

func (h *Handler) Shutdown(ctx context.Context)

Shutdown implements service.Shutdowner. It cancels in-flight job run transition goroutines and waits for them to drain, bounded by ctx.

func (*Handler) Snapshot 

func (h *Handler) Snapshot(ctx context.Context) []byte

Snapshot implements persistence.Persistable by delegating to the backend. Handler previously had no Snapshot/Restore of its own -- and neither did InMemoryBackend -- so cli.go's generic setupPersistence (which type-asserts the registered service.Registerable, i.e. the Handler, for a Snapshot/Restore pair) never picked DataBrew up at all: dead wiring, with no persistence underneath it either. This delegation (matching the codecommit/codepipeline/cleanrooms pattern) is what wires DataBrew into persistence for the first time.

func (*Handler) StartWorker 

func (h *Handler) StartWorker(_ context.Context) error

type InMemoryBackend 

type InMemoryBackend struct {
	// contains filtered or unexported fields
}

InMemoryBackend stores DataBrew state in memory.

All resource collections are nested by region (outer key = region) so that same-named resources are isolated across regions. datasets/recipes/ projects/jobs/rulesets/schedules each hold one *store.Table[T] per region, created lazily via the *Table accessors in store_setup.go — mirroring the lazy map creation the hand-rolled *Store helpers did before the Phase 3.3 pkgs/store conversion (see store_setup.go's package doc for why jobRuns is NOT converted). Callers must hold b.mu while accessing any of them.

func NewInMemoryBackend 

func NewInMemoryBackend(accountID, region string) *InMemoryBackend

NewInMemoryBackend creates a new in-memory DataBrew backend with a background lifecycle context.

func NewInMemoryBackendWithContext 

func NewInMemoryBackendWithContext(
	svcCtx context.Context,
	accountID, region string,
) *InMemoryBackend

NewInMemoryBackendWithContext creates a new in-memory DataBrew backend whose delayed lifecycle goroutines are tied to svcCtx. When svcCtx (or the backend's Shutdown) is cancelled, in-flight transition goroutines exit promptly. If svcCtx is nil, context.Background is used.

func (*InMemoryBackend) AccountID 

func (b *InMemoryBackend) AccountID() string

func (*InMemoryBackend) BatchDeleteRecipeVersion added in v1.2.0 

func (b *InMemoryBackend) BatchDeleteRecipeVersion(
	ctx context.Context,
	name string,
	toDelete []string,
) ([]RecipeVersionErrorDetail, error)

BatchDeleteRecipeVersion deletes multiple recipe versions at once. Unlike DeleteRecipeVersion, an individual version that doesn't exist (or is a LATEST_WORKING that can't yet be deleted) is a documented PARTIAL failure -- the overall call still succeeds and that version is reported in the returned []RecipeVersionErrorDetail -- whereas the recipe not existing at all, an empty/oversized/duplicate-containing list, or a syntactically invalid version identifier reject the WHOLE request (returned as an error), per aws-sdk-go-v2/service/databrew's BatchDeleteRecipeVersion doc comment.

func (*InMemoryBackend) CreateDataset 

func (b *InMemoryBackend) CreateDataset(
	ctx context.Context,
	name, format string,
	input DatasetInput,
	formatOpts DatasetFormatOptions,
	tags map[string]string,
	pathOptions *PathOptions,
) (*Dataset, error)

func (*InMemoryBackend) CreateJob 

func (b *InMemoryBackend) CreateJob(
	ctx context.Context,
	name, jobType, datasetName, projectName, recipeName, roleArn string,
	outputs []Output,
	tags map[string]string,
	extra JobExtras,
) (*Job, error)

func (*InMemoryBackend) CreateProject 

func (b *InMemoryBackend) CreateProject(
	ctx context.Context,
	name, datasetName, recipeName, roleArn string,
	sample Sample,
	tags map[string]string,
) (*Project, error)

func (*InMemoryBackend) CreateRecipe 

func (b *InMemoryBackend) CreateRecipe(
	ctx context.Context,
	name, description string,
	steps []RecipeStep,
	tags map[string]string,
) (*Recipe, error)

func (*InMemoryBackend) CreateRuleset 

func (b *InMemoryBackend) CreateRuleset(
	ctx context.Context,
	name, description, targetArn string,
	rules []Rule,
	tags map[string]string,
) (*Ruleset, error)

func (*InMemoryBackend) CreateSchedule 

func (b *InMemoryBackend) CreateSchedule(
	ctx context.Context,
	name string,
	jobNames []string,
	cron string,
	tags map[string]string,
) (*Schedule, error)

func (*InMemoryBackend) DeleteDataset 

func (b *InMemoryBackend) DeleteDataset(ctx context.Context, name string) error

func (*InMemoryBackend) DeleteJob 

func (b *InMemoryBackend) DeleteJob(ctx context.Context, name string) error

func (*InMemoryBackend) DeleteProject 

func (b *InMemoryBackend) DeleteProject(ctx context.Context, name string) error

func (*InMemoryBackend) DeleteRecipe 

func (b *InMemoryBackend) DeleteRecipe(ctx context.Context, name string) error

DeleteRecipe deletes a recipe's working draft and cascades to its entire published version history, so no orphaned recipeVersions rows survive the recipe itself.

func (*InMemoryBackend) DeleteRecipeVersion added in v1.2.0 

func (b *InMemoryBackend) DeleteRecipeVersion(ctx context.Context, name, version string) error

DeleteRecipeVersion deletes a single recipe version. version must be a numeric "X.Y" identifier or "LATEST_WORKING" (LATEST_PUBLISHED is rejected -- see isValidRecipeVersionID). Deleting LATEST_WORKING only succeeds if the recipe has no published versions (in which case the whole recipe, having no remaining version, is removed); this mirrors BatchDeleteRecipeVersion's documented constraint that LATEST_WORKING is only deleted if the recipe has no other versions.

func (*InMemoryBackend) DeleteRuleset 

func (b *InMemoryBackend) DeleteRuleset(ctx context.Context, name string) error

func (*InMemoryBackend) DeleteSchedule 

func (b *InMemoryBackend) DeleteSchedule(ctx context.Context, name string) error

func (*InMemoryBackend) DescribeDataset 

func (b *InMemoryBackend) DescribeDataset(ctx context.Context, name string) (*Dataset, error)

func (*InMemoryBackend) DescribeJob 

func (b *InMemoryBackend) DescribeJob(ctx context.Context, name string) (*Job, error)

func (*InMemoryBackend) DescribeJobRun 

func (b *InMemoryBackend) DescribeJobRun(ctx context.Context, name, runID string) (*JobRun, error)

func (*InMemoryBackend) DescribeProject 

func (b *InMemoryBackend) DescribeProject(ctx context.Context, name string) (*Project, error)

func (*InMemoryBackend) DescribeRecipe 

func (b *InMemoryBackend) DescribeRecipe(ctx context.Context, name, version string) (*Recipe, error)

DescribeRecipe returns the recipe version identified by version:

  • "" or "LATEST_PUBLISHED": the most recently published numeric version. If version is "" and the recipe has never been published (no numeric version exists yet), the working draft is returned instead -- the real API's documented default is "the latest published version", but DescribeRecipe on a freshly created, unpublished recipe (no version filter) is a common, supported call that must not 404. Passing "LATEST_PUBLISHED" explicitly does 404 in that case, matching the literal filter semantics.
  • "LATEST_WORKING": the current (possibly unpublished) draft.
  • a numeric "X.Y" version: that specific published snapshot.

func (*InMemoryBackend) DescribeRuleset 

func (b *InMemoryBackend) DescribeRuleset(ctx context.Context, name string) (*Ruleset, error)

func (*InMemoryBackend) DescribeSchedule 

func (b *InMemoryBackend) DescribeSchedule(ctx context.Context, name string) (*Schedule, error)

func (*InMemoryBackend) FindTagsByArn 

func (b *InMemoryBackend) FindTagsByArn(
	ctx context.Context,
	arnVal string,
) (map[string]string, error)

FindTagsByArn searches all resources in the request region for a specific ARN and returns its tags.

func (*InMemoryBackend) ListDatasets 

func (b *InMemoryBackend) ListDatasets(
	ctx context.Context,
	maxResults int,
	nextToken string,
) ([]*Dataset, string)

func (*InMemoryBackend) ListJobRuns 

func (b *InMemoryBackend) ListJobRuns(
	ctx context.Context,
	jobName string,
	maxResults int,
	nextToken string,
) ([]*JobRun, string, error)

func (*InMemoryBackend) ListJobs 

func (b *InMemoryBackend) ListJobs(
	ctx context.Context,
	maxResults int,
	nextToken,
	datasetName,
	projectName string,
) ([]*Job, string)

func (*InMemoryBackend) ListProjects 

func (b *InMemoryBackend) ListProjects(
	ctx context.Context,
	maxResults int,
	nextToken string,
) ([]*Project, string)

func (*InMemoryBackend) ListRecipeVersions added in v1.2.0 

func (b *InMemoryBackend) ListRecipeVersions(
	ctx context.Context,
	name string,
	maxResults int,
	nextToken string,
) ([]*Recipe, string, error)

ListRecipeVersions returns the published version history for name (never including LATEST_WORKING, matching the real op's doc comment: "Lists the versions of a particular DataBrew recipe, except for LATEST_WORKING"). The returned slice is never nil (an unpublished recipe returns an empty, non-nil slice) so callers marshal "Recipes":[] rather than "Recipes":null.

func (*InMemoryBackend) ListRecipes 

func (b *InMemoryBackend) ListRecipes(
	ctx context.Context,
	maxResults int,
	nextToken, versionFilter string,
) ([]*Recipe, string)

ListRecipes lists recipes. versionFilter mirrors ListRecipesInput's RecipeVersion field: "LATEST_WORKING" returns every recipe's working draft; "" or "LATEST_PUBLISHED" (the real API's documented default) returns only recipes that have at least one published version, each represented by its most recent published snapshot -- a never-published recipe does not appear in the default listing (confirmed against aws-sdk-go-v2/service/databrew/api_op_ListRecipes.go's ListRecipesInput doc comment: "If RecipeVersion is omitted, ListRecipes returns all of the LATEST_PUBLISHED recipe versions.").

func (*InMemoryBackend) ListRulesets 

func (b *InMemoryBackend) ListRulesets(
	ctx context.Context,
	maxResults int,
	nextToken, targetArn string,
) ([]*Ruleset, string)

func (*InMemoryBackend) ListSchedules 

func (b *InMemoryBackend) ListSchedules(
	ctx context.Context,
	maxResults int,
	nextToken string,
) ([]*Schedule, string)

func (*InMemoryBackend) PublishRecipe 

func (b *InMemoryBackend) PublishRecipe(ctx context.Context, name, description string) error

PublishRecipe snapshots the current working draft as a new numbered published version ("N.0", where N is the count of prior published versions plus one -- this backend does not model minor-version publishes).

func (*InMemoryBackend) Region 

func (b *InMemoryBackend) Region() string

func (*InMemoryBackend) Reset 

func (b *InMemoryBackend) Reset()

func (*InMemoryBackend) Restore 

func (b *InMemoryBackend) Restore(ctx context.Context, data []byte) error

Restore loads backend state from a JSON snapshot. It implements persistence.Persistable.

func (*InMemoryBackend) Shutdown 

func (b *InMemoryBackend) Shutdown(ctx context.Context)

Shutdown cancels the backend's lifecycle context and waits for in-flight delayed goroutines to finish, bounded by ctx. After Shutdown the backend no longer schedules state transitions.

func (*InMemoryBackend) Snapshot 

func (b *InMemoryBackend) Snapshot(ctx context.Context) []byte

Snapshot serialises the backend state to JSON. It implements persistence.Persistable.

func (*InMemoryBackend) StartJobRun 

func (b *InMemoryBackend) StartJobRun(ctx context.Context, jobName string) (*JobRun, error)

StartJobRun creates a new job run with STARTING state, transitioning to SUCCEEDED asynchronously.

func (*InMemoryBackend) StopJobRun 

func (b *InMemoryBackend) StopJobRun(ctx context.Context, name, runID string) (*JobRun, error)

func (*InMemoryBackend) UpdateDataset 

func (b *InMemoryBackend) UpdateDataset(
	ctx context.Context,
	name, format string,
	input DatasetInput,
	formatOpts DatasetFormatOptions,
	pathOptions *PathOptions,
) error

func (*InMemoryBackend) UpdateJob 

func (b *InMemoryBackend) UpdateJob(
	ctx context.Context,
	name, roleArn string,
	outputs []Output,
	maxCapacity, maxRetries, timeout int,
	extra JobExtras,
) error

func (*InMemoryBackend) UpdateProject 

func (b *InMemoryBackend) UpdateProject(
	ctx context.Context,
	name, roleArn string,
	sample Sample,
) error

UpdateProject modifies a project's RoleArn and Sample. DatasetName is deliberately NOT settable here: aws-sdk-go-v2/service/databrew's UpdateProjectInput has no DatasetName member (only Name/RoleArn/Sample) -- a project's dataset is fixed at creation and is not one of the documented updatable fields.

func (*InMemoryBackend) UpdateRecipe 

func (b *InMemoryBackend) UpdateRecipe(
	ctx context.Context,
	name, description string,
	steps []RecipeStep,
) error

UpdateRecipe modifies the definition of the LATEST_WORKING version only, matching aws-sdk-go-v2/service/databrew's UpdateRecipe doc comment; published version snapshots are immutable once created by PublishRecipe.

func (*InMemoryBackend) UpdateRuleset 

func (b *InMemoryBackend) UpdateRuleset(
	ctx context.Context,
	name, description string,
	rules []Rule,
) error

func (*InMemoryBackend) UpdateSchedule 

func (b *InMemoryBackend) UpdateSchedule(
	ctx context.Context,
	name string,
	jobNames []string,
	cron string,
) error

func (*InMemoryBackend) UpdateTagsByArn 

func (b *InMemoryBackend) UpdateTagsByArn(
	ctx context.Context,
	arnVal string,
	add map[string]string,
	remove []string,
) error

UpdateTagsByArn searches all resources in the request region and applies tags additions/removals.

type Job 

type Job struct {
	ProfileConfiguration     map[string]any    `json:"ProfileConfiguration,omitempty"`
	JobSample                map[string]any    `json:"JobSample,omitempty"`
	Tags                     map[string]string `json:"Tags,omitempty"`
	RecipeReference          *RecipeRef        `json:"RecipeReference,omitempty"`
	EncryptionMode           string            `json:"EncryptionMode,omitempty"`
	EncryptionKeyArn         string            `json:"EncryptionKeyArn,omitempty"`
	DatasetName              string            `json:"DatasetName,omitempty"`
	ProjectName              string            `json:"ProjectName,omitempty"`
	Name                     string            `json:"Name"`
	CreatedBy                string            `json:"CreatedBy,omitempty"`
	AccountID                string            `json:"AccountId,omitempty"`
	RecipeName               string            `json:"-"`
	RoleArn                  string            `json:"RoleArn,omitempty"`
	LogSubscription          string            `json:"LogSubscription,omitempty"`
	Type                     string            `json:"Type,omitempty"`
	LastModifiedBy           string            `json:"LastModifiedBy,omitempty"`
	Arn                      string            `json:"ResourceArn"`
	ValidationConfigurations []map[string]any  `json:"ValidationConfigurations,omitempty"`
	DataCatalogOutputs       []map[string]any  `json:"DataCatalogOutputs,omitempty"`
	DatabaseOutputs          []map[string]any  `json:"DatabaseOutputs,omitempty"`
	Outputs                  []Output          `json:"Outputs,omitempty"`
	Timeout                  int               `json:"Timeout,omitempty"`
	MaxRetries               int               `json:"MaxRetries,omitempty"`
	MaxCapacity              int               `json:"MaxCapacity,omitempty"`
	LastModifiedDate         float64           `json:"LastModifiedDate,omitempty"`
	CreateDate               float64           `json:"CreateDate,omitempty"`
}

Job represents a DataBrew job. AccountID mirrors aws-sdk-go-v2/service/databrew/types.Job's AccountId member -- see Dataset's AccountID doc comment for why it's safe to include on Describe responses too.

type JobExtras added in v1.2.0 

type JobExtras struct {
	ProfileConfiguration     map[string]any
	JobSample                map[string]any
	EncryptionMode           string
	EncryptionKeyArn         string
	LogSubscription          string
	DataCatalogOutputs       []map[string]any
	DatabaseOutputs          []map[string]any
	ValidationConfigurations []map[string]any
	MaxCapacity              int
	MaxRetries               int
	Timeout                  int
}

JobExtras bundles the optional job fields that are specific to one of the two job types (profile vs. recipe) but modeled on the shared Job entity, so CreateJob/UpdateJob's core positional signature doesn't grow further. ProfileConfiguration/JobSample/ValidationConfigurations are populated only by the profile-job handlers; DataCatalogOutputs/DatabaseOutputs only by the recipe-job handlers. EncryptionMode/EncryptionKeyArn/LogSubscription are accepted by both real job types. An unset (zero-value) field on UpdateJob leaves the corresponding Job field unchanged. MaxCapacity/ MaxRetries/Timeout are here (rather than positional, unlike UpdateJob) purely because CreateJob has no existing positional slots for them -- CreateProfileJobInput/CreateRecipeJobInput both accept all three but the pre-existing CreateJob signature silently dropped them.

type JobRun 

type JobRun struct {
	DatasetName   string  `json:"DatasetName,omitempty"`
	JobName       string  `json:"JobName"`
	RunID         string  `json:"RunId"`
	State         string  `json:"State"`
	LogGroupName  string  `json:"LogGroupName,omitempty"`
	StartedOn     float64 `json:"StartedOn,omitempty"`
	CompletedOn   float64 `json:"CompletedOn,omitempty"`
	ExecutionTime int     `json:"ExecutionTime,omitempty"`
}

JobRun represents a single execution of a DataBrew job.

type Output 

type Output struct {
	FormatOptions     map[string]any `json:"FormatOptions,omitempty"`
	Location          S3Location     `json:"Location,omitzero"`
	Format            string         `json:"Format,omitempty"`
	CompressionFormat string         `json:"CompressionFormat,omitempty"`
	PartitionColumns  []string       `json:"PartitionColumns,omitempty"`
	MaxOutputFiles    int            `json:"MaxOutputFiles,omitempty"`
	Overwrite         bool           `json:"Overwrite,omitempty"`
}

Output describes a DataBrew job output destination.

type PathOptions added in v1.2.0 

type PathOptions struct {
	FilesLimit                *FilesLimit                 `json:"FilesLimit,omitempty"`
	LastModifiedDateCondition *FilterExpression           `json:"LastModifiedDateCondition,omitempty"`
	Parameters                map[string]DatasetParameter `json:"Parameters,omitempty"`
}

PathOptions defines how DataBrew selects files for a given Amazon S3 path in a dataset (aws-sdk-go-v2/service/databrew/types.PathOptions).

type Project 

type Project struct {
	Tags             map[string]string `json:"Tags,omitempty"`
	Name             string            `json:"Name"`
	Arn              string            `json:"ResourceArn"`
	DatasetName      string            `json:"DatasetName,omitempty"`
	RecipeName       string            `json:"RecipeName"`
	RoleArn          string            `json:"RoleArn,omitempty"`
	SessionStatus    string            `json:"SessionStatus,omitempty"`
	CreatedBy        string            `json:"CreatedBy,omitempty"`
	LastModifiedBy   string            `json:"LastModifiedBy,omitempty"`
	AccountID        string            `json:"AccountId,omitempty"`
	Sample           Sample            `json:"Sample,omitzero"`
	CreateDate       float64           `json:"CreateDate,omitempty"`
	LastModifiedDate float64           `json:"LastModifiedDate,omitempty"`
}

Project represents a DataBrew project. AccountID mirrors aws-sdk-go-v2/service/databrew/types.Project's AccountId member -- see Dataset's AccountID doc comment for why it's safe to include on Describe responses too.

type Provider 

type Provider struct{}

Provider implements service.Provider for AWS Glue DataBrew.

func (*Provider) Init 

func (p *Provider) Init(ctx *service.AppContext) (service.Registerable, error)

Init initializes the DataBrew service backend and handler.

func (*Provider) Name 

func (p *Provider) Name() string

Name returns the provider name.

type Recipe 

type Recipe struct {
	Tags             map[string]string `json:"Tags,omitempty"`
	Name             string            `json:"Name"`
	Arn              string            `json:"ResourceArn"`
	Description      string            `json:"Description,omitempty"`
	PublishedBy      string            `json:"PublishedBy,omitempty"`
	RecipeVersion    string            `json:"RecipeVersion,omitempty"`
	CreatedBy        string            `json:"CreatedBy,omitempty"`
	LastModifiedBy   string            `json:"LastModifiedBy,omitempty"`
	Steps            []RecipeStep      `json:"Steps,omitempty"`
	PublishedDate    float64           `json:"PublishedDate,omitempty"`
	CreateDate       float64           `json:"CreateDate,omitempty"`
	LastModifiedDate float64           `json:"LastModifiedDate,omitempty"`
}

Recipe represents a DataBrew recipe.

type RecipeRef 

type RecipeRef struct {
	Name          string `json:"Name"`
	RecipeVersion string `json:"RecipeVersion,omitempty"`
}

RecipeRef holds a reference to a DataBrew recipe and optional version.

type RecipeStep 

type RecipeStep struct {
	Action               map[string]any   `json:"Action,omitempty"`
	ConditionExpressions []map[string]any `json:"ConditionExpressions,omitempty"`
}

RecipeStep is one transformation step in a recipe.

type RecipeVersionErrorDetail added in v1.2.0 

type RecipeVersionErrorDetail struct {
	RecipeVersion string `json:"RecipeVersion,omitempty"`
	ErrorCode     string `json:"ErrorCode,omitempty"`
	ErrorMessage  string `json:"ErrorMessage,omitempty"`
}

RecipeVersionErrorDetail describes a single recipe version's failure within a BatchDeleteRecipeVersion partial-failure response.

type Rule 

type Rule struct {
	SubstitutionMap map[string]string `json:"SubstitutionMap,omitempty"`
	Threshold       map[string]any    `json:"Threshold,omitempty"`
	Name            string            `json:"Name"`
	CheckExpression string            `json:"CheckExpression"`
	ColumnSelectors []map[string]any  `json:"ColumnSelectors,omitempty"`
	Disabled        bool              `json:"Disabled,omitempty"`
}

Rule represents a data quality rule.

type Ruleset 

type Ruleset struct {
	Tags             map[string]string `json:"Tags,omitempty"`
	Name             string            `json:"Name"`
	Arn              string            `json:"ResourceArn"`
	Description      string            `json:"Description,omitempty"`
	TargetArn        string            `json:"TargetArn"`
	CreatedBy        string            `json:"CreatedBy,omitempty"`
	LastModifiedBy   string            `json:"LastModifiedBy,omitempty"`
	AccountID        string            `json:"AccountId,omitempty"`
	Rules            []Rule            `json:"Rules"`
	RuleCount        int               `json:"RuleCount"`
	CreateDate       float64           `json:"CreateDate,omitempty"`
	LastModifiedDate float64           `json:"LastModifiedDate,omitempty"`
}

Ruleset represents a DataBrew data quality ruleset.

ListRulesets and DescribeRuleset use genuinely different wire shapes in the real SDK: DescribeRulesetOutput carries Rules (the full rule list, no AccountId/RuleCount), while ListRulesetsOutput.Rulesets is []types.RulesetItem (AccountId + RuleCount -- an integer count -- instead of the full Rules list; confirmed against awsRestjson1_deserializeDocumentRulesetItem, whose key switch has "RuleCount", not "Rules"). Rather than maintaining two marshal shapes, this type carries both Rules and RuleCount together: DescribeRuleset's real client ignores the extra RuleCount/AccountId keys it doesn't recognize, and ListRulesets' real client ignores the extra Rules key it doesn't recognize, so one shared struct is wire-safe both ways.

type S3Location 

type S3Location struct {
	Bucket string `json:"Bucket"`
	Key    string `json:"Key,omitempty"`
}

S3Location references an S3 path.

type Sample 

type Sample struct {
	Type string `json:"Type,omitempty"`
	Size int    `json:"Size,omitempty"`
}

Sample describes a data sample for a project.

type Schedule 

type Schedule struct {
	Tags             map[string]string `json:"Tags,omitempty"`
	Name             string            `json:"Name"`
	Arn              string            `json:"ResourceArn"`
	CronExpression   string            `json:"CronExpression"`
	CreatedBy        string            `json:"CreatedBy,omitempty"`
	LastModifiedBy   string            `json:"LastModifiedBy,omitempty"`
	AccountID        string            `json:"AccountId,omitempty"`
	JobNames         []string          `json:"JobNames,omitempty"`
	CreateDate       float64           `json:"CreateDate,omitempty"`
	LastModifiedDate float64           `json:"LastModifiedDate,omitempty"`
}

Schedule represents a DataBrew schedule. AccountID mirrors aws-sdk-go-v2/service/databrew/types.Schedule's AccountId member -- see Dataset's AccountID doc comment for why it's safe to include on Describe responses too.

type StorageBackend 

type StorageBackend interface {
	Region() string
	AccountID() string
	Reset()

	// Snapshot and Restore implement persistence.Persistable. Handler
	// delegates to them (see persistence.go) so cli.go's generic
	// setupPersistence picks DataBrew up.
	Snapshot(ctx context.Context) []byte
	Restore(ctx context.Context, data []byte) error

	// Dataset operations.
	CreateDataset(
		ctx context.Context,
		name, format string,
		input DatasetInput,
		formatOpts DatasetFormatOptions,
		tags map[string]string,
		pathOptions *PathOptions,
	) (*Dataset, error)
	DescribeDataset(ctx context.Context, name string) (*Dataset, error)
	ListDatasets(ctx context.Context, maxResults int, nextToken string) ([]*Dataset, string)
	UpdateDataset(
		ctx context.Context,
		name, format string,
		input DatasetInput,
		formatOpts DatasetFormatOptions,
		pathOptions *PathOptions,
	) error
	DeleteDataset(ctx context.Context, name string) error

	// Recipe operations.
	CreateRecipe(
		ctx context.Context,
		name, description string,
		steps []RecipeStep,
		tags map[string]string,
	) (*Recipe, error)
	DescribeRecipe(ctx context.Context, name, version string) (*Recipe, error)
	ListRecipes(ctx context.Context, maxResults int, nextToken, versionFilter string) ([]*Recipe, string)
	ListRecipeVersions(ctx context.Context, name string, maxResults int, nextToken string) ([]*Recipe, string, error)
	PublishRecipe(ctx context.Context, name, description string) error
	UpdateRecipe(ctx context.Context, name, description string, steps []RecipeStep) error
	DeleteRecipe(ctx context.Context, name string) error
	DeleteRecipeVersion(ctx context.Context, name, version string) error
	BatchDeleteRecipeVersion(ctx context.Context, name string, versions []string) ([]RecipeVersionErrorDetail, error)

	// Project operations.
	CreateProject(
		ctx context.Context,
		name, datasetName, recipeName, roleArn string,
		sample Sample,
		tags map[string]string,
	) (*Project, error)
	DescribeProject(ctx context.Context, name string) (*Project, error)
	ListProjects(ctx context.Context, maxResults int, nextToken string) ([]*Project, string)
	UpdateProject(ctx context.Context, name, roleArn string, sample Sample) error
	DeleteProject(ctx context.Context, name string) error

	// Job operations.
	CreateJob(
		ctx context.Context,
		name, jobType, datasetName, projectName, recipeName, roleArn string,
		outputs []Output,
		tags map[string]string,
		extra JobExtras,
	) (*Job, error)
	DescribeJob(ctx context.Context, name string) (*Job, error)
	ListJobs(ctx context.Context, maxResults int, nextToken, datasetName, projectName string) ([]*Job, string)
	UpdateJob(
		ctx context.Context,
		name, roleArn string,
		outputs []Output,
		maxCapacity, maxRetries, timeout int,
		extra JobExtras,
	) error
	DeleteJob(ctx context.Context, name string) error
	StartJobRun(ctx context.Context, jobName string) (*JobRun, error)
	ListJobRuns(
		ctx context.Context,
		jobName string,
		maxResults int,
		nextToken string,
	) ([]*JobRun, string, error)
	DescribeJobRun(ctx context.Context, name, runID string) (*JobRun, error)
	StopJobRun(ctx context.Context, name, runID string) (*JobRun, error)

	// Ruleset operations.
	CreateRuleset(
		ctx context.Context,
		name, description, targetArn string,
		rules []Rule,
		tags map[string]string,
	) (*Ruleset, error)
	DescribeRuleset(ctx context.Context, name string) (*Ruleset, error)
	ListRulesets(ctx context.Context, maxResults int, nextToken, targetArn string) ([]*Ruleset, string)
	UpdateRuleset(ctx context.Context, name, description string, rules []Rule) error
	DeleteRuleset(ctx context.Context, name string) error

	// Schedule operations.
	CreateSchedule(
		ctx context.Context,
		name string,
		jobNames []string,
		cron string,
		tags map[string]string,
	) (*Schedule, error)
	DescribeSchedule(ctx context.Context, name string) (*Schedule, error)
	ListSchedules(ctx context.Context, maxResults int, nextToken string) ([]*Schedule, string)
	UpdateSchedule(ctx context.Context, name string, jobNames []string, cron string) error
	DeleteSchedule(ctx context.Context, name string) error

	// Tag operations.
	FindTagsByArn(ctx context.Context, arn string) (map[string]string, error)
	UpdateTagsByArn(ctx context.Context, arn string, add map[string]string, remove []string) error
}

StorageBackend defines the interface for all DataBrew backend operations.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.