databrew

package
v1.4.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Aug 19, 2026 License: MIT Imports: 21 Imported by: 0

README

Glue DataBrew

Parity grade: A · SDK aws-sdk-go-v2/service/databrew@v1.42.4 · last audited 2026-07-31 (782e2a93)

Coverage

Metric Value
Operations audited 44 (44 ok)
Feature families 8 (8 ok)
Known gaps 4
Deferred items 0
Resource leaks clean
Known gaps
  • ProfileConfiguration (CreateProfileJob/UpdateProfileJob's Configuration field) remains map[string]any pass-through -- see families.job_extras_typing for the depth measurement behind that call. Wire-compatible (arbitrary nested JSON round-trips byte-for-byte) but not validated.
  • StartProjectSession/SendProjectSessionAction's interactive session lifecycle (view frames, recipe-step preview/apply) is not modeled -- structural, not a stub gap: there's no session state to be incomplete. What was fixable (rejecting a project name that doesn't exist) was fixed 2026-08-10; OpenDate was fixed 2026-08-15 (see families.session_status_fabrication).
  • Project.OpenedBy (real member) is never populated -- see families.session_status_fabrication. No caller-identity infrastructure exists anywhere in this package to derive it from (same root cause as CreatedBy/LastModifiedBy staying empty across every entity).
  • JobRun.ErrorMessage/StartedBy (real members) are never populated -- see families.jobrun_job_snapshot. ErrorMessage has no FAILED path to source a message from (StartJobRun always succeeds); StartedBy has the same no-identity-infrastructure root cause as OpenedBy above.

More

Documentation

Overview

Package databrew implements an in-memory AWS Glue DataBrew service backend.

Index

Constants

This section is empty.

Variables

View Source 
var (
	// ErrNotFound is returned when a requested resource does not exist.
	ErrNotFound = awserr.New(errCodeResourceNotFound, awserr.ErrNotFound)
	// ErrAlreadyExists is returned when a resource already exists.
	ErrAlreadyExists = awserr.New("ConflictException", awserr.ErrAlreadyExists)
	// ErrValidation is returned when input validation fails.
	ErrValidation = awserr.New(errCodeValidation, awserr.ErrInvalidParameter)
)
View Source 
var ErrNilAppContext = errors.New("nil AppContext passed to DataBrew Provider.Init")

ErrNilAppContext is returned by Init when a nil AppContext is passed.

Functions

This section is empty.

Types

type CsvOptions added in v1.3.1 

type CsvOptions struct {
	HeaderRow *bool  `json:"HeaderRow,omitempty"`
	Delimiter string `json:"Delimiter,omitempty"`
}

CsvOptions defines how DataBrew reads a CSV dataset input (aws-sdk-go-v2/service/databrew/types.CsvOptions).

type DataCatalogInput 

type DataCatalogInput struct {
	DatabaseName string `json:"DatabaseName"`
	TableName    string `json:"TableName"`
}

DataCatalogInput references a Glue Data Catalog table.

type DataCatalogOutput added in v1.3.1 

type DataCatalogOutput struct {
	DatabaseOptions *DatabaseTableOutputOptions `json:"DatabaseOptions,omitempty"`
	S3Options       *S3TableOutputOptions       `json:"S3Options,omitempty"`
	DatabaseName    string                      `json:"DatabaseName"`
	TableName       string                      `json:"TableName"`
	CatalogID       string                      `json:"CatalogId,omitempty"`
	Overwrite       bool                        `json:"Overwrite,omitempty"`
}

DataCatalogOutput represents where a recipe job writes Glue Data Catalog output (aws-sdk-go-v2/service/databrew/types.DataCatalogOutput).

type DatabaseInput 

type DatabaseInput struct {
	GlueConnectionName string `json:"GlueConnectionName"`
	DatabaseTableName  string `json:"DatabaseTableName"`
}

DatabaseInput references a database table.

type DatabaseOutput added in v1.3.1 

type DatabaseOutput struct {
	DatabaseOptions    *DatabaseTableOutputOptions `json:"DatabaseOptions"`
	GlueConnectionName string                      `json:"GlueConnectionName"`
	DatabaseOutputMode string                      `json:"DatabaseOutputMode,omitempty"`
}

DatabaseOutput represents a JDBC database output destination for a recipe job (aws-sdk-go-v2/service/databrew/types.DatabaseOutput).

type DatabaseTableOutputOptions added in v1.3.1 

type DatabaseTableOutputOptions struct {
	TempDirectory *S3Location `json:"TempDirectory,omitempty"`
	TableName     string      `json:"TableName"`
}

DatabaseTableOutputOptions specifies how and where a recipe job writes database output (aws-sdk-go-v2/service/databrew/types.DatabaseTableOutputOptions).

type Dataset 

type Dataset struct {
	PathOptions      *PathOptions         `json:"PathOptions,omitempty"`
	FormatOptions    DatasetFormatOptions `json:"FormatOptions,omitzero"`
	Input            DatasetInput         `json:"Input,omitzero"`
	Tags             map[string]string    `json:"Tags,omitempty"`
	Name             string               `json:"Name"`
	Arn              string               `json:"ResourceArn"`
	Format           string               `json:"Format,omitempty"`
	Source           string               `json:"Source,omitempty"`
	CreatedBy        string               `json:"CreatedBy,omitempty"`
	LastModifiedBy   string               `json:"LastModifiedBy,omitempty"`
	AccountID        string               `json:"AccountId,omitempty"`
	CreateDate       float64              `json:"CreateDate,omitempty"`
	LastModifiedDate float64              `json:"LastModifiedDate,omitempty"`
}

Dataset represents a DataBrew dataset. AccountID mirrors aws-sdk-go-v2/service/databrew/types.Dataset's AccountId member -- present on ListDatasets items only. DescribeDatasetOutput (api_op_DescribeDataset.go:39-88) has no AccountId member at all, so handleDescribeDataset clears it before marshaling; a raw-body or non-SDK caller would otherwise see a field the real API never sends.

type DatasetFormatOptions 

type DatasetFormatOptions struct {
	Csv   *CsvOptions   `json:"Csv,omitempty"`
	Excel *ExcelOptions `json:"Excel,omitempty"`
	JSON  *JSONOptions  `json:"Json,omitempty"`
}

DatasetFormatOptions holds format-specific options for a dataset.

The JSON field's wire key is "Json" (mixed case), NOT "JSON" -- confirmed against aws-sdk-go-v2/service/databrew's deserializer (awsRestjson1_deserializeDocumentFormatOptions switches on the exact, case-sensitive key "Json"). A response emitting the Go-idiomatic "JSON" falls through that switch's default case and the client silently drops the field, so a dataset created with JSON format options would appear to have none on describe/list.

type DatasetInput 

type DatasetInput struct {
	S3InputDefinition          *S3Location       `json:"S3InputDefinition,omitempty"`
	DataCatalogInputDefinition *DataCatalogInput `json:"DataCatalogInputDefinition,omitempty"`
	DatabaseInputDefinition    *DatabaseInput    `json:"DatabaseInputDefinition,omitempty"`
}

DatasetInput holds the data source for a dataset.

type DatasetParameter added in v1.2.0 

type DatasetParameter struct {
	DatetimeOptions *DatetimeOptions  `json:"DatetimeOptions,omitempty"`
	Filter          *FilterExpression `json:"Filter,omitempty"`
	Name            string            `json:"Name"`
	Type            string            `json:"Type"`
	CreateColumn    bool              `json:"CreateColumn,omitempty"`
}

DatasetParameter maps a name used in a dataset's Amazon S3 path to its definition.

type DatetimeOptions added in v1.2.0 

type DatetimeOptions struct {
	Format         string `json:"Format"`
	LocaleCode     string `json:"LocaleCode,omitempty"`
	TimezoneOffset string `json:"TimezoneOffset,omitempty"`
}

DatetimeOptions holds additional options for interpreting datetime parameters used in a dataset's Amazon S3 path.

type ExcelOptions added in v1.3.1 

type ExcelOptions struct {
	HeaderRow    *bool    `json:"HeaderRow,omitempty"`
	SheetIndexes []int32  `json:"SheetIndexes,omitempty"`
	SheetNames   []string `json:"SheetNames,omitempty"`
}

ExcelOptions defines how DataBrew reads an Excel dataset input (aws-sdk-go-v2/service/databrew/types.ExcelOptions).

type FilesLimit added in v1.2.0 

type FilesLimit struct {
	Order     string `json:"Order,omitempty"`
	OrderedBy string `json:"OrderedBy,omitempty"`
	MaxFiles  int    `json:"MaxFiles"`
}

FilesLimit imposes a limit on the number of Amazon S3 files selected for a dataset from a connected Amazon S3 path.

type FilterExpression added in v1.2.0 

type FilterExpression struct {
	ValuesMap  map[string]string `json:"ValuesMap"`
	Expression string            `json:"Expression"`
}

FilterExpression defines parameter-matching conditions (e.g. for dynamic dataset paths or datetime parameter filters).

type Handler 

type Handler struct {
	Backend   StorageBackend
	AccountID string
	Region    string
}

Handler is the HTTP handler for AWS Glue DataBrew operations.

func NewHandler 

func NewHandler(backend StorageBackend) *Handler

NewHandler creates a new DataBrew handler.

func (*Handler) ExtractOperation 

func (h *Handler) ExtractOperation(c *echo.Context) string

func (*Handler) ExtractResource 

func (h *Handler) ExtractResource(c *echo.Context) string

func (*Handler) GetSupportedOperations 

func (h *Handler) GetSupportedOperations() []string

func (*Handler) Handler 

func (h *Handler) Handler() echo.HandlerFunc

func (*Handler) MatchPriority 

func (h *Handler) MatchPriority() int

MatchPriority returns PriorityPathVersioned+1 so DataBrew is evaluated before IoT Analytics, which also claims /datasets at PriorityPathVersioned.

func (*Handler) Name 

func (h *Handler) Name() string

func (*Handler) Reset 

func (h *Handler) Reset()

func (*Handler) Restore 

func (h *Handler) Restore(ctx context.Context, data []byte) error

Restore implements persistence.Persistable by delegating to the backend.

func (*Handler) RouteMatcher 

func (h *Handler) RouteMatcher() service.Matcher

func (*Handler) Shutdown 

func (h *Handler) Shutdown(ctx context.Context)

Shutdown implements service.Shutdowner. It cancels in-flight job run transition goroutines and waits for them to drain, bounded by ctx.

func (*Handler) Snapshot 

func (h *Handler) Snapshot(ctx context.Context) []byte

Snapshot implements persistence.Persistable by delegating to the backend. Handler previously had no Snapshot/Restore of its own -- and neither did InMemoryBackend -- so cli.go's generic setupPersistence (which type-asserts the registered service.Registerable, i.e. the Handler, for a Snapshot/Restore pair) never picked DataBrew up at all: dead wiring, with no persistence underneath it either. This delegation (matching the codecommit/codepipeline/cleanrooms pattern) is what wires DataBrew into persistence for the first time.

func (*Handler) StartWorker 

func (h *Handler) StartWorker(_ context.Context) error

type InMemoryBackend 

type InMemoryBackend struct {
	// contains filtered or unexported fields
}

InMemoryBackend stores DataBrew state in memory.

All resource collections are nested by region (outer key = region) so that same-named resources are isolated across regions. datasets/recipes/ projects/jobs/rulesets/schedules each hold one *store.Table[T] per region, created lazily via the *Table accessors in store_setup.go — mirroring the lazy map creation the hand-rolled *Store helpers did before the Phase 3.3 pkgs/store conversion (see store_setup.go's package doc for why jobRuns is NOT converted). Callers must hold b.mu while accessing any of them.

func NewInMemoryBackend 

func NewInMemoryBackend(accountID, region string) *InMemoryBackend

NewInMemoryBackend creates a new in-memory DataBrew backend with a background lifecycle context.

func NewInMemoryBackendWithContext 

func NewInMemoryBackendWithContext(
	svcCtx context.Context,
	accountID, region string,
) *InMemoryBackend

NewInMemoryBackendWithContext creates a new in-memory DataBrew backend whose delayed lifecycle goroutines are tied to svcCtx. When svcCtx (or the backend's Shutdown) is cancelled, in-flight transition goroutines exit promptly. If svcCtx is nil, context.Background is used.

func (*InMemoryBackend) AccountID 

func (b *InMemoryBackend) AccountID() string

func (*InMemoryBackend) BatchDeleteRecipeVersion added in v1.2.0 

func (b *InMemoryBackend) BatchDeleteRecipeVersion(
	ctx context.Context,
	name string,
	toDelete []string,
) ([]RecipeVersionErrorDetail, error)

BatchDeleteRecipeVersion deletes multiple recipe versions at once. Unlike DeleteRecipeVersion, an individual version that doesn't exist (or is a LATEST_WORKING that can't yet be deleted) is a documented PARTIAL failure -- the overall call still succeeds and that version is reported in the returned []RecipeVersionErrorDetail -- whereas the recipe not existing at all, an empty/oversized/duplicate-containing list, or a syntactically invalid version identifier reject the WHOLE request (returned as an error), per aws-sdk-go-v2/service/databrew's BatchDeleteRecipeVersion doc comment.

func (*InMemoryBackend) CreateDataset 

func (b *InMemoryBackend) CreateDataset(
	ctx context.Context,
	name, format string,
	input DatasetInput,
	formatOpts DatasetFormatOptions,
	tags map[string]string,
	pathOptions *PathOptions,
) (*Dataset, error)

func (*InMemoryBackend) CreateJob 

func (b *InMemoryBackend) CreateJob(
	ctx context.Context,
	name, jobType, datasetName, projectName, recipeName, roleArn string,
	outputs []Output,
	tags map[string]string,
	extra JobExtras,
) (*Job, error)

func (*InMemoryBackend) CreateProject 

func (b *InMemoryBackend) CreateProject(
	ctx context.Context,
	name, datasetName, recipeName, roleArn string,
	sample Sample,
	tags map[string]string,
) (*Project, error)

func (*InMemoryBackend) CreateRecipe 

func (b *InMemoryBackend) CreateRecipe(
	ctx context.Context,
	name, description string,
	steps []RecipeStep,
	tags map[string]string,
) (*Recipe, error)

func (*InMemoryBackend) CreateRuleset 

func (b *InMemoryBackend) CreateRuleset(
	ctx context.Context,
	name, description, targetArn string,
	rules []Rule,
	tags map[string]string,
) (*Ruleset, error)

func (*InMemoryBackend) CreateSchedule 

func (b *InMemoryBackend) CreateSchedule(
	ctx context.Context,
	name string,
	jobNames []string,
	cron string,
	tags map[string]string,
) (*Schedule, error)

func (*InMemoryBackend) DeleteDataset 

func (b *InMemoryBackend) DeleteDataset(ctx context.Context, name string) error

func (*InMemoryBackend) DeleteJob 

func (b *InMemoryBackend) DeleteJob(ctx context.Context, name string) error

func (*InMemoryBackend) DeleteProject 

func (b *InMemoryBackend) DeleteProject(ctx context.Context, name string) error

func (*InMemoryBackend) DeleteRecipe 

func (b *InMemoryBackend) DeleteRecipe(ctx context.Context, name string) error

DeleteRecipe deletes a recipe's working draft and cascades to its entire published version history, so no orphaned recipeVersions rows survive the recipe itself.

func (*InMemoryBackend) DeleteRecipeVersion added in v1.2.0 

func (b *InMemoryBackend) DeleteRecipeVersion(ctx context.Context, name, version string) error

DeleteRecipeVersion deletes a single recipe version. version must be a numeric "X.Y" identifier or "LATEST_WORKING" (LATEST_PUBLISHED is rejected -- see isValidRecipeVersionID). Deleting LATEST_WORKING only succeeds if the recipe has no published versions (in which case the whole recipe, having no remaining version, is removed); this mirrors BatchDeleteRecipeVersion's documented constraint that LATEST_WORKING is only deleted if the recipe has no other versions.

func (*InMemoryBackend) DeleteRuleset 

func (b *InMemoryBackend) DeleteRuleset(ctx context.Context, name string) error

func (*InMemoryBackend) DeleteSchedule 

func (b *InMemoryBackend) DeleteSchedule(ctx context.Context, name string) error

func (*InMemoryBackend) DescribeDataset 

func (b *InMemoryBackend) DescribeDataset(ctx context.Context, name string) (*Dataset, error)

func (*InMemoryBackend) DescribeJob 

func (b *InMemoryBackend) DescribeJob(ctx context.Context, name string) (*Job, error)

func (*InMemoryBackend) DescribeJobRun 

func (b *InMemoryBackend) DescribeJobRun(ctx context.Context, name, runID string) (*JobRun, error)

func (*InMemoryBackend) DescribeProject 

func (b *InMemoryBackend) DescribeProject(ctx context.Context, name string) (*Project, error)

func (*InMemoryBackend) DescribeRecipe 

func (b *InMemoryBackend) DescribeRecipe(ctx context.Context, name, version string) (*Recipe, error)

DescribeRecipe returns the recipe version identified by version:

  • "" or "LATEST_PUBLISHED": the most recently published numeric version. If version is "" and the recipe has never been published (no numeric version exists yet), the working draft is returned instead -- the real API's documented default is "the latest published version", but DescribeRecipe on a freshly created, unpublished recipe (no version filter) is a common, supported call that must not 404. Passing "LATEST_PUBLISHED" explicitly does 404 in that case, matching the literal filter semantics.
  • "LATEST_WORKING": the current (possibly unpublished) draft.
  • a numeric "X.Y" version: that specific published snapshot.

func (*InMemoryBackend) DescribeRuleset 

func (b *InMemoryBackend) DescribeRuleset(ctx context.Context, name string) (*Ruleset, error)

func (*InMemoryBackend) DescribeSchedule 

func (b *InMemoryBackend) DescribeSchedule(ctx context.Context, name string) (*Schedule, error)

func (*InMemoryBackend) FindTagsByArn 

func (b *InMemoryBackend) FindTagsByArn(
	ctx context.Context,
	arnVal string,
) (map[string]string, error)

FindTagsByArn searches all resources in the request region for a specific ARN and returns its tags.

func (*InMemoryBackend) ListDatasets 

func (b *InMemoryBackend) ListDatasets(
	ctx context.Context,
	maxResults int,
	nextToken string,
) ([]*Dataset, string)

func (*InMemoryBackend) ListJobRuns 

func (b *InMemoryBackend) ListJobRuns(
	ctx context.Context,
	jobName string,
	maxResults int,
	nextToken string,
) ([]*JobRun, string, error)

func (*InMemoryBackend) ListJobs 

func (b *InMemoryBackend) ListJobs(
	ctx context.Context,
	maxResults int,
	nextToken,
	datasetName,
	projectName string,
) ([]*Job, string)

func (*InMemoryBackend) ListProjects 

func (b *InMemoryBackend) ListProjects(
	ctx context.Context,
	maxResults int,
	nextToken string,
) ([]*Project, string)

func (*InMemoryBackend) ListRecipeVersions added in v1.2.0 

func (b *InMemoryBackend) ListRecipeVersions(
	ctx context.Context,
	name string,
	maxResults int,
	nextToken string,
) ([]*Recipe, string, error)

ListRecipeVersions returns the published version history for name (never including LATEST_WORKING, matching the real op's doc comment: "Lists the versions of a particular DataBrew recipe, except for LATEST_WORKING"). The returned slice is never nil (an unpublished recipe returns an empty, non-nil slice) so callers marshal "Recipes":[] rather than "Recipes":null.

func (*InMemoryBackend) ListRecipes 

func (b *InMemoryBackend) ListRecipes(
	ctx context.Context,
	maxResults int,
	nextToken, versionFilter string,
) ([]*Recipe, string)

ListRecipes lists recipes. versionFilter mirrors ListRecipesInput's RecipeVersion field: "LATEST_WORKING" returns every recipe's working draft; "" or "LATEST_PUBLISHED" (the real API's documented default) returns only recipes that have at least one published version, each represented by its most recent published snapshot -- a never-published recipe does not appear in the default listing (confirmed against aws-sdk-go-v2/service/databrew/api_op_ListRecipes.go's ListRecipesInput doc comment: "If RecipeVersion is omitted, ListRecipes returns all of the LATEST_PUBLISHED recipe versions.").

func (*InMemoryBackend) ListRulesets 

func (b *InMemoryBackend) ListRulesets(
	ctx context.Context,
	maxResults int,
	nextToken, targetArn string,
) ([]*Ruleset, string)

func (*InMemoryBackend) ListSchedules 

func (b *InMemoryBackend) ListSchedules(
	ctx context.Context,
	maxResults int,
	nextToken string,
) ([]*Schedule, string)

func (*InMemoryBackend) OpenProjectSession added in v1.3.1 

func (b *InMemoryBackend) OpenProjectSession(ctx context.Context, name string) (*Project, error)

OpenProjectSession records that a project session was started against name, setting OpenDate (a real types.Project member, deserializers.go's awsRestjson1_deserializeDocumentProject case "OpenDate") to now. Real AWS sets it when a working session is opened via StartProjectSession, the only such backend event this in-memory emulator has.

func (*InMemoryBackend) PublishRecipe 

func (b *InMemoryBackend) PublishRecipe(ctx context.Context, name, description string) error

PublishRecipe snapshots the current working draft as a new numbered published version ("N.0", where N is the count of prior published versions plus one -- this backend does not model minor-version publishes).

func (*InMemoryBackend) Region 

func (b *InMemoryBackend) Region() string

func (*InMemoryBackend) Reset 

func (b *InMemoryBackend) Reset()

func (*InMemoryBackend) Restore 

func (b *InMemoryBackend) Restore(ctx context.Context, data []byte) error

Restore loads backend state from a JSON snapshot. It implements persistence.Persistable.

func (*InMemoryBackend) Shutdown 

func (b *InMemoryBackend) Shutdown(ctx context.Context)

Shutdown cancels the backend's lifecycle context and waits for in-flight delayed goroutines to finish, bounded by ctx. After Shutdown the backend no longer schedules state transitions.

func (*InMemoryBackend) Snapshot 

func (b *InMemoryBackend) Snapshot(ctx context.Context) []byte

Snapshot serialises the backend state to JSON. It implements persistence.Persistable.

func (*InMemoryBackend) StartJobRun 

func (b *InMemoryBackend) StartJobRun(ctx context.Context, jobName string) (*JobRun, error)

StartJobRun creates a new job run with STARTING state, transitioning to SUCCEEDED asynchronously.

func (*InMemoryBackend) StopJobRun 

func (b *InMemoryBackend) StopJobRun(ctx context.Context, name, runID string) (*JobRun, error)

func (*InMemoryBackend) UpdateDataset 

func (b *InMemoryBackend) UpdateDataset(
	ctx context.Context,
	name, format string,
	input DatasetInput,
	formatOpts DatasetFormatOptions,
	pathOptions *PathOptions,
) error

func (*InMemoryBackend) UpdateJob 

func (b *InMemoryBackend) UpdateJob(
	ctx context.Context,
	name, roleArn string,
	outputs []Output,
	maxCapacity, maxRetries, timeout int,
	extra JobExtras,
) error

func (*InMemoryBackend) UpdateProject 

func (b *InMemoryBackend) UpdateProject(
	ctx context.Context,
	name, roleArn string,
	sample Sample,
) error

UpdateProject modifies a project's RoleArn and Sample. DatasetName is deliberately NOT settable here: aws-sdk-go-v2/service/databrew's UpdateProjectInput has no DatasetName member (only Name/RoleArn/Sample) -- a project's dataset is fixed at creation and is not one of the documented updatable fields.

func (*InMemoryBackend) UpdateRecipe 

func (b *InMemoryBackend) UpdateRecipe(
	ctx context.Context,
	name, description string,
	steps []RecipeStep,
) error

UpdateRecipe modifies the definition of the LATEST_WORKING version only, matching aws-sdk-go-v2/service/databrew's UpdateRecipe doc comment; published version snapshots are immutable once created by PublishRecipe.

func (*InMemoryBackend) UpdateRuleset 

func (b *InMemoryBackend) UpdateRuleset(
	ctx context.Context,
	name, description string,
	rules []Rule,
) error

func (*InMemoryBackend) UpdateSchedule 

func (b *InMemoryBackend) UpdateSchedule(
	ctx context.Context,
	name string,
	jobNames []string,
	cron string,
) error

func (*InMemoryBackend) UpdateTagsByArn 

func (b *InMemoryBackend) UpdateTagsByArn(
	ctx context.Context,
	arnVal string,
	add map[string]string,
	remove []string,
) error

UpdateTagsByArn searches all resources in the request region and applies tags additions/removals.

type JSONOptions added in v1.3.1 

type JSONOptions struct {
	MultiLine bool `json:"MultiLine,omitempty"`
}

JSONOptions defines how DataBrew reads a JSON dataset input (aws-sdk-go-v2/service/databrew/types.JSONOptions).

type Job 

type Job struct {
	// ProfileConfiguration is left untyped: it nests 4 levels deep
	// (ProfileConfiguration -> ColumnStatisticsConfigurations ->
	// Statistics(StatisticsConfiguration) -> Overrides([]StatisticOverride)),
	// spans 6 distinct struct shapes, and carries two independent
	// list-of-struct branches (column overrides and entity-detector
	// AllowedStatistics) -- deep enough that a partial model risks silently
	// dropping fields a client can't tell were never implemented.
	ProfileConfiguration     map[string]any      `json:"ProfileConfiguration,omitempty"`
	JobSample                *JobSample          `json:"JobSample,omitempty"`
	Tags                     map[string]string   `json:"Tags,omitempty"`
	RecipeReference          *RecipeRef          `json:"RecipeReference,omitempty"`
	EncryptionMode           string              `json:"EncryptionMode,omitempty"`
	EncryptionKeyArn         string              `json:"EncryptionKeyArn,omitempty"`
	DatasetName              string              `json:"DatasetName,omitempty"`
	ProjectName              string              `json:"ProjectName,omitempty"`
	Name                     string              `json:"Name"`
	CreatedBy                string              `json:"CreatedBy,omitempty"`
	AccountID                string              `json:"AccountId,omitempty"`
	RecipeName               string              `json:"-"`
	RoleArn                  string              `json:"RoleArn,omitempty"`
	LogSubscription          string              `json:"LogSubscription,omitempty"`
	Type                     string              `json:"Type,omitempty"`
	LastModifiedBy           string              `json:"LastModifiedBy,omitempty"`
	Arn                      string              `json:"ResourceArn"`
	ValidationConfigurations []map[string]any    `json:"ValidationConfigurations,omitempty"`
	DataCatalogOutputs       []DataCatalogOutput `json:"DataCatalogOutputs,omitempty"`
	DatabaseOutputs          []DatabaseOutput    `json:"DatabaseOutputs,omitempty"`
	Outputs                  []Output            `json:"Outputs,omitempty"`
	Timeout                  int                 `json:"Timeout,omitempty"`
	MaxRetries               int                 `json:"MaxRetries,omitempty"`
	MaxCapacity              int                 `json:"MaxCapacity,omitempty"`
	LastModifiedDate         float64             `json:"LastModifiedDate,omitempty"`
	CreateDate               float64             `json:"CreateDate,omitempty"`
}

Job represents a DataBrew job. AccountID mirrors aws-sdk-go-v2/service/databrew/types.Job's AccountId member -- see Dataset's AccountID doc comment; DescribeJobOutput (api_op_DescribeJob.go:39+) has no AccountId member either.

type JobExtras added in v1.2.0 

type JobExtras struct {
	// ProfileConfiguration stays untyped -- see Job.ProfileConfiguration's
	// doc comment.
	ProfileConfiguration     map[string]any
	JobSample                *JobSample
	EncryptionMode           string
	EncryptionKeyArn         string
	LogSubscription          string
	DataCatalogOutputs       []DataCatalogOutput
	DatabaseOutputs          []DatabaseOutput
	ValidationConfigurations []map[string]any
	MaxCapacity              int
	MaxRetries               int
	Timeout                  int
}

JobExtras bundles the optional job fields that are specific to one of the two job types (profile vs. recipe) but modeled on the shared Job entity, so CreateJob/UpdateJob's core positional signature doesn't grow further. ProfileConfiguration/JobSample/ValidationConfigurations are populated only by the profile-job handlers; DataCatalogOutputs/DatabaseOutputs only by the recipe-job handlers. EncryptionMode/EncryptionKeyArn/LogSubscription are accepted by both real job types. An unset (zero-value) field on UpdateJob leaves the corresponding Job field unchanged. MaxCapacity/ MaxRetries/Timeout are here (rather than positional, unlike UpdateJob) purely because CreateJob has no existing positional slots for them -- CreateProfileJobInput/CreateRecipeJobInput both accept all three but the pre-existing CreateJob signature silently dropped them.

type JobRun 

type JobRun struct {
	RecipeReference    *RecipeRef          `json:"RecipeReference,omitempty"`
	JobSample          *JobSample          `json:"JobSample,omitempty"`
	DatasetName        string              `json:"DatasetName,omitempty"`
	JobName            string              `json:"JobName"`
	RunID              string              `json:"RunId"`
	State              string              `json:"State"`
	LogGroupName       string              `json:"LogGroupName,omitempty"`
	LogSubscription    string              `json:"LogSubscription,omitempty"`
	ErrorMessage       string              `json:"ErrorMessage,omitempty"`
	StartedBy          string              `json:"StartedBy,omitempty"`
	DataCatalogOutputs []DataCatalogOutput `json:"DataCatalogOutputs,omitempty"`
	DatabaseOutputs    []DatabaseOutput    `json:"DatabaseOutputs,omitempty"`
	Outputs            []Output            `json:"Outputs,omitempty"`
	StartedOn          float64             `json:"StartedOn,omitempty"`
	CompletedOn        float64             `json:"CompletedOn,omitempty"`
	ExecutionTime      int                 `json:"ExecutionTime,omitempty"`
	Attempt            int                 `json:"Attempt,omitempty"`
}

JobRun represents a single execution of a DataBrew job. Attempt/ DataCatalogOutputs/DatabaseOutputs/JobSample/LogSubscription/Outputs/ RecipeReference mirror real types.JobRun members (deserializers.go's awsRestjson1_deserializeDocumentJobRun) that were previously never emitted at all; StartJobRun now snapshots them from the parent Job, the only backend state they could come from. ErrorMessage/StartedBy are also real members, left always-unpopulated and disclosed in PARITY.md: this backend's StartJobRun always transitions STARTING->SUCCEEDED (see jobRunTransitionDelay) with no FAILED path to source an error message from, and, like CreatedBy/LastModifiedBy elsewhere in this package, there is no caller-identity infrastructure to derive StartedBy from.

type JobSample added in v1.3.1 

type JobSample struct {
	Mode string `json:"Mode,omitempty"`
	Size int64  `json:"Size,omitempty"`
}

JobSample determines how many rows a profile job processes (aws-sdk-go-v2/service/databrew/types.JobSample).

type Output 

type Output struct {
	FormatOptions     map[string]any `json:"FormatOptions,omitempty"`
	Location          S3Location     `json:"Location,omitzero"`
	Format            string         `json:"Format,omitempty"`
	CompressionFormat string         `json:"CompressionFormat,omitempty"`
	PartitionColumns  []string       `json:"PartitionColumns,omitempty"`
	MaxOutputFiles    int            `json:"MaxOutputFiles,omitempty"`
	Overwrite         bool           `json:"Overwrite,omitempty"`
}

Output describes a DataBrew job output destination.

type PathOptions added in v1.2.0 

type PathOptions struct {
	FilesLimit                *FilesLimit                 `json:"FilesLimit,omitempty"`
	LastModifiedDateCondition *FilterExpression           `json:"LastModifiedDateCondition,omitempty"`
	Parameters                map[string]DatasetParameter `json:"Parameters,omitempty"`
}

PathOptions defines how DataBrew selects files for a given Amazon S3 path in a dataset (aws-sdk-go-v2/service/databrew/types.PathOptions).

type Project 

type Project struct {
	Tags             map[string]string `json:"Tags,omitempty"`
	Name             string            `json:"Name"`
	Arn              string            `json:"ResourceArn"`
	DatasetName      string            `json:"DatasetName,omitempty"`
	RecipeName       string            `json:"RecipeName"`
	RoleArn          string            `json:"RoleArn,omitempty"`
	CreatedBy        string            `json:"CreatedBy,omitempty"`
	LastModifiedBy   string            `json:"LastModifiedBy,omitempty"`
	AccountID        string            `json:"AccountId,omitempty"`
	OpenedBy         string            `json:"OpenedBy,omitempty"`
	Sample           Sample            `json:"Sample,omitzero"`
	CreateDate       float64           `json:"CreateDate,omitempty"`
	LastModifiedDate float64           `json:"LastModifiedDate,omitempty"`
	OpenDate         float64           `json:"OpenDate,omitempty"`
}

Project represents a DataBrew project. AccountID mirrors aws-sdk-go-v2/service/databrew/types.Project's AccountId member -- see Dataset's AccountID doc comment; DescribeProjectOutput (api_op_DescribeProject.go:39-97) has no AccountId member either.

OpenDate/OpenedBy mirror types.Project's real members (deserializers.go's awsRestjson1_deserializeDocumentProject, cases "OpenDate"/"OpenedBy") -- there is no "SessionStatus" member on the real type at all (confirmed against that same deserializer's full case list: no such key exists), so the field this struct previously carried under that name was fabricated, a real API caller never sends it, and it has been removed. OpenDate is set by StartProjectSession, the real trigger for it (see InMemoryBackend.OpenProjectSession). OpenedBy stays unpopulated -- like CreatedBy/LastModifiedBy above, this backend has no caller-identity infrastructure to derive it from.

type Provider 

type Provider struct{}

Provider implements service.Provider for AWS Glue DataBrew.

func (*Provider) Init 

func (p *Provider) Init(ctx *service.AppContext) (service.Registerable, error)

Init initializes the DataBrew service backend and handler.

func (*Provider) Name 

func (p *Provider) Name() string

Name returns the provider name.

type Recipe 

type Recipe struct {
	Tags             map[string]string `json:"Tags,omitempty"`
	Name             string            `json:"Name"`
	Arn              string            `json:"ResourceArn"`
	Description      string            `json:"Description,omitempty"`
	ProjectName      string            `json:"ProjectName,omitempty"`
	PublishedBy      string            `json:"PublishedBy,omitempty"`
	RecipeVersion    string            `json:"RecipeVersion,omitempty"`
	CreatedBy        string            `json:"CreatedBy,omitempty"`
	LastModifiedBy   string            `json:"LastModifiedBy,omitempty"`
	Steps            []RecipeStep      `json:"Steps,omitempty"`
	PublishedDate    float64           `json:"PublishedDate,omitempty"`
	CreateDate       float64           `json:"CreateDate,omitempty"`
	LastModifiedDate float64           `json:"LastModifiedDate,omitempty"`
}

Recipe represents a DataBrew recipe. ProjectName mirrors aws-sdk-go-v2/service/databrew/types.Recipe's ProjectName member (deserializers.go's awsRestjson1_deserializeDocumentRecipe, case "ProjectName") -- this backend does not store the association on the recipe itself, so it is derived at read time from the reverse link CreateProject already stores (Project.RecipeName); see InMemoryBackend.recipeProjectName.

type RecipeRef 

type RecipeRef struct {
	Name          string `json:"Name"`
	RecipeVersion string `json:"RecipeVersion,omitempty"`
}

RecipeRef holds a reference to a DataBrew recipe and optional version.

type RecipeStep 

type RecipeStep struct {
	Action               map[string]any   `json:"Action,omitempty"`
	ConditionExpressions []map[string]any `json:"ConditionExpressions,omitempty"`
}

RecipeStep is one transformation step in a recipe.

type RecipeVersionErrorDetail added in v1.2.0 

type RecipeVersionErrorDetail struct {
	RecipeVersion string `json:"RecipeVersion,omitempty"`
	ErrorCode     string `json:"ErrorCode,omitempty"`
	ErrorMessage  string `json:"ErrorMessage,omitempty"`
}

RecipeVersionErrorDetail describes a single recipe version's failure within a BatchDeleteRecipeVersion partial-failure response.

type Rule 

type Rule struct {
	SubstitutionMap map[string]string `json:"SubstitutionMap,omitempty"`
	Threshold       map[string]any    `json:"Threshold,omitempty"`
	Name            string            `json:"Name"`
	CheckExpression string            `json:"CheckExpression"`
	ColumnSelectors []map[string]any  `json:"ColumnSelectors,omitempty"`
	Disabled        bool              `json:"Disabled,omitempty"`
}

Rule represents a data quality rule.

type Ruleset 

type Ruleset struct {
	Tags             map[string]string `json:"Tags,omitempty"`
	Name             string            `json:"Name"`
	Arn              string            `json:"ResourceArn"`
	Description      string            `json:"Description,omitempty"`
	TargetArn        string            `json:"TargetArn"`
	CreatedBy        string            `json:"CreatedBy,omitempty"`
	LastModifiedBy   string            `json:"LastModifiedBy,omitempty"`
	AccountID        string            `json:"AccountId,omitempty"`
	Rules            []Rule            `json:"Rules"`
	RuleCount        int               `json:"RuleCount"`
	CreateDate       float64           `json:"CreateDate,omitempty"`
	LastModifiedDate float64           `json:"LastModifiedDate,omitempty"`
}

Ruleset is the internal storage representation of a DataBrew data quality ruleset. It is never marshaled directly.

ListRulesets and DescribeRuleset use genuinely different wire shapes in the real SDK: DescribeRulesetOutput (api_op_DescribeRuleset.go:39-77) carries Rules (the full rule list), no AccountId/RuleCount at all, while ListRulesetsOutput.Rulesets is []types.RulesetItem (types.go:1020) -- AccountId + RuleCount (an integer count), no Rules field at all (confirmed against awsRestjson1_deserializeDocumentRulesetItem, deserializers.go:11521, whose key switch has "RuleCount", not "Rules"). A real client silently ignores unrecognized keys, but a raw-body or non-SDK caller reading DescribeRuleset's response would see a fabricated AccountId/RuleCount, and one reading ListRulesets would see every ruleset's full rule text leaked even though real ListRulesets never sends it. newRulesetDescribeView and newRulesetListItem below project this type into each op's real shape.

type RulesetDescribeView added in v1.3.1 

type RulesetDescribeView struct {
	Tags             map[string]string `json:"Tags,omitempty"`
	Name             string            `json:"Name"`
	Arn              string            `json:"ResourceArn"`
	Description      string            `json:"Description,omitempty"`
	TargetArn        string            `json:"TargetArn"`
	CreatedBy        string            `json:"CreatedBy,omitempty"`
	LastModifiedBy   string            `json:"LastModifiedBy,omitempty"`
	Rules            []Rule            `json:"Rules"`
	CreateDate       float64           `json:"CreateDate,omitempty"`
	LastModifiedDate float64           `json:"LastModifiedDate,omitempty"`
}

RulesetDescribeView is the wire shape for DescribeRulesetOutput (api_op_DescribeRuleset.go:39-77): Rules, no AccountId/RuleCount.

type RulesetListItem added in v1.3.1 

type RulesetListItem struct {
	Tags             map[string]string `json:"Tags,omitempty"`
	Name             string            `json:"Name"`
	Arn              string            `json:"ResourceArn"`
	Description      string            `json:"Description,omitempty"`
	TargetArn        string            `json:"TargetArn"`
	CreatedBy        string            `json:"CreatedBy,omitempty"`
	LastModifiedBy   string            `json:"LastModifiedBy,omitempty"`
	AccountID        string            `json:"AccountId,omitempty"`
	RuleCount        int               `json:"RuleCount"`
	CreateDate       float64           `json:"CreateDate,omitempty"`
	LastModifiedDate float64           `json:"LastModifiedDate,omitempty"`
}

RulesetListItem is the wire shape for ListRulesetsOutput.Rulesets (types.RulesetItem, types/types.go:1020): AccountId + RuleCount, no Rules.

type S3Location 

type S3Location struct {
	Bucket      string `json:"Bucket"`
	Key         string `json:"Key,omitempty"`
	BucketOwner string `json:"BucketOwner,omitempty"`
}

S3Location references an S3 path.

type S3TableOutputOptions added in v1.3.1 

type S3TableOutputOptions struct {
	Location S3Location `json:"Location,omitzero"`
}

S3TableOutputOptions specifies the S3 location for a recipe job's Glue Data Catalog output (aws-sdk-go-v2/service/databrew/types.S3TableOutputOptions).

type Sample 

type Sample struct {
	Type string `json:"Type,omitempty"`
	Size int    `json:"Size,omitempty"`
}

Sample describes a data sample for a project.

type Schedule 

type Schedule struct {
	Tags             map[string]string `json:"Tags,omitempty"`
	Name             string            `json:"Name"`
	Arn              string            `json:"ResourceArn"`
	CronExpression   string            `json:"CronExpression"`
	CreatedBy        string            `json:"CreatedBy,omitempty"`
	LastModifiedBy   string            `json:"LastModifiedBy,omitempty"`
	AccountID        string            `json:"AccountId,omitempty"`
	JobNames         []string          `json:"JobNames,omitempty"`
	CreateDate       float64           `json:"CreateDate,omitempty"`
	LastModifiedDate float64           `json:"LastModifiedDate,omitempty"`
}

Schedule represents a DataBrew schedule. AccountID mirrors aws-sdk-go-v2/service/databrew/types.Schedule's AccountId member -- see Dataset's AccountID doc comment; DescribeScheduleOutput (api_op_DescribeSchedule.go:38-76) has no AccountId member either.

type StorageBackend 

type StorageBackend interface {
	Region() string
	AccountID() string
	Reset()

	// Snapshot and Restore implement persistence.Persistable. Handler
	// delegates to them (see persistence.go) so cli.go's generic
	// setupPersistence picks DataBrew up.
	Snapshot(ctx context.Context) []byte
	Restore(ctx context.Context, data []byte) error

	// Dataset operations.
	CreateDataset(
		ctx context.Context,
		name, format string,
		input DatasetInput,
		formatOpts DatasetFormatOptions,
		tags map[string]string,
		pathOptions *PathOptions,
	) (*Dataset, error)
	DescribeDataset(ctx context.Context, name string) (*Dataset, error)
	ListDatasets(ctx context.Context, maxResults int, nextToken string) ([]*Dataset, string)
	UpdateDataset(
		ctx context.Context,
		name, format string,
		input DatasetInput,
		formatOpts DatasetFormatOptions,
		pathOptions *PathOptions,
	) error
	DeleteDataset(ctx context.Context, name string) error

	// Recipe operations.
	CreateRecipe(
		ctx context.Context,
		name, description string,
		steps []RecipeStep,
		tags map[string]string,
	) (*Recipe, error)
	DescribeRecipe(ctx context.Context, name, version string) (*Recipe, error)
	ListRecipes(ctx context.Context, maxResults int, nextToken, versionFilter string) ([]*Recipe, string)
	ListRecipeVersions(ctx context.Context, name string, maxResults int, nextToken string) ([]*Recipe, string, error)
	PublishRecipe(ctx context.Context, name, description string) error
	UpdateRecipe(ctx context.Context, name, description string, steps []RecipeStep) error
	DeleteRecipe(ctx context.Context, name string) error
	DeleteRecipeVersion(ctx context.Context, name, version string) error
	BatchDeleteRecipeVersion(ctx context.Context, name string, versions []string) ([]RecipeVersionErrorDetail, error)

	// Project operations.
	CreateProject(
		ctx context.Context,
		name, datasetName, recipeName, roleArn string,
		sample Sample,
		tags map[string]string,
	) (*Project, error)
	DescribeProject(ctx context.Context, name string) (*Project, error)
	ListProjects(ctx context.Context, maxResults int, nextToken string) ([]*Project, string)
	UpdateProject(ctx context.Context, name, roleArn string, sample Sample) error
	DeleteProject(ctx context.Context, name string) error
	OpenProjectSession(ctx context.Context, name string) (*Project, error)

	// Job operations.
	CreateJob(
		ctx context.Context,
		name, jobType, datasetName, projectName, recipeName, roleArn string,
		outputs []Output,
		tags map[string]string,
		extra JobExtras,
	) (*Job, error)
	DescribeJob(ctx context.Context, name string) (*Job, error)
	ListJobs(ctx context.Context, maxResults int, nextToken, datasetName, projectName string) ([]*Job, string)
	UpdateJob(
		ctx context.Context,
		name, roleArn string,
		outputs []Output,
		maxCapacity, maxRetries, timeout int,
		extra JobExtras,
	) error
	DeleteJob(ctx context.Context, name string) error
	StartJobRun(ctx context.Context, jobName string) (*JobRun, error)
	ListJobRuns(
		ctx context.Context,
		jobName string,
		maxResults int,
		nextToken string,
	) ([]*JobRun, string, error)
	DescribeJobRun(ctx context.Context, name, runID string) (*JobRun, error)
	StopJobRun(ctx context.Context, name, runID string) (*JobRun, error)

	// Ruleset operations.
	CreateRuleset(
		ctx context.Context,
		name, description, targetArn string,
		rules []Rule,
		tags map[string]string,
	) (*Ruleset, error)
	DescribeRuleset(ctx context.Context, name string) (*Ruleset, error)
	ListRulesets(ctx context.Context, maxResults int, nextToken, targetArn string) ([]*Ruleset, string)
	UpdateRuleset(ctx context.Context, name, description string, rules []Rule) error
	DeleteRuleset(ctx context.Context, name string) error

	// Schedule operations.
	CreateSchedule(
		ctx context.Context,
		name string,
		jobNames []string,
		cron string,
		tags map[string]string,
	) (*Schedule, error)
	DescribeSchedule(ctx context.Context, name string) (*Schedule, error)
	ListSchedules(ctx context.Context, maxResults int, nextToken string) ([]*Schedule, string)
	UpdateSchedule(ctx context.Context, name string, jobNames []string, cron string) error
	DeleteSchedule(ctx context.Context, name string) error

	// Tag operations.
	FindTagsByArn(ctx context.Context, arn string) (map[string]string, error)
	UpdateTagsByArn(ctx context.Context, arn string, add map[string]string, remove []string) error
}

StorageBackend defines the interface for all DataBrew backend operations.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.