athena

package
v0.15.2 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jun 29, 2026 License: Apache-2.0 Imports: 23 Imported by: 0

Documentation

Overview

Package athena implements the scrapper.Scrapper interface for Amazon Athena.

V1 scope is metadata-only: catalog, tables, databases, view definitions. Table metrics, query logs, freshness, table constraints, and table comments are deferred — those methods either return ErrUnsupported or empty slices.

Athena exposes a Presto-derived SQL surface, so most queries look like Trino's but talk to Athena's information_schema (no system.metadata.* tables, no per-catalog prefix needed — Athena's information_schema is already catalog-scoped to the data catalog the connection is bound to).

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func ScopeFromConf 

func ScopeFromConf(conf *AthenaScrapperConf) *scope.ScopeFilter

ScopeFromConf returns the configured ScopeFilter as-is. Athena is the first warehouse to use synq.common.v1.ScopeFilter directly on both internal and agent protos (no legacy `repeated string databases` field to translate from), so this is just a passthrough — kept for symmetry with the other scrappers' ScopeFromConf functions.

Mapping note: Athena's hierarchy is Glue catalog → Glue database → table. In ScopeFilter terms (mirroring BigQuery's project/dataset shape): 

ScopeRule.database = Glue catalog (almost always 'AwsDataCatalog')
ScopeRule.schema   = Glue database
ScopeRule.table    = Glue table / view

Types

type AthenaScrapper 

type AthenaScrapper struct {
	// contains filtered or unexported fields
}

func NewAthenaScrapper 

func NewAthenaScrapper(ctx context.Context, conf *AthenaScrapperConf) (*AthenaScrapper, error)

func (*AthenaScrapper) Capabilities 

func (e *AthenaScrapper) Capabilities() scrapper.Capabilities

func (*AthenaScrapper) Close 

func (e *AthenaScrapper) Close() error

func (*AthenaScrapper) DialectType 

func (e *AthenaScrapper) DialectType() string

func (*AthenaScrapper) FetchQueryLogs 

func (e *AthenaScrapper) FetchQueryLogs(
	ctx context.Context,
	from, to time.Time,
	obfuscator querylogs.QueryObfuscator,
) (querylogs.QueryLogIterator, error)

FetchQueryLogs streams the workgroup's query history from the Athena management APIs (athena:ListQueryExecutions + BatchGetQueryExecution).

These calls do NOT bill per data scanned — they're metadata operations. Page size is 50 (the AWS hard cap for both ListQueryExecutions and BatchGetQueryExecution); the iterator stops paging once it sees an execution older than `from`. AWS returns executions newest-first.

Athena retains query history for 45 days by default per workgroup; older `from` values silently truncate to that window.

func (*AthenaScrapper) IsPermissionError 

func (e *AthenaScrapper) IsPermissionError(err error) bool

func (*AthenaScrapper) QueryCatalog 

func (e *AthenaScrapper) QueryCatalog(ctx context.Context) ([]*scrapper.CatalogColumnRow, error)

func (*AthenaScrapper) QueryCustomMetrics 

func (e *AthenaScrapper) QueryCustomMetrics(ctx context.Context, sql string, args ...any) ([]*scrapper.CustomMetricsRow, error)

func (*AthenaScrapper) QueryDatabases 

func (e *AthenaScrapper) QueryDatabases(ctx context.Context) ([]*scrapper.DatabaseRow, error)

QueryDatabases lists Glue databases visible to the connection.

Note on terminology: ScopeRule.database is the Glue Data Catalog name (almost always 'AwsDataCatalog'); ScopeRule.schema is the Glue database. We populate DatabaseRow.Database with the *Glue database* name here so downstream code that treats DatabaseRow as the top-level container sees what the user actually filters on. The real catalog lives on the row's Instance (and the catalog string is captured via the executor's Catalog()).

func (*AthenaScrapper) QuerySchemas added in v0.14.5 

func (e *AthenaScrapper) QuerySchemas(ctx context.Context) ([]*scrapper.SchemaRow, error)

QuerySchemas lists the Glue databases (mapped to schemas) visible to the connection. ScopeRule.database is the Glue Data Catalog name and ScopeRule.schema is the Glue database, so we populate SchemaRow.Database with the catalog and SchemaRow.Schema with the Glue database — mirroring QueryTables.

func (*AthenaScrapper) QuerySegments 

func (e *AthenaScrapper) QuerySegments(ctx context.Context, sql string, args ...any) ([]*scrapper.SegmentRow, error)

func (*AthenaScrapper) QueryShape 

func (e *AthenaScrapper) QueryShape(ctx context.Context, sql string) ([]*scrapper.QueryShapeColumn, error)

func (*AthenaScrapper) QuerySqlDefinitions 

func (e *AthenaScrapper) QuerySqlDefinitions(ctx context.Context) ([]*scrapper.SqlDefinitionRow, error)

QuerySqlDefinitions returns view (and optionally table) DDLs.

Default: pulls view bodies from information_schema.views.view_definition — one bulk query, free (DDL).

With UseShowCreateView/UseShowCreateTable enabled, additionally calls SHOW CREATE VIEW / SHOW CREATE TABLE per object in an 8-way errgroup pool. SHOW CREATE on Athena is the only way to retrieve table DDL (CTAS bodies, Iceberg TBLPROPERTIES, Hive external LOCATION/SerDe) and the only way to get the full CREATE OR REPLACE VIEW DDL with original column declarations. Each call is one Athena query execution (~$0.00005 at the 10MB scan minimum).

func (*AthenaScrapper) QueryTableConstraints 

func (e *AthenaScrapper) QueryTableConstraints(ctx context.Context) ([]*scrapper.TableConstraintRow, error)

QueryTableConstraints exposes partition columns as `PARTITION BY` constraint rows — the same shape BigQuery uses for time/range partitioning.

Athena / Hive / Iceberg metastore has no traditional PK / FK / UNIQUE / CHECK concept; partitioning is the only constraint-shaped metadata Athena exposes that's worth surfacing.

Sources, per row:

  1. Glue `Table.PartitionKeys` — populated for Hive-style partitioned tables (`PARTITIONED BY (col TYPE)`), free along with the metrics fetch.
  2. Iceberg `$partitions` metadata table — Glue does NOT populate `PartitionKeys` for Iceberg tables (the partition spec lives in the `metadata.json` on S3, not in Glue), so we read the spec column names from `SELECT * FROM "<db>"."<table>$partitions" LIMIT 1` when `UseIcebergMetricsScan` is enabled. One Athena query per Iceberg table (~$0.00005 each at the 10MB scan minimum).

Iceberg transforms (`day(...)`, `bucket(N, ...)`, `truncate(L, ...)`) are flattened to their source column names — the per-column constraint row can't represent the transform. To recover the full transform string, enable `UseShowCreateTable` and parse the `PARTITIONED BY` clause from the DDL returned by `QuerySqlDefinitions`.

func (*AthenaScrapper) QueryTableMetrics 

func (e *AthenaScrapper) QueryTableMetrics(ctx context.Context, lastFetchTime time.Time) ([]*scrapper.TableMetricsRow, error)

QueryTableMetrics returns row counts, total size in bytes, and last-updated timestamps for tables visible to the configured Glue catalog.

Sources, in order of priority per row:

  1. Glue table parameters (Hive table-level statistics): `numRows`, `totalSize`, `transient_lastDdlTime`. Populated by `ANALYZE TABLE COMPUTE STATISTICS` or by Glue crawlers with stats enabled. Free — one paginated `glue:GetTables` call per Glue database, no Athena query cost.
  2. Iceberg `$files` metadata table aggregate (only when `UseIcebergMetricsScan` is enabled and the Glue parameters didn't already supply row count + size). One Athena query per Iceberg table (~$0.00005 each at the 10MB scan minimum).

Tables with no usable stats from either source are skipped — we'd rather omit a row than emit zeros that downstream consumers treat as a real "0 rows".

`lastFetchTime` is honoured against Glue's `UpdateTime`: tables that haven't changed since the last fetch are skipped entirely, including their Iceberg scan.

func (*AthenaScrapper) QueryTables 

func (e *AthenaScrapper) QueryTables(ctx context.Context, opts ...scrapper.QueryTablesOption) ([]*scrapper.TableRow, error)

func (*AthenaScrapper) RunRawQuery 

func (e *AthenaScrapper) RunRawQuery(ctx context.Context, sql string) (scrapper.RawQueryRowIterator, error)

func (*AthenaScrapper) SqlDialect 

func (e *AthenaScrapper) SqlDialect() sqldialect.Dialect

func (*AthenaScrapper) ValidateConfiguration 

func (e *AthenaScrapper) ValidateConfiguration(ctx context.Context) ([]string, error)

type AthenaScrapperConf 

type AthenaScrapperConf struct {
	*dwhexecathena.AthenaConf

	// UseShowCreateView, when true, calls `SHOW CREATE VIEW` per view to
	// retrieve the full `CREATE OR REPLACE VIEW … AS …` DDL instead of just
	// the rewritten Presto body from `information_schema.views.view_definition`.
	// Each call is one Athena query (~$0.00005 at the 10MB scan minimum).
	UseShowCreateView bool

	// UseShowCreateTable, when true, calls `SHOW CREATE TABLE` per non-view
	// object to retrieve the table DDL. This is the only way to get table DDL
	// on Athena — `information_schema` exposes no equivalent. Required to see
	// CTAS bodies (lineage), Iceberg `TBLPROPERTIES`, partitioning, and Hive
	// external `LOCATION`/SerDe info. Each call is one Athena query.
	UseShowCreateTable bool

	// UseIcebergMetricsScan, when true, fans out per-Iceberg-table Athena
	// queries against the table's metadata tables to recover information that
	// Glue does not expose:
	//
	//  - `$files` aggregate → row count + total size.
	//  - `$snapshots` MAX(committed_at) → exact data-freshness timestamp (the
	//    moment of the most recent INSERT / MERGE / UPDATE / DELETE). Glue's
	//    `UpdateTime` usually moves with the metadata pointer too, but the
	//    snapshot timestamp is authoritative.
	//  - `$partitions` `typeof(partition)` → partition-spec column names (Glue's
	//    `PartitionKeys` is empty for native Iceberg tables since the spec
	//    lives in the metadata.json).
	//
	// Per Iceberg table: one combined query for row count + size + freshness,
	// plus one query for the partition spec on partitioned tables. Each query
	// costs ~$0.00005 at Athena's 10MB scan minimum. Hive-style tables and
	// views are unaffected — those metrics / partitions / freshness come from
	// Glue parameters and `PartitionKeys` for free. Default is false.
	UseIcebergMetricsScan bool

	// Scope is the include/exclude filter for Glue catalog/database/table.
	// Nil means accept-all. The cloud and agent protos both carry this as
	// a synq.common.v1.ScopeFilter; callers translate via ScopeFromProto.
	Scope *scope.ScopeFilter
}

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.