Documentation
¶
Overview ¶
Discovery phase for create_data_quality_job. Folded in from the former prepare_create_data_quality_job tool: given whatever the caller knows so far (a connection, and optionally a data source / schema / table, or a catalog Table asset), it walks the same chain the DQ wizard uses — resolve the connection, detect the job type, and enumerate data sources, schemas, tables. Only the FIRST missing location field is enumerated (returned as options); fields the caller already supplied are trusted as-is (the public create validates them server-side). This keeps the fully-specified path a single build+preview with no extra browse round-trips, while an under-specified call still gets one-field-at-a-time discovery.
Package create_data_quality_job implements the create_data_quality_job MCP tool — a single tool that spans discovery, preview, and create of a Collibra data-quality job for a table (and, per the wizard, queues a run). Discovery (connection / data source / schema / table enumeration) lives in discover.go and uses the INTERNAL monitoring/edge browse endpoints; the create itself POSTs to the public DQ API.
IT POSTS TO THE PUBLIC DQ API: POST /rest/dq/1.0/jobs (clients.CreateDqJob / clients.CreateDqJobRequest, contract dq/udq-app-client/oas/dq-v1-public-oas-spec.yaml). The public create accepts the full job definition — sourceQuery, runDate window, monitors, schedule, back-runs, notifications, and pullup/pushdown settings — so the internal BFF endpoint is no longer needed. (The public DQ API exposes no connection/edge metadata browse, so discovery uses the internal endpoints.)
HOW SCHEDULING + TIME-SLICE + ${rd} FIT TOGETHER: A scheduled job re-runs on a cadence; each run scans one date-column slice. The slice lives in the job SQL's WHERE as `<col> >= '${rd}' AND <col> < '${rdEnd}'`. The public API has NO structured time-slice object — instead the engine substitutes ${rd}/${rdEnd} from runDate/ runDateEnd (formatted per jobSettings.dateFormat), and the scheduler adjusts the run date per run from the schedule's *Offset enums. So this tool writes the ${rd} predicate into sourceQuery whenever a timeSliceColumn is set and seeds runDate/runDateEnd to the most recent slice; without a timeSliceColumn every scheduled run rescans the whole table.
It is built around a confirm checkpoint: confirm=false (default) returns a PREVIEW of the exact request without creating; confirm=true submits.
Index ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
Types ¶
type ColumnInfo ¶
type ColumnInfo struct {
Name string `json:"name" jsonschema:"Column name."`
Type string `json:"type" jsonschema:"Source column type (e.g. int4, text, numeric, timestamp)."`
}
ColumnInfo is one column of the resolved table.
type ConnectionOption ¶
type ConnectionOption struct {
ConnectionID string `json:"connectionId" jsonschema:"Connection UUID."`
ConnectionName string `json:"connectionName" jsonschema:"Connection display name."`
CapabilityTypes []string `` /* 126-byte string literal not displayed */
DatabaseProduct string `json:"databaseProductName,omitempty" jsonschema:"Database vendor (e.g. POSTGRES)."`
}
ConnectionOption is one selectable connection.
type Input ¶
type Input struct {
// --- Catalog table-asset entry point (optional). Identify the source table by a DGC catalog Table
// asset instead of connection/data-source/schema/table names; the location is resolved from it (and
// the success result gets a catalog deep link). tableAssetId wins, then URL, then name. ---
TableAssetID string `` /* 249-byte string literal not displayed */
TableAssetURL string `` /* 144-byte string literal not displayed */
TableAssetName string `` /* 238-byte string literal not displayed */
TableAssetDomain string `` /* 153-byte string literal not displayed */
// --- Data location. Supply a connection (UUID or name) to begin; omit dataSource/schema/table to
// have the tool list the options for the next one (status=incomplete). All four resolved = the tool
// previews. ---
Connection string `` /* 235-byte string literal not displayed */
DataSourceName string `` /* 165-byte string literal not displayed */
SchemaName string `json:"schemaName,omitempty" jsonschema:"Schema within the data source, e.g. 'sales'. Omit to list the schemas."`
TableName string `` /* 126-byte string literal not displayed */
JobType string `` /* 174-byte string literal not displayed */
JobName string `` /* 131-byte string literal not displayed */
// --- Column selection. Omit to monitor ALL columns (the default). Provide a subset to
// scope profiling/monitoring to just those columns (each sent with selected=true). ---
SelectedColumns []string `` /* 350-byte string literal not displayed */
// --- Monitors. Omit to use the default set; provide an authoritative set of monitor keys
// to enable (anything not listed is turned off). descriptiveStatistics unmasks sensitive
// data — only include it after explicit user confirmation. ---
Monitors []string `` /* 428-byte string literal not displayed */
// --- Advanced monitor settings (adaptive behavior). Omit both to use the wizard defaults
// (lookback 10, learning phase 4). Setting either sends a structured `settings` object. ---
DataLookback int `` /* 206-byte string literal not displayed */
LearningPhase int `` /* 142-byte string literal not displayed */
// --- Row filter (the wizard's single-column predicate). Scopes the job to rows matching
// "filterColumn filterOperator filterValue" (e.g. amount > 100). Omit to scan all rows.
// This is ONE predicate — not compound AND/OR or free-form SQL (mirrors the DQ wizard). ---
FilterColumn string `` /* 257-byte string literal not displayed */
FilterOperator string `` /* 258-byte string literal not displayed */
FilterValue string `` /* 342-byte string literal not displayed */
// --- Row sampling. Omit to scan all matching rows; a positive size enables sampling. The
// engine turns this into a dialect-correct RANDOM()/LIMIT query at dispatch. ---
SampleSize int `` /* 230-byte string literal not displayed */
// --- Time slice (incremental scanning). Pick a DATE COLUMN; when set, the tool writes a
// "<col> >= '${rd}' AND <col> < '${rdEnd}'" predicate into the job SQL's WHERE. The scheduler
// substitutes ${rd}/${rdEnd} per run, so each run scans only that slice, not the whole table. ---
TimeSliceColumn string `` /* 413-byte string literal not displayed */
TimeSliceSize int `` /* 175-byte string literal not displayed */
TimeSliceUnit string `json:"timeSliceUnit,omitempty" jsonschema:"HOURS | DAYS | WEEKS | MONTHS (default DAYS)."`
// --- Schedule (recurrence). Omit/NEVER = run once now. To slice incrementally per run, set a
// timeSliceColumn too (the schedule provides the cadence; the ${rd} window provides the slice). ---
ScheduleRepeat string `` /* 211-byte string literal not displayed */
ScheduleRunTime string `` /* 144-byte string literal not displayed */
ScheduleDaysOfWeek []string `` /* 193-byte string literal not displayed */
ScheduleDayOfMonth int `json:"scheduleDayOfMonth,omitempty" jsonschema:"For MONTHLY with scheduleMonthlyMode=DAY (the default): day of month 1-28."`
ScheduleMonthlyMode string `` /* 163-byte string literal not displayed */
RunDateOffset string `` /* 424-byte string literal not displayed */
// --- Back runs (historical backfill, walks forward one slice/run from runDate - binValue*timeBin). ---
BackrunEnabled bool `json:"backrunEnabled,omitempty" jsonschema:"Enable historical backfill runs."`
BackrunBinValue int `json:"backrunBinValue,omitempty" jsonschema:"Number of prior bins to backfill."`
BackrunTimeBin string `json:"backrunTimeBin,omitempty" jsonschema:"DAY | MONTH | YEAR."`
// --- Notifications (optional). Configured when `notify` or `notifyRecipients` is set; otherwise
// no notifications. The invoking user is always the default recipient. ---
Notify []string `` /* 312-byte string literal not displayed */
NotifyRowsBelow int `json:"notifyRowsBelow,omitempty" jsonschema:"Threshold for rowsBelow — alert when row count <= this. Default 1."`
NotifyScoreBelow int `json:"notifyScoreBelow,omitempty" jsonschema:"Threshold for scoreBelow — alert when score (0-100) <= this. Default 75."`
NotifyRunTimeAboveMinutes int `` /* 134-byte string literal not displayed */
NotifyRunsWithoutData int `` /* 134-byte string literal not displayed */
NotifyDaysWithoutData int `` /* 134-byte string literal not displayed */
NotifyMessage string `json:"notifyMessage,omitempty" jsonschema:"Optional global message applied to the enabled notifications."`
NotifyRecipients []string `` /* 211-byte string literal not displayed */
NotifyProceedWithoutUnresolved bool `` /* 282-byte string literal not displayed */
// --- Per-notification message overrides. Keyed by notification key (see prepare's notifications);
// a per-key message overrides notifyMessage for just that notification. ---
NotifyMessages map[string]string `` /* 313-byte string literal not displayed */
// --- Sizing (PULLUP only — the wizard's "Size Job Resources" step). Default = automatic
// ("Automate job resources"). Setting ANY manual field below switches to manual sizing (advanced;
// for large datasets). Memory fields are GB (e.g. "1", "2"). ---
SizingMaxExecutors int `` /* 189-byte string literal not displayed */
SizingExecutorCores int `` /* 151-byte string literal not displayed */
SizingExecutorMemoryGb string `` /* 168-byte string literal not displayed */
SizingDriverCores int `json:"sizingDriverCores,omitempty" jsonschema:"PULLUP manual sizing: driver cores. Default 1 when manual sizing is engaged."`
SizingDriverMemoryGb string `` /* 160-byte string literal not displayed */
SizingMemoryOverheadGb string `` /* 164-byte string literal not displayed */
SizingNumPartitions int `json:"sizingNumPartitions,omitempty" jsonschema:"PULLUP: number of partitions (load options). Omit/0 lets Spark decide."`
// --- Parallel JDBC (PULLUP advanced sizing). mode AUTO (column+count auto) | AUTO_COLUMN (column
// auto, count required) | MANUAL (column+count required). Mode is inferred when omitted: a
// partition column implies MANUAL, a partition count alone implies AUTO_COLUMN. ---
ParallelJdbcMode string `` /* 296-byte string literal not displayed */
ParallelJdbcPartitionColumn string `` /* 231-byte string literal not displayed */
ParallelJdbcPartitionNumber int `` /* 144-byte string literal not displayed */
SparkSqlProperties map[string]string `json:"sparkSqlProperties,omitempty" jsonschema:"PULLUP: additional Spark SQL key/value properties to set on the job."`
// --- Compute settings (PUSHDOWN only — the wizard's Review step). Defaults connections 10
// (range 1-50), threads 2 (range 1-10). ---
PushdownConnections int `json:"pushdownConnections,omitempty" jsonschema:"PUSHDOWN compute: number of connections (1-50). Default 10 when omitted."`
PushdownThreads int `json:"pushdownThreads,omitempty" jsonschema:"PUSHDOWN compute: number of threads (1-10). Default 2 when omitted."`
// AcknowledgeDescriptiveStatistics must be true to enable the descriptiveStatistics monitor — it
// UNMASKS sensitive values. Without it the tool refuses to proceed (the wizard's explicit-confirm).
AcknowledgeDescriptiveStatistics bool `` /* 219-byte string literal not displayed */
Confirm bool `` /* 171-byte string literal not displayed */
}
Input — the tool discovers the data location one field at a time: supply what you know and omit the rest, and the response returns the options for the first missing field (status=incomplete) until the location (connection + data source + schema + table) is complete, at which point it previews. Time-slice / schedule / back-run fields are optional and structured.
type Output ¶
type Output struct {
Status Status `` /* 131-byte string literal not displayed */
Message string `json:"message" jsonschema:"Human-readable outcome and what to do next."`
// --- Discovery (status=incomplete): options for the next location field to choose. ---
ConnectionOptions []ConnectionOption `` /* 179-byte string literal not displayed */
TableAssetOptions []TableAssetOption `` /* 159-byte string literal not displayed */
DataSourceOptions []string `` /* 168-byte string literal not displayed */
SchemaOptions []string `` /* 151-byte string literal not displayed */
TableOptions []string `` /* 147-byte string literal not displayed */
OptionsTruncated bool `json:"optionsTruncated,omitempty" jsonschema:"True when an options list was truncated below the instance's true total."`
// --- Option catalogs (status=preview): surfaced so you can offer the config choices before confirm. ---
Columns []ColumnInfo `` /* 180-byte string literal not displayed */
Monitors []clients.DqMonitorInfo `` /* 237-byte string literal not displayed */
AdaptiveMonitorSettings []clients.DqAdaptiveMonitorSetting `` /* 254-byte string literal not displayed */
Notifications []clients.DqNotificationInfo `` /* 215-byte string literal not displayed */
// --- Preview / create result. ---
Request *clients.CreateDqJobRequest `` /* 140-byte string literal not displayed */
JobName string `json:"jobName,omitempty" jsonschema:"Final job name."`
JobID string `json:"jobId,omitempty" jsonschema:"Created job id (the internal create returns jobId, not a run id)."`
JobType string `json:"jobType,omitempty" jsonschema:"Resolved job type."`
JobDetailsLink string `` /* 165-byte string literal not displayed */
TableAssetLink string `` /* 186-byte string literal not displayed */
Warnings []string `` /* 138-byte string literal not displayed */
AffectedStep string `` /* 185-byte string literal not displayed */
UnsupportedOptions []string `` /* 138-byte string literal not displayed */
Guidance string `json:"guidance,omitempty" jsonschema:"On needs_input/error, what to do next."`
}
type Status ¶
type Status string
const ( // StatusIncomplete means a required location field is missing; the response carries the // pre-fetched options for the next field to choose (connection/dataSource/schema/table). StatusIncomplete Status = "incomplete" StatusPreview Status = "preview" StatusCreated Status = "created" StatusNeedsInput Status = "needs_input" StatusError Status = "error" )
type TableAssetOption ¶
type TableAssetOption struct {
AssetID string `json:"assetId" jsonschema:"Catalog Table asset UUID — pass as tableAssetId to select it."`
DisplayName string `json:"displayName" jsonschema:"Table signifier (name)."`
Domain string `json:"domain,omitempty" jsonschema:"Domain/path of the asset (helps tell duplicates apart)."`
FullName string `json:"fullName,omitempty" jsonschema:"Fully-qualified asset name."`
}
TableAssetOption is one catalog Table asset candidate for disambiguation.