README
¶
Stencil
Register entity column schemas with a Stencil schema registry in JSON Schema or Avro format.
Usage
sinks:
- name: stencil
config:
host: https://stencil.com
namespace_id: my-namespace
format: json
Configuration
| Key | Type | Example | Description | |
|---|---|---|---|---|
host |
string |
https://stencil.com |
Hostname of the Stencil service | required |
namespace_id |
string |
my-namespace |
Namespace in which schemas are created | required |
format |
string |
json |
Schema format: json or avro. Default: json. |
optional |
Behavior
For each Record whose Entity properties contain a columns list, the sink builds a schema (JSON Schema or Avro) from the column metadata (name, data type, nullability, description). The schema is posted to POST /v1beta1/namespaces/{namespace_id}/schemas/{schema_id}, where schema_id is derived from the Entity URN.
Entities without columns are skipped. Type mapping from source-specific column types (BigQuery, Postgres) to JSON Schema / Avro types is handled automatically. Server errors (5xx) are returned as retryable errors.
Contributing
Refer to the contribution guidelines for information on contributing to this module.
Documentation
¶
Index ¶
Constants ¶
const ( JSONTypeObject JSONType = "object" JSONTypeString JSONType = "string" JSONTypeNumber JSONType = "number" JSONTypeArray JSONType = "array" JSONTypeBoolean JSONType = "boolean" JSONTypeNull JSONType = "null" AvroTypeNull AvroType = "null" AvroTypeBoolean AvroType = "boolean" AvroTypeInteger AvroType = "int" AvroTypeLong AvroType = "long" AvroTypeFloat AvroType = "float" AvroTypeDouble AvroType = "double" AvroTypeBytes AvroType = "bytes" AvroTypeString AvroType = "string" AvroTypeRecord AvroType = "record" AvroTypeArray AvroType = "array" AvroTypeMap AvroType = "map" )
Variables ¶
This section is empty.
Functions ¶
Types ¶
type AvroFields ¶
type AvroSchema ¶
type AvroSchema struct {
Type string `json:"type"`
Namespace string `json:"namespace"`
Name string `json:"name"`
Fields []AvroFields `json:"fields"`
}
type Config ¶
type Config struct {
Host string `mapstructure:"host" validate:"required"`
NamespaceID string `mapstructure:"namespace_id" validate:"required"`
Format string `mapstructure:"format" validate:"oneof=json avro" default:"json"`
}
Config holds the set of configuration options for the sink
Source Files