README
¶
application_yaml
Extractor for Application from YAML file.
Usage
source:
name: application_yaml
scope: applications-stg
config:
file: "./path/to/meteor.app.yaml"
env_prefix: CI
Inputs
| Key | Value | Example | Description | Required? |
|---|---|---|---|---|
file |
string |
meteor.app.yaml |
File path of application.yaml |
✅ |
env_prefix |
string |
CI |
Prefix for environment variables. These are made available as variables in application.yaml with the prefix trimmed. |
✘ |
application.yaml format
name: "string" # REQUIRED
id: "string" # REQUIRED
team:
id: "string"
name: "string"
email: "string"
description: "string"
url: "string"
version: "string"
inputs: # OPTIONAL
# Format: "urn:{service}:{scope}:{type}:{name}"
- urn:bigquery:bq-raw-internal:table:bq-raw-internal:dagstream.production_feast09_s2id13_30min_demand
- urn:kafka:int-dagstream-kafka.yonkou.io:topic:staging_feast09_s2id13_30min_demand
outputs: # OPTIONAL
# Format: "urn:{service}:{scope}:{type}:{name}"
- urn:kafka:1-my-kafka.com:topic:staging_feast09_mixed_granularity_demand_forecast_3es
create_time: "2006-01-02T15:04:05Z"
update_time: "2006-01-02T15:04:05Z"
labels:
x: "y"
For an example,
see application.detailed.yaml
Environment variables that are commonly available in CI can be used for
populating the fields. The default prefix for environment variables is CI but
this can be overridden using env_prefix config.
For an example,
see application.envvars.yaml. The
following env vars are utilised for it:
CI_PROJECT_NAMECI_PROJECT_URLCI_COMMIT_SHORT_SHA
Outputs
The application is mapped to an Asset with model specific
metadata stored using Application. Please refer the
proto definitions for more information.
| Field | Value | Sample Value |
|---|---|---|
resource.urn |
urn:application_yaml:{scope}:application:{application.name} |
urn:application_yaml:integration:application:order-manager |
resource.name |
{application.name} |
order-manager |
resource.service |
application_yaml |
application_yaml |
resource.type |
application |
application |
resource.url |
{application.url} |
https://github.com/mycompany/order-manager |
resource.description |
{application.description |
Order-Manager is the order management system for MyCompany |
application_id |
application.id |
0adf3214-676c-4a74-ab37-9d4a4b8ade0e |
version |
application.version |
d6ec883 |
create_time |
{application.create_time} |
2022-08-08T03:17:54Z |
update_time |
{application.update_time} |
2022-08-08T03:57:54Z |
ownership.owners[0].urn |
{application.team.id} |
9ebcc2f8-5894-47c6-83a9-160b7eaa3f6b |
ownership.owners[0].name |
{application.team.name} |
Search |
ownership.owners[0].email |
{application.team.email} |
search@mycompany.com |
lineage.upstreams[].urn |
{application.inputs[]} |
urn:kafka:int-kafka.yonkou.io:topic:staging_30min_demand |
lineage.downstreams[].urn |
{application.outputs[]} |
urn:bigquery:bq-internal:table:bq-internal:dagstream.production_30min_demand |
resource.labels |
map[string]string |
{"team": "Booking Experience"} |
Contributing
Refer to the contribution guidelines for information on contributing to this module.
Documentation
¶
Index ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type Application ¶
type Application struct {
Name string `json:"name" yaml:"name" validate:"required"`
ID string `json:"id" yaml:"id" validate:"required"`
Team struct {
ID string `json:"id" yaml:"id"`
Name string `json:"name" yaml:"name"`
Email string `json:"email" yaml:"email"`
} `json:"team" yaml:"team"`
Description string `json:"description" yaml:"description"`
URL string `json:"url" yaml:"url" validate:"omitempty,url"`
Version string `json:"version" yaml:"version"`
Inputs []string `json:"inputs" yaml:"inputs"`
Outputs []string `json:"outputs" yaml:"outputs"`
CreateTime time.Time `json:"create_time" yaml:"create_time"`
UpdateTime time.Time `json:"update_time" yaml:"update_time"`
Labels map[string]string `json:"labels" yaml:"labels"`
}
type Config ¶
type Config struct {
File string `mapstructure:"file" validate:"required,file"`
EnvPrefix string `mapstructure:"env_prefix" default:"CI" validate:"required"`
}
Config holds the set of configuration for the CaraML Store extractor
type Extractor ¶
type Extractor struct {
plugins.BaseExtractor
// contains filtered or unexported fields
}
Extractor is the extractor instance for application YAML file.
Source Files
Click to show internal directories.
Click to hide internal directories.