pluginsdk

package module
v0.0.0-...-7aecacb Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jul 11, 2026 License: MIT Imports: 26 Imported by: 0

README

plugin-sdk

SDK for building external Marmot plugins.

Marmot plugins are standalone binaries that the Marmot host launches on demand via go-plugin and talks to over gRPC. This SDK provides everything both sides need: the wire protocol, the plugin-facing types and helpers, and the host-side client.

Writing a plugin

A plugin implements the Source interface and hands it to Serve in its main function:

package main

import (
	"context"

	pluginsdk "github.com/marmotdata/plugin-sdk"
)

type Config struct {
	pluginsdk.BaseConfig `json:",inline"`

	ProjectID string `json:"project_id" description:"Project to discover" validate:"required"`
}

type Source struct{}

func (s *Source) Validate(raw pluginsdk.RawConfig) (pluginsdk.RawConfig, error) {
	config, err := pluginsdk.UnmarshalConfig[Config](raw)
	if err != nil {
		return nil, err
	}
	pluginsdk.ApplyDefaults(config, raw)
	if err := pluginsdk.ValidateStruct(config); err != nil {
		return nil, err
	}
	return raw, nil
}

func (s *Source) Discover(ctx context.Context, raw pluginsdk.RawConfig) (*pluginsdk.DiscoveryResult, error) {
	// discover assets, lineage, and documentation here
	return &pluginsdk.DiscoveryResult{}, nil
}

func main() {
	pluginsdk.Serve(&pluginsdk.ServeConfig{
		Meta: pluginsdk.Meta{
			ID:          "example",
			Name:        "Example",
			Description: "Discovers things from Example",
			Icon:        "example",
			Category:    "storage",
			ConfigSpec:  pluginsdk.GenerateConfigSpec(Config{}),
		},
		Source: &Source{},
	})
}

Build the binary with a marmot-plugin- name prefix and drop it in ~/.marmot/plugins (or the directory set by MARMOT_PLUGINS_DIR). Marmot discovers it at startup, fetches its metadata, and registers it; a local binary shadows a downloaded core plugin with the same ID. Plugin processes are short-lived: the host spawns the binary per call and kills it, and the SDK runs Validate before Discover in each process, so state your Validate sets on the Source is there when Discover runs.

A Source can optionally implement DataFetcher to power sample-row previews on asset pages; Serve detects it automatically. Test your built binary over the real wire protocol with the plugintest package.

See marmot-plugin-gcs for a complete real-world plugin, and Creating a Marmot Plugin for a step-by-step guide.

The packages

Package Contents
pluginsdk Source, DataFetcher, and Meta; the plugin-facing types (Asset, DiscoveryResult, LineageEdge, ...); config helpers (UnmarshalConfig, ApplyDefaults, ValidateStruct, GenerateConfigSpec, InterpolateTags); AWS helpers; Serve for plugin binaries and Open for hosts
filesource Resolve file paths from local disk, s3://bucket/prefix, or git:: URLs into a local directory
mrn Build and parse Marmot Resource Names (mrn://bucket/gcs/my-bucket)
plugintest End-to-end test helpers that run a built plugin binary over the wire protocol
proto The gRPC wire protocol (GetMeta/Validate/Discover/FetchSampleData); payloads are JSON so the protocol stays stable while types evolve

Config specs

GenerateConfigSpec reflects over your config struct's tags to produce the settings form Marmot renders in its UI:

  • json: field name
  • description: help text
  • label: display label (defaults to a title-cased field name)
  • validate: required marks the field required; oneof=a b c renders a dropdown
  • sensitive:"true": renders as a password field and masks the value
  • default: default value
  • show_when:"field:value": conditional visibility

The default tag pre-fills the UI form; call ApplyDefaults(config, raw) in Validate so defaults also apply to configs written by hand.

BaseConfig (embed it inline) adds the standard tags, external_links, and filter fields every plugin supports. Filtering is applied by the host after discovery; plugins only carry the config.

Protocol

The handshake pins ProtocolVersion (currently 1). Bump it on breaking changes to the wire protocol; hosts refuse plugins built against a different version. Adding an RPC is not breaking: old plugins answer it with Unimplemented. Regenerate the gRPC code after editing proto/plugin.proto:

protoc --go_out=. --go_opt=paths=source_relative \
  --go-grpc_out=. --go-grpc_opt=paths=source_relative \
  proto/plugin.proto

Documentation

Overview

Package pluginsdk is the SDK for building external Marmot plugins.

Plugins are standalone binaries that Marmot launches on demand via go-plugin and talks to over gRPC. A plugin implements the Source interface and hands it to Serve in its main function.

The types in this package mirror the JSON shapes of Marmot's core types (assets, lineage, documentation), so results cross the process boundary as JSON without plugins importing Marmot internals.

Index

Constants

View Source 
const DumpMetadataFlag = "--dump-metadata"
View Source 
const PluginSetName = "source"

PluginSetName is the key under which the source plugin is registered in the go-plugin plugin map.

View Source 
const ProtocolVersion = 1

ProtocolVersion is the Marmot plugin protocol version. Bump it on breaking changes to the wire protocol; the go-plugin handshake rejects plugins built against a different version.

View Source 
const SensitiveMask = "********"

SensitiveMask replaces sensitive values in metadata and config specs.

Variables

View Source 
var ErrEndpointNotFound = fmt.Errorf("endpoint not found")
View Source 
var Handshake = goplugin.HandshakeConfig{
	ProtocolVersion:  ProtocolVersion,
	MagicCookieKey:   "MARMOT_PLUGIN",
	MagicCookieValue: "8b1c9e2f4a7d4e6b9c3f5a8d2e7b1c4f",
}

Handshake is the shared handshake config between the Marmot host and its plugins. The magic cookie is not a security measure; it only prevents plugin binaries from being executed as normal CLIs by mistake.

Functions

func ApplyDefaults 

func ApplyDefaults(config interface{}, raw RawConfig)

ApplyDefaults sets config fields to their `default:"..."` struct tag value when the corresponding key is absent from raw. It recurses into embedded (inline) config sections. Call it from Validate after UnmarshalConfig: 

config, err := pluginsdk.UnmarshalConfig[Config](raw)
...
pluginsdk.ApplyDefaults(config, raw)

Default tag values are parsed as JSON (`default:"true"`, `default:"10"`, `default:"[\"a\",\"b\"]"`); values that are not valid JSON apply verbatim to string fields (`default:"production"`).

func GetValidator 

func GetValidator() *validator.Validate

GetValidator returns a configured validator instance

func InterpolateTags 

func InterpolateTags(tags TagsConfig, metadata map[string]interface{}) []string

InterpolateTags processes tags and replaces variables with values from metadata

func MapToMetadata 

func MapToMetadata(source interface{}) map[string]interface{}

MapToMetadata converts a struct with metadata tags to a metadata map. Fields are included when they carry a `metadata:"<key>"` tag and hold a non-zero value; fields tagged `sensitive` are masked. Nested structs and slices of structs flatten under dotted keys.

func ProcessAWSTags 

func ProcessAWSTags(tagsToMetadata bool, includeTags []string, tags map[string]string) map[string]interface{}

func Serve 

func Serve(config *ServeConfig)

Serve hands the plugin over to go-plugin and blocks until the host disconnects. Call it from the plugin's main function: 

func main() {
    pluginsdk.Serve(&pluginsdk.ServeConfig{
        Meta:   meta,
        Source: &Source{},
    })
}

func ShouldIncludeResource 

func ShouldIncludeResource(name string, filter Filter) bool

func UnmarshalConfig 

func UnmarshalConfig[T any](raw RawConfig) (*T, error)

UnmarshalConfig unmarshals a raw config into a specific plugin config type.

func ValidateStruct 

func ValidateStruct(s interface{}) error

ValidateStruct validates a struct and returns user-friendly validation errors

Types

type AWSConfig 

type AWSConfig struct {
	Credentials    AWSCredentials `json:"credentials" description:"AWS credentials configuration"`
	TagsToMetadata bool           `json:"tags_to_metadata,omitempty" description:"Convert AWS tags to Marmot metadata"`
	IncludeTags    []string       `json:"include_tags,omitempty" description:"List of AWS tags to include as metadata. By default, all tags are included."`
}

func ExtractAWSConfig 

func ExtractAWSConfig(rawConfig map[string]interface{}) (*AWSConfig, error)

func (*AWSConfig) NewAWSConfig 

func (a *AWSConfig) NewAWSConfig(ctx context.Context) (aws.Config, error)

func (*AWSConfig) Validate 

func (a *AWSConfig) Validate() error

type AWSCredentialStatus 

type AWSCredentialStatus struct {
	Available bool     `json:"available"`
	Sources   []string `json:"sources"`
	Error     string   `json:"error,omitempty"`

} // @name AWSCredentialStatus

func DetectAWSCredentials 

func DetectAWSCredentials(ctx context.Context) *AWSCredentialStatus

DetectAWSCredentials checks if AWS credentials are available from environment or config files

type AWSCredentials 

type AWSCredentials struct {
	UseDefault     bool   `` /* 127-byte string literal not displayed */
	ID             string `json:"id,omitempty" description:"AWS access key ID"`
	Secret         string `json:"secret,omitempty" description:"AWS secret access key" sensitive:"true"`
	Token          string `json:"token,omitempty" description:"AWS session token" sensitive:"true"`
	Profile        string `json:"profile,omitempty" description:"AWS profile to use from shared credentials file"`
	Role           string `json:"role,omitempty" description:"AWS IAM role ARN to assume"`
	RoleExternalID string `json:"role_external_id,omitempty" description:"External ID for cross-account role assumption"`
	Region         string `json:"region,omitempty" description:"AWS region for services"`
	Endpoint       string `json:"endpoint,omitempty" description:"Custom endpoint URL for AWS services" validate:"omitempty,url"`
}

type AWSPlugin 

type AWSPlugin struct {
	AWSConfig  `json:",inline"`
	BaseConfig `json:",inline"`
}

type Asset 

type Asset struct {
	ParentMRN     *string                `json:"parent_mrn,omitempty"`
	Name          *string                `json:"name,omitempty"`
	Description   *string                `json:"description,omitempty"`
	Type          string                 `json:"type"`
	Providers     []string               `json:"providers"`
	MRN           *string                `json:"mrn,omitempty"`
	Schema        map[string]string      `json:"schema,omitempty"`
	Metadata      map[string]interface{} `json:"metadata,omitempty"`
	Sources       []AssetSource          `json:"sources,omitempty"`
	Tags          []string               `json:"tags,omitempty"`
	Environments  map[string]Environment `json:"environments,omitempty"`
	Query         *string                `json:"query,omitempty"`
	QueryLanguage *string                `json:"query_language,omitempty"`
	ExternalLinks []AssetExternalLink    `json:"external_links,omitempty"`
}

Asset is a discovered catalog asset. It mirrors Marmot's core asset JSON shape. 

type AssetExternalLink struct {
	Name string `json:"name"`
	Icon string `json:"icon,omitempty"`
	URL  string `json:"url"`
}

AssetExternalLink is an external link attached to a single asset.

type AssetRunHistory 

type AssetRunHistory struct {
	AssetMRN string            `json:"asset_mrn"`
	Runs     []RunHistoryEvent `json:"runs"`
}

AssetRunHistory contains run history events for an asset.

type AssetSource 

type AssetSource struct {
	Name       string                 `json:"name"`
	LastSyncAt time.Time              `json:"last_sync_at"`
	Properties map[string]interface{} `json:"properties"`
	Priority   int                    `json:"priority"`
}

AssetSource records which source contributed an asset's properties.

type BaseConfig 

type BaseConfig struct {
	Tags          TagsConfig     `json:"tags,omitempty" description:"Tags to apply to discovered assets"`
	ExternalLinks []ExternalLink `json:"external_links,omitempty" description:"External links to show on all assets"`
	Filter        *Filter        `json:"filter,omitempty" description:"Filter discovered assets by name (regex)"`
}

BaseConfig holds the config fields shared by every plugin. Embed it inline in your plugin's Config struct: 

type Config struct {
    pluginsdk.BaseConfig `json:",inline"`
    ...
}

type ConfigField 

type ConfigField struct {
	Name        string        `json:"name"`
	Type        FieldType     `json:"type"`
	Label       string        `json:"label"`
	Description string        `json:"description"`
	Required    bool          `json:"required"`
	Default     interface{}   `json:"default,omitempty"`
	Options     []FieldOption `json:"options,omitempty"`
	Validation  *Validation   `json:"validation,omitempty"`
	Sensitive   bool          `json:"sensitive"`
	Placeholder string        `json:"placeholder,omitempty"`
	Fields      []ConfigField `json:"fields,omitempty"`
	IsArray     bool          `json:"is_array,omitempty"`
	ShowWhen    *ShowWhen     `json:"show_when,omitempty"`
	Hidden      bool          `json:"hidden,omitempty"`
}

func CloneConfigSpec 

func CloneConfigSpec(spec []ConfigField) []ConfigField

CloneConfigSpec deep-copies a config spec so it can be mutated without affecting the original. DeriveSpec starts from a config type and produces a fresh spec so most callers do not need this. It is exported for advanced use where a spec is derived from another spec rather than from a struct.

func DeriveSpec 

func DeriveSpec(config interface{}, opts ...SpecOption) []ConfigField

DeriveSpec builds a config spec from a plugin config struct and applies the given options in order. It is the standard way to build an alias plugin's spec, e.g. a Confluent Cloud plugin that reuses the Kafka config struct but hides TLS and pins the SASL mechanism: 

spec := pluginsdk.DeriveSpec(kafka.Config{},
    pluginsdk.Hide("tls", "authentication.type"),
    pluginsdk.Override("bootstrap_servers",
        pluginsdk.Placeholder("pkc-xxxxx.confluent.cloud:9092"),
    ),
)

DeriveSpec panics if any option references a field that does not exist in the generated spec. Alias plugins configure this at startup, so a bad dotted path is programmer error and should surface loudly rather than silently drop the override.

func GenerateConfigSpec 

func GenerateConfigSpec(configType interface{}) []ConfigField

GenerateConfigSpec builds a config spec from a plugin config struct using reflection over its json/description/validate/... struct tags.

type DataFetcher 

type DataFetcher interface {
	FetchSampleData(ctx context.Context, config RawConfig, a *Asset) (columnNames []string, rows [][]interface{}, err error)
}

DataFetcher is an optional interface a Source can implement to support fetching sample data for asset previews. It matches the semantics of Marmot's in-process DataFetcher interface. Plugins whose Source implements it advertise the capability via Meta.SupportsDataPreview.

type DiscoveryResult 

type DiscoveryResult struct {
	Assets        []Asset           `json:"assets"`
	Lineage       []LineageEdge     `json:"lineage"`
	Documentation []Documentation   `json:"documentation"`
	Statistics    []Statistic       `json:"statistics"`
	RunHistory    []AssetRunHistory `json:"run_history,omitempty"`
}

DiscoveryResult contains everything a plugin discovered in one run.

type Documentation 

type Documentation struct {
	MRN     string `json:"mrn"`
	Content string `json:"content"`
	Source  string `json:"source"`
}

Documentation is markdown documentation attached to an asset.

type Environment 

type Environment struct {
	Name     string                 `json:"name"`
	Path     string                 `json:"path"`
	Metadata map[string]interface{} `json:"metadata,omitempty"`
}

Environment describes an asset's presence in a named environment. 

type ExternalLink struct {
	Name string `json:"name" description:"Display name for the link" validate:"required"`
	Icon string `json:"icon,omitempty" description:"Icon identifier for the link"`
	URL  string `json:"url" description:"URL to the external resource" validate:"required,url"`
}

ExternalLink defines an external resource link.

type FieldOption 

type FieldOption struct {
	Label string `json:"label"`
	Value string `json:"value"`
}

type FieldOverride 

type FieldOverride func(field *ConfigField)

FieldOverride mutates a single ConfigField in place. Build them with Placeholder, Default, Required or Description and pass them to Override.

func Default 

func Default(value interface{}) FieldOverride

Default sets the default value for the field.

func Description 

func Description(text string) FieldOverride

Description overrides the field's description shown in the UI.

func Placeholder 

func Placeholder(text string) FieldOverride

Placeholder sets the placeholder text shown in the UI form.

func Required 

func Required(required bool) FieldOverride

Required marks the field as required, or clears the required flag when passed false.

type FieldType 

type FieldType string
const (
	FieldTypeString      FieldType = "string"
	FieldTypeInt         FieldType = "int"
	FieldTypeBool        FieldType = "bool"
	FieldTypeSelect      FieldType = "select"
	FieldTypeMultiselect FieldType = "multiselect"
	FieldTypePassword    FieldType = "password"
	FieldTypeObject      FieldType = "object"
)

type Filter 

type Filter struct {
	Include []string `json:"include,omitempty" description:"Include patterns for resource names (regex)"`
	Exclude []string `json:"exclude,omitempty" description:"Exclude patterns for resource names (regex)"`
}

Filter filters discovered assets by name. Filtering is applied by the Marmot host after discovery; plugins only need to carry the field.

type GCPConfig 

type GCPConfig struct {
	Credentials GCPCredentials `json:"credentials" description:"GCP credentials configuration"`
}

GCPConfig holds the Google Cloud configuration shared across plugins, mirroring AWSConfig. Embed it inline in a plugin's Config struct.

func ExtractGCPConfig 

func ExtractGCPConfig(rawConfig map[string]interface{}) (*GCPConfig, error)

ExtractGCPConfig pulls the GCP configuration out of a raw plugin config.

func (*GCPConfig) TokenSource 

func (g *GCPConfig) TokenSource(ctx context.Context, scopes ...string) (oauth2.TokenSource, error)

TokenSource returns an OAuth2 token source for the given scopes. It uses an explicit service account key (JSON content or file) when provided, otherwise Application Default Credentials (Workload Identity, a Cloud Run/GCE service account, or GOOGLE_APPLICATION_CREDENTIALS).

func (*GCPConfig) Validate 

func (g *GCPConfig) Validate() error

type GCPCredentials 

type GCPCredentials struct {
	CredentialsJSON string `json:"credentials_json,omitempty" description:"Service account key JSON content" sensitive:"true"`
	CredentialsFile string `json:"credentials_file,omitempty" description:"Path to a service account key JSON file"`
}

GCPCredentials configures how a plugin authenticates to Google Cloud. With neither field set, Application Default Credentials are used (Workload Identity, a Cloud Run/GCE service account, or GOOGLE_APPLICATION_CREDENTIALS).

type LineageEdge 

type LineageEdge struct {
	Source string `json:"source"`
	Target string `json:"target"`
	Type   string `json:"type"`
	JobMRN string `json:"job_mrn,omitempty"`
}

LineageEdge is a lineage relationship between two assets.

type Meta 

type Meta struct {
	ID                  string        `json:"id"`
	Name                string        `json:"name"`
	Description         string        `json:"description"`
	Icon                string        `json:"icon"`
	Category            string        `json:"category"`
	Status              string        `json:"status"`
	Features            []string      `json:"features,omitempty"`
	ConfigSpec          []ConfigField `json:"config_spec"`
	SupportsDataPreview bool          `json:"supports_data_preview,omitempty"`
}

Meta describes a plugin to the Marmot host: identity, display information, and the config spec used to render its settings form.

Status and Features drive documentation rendering. Status is one of "stable", "beta", or "experimental". Features is the list of asset kinds the plugin produces (e.g. "Assets", "Lineage", "Run History").

SupportsDataPreview is set by Serve when the plugin's Source implements DataFetcher; plugin authors never set it themselves.

type PluginProcess 

type PluginProcess struct {
	Source RemoteSource
	// contains filtered or unexported fields
}

PluginProcess is a handle to a running plugin process, created by the Marmot host with Open. Callers must Kill it when done; plugin processes are meant to be short-lived (open, call, kill).

func Open 

func Open(path string, logger hclog.Logger) (*PluginProcess, error)

Open launches the plugin binary at path and connects to it over gRPC.

func (*PluginProcess) Kill 

func (p *PluginProcess) Kill()

Kill terminates the plugin process.

type RawConfig 

type RawConfig map[string]interface{}

RawConfig holds the raw configuration for a plugin run. It uses a map[string]interface{} to carry arbitrary user-provided YAML/JSON.

type RemoteSource 

type RemoteSource interface {
	GetMeta(ctx context.Context) (*Meta, error)
	Validate(ctx context.Context, config RawConfig) (RawConfig, error)
	Discover(ctx context.Context, config RawConfig) (*DiscoveryResult, error)
	FetchSampleData(ctx context.Context, config RawConfig, a *Asset) ([]string, [][]interface{}, error)
}

RemoteSource is the host-facing view of a plugin process. It mirrors Source but is context-aware and exposes the plugin's metadata. FetchSampleData fails with an Unimplemented gRPC status when the plugin's Source does not implement DataFetcher; hosts should check Meta.SupportsDataPreview before calling it.

type RunHistoryEvent 

type RunHistoryEvent struct {
	RunID        string                 `json:"run_id"`
	JobNamespace string                 `json:"job_namespace"`
	JobName      string                 `json:"job_name"`
	EventType    string                 `json:"event_type"`
	EventTime    time.Time              `json:"event_time"`
	RunFacets    map[string]interface{} `json:"run_facets,omitempty"`
	JobFacets    map[string]interface{} `json:"job_facets,omitempty"`
}

RunHistoryEvent represents a single run event (START, COMPLETE, FAIL, etc.)

type SampleData 

type SampleData struct {
	ColumnNames []string        `json:"column_names"`
	Rows        [][]interface{} `json:"rows"`
}

SampleData is the wire shape of a FetchSampleData result: a set of column names and the sampled rows.

type ServeConfig 

type ServeConfig struct {
	Meta   Meta
	Source Source
}

ServeConfig holds everything a plugin binary needs to serve itself to the Marmot host.

type ShowWhen 

type ShowWhen struct {
	Field string `json:"field"`
	Value string `json:"value"`
}

type Source 

type Source interface {
	Validate(config RawConfig) (RawConfig, error)
	Discover(ctx context.Context, config RawConfig) (*DiscoveryResult, error)
}

Source is the interface a Marmot plugin implements. It matches the semantics of Marmot's in-process plugin interface: Validate checks a raw config and returns it (or an error), Discover performs asset discovery with a validated config.

type SpecOption 

type SpecOption func(spec []ConfigField) []ConfigField

SpecOption transforms a config spec. Build them with Hide and Override and pass them to DeriveSpec.

func Hide 

func Hide(names ...string) SpecOption

Hide removes fields from the spec. Nested fields use dot notation (e.g. "authentication.type"). Hiding a field that does not exist panics.

func Override 

func Override(name string, overrides ...FieldOverride) SpecOption

Override applies one or more FieldOverrides to a single field. Nested fields use dot notation (e.g. "authentication.type"). Overriding a field that does not exist panics.

type Statistic 

type Statistic struct {
	AssetMRN   string  `json:"asset_mrn"`
	MetricName string  `json:"metric_name"`
	Value      float64 `json:"value"`
}

Statistic is a single metric value for an asset.

type TagsConfig 

type TagsConfig []string

TagsConfig is a string slice that supports interpolation

type Validation 

type Validation struct {
	Pattern string `json:"pattern,omitempty"`
	Min     *int   `json:"min,omitempty"`
	Max     *int   `json:"max,omitempty"`
	MinLen  *int   `json:"min_len,omitempty"`
	MaxLen  *int   `json:"max_len,omitempty"`
}

type ValidationError 

type ValidationError struct {
	Field   string `json:"field"`
	Message string `json:"message"`
}

ValidationError represents a field-level validation error

type ValidationErrors 

type ValidationErrors struct {
	Errors []ValidationError `json:"errors"`
}

ValidationErrors is a collection of validation errors

func (ValidationErrors) Error 

func (v ValidationErrors) Error() string

Source Files

View all Source files

Directories

Path Synopsis
Package filesource resolves plugin file paths from local, S3 or Git sources.
 Package filesource resolves plugin file paths from local, S3 or Git sources.
Package mrn builds and parses Marmot Resource Names (MRNs).
 Package mrn builds and parses Marmot Resource Names (MRNs).
Package plugintest helps plugin modules test their own built binary end to end, over the same wire protocol the Marmot host uses.
 Package plugintest helps plugin modules test their own built binary end to end, over the same wire protocol the Marmot host uses.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.