Documentation
¶
Index ¶
- Constants
- Variables
- func GetEventMetadata(message *sarama.ConsumerMessage, kafka *Kafka) map[string]string
- func GetSyncProducer(config sarama.Config, brokers []string, pc ProducerConfig) (sarama.SyncProducer, error)
- type AwsClients
- type BulkEventHandler
- type EventHandler
- type Kafka
- func (k *Kafka) BulkPublish(_ context.Context, topic string, entries []pubsub.BulkMessageEntry, ...) (pubsub.BulkPublishResponse, error)
- func (k *Kafka) Close() error
- func (k *Kafka) DeserializeValue(message *sarama.ConsumerMessage, config SubscriptionHandlerConfig) ([]byte, error)
- func (k *Kafka) GetTopicHandlerConfig(topic string) (SubscriptionHandlerConfig, error)
- func (k *Kafka) Init(ctx context.Context, metadata map[string]string) error
- func (k *Kafka) Pause(ctx context.Context) error
- func (k *Kafka) Ping(ctx context.Context) error
- func (k *Kafka) Publish(_ context.Context, topic string, data []byte, metadata map[string]string) error
- func (k *Kafka) Resume(ctx context.Context) error
- func (k *Kafka) SerializeValue(topic string, data []byte, metadata map[string]string) ([]byte, error)
- func (k *Kafka) Subscribe(ctx context.Context, handlerConfig SubscriptionHandlerConfig, topics ...string)
- func (k *Kafka) ValidateAWS(metadata map[string]string) (awsAuth.Options, error)
- type KafkaBulkMessage
- type KafkaBulkMessageEntry
- type KafkaMetadata
- type KafkaOptions
- type NewEvent
- type OAuthTokenSource
- type OAuthTokenSourcePrivateKeyJWT
- type ProducerConfig
- type SaramaLogBridge
- type SchemaCacheEntry
- type SchemaType
- type SubscriptionHandlerConfig
- type TopicHandlerConfig
- type XDGSCRAMClient
Constants ¶
const ( // DefaultMaxBulkSubCount is the default max bulk count for kafka pubsub component // if the MaxBulkCountKey is not set in the metadata. DefaultMaxBulkSubCount = 80 // DefaultMaxBulkSubAwaitDurationMs is the default max bulk await duration for kafka pubsub component // if the MaxBulkAwaitDurationKey is not set in the metadata. DefaultMaxBulkSubAwaitDurationMs = 10000 )
Variables ¶
var ( SHA256 scram.HashGeneratorFcn = sha256.New SHA512 scram.HashGeneratorFcn = sha512.New )
Functions ¶
func GetEventMetadata ¶
func GetEventMetadata(message *sarama.ConsumerMessage, kafka *Kafka) map[string]string
func GetSyncProducer ¶ added in v1.15.0
func GetSyncProducer(config sarama.Config, brokers []string, pc ProducerConfig) (sarama.SyncProducer, error)
GetSyncProducer creates a new Sarama SyncProducer using the provided base config and producer tunables. RequiredAcks and RetryMax are applied from ProducerConfig so callers can override the previously hard-coded defaults (WaitForAll, 5 retries) via component metadata.
Types ¶
type AwsClients ¶ added in v1.16.0
type AwsClients struct {
ConsumerGroup sarama.ConsumerGroup
Producer sarama.SyncProducer
// contains filtered or unexported fields
}
func InitAwsClients ¶ added in v1.16.0
func InitAwsClients(opts KafkaOptions) *AwsClients
type BulkEventHandler ¶
type BulkEventHandler func(ctx context.Context, msg *KafkaBulkMessage) ([]pubsub.BulkSubscribeResponseEntry, error)
BulkEventHandler is the handler used to handle the subscribed bulk event.
type EventHandler ¶
EventHandler is the handler used to handle the subscribed event.
type Kafka ¶
type Kafka struct {
// The default value should be true for kafka pubsub component and false for kafka binding component
// This default value can be overridden by metadata consumeRetryEnabled
DefaultConsumeRetryEnabled bool
// contains filtered or unexported fields
}
Kafka allows reading/writing to a Kafka consumer group.
func (*Kafka) BulkPublish ¶
func (*Kafka) DeserializeValue ¶
func (k *Kafka) DeserializeValue(message *sarama.ConsumerMessage, config SubscriptionHandlerConfig) ([]byte, error)
func (*Kafka) GetTopicHandlerConfig ¶
func (k *Kafka) GetTopicHandlerConfig(topic string) (SubscriptionHandlerConfig, error)
GetTopicBulkHandler returns the handlerConfig for a topic
func (*Kafka) Pause ¶ added in v1.17.4
Pause stops fetching new messages from the broker for all active subscriptions on this component. The Sarama session and partition assignments are preserved so messages already buffered in claim queues can still be processed by handlers. Idempotent.
func (*Kafka) Ping ¶ added in v1.18.2
Ping performs a connectivity check against the configured Kafka brokers by opening a short-lived Sarama client, issuing a single metadata request, and then closing it immediately.
Note the metadata request is cluster-wide: RefreshMetadata() called with no topics refreshes metadata for all topics regardless of Metadata.Full, so on clusters with very many topics this is not a cheap probe. Metadata.Full=false only stops sarama.NewClient from performing its own construction-time fetch; the explicit RefreshMetadata() call below is what actually probes connectivity.
The check is fully bounded and cancellable:
- Pass a context with a deadline to enforce a wall-clock limit. When a deadline is present, all Net and Metadata timeouts are clamped to the remaining time so the call cannot block beyond the deadline.
- Metadata retries are always disabled for the probe so a failure returns promptly instead of retrying against an unhealthy cluster (Sarama would otherwise apply its default of 3 retries, adding tens of seconds when no deadline is supplied).
- Context cancellation is honoured mid-flight: sarama.NewClient is run in a goroutine and the function returns ctx.Err() as soon as ctx is done. Any partially-opened client is closed in the background to avoid leaks.
Ping does NOT modify any internal component state and is safe to call from concurrent goroutines after Init() has completed.
func (*Kafka) Publish ¶
func (k *Kafka) Publish(_ context.Context, topic string, data []byte, metadata map[string]string) error
Publish message to Kafka cluster.
func (*Kafka) Resume ¶ added in v1.17.4
Resume resumes fetching new messages from the broker for all active subscriptions on this component. Idempotent.
func (*Kafka) SerializeValue ¶
type KafkaBulkMessage ¶
type KafkaBulkMessage struct {
Entries []KafkaBulkMessageEntry `json:"entries"`
Topic string `json:"topic"`
Metadata map[string]string `json:"metadata"`
}
KafkaBulkMessage is a bulk event arriving from a message bus instance.
type KafkaBulkMessageEntry ¶
type KafkaBulkMessageEntry struct {
EntryId string `json:"entryId"` //nolint:stylecheck
Event []byte `json:"event"`
ContentType string `json:"contentType,omitempty"`
Metadata map[string]string `json:"metadata"`
}
KafkaBulkMessageEntry is an item contained inside bulk event arriving from a message bus instance.
type KafkaMetadata ¶
type KafkaMetadata struct {
Brokers string `mapstructure:"brokers"`
ConsumerGroup string `mapstructure:"consumerGroup"`
ClientID string `mapstructure:"clientId"`
AuthType string `mapstructure:"authType"`
SaslUsername string `mapstructure:"saslUsername"`
SaslPassword string `mapstructure:"saslPassword"`
SaslMechanism string `mapstructure:"saslMechanism"`
InitialOffset string `mapstructure:"initialOffset"`
MaxMessageBytes int `mapstructure:"maxMessageBytes"`
OidcTokenEndpoint string `mapstructure:"oidcTokenEndpoint"`
OidcClientID string `mapstructure:"oidcClientID"`
OidcClientSecret string `mapstructure:"oidcClientSecret"`
OidcScopes string `mapstructure:"oidcScopes"`
OidcExtensions string `mapstructure:"oidcExtensions"`
OidcClientAssertionCert string `mapstructure:"oidcClientAssertionCert"`
OidcClientAssertionKey string `mapstructure:"oidcClientAssertionKey"`
OidcResource string `mapstructure:"oidcResource"`
OidcAudience string `mapstructure:"oidcAudience"`
OidcKid string `mapstructure:"oidcKid"`
TLSDisable bool `mapstructure:"disableTls"`
TLSSkipVerify bool `mapstructure:"skipVerify"`
TLSCaCert string `mapstructure:"caCert"`
TLSClientCert string `mapstructure:"clientCert"`
TLSClientKey string `mapstructure:"clientKey"`
ConsumeRetryEnabled bool `mapstructure:"consumeRetryEnabled"`
ConsumeRetryInterval time.Duration `mapstructure:"consumeRetryInterval"`
HeartbeatInterval time.Duration `mapstructure:"heartbeatInterval"`
SessionTimeout time.Duration `mapstructure:"sessionTimeout"`
Version string `mapstructure:"version"`
EscapeHeaders bool `mapstructure:"escapeHeaders"`
// configs for kafka client
ClientConnectionTopicMetadataRefreshInterval time.Duration `mapstructure:"clientConnectionTopicMetadataRefreshInterval"`
ClientConnectionKeepAliveInterval time.Duration `mapstructure:"clientConnectionKeepAliveInterval"`
// Net timeout tunables.
// Defaults preserve Sarama's built-in values (30 s each) so unset
// deployments behave identically to previous releases.
DialTimeout time.Duration `mapstructure:"dialTimeout"`
ReadTimeout time.Duration `mapstructure:"readTimeout"`
WriteTimeout time.Duration `mapstructure:"writeTimeout"`
// MetadataTimeout is the per-request metadata timeout passed to
// Sarama's Metadata.Timeout. When 0 (default) Sarama's own default
// applies (disabled, computed from Net timeouts × retry count).
MetadataTimeout time.Duration `mapstructure:"metadataTimeout"`
ConsumerGroupRebalanceStrategy string `mapstructure:"consumerGroupRebalanceStrategy"`
// configs for kafka producer
Compression string `mapstructure:"compression"`
// Producer tunables.
// ProducerRequiredAcks accepts "all" (default), "local", or "none".
ProducerRequiredAcks string `mapstructure:"producerRequiredAcks"`
// ProducerRetryMax is the number of times to retry sending a message (default 5).
ProducerRetryMax int `mapstructure:"producerRetryMax"`
// schema registry
SchemaRegistryURL string `mapstructure:"schemaRegistryURL"`
SchemaRegistryAPIKey string `mapstructure:"schemaRegistryAPIKey"`
SchemaRegistryAPISecret string `mapstructure:"schemaRegistryAPISecret"`
SchemaCachingEnabled bool `mapstructure:"schemaCachingEnabled"`
SchemaLatestVersionCacheTTL time.Duration `mapstructure:"schemaLatestVersionCacheTTL"`
UseAvroJSON bool `mapstructure:"useAvroJSON"`
// header from/to metadata excluded keys regex
ExcludeHeaderMetaRegex string `mapstructure:"excludeHeaderMetaRegex"`
// contains filtered or unexported fields
}
type KafkaOptions ¶ added in v1.16.0
type KafkaOptions struct {
Config *sarama.Config
ConsumerGroup string
Brokers []string
ProducerConfig ProducerConfig
}
type NewEvent ¶
type NewEvent struct {
Data []byte `json:"data"`
Topic string `json:"topic"`
Metadata map[string]string `json:"metadata"`
ContentType *string `json:"contentType,omitempty"`
}
NewEvent is an event arriving from a message bus instance.
type OAuthTokenSource ¶
type OAuthTokenSource struct {
CachedToken oauth2.Token
Extensions map[string]string
TokenEndpoint oauth2.Endpoint
ClientID string
ClientSecret string
Scopes []string
// contains filtered or unexported fields
}
func (*OAuthTokenSource) Token ¶
func (ts *OAuthTokenSource) Token() (*sarama.AccessToken, error)
type OAuthTokenSourcePrivateKeyJWT ¶ added in v1.17.0
type OAuthTokenSourcePrivateKeyJWT struct {
CachedToken oauth2.Token
Extensions map[string]string
TokenEndpoint oauth2.Endpoint
ClientID string
ClientSecret string
Scopes []string
ClientAuthMethod string
ClientAssertionCert string
ClientAssertionKey string
Resource string
Audience string
Kid string
// contains filtered or unexported fields
}
func (*OAuthTokenSourcePrivateKeyJWT) Token ¶ added in v1.17.0
func (ts *OAuthTokenSourcePrivateKeyJWT) Token() (*sarama.AccessToken, error)
At the time of writing this, the oauth2 package does not support the client assertion authentication method. Ref: https://github.com/golang/oauth2/issues/744
type ProducerConfig ¶ added in v1.18.2
type ProducerConfig struct {
RequiredAcks sarama.RequiredAcks
RetryMax int
MaxMessageBytes int
}
ProducerConfig holds the producer-specific tunables derived from component metadata. Separating them avoids threading the entire KafkaMetadata struct through callers that only have access to the sarama.Config copy.
type SaramaLogBridge ¶
type SaramaLogBridge struct {
// contains filtered or unexported fields
}
func (SaramaLogBridge) Print ¶
func (b SaramaLogBridge) Print(v ...interface{})
func (SaramaLogBridge) Printf ¶
func (b SaramaLogBridge) Printf(format string, v ...interface{})
func (SaramaLogBridge) Println ¶
func (b SaramaLogBridge) Println(v ...interface{})
type SchemaCacheEntry ¶
type SchemaCacheEntry struct {
// contains filtered or unexported fields
}
type SchemaType ¶
type SchemaType int
const ( None SchemaType = iota Avro )
func GetValueSchemaType ¶
func GetValueSchemaType(metadata map[string]string) (SchemaType, error)
type SubscriptionHandlerConfig ¶
type SubscriptionHandlerConfig struct {
IsBulkSubscribe bool
SubscribeConfig pubsub.BulkSubscribeConfig
BulkHandler BulkEventHandler
Handler EventHandler
ValueSchemaType SchemaType
}
SubscriptionHandlerConfig is the handler and configuration for subscription.
type TopicHandlerConfig ¶
type TopicHandlerConfig map[string]SubscriptionHandlerConfig
TopicHandlerConfig is the map of topics and sruct containing handler and their config.
func (TopicHandlerConfig) TopicList ¶
func (tbh TopicHandlerConfig) TopicList() []string
TopicList returns the list of topics
type XDGSCRAMClient ¶
type XDGSCRAMClient struct {
*scram.Client
*scram.ClientConversation
scram.HashGeneratorFcn
}
func (*XDGSCRAMClient) Begin ¶
func (x *XDGSCRAMClient) Begin(userName, password, authzID string) (err error)
func (*XDGSCRAMClient) Done ¶
func (x *XDGSCRAMClient) Done() bool