kafka

package module
v1.13.7 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Nov 12, 2026 License: Apache-2.0 Imports: 18 Imported by: 0

README

Forge Kafka, an Apache Kafka producer, consumer and consumer group client for Go applications built on Forge

Forge Kafka: Apache Kafka producer and consumer for Go

Forge Kafka is the Apache Kafka extension for Forge, the Go backend framework. You add it to your app and get a Kafka client in the container, with sync and async producers, partition consumers, consumer groups, topic administration, TLS and SASL (PLAIN, SCRAM-SHA-256, SCRAM-SHA-512) authentication. It is built on IBM Sarama, and you can reach the raw Sarama client, producers and consumers whenever you need something the wrapper does not cover.

Go CI Go Reference Apache 2.0

Documentation · Quick start · Features · Producing · Consuming · Topics · Security · Configuration · FAQ

Quick start

You need Go 1.26 or later and a Kafka broker. Add the extension to your module:

go get github.com/xraph/forge@latest
go get github.com/xraph/forge/extensions/kafka@latest
go get github.com/IBM/sarama

This app publishes the body of every POST /orders to an orders topic and logs each message it reads back:

package main

import (
	"context"
	"io"
	"log"
	"net/http"

	"github.com/IBM/sarama"
	"github.com/xraph/forge"
	"github.com/xraph/forge/extensions/kafka"
)

func main() {
	app := forge.NewApp(forge.AppConfig{
		Name:        "orders",
		HTTPAddress: ":8080",
		Extensions: []forge.Extension{
			kafka.NewExtension(
				kafka.WithBrokers("localhost:9092"),
				kafka.WithClientID("orders"),
			),
		},
	})

	// Start reading the topic once every extension is up.
	err := app.RegisterHookFn(forge.PhaseAfterStart, "consume-orders",
		func(ctx context.Context, a forge.App) error {
			k, err := forge.Inject[*kafka.KafkaService](a.Container())
			if err != nil {
				return err
			}

			return k.Consume(ctx, []string{"orders"}, func(msg *sarama.ConsumerMessage) error {
				log.Printf("partition %d offset %d: %s", msg.Partition, msg.Offset, msg.Value)

				return nil
			})
		})
	if err != nil {
		log.Fatal(err)
	}

	// POST a body and it lands on the "orders" topic.
	err = app.Router().POST("/orders", func(ctx forge.Context) error {
		k, err := forge.Inject[*kafka.KafkaService](app.Container())
		if err != nil {
			return err
		}

		body, err := io.ReadAll(ctx.Request().Body)
		if err != nil {
			return err
		}

		if err := k.SendMessage("orders", nil, body); err != nil {
			return err
		}

		return ctx.JSON(http.StatusAccepted, map[string]string{"status": "queued"})
	})
	if err != nil {
		log.Fatal(err)
	}

	if err := app.Run(); err != nil {
		log.Fatal(err)
	}
}

If you don't have a broker running, start a single-node one with Docker and create the topic:

docker run -d --name kafka -p 9092:9092 apache/kafka:latest
docker exec kafka /opt/kafka/bin/kafka-topics.sh \
  --bootstrap-server localhost:9092 --create --topic orders

Then run the app and post something:

go mod tidy
go run .

# in another terminal
curl -X POST localhost:8080/orders -d '{"id": 42}'

The app's log shows the message coming back from the orders topic. If no broker answers on localhost:9092, the after-start hook can't create the client, so app.Run() returns that error and nothing listens on port 8080.

Features

You need Forge Kafka gives you
To publish events from handlers SendMessage (waits for the broker), SendMessageAsync (fire and forget) and SendMessages for a batch, each with a *Context twin
To read a topic Consume across every partition, or ConsumePartition from an offset you choose, with ConsumeContext and ConsumePartitionContext for handlers that take a context
Load-balanced consumers with commits JoinConsumerGroup with your own Sarama ConsumerGroupHandler
Topic setup from code CreateTopic, DeleteTopic, ListTopics, DescribeTopic, GetPartitions, GetOffset
Secure clusters TLS with client certificates and a custom CA, SASL PLAIN and SCRAM
Throughput tuning gzip, snappy, lz4 and zstd compression, flush batching, acks and fetch sizes
Visibility Forge metrics per topic and client counters from GetStats
Anything Sarama can do GetClient, GetProducer, GetAsyncProducer, GetConsumer, GetConsumerGroup

The extension registers one service, *kafka.KafkaService, by type and under the alias kafka. It implements the kafka.Kafka interface, so you can pass it to code that only needs the interface. Resolve it wherever you have a container:

k, err := forge.Inject[*kafka.KafkaService](app.Container())

// The same service, by its alias:
k, err = forge.Resolve[*kafka.KafkaService](app.Container(), "kafka")

The service is built the first time something resolves it. That's when the client connects, creates the producers and consumer you enabled, and pings the brokers, so the first Inject is where a connection error shows up. When the app stops, the container closes every producer, consumer and consumer group, then the client.

Produce messages

When you call SendMessage, it goes through Sarama's sync producer and returns once the broker acknowledges the write (producer_acks decides how many replicas have to acknowledge it), so a failed write comes back to you as an error in the handler that sent it.

err := k.SendMessage("orders", []byte("customer-17"), payload)

SendMessageAsync drops the message on the async producer's input channel and returns. Delivery errors don't come back to you: the extension logs them and counts them in GetStats().Errors.

err := k.SendMessageAsync("clicks", nil, payload)

SendMessages sends a batch through the sync producer. Each ProducerMessage carries its own topic, key, value and partition:

err := k.SendMessages([]*kafka.ProducerMessage{
	{Topic: "orders", Key: []byte("customer-17"), Value: first},
	{Topic: "orders", Key: []byte("customer-17"), Value: second},
})

SendMessages sends each message's Headers and Timestamp along with the rest. A zero Timestamp leaves the choice to the producer, as it does for SendMessage. The Partition you set is copied but not honoured: Sarama's default hash partitioner picks the partition from the key, and only a manual partitioner reads the field.

Each of the three has a context-aware twin: SendMessageContext, SendMessageAsyncContext and SendMessagesContext. They take the ctx of the request you're serving as their first argument, and the plain methods call them with context.Background(). Use the twins in handler code: they're where the context you pass in gets picked up.

err := k.SendMessageContext(ctx, "orders", []byte("customer-17"), payload)

Sarama's default partitioner hashes the key. If you give related events the same key, they land on one partition and you read them back in the order you sent them.

Consume messages

You have two ways to read a topic, and they behave differently.

Consume and ConsumePartition read partitions directly, with no consumer group and no committed offsets. Consume finds every partition of each topic and starts reading from the newest offset, so it only sees messages produced after it starts. ConsumePartition lets you pick the offset, including sarama.OffsetOldest:

err := k.ConsumePartition(ctx, "orders", 0, sarama.OffsetOldest,
	func(msg *sarama.ConsumerMessage) error {
		return process(msg.Value)
	})

If your handler needs a context, use ConsumeContext and ConsumePartitionContext. They take a kafka.ContextMessageHandler, func(ctx context.Context, msg *sarama.ConsumerMessage) error, and pass it a context derived from the one you started the consumer with:

err := k.ConsumePartitionContext(ctx, "orders", 0, sarama.OffsetOldest,
	func(ctx context.Context, msg *sarama.ConsumerMessage) error {
		return process(ctx, msg.Value)
	})

Both return as soon as the partition readers are running. Each partition gets its own goroutine. If your handler returns an error, the extension logs it with the topic, partition and offset and moves on to the next message, so nothing is retried and you decide what a failed message means for your service. To stop reading, cancel the context you passed in. k.StopConsume() only cancels the readers started by the most recent Consume call, so it won't stop a ConsumePartition reader, or the readers from an earlier Consume.

JoinConsumerGroup is the one you want when several instances of a service share a topic, or when a restart should pick up where it left off. You write a Sarama ConsumerGroupHandler, and you mark messages as processed yourself:

type orderHandler struct{}

func (orderHandler) Setup(sarama.ConsumerGroupSession) error   { return nil }
func (orderHandler) Cleanup(sarama.ConsumerGroupSession) error { return nil }

func (orderHandler) ConsumeClaim(s sarama.ConsumerGroupSession, c sarama.ConsumerGroupClaim) error {
	for msg := range c.Messages() {
		if err := process(msg.Value); err != nil {
			return err
		}

		s.MarkMessage(msg, "")
	}

	return nil
}

err := k.JoinConsumerGroup(ctx, "order-workers", []string{"orders"}, orderHandler{})

The call returns once the group's consume loop is running in the background. A new group with no committed offsets starts from consumer_offsets (newest or oldest), and partitions are assigned with consumer_group_rebalance (range, roundrobin or sticky). One client joins one group at a time. Call k.LeaveConsumerGroup(ctx) to leave it before joining another.

Tracing

With enable_tracing on (it is by default), every send starts a producer span and writes the trace context into the record's headers as traceparent, plus tracestate when there is one and baggage when the context has any. SendMessage and SendMessageAsync get a send {topic} span. The async span covers the hand-off to the producer only, because delivery errors show up later on the producer's own error channel.

SendMessages gets one create {topic} span per record, injected into that record, and a single send {topic} span that links to all of them. A batch that spans several topics gets a plain send span with no destination. Pass the ctx of the request you're serving to the *Context methods and these spans join its trace.

On the consuming side, Consume, ConsumePartition and their *Context twins run every handler inside a process {topic} span that continues the producer's trace. A record with no traceparent header starts a new trace. If your handler takes a context, that context carries the consumer span, so anything you start in it is a child. A handler without one still gets the span and the timing.

Consumer groups are your own ConsumerGroupHandler, so the extension can't wrap them. Use kafka.ConsumerHeaderCarrier with forge/messaging to do the same thing in ConsumeClaim:

func (h handler) ConsumeClaim(sess sarama.ConsumerGroupSession, claim sarama.ConsumerGroupClaim) error {
	mt := messaging.New("kafka", messaging.WithMetrics(h.metrics))

	for msg := range claim.Messages() {
		err := mt.Process(sess.Context(), kafka.ConsumerHeaderCarrier(msg), msg.Topic, "process",
			func(ctx context.Context) error { return h.handle(ctx, msg) })
		if err == nil {
			sess.MarkMessage(msg, "")
		}
	}

	return nil
}

kafka.ProducerHeaderCarrier is the same adapter for an outgoing *sarama.ProducerMessage, if you build records yourself and send them through GetProducer().

Manage topics

err := k.CreateTopic("orders", kafka.TopicConfig{
	NumPartitions:     6,
	ReplicationFactor: 3,
})

topics, err := k.ListTopics()
meta, err := k.DescribeTopic("orders")         // partitions, leaders, replicas
parts, err := k.GetPartitions("orders")
next, err := k.GetOffset("orders", 0, sarama.OffsetNewest)
err = k.DeleteTopic("orders")

CreateTopic and DeleteTopic send the admin request to the first broker in the client's list. TopicConfig.ConfigEntries passes topic settings such as retention.ms straight through.

TLS and SASL

kafka.NewExtension(
	kafka.WithBrokers("broker-1:9093", "broker-2:9093"),
	kafka.WithTLS("/etc/kafka/client.pem", "/etc/kafka/client-key.pem", "/etc/kafka/ca.pem", false),
	kafka.WithSASL("SCRAM-SHA-512", "orders-service", os.Getenv("KAFKA_PASSWORD")),
)

WithTLS takes a client certificate, its key, a CA file and a skip-verify flag. With verification on, the config check requires both the certificate and the key, so a TLS-only setup without a client certificate doesn't pass validation today. WithSASL accepts PLAIN, SCRAM-SHA-256 or SCRAM-SHA-512, and needs a username and password.

Configuration

Configure the extension in code. Pass options to NewExtension, or start from kafka.DefaultConfig(), change the fields you need and hand the struct to WithConfig (or NewExtensionWithConfig):

cfg := kafka.DefaultConfig()
cfg.Brokers = []string{"broker-1:9092", "broker-2:9092"}
cfg.ClientID = "orders"
cfg.ProducerCompression = "lz4"
cfg.ProducerAcks = "all"
cfg.ConsumerOffsets = "oldest"
cfg.ConsumerGroupRebalance = "sticky"

ext := kafka.NewExtension(kafka.WithConfig(cfg))

The options cover the common fields: WithBrokers, WithClientID, WithVersion, WithTLS, WithSASL, WithProducer, WithConsumer, WithConsumerGroup, WithCompression, WithIdempotent, WithMetrics, WithTracing, WithRequireConfig and WithConfig.

The extension also reads extensions.kafka from your config file, falling back to a top-level kafka key, and WithRequireConfig(true) turns a missing section into a startup error. The file overrides the defaults and your options override the file, so brokers or sasl_mechanism in YAML takes effect with a plain kafka.NewExtension().

These are the fields of kafka.Config, with their YAML names:

Key Default What it does
brokers ["localhost:9092"] Seed brokers
client_id forge-kafka-client Client ID sent to the brokers
version 3.0.0 Kafka protocol version Sarama speaks
dial_timeout, read_timeout, write_timeout 30s each Network timeouts
producer_enabled true Create the sync and async producers
producer_compression snappy none, gzip, snappy, lz4 or zstd
producer_acks local none, local (leader only) or all
producer_flush_messages, producer_flush_frequency 10, 100ms Batch triggers for the producers
producer_retry_max 3 Send retries
producer_max_message_bytes 1000000 Largest message the producer accepts
producer_idempotent false Idempotent producer (see the FAQ first)
consumer_enabled true Create the partition consumer
consumer_offsets newest Where a new consumer group starts: newest or oldest
consumer_isolation read_uncommitted read_uncommitted or read_committed
consumer_max_wait, consumer_fetch_min, consumer_fetch_max 500ms, 1, 1MiB Fetch tuning
consumer_group_rebalance range range, roundrobin or sticky
consumer_group_session, consumer_group_heartbeat 10s, 3s Group session timeout and heartbeat
metadata_retry_max, metadata_retry_backoff, metadata_refresh_freq 3, 250ms, 10m Metadata refresh
enable_tls, tls_cert_file, tls_key_file, tls_ca_file, tls_skip_verify off TLS
enable_sasl, sasl_mechanism, sasl_username, sasl_password off, PLAIN SASL
enable_metrics true Record the metrics below
enable_tracing true Start a send span for each send and a process span around each handler, and carry traceparent in the record headers. Off means no spans; the headers still pass the trace through
enable_logging true Log failed and successful sync sends

Unknown values for compression, acks, offsets, isolation or rebalance fail validation when the extension registers, so a typo stops the app at startup. consumer_group_id is stored on the config but nothing reads it yet: the group ID you pass to JoinConsumerGroup is the one that counts.

With enable_metrics on, the extension records two duration histograms, messaging.client.operation.duration for sends and messaging.process.duration for handlers. A scrape spells them messaging_client_operation_duration_seconds and messaging_process_duration_seconds, and they carry messaging_system="kafka" along with the destination, operation and error type. It also keeps the kafka.bytes.sent and kafka.bytes.received gauges (the size of the last message), labelled by topic.

The kafka.messages.sent and kafka.messages.received counters are gone. The _count series of the two histograms replaces them, so a query like sum(rate(messaging_process_duration_seconds_count{messaging_system="kafka"}[5m])) gives you the receive rate.

GetStats() returns running totals for the client: messages and bytes in both directions, error count and the last error.

Documentation

Topic Where
Overview and what the extension registers Kafka extension docs
Producers, consumers, groups and admin Features
Every config field and option Configuration
Method reference Functions and pkg.go.dev
How extensions load config and join the lifecycle Forge extension system

FAQ

How do I use Kafka in a Go web app built with Forge?

Add kafka.NewExtension(...) to forge.AppConfig.Extensions, then resolve *kafka.KafkaService with forge.Inject from a handler or a lifecycle hook, as in the quick start. The client is shared, so you don't open a connection per request.

Does Consume commit offsets?

No. Consume and ConsumePartition read partitions without a consumer group and never commit, and Consume always starts at the newest offset. If a restarted service should continue where it stopped, or several instances should split a topic, use JoinConsumerGroup and call MarkMessage in your handler.

Can I turn on the idempotent producer?

Not yet. Sarama requires Net.MaxOpenRequests to be 1 for an idempotent producer, and the extension leaves it at Sarama's default of 5, so WithIdempotent(true) makes client creation fail. Keep it off for now.

What happens if Kafka is down when my app starts?

The app itself starts, because the client is only created when something first resolves *kafka.KafkaService. That forge.Inject call returns the connection error. If you resolve it in an after-start hook, as the quick start does, the hook's error stops app.Run(), which is usually what you want.

Which Kafka versions does it support?

Whatever Sarama supports. Set version to your broker's version (it defaults to 3.0.0) so Sarama uses the right protocol features.

Contributing

The extension is its own Go module inside the Forge repository. Test it from its directory:

cd extensions/kafka
GOWORK=off go build ./...
GOWORK=off go vet ./...
GOWORK=off go test -short -count=1 ./...

The tests don't need a broker. Run golangci-lint run ./... from the same directory before you open a pull request, and see AGENTS.md for the toolchain, lint rules and commit format.

License

Forge Kafka is part of Forge and is available under the Apache License 2.0.

Documentation

Index

Constants

View Source
const (
	// ServiceKey is the DI key for the Kafka service.
	ServiceKey = "kafka"
)

DI container keys for Kafka extension services.

Variables

View Source
var (
	// ErrClientNotInitialized is returned when client is not initialized.
	ErrClientNotInitialized = errors.New("kafka: client not initialized")

	// ErrProducerNotEnabled is returned when producer operations are attempted but producer is disabled.
	ErrProducerNotEnabled = errors.New("kafka: producer not enabled")

	// ErrConsumerNotEnabled is returned when consumer operations are attempted but consumer is disabled.
	ErrConsumerNotEnabled = errors.New("kafka: consumer not enabled")

	// ErrAlreadyConsuming is returned when attempting to start consuming while already consuming.
	ErrAlreadyConsuming = errors.New("kafka: already consuming")

	// ErrNotConsuming is returned when attempting to stop consuming while not consuming.
	ErrNotConsuming = errors.New("kafka: not consuming")

	// ErrInConsumerGroup is returned when attempting consumer operations while in consumer group.
	ErrInConsumerGroup = errors.New("kafka: already in consumer group")

	// ErrNotInConsumerGroup is returned when attempting to leave consumer group while not in one.
	ErrNotInConsumerGroup = errors.New("kafka: not in consumer group")

	// ErrSendFailed is returned when message send fails.
	ErrSendFailed = errors.New("kafka: send failed")

	// ErrConsumeFailed is returned when message consumption fails.
	ErrConsumeFailed = errors.New("kafka: consume failed")

	// ErrTopicNotFound is returned when topic doesn't exist.
	ErrTopicNotFound = errors.New("kafka: topic not found")

	// ErrInvalidPartition is returned when partition is invalid.
	ErrInvalidPartition = errors.New("kafka: invalid partition")

	// ErrConnectionFailed is returned when connection fails.
	ErrConnectionFailed = errors.New("kafka: connection failed")

	// ErrClientClosed is returned when operations are attempted on closed client.
	ErrClientClosed = errors.New("kafka: client closed")
)

Functions

func ConsumerHeaderCarrier added in v1.12.9

func ConsumerHeaderCarrier(msg *sarama.ConsumerMessage) propagation.TextMapCarrier

ConsumerHeaderCarrier adapts a consumed record's headers to a TextMapCarrier. Use it in consumer-group handlers you write yourself:

err := mt.Process(sess.Context(), kafka.ConsumerHeaderCarrier(msg), msg.Topic, "process", handle)

func NewExtension

func NewExtension(opts ...ConfigOption) forge.Extension

NewExtension creates a new Kafka extension.

func NewExtensionWithConfig

func NewExtensionWithConfig(config Config) forge.Extension

NewExtensionWithConfig creates a new Kafka extension with a complete config.

func ProducerHeaderCarrier added in v1.12.9

func ProducerHeaderCarrier(msg *sarama.ProducerMessage) propagation.TextMapCarrier

ProducerHeaderCarrier adapts an outgoing record's headers to a TextMapCarrier.

Types

type ClientStats

type ClientStats struct {
	Connected        bool
	ConnectTime      time.Time
	MessagesSent     int64
	MessagesReceived int64
	BytesSent        int64
	BytesReceived    int64
	Errors           int64
	LastError        error
	LastErrorTime    time.Time
	ActiveConsumers  int
	ActiveProducers  int
}

ClientStats contains client statistics.

type Config

type Config struct {
	// Connection settings
	Brokers      []string      `json:"brokers"       mapstructure:"brokers"       yaml:"brokers"`
	ClientID     string        `json:"client_id"     mapstructure:"client_id"     yaml:"client_id"`
	Version      string        `json:"version"       mapstructure:"version"       yaml:"version"` // Kafka version (e.g., "3.0.0")
	DialTimeout  time.Duration `json:"dial_timeout"  mapstructure:"dial_timeout"  yaml:"dial_timeout"`
	ReadTimeout  time.Duration `json:"read_timeout"  mapstructure:"read_timeout"  yaml:"read_timeout"`
	WriteTimeout time.Duration `json:"write_timeout" mapstructure:"write_timeout" yaml:"write_timeout"`
	KeepAlive    time.Duration `json:"keep_alive"    mapstructure:"keep_alive"    yaml:"keep_alive"`

	// TLS/SASL
	EnableTLS     bool   `json:"enable_tls"              mapstructure:"enable_tls"      yaml:"enable_tls"`
	TLSCertFile   string `json:"tls_cert_file,omitempty" mapstructure:"tls_cert_file"   yaml:"tls_cert_file,omitempty"`
	TLSKeyFile    string `json:"tls_key_file,omitempty"  mapstructure:"tls_key_file"    yaml:"tls_key_file,omitempty"`
	TLSCAFile     string `json:"tls_ca_file,omitempty"   mapstructure:"tls_ca_file"     yaml:"tls_ca_file,omitempty"`
	TLSSkipVerify bool   `json:"tls_skip_verify"         mapstructure:"tls_skip_verify" yaml:"tls_skip_verify"`

	EnableSASL    bool   `json:"enable_sasl"              mapstructure:"enable_sasl"    yaml:"enable_sasl"`
	SASLMechanism string `json:"sasl_mechanism,omitempty" mapstructure:"sasl_mechanism" yaml:"sasl_mechanism,omitempty"` // PLAIN, SCRAM-SHA-256, SCRAM-SHA-512
	SASLUsername  string `json:"sasl_username,omitempty"  mapstructure:"sasl_username"  yaml:"sasl_username,omitempty"`
	SASLPassword  string `json:"sasl_password,omitempty"  mapstructure:"sasl_password"  yaml:"sasl_password,omitempty"`

	// Producer settings
	ProducerEnabled         bool          `json:"producer_enabled"           mapstructure:"producer_enabled"           yaml:"producer_enabled"`
	ProducerMaxMessageBytes int           `json:"producer_max_message_bytes" mapstructure:"producer_max_message_bytes" yaml:"producer_max_message_bytes"`
	ProducerCompression     string        `json:"producer_compression"       mapstructure:"producer_compression"       yaml:"producer_compression"` // none, gzip, snappy, lz4, zstd
	ProducerFlushMessages   int           `json:"producer_flush_messages"    mapstructure:"producer_flush_messages"    yaml:"producer_flush_messages"`
	ProducerFlushFrequency  time.Duration `json:"producer_flush_frequency"   mapstructure:"producer_flush_frequency"   yaml:"producer_flush_frequency"`
	ProducerRetryMax        int           `json:"producer_retry_max"         mapstructure:"producer_retry_max"         yaml:"producer_retry_max"`
	ProducerIdempotent      bool          `json:"producer_idempotent"        mapstructure:"producer_idempotent"        yaml:"producer_idempotent"`
	ProducerAcks            string        `json:"producer_acks"              mapstructure:"producer_acks"              yaml:"producer_acks"` // none, local, all

	// Consumer settings
	ConsumerEnabled   bool          `json:"consumer_enabled"            mapstructure:"consumer_enabled"   yaml:"consumer_enabled"`
	ConsumerGroupID   string        `json:"consumer_group_id,omitempty" mapstructure:"consumer_group_id"  yaml:"consumer_group_id,omitempty"`
	ConsumerOffsets   string        `json:"consumer_offsets"            mapstructure:"consumer_offsets"   yaml:"consumer_offsets"` // newest, oldest
	ConsumerMaxWait   time.Duration `json:"consumer_max_wait"           mapstructure:"consumer_max_wait"  yaml:"consumer_max_wait"`
	ConsumerFetchMin  int32         `json:"consumer_fetch_min"          mapstructure:"consumer_fetch_min" yaml:"consumer_fetch_min"`
	ConsumerFetchMax  int32         `json:"consumer_fetch_max"          mapstructure:"consumer_fetch_max" yaml:"consumer_fetch_max"`
	ConsumerIsolation string        `json:"consumer_isolation"          mapstructure:"consumer_isolation" yaml:"consumer_isolation"` // read_uncommitted, read_committed

	// Consumer group settings
	ConsumerGroupRebalance string        `json:"consumer_group_rebalance" mapstructure:"consumer_group_rebalance" yaml:"consumer_group_rebalance"` // range, roundrobin, sticky
	ConsumerGroupSession   time.Duration `json:"consumer_group_session"   mapstructure:"consumer_group_session"   yaml:"consumer_group_session"`
	ConsumerGroupHeartbeat time.Duration `json:"consumer_group_heartbeat" mapstructure:"consumer_group_heartbeat" yaml:"consumer_group_heartbeat"`

	// Metadata settings
	MetadataRetryMax     int           `json:"metadata_retry_max"     mapstructure:"metadata_retry_max"     yaml:"metadata_retry_max"`
	MetadataRetryBackoff time.Duration `json:"metadata_retry_backoff" mapstructure:"metadata_retry_backoff" yaml:"metadata_retry_backoff"`
	MetadataRefreshFreq  time.Duration `json:"metadata_refresh_freq"  mapstructure:"metadata_refresh_freq"  yaml:"metadata_refresh_freq"`
	MetadataFullRefresh  bool          `json:"metadata_full_refresh"  mapstructure:"metadata_full_refresh"  yaml:"metadata_full_refresh"`

	// Observability
	EnableMetrics bool `json:"enable_metrics" mapstructure:"enable_metrics" yaml:"enable_metrics"`
	EnableTracing bool `json:"enable_tracing" mapstructure:"enable_tracing" yaml:"enable_tracing"`
	EnableLogging bool `json:"enable_logging" mapstructure:"enable_logging" yaml:"enable_logging"`

	// Config loading flags
	RequireConfig bool `json:"-" mapstructure:"-" yaml:"-"`
}

Config contains configuration for the Kafka extension.

func DefaultConfig

func DefaultConfig() Config

DefaultConfig returns default Kafka configuration.

func (*Config) ToSaramaConfig

func (c *Config) ToSaramaConfig() (*sarama.Config, error)

ToSaramaConfig converts to Sarama configuration.

func (*Config) Validate

func (c *Config) Validate() error

Validate validates the configuration.

type ConfigOption

type ConfigOption func(*Config)

ConfigOption is a functional option for Config.

func WithBrokers

func WithBrokers(brokers ...string) ConfigOption

func WithClientID

func WithClientID(clientID string) ConfigOption

func WithCompression

func WithCompression(compression string) ConfigOption

func WithConfig

func WithConfig(config Config) ConfigOption

func WithConsumer

func WithConsumer(enabled bool) ConfigOption

func WithConsumerGroup

func WithConsumerGroup(groupID string) ConfigOption

func WithIdempotent

func WithIdempotent(enabled bool) ConfigOption

func WithMetrics

func WithMetrics(enable bool) ConfigOption

func WithProducer

func WithProducer(enabled bool) ConfigOption

func WithRequireConfig

func WithRequireConfig(require bool) ConfigOption

func WithSASL

func WithSASL(mechanism, username, password string) ConfigOption

func WithTLS

func WithTLS(certFile, keyFile, caFile string, skipVerify bool) ConfigOption

func WithTracing

func WithTracing(enable bool) ConfigOption

func WithVersion

func WithVersion(version string) ConfigOption

type ConsumerGroupHandler

type ConsumerGroupHandler interface {
	Setup(session sarama.ConsumerGroupSession) error
	Cleanup(session sarama.ConsumerGroupSession) error
	ConsumeClaim(session sarama.ConsumerGroupSession, claim sarama.ConsumerGroupClaim) error
}

ConsumerGroupHandler handles consumer group messages.

type ContextMessageHandler added in v1.12.9

type ContextMessageHandler func(ctx context.Context, message *sarama.ConsumerMessage) error

ContextMessageHandler processes a Kafka message with a context that carries the consumer span, so the spans you start in it join the producer's trace.

type Extension

type Extension struct {
	*forge.BaseExtension
	// contains filtered or unexported fields
}

Extension implements forge.Extension for Kafka functionality. The extension is now a lightweight facade that loads config and registers services.

func (*Extension) Health

func (e *Extension) Health(ctx context.Context) error

Health checks the extension health.

func (*Extension) Register

func (e *Extension) Register(app forge.App) error

Register registers the Kafka extension with the app.

func (*Extension) Start

func (e *Extension) Start(ctx context.Context) error

Start marks the extension as started. The actual client is started by Vessel calling KafkaService.Start().

func (*Extension) Stop

func (e *Extension) Stop(ctx context.Context) error

Stop marks the extension as stopped. The actual client is stopped by Vessel calling KafkaService.Stop().

type HashGeneratorFcn

type HashGeneratorFcn func() hash.Hash

HashGeneratorFcn is a function type for generating hash functions.

var (
	// SHA256 is a hash generator for SHA-256.
	SHA256 HashGeneratorFcn = sha256.New

	// SHA512 is a hash generator for SHA-512.
	SHA512 HashGeneratorFcn = sha512.New
)

type Kafka

type Kafka interface {
	// Producer operations
	SendMessage(topic string, key, value []byte) error
	SendMessageAsync(topic string, key, value []byte) error
	SendMessages(messages []*ProducerMessage) error
	SendMessageContext(ctx context.Context, topic string, key, value []byte) error
	SendMessageAsyncContext(ctx context.Context, topic string, key, value []byte) error
	SendMessagesContext(ctx context.Context, messages []*ProducerMessage) error

	// Consumer operations
	Consume(ctx context.Context, topics []string, handler MessageHandler) error
	ConsumePartition(ctx context.Context, topic string, partition int32, offset int64, handler MessageHandler) error
	ConsumeContext(ctx context.Context, topics []string, handler ContextMessageHandler) error
	ConsumePartitionContext(ctx context.Context, topic string, partition int32, offset int64, handler ContextMessageHandler) error
	StopConsume() error

	// Consumer group operations
	JoinConsumerGroup(ctx context.Context, groupID string, topics []string, handler ConsumerGroupHandler) error
	LeaveConsumerGroup(ctx context.Context) error

	// Admin operations
	CreateTopic(topic string, config TopicConfig) error
	DeleteTopic(topic string) error
	ListTopics() ([]string, error)
	DescribeTopic(topic string) (*TopicMetadata, error)
	GetPartitions(topic string) ([]int32, error)
	GetOffset(topic string, partition int32, time int64) (int64, error)

	// Client info
	GetProducer() sarama.SyncProducer
	GetAsyncProducer() sarama.AsyncProducer
	GetConsumer() sarama.Consumer
	GetConsumerGroup() sarama.ConsumerGroup
	GetClient() sarama.Client
	GetStats() ClientStats

	// Lifecycle
	Close() error

	// Health
	Ping(ctx context.Context) error
}

Kafka represents a unified Kafka client interface.

func NewKafkaClient

func NewKafkaClient(config Config, logger forge.Logger, metrics forge.Metrics) (Kafka, error)

NewKafkaClient creates a new Kafka client.

type KafkaService

type KafkaService struct {
	// contains filtered or unexported fields
}

KafkaService wraps a Kafka client and provides lifecycle management. It implements vessel's di.Service interface so Vessel can manage its lifecycle.

func NewKafkaService

func NewKafkaService(config Config, logger forge.Logger, metrics forge.Metrics) (*KafkaService, error)

NewKafkaService creates a new Kafka service with the given configuration. This is the constructor that will be registered with the DI container.

func (*KafkaService) Client

func (s *KafkaService) Client() Kafka

Client returns the underlying Kafka client.

func (*KafkaService) Close

func (s *KafkaService) Close() error

func (*KafkaService) Consume

func (s *KafkaService) Consume(ctx context.Context, topics []string, handler MessageHandler) error

func (*KafkaService) ConsumeContext added in v1.12.9

func (s *KafkaService) ConsumeContext(ctx context.Context, topics []string, handler ContextMessageHandler) error

func (*KafkaService) ConsumePartition added in v0.9.7

func (s *KafkaService) ConsumePartition(ctx context.Context, topic string, partition int32, offset int64, handler MessageHandler) error

func (*KafkaService) ConsumePartitionContext added in v1.12.9

func (s *KafkaService) ConsumePartitionContext(ctx context.Context, topic string, partition int32, offset int64, handler ContextMessageHandler) error

func (*KafkaService) CreateTopic

func (s *KafkaService) CreateTopic(topic string, config TopicConfig) error

func (*KafkaService) DeleteTopic

func (s *KafkaService) DeleteTopic(topic string) error

func (*KafkaService) DescribeTopic added in v0.9.7

func (s *KafkaService) DescribeTopic(topic string) (*TopicMetadata, error)

func (*KafkaService) GetAsyncProducer added in v0.9.7

func (s *KafkaService) GetAsyncProducer() sarama.AsyncProducer

func (*KafkaService) GetClient added in v0.9.7

func (s *KafkaService) GetClient() sarama.Client

func (*KafkaService) GetConsumer added in v0.9.7

func (s *KafkaService) GetConsumer() sarama.Consumer

func (*KafkaService) GetConsumerGroup added in v0.9.7

func (s *KafkaService) GetConsumerGroup() sarama.ConsumerGroup

func (*KafkaService) GetOffset added in v0.9.7

func (s *KafkaService) GetOffset(topic string, partition int32, time int64) (int64, error)

func (*KafkaService) GetPartitions added in v0.9.7

func (s *KafkaService) GetPartitions(topic string) ([]int32, error)

func (*KafkaService) GetProducer added in v0.9.7

func (s *KafkaService) GetProducer() sarama.SyncProducer

func (*KafkaService) GetStats added in v0.9.7

func (s *KafkaService) GetStats() ClientStats

func (*KafkaService) Health

func (s *KafkaService) Health(ctx context.Context) error

Health checks if the Kafka service is healthy.

func (*KafkaService) JoinConsumerGroup added in v0.9.7

func (s *KafkaService) JoinConsumerGroup(ctx context.Context, groupID string, topics []string, handler ConsumerGroupHandler) error

func (*KafkaService) LeaveConsumerGroup added in v0.9.7

func (s *KafkaService) LeaveConsumerGroup(ctx context.Context) error

func (*KafkaService) ListTopics

func (s *KafkaService) ListTopics() ([]string, error)

func (*KafkaService) Name

func (s *KafkaService) Name() string

Name returns the service name for Vessel's lifecycle management.

func (*KafkaService) Ping

func (s *KafkaService) Ping(ctx context.Context) error

func (*KafkaService) SendMessage added in v0.9.7

func (s *KafkaService) SendMessage(topic string, key, value []byte) error

func (*KafkaService) SendMessageAsync added in v0.9.7

func (s *KafkaService) SendMessageAsync(topic string, key, value []byte) error

func (*KafkaService) SendMessageAsyncContext added in v1.12.9

func (s *KafkaService) SendMessageAsyncContext(ctx context.Context, topic string, key, value []byte) error

func (*KafkaService) SendMessageContext added in v1.12.9

func (s *KafkaService) SendMessageContext(ctx context.Context, topic string, key, value []byte) error

func (*KafkaService) SendMessages added in v0.9.7

func (s *KafkaService) SendMessages(messages []*ProducerMessage) error

func (*KafkaService) SendMessagesContext added in v1.12.9

func (s *KafkaService) SendMessagesContext(ctx context.Context, messages []*ProducerMessage) error

func (*KafkaService) Start

func (s *KafkaService) Start(ctx context.Context) error

Start starts the Kafka service by verifying connection. This is called automatically by Vessel during container.Start().

func (*KafkaService) Stop

func (s *KafkaService) Stop(_ context.Context) error

Stop stops the Kafka service by closing the client. This is called automatically by Vessel during container.Stop().

func (*KafkaService) StopConsume added in v0.9.7

func (s *KafkaService) StopConsume() error

type MessageHandler

type MessageHandler func(message *sarama.ConsumerMessage) error

MessageHandler processes incoming Kafka messages.

type MessageHeader

type MessageHeader struct {
	Key   string
	Value []byte
}

MessageHeader represents a Kafka message header.

type PartitionMetadata

type PartitionMetadata struct {
	ID       int32
	Leader   int32
	Replicas []int32
	Isr      []int32
}

PartitionMetadata contains partition metadata.

type ProducerMessage

type ProducerMessage struct {
	Topic     string
	Key       []byte
	Value     []byte
	Headers   []MessageHeader
	Partition int32
	Offset    int64
	Timestamp time.Time
}

ProducerMessage represents a message to be produced.

type TopicConfig

type TopicConfig struct {
	NumPartitions     int32
	ReplicationFactor int16
	ConfigEntries     map[string]*string
}

TopicConfig contains topic configuration.

type TopicMetadata

type TopicMetadata struct {
	Name       string
	Partitions []PartitionMetadata
	Config     map[string]string
}

TopicMetadata contains topic metadata.

type XDGSCRAMClient

type XDGSCRAMClient struct {
	*scram.Client
	*scram.ClientConversation

	HashGeneratorFcn scram.HashGeneratorFcn
}

XDGSCRAMClient implements the SCRAM client interface.

func (*XDGSCRAMClient) Begin

func (x *XDGSCRAMClient) Begin(userName, password, authzID string) (err error)

Begin starts the SCRAM authentication process.

func (*XDGSCRAMClient) Done

func (x *XDGSCRAMClient) Done() bool

Done returns true if the authentication is complete.

func (*XDGSCRAMClient) Step

func (x *XDGSCRAMClient) Step(challenge string) (response string, err error)

Step processes one step of the SCRAM authentication.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL