Documentation
¶
Index ¶
Constants ¶
This section is empty.
Variables ¶
var (
ErrInvalidOffsetFormat = errors.New("invalid format for kafka offset")
)
Functions ¶
This section is empty.
Types ¶
type ConnConfig ¶
type ConnConfig struct {
Servers []string
Topic TopicConfig
TLS tlslib.Config
SASL SASLConfig
}
type MessageReader ¶
type MessageWriter ¶
type OffsetParser ¶
type Reader ¶
type Reader struct {
// contains filtered or unexported fields
}
func (*Reader) CommitOffsets ¶
func (*Reader) FetchMessage ¶
FetchMessage returns the next message from the reader. This call will block until a message is available, or an error occurs. It can be stopped by canceling the context. The message offset needs to be explicitly committed by using CommitMessages.
type ReaderConfig ¶
type ReaderConfig struct {
Conn ConnConfig
// ConsumerGroupID is the ID of the consumer group to use. If not set,
// defaults to "pgstream-consumer-group".
ConsumerGroupID string
// ConsumerGroupStartOffset is the offset to start consuming from. If not
// set, defaults to "earliest".
ConsumerGroupStartOffset string
}
type SASLConfig ¶ added in v1.5.0
type SASLConfig struct {
// Enabled defines if the connection uses SASL authentication. Defaults to
// false.
Enabled bool
// Mechanism is the SASL mechanism to use. It must be one of "plain",
// "scram-sha-256" or "scram-sha-512". It is required when SASL is enabled.
Mechanism string
// Username is the user for the SASL authentication. It is required when
// SASL is enabled.
Username string
// Password is the password for the SASL authentication. It is required when
// SASL is enabled.
Password string
}
SASLConfig configures the SASL authentication used to connect to the Kafka servers.
type TopicConfig ¶
type TopicConfig struct {
Name string
// Number of partitions to be created for the topic. Defaults to 1.
NumPartitions int
// Replication factor for the topic. Defaults to 1.
ReplicationFactor int
// AutoCreate defines if the topic should be created if it doesn't exist.
// Defaults to false.
AutoCreate bool
}
type Writer ¶
type Writer struct {
// contains filtered or unexported fields
}
Writer is a wrapper around the kafkago library writer
func NewWriter ¶
func NewWriter(config WriterConfig, logger loglib.Logger) (*Writer, error)
NewWriter returns a kafka writer that produces messages to the configured topic, using the CRC32 hash function to determine which partition to route messages to. This ensures that messages with the same key are routed to the same partition.
If the topic auto create setting is enabled in the config, it will create it.
type WriterConfig ¶
type WriterConfig struct {
Conn ConnConfig
// BatchTimeout is the time limit on how often incomplete message batches
// will be flushed to kafka. Defaults to 1s.
BatchTimeout time.Duration
// BatchBytes limits the maximum size of a request in bytes before being
// sent to a partition. Defaults to 1048576 bytes.
BatchBytes int64
// BatchSize limits how many messages will be buffered before being sent to
// a partition. Defaults to 100 messages.
BatchSize int
}