Documentation
¶
Overview ¶
Package morpc provides a high-performance RPC client with automatic backend management, circuit breaker, retry policies, and bounded wait for backend creation.
Backend State Machine & Error Semantics ¶
The client maintains a pool of backends for each remote address. Backend states and corresponding errors guide retry behavior:
┌──────────────────────────────────────────────────────────────────────────────┐ │ Backend State │ Error │ Should Retry? │ ├──────────────────────────────┼────────────────────────────┼─────────────────┤ │ Creating (async) │ ErrBackendCreating │ Yes (transient) │ │ Pool empty, cannot create │ ErrNoAvailableBackend │ Yes (transient) │ │ Pool has backends but down │ ErrBackendUnavailable │ No (permanent) │ │ Create timeout exceeded │ ErrBackendCreateTimeout │ No (permanent) │ │ Circuit breaker open │ ErrCircuitOpen │ No (permanent) │ │ Circuit breaker half-open │ ErrCircuitHalfOpen │ Maybe (probe) │ │ Client closing │ ErrClientClosing │ No (permanent) │ │ Client closed │ ErrClientClosed │ No (permanent) │ └──────────────────────────────────────────────────────────────────────────────┘
Bounded Wait for Auto-Create ¶
When auto-create is enabled and a backend is being created asynchronously, callers can configure a bounded wait timeout:
- autoCreateWaitTimeout = 0 (default): Wait until context deadline (legacy behavior)
- autoCreateWaitTimeout > 0: Wait up to specified duration, then return ErrBackendClosed
Example: lockservice sets 500ms timeout for fast failure detection in orphan transaction cleanup.
Retry Policy ¶
DefaultRetryPolicy retries indefinitely (MaxRetries=0) with exponential backoff. The retry loop exits when:
- Context is cancelled/timeout
- Non-retryable error (ErrBackendClosed, ErrCircuitOpen, ErrClientClosed)
- Bounded wait timeout exceeded (if configured)
Usage Example ¶
// Default behavior (wait until context timeout)
client, _ := NewClient("my-service", cfg, factory)
// With bounded wait (fast failure detection)
client, _ := NewClient("my-service", cfg, factory,
WithClientAutoCreateWaitTimeout(500*time.Millisecond))
Observability ¶
Metrics:
- mo_rpc_backend_auto_create_timeout_total: Auto-create wait timeouts
- mo_rpc_backend_create_total: Backend creation attempts
- mo_rpc_backend_connect_total: Connection attempts (total/failed)
Logs:
- "waiting for backend creation": Sparse logging (1st, then every 10th retry)
- "auto-create backend timed out": When bounded wait timeout exceeded
Index ¶
- Constants
- Variables
- func GetMessageSize() int
- func InitGlobalGCManager(checkInterval time.Duration, channelBufferSize int)
- func IsCancelled(err error) bool
- func IsCircuitOpen(err error) bool
- func IsConnectionError(err error) bool
- func IsDefinitiveFailure(err error) bool
- func IsRemoteUnavailable(err error) bool
- func IsRetryableForCDC(err error) bool
- func IsTransient(err error) bool
- func IsUnavailable(err error) bool
- func ResetGlobalClientGCManagerForTest()
- type Backend
- type BackendFactory
- type BackendOption
- func WithBackendBatchSendSize(size int) BackendOption
- func WithBackendBufferSize(size int) BackendOption
- func WithBackendBusyBufferSize(size int) BackendOption
- func WithBackendConnectTimeout(timeout time.Duration) BackendOption
- func WithBackendFilter(filter func(Message, string) bool) BackendOption
- func WithBackendFreeOrphansResponse(value func(Message)) BackendOption
- func WithBackendGoettyOptions(options ...goetty.Option) BackendOption
- func WithBackendHasPayloadResponse() BackendOption
- func WithBackendLogger(logger *zap.Logger) BackendOption
- func WithBackendMetrics(metrics *metrics) BackendOption
- func WithBackendReadTimeout(value time.Duration) BackendOption
- func WithBackendStreamBufferSize(value int) BackendOption
- func WithDisconnectAfterRead(n int) BackendOption
- type Buffer
- func (b *Buffer) Close()
- func (b *Buffer) EncodeBytes(bys []byte) []byte
- func (b *Buffer) EncodeInt(v int) []byte
- func (b *Buffer) EncodeUint16(v uint16) []byte
- func (b *Buffer) EncodeUint64(v uint64) []byte
- func (b *Buffer) GetMarkedData() []byte
- func (b *Buffer) Mark()
- func (b Buffer) TypeName() string
- type CircuitBreaker
- type CircuitBreakerConfig
- type CircuitBreakerManager
- func (m *CircuitBreakerManager) Allow(backend string) bool
- func (m *CircuitBreakerManager) GetBreaker(backend string) *CircuitBreaker
- func (m *CircuitBreakerManager) GetError(backend string) error
- func (m *CircuitBreakerManager) RecordFailure(backend string)
- func (m *CircuitBreakerManager) RecordSuccess(backend string)
- func (m *CircuitBreakerManager) RemoveBreaker(backend string)
- func (m *CircuitBreakerManager) Stats() map[string]CircuitBreakerStats
- type CircuitBreakerStats
- type CircuitState
- type ClientOption
- func WithClientAutoCreateWaitTimeout(timeout time.Duration) ClientOption
- func WithClientCircuitBreaker(config CircuitBreakerConfig) ClientOption
- func WithClientCreateTaskChanSize(size int) ClientOption
- func WithClientDisableAutoCreateBackend() ClientOption
- func WithClientDisableCircuitBreaker() ClientOption
- func WithClientDisableRetry() ClientOption
- func WithClientEnableAutoCreateBackend() ClientOption
- func WithClientInitBackends(backends []string, counts []int) ClientOption
- func WithClientLogger(logger *zap.Logger) ClientOption
- func WithClientMaxBackendMaxIdleDuration(value time.Duration) ClientOption
- func WithClientMaxBackendPerHost(maxBackendsPerHost int) ClientOption
- func WithClientRetryPolicy(policy RetryPolicy) ClientOption
- type ClientSession
- type Codec
- type CodecOption
- type Config
- type Future
- type HandleFunc
- type HandlerOption
- type HeaderCodec
- type Message
- type MessageCache
- type MessagePool
- type MethodBasedClient
- type MethodBasedMessage
- type MethodBasedServer
- type PayloadMessage
- type RPCClient
- type RPCMessage
- type RPCServer
- type RetryPolicy
- type ServerOption
- func WithServerBatchSendSize(size int) ServerOption
- func WithServerDisableAutoCancelContext() ServerOption
- func WithServerGoettyOptions(options ...goetty.Option) ServerOption
- func WithServerHandler(...) ServerOption
- func WithServerLogger(logger *zap.Logger) ServerOption
- func WithServerMessageReleaseFunc(release func(Message)) ServerOption
- func WithServerSessionBufferSize(size int) ServerOption
- func WithServerWriteFilter(filter func(Message) bool) ServerOption
- type StatusCategory
- type Stream
Constants ¶
const ( // DefaultGCIdleCheckInterval is the default interval for checking idle backends. // This value is shared between client_gc.go and cfg.go. DefaultGCIdleCheckInterval = time.Second // DefaultGCInactiveCheckInterval is the interval for proactively removing // explicitly closed (inactive) backends. Such backends are removed within // this period instead of waiting for the normal idle timeout (e.g. 1 minute). DefaultGCInactiveCheckInterval = 10 * time.Second // DefaultGCChannelBufferSize is the default buffer size for GC task channels. // Increased from 1024 to 4096 to reduce request drops in high-load scenarios. DefaultGCChannelBufferSize = 4096 )
Variables ¶
var ( // DefaultCircuitBreakerConfig disables circuit breaker by default. // This preserves existing behavior where callers rely on context timeout // and retry logic. Use EnabledCircuitBreakerConfig to opt-in. DefaultCircuitBreakerConfig = CircuitBreakerConfig{ Enabled: false, } // EnabledCircuitBreakerConfig is a recommended circuit breaker configuration. // It opens the circuit after 5 consecutive failures, waits 10 seconds before // allowing probe requests, and allows 3 probe requests in half-open state. // Use WithClientCircuitBreaker(EnabledCircuitBreakerConfig) to enable. EnabledCircuitBreakerConfig = CircuitBreakerConfig{ Enabled: true, FailureThreshold: 5, ResetTimeout: 10 * time.Second, HalfOpenMaxRequests: 3, } // DisabledCircuitBreakerConfig explicitly disables the circuit breaker. DisabledCircuitBreakerConfig = CircuitBreakerConfig{ Enabled: false, } )
var ( // DefaultRetryPolicy is the default retry policy for morpc client. // It retries indefinitely (MaxRetries=0) with exponential backoff starting at 10ms, // maxing out at 1s, with 20% jitter. The retry loop exits when context is cancelled. // This matches the original design intent where context timeout is the exit mechanism. DefaultRetryPolicy = RetryPolicy{ MaxRetries: 0, InitialBackoff: 10 * time.Millisecond, MaxBackoff: 1 * time.Second, Multiplier: 2.0, JitterFraction: 0.2, } // NoRetryPolicy disables retry (only 1 attempt). NoRetryPolicy = RetryPolicy{ MaxRetries: 1, InitialBackoff: 0, MaxBackoff: 0, Multiplier: 1.0, JitterFraction: 0, } // ErrBackendCreating indicates that the backend is being created asynchronously. // Callers can distinguish "creation in progress" from "backend closed/unavailable". // This is a high-frequency expected error (NoCtx to avoid log spam). ErrBackendCreating = moerr.NewInternalErrorNoCtx("morpc backend is being created") // This typically means network partition, service crash, or all backends inactive. // This is a high-frequency expected error (NoCtx to avoid log spam). // Uses ErrBackendClosed code for compatibility with existing error handling. ErrBackendUnavailable = moerr.NewBackendClosedNoCtx() // ErrBackendCreateTimeout indicates that auto-create wait timeout exceeded. // This typically means backend creation is too slow or queue congestion. // This is a boundary condition error (NoCtx + Counter for monitoring). // Uses ErrBackendClosed code for compatibility with existing error handling. ErrBackendCreateTimeout = moerr.NewBackendClosedNoCtx() // ErrClientClosing indicates that the client is in the process of closing. // New requests should fail fast rather than waiting for backend creation. // This is a high-frequency expected error during shutdown (NoCtx to avoid log spam). ErrClientClosing = moerr.NewClientClosedNoCtx() )
var ErrCircuitHalfOpen = moerr.NewServiceUnavailableNoCtx("circuit breaker is half-open")
ErrCircuitHalfOpen indicates that the circuit breaker is in half-open state. Only limited probe requests are allowed in this state.
var ErrCircuitOpen = moerr.NewServiceUnavailableNoCtx("circuit breaker is open")
ErrCircuitOpen is returned when the circuit breaker is open and rejecting requests. ErrCircuitOpen indicates that the circuit breaker is open and rejecting all requests.
Functions ¶
func GetMessageSize ¶ added in v0.6.0
func GetMessageSize() int
func InitGlobalGCManager ¶
InitGlobalGCManager initializes the global GC manager with config values. This should be called before creating any clients if you want to use config values. If not called, environment variables or defaults will be used.
Note: This function should be called before any clients are created. If called after the GC manager has started, it will only update the interval (if manager hasn't started yet), but channel buffer size cannot be changed after initialization.
Thread-safety: This function is safe to call concurrently. It uses a separate mutex to protect the global manager initialization to avoid lock issues when replacing the manager instance.
func IsCancelled ¶
IsCancelled returns true if the error indicates client-side cancellation. Examples: client closed, context cancelled.
Usage:
- During shutdown, this is expected and should not trigger retries
- CDC may choose to retry by creating a new client
func IsCircuitOpen ¶
IsCircuitOpen checks if the error indicates a circuit breaker rejection. It checks both identity comparison and error code for wrapped errors.
func IsConnectionError ¶
IsConnectionError returns true if the error is related to connection issues. This includes both transient network problems and unavailable backends. Convenience function combining transient and unavailable categories.
func IsDefinitiveFailure ¶
IsDefinitiveFailure returns true if the error indicates a definitive failure where the remote service is considered unavailable. This is the opposite of "transient" - the failure is not temporary.
This is a BROAD definition that includes:
- StatusUnavailable: ErrBackendClosed, ErrBackendCannotConnect, ErrNoAvailableBackend
- StatusCancelled: ErrClientClosed, context.Canceled
Usage (CDC/general style):
- If true: The remote is definitely gone, take action
- If false: Be conservative, assume things might still be OK
For lockservice-specific semantics (more conservative), use IsRemoteUnavailable().
func IsRemoteUnavailable ¶
IsRemoteUnavailable returns true ONLY if the specific remote backend is definitely unreachable. This matches lockservice's original isRetryError semantics.
Only these errors qualify:
- ErrBackendClosed: The specific backend connection is closed
- ErrBackendCannotConnect: Cannot connect to the specific backend
This is MORE CONSERVATIVE than IsDefinitiveFailure() because:
- ErrNoAvailableBackend is NOT included (could be temporary routing issue)
- ErrClientClosed is NOT included (local shutdown, not remote failure)
Usage (lockservice style):
- If true: The SPECIFIC remote backend is gone, refresh bindings immediately
- If false: Be conservative, the remote might still be there
func IsRetryableForCDC ¶
IsRetryableForCDC returns true if the error should trigger retry in CDC-style callers. This is a convenience function for callers that want to retry on any connection-related error, including client cancellation (for reconnection).
Note: This differs from lockservice semantics where ErrClientClosed means the local client is shutting down and should not retry.
func IsTransient ¶
IsTransient returns true if the error is a transient failure. Transient errors are temporary and may succeed on immediate retry. Examples: timeout, temporary network issues.
func IsUnavailable ¶
IsUnavailable returns true if the error indicates service unavailability. This means the backend/connection is currently down. By the time caller receives this, morpc has already exhausted internal retries.
Usage:
- lockservice: Use this to detect when lock table bindings should be refreshed
- txn/rpc: Use this to detect when to switch to a different TN
func ResetGlobalClientGCManagerForTest ¶
func ResetGlobalClientGCManagerForTest()
ResetGlobalClientGCManagerForTest resets the global GC manager for testing purposes. This stops the current manager and creates a new one. WARNING: Only use this in tests!
Types ¶
type Backend ¶
type Backend interface {
// Send send the request for future to the corresponding backend.
// moerr.ErrBackendClosed returned if backend is closed.
Send(ctx context.Context, request Message) (*Future, error)
// SendInternal is similar to Send, but perform on internal message
SendInternal(ctx context.Context, request Message) (*Future, error)
// NewStream create a stream used to asynchronous stream of sending and receiving messages.
// If the underlying connection is reset during the duration of the stream, then the stream
// will be closed.
NewStream(unlockAfterClose bool) (Stream, error)
// Close close the backend.
Close()
// Busy the backend receives a lot of requests concurrently during operation, but when the number
// of requests waiting to be sent reaches some threshold, the current backend is busy.
Busy() bool
// LastActiveTime returns last active time
LastActiveTime() time.Time
// Lock other I/O operations can not use this backend if the backend is locked
Lock()
// Unlock the backend can used by other I/O operations after unlock
Unlock()
// Locked indicates if backend is locked
Locked() bool
}
Backend backend represents a wrapper for a client communicating with a remote server.
func NewRemoteBackend ¶
func NewRemoteBackend( remote string, codec Codec, options ...BackendOption) (Backend, error)
NewRemoteBackend create a goetty connection based backend. This backend will start 2 goroutine, one for read and one for write. If there is a network error in the underlying goetty connection, it will automatically retry until the Future times out.
type BackendFactory ¶
type BackendFactory interface {
// Create create the corresponding backend based on the given address.
Create(address string, extraOptions ...BackendOption) (Backend, error)
}
BackendFactory backend factory
func NewGoettyBasedBackendFactory ¶
func NewGoettyBasedBackendFactory(codec Codec, options ...BackendOption) BackendFactory
type BackendOption ¶
type BackendOption func(*remoteBackend)
BackendOption options for create remote backend
func WithBackendBatchSendSize ¶
func WithBackendBatchSendSize(size int) BackendOption
WithBackendBatchSendSize set the maximum number of messages to be sent together at each batch. Default is 8.
func WithBackendBufferSize ¶
func WithBackendBufferSize(size int) BackendOption
WithBackendBufferSize set the buffer size of the wait send chan. Default is 1024.
func WithBackendBusyBufferSize ¶
func WithBackendBusyBufferSize(size int) BackendOption
WithBackendBusyBufferSize if len(writeC) >= size, backend is busy. Default is 3/4 buffer size.
func WithBackendConnectTimeout ¶
func WithBackendConnectTimeout(timeout time.Duration) BackendOption
WithBackendConnectTimeout set the timeout for connect to remote. Default 10s.
func WithBackendFilter ¶
func WithBackendFilter(filter func(Message, string) bool) BackendOption
WithBackendFilter set send filter func. Input ready to send futures, output is really need to be send futures.
func WithBackendFreeOrphansResponse ¶ added in v1.1.0
func WithBackendFreeOrphansResponse(value func(Message)) BackendOption
WithBackendFreeOrphansResponse setup free orphans response func
func WithBackendGoettyOptions ¶
func WithBackendGoettyOptions(options ...goetty.Option) BackendOption
WithBackendGoettyOptions set goetty connection options. e.g. set read/write buffer size, adjust net.Conn attribute etc.
func WithBackendHasPayloadResponse ¶ added in v0.6.0
func WithBackendHasPayloadResponse() BackendOption
WithBackendHasPayloadResponse has payload response means read a response that hold a slice of data in the read buffer to avoid data copy.
func WithBackendLogger ¶
func WithBackendLogger(logger *zap.Logger) BackendOption
WithBackendLogger set the backend logger
func WithBackendMetrics ¶ added in v1.0.0
func WithBackendMetrics(metrics *metrics) BackendOption
WithBackendMetrics setup backend metrics
func WithBackendReadTimeout ¶ added in v1.0.0
func WithBackendReadTimeout(value time.Duration) BackendOption
WithBackendReadTimeout set read timeout for read loop.
func WithBackendStreamBufferSize ¶ added in v0.5.1
func WithBackendStreamBufferSize(value int) BackendOption
WithBackendStreamBufferSize set buffer size for stream receive message chan
func WithDisconnectAfterRead ¶ added in v1.2.0
func WithDisconnectAfterRead(n int) BackendOption
WithDisconnectAfterRead used for testing. Close the connection after read N messages.
type Buffer ¶
type Buffer struct {
// contains filtered or unexported fields
}
func (*Buffer) EncodeBytes ¶
func (*Buffer) EncodeUint16 ¶
func (*Buffer) EncodeUint64 ¶
func (*Buffer) GetMarkedData ¶
type CircuitBreaker ¶
type CircuitBreaker struct {
// contains filtered or unexported fields
}
CircuitBreaker implements the circuit breaker pattern for a single backend. It tracks failures and prevents cascading failures by temporarily rejecting requests to a failing backend.
func NewCircuitBreaker ¶
func NewCircuitBreaker(config CircuitBreakerConfig, logger *zap.Logger) *CircuitBreaker
NewCircuitBreaker creates a new CircuitBreaker with the given configuration.
func (*CircuitBreaker) Allow ¶
func (cb *CircuitBreaker) Allow() bool
Allow checks if a request should be allowed. Returns true if the request is allowed, false if rejected due to open circuit.
func (*CircuitBreaker) RecordFailure ¶
func (cb *CircuitBreaker) RecordFailure()
RecordFailure records a failed request.
func (*CircuitBreaker) RecordSuccess ¶
func (cb *CircuitBreaker) RecordSuccess()
RecordSuccess records a successful request.
func (*CircuitBreaker) Reset ¶
func (cb *CircuitBreaker) Reset()
Reset resets the circuit breaker to closed state.
func (*CircuitBreaker) State ¶
func (cb *CircuitBreaker) State() CircuitState
State returns the current circuit state.
func (*CircuitBreaker) Stats ¶
func (cb *CircuitBreaker) Stats() CircuitBreakerStats
Stats returns circuit breaker statistics.
type CircuitBreakerConfig ¶
type CircuitBreakerConfig struct {
// Enabled determines if circuit breaker is active.
Enabled bool
// FailureThreshold is the number of consecutive failures before opening the circuit.
FailureThreshold int32
// ResetTimeout is the duration to wait before transitioning from open to half-open.
ResetTimeout time.Duration
// HalfOpenMaxRequests is the maximum number of requests allowed in half-open state.
HalfOpenMaxRequests int32
}
CircuitBreakerConfig defines the configuration for a circuit breaker.
type CircuitBreakerManager ¶
type CircuitBreakerManager struct {
// contains filtered or unexported fields
}
CircuitBreakerManager manages circuit breakers for multiple backends.
func NewCircuitBreakerManager ¶
func NewCircuitBreakerManager(name string, config CircuitBreakerConfig, logger *zap.Logger) *CircuitBreakerManager
NewCircuitBreakerManager creates a new CircuitBreakerManager.
func (*CircuitBreakerManager) Allow ¶
func (m *CircuitBreakerManager) Allow(backend string) bool
Allow checks if a request to the given backend should be allowed. Returns an error if the request should be rejected.
func (*CircuitBreakerManager) GetBreaker ¶
func (m *CircuitBreakerManager) GetBreaker(backend string) *CircuitBreaker
GetBreaker returns the circuit breaker for the given backend address. Creates a new breaker if one doesn't exist.
func (*CircuitBreakerManager) GetError ¶
func (m *CircuitBreakerManager) GetError(backend string) error
GetError returns the appropriate error based on circuit breaker state. Returns nil if circuit is closed, ErrCircuitOpen if open, ErrCircuitHalfOpen if half-open.
func (*CircuitBreakerManager) RecordFailure ¶
func (m *CircuitBreakerManager) RecordFailure(backend string)
RecordFailure records a failed request to the given backend.
func (*CircuitBreakerManager) RecordSuccess ¶
func (m *CircuitBreakerManager) RecordSuccess(backend string)
RecordSuccess records a successful request to the given backend.
func (*CircuitBreakerManager) RemoveBreaker ¶
func (m *CircuitBreakerManager) RemoveBreaker(backend string)
RemoveBreaker removes the circuit breaker for the given backend.
func (*CircuitBreakerManager) Stats ¶
func (m *CircuitBreakerManager) Stats() map[string]CircuitBreakerStats
Stats returns statistics for all circuit breakers.
type CircuitBreakerStats ¶
type CircuitBreakerStats struct {
State CircuitState
Failures int32
LastFailure time.Time
OpenCount int64
RejectCount int64
SuccessCount int64
FailureCount int64
}
CircuitBreakerStats contains statistics for a circuit breaker.
type CircuitState ¶
type CircuitState int32
CircuitState represents the state of a circuit breaker.
const ( // CircuitClosed is the normal state where requests are allowed. CircuitClosed CircuitState = iota // CircuitOpen is the state where all requests are rejected. CircuitOpen // CircuitHalfOpen is the state where limited requests are allowed for probing. CircuitHalfOpen )
func (CircuitState) String ¶
func (s CircuitState) String() string
String returns the string representation of the circuit state.
type ClientOption ¶
type ClientOption func(*client)
ClientOption client options for create client
func WithClientAutoCreateWaitTimeout ¶
func WithClientAutoCreateWaitTimeout(timeout time.Duration) ClientOption
WithClientAutoCreateWaitTimeout sets how long Send/NewStream/Ping will wait for an auto-created backend before giving up. Zero keeps legacy behavior (wait until context deadline).
func WithClientCircuitBreaker ¶
func WithClientCircuitBreaker(config CircuitBreakerConfig) ClientOption
WithClientCircuitBreaker sets the circuit breaker configuration for the client. If not set, DefaultCircuitBreakerConfig is used.
func WithClientCreateTaskChanSize ¶
func WithClientCreateTaskChanSize(size int) ClientOption
WithClientCreateTaskChanSize set the buffer size of the chan that creates the Backend Task.
func WithClientDisableAutoCreateBackend ¶
func WithClientDisableAutoCreateBackend() ClientOption
WithClientDisableAutoCreateBackend disable client from automatically creating backends. By default, auto-create is enabled. Use this option to disable it.
func WithClientDisableCircuitBreaker ¶
func WithClientDisableCircuitBreaker() ClientOption
WithClientDisableCircuitBreaker disables the circuit breaker for the client.
func WithClientDisableRetry ¶
func WithClientDisableRetry() ClientOption
WithClientDisableRetry disables retry for the client.
func WithClientEnableAutoCreateBackend ¶ added in v1.1.1
func WithClientEnableAutoCreateBackend() ClientOption
WithClientEnableAutoCreateBackend enable client to automatically create a backend in the background, when the links in the connection pool are used, if the pool has not reached the maximum number of links, it will automatically create them in the background to improve the latency of link creation.
func WithClientInitBackends ¶
func WithClientInitBackends(backends []string, counts []int) ClientOption
WithClientInitBackends set the number of connections for the initialized backends.
func WithClientLogger ¶
func WithClientLogger(logger *zap.Logger) ClientOption
WithClientLogger set client logger
func WithClientMaxBackendMaxIdleDuration ¶ added in v0.6.0
func WithClientMaxBackendMaxIdleDuration(value time.Duration) ClientOption
WithClientMaxBackendMaxIdleDuration set the maximum idle duration of the backend connection. Backend connection that exceed this time will be automatically closed. 0 means no idle time limit.
Note: To avoid "thundering herd" effect where many connections expire simultaneously, a small random jitter (±10%) is automatically applied to positive durations. This spreads connection expiration times across a time window, reducing the impact of simultaneous connection closures. When value is 0 (disabled), no jitter is applied.
func WithClientMaxBackendPerHost ¶
func WithClientMaxBackendPerHost(maxBackendsPerHost int) ClientOption
WithClientMaxBackendPerHost maximum number of connections per host
func WithClientRetryPolicy ¶
func WithClientRetryPolicy(policy RetryPolicy) ClientOption
WithClientRetryPolicy sets the retry policy for the client. If not set, DefaultRetryPolicy is used.
type ClientSession ¶
type ClientSession interface {
// Close close the client session
Close() error
// SessionCtx get the session context, if session is closed the context will be canceled.
SessionCtx() context.Context
// Write writing the response message to the client.
Write(ctx context.Context, response Message) error
// AsyncWrite only put message into write queue, and return immediately.
AsyncWrite(response Message) error
// CreateCache create a message cache using cache ID. Cache will removed if
// context is done.
CreateCache(ctx context.Context, cacheID uint64) (MessageCache, error)
// DeleteCache delete cache using the spec cacheID
DeleteCache(cacheID uint64)
// GetCache returns the message cache
GetCache(cacheID uint64) (MessageCache, error)
// RemoteAddress returns remote address, include ip and port
RemoteAddress() string
}
ClientSession client session, which is used to send the response message. Note that it is not thread-safe.
type Codec ¶
type Codec interface {
codec.Codec
// Valid valid the message is valid
Valid(message Message) error
// AddHeaderCodec add header codec. The HeaderCodecs are added sequentially and the headercodecs are
// executed in the order in which they are added at codec time.
AddHeaderCodec(HeaderCodec)
}
Codec codec
func NewMessageCodec ¶
func NewMessageCodec( sid string, messageFactory func() Message, options ...CodecOption, ) Codec
NewMessageCodec create message codec. The message encoding format consists of a message header and a message body. Format:
- Size, 4 bytes, required. Inlucde header and body.
- Message header 2.1. Flag, 1 byte, required. 2.2. Checksum, 8 byte, optional. Set if has a checksun flag 2.3. PayloadSize, 4 byte, optional. Set if the message is a morpc.PayloadMessage. 2.4. Streaming sequence, 4 byte, optional. Set if the message is in a streaming. 2.5. Custom headers, optional. Set if has custom header codecs
- Message body 3.1. message body, required. 3.2. payload, optional. Set if has paylad flag.
type CodecOption ¶ added in v0.6.0
type CodecOption func(*messageCodec)
CodecOption codec options
func WithCodecEnableChecksum ¶ added in v0.6.0
func WithCodecEnableChecksum() CodecOption
WithCodecEnableChecksum enable checksum
func WithCodecEnableCompress ¶ added in v0.7.0
func WithCodecEnableCompress(allocator malloc.Allocator) CodecOption
WithCodecEnableCompress enable compress body and payload
func WithCodecIntegrationHLC ¶ added in v0.6.0
func WithCodecIntegrationHLC(clock clock.Clock) CodecOption
WithCodecIntegrationHLC intrgration hlc
func WithCodecMaxBodySize ¶ added in v0.6.0
func WithCodecMaxBodySize(size int) CodecOption
WithCodecMaxBodySize set rpc max body size
func WithCodecPayloadCopyBufferSize ¶ added in v0.6.0
func WithCodecPayloadCopyBufferSize(value int) CodecOption
WithCodecPayloadCopyBufferSize set payload copy buffer size, if is a PayloadMessage
type Config ¶ added in v0.8.0
type Config struct {
// MaxConnections maximum number of connections to communicate with each DNStore.
// Default is 400.
MaxConnections int `toml:"max-connections"`
// MaxIdleDuration maximum connection idle time, connection will be closed automatically
// if this value is exceeded. Default is 1 min.
MaxIdleDuration toml.Duration `toml:"max-idle-duration"`
// SendQueueSize maximum capacity of the send request queue per connection, when the
// queue is full, the send request will be blocked. Default is 10240.
SendQueueSize int `toml:"send-queue-size"`
// BusyQueueSize when the length of the send queue reaches the currently set value, the
// current connection is busy with high load. When any connection with Busy status exists,
// a new connection will be created until the value set by MaxConnections is reached.
// Default is 3/4 of SendQueueSize.
BusyQueueSize int `toml:"busy-queue-size"`
// WriteBufferSize buffer size for write messages per connection. Default is 1kb
WriteBufferSize toml.ByteSize `toml:"write-buffer-size"`
// ReadBufferSize buffer size for read messages per connection. Default is 1kb
ReadBufferSize toml.ByteSize `toml:"read-buffer-size"`
// MaxMessageSize max message size for rpc. Default is 100M
MaxMessageSize toml.ByteSize `toml:"max-message-size"`
// PayloadCopyBufferSize buffer size for copy payload to socket. Default is 16kb
PayloadCopyBufferSize toml.ByteSize `toml:"payload-copy-buffer-size"`
// EnableCompress enable compress message
EnableCompress bool `toml:"enable-compress"`
// ServerWorkers number of server workers for handle requests
ServerWorkers int `toml:"server-workers"`
// ServerBufferQueueSize queue size for server buffer requetsts
ServerBufferQueueSize int `toml:"server-buffer-queue-size"`
// GCIdleCheckInterval interval for global GC manager to check idle backends.
// Default is 1 second. This controls how often the GC manager checks for
// idle backends, not the idle timeout (which is controlled by MaxIdleDuration).
GCIdleCheckInterval toml.Duration `toml:"gc-idle-check-interval"`
// GCChannelBufferSize buffer size for GC task channels (gcInactiveC and createC).
// Default is 1024. When channel is full, requests are dropped to avoid blocking.
GCChannelBufferSize int `toml:"gc-channel-buffer-size"`
// BackendOptions extra backend options
BackendOptions []BackendOption `toml:"-"`
// ClientOptions extra client options
ClientOptions []ClientOption `toml:"-"`
// CodecOptions extra codec options
CodecOptions []CodecOption `toml:"-"`
}
Config rpc client config
func (*Config) Adjust ¶ added in v0.8.0
func (c *Config) Adjust()
Adjust adjust config, fill default value
type Future ¶
type Future struct {
// contains filtered or unexported fields
}
Future is used to obtain response data synchronously.
type HandleFunc ¶ added in v0.8.0
type HandleFunc[REQ, RESP MethodBasedMessage] func(context.Context, REQ, RESP, *Buffer) error
HandleFunc request handle func
type HandlerOption ¶ added in v0.8.0
type HandlerOption[REQ, RESP MethodBasedMessage] func(*methodBasedServer[REQ, RESP])
HandlerOption message handler option
func WithHandleMessageFilter ¶ added in v0.8.0
func WithHandleMessageFilter[REQ, RESP MethodBasedMessage](filter func(REQ) bool) HandlerOption[REQ, RESP]
WithHandleMessageFilter set filter func. Requests can be modified or filtered out by the filter before they are processed by the handler.
func WithHandlerRespReleaseFunc ¶ added in v1.2.0
func WithHandlerRespReleaseFunc[REQ, RESP MethodBasedMessage](f func(message Message)) HandlerOption[REQ, RESP]
WithHandlerRespReleaseFunc sets the respReleaseFunc of the handler.
type HeaderCodec ¶ added in v0.6.0
type HeaderCodec interface {
// Encode encode header into output buffer
Encode(*RPCMessage, *buf.ByteBuf) (int, error)
// Decode decode header from input buffer
Decode(*RPCMessage, []byte) (int, error)
}
HeaderCodec encode and decode header
type Message ¶
type Message interface {
// SetID each message has a unique ID in a RPCClient Backend. If it is a message transmitted
// in stream, the ID must be set to Stream.ID.
SetID(uint64)
// GetID returns ID of the message
GetID() uint64
// DebugString return debug string
DebugString() string
// Size size of message after marshal
ProtoSize() int
// MarshalTo marshal to target byte slice
MarshalTo(data []byte) (int, error)
// Unmarshal unmarshal from data
Unmarshal(data []byte) error
}
Message morpc is not a normal remote method call, rather it is a message-based asynchronous driven framework.
type MessageCache ¶ added in v0.8.0
type MessageCache interface {
// Add FIFO add message to cache
Add(value Message) error
// Len returns message count in the cache
Len() (int, error)
// Pop pop the first message in the cache, return false means no message in cache
Pop() (Message, bool, error)
// Close close the cache
Close()
}
MessageCache the client uses stream to send messages to the server, and when the server thinks it has not received enough messages, it can cache the messages sent by the client.
type MessagePool ¶ added in v0.8.0
type MessagePool[REQ, RESP MethodBasedMessage] interface { AcquireRequest() REQ ReleaseRequest(REQ) AcquireResponse() RESP ReleaseResponse(RESP) }
MessagePool message pool is used to reuse request and response to avoid allocate.
func NewMessagePool ¶ added in v0.8.0
func NewMessagePool[REQ, RESP MethodBasedMessage]( requestFactory func() REQ, responseFactory func() RESP, ) MessagePool[REQ, RESP]
NewMessagePool create message pool
type MethodBasedClient ¶
type MethodBasedClient[REQ, RESP MethodBasedMessage] interface { // RegisterMethod register remote getter func, used to get remote address by method. RegisterMethod(method uint32, remoteGetter func(REQ) (string, error)) // Send send request to remote, and wait for a response synchronously. Send(context.Context, REQ) (RESP, error) // AsyncSend async send request to remote. AsyncSend(context.Context, REQ) (*Future, error) // Close close the client Close() error }
MethodBasedClient is used for sending request by method.
func NewMethodBasedClient ¶
func NewMethodBasedClient[REQ, RESP MethodBasedMessage]( sid string, name string, cfg Config, pool MessagePool[REQ, RESP], ) (MethodBasedClient[REQ, RESP], error)
type MethodBasedMessage ¶ added in v0.8.0
type MethodBasedMessage interface {
Message
// Reset reset message
Reset()
// Method message type
Method() uint32
// SetMethod set message type.
SetMethod(uint32)
// WrapError wrap error into message, and transport to remote endpoint.
WrapError(error)
// UnwrapError parse error from the message.
UnwrapError() error
}
MethodBasedMessage defines messages based on Request and Response patterns in RPC. And different processing logic can be implemented according to the Method in Request.
type MethodBasedServer ¶
type MethodBasedServer[REQ, RESP MethodBasedMessage] interface { // Start start the txn server Start() error // Close the txn server Close() error // RegisterMethod register request handler func RegisterMethod(method uint32, handleFunc HandleFunc[REQ, RESP], async bool) MethodBasedServer[REQ, RESP] // Handle handle at local Handle(ctx context.Context, req REQ, buf *Buffer) RESP }
MethodBasedServer receives and handle requests from MethodBasedClient.
func NewMessageHandler ¶ added in v0.8.0
func NewMessageHandler[REQ, RESP MethodBasedMessage]( sid string, name string, address string, cfg Config, pool MessagePool[REQ, RESP], opts ...HandlerOption[REQ, RESP], ) (MethodBasedServer[REQ, RESP], error)
NewMessageHandler create a message handler.
type PayloadMessage ¶
type PayloadMessage interface {
Message
// GetPayloadField return the payload data
GetPayloadField() []byte
// SetPayloadField set the payload data
SetPayloadField(data []byte)
}
PayloadMessage is similar message, but has a large payload field. To avoid redundant copy of memory, the encoding is msgTotalSize(4 bytes) + flag(1 byte) + messageWithoutPayloadSize(4 bytes) + messageWithoutPayload + payload, all fields except payload will be written to the buffer of each link before being written to the socket. payload, being a []byte, can be written directly to the socket to avoid a copy from the buffer to the socket.
Note: When decoding, all the socket data will be read into the buffer, the payload data will not be copied from the buffer once, but directly using the slice of the buffer data to call SetPayloadField. so this is not safe and needs to be used very carefully, i.e. after processing the message back to the rpc framework, this data cannot be held.
type RPCClient ¶
type RPCClient interface {
// Send send a request message to the corresponding server and return a Future to get the
// response message.
Send(ctx context.Context, backend string, request Message) (*Future, error)
// NewStream create a stream used to asynchronous stream of sending and receiving messages.
// If the underlying connection is reset during the duration of the stream, then the stream will
// be closed. The context is used to control cancellation of stream creation; if the context
// is cancelled during connection establishment or retry backoff, NewStream will return immediately.
NewStream(ctx context.Context, backend string, lock bool) (Stream, error)
// Ping is used to check if the remote service is available. The remote service will reply with
// a pong when it receives the ping.
Ping(ctx context.Context, backend string) error
// Close close the client
Close() error
CloseBackend() error
}
RPCClient morpc is not a normal remote method call, rather it is a message-based asynchronous driven framework. Each message has a unique ID, and the response to this message must have the same ID.
func NewClient ¶
func NewClient( name string, factory BackendFactory, options ...ClientOption) (RPCClient, error)
NewClient create rpc client with options
type RPCMessage ¶ added in v0.6.0
type RPCMessage struct {
// Ctx context
Ctx context.Context
Cancel context.CancelFunc
// Message raw rpc message
Message Message
// contains filtered or unexported fields
}
RPCMessage any message sent via morpc needs to have a Context set, which is transmitted across the network. So messages sent and received at the network level are RPCMessage.
func (RPCMessage) GetTimeoutFromContext ¶ added in v0.6.0
func (m RPCMessage) GetTimeoutFromContext() (time.Duration, error)
GetTimeoutFromContext returns the timeout duration from context.
func (RPCMessage) InternalMessage ¶ added in v0.7.0
func (m RPCMessage) InternalMessage() bool
InternalMessage returns true means the rpc message is the internal message in morpc.
func (RPCMessage) Timeout ¶ added in v0.6.0
func (m RPCMessage) Timeout() bool
Timeout return true if the message is timeout
type RPCServer ¶
type RPCServer interface {
// Start start listening and wait for client messages. After each client link is established,
// a separate goroutine is assigned to handle the Read, and the Read-to message is handed over
// to the Handler for processing.
Start() error
// Close close the rpc server
Close() error
// RegisterRequestHandler register the request handler. The request handler is processed in the
// read goroutine of the current client connection. Sequence is the sequence of message received
// by the current client connection. If error returned by handler, client connection will closed.
// Handler can use the ClientSession to write response, both synchronous and asynchronous.
RegisterRequestHandler(func(ctx context.Context, request RPCMessage, sequence uint64, cs ClientSession) error)
}
RPCServer RPC server implementation corresponding to RPCClient.
func NewRPCServer ¶
func NewRPCServer( name, address string, codec Codec, options ...ServerOption) (RPCServer, error)
NewRPCServer create rpc server with options. After the rpc server starts, one link corresponds to two goroutines, one read and one write. All messages to be written are first written to a buffer chan and sent to the client by the write goroutine.
type RetryPolicy ¶
type RetryPolicy struct {
// MaxRetries is the maximum number of retry attempts.
// 0 means unlimited (rely on context timeout), which is the default behavior.
MaxRetries int
// InitialBackoff is the initial backoff duration before the first retry.
InitialBackoff time.Duration
// MaxBackoff is the maximum backoff duration.
MaxBackoff time.Duration
// Multiplier is the factor by which backoff increases after each retry.
Multiplier float64
// JitterFraction adds randomness to backoff (0.2 means ±20%).
JitterFraction float64
}
RetryPolicy defines retry behavior for morpc client operations.
type ServerOption ¶
type ServerOption func(*server)
ServerOption server options for create rpc server
func WithServerBatchSendSize ¶
func WithServerBatchSendSize(size int) ServerOption
WithServerBatchSendSize set the maximum number of messages to be sent together at each batch. Default is 8.
func WithServerDisableAutoCancelContext ¶ added in v0.6.0
func WithServerDisableAutoCancelContext() ServerOption
WithServerDisableAutoCancelContext disable automatic cancel messaging for the context. The server will receive RPC messages from the client, each message comes with a Context, and morpc will call the handler to process it, and when the handler returns, the Context will be auto cancel the context. But in some scenarios, the handler is asynchronous, so morpc can't directly cancel the context after the handler returns, otherwise many strange problems will occur.
func WithServerGoettyOptions ¶
func WithServerGoettyOptions(options ...goetty.Option) ServerOption
WithServerGoettyOptions set write filter func. Input ready to send Messages, output is really need to be send Messages.
func WithServerHandler ¶
func WithServerHandler( h func( ctx context.Context, request RPCMessage, sequence uint64, cs ClientSession, ) error, ) ServerOption
WithServerHandler sets the server handler. It is used in tests for now.
func WithServerLogger ¶
func WithServerLogger(logger *zap.Logger) ServerOption
WithServerLogger set rpc server logger
func WithServerMessageReleaseFunc ¶
func WithServerMessageReleaseFunc(release func(Message)) ServerOption
WithServerMessageReleaseFunc sets the release callback used for responses dropped before goetty takes ownership of them.
func WithServerSessionBufferSize ¶
func WithServerSessionBufferSize(size int) ServerOption
WithServerSessionBufferSize set the buffer size of the write response chan. Default is 16.
func WithServerWriteFilter ¶
func WithServerWriteFilter(filter func(Message) bool) ServerOption
WithServerWriteFilter set write filter func. Input ready to send Messages, output is really need to be send Messages.
type StatusCategory ¶
type StatusCategory int
StatusCategory classifies RPC errors into semantic categories. This allows callers to make informed decisions about error handling without checking individual error codes.
const ( // StatusOK indicates no error (success). StatusOK StatusCategory = iota // StatusTransient indicates temporary failures that may succeed on immediate retry. // Examples: timeout waiting for response, temporary network hiccup. // These errors suggest the system is working but the specific request failed. StatusTransient // Examples: backend closed, cannot connect, no available backend. // These errors suggest a connection-level problem that may recover over time. // Note: By the time caller receives this, morpc has already exhausted internal retries. StatusUnavailable // StatusCancelled indicates the operation was cancelled by the client side. // Examples: client closed, context cancelled. // Whether to retry depends on the caller's shutdown logic. StatusCancelled // StatusUnknown indicates the error category cannot be determined. // Callers should handle these conservatively based on their specific requirements. StatusUnknown )
func GetStatusCategory ¶
func GetStatusCategory(err error) StatusCategory
GetStatusCategory classifies an error into a semantic category. This provides a unified way to understand RPC errors without checking individual error codes.
Usage patterns:
- lockservice: Use IsUnavailable() to detect definitive failures
- CDC: Use IsTransient() || IsUnavailable() to detect retryable errors
func (StatusCategory) String ¶
func (s StatusCategory) String() string
String returns the string representation of the status category.
type Stream ¶
type Stream interface {
// ID returns the stream ID. All messages transmitted on the current stream need to use the
// stream ID as the message ID
ID() uint64
// Send send message to stream
Send(ctx context.Context, request Message) error
// Receive returns a channel to read stream message from server. If nil is received, the receive
// loop needs to exit. In any case, Stream.Close needs to be called.
Receive() (chan Message, error)
// Close close the stream. If closeConn is true, the underlying connection will be closed.
Close(closeConn bool) error
}
Stream used to asynchronous stream of sending and receiving messages