rdma

package
v1.8.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Sep 4, 2026 License: Apache-2.0 Imports: 6 Imported by: 0

Documentation

Overview

Package rdma provides Go bindings to libcuobjserver via CGO.

Index

Constants

View Source
const MaxTransferSize = 1 << 30

MaxTransferSize is the cuObjServer limit per RDMA operation (1 GiB).

Variables

This section is empty.

Functions

func ConfigureTelemetry

func ConfigureTelemetry(debug bool)

ConfigureTelemetry controls cuObjServer telemetry logging. When debug is true, enable info+debug+error logs to stderr/stdout path.

func DebugTelemetryEnabled

func DebugTelemetryEnabled() bool

DebugTelemetryEnabled reports whether verbose RDMA diagnostics are enabled.

Types

type Buffer

type Buffer struct {
	// contains filtered or unexported fields
}

Buffer wraps an RDMA-registered memory region.

func (*Buffer) HostPtr

func (b *Buffer) HostPtr() unsafe.Pointer

HostPtr returns the underlying host memory pointer.

func (*Buffer) Size

func (b *Buffer) Size() int

Size returns the buffer size in bytes.

func (*Buffer) Slice

func (b *Buffer) Slice() []byte

Slice returns the buffer contents as a Go byte slice backed by C-allocated memory. Returns nil after DeregisterBuffer.

type RDMATunables

type RDMATunables struct {
	// NumDCIs controls the maximum number of concurrent RDMA connections
	// (Dynamic Connection Interfaces). Default: 128.
	NumDCIs int
	// CQDepth is the completion queue depth, limiting outstanding ops.
	// Default: 640.
	CQDepth uint32
	// DCKey is the InfiniBand Dynamic Connection security key. All clients
	// must use a matching key. Change this from the default in production.
	// Default: 0xffeeddcc.
	DCKey uint64
	// ServiceLevel sets the IB QoS service level. Default: 0.
	ServiceLevel int
	// Timeout is the QP ACK timeout exponent: 4.096 * 2^Timeout µs.
	// Default: 16 (~268 ms).
	Timeout uint8
	// HopLimit is the IB packet hop limit (analogous to IP TTL). Default: 4.
	HopLimit uint32
	// PKeyIndex is the IB partition key index. Default: 0.
	PKeyIndex int
	// DelayInterval is the polling delay in nanoseconds. Default: 5000.
	DelayInterval uint32
	// DelayMode selects the polling strategy:
	//   0 = none, 1 = batch (default), 2 = per-entry, 3 = adaptive.
	DelayMode int
	// RetryCount is the QP retry count (0–7). Default: 7.
	RetryCount uint8
	// QPResetOnFailure controls whether the QP is reset after an RDMA failure.
	// Default: true.
	QPResetOnFailure bool
	// TrafficClass sets the IB traffic class / DSCP bits. Default: 96.
	TrafficClass uint32
	// MaxRdAtomic is the max outstanding RDMA reads per DCI QP.
	// 0 means auto-detect from device capabilities. Default: 0.
	MaxRdAtomic int
}

RDMATunables holds RDMA connection tuning parameters that map to cuObjRDMATunableParam. Pass a non-nil pointer in CuServerOpts to apply custom settings before the RDMA session starts; nil means library defaults.

Use DefaultRDMATunables() to start from the library defaults and adjust only the fields you care about.

func DefaultRDMATunables

func DefaultRDMATunables() RDMATunables

DefaultRDMATunables returns a RDMATunables pre-populated with the cuObjServer library defaults. Use this as a starting point and override only the fields you need.

type Server

type Server struct {
	// contains filtered or unexported fields
}

Server wraps a cuObjServer instance.

func NewServer

func NewServer(ip string, port uint16, tunables *RDMATunables) (*Server, error)

NewServer creates a cuObjServer bound to the given RDMA IP and port. Uses CUOBJ_PROTO_RDMA_DC_V1 (1001). If tunables is non-nil, the 4-argument constructor is used so the tunable parameters apply to the initial session started by the constructor. Pass nil to use library defaults.

func (*Server) AllocHostBuffer

func (s *Server) AllocHostBuffer(size int) (unsafe.Pointer, error)

AllocHostBuffer allocates a 4KB-aligned host buffer of the given size.

func (*Server) AllocateChannel

func (s *Server) AllocateChannel() (uint16, error)

AllocateChannel obtains a unique channel ID for concurrent RDMA operations.

func (*Server) Close

func (s *Server) Close()

Close destroys the cuObjServer instance. The Server must not be used afterward.

func (*Server) CloseSession

func (s *Server) CloseSession()

CloseSession tears down the RDMA session.

func (*Server) DeregisterBuffer

func (s *Server) DeregisterBuffer(buf *Buffer)

DeregisterBuffer deregisters a previously registered RDMA buffer.

func (*Server) FreeChannel

func (s *Server) FreeChannel(id uint16)

FreeChannel releases a previously allocated channel ID.

func (*Server) FreeHostBuffer

func (s *Server) FreeHostBuffer(ptr unsafe.Pointer)

FreeHostBuffer releases a buffer previously allocated by AllocHostBuffer.

func (*Server) HandleGet

func (s *Server) HandleGet(key string, buf *Buffer, remoteStart uint64, size int64, rdmaDescr string, channel uint16) (int64, error)

HandleGet performs an RDMA WRITE (server→client) to serve a GET request. The local buffer must already contain the data to send. Returns bytes transferred.

func (*Server) HandlePut

func (s *Server) HandlePut(key string, buf *Buffer, remoteStart uint64, size int64, rdmaDescr string, channel uint16) (int64, error)

HandlePut performs an RDMA READ (client→server) to serve a PUT request. After return, the local buffer contains the data read from the client. Returns bytes transferred.

func (*Server) InitRDMAConfig

func (s *Server) InitRDMAConfig(t RDMATunables) error

InitRDMAConfig applies RDMA tuning parameters. Must be called before StartSession.

func (*Server) IsConnected

func (s *Server) IsConnected() bool

IsConnected returns the RDMA connection status.

func (*Server) RegisterBuffer

func (s *Server) RegisterBuffer(ptr unsafe.Pointer, size int) (*Buffer, error)

RegisterBuffer registers a host memory region for RDMA and returns a Buffer handle.

func (*Server) StartSession

func (s *Server) StartSession() error

StartSession initiates the RDMA listening session. StartSession must not be called concurrently with CloseSession or Close.

Directories

Path Synopsis
Package bufferpool provides a pool of pre-registered RDMA buffers to avoid per-request allocation and registration overhead.
Package bufferpool provides a pool of pre-registered RDMA buffers to avoid per-request allocation and registration overhead.
Package hostclient implements the client side of the cuObject RDMA DC protocol using libibverbs + mlx5 direct verbs, with no CUDA/GPU dependency.
Package hostclient implements the client side of the cuObject RDMA DC protocol using libibverbs + mlx5 direct verbs, with no CUDA/GPU dependency.
Package rcroutes serves the /.hipobj-rc/{prepare,ready,cancel} terminal routes of the hipobj-rc-v2 two-phase transfer protocol.
Package rcroutes serves the /.hipobj-rc/{prepare,ready,cancel} terminal routes of the hipobj-rc-v2 two-phase transfer protocol.
Package rcserver binds the RC session server (rdma/librcserver.a) that implements the hipobj-rc-v2 two-phase transfer protocol.
Package rcserver binds the RC session server (rdma/librcserver.a) that implements the hipobj-rc-v2 two-phase transfer protocol.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL