haproxy

package module
v0.0.6 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Sep 1, 2022 License: MIT Imports: 8 Imported by: 3

README

godoc

HAProxy utilities and helpers for Golang

parsing HTTP log from syslog format (WiP)

Most of the fields are decoded, log accepts format of log 127.0.0.1:50514 local3 debug. The syslog facility is ignored but the app name must be haproxy

That's how basic ingestor can look like:

func (i *Ingest) ingestor(conn *net.UDPConn, ch chan haproxy.HTTPRequest) {
	buf := make([]byte, 65535)
	for {
		n, _, err := conn.ReadFromUDP(buf)
		log_str := string(buf[0:n])
		if err != nil {
			i.l.Errorf("Error: %s", err)
		}
		if strings.Contains(log_str, " SSL handshake") {
			continue
		}
		req, err := haproxy.DecodeHTTPLog(log_str)
		if err != nil {
			continue
		}
		ch <- req
	}
}
Quirks
Request/response variable capture

This is how capture request only block looks like: {}

This is how capture response only block looks like: {}

This is how config with both looks like in log: {} {}

This code is not and will not guess on that, it just dumps the whole into CapturedHeaders string

Handling truncated requests

The usual reason is the last part, HTTP request part, not fitting the packet.

It will be parsed on best effort basis and will have Truncated=true set in the structure

Time handling

HAProxy sends logs in local time, so we decode it in local time. There is a global variable HaproxyLogTimezone that will set the zone used to decode time.

Testing

add your local log lines to t-data/haproxy_log_local and they will be used instead of ones in repo (that file is in gitignore)

to generate it just run haproxy with log 127.0.0.1:50514 local3 debug in global section and record it with nc -l -u 50514 > /tmp/log

Documentation

Index

Examples

Constants

View Source 
const (
	TerminationClientAbort     = 'C'
	TerminationServerAbort     = 'S'
	TerminationDeny            = 'P'
	TerminationLocal           = 'L'
	TerminationExhausted       = 'R'
	TerminationInternalError   = 'I'
	TerminationServerGoingDown = 'D'
	TerminationActiveUp        = 'U'
	TerminationAdmin           = 'K'
	TerminationClientWait      = 'c'
	TerminationServerWait      = 's'
	TerminationNone            = '-'
)

https://cbonte.github.io/haproxy-dconv/2.0/configuration.html#8.5

View Source 
const (
	SessionCloseRequest    = 'R'
	SessionCloseQueue      = 'Q'
	SessionCloseConnection = 'C'
	SessionCloseHeaders    = 'H'
	SessionCloseData       = 'D'
	SessionCloseLast       = 'L'
	SessionCloseTarpit     = 'T'
	SessionCloseNone       = '-'
)
View Source 
const (
	// TYPED-DATA    : <TYPE:4 bits><FLAGS:4 bits><DATA>
	DataTypeNULL   = 0 //<0>
	DataTypeBOOL   = 1 //<1+FLAG>
	DataTypeINT32  = 2 //<2><VALUE:varint>
	DataTypeUINT32 = 3 //<3><VALUE:varint>
	DataTypeINT64  = 4 //<4><VALUE:varint>
	DataTypeUNIT64 = 5 //<5><VALUE:varint>
	DataTypeIPV4   = 6 //<6><STRUCT IN_ADDR:4 bytes>
	DataTypeIPV6   = 7 //<7><STRUCT IN_ADDR6:16 bytes>
	DataTypeSTRING = 8 //<8><LENGTH:varint><BYTES>
	DataTypeBINARY = 9 //<9><LENGTH:varint><BYTES>
	DataTypeRSVD1  = 10
	DataTypeRSVD2  = 11
	DataTypeRSVD3  = 12
	DataTypeRSVD4  = 13
	DataTypeRSVD5  = 14
	DataTypeRSVD6  = 15
)

4 bits TYPED-DATA : <TYPE:4 bits><FLAGS:4 bits><DATA>

Variable-length integer (varint) are encoded using Peers encoding: 

      0  <= X < 240        : 1 byte  (7.875 bits)  [ XXXX XXXX ]
     240 <= X < 2288       : 2 bytes (11 bits)     [ 1111 XXXX ] [ 0XXX XXXX ]
    2288 <= X < 264432     : 3 bytes (18 bits)     [ 1111 XXXX ] [ 1XXX XXXX ]   [ 0XXX XXXX ]
  264432 <= X < 33818864   : 4 bytes (25 bits)     [ 1111 XXXX ] [ 1XXX XXXX ]*2 [ 0XXX XXXX ]
33818864 <= X < 4328786160 : 5 bytes (32 bits)     [ 1111 XXXX ] [ 1XXX XXXX ]*3 [ 0XXX XXXX ]
View Source 
const (
	FrameHaproxyHello      = 1   // Sent by HAProxy when it opens a connection on an agent.
	FrameHaproxyDisconnect = 2   // Sent by HAProxy when it want to close the connection or in reply to an AGENT-DISCONNECT frame
	FrameNotify            = 3   // Sent by HAProxy to pass information to an agent
	FrameAgentHello        = 101 // Reply to a HAPROXY-HELLO frame, when the connection is established
	FrameAgentDDisconnect  = 102 // Sent by an agent just before closing the connection
	FrameAck               = 103 // Sent to acknowledge a NOTIFY frame
)

Variables

View Source 
var HaproxyLogTimezone = time.Local

HaproxyLogTimezone specifies in what timezone emitted logs are defaults to local as haproxy by default uses localtime in logs

Functions

This section is empty.

Types

type ACL 

type ACL struct {
	// haproxy-assigned ID
	ID int `json:"id"`
	// path of external file that is source of this ACL entries
	SourceFile string `json:"source_file"`
	// type of ACL if it is from config or "file" if it sourced from file (haproxy doesnt return type here)
	Type string `json:"type"`
	// Line of inline ACL
	Line int `json:"line"`
}

type Conn 

type Conn struct {
	// contains filtered or unexported fields
}

HAProxy socket interface

func New 

func New(path string) Conn

Setup new connection accepts path to haproxy unix socket

Example 
// start
ha := New("/var/run/haproxy.sock")
// clear all entries in ACL
_ = ha.ClearACL("inc/blacklist.lst")

func (*Conn) AddACL 

func (c *Conn) AddACL(acl string, pattern string) error

Add new entry to acl ACL name is either file path ( if haproxy config uses -f option to load acls from file ) or ACL id prepended with hash

Example 
// Initialize
ha := New("/var/run/haproxy.sock")

// Get ACL entries from file in config (via -f in haproxy cfg)
acls, _ := ha.GetACL("inc/blacklist.lst")

// Check if it exists and add
if acls["/bad/path"] == "" {
	ha.AddACL("inc/blacklist.lst", acls["/bad/path"])
}

func (*Conn) ClearACL 

func (c *Conn) ClearACL(acl string) error

Clear all entries in ACL

Example 
// Initialize
ha := New("/var/run/haproxy.sock")

// clear all entries in ACL
_ = ha.ClearACL("inc/blacklist.lst")

func (*Conn) DeleteACL 

func (c *Conn) DeleteACL(acl string, id string) error
Example 
// Initialize
ha := New("/var/run/haproxy.sock")

// Get ACL entries from file in config (via -f in haproxy cfg)
acls, _ := ha.GetACL("inc/blacklist.lst")

// Check if it exists and delete
if acls["/bad/path"] != "" {
	ha.DeleteACL("inc/blacklist.lst", acls["/bad/path"])
}

func (*Conn) GetACL 

func (c *Conn) GetACL(acl string) (map[string]string, error)
Example 
// Initialize
ha := New("/var/run/haproxy.sock")

// Get ACL entries from file in config (via -f in haproxy cfg)
acls, _ := ha.GetACL("inc/blacklist.lst")

for k, v := range acls {
	fmt.Println(k, ":\t", v)
}

func (*Conn) ListACL 

func (c *Conn) ListACL() ([]ACL, error)

List all ACLs that haproxy currenty uses

func (*Conn) ListACLFiles 

func (c *Conn) ListACLFiles() (map[string]ACL, error)

Return map with all external files that are used as ACL entry source with first occurence of ACL as a value

func (*Conn) RunCmd 

func (c *Conn) RunCmd(cmd string) ([]string, error)

Run arbitrary haproxy command and return output

type Frame 

type Frame struct {
	Length    uint32
	FrameType uint8
	Frame     struct {
		Metadata uint64 `type:"varint"`
	}
}

type FrameMetadata 

type FrameMetadata struct {
	Flags    [4]byte
	StreamId uint64 `type:"varint"`
	FrameId  uint64 `type:"varint"`
}

type HTTPRequest 

type HTTPRequest struct {
	TS                     int64  `json:"ts_us"`
	PID                    int    `json:"pid"` // necessary to distinguish conns hitting different processes
	ClientIP               string `json:"client_ip"`
	ClientPort             uint16 `json:"client_port"`
	ClientSSL              bool   `json:"client_ssl"`
	FrontendName           string `json:"frontend_name"`
	BackendName            string `json:"backend_name"`
	ServerName             string `json:"server_name"`
	StatusCode             int16  `json:"status_code"`
	BytesRead              uint64 `json:"bytes_read"`
	CapturedRequestCookie  string `json:"captured_request_cookie"`
	CapturedResponseCookie string `json:"captured_response_cookie"`
	// It is not possible to distinguish between "just captured request headers" and "just captured response headers"
	// so we let user sort it out
	CapturedSamples string
	// timings
	// aborted connections are marked via -1 by haproxy
	RequestHeaderDurationMs  int `json:"request_header_duration_ms"`  // Tq
	QueueDurationMs          int `json:"queue_duration_ms"`           // Tw
	ServerConnDurationMs     int `json:"server_conn_duration_ms"`     // Tc
	ResponseHeaderDurationMs int `json:"response_header_duration_ms"` // Tr
	TotalDurationMs          int `json:"total_duration_ms"`           // Tt

	RequestPath   string `json:"http_path"`
	RequestMethod string `json:"http_method"`
	HTTPVersion   string `json:"http_version"`

	// Connection state
	TerminationReason      rune `json:"termination_reason"`
	SessionCloseState      rune `json:"session_close_state"`
	ClientPersistenceState rune `json:"client_persistence_state"`
	PersistenceCookieState rune `json:"persistence_cookie"`

	// conn count stats (per-pid
	TotalConn    uint `json:"total_conn"`
	FrontendConn uint `json:"frontend_conn"`
	BackendConn  uint `json:"backend_conn"`
	ServerConn   uint `json:"server_conn"`
	Retries      uint `json:"retries"`
	ServerQueue  uint `json:"server_queue"`
	BackendQueue uint `json:"backend_queue"`

	// flags
	BadReq    bool `json:"bad_request"`
	Truncated bool `json:"truncated"`
}

HAProxy http log format https://cbonte.github.io/haproxy-dconv/configuration-1.5.html#8.2.3

func DecodeHTTPLog 

func DecodeHTTPLog(s string) (HTTPRequest, error)

func (HTTPRequest) Timestamp added in v0.0.5 

func (h HTTPRequest) Timestamp() time.Time

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.