logstore

package
v0.4.89 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Nov 7, 2026 License: Apache-2.0 Imports: 21 Imported by: 0

Documentation

Overview

Package logstore keeps container logs as per-day text files. Standard library only.

Index

Constants

View Source
const MaxLineBytes = 16 << 10
View Source
const MaxPageBytes = 2 << 20
View Source
const QueueSize = 10_000

QueueSize is the per-subscriber backlog; beyond it the oldest message is dropped.

Variables

View Source
var (
	ErrBadFilter      = errors.New("bad filter")
	ErrBothDirections = errors.New("since and before do not combine")
)
View Source
var Kinds = []string{"created", "started", "stopped", "died", "oom-killed", "restarting", "unhealthy", "healthy", "installed", "uninstalled", "updated", "image-pulled"}

Functions

func AppendEvent

func AppendEvent(root string, e Event) error

AppendEvent writes the event as one line in a single Write call, so concurrent appenders do not interleave.

func DayFile

func DayFile(root, app, service, container string, day time.Time) string

func EventsFile

func EventsFile(root string, day time.Time) string

func FormatLine

func FormatLine(at time.Time, stream, text string) []byte

FormatLine returns "<RFC3339Nano> <o|e> <text>\n" with \r and \n in text replaced by spaces.

func GuessLevel

func GuessLevel(stream, text string) string

func LastTimestamp

func LastTimestamp(root, app, service, container string) (time.Time, bool)

LastTimestamp returns the time of the last parsable line of the container's newest day file.

func ParseLine

func ParseLine(b []byte) (at time.Time, stream, text string, ok bool)

ParseLine reads a line produced by FormatLine (without its trailing newline).

func PlanPurge

func PlanPurge(files []FileInfo, now time.Time, retentionDays int, maxBytes int64) []string

PlanPurge returns the paths to remove: days older than the retention window, then whole oldest days until the rest fits maxBytes. Files with a zero Day are never planned. retentionDays<=0 and maxBytes<=0 disable their rule.

func Purge

func Purge(root string, now time.Time, retentionDays int, maxBytes int64) (removed int64, err error)

Purge applies PlanPurge to the store on disk, drops <app>/<service> and <app> directories it emptied, and returns the bytes of the files removed.

func SafeName

func SafeName(s string) string

func Truncate

func Truncate(text string) string

func Usage

func Usage(root string) (bytes int64, oldest time.Time, err error)

Types

type AppSummary

type AppSummary struct {
	App      string           `json:"app"`
	Services []ServiceSummary `json:"services"`
}

func ListApps

func ListApps(root string) ([]AppSummary, error)

type Cursor

type Cursor struct {
	At   time.Time
	File string // path relative to root
	Seq  int64  // byte offset of the line's start in File: ordered like the lines, stable under appends, seekable
}

func ParseCursor

func ParseCursor(s string) (Cursor, error)

ParseCursor wraps its error in ErrBadFilter: a cursor comes from the client like the filter does.

func (Cursor) String

func (c Cursor) String() string

type Event

type Event struct {
	At        time.Time `json:"at"`
	App       string    `json:"app"`
	Service   string    `json:"service"`
	Container string    `json:"container"`
	Kind      string    `json:"kind"`
	ExitCode  string    `json:"exit_code,omitempty"`
	FromImage string    `json:"from_image,omitempty"`
	ToImage   string    `json:"to_image,omitempty"`
	By        string    `json:"by"` // "operation" | "docker"
	Detail    string    `json:"detail,omitempty"`
}

func ReadEvents

func ReadEvents(root string, q EventQuery) ([]Event, error)

ReadEvents returns events newest first. Day files outside the Since/Until days are never opened.

type EventQuery

type EventQuery struct {
	Apps  []string
	Kinds []string
	Since time.Time // zero = no bound
	Until time.Time // zero = no bound
	Limit int       // ≤ 1000, default 500
}

type FileInfo

type FileInfo struct {
	Path string
	Day  time.Time // UTC midnight parsed from the name; zero for a file the layout does not know
	Size int64
}

func ListFiles

func ListFiles(root string) ([]FileInfo, error)

ListFiles returns every file under root; only <app>/<service>/<day>.*.log and _events/<day>.jsonl get a Day. A missing root is an empty store.

type Filter

type Filter struct {
	Apps, Services []string
	Stream         string // "", "o", "e"
	Level          string // "", "info", "warn", "error" (minimum level)
	Q              string
	Regex          bool
}

func (Filter) Compile

func (f Filter) Compile() (*Matcher, error)

type Hub

type Hub struct {
	// contains filtered or unexported fields
}

Hub fans messages out to subscribers. Publish never blocks: each subscriber has its own bounded queue.

func NewHub

func NewHub() *Hub

func (*Hub) Publish

func (h *Hub) Publish(m Message)

func (*Hub) Subscribe

func (h *Hub) Subscribe(m *Matcher) (<-chan Message, func())

type Line

type Line struct {
	At        time.Time `json:"at"`
	App       string    `json:"app"`
	Service   string    `json:"service"`
	Container string    `json:"container"` // 12 hex chars
	Stream    string    `json:"stream"`    // "o" or "e"
	Text      string    `json:"text"`
	Cursor    string    `json:"cursor,omitempty"` // set by Read only: where the line is, to pass as since or before
}

type Matcher

type Matcher struct {
	// contains filtered or unexported fields
}

func (*Matcher) Match

func (m *Matcher) Match(l Line) bool

func (*Matcher) MatchEvent

func (m *Matcher) MatchEvent(e Event) bool

type Message

type Message struct {
	T       string `json:"t"` // "line" | "event" | "lag"
	Line    *Line  `json:"line,omitempty"`
	Event   *Event `json:"event,omitempty"`
	Dropped int    `json:"dropped,omitempty"`
}

type Page

type Page struct {
	Lines  []Line  `json:"lines"`
	Events []Event `json:"events"`
	Next   string  `json:"next,omitempty"` // "" when exhausted
}

func Read

func Read(ctx context.Context, root string, q Query) (Page, error)

Read returns one page of history. Lines are ordered on (At, File, Seq), descending without After, ascending with it; a day file's lines are assumed to be in time order, as the writer appends them. Forwards, the lines of the last second are held back (see holdBack).

type Query

type Query struct {
	Filter
	Since  time.Time // with After: lower bound of the window; otherwise zero
	Until  time.Time
	Before *Cursor // backwards (newest→oldest) strictly before; nil = from the end
	After  *Cursor // forwards (oldest→newest) strictly after; nil = backwards mode
	Limit  int     // clamped to [1,1000], default 500
}

type ServiceSummary

type ServiceSummary struct {
	Service   string    `json:"service"`
	Container string    `json:"container"`
	Running   bool      `json:"running"` // set by the caller (service layer)
	FirstAt   time.Time `json:"first_at"`
	LastAt    time.Time `json:"last_at"`
	SizeBytes int64     `json:"size_bytes"`
}

type Store

type Store struct {
	Root string
	// OnError, if set before use, receives every writer's flush error (nil = ignored).
	OnError func(error)
	// contains filtered or unexported fields
}

Store is the single entry point of the package: it owns the writers, the hub and the root directory.

func Open

func Open(ctx context.Context, root string) (*Store, error)

Open creates root if needed, but never its grandparent (/DATA/AppData for /DATA/AppData/casaos/logs): while the data disk is missing nothing is written to the system disk, and the caller retries. An existing but unmounted mount point cannot be told apart from the disk itself. A symlinked root is resolved, so the purge walks the files it points to. ctx bounds the writers' flush loops.

func (*Store) Apps

func (s *Store) Apps() ([]AppSummary, error)

func (*Store) Close

func (s *Store) Close() error

Close flushes and closes every writer and returns the first error.

func (*Store) CloseWriter

func (s *Store) CloseWriter(container string) error

CloseWriter flushes, closes and forgets the container's writer; unknown ids are a no-op.

func (*Store) Events

func (s *Store) Events(q EventQuery) ([]Event, error)

func (*Store) Purge

func (s *Store) Purge(now time.Time, retentionDays int, maxBytes int64) (int64, error)

Purge makes every writer drop its open file afterwards: a removed file still held open would keep taking lines no reader sees, on space never freed; the next flush creates the file again. The files are also dropped before, so that a system refusing to remove an open file removes it.

func (*Store) Read

func (s *Store) Read(ctx context.Context, q Query) (Page, error)

Read flushes every writer first: any line's cursor may start a forwards poll, which must not pass a line still in a buffer.

func (*Store) Record

func (s *Store) Record(e Event) error

Record appends the event to the journal, then publishes it; nothing is published on error.

func (*Store) Subscribe

func (s *Store) Subscribe(m *Matcher) (<-chan Message, func())

func (*Store) Usage

func (s *Store) Usage() (int64, time.Time, error)

func (*Store) Writer

func (s *Store) Writer(app, service, container string) *Writer

Writer returns the container's writer, creating and starting it on first use.

type Writer

type Writer struct {

	// OnError, if set before use, receives every flush error (nil = ignored). The failed buffer is dropped.
	OnError func(error)
	// contains filtered or unexported fields
}

Writer appends one container's lines to its per-day files through a buffer.

func NewWriter

func NewWriter(root, app, service, container string, onLine func(Line)) *Writer

onLine runs synchronously inside Write and must stay cheap; nil is allowed.

func (*Writer) Close

func (w *Writer) Close() error

func (*Writer) Flush

func (w *Writer) Flush() error

Flush does not fsync: the 1 s crash window is accepted by the spec.

func (*Writer) Run

func (w *Writer) Run(ctx context.Context)

Run does not close the writer when ctx ends; the caller still calls Close.

func (*Writer) Write

func (w *Writer) Write(at time.Time, stream, text string) error

Write always buffers the line and always calls onLine. A flush error (day change or 64 KiB) is returned and sent to OnError; it concerns earlier lines only.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL