logging

package
v0.9.1 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Apr 23, 2026 License: Apache-2.0 Imports: 24 Imported by: 0

Documentation

Overview

Package logging provides dataset-partitioned JSONL log retention, context-propagated structured logging, and a queryable SQLite index for Thane.

Structured filesystem retention is written as append-only JSONL datasets, partitioned by dataset/date/hour: 

logs/
  events/2026-04-21/15.jsonl
  requests/2026-04-21/15.jsonl
  access/2026-04-21/15.jsonl
  loops/2026-04-21/15.jsonl
  delegates/2026-04-21/15.jsonl
  envelopes/2026-04-21/15.jsonl
  logs.db

The WithLogger / Logger helpers thread a *slog.Logger through context.Context so that every log line in a request chain automatically carries trace fields (request_id, session, conversation, subsystem, iteration index).

ShortenSource strips the module prefix from source file paths when slog's AddSource option is enabled, keeping log lines compact.

The IndexHandler wraps any slog.Handler and simultaneously indexes every log record into a SQLite database. Promoted fields (request_id, session_id, conversation_id, subsystem, tool, model) are extracted into indexed columns for fast queries; remaining attributes go into a JSON catch-all. Use Prune to manage index retention while preserving the dataset files as the canonical record.

Index

Constants

View Source 
const (
	SubsystemAgent     = "agent"
	SubsystemDelegate  = "delegate"
	SubsystemSignal    = "signal"
	SubsystemScheduler = "scheduler"
	SubsystemMetacog   = "metacog"
	SubsystemLoop      = "loop"
	SubsystemAPI       = "api"
)

Standard subsystem names for structured log filtering.

View Source 
const (
	// DatasetEvents is the low-volume operator-significant lifecycle stream.
	DatasetEvents = "events"
	// DatasetRequests captures request lifecycle and model/tool activity.
	DatasetRequests = "requests"
	// DatasetAccess captures HTTP access-style request traffic.
	DatasetAccess = "access"
	// DatasetLoops captures loop lifecycle events.
	DatasetLoops = "loops"
	// DatasetDelegates captures delegate lifecycle events.
	DatasetDelegates = "delegates"
	// DatasetEnvelopes captures message-envelope delivery audit records.
	DatasetEnvelopes = "envelopes"
)
View Source 
const (
	// KindHTTPAccess labels HTTP access-log records for the access dataset.
	KindHTTPAccess = "http_access"
	// KindRequestReceived is an alternate access-log kind for legacy
	// compatibility with earlier call sites.
	KindRequestReceived = "request_received"

	// ComponentMessageBus labels slog records originating from the
	// envelope message bus plumbing.
	ComponentMessageBus = "message_bus"
)

Shared log-attribute keys and values used to route slog records into the right dataset. Kept in one place so the handler classifier and the HTTP access middleware cannot drift against each other.

View Source 
const DefaultLiveRequestStoreSize = 512

DefaultLiveRequestStoreSize bounds the number of recent request detail records kept in memory for live inspection when archival storage is disabled.

Variables

This section is empty.

Functions

func ChainReplaceAttr 

func ChainReplaceAttr(fns ...func([]string, slog.Attr) slog.Attr) func([]string, slog.Attr) slog.Attr

ChainReplaceAttr composes multiple slog.HandlerOptions.ReplaceAttr functions into one. Each function is applied in order, so earlier functions can transform the attribute before later ones see it.

func Logger 

func Logger(ctx context.Context) *slog.Logger

Logger extracts the *slog.Logger stored by WithLogger. If no logger is present (or nil was stored), it returns slog.Default as a safe fallback so callers never need nil checks.

func Migrate 

func Migrate(db *sql.DB) error

Migrate creates or upgrades the log_entries table and indexes. Call this once after opening the database and before logging begins.

func Prune 

func Prune(db *sql.DB, maxAge time.Duration, minKeepLevel slog.Level) (int64, error)

Prune deletes log index entries older than maxAge whose level is strictly below minKeepLevel. For example, passing slog.LevelInfo prunes DEBUG and TRACE entries while keeping INFO, WARN, and ERROR. Returns the number of rows deleted.

func ShortenSource 

func ShortenSource(_ []string, a slog.Attr) slog.Attr

ShortenSource is a slog.HandlerOptions.ReplaceAttr function that strips the module prefix from source file and function paths when AddSource is enabled. This yields compact output like "internal/agent/loop.go:730" and "internal/agent.(*Loop).Run" instead of fully qualified module paths. Requires -trimpath in the build flags so Go embeds module-relative paths rather than absolute filesystem paths.

func WithLogger 

func WithLogger(ctx context.Context, logger *slog.Logger) context.Context

WithLogger returns a copy of ctx carrying logger. Retrieve it with Logger. Typically called at request entry points to inject a logger pre-enriched with trace fields (request_id, subsystem, etc.), then again at iteration boundaries to add the iteration index.

Types

type AccessResponseWriter added in v0.9.1 

type AccessResponseWriter struct {
	http.ResponseWriter
	// contains filtered or unexported fields
}

AccessResponseWriter captures HTTP status code and bytes written while preserving the optional interfaces needed by streaming handlers.

func NewAccessResponseWriter added in v0.9.1 

func NewAccessResponseWriter(w http.ResponseWriter) *AccessResponseWriter

NewAccessResponseWriter wraps w and records status/byte counts.

func (*AccessResponseWriter) BytesWritten added in v0.9.1 

func (w *AccessResponseWriter) BytesWritten() int64

BytesWritten returns the number of response bytes written so far.

func (*AccessResponseWriter) Flush added in v0.9.1 

func (w *AccessResponseWriter) Flush()

Flush preserves streaming support when the underlying writer supports it.

func (*AccessResponseWriter) Hijack added in v0.9.1 

func (w *AccessResponseWriter) Hijack() (net.Conn, *bufio.ReadWriter, error)

Hijack preserves websocket and raw-connection support when available.

func (*AccessResponseWriter) Push added in v0.9.1 

func (w *AccessResponseWriter) Push(target string, opts *http.PushOptions) error

Push preserves HTTP/2 server push when the underlying writer supports it.

func (*AccessResponseWriter) ReadFrom added in v0.9.1 

func (w *AccessResponseWriter) ReadFrom(src io.Reader) (int64, error)

ReadFrom preserves the optimized io.ReaderFrom path when available.

func (*AccessResponseWriter) StatusCode added in v0.9.1 

func (w *AccessResponseWriter) StatusCode() int

StatusCode returns the final response status, defaulting to 200 when the handler never called WriteHeader explicitly.

func (*AccessResponseWriter) Unwrap added in v0.9.1 

func (w *AccessResponseWriter) Unwrap() http.ResponseWriter

Unwrap returns the underlying http.ResponseWriter. This is the convention http.NewResponseController uses to walk through middleware wrappers to reach optional interfaces like http.ResponseController.SetReadDeadline and http.ResponseController.SetWriteDeadline. Without it, SSE and other streaming handlers behind this middleware cannot adjust their deadlines.

func (*AccessResponseWriter) Write added in v0.9.1 

func (w *AccessResponseWriter) Write(p []byte) (int, error)

Write records bytes written and defaults the status to 200 when needed.

func (*AccessResponseWriter) WriteHeader added in v0.9.1 

func (w *AccessResponseWriter) WriteHeader(code int)

WriteHeader records the status before delegating.

type Archiver added in v0.9.1 

type Archiver struct {
	// contains filtered or unexported fields
}

Archiver exports old log_request_content rows to monthly JSONL files and removes them from the database. Each archived line is a self-contained JSON object (RequestDetail with nested ToolCalls and the resolved system prompt) so the flat files need no external index to be useful.

Archive files are written to {dir}/archive/YYYY-MM.jsonl and are appended to on each run, so the operation is safe to repeat. If the process is interrupted after writing but before the database delete, some rows will be re-archived on the next run (harmless duplicate lines in the JSONL).

log_prompts rows are never deleted — they are content-addressed and bounded by system prompt variation, not request volume.

func NewArchiver added in v0.9.1 

func NewArchiver(db *sql.DB, dir string, logger *slog.Logger) *Archiver

NewArchiver creates an Archiver that writes JSONL files directly into dir. The caller is responsible for resolving the directory path (see LoggingConfig.ContentArchiveDirPath for the default).

func (*Archiver) Archive added in v0.9.1 

func (a *Archiver) Archive(ctx context.Context, before time.Time) (int, error)

Archive exports all log_request_content rows with created_at older than before to monthly JSONL files in a.dir, then deletes them (and their associated log_tool_content rows) from the database. Returns the number of requests archived.

type ContentWriter added in v0.9.1 

type ContentWriter struct {
	// contains filtered or unexported fields
}

ContentWriter writes request-level content (system prompts, tool call details, message bodies) to the log index database. It is safe for concurrent use.

func NewContentWriter added in v0.9.1 

func NewContentWriter(db *sql.DB, maxLen int, logger *slog.Logger) (*ContentWriter, error)

NewContentWriter creates a writer for the given logs.db connection. maxLen controls the maximum character count for retained content fields (tool results, message bodies). Pass 0 for unlimited.

func (*ContentWriter) Close added in v0.9.1 

func (w *ContentWriter) Close() error

Close releases prepared statements.

func (*ContentWriter) WriteRequest added in v0.9.1 

func (w *ContentWriter) WriteRequest(ctx context.Context, rc RequestContent)

WriteRequest persists a completed request's content. The system prompt is stored content-addressed (deduplicated by SHA-256 hash). Tool call arguments and results are extracted from the message history. Errors are logged but not returned — content retention is best-effort and must never block request processing.

type DatasetHandler added in v0.9.1 

type DatasetHandler struct {
	// contains filtered or unexported fields
}

DatasetHandler routes slog records into structured JSONL datasets and a separately filtered stdout handler.

func NewDatasetHandler added in v0.9.1 

func NewDatasetHandler(inner slog.Handler, writer *DatasetWriter, options DatasetHandlerOptions) *DatasetHandler

NewDatasetHandler creates a handler that writes structured dataset records and forwards only operator-significant records to stdout.

func (*DatasetHandler) Enabled added in v0.9.1 

func (h *DatasetHandler) Enabled(ctx context.Context, level slog.Level) bool

Enabled reports whether either stdout or dataset retention needs this level.

func (*DatasetHandler) Handle added in v0.9.1 

func (h *DatasetHandler) Handle(ctx context.Context, r slog.Record) error

Handle routes one slog record into the configured dataset stream and stdout. Both sinks are independent: a failure writing to the dataset file must not suppress operator-facing stdout, so errors are aggregated and returned together rather than short-circuiting.

func (*DatasetHandler) WithAttrs added in v0.9.1 

func (h *DatasetHandler) WithAttrs(attrs []slog.Attr) slog.Handler

WithAttrs returns a derived handler with attrs applied to both sinks.

func (*DatasetHandler) WithGroup added in v0.9.1 

func (h *DatasetHandler) WithGroup(name string) slog.Handler

WithGroup returns a derived handler with the given group.

type DatasetHandlerOptions added in v0.9.1 

type DatasetHandlerOptions struct {
	DatasetLevel    slog.Level
	StdoutLevel     slog.Level
	StdoutEnabled   bool
	EventsEnabled   bool
	RequestsEnabled bool
	AccessEnabled   bool
}

DatasetHandlerOptions controls how slog records are split between filesystem datasets and operator-facing stdout.

type DatasetRecord added in v0.9.1 

type DatasetRecord struct {
	EventID        string         `json:"event_id"`
	Timestamp      time.Time      `json:"ts"`
	Dataset        string         `json:"dataset"`
	Kind           string         `json:"kind"`
	SchemaVersion  int            `json:"schema_version"`
	RequestID      string         `json:"request_id,omitempty"`
	SessionID      string         `json:"session_id,omitempty"`
	ConversationID string         `json:"conversation_id,omitempty"`
	LoopID         string         `json:"loop_id,omitempty"`
	DelegateID     string         `json:"delegate_id,omitempty"`
	Source         string         `json:"source,omitempty"`
	Severity       string         `json:"severity,omitempty"`
	Payload        map[string]any `json:"payload,omitempty"`
}

DatasetRecord is one append-only structured JSONL record in a dataset stream.

func DatasetRecordFromEnvelopeAudit added in v0.9.1 

func DatasetRecordFromEnvelopeAudit(now time.Time, env messages.Envelope, result *messages.DeliveryResult, deliveryErr error) DatasetRecord

DatasetRecordFromEnvelopeAudit converts one envelope bus delivery attempt into an append-only envelopes dataset record.

func DatasetRecordFromOperationalEvent added in v0.9.1 

func DatasetRecordFromOperationalEvent(event events.Event) (DatasetRecord, bool)

DatasetRecordFromOperationalEvent converts a structured operational bus event into a dataset record for loops or delegates.

type DatasetWriter added in v0.9.1 

type DatasetWriter struct {
	// contains filtered or unexported fields
}

DatasetWriter appends structured JSONL records into dataset/date/hour segments. It keeps at most one active segment open per dataset.

func OpenDatasetWriter added in v0.9.1 

func OpenDatasetWriter(root string) (*DatasetWriter, error)

OpenDatasetWriter creates a filesystem-backed writer under root.

func (*DatasetWriter) Close added in v0.9.1 

func (w *DatasetWriter) Close() error

Close closes all active dataset segment files.

func (*DatasetWriter) Root added in v0.9.1 

func (w *DatasetWriter) Root() string

Root returns the dataset root directory.

func (*DatasetWriter) WriteRecord added in v0.9.1 

func (w *DatasetWriter) WriteRecord(record DatasetRecord) error

WriteRecord appends one structured record to the correct dataset/date/hour segment.

type IndexHandler 

type IndexHandler struct {
	// contains filtered or unexported fields
}

IndexHandler is an slog.Handler that wraps another handler and simultaneously indexes every log record into a SQLite database. The wrapped handler produces the canonical raw log output (file and/or stdout); the SQLite index enables fast queries by time, level, request ID, subsystem, etc.

Writes to SQLite happen asynchronously via a buffered channel so that logging never blocks on database I/O. The channel is drained by a single background goroutine started by NewIndexHandler. Call IndexHandler.Close to flush pending entries and stop the background goroutine.

func NewIndexHandler 

func NewIndexHandler(inner slog.Handler, db *sql.DB) *IndexHandler

NewIndexHandler wraps inner with a SQLite indexing handler. The db must be an open SQLite connection (typically from [database.Open]).

The caller must call IndexHandler.Close on shutdown to flush pending entries and release the background goroutine.

func (*IndexHandler) Close 

func (h *IndexHandler) Close()

Close flushes pending index entries and stops the background goroutine. It is safe to call multiple times.

func (*IndexHandler) Enabled 

func (h *IndexHandler) Enabled(ctx context.Context, level slog.Level) bool

Enabled reports whether the handler handles records at the given level.

func (*IndexHandler) Handle 

func (h *IndexHandler) Handle(ctx context.Context, r slog.Record) error

Handle delegates to the wrapped handler and then asynchronously indexes the record into SQLite.

func (*IndexHandler) WithAttrs 

func (h *IndexHandler) WithAttrs(attrs []slog.Attr) slog.Handler

WithAttrs returns a new handler with the given attributes pre-set.

func (*IndexHandler) WithGroup 

func (h *IndexHandler) WithGroup(name string) slog.Handler

WithGroup returns a new handler with the given group name.

type LiveRequestStore added in v0.9.1 

type LiveRequestStore struct {
	// contains filtered or unexported fields
}

LiveRequestStore keeps a bounded in-memory buffer of recent request details for live forensics independently of any persistent log index.

func NewLiveRequestStore added in v0.9.1 

func NewLiveRequestStore(maxEntries, maxLen int) *LiveRequestStore

NewLiveRequestStore creates a bounded in-memory request detail store. maxEntries defaults to DefaultLiveRequestStoreSize when non-positive. maxLen follows the same semantics as logging max content length: non-positive means unlimited.

func (*LiveRequestStore) QueryRequestDetail added in v0.9.1 

func (s *LiveRequestStore) QueryRequestDetail(requestID string) (*RequestDetail, error)

QueryRequestDetail returns a copy of the stored request detail, or nil when the request is no longer present in the live buffer.

func (*LiveRequestStore) WriteRequest added in v0.9.1 

func (s *LiveRequestStore) WriteRequest(_ context.Context, rc RequestContent)

WriteRequest stores the latest request detail snapshot in memory.

type LogEntry 

type LogEntry struct {
	ID             int64
	Timestamp      time.Time
	Level          string
	Msg            string
	RequestID      string
	SessionID      string
	ConversationID string
	Subsystem      string
	Tool           string
	Model          string
	LoopID         string
	LoopName       string
	Attrs          string
	SourceFile     string
	SourceLine     int
}

LogEntry is an exported representation of a log index row suitable for display in the web dashboard and tool queries.

func Query added in v0.8.3 

func Query(db *sql.DB, params QueryParams) ([]LogEntry, error)

Query returns log entries matching the given filter parameters, ordered by timestamp ascending (chronological). The limit defaults to 50 and is capped at 200. Level filtering is "minimum level" — e.g., WARN returns WARN and ERROR entries.

func QueryBySession 

func QueryBySession(db *sql.DB, sessionID, level, subsystem string, limit int) ([]LogEntry, error)

QueryBySession returns log entries matching the given session ID, ordered by timestamp ascending (chronological). When limit is positive, only the most recent limit entries are returned — the query selects newest-first and then reverses in Go so callers always receive chronological order. Optional filters narrow by level and/or subsystem.

type QueryParams added in v0.8.3 

type QueryParams struct {
	SessionID             string
	ConversationID        string
	RequestID             string
	Subsystem             string
	Tool                  string
	Model                 string
	LoopID                string
	LoopName              string
	Level                 string    // minimum level: ERROR > WARN > INFO > DEBUG
	Since                 time.Time // zero = no lower bound
	Until                 time.Time // zero = defaults to now
	Pattern               string    // substring match on msg
	SourceFilePrefix      string    // prefix match on source_file (e.g., "cmd/thane/")
	ExcludeSourcePrefixes []string  // exclude entries whose source_file starts with any of these
	Limit                 int       // default 50, max 200
}

QueryParams holds filter criteria for querying the log index. All fields are optional — zero values are ignored. Level is treated as a minimum severity: WARN returns WARN and ERROR entries, DEBUG returns everything including TRACE.

type RequestContent added in v0.9.1 

type RequestContent struct {
	RequestID    string
	SystemPrompt string // full assembled system prompt
	UserContent  string // inbound user message
	Model        string

	// From iterate.Result:
	AssistantContent string
	IterationCount   int
	InputTokens      int
	OutputTokens     int
	ToolsUsed        map[string]int
	Exhausted        bool
	ExhaustReason    string

	// Full message history for tool call extraction.
	Messages []llm.Message
}

RequestContent holds the data to persist for a completed request.

type RequestDetail added in v0.9.1 

type RequestDetail struct {
	RequestID        string         `json:"request_id"`
	PromptHash       string         `json:"prompt_hash,omitempty"`
	SystemPrompt     string         `json:"system_prompt,omitempty"`
	UserContent      string         `json:"user_content,omitempty"`
	AssistantContent string         `json:"assistant_content,omitempty"`
	Model            string         `json:"model,omitempty"`
	IterationCount   int            `json:"iteration_count"`
	InputTokens      int            `json:"input_tokens"`
	OutputTokens     int            `json:"output_tokens"`
	ToolsUsed        map[string]int `json:"tools_used,omitempty"`
	Exhausted        bool           `json:"exhausted"`
	ExhaustReason    string         `json:"exhaust_reason,omitempty"`
	CreatedAt        string         `json:"created_at"`
	ToolCalls        []ToolDetail   `json:"tool_calls"`
}

RequestDetail holds the full content retained for a single request, ready for JSON serialization by the web API.

func QueryRequestDetail added in v0.9.1 

func QueryRequestDetail(db *sql.DB, requestID string) (*RequestDetail, error)

QueryRequestDetail fetches the full retained content for a request by its request ID. Returns nil, nil if the request is not found or if content retention was not active when the request was processed.

type RequestRecordFunc added in v0.9.1 

type RequestRecordFunc func(ctx context.Context, rc RequestContent)

RequestRecordFunc captures completed request content for later inspection.

func CombineRequestRecorders added in v0.9.1 

func CombineRequestRecorders(recorders ...RequestRecordFunc) RequestRecordFunc

CombineRequestRecorders fan-outs request content to every non-nil recorder. It returns nil when no recorders are provided.

type ToolDetail added in v0.9.1 

type ToolDetail struct {
	ToolCallID     string `json:"tool_call_id,omitempty"`
	ToolName       string `json:"tool_name"`
	Arguments      string `json:"arguments,omitempty"`
	Result         string `json:"result,omitempty"`
	IterationIndex int    `json:"iteration_index"`
}

ToolDetail holds the retained content for a single tool invocation.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.