ingestlog

package
v0.0.0-...-9a2a36c Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Aug 8, 2026 License: Apache-2.0 Imports: 10 Imported by: 0

Documentation

Overview

Package ingestlog provides structured logging for ingest endpoint operations.

Package ingestlog provides structured logging for ingest endpoint operations.

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func CaptureEndpointName

func CaptureEndpointName(endpoint string) (string, error)

CaptureEndpointName creates an endpoint string with validation. This helper function accepts an endpoint parameter and returns it with validation.

Parameters:

  • endpoint: Endpoint identifier (e.g., "github-username-resolution", required)

Returns:

  • The endpoint string
  • An error if validation fails (empty endpoint)

func CaptureMethod

func CaptureMethod(method string) (string, error)

CaptureMethod creates a method string with validation. This helper function accepts an HTTP method parameter and returns it with validation.

Parameters:

  • method: HTTP method (GET, POST, etc., required)

Returns:

  • The method string
  • An error if validation fails (empty method)

func CapturePath

func CapturePath(path string) (string, error)

CapturePath creates a path string with validation. This helper function accepts a path parameter and returns it with validation.

Parameters:

  • path: Request path (required)

Returns:

  • The path string
  • An error if validation fails (empty path)

func CaptureRequestID

func CaptureRequestID(requestID string) string

CaptureRequestID creates a requestID string with validation. This helper function accepts a requestID parameter and returns it with validation.

Parameters:

  • requestID: Current request identifier (optional, can be empty string)

Returns:

  • The requestID string (empty string if not provided)

func CaptureSessionID

func CaptureSessionID(sessionID string) string

CaptureSessionID creates a sessionID string with validation. This helper function accepts a sessionID parameter and returns it with validation.

Parameters:

  • sessionID: User's current session identifier (optional, can be empty string)

Returns:

  • The sessionID string (empty string if not provided)

func CaptureUserID

func CaptureUserID(userID string) string

CaptureUserID creates a userID string with validation. This helper function accepts a userID parameter and returns it with validation.

Parameters:

  • userID: User's unique identifier (optional, can be empty string)

Returns:

  • The userID string (empty string if not provided)
  • nil (always succeeds - userID is optional)

func GetErrorChain

func GetErrorChain(err error) []string

GetErrorChain returns the chain of wrapped errors as a slice of error types. This is useful for understanding the full error wrapping hierarchy.

Parameters:

  • err: The error to analyze (can be nil)

Returns:

  • []string: Slice of error type names in the chain, from outermost to innermost

func LogFailureInline

func LogFailureInline(email, githubUsername, endpointURL string, attemptNumber, maxRetries int, statusCode int, responseBody, errorType, errorMessage, stacktrace string, totalDurationMs int64)

LogFailureInline is a convenience function for logging failures without a Logger instance.

func LogFailureInlineWithEntry

func LogFailureInlineWithEntry(entry LogEntry)

LogFailureInlineWithEntry is a convenience function for logging failures using the new LogEntry format.

func LogIngestError

func LogIngestError(logger *Logger, email, githubUsername, userID, sessionID, requestID, endpoint, method, path, endpointURL string, err error, statusCode int, responseBody string, attemptNumber, maxRetries, retryDelayMs int, totalDurationMs int64, eventType string, metadata RequestMetadata) error

LogIngestError is the main logging function that integrates all components and writes formatted log entries for ingest endpoint operations.

This function brings together error serialization (from cg-2iff2), context capture helpers (from cg-4zz54), and the Logger to create a complete ingest error logging solution.

Parameters:

  • logger: The Logger instance to write the log entry (can be nil, will use default)
  • email: User's email address being resolved (required)
  • githubUsername: Target GitHub username for resolution (required)
  • userID: User's unique identifier (optional, can be empty string)
  • sessionID: User's current session identifier (optional, can be empty string)
  • requestID: Current request identifier (optional, can be empty string)
  • endpoint: Endpoint identifier (e.g., "github-username-resolution", required)
  • method: HTTP method (GET, POST, etc., required)
  • path: Request path (required)
  • endpointURL: Full HTTP endpoint URL being called (required)
  • err: The error that occurred (can be nil for success cases)
  • statusCode: HTTP status code received (0 if not applicable)
  • responseBody: Response body content (empty if not available)
  • attemptNumber: Current retry attempt (1-based, required)
  • maxRetries: Maximum number of retry attempts configured (required)
  • retryDelayMs: Delay before next retry in milliseconds (0 for final failure)
  • totalDurationMs: Total time spent attempting in milliseconds
  • eventType: Type of event ("retry", "failure", or "success")
  • metadata: Optional metadata map for additional context (can be nil)

Returns:

  • error: Any error that occurred during logging (nil indicates success)

The function handles logging failures gracefully by returning the error to the caller while ensuring the log entry is properly formatted and written.

func LogIngestErrorExtended

func LogIngestErrorExtended(logger *Logger, err error, userCtx ExtendedUserContext, endpointCtx ExtendedEndpointContext, metadata RequestMetadata) error

LogIngestErrorExtended is the enhanced logging function that accepts structured context.

This function brings together error serialization (from cg-2iff2), context capture helpers (from cg-4zz54), and the Logger to create a complete ingest error logging solution with enhanced user and endpoint context.

Parameters:

  • logger: The Logger instance to write the log entry (can be nil, will use default)
  • err: The error that occurred (error interface, can be nil for success cases)
  • userCtx: Extended user context containing userID, sessionID, requestID, and optional email/username
  • endpointCtx: Extended endpoint context containing endpoint, method, path, URL, and response details
  • metadata: Optional metadata map for additional context (can be nil)

Returns:

  • error: Any error that occurred during logging (nil indicates success)

The function handles logging failures gracefully by returning the error to the caller while ensuring the log entry is properly formatted and written.

Integration Points (TODO):

  • cg-2iff2: Error serialization and type classification
  • cg-4zz54: User and endpoint context capture helpers
  • Future: Integration with monitoring and alerting systems
  • Future: Integration with distributed tracing systems

func LogRetryInline

func LogRetryInline(email, githubUsername, endpointURL string, attemptNumber, maxRetries int, statusCode int, responseBody, errorType, errorMessage, stacktrace string, retryDelayMs int, totalDurationMs int64)

LogRetryInline is a convenience function for logging retry attempts without a Logger instance.

func LogRetryInlineWithEntry

func LogRetryInlineWithEntry(entry LogEntry)

LogRetryInlineWithEntry is a convenience function for logging retry attempts using the new LogEntry format.

func ValidateMetadataKeys

func ValidateMetadataKeys(metadata RequestMetadata) error

ValidateMetadataKeys validates that metadata keys do not collide with reserved LogEntry fields. This prevents metadata from overwriting core LogEntry fields during JSON marshaling.

Parameters:

  • metadata: The metadata map to validate (can be nil or empty)

Returns:

  • error: An error if a reserved key is found, nil otherwise

The reserved keys correspond to the top-level JSON fields in LogEntry:

  • timestamp, event_type, user, endpoint, error, max_retries, retry_delay_ms, total_duration_ms, metadata

Example:

metadata := RequestMetadata{"batch_id": "123"} // valid
metadata := RequestMetadata{"user": "collision"} // returns error

Types

type AggregateStats

type AggregateStats struct {
	TotalProcessed int // Total records attempted
	TotalSkipped   int // Total records skipped (e.g., empty login, validation failures)
	TotalIngested  int // Total records successfully ingested
	TotalRetries   int // Total retry attempts
	TotalFailures  int // Total final failures (after all retries)
	StartTime      time.Time
	LastUpdateTime time.Time
}

AggregateStats tracks aggregate statistics for ingest operations.

func NewAggregateStats

func NewAggregateStats() *AggregateStats

NewAggregateStats creates a new AggregateStats instance.

type BatchProgress

type BatchProgress struct {
	BatchNum      int           // Current batch number (1-based)
	TotalBatches  int           // Total number of batches
	ProcessedRows int           // Total rows processed so far
	TotalRows     int           // Total rows to process
	BatchElapsed  time.Duration // Time taken for this batch
	TotalElapsed  time.Duration // Total elapsed time
}

BatchProgress represents progress information for batch processing.

type EndpointContext

type EndpointContext struct {
	Endpoint      string `json:"endpoint"`                // Endpoint identifier (e.g., "github-username-resolution")
	Method        string `json:"method"`                  // HTTP method (GET, POST, etc.)
	Path          string `json:"path"`                    // Request path
	URL           string `json:"url"`                     // Full HTTP endpoint URL being called
	AttemptNumber int    `json:"attempt_number"`          // Current retry attempt (1-based)
	StatusCode    int    `json:"status_code,omitempty"`   // HTTP status code received
	ResponseBody  string `json:"response_body,omitempty"` // Response body content (if available)
}

EndpointContext contains HTTP endpoint interaction details.

func CaptureEndpointContext

func CaptureEndpointContext(endpoint, method, path, url string, attemptNumber int, statusCode int, responseBody string) (EndpointContext, error)

CaptureEndpointContext creates an EndpointContext struct with validation. This helper function accepts endpoint interaction parameters and returns a populated EndpointContext struct for use in log entries.

Parameters:

  • endpoint: Endpoint identifier (e.g., "github-username-resolution", required)
  • method: HTTP method (GET, POST, etc., required)
  • path: Request path (required)
  • url: Full HTTP endpoint URL being called (required)
  • attemptNumber: Current retry attempt (1-based, required)
  • statusCode: HTTP status code received (0 if not applicable, defaults to 0)
  • responseBody: Response body content (empty if not available, defaults to empty string)

Returns:

  • A populated EndpointContext struct
  • An error if validation fails (empty endpoint, method, path, url or zero/negative attempt number)

type ErrorContext

type ErrorContext struct {
	Type       string `json:"type"`                  // Type of error (network, timeout, client_error, server_error, parse_error, unknown)
	Message    string `json:"message"`               // Human-readable error message
	StackTrace string `json:"stack_trace,omitempty"` // Stack trace captured at error time
}

ErrorContext contains error details for ingest operations.

func SerializeError

func SerializeError(err error) ErrorContext

SerializeError serializes an error into an ErrorContext structure. This function extracts error type, message, and stack trace from the provided error.

Parameters:

  • err: The error to serialize (can be nil)

Returns:

  • ErrorContext: Populated error context with type, message, and stack trace

The function handles nil errors gracefully by returning an empty ErrorContext. Error type is extracted using reflection to get the underlying type name. Stack trace is captured at the point of this call for debugging purposes.

func SerializeErrorWithCaller

func SerializeErrorWithCaller(err error, callerDepth int) ErrorContext

SerializeErrorWithCaller serializes an error into an ErrorContext structure, capturing the stack trace from a specific caller depth.

Parameters:

  • err: The error to serialize (can be nil)
  • callerDepth: The number of stack frames to skip (0 = this function, 1 = direct caller, etc.)

Returns:

  • ErrorContext: Populated error context with type, message, and stack trace

Use this function when you want to capture the stack trace from a specific call site, such as the function that originated the error.

func SerializeErrorWithOptions

func SerializeErrorWithOptions(err error, opts *SerializationOptions) ErrorContext

SerializeErrorWithOptions serializes an error into an ErrorContext structure with additional options for customization.

Parameters:

  • err: The error to serialize (can be nil)
  • opts: Serialization options (can be nil for defaults)

Returns:

  • ErrorContext: Populated error context based on provided options

type ErrorRecovery

type ErrorRecovery struct {
	Suggestion string // Actionable recovery suggestion
	Severity   string // "low", "medium", "high"
}

ErrorRecovery provides recovery suggestions for different error types.

func GetErrorRecovery

func GetErrorRecovery(errorType string, statusCode int) ErrorRecovery

GetErrorRecovery returns actionable recovery suggestions based on error type.

type Event

type Event struct {
	Timestamp       time.Time `json:"timestamp"`
	EventType       string    `json:"event_type"` // "retry" or "failure"
	Email           string    `json:"email"`
	GithubUsername  string    `json:"github_username"`
	EndpointURL     string    `json:"endpoint_url"`
	AttemptNumber   int       `json:"attempt_number"`
	MaxRetries      int       `json:"max_retries"`
	StatusCode      int       `json:"status_code,omitempty"`       // HTTP status code if available
	ResponseBody    string    `json:"response_body,omitempty"`     // Response body if available
	ErrorType       string    `json:"error_type,omitempty"`        // Type of error (network, timeout, client_error, server_error)
	ErrorMessage    string    `json:"error_message,omitempty"`     // Error message
	Stacktrace      string    `json:"stacktrace,omitempty"`        // Stack trace if available
	RetryDelayMs    int       `json:"retry_delay_ms,omitempty"`    // Delay before next retry in milliseconds
	TotalDurationMs int64     `json:"total_duration_ms,omitempty"` // Total time spent attempting
}

Event represents a single ingest log event.

func EventFromError

func EventFromError(email, githubUsername, endpointURL string, err error, statusCode int, responseBody string, attemptNumber, maxRetries, retryDelayMs int, totalDurationMs int64) Event

EventFromError creates an Event from error context, extracting error details. This function accepts all required context parameters and serializes error information (type, message, stack trace) into a structured Event for logging.

Parameters:

  • email: User's email address being resolved
  • githubUsername: Target GitHub username for resolution
  • endpointURL: Full HTTP endpoint URL being called
  • err: The error that occurred (can be nil)
  • statusCode: HTTP status code received (0 if not applicable)
  • responseBody: Response body content (empty if not available)
  • attemptNumber: Current retry attempt (1-based)
  • maxRetries: Maximum number of retry attempts
  • retryDelayMs: Delay before next retry in milliseconds (0 for final failure)
  • totalDurationMs: Total time spent attempting in milliseconds

Returns:

  • A populated Event struct with error context extracted from the error

func (*Event) ToLogEntry

func (e *Event) ToLogEntry() LogEntry

ToLogEntry converts a legacy Event to the new structured LogEntry format.

type ExtendedEndpointContext

type ExtendedEndpointContext struct {
	Endpoint     string `json:"endpoint"`                // Endpoint identifier (e.g., "github-username-resolution")
	Method       string `json:"method"`                  // HTTP method (GET, POST, etc.)
	Path         string `json:"path"`                    // Request path
	URL          string `json:"url"`                     // Full HTTP endpoint URL
	StatusCode   int    `json:"status_code"`             // HTTP status code received
	ResponseBody string `json:"response_body,omitempty"` // Response body content (if available)
}

ExtendedEndpointContext contains detailed HTTP endpoint interaction information.

type ExtendedUserContext

type ExtendedUserContext struct {
	UserID    string `json:"user_id"`    // User's unique identifier
	SessionID string `json:"session_id"` // Current session identifier
	RequestID string `json:"request_id"` // Current request identifier
	Email     string `json:"email"`      // User's email address (optional)
	Username  string `json:"username"`   // User's username (optional)
}

ExtendedUserContext contains enhanced user identification information.

type LogEntry

type LogEntry struct {
	Timestamp       time.Time       `json:"timestamp"`                   // UTC timestamp when the event occurred
	EventType       string          `json:"event_type"`                  // "retry", "failure", or "success"
	User            UserContext     `json:"user"`                        // User identification context
	Endpoint        EndpointContext `json:"endpoint"`                    // Endpoint interaction context
	Error           ErrorContext    `json:"error,omitempty"`             // Error details (if applicable)
	MaxRetries      int             `json:"max_retries"`                 // Maximum number of retry attempts configured
	RetryDelayMs    int             `json:"retry_delay_ms,omitempty"`    // Delay before next retry in milliseconds
	TotalDurationMs int64           `json:"total_duration_ms,omitempty"` // Total time spent attempting in milliseconds
	Metadata        RequestMetadata `json:"metadata,omitempty"`          // Optional metadata for additional context
}

LogEntry represents a structured ingest log entry with nested contexts.

func LogEntryFromError

func LogEntryFromError(email, githubUsername, endpointURL string, err error, statusCode int, responseBody string, attemptNumber, maxRetries, retryDelayMs int, totalDurationMs int64) LogEntry

LogEntryFromError creates a LogEntry from error context, extracting error details. This function accepts all required context parameters and serializes error information (type, message, stack trace) into a structured LogEntry for logging.

Parameters:

  • email: User's email address being resolved
  • githubUsername: Target GitHub username for resolution
  • endpointURL: Full HTTP endpoint URL being called
  • err: The error that occurred (can be nil)
  • statusCode: HTTP status code received (0 if not applicable)
  • responseBody: Response body content (empty if not available)
  • attemptNumber: Current retry attempt (1-based)
  • maxRetries: Maximum number of retry attempts
  • retryDelayMs: Delay before next retry in milliseconds (0 for final failure)
  • totalDurationMs: Total time spent attempting in milliseconds

Returns:

  • A populated LogEntry struct with error context extracted from the error

func (*LogEntry) ToEvent

func (le *LogEntry) ToEvent() Event

ToEvent converts a LogEntry to the legacy Event format for backward compatibility.

type Logger

type Logger struct {
	// contains filtered or unexported fields
}

Logger writes structured logs for ingest endpoint operations and errors.

func NewLogger

func NewLogger() *Logger

NewLogger creates a new ingest logger that writes to stderr (or a configured output).

func NewLoggerWithOutput

func NewLoggerWithOutput(output *log.Logger) *Logger

NewLoggerWithOutput creates a new ingest logger with a custom output writer.

func (*Logger) GetStats

func (l *Logger) GetStats() *AggregateStats

GetStats returns the current aggregate statistics.

func (*Logger) LogBatchProgress

func (l *Logger) LogBatchProgress(progress BatchProgress)

LogBatchProgress logs batch processing progress with rate and ETA calculations. This is suitable for monitoring large production runs (e.g., 349,425 records).

func (*Logger) LogErrorWithRecovery

func (l *Logger) LogErrorWithRecovery(entry *LogEntry) error

LogErrorWithRecovery logs an error with actionable recovery suggestions.

func (*Logger) LogFailure

func (l *Logger) LogFailure(event *Event) error

LogFailure logs a final failure after all retries exhausted. The EventType and (if unset) Timestamp fields are populated on the caller's Event via the pointer receiver.

func (*Logger) LogFailureWithEntry

func (l *Logger) LogFailureWithEntry(entry *LogEntry) error

LogFailureWithEntry logs a final failure using the new structured LogEntry format.

func (*Logger) LogRetry

func (l *Logger) LogRetry(event *Event) error

LogRetry logs a retry attempt with full context. The EventType and (if unset) Timestamp fields are populated on the caller's Event via the pointer receiver.

func (*Logger) LogRetryWithEntry

func (l *Logger) LogRetryWithEntry(entry *LogEntry) error

LogRetryWithEntry logs a retry attempt using the new structured LogEntry format.

func (*Logger) LogStats

func (l *Logger) LogStats(title string)

LogStats logs the current aggregate statistics in a formatted summary.

func (*Logger) LogStatsJSON

func (l *Logger) LogStatsJSON(title string) error

LogStatsJSON logs the current aggregate statistics as a structured, machine-readable JSON summary (as opposed to LogStats, which logs a human-readable summary). Percentage and rate calculations are guarded against division by zero.

func (*Logger) LogStatusReport

func (l *Logger) LogStatusReport(report StatusReport)

LogStatusReport logs a comprehensive status report suitable for periodic monitoring. This provides production-ready visibility into long-running operations.

func (*Logger) LogSuccess

func (l *Logger) LogSuccess(event *Event) error

LogSuccess logs a successful resolution (optional, for debugging). The EventType and (if unset) Timestamp fields are populated on the caller's Event via the pointer receiver.

func (*Logger) LogSuccessWithEntry

func (l *Logger) LogSuccessWithEntry(entry *LogEntry) error

LogSuccessWithEntry logs a successful resolution using the new structured LogEntry format.

func (*Logger) RecordFailure

func (l *Logger) RecordFailure(entry *LogEntry) error

RecordFailure records a final failure after all retries exhausted. This counts as a record "attempted" (TotalProcessed), in addition to bumping TotalFailures.

func (*Logger) RecordProcessed

func (l *Logger) RecordProcessed()

RecordProcessed records a record as it enters the ingest flow. This increments the TotalProcessed counter and updates the LastUpdateTime.

func (*Logger) RecordRetry

func (l *Logger) RecordRetry(entry *LogEntry) error

RecordRetry records a retry attempt. This counts as a record "attempted" (TotalProcessed), in addition to bumping TotalRetries.

func (*Logger) RecordSkipped

func (l *Logger) RecordSkipped(reason string)

RecordSkipped records a skipped record (e.g., empty login, validation failure).

type PeriodicReporter

type PeriodicReporter struct {
	// contains filtered or unexported fields
}

PeriodicReporter provides automatic periodic status reporting.

func NewPeriodicReporter

func NewPeriodicReporter(logger *Logger, report *StatusReport, interval time.Duration) *PeriodicReporter

NewPeriodicReporter creates a new periodic reporter. The report is held by pointer so that AddError (and the periodic reports themselves) mutate the same StatusReport the caller holds.

func (*PeriodicReporter) AddError

func (pr *PeriodicReporter) AddError(errorMsg string)

AddError records an error for inclusion in the next status report.

func (*PeriodicReporter) Start

func (pr *PeriodicReporter) Start()

Start begins periodic status reporting in a background goroutine.

func (*PeriodicReporter) Stop

func (pr *PeriodicReporter) Stop()

Stop stops the periodic reporter, waits for the background goroutine to fully exit, and logs a final status report. After Stop returns, it is safe for the caller to read/mutate the StatusReport passed to NewPeriodicReporter without further synchronization.

type RequestMetadata

type RequestMetadata map[string]interface{}

RequestMetadata contains optional metadata for ingest requests.

type SerializationOptions

type SerializationOptions struct {
	IncludeStackTrace bool // Whether to capture stack trace
	CallerDepth       int  // Stack frame skip depth for stack trace capture
}

SerializationOptions configures error serialization behavior.

type StatsJSON

type StatsJSON struct {
	Timestamp   string               `json:"timestamp"`
	Title       string               `json:"title"`
	Records     StatsJSONRecords     `json:"records"`
	Percentages StatsJSONPercentages `json:"percentages"`
	Performance StatsJSONPerformance `json:"performance"`
	Retries     StatsJSONRetries     `json:"retries"`
}

StatsJSON is the JSON-serializable representation of aggregate ingest statistics, as produced by LogStatsJSON.

type StatsJSONPercentages

type StatsJSONPercentages struct {
	SkippedPercent  float64 `json:"skipped_percent"`
	IngestedPercent float64 `json:"ingested_percent"`
}

StatsJSONPercentages holds the derived percentages in a JSON stats summary.

type StatsJSONPerformance

type StatsJSONPerformance struct {
	ElapsedSeconds float64 `json:"elapsed_seconds"`
	RatePerSec     float64 `json:"rate_per_sec"`
}

StatsJSONPerformance holds timing/throughput information in a JSON stats summary.

type StatsJSONRecords

type StatsJSONRecords struct {
	Processed int `json:"processed"`
	Skipped   int `json:"skipped"`
	Ingested  int `json:"ingested"`
}

StatsJSONRecords holds the raw record counts in a JSON stats summary.

type StatsJSONRetries

type StatsJSONRetries struct {
	TotalAttempts int `json:"total_attempts"`
	FinalFailures int `json:"final_failures"`
}

StatsJSONRetries holds retry/failure counters in a JSON stats summary.

type StatusReport

type StatusReport struct {
	Title           string    // Report title
	Logger          *Logger   // Logger instance with stats
	ProcessedRows   int       // Total rows processed
	TotalRows       int       // Total rows to process
	CurrentBatch    int       // Current batch number
	TotalBatches    int       // Total number of batches
	LastError       string    // Last error message
	LastErrorTime   time.Time // Time of last error
	ErrorCount      int       // Number of errors since last report
	IncludeProgress bool      // Whether to include progress percentage
}

StatusReport contains comprehensive status information for periodic reporting.

type UserContext

type UserContext struct {
	UserID         string `json:"user_id"`         // User's unique identifier
	SessionID      string `json:"session_id"`      // User's current session identifier
	RequestID      string `json:"request_id"`      // Current request identifier
	Email          string `json:"email"`           // User's email address being resolved
	GithubUsername string `json:"github_username"` // Target GitHub username for resolution
}

UserContext contains user identification information.

func CaptureUserContext

func CaptureUserContext(email, githubUsername string) (UserContext, error)

CaptureUserContext creates a UserContext struct with validation. This helper function accepts user identification parameters and returns a populated UserContext struct for use in log entries.

Parameters:

  • email: User's email address being resolved (required)
  • githubUsername: Target GitHub username for resolution (required)

Returns:

  • A populated UserContext struct
  • An error if validation fails (empty email or githubUsername)

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL