backfill

package
v0.24.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Jul 14, 2026 License: AGPL-3.0 Imports: 22 Imported by: 0

Documentation

Overview

Package backfill extracts token usage from Claude Code transcripts and backfills historical nodes in the tapes database.

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func ScanTranscriptDir

func ScanTranscriptDir(dir string) ([]string, error)

ScanTranscriptDir finds all JSONL files under the given directory.

Types

type APIRunRequest added in v0.5.0

type APIRunRequest struct {
	TranscriptDir string `json:"transcript_dir"`
	DryRun        bool   `json:"dry_run,omitempty"`
	Verbose       bool   `json:"verbose,omitempty"`
	Sessions      bool   `json:"sessions,omitempty"`
	OrgID         string `json:"org_id,omitempty"`
	AuthSubject   string `json:"auth_subject,omitempty"`
}

APIRunRequest is the request payload for the API-backed usage sync flow.

type Backfiller

type Backfiller struct {
	// contains filtered or unexported fields
}

Backfiller matches Claude Code transcript usage data to tapes DB nodes.

func NewBackfillerWithDriver added in v0.5.0

func NewBackfillerWithDriver(driver storage.Driver, opts Options) *Backfiller

NewBackfillerWithDriver creates a Backfiller using an existing storage driver.

func (*Backfiller) Run

func (b *Backfiller) Run(ctx context.Context, transcriptDir string) (*Result, error)

Run scans transcripts and backfills usage data into the database.

type Options

type Options struct {
	DryRun      bool
	Verbose     bool
	Sessions    bool
	OrgID       string
	AuthSubject string
}

Options configures backfill behavior.

type Result

type Result struct {
	Matched               int `json:"matched"`
	Skipped               int `json:"skipped"`
	Unmatched             int `json:"unmatched"`
	TotalTokensBackfilled int `json:"total_tokens_backfilled"`
	TranscriptFiles       int `json:"transcript_files"`
	TranscriptEntries     int `json:"transcript_entries"`
	SessionsBackfilled    int `json:"sessions_backfilled,omitempty"`
	NodesLinked           int `json:"nodes_linked,omitempty"`
}

Result contains statistics from a backfill run.

func RunViaAPI added in v0.5.0

func RunViaAPI(ctx context.Context, apiTarget string, transcriptDir string, opts Options) (*Result, error)

RunViaAPI asks a tapes API server to perform the usage sync.

func (*Result) Summary

func (r *Result) Summary() string

Summary returns a human-readable summary of the sync result.

type TranscriptBlock

type TranscriptBlock struct {
	Type string `json:"type"`
	Text string `json:"text"`
}

TranscriptBlock represents a content block in a transcript message.

type TranscriptEntry

type TranscriptEntry struct {
	Type       string             `json:"type"`
	UUID       string             `json:"uuid"`
	ParentUUID *string            `json:"parentUuid"`
	Timestamp  string             `json:"timestamp"`
	SessionID  string             `json:"sessionId"`
	Cwd        string             `json:"cwd"`
	Version    string             `json:"version"`
	Message    *TranscriptMessage `json:"message"`
	SourcePath string             `json:"-"`
}

TranscriptEntry represents a single line in a Claude Code JSONL transcript.

func ParseTranscript

func ParseTranscript(path string) ([]TranscriptEntry, error)

ParseTranscript reads a JSONL file and returns assistant entries with usage data. It deduplicates by message ID, keeping the last (most complete) entry per message.

func (*TranscriptEntry) TextContent

func (e *TranscriptEntry) TextContent() string

TextContent extracts the concatenated text from all text content blocks.

type TranscriptMessage

type TranscriptMessage struct {
	ID         string            `json:"id"`
	Role       string            `json:"role"`
	Model      string            `json:"model"`
	Content    []TranscriptBlock `json:"content"`
	Usage      *TranscriptUsage  `json:"usage"`
	StopReason json.RawMessage   `json:"stop_reason"`
}

TranscriptMessage represents the message field within a JSONL entry.

type TranscriptUploadOptions added in v0.16.0

type TranscriptUploadOptions struct {
	// ProjectDir is one Claude Code project directory
	// (~/.claude/projects/<flattened-cwd>) holding <session>.jsonl
	// files and per-session subagents/ directories.
	ProjectDir string

	// SessionIDs filters which sessions upload; empty means every
	// .jsonl in the directory.
	SessionIDs []string

	// IngestURL is the tapes-ingest base URL.
	IngestURL string

	// HarnessID tags the session envelope; Claude Code transcripts are
	// "claude".
	HarnessID string

	Verbose bool
	Logf    func(format string, args ...any)
}

TranscriptUploadOptions configures a harness-transcript upload run.

type TranscriptUploadResult added in v0.16.0

type TranscriptUploadResult struct {
	Sessions int      `json:"sessions"`
	Files    int      `json:"files"`
	Uploaded int      `json:"uploaded"`
	Deduped  int      `json:"deduped"`
	Failed   int      `json:"failed"`
	Failures []string `json:"failures,omitempty"`
}

TranscriptUploadResult summarizes an upload run.

func UploadTranscripts added in v0.16.0

func UploadTranscripts(ctx context.Context, opts TranscriptUploadOptions) (*TranscriptUploadResult, error)

UploadTranscripts pushes harness transcripts (main + subagents) into the tapes raw layer via POST /v1/ingest/transcript. Idempotent: the server dedups by content version, so re-running uploads nothing for unchanged files and a new version for grown ones.

type TranscriptUsage

type TranscriptUsage struct {
	InputTokens              int `json:"input_tokens"`
	OutputTokens             int `json:"output_tokens"`
	CacheCreationInputTokens int `json:"cache_creation_input_tokens"`
	CacheReadInputTokens     int `json:"cache_read_input_tokens"`
}

TranscriptUsage contains token counts from a Claude Code transcript entry.

type WireTraceOptions added in v0.16.0

type WireTraceOptions struct {
	// CapturesDir is the paperd wire-trace root holding turn-* bundles
	// (request.json + response.sse + meta.json per turn).
	CapturesDir string

	// IngestURL is the base URL of a tapes-ingest server, e.g.
	// "http://127.0.0.1:8090". The backfill POSTs one envelope per
	// captured turn to {IngestURL}/v1/ingest.
	IngestURL string

	// SessionIDs filters replay to bundles whose captured
	// X-Tapes-Harness-Session-Id matches; empty replays everything.
	SessionIDs []string

	// DryRun parses and reduces every bundle but skips the POST.
	DryRun bool

	// Verbose logs each turn's outcome to Logf.
	Verbose bool

	// Logf receives progress output; defaults to a no-op.
	Logf func(format string, args ...any)
}

WireTraceOptions configures a wire-trace → ingest backfill run.

type WireTraceResult added in v0.16.0

type WireTraceResult struct {
	Scanned  int      `json:"scanned"`
	Posted   int      `json:"posted"`
	RawOnly  int      `json:"raw_only"`
	Skipped  int      `json:"skipped"`
	Failed   int      `json:"failed"`
	Failures []string `json:"failures,omitempty"`
}

WireTraceResult summarizes a wire-trace backfill run.

func WireTrace added in v0.16.0

func WireTrace(ctx context.Context, opts WireTraceOptions) (*WireTraceResult, error)

WireTrace replays paperd wire-trace capture bundles through a tapes-ingest server, reconstructing the envelope tapes-extproc would have dispatched live: verbatim request bytes, response reduced with the same shared capture reducer, meta rebuilt from the bundle, and the session block recovered from the captured X-Tapes-* headers.

The whole flow is idempotent end to end: the raw layer dedupes on (org, request_id) — the turn directory name, stable across re-runs — and node inserts are content-addressed ON CONFLICT DO NOTHING, so turns that already landed via live capture reproduce their existing nodes byte for byte and only gain the previously-missing raw row.

Only provider chat-completion calls are replayed (…/v1/messages); auxiliary traffic in the trace (count_tokens, tapes API reads) is skipped.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL