locomo

package
v1.2.2 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Apr 13, 2026 License: MIT Imports: 9 Imported by: 0

Documentation

Overview

Package locomo implements the LOCOMO benchmark runner for Cortex.

LOCOMO evaluates memory systems on 1,986 questions across 10 long-term conversations with 5 category types (1-5).

Dataset: https://github.com/snap-research/locomo

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func Run

func Run(cfg Config) (*common.BenchmarkResult, error)

Run executes the LOCOMO benchmark against Cortex.

Types

type Config

type Config struct {
	DataPath     string
	Limit        int // Max questions (0 = all)
	JudgeCfg     *common.JudgeConfig
	GraphBoost   bool
	EmbeddingCfg *embedding.Config // If set, enables vector search
}

Config controls the benchmark run.

type Conversation

type Conversation struct {
	SampleID     string           `json:"sample_id"`
	Conversation ConversationData `json:"conversation"`
	QA           []QA             `json:"qa"`
	Observation  json.RawMessage  `json:"observation"` // Complex nested structure, parsed manually
}

Conversation represents a LOCOMO conversation sample.

type ConversationData

type ConversationData struct {
	SpeakerA string            `json:"speaker_a"`
	SpeakerB string            `json:"speaker_b"`
	Sessions map[string][]Turn // Parsed from session_N keys
	Dates    map[string]string // Parsed from session_N_date_time keys
}

ConversationData holds the dialogue sessions and metadata.

func (*ConversationData) UnmarshalJSON

func (cd *ConversationData) UnmarshalJSON(data []byte) error

UnmarshalJSON custom-parses the conversation dict with dynamic session keys.

type QA

type QA struct {
	Question string          `json:"question"`
	Answer   json.RawMessage `json:"answer"` // Can be string or number
	Evidence []string        `json:"evidence"`
	Category int             `json:"category"`
}

QA represents a question-answer pair.

func (QA) AnswerString

func (qa QA) AnswerString() string

AnswerString returns the answer as a string regardless of JSON type.

type Turn

type Turn struct {
	Speaker string `json:"speaker"`
	DiaID   string `json:"dia_id"`
	Text    string `json:"text"`
}

Turn represents a single dialogue turn.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL