Documentation
¶
Overview ¶
Package longmemeval implements the LongMemEval benchmark runner for Cortex.
LongMemEval tests 500 questions across 5 memory abilities: Information Extraction, Multi-Session Reasoning, Temporal Reasoning, Knowledge Updates, and Abstention.
Reference: arXiv:2410.10813
Index ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
Types ¶
type AbilityCategory ¶
AbilityCategory retains a LongMemEval ability code and gives it a stable, human-readable name for evidence reports.
type BaselineEvidence ¶
type BaselineEvidence struct {
DatasetSourceID string
QuerySourceID string
SourceURL string
LicenseNotice string
Ability AbilityCategory
TemporalMetadata []TemporalLocator
IsAbstention bool
EvidenceOrigin EvidenceOrigin
CortexReproduction bool
RetrievalLabels RetrievalLabelStatus
LegacyMetrics LegacyMetricComparability
Limitations []string
}
BaselineEvidence is a reporting-only view of one LongMemEval question. It does not execute retrieval or alter the legacy runner's F1/judge behavior.
func AdaptBaselineEvidence ¶
func AdaptBaselineEvidence(provenance DatasetProvenance, question Question) (BaselineEvidence, error)
AdaptBaselineEvidence converts upstream metadata into an evidence report. It deliberately leaves retrieval IDs and spans absent because LongMemEval's answer-oriented labels do not establish Cortex stable-ID relevance.
type ChatTurn ¶
type ChatTurn struct {
Role string `json:"role"`
Content string `json:"content"`
SessionID int `json:"session_id"`
Timestamp string `json:"timestamp"`
}
ChatTurn represents a single turn in the chat history.
type Config ¶
type Config struct {
DataPath string
Limit int
JudgeCfg *common.JudgeConfig
GraphBoost bool
}
Config controls the benchmark run.
type Dataset ¶
type Dataset struct {
Questions []Question `json:"questions"`
}
Dataset represents the LongMemEval dataset structure.
type DatasetProvenance ¶
DatasetProvenance identifies the exact upstream dataset distribution and retains its source and licensing notice without interpreting either.
type EvidenceOrigin ¶
type EvidenceOrigin string
EvidenceOrigin identifies whether evidence was produced by Cortex or comes from an external benchmark source.
const ( // EvidenceOriginExternal marks evidence that Cortex has not reproduced. EvidenceOriginExternal EvidenceOrigin = "external" )
type LegacyMetricComparability ¶
type LegacyMetricComparability struct {
AnswerTokenF1ComparableToLabelledRetrieval bool
JudgeComparableToLabelledRetrieval bool
}
LegacyMetricComparability prevents answer scoring from being interpreted as labelled retrieval relevance.
type Question ¶
type Question struct {
ID string `json:"id"`
Question string `json:"question"`
Answer string `json:"answer"`
Category string `json:"category"` // IE, MR, TR, KU, ABS
ChatHistory []ChatTurn `json:"chat_history"`
}
Question represents a single evaluation question.
type RetrievalLabelStatus ¶
type RetrievalLabelStatus struct {
Available bool
RelevantStableIDs []string
EvidenceSpans []string
ReleaseEvidenceEligible bool
}
RetrievalLabelStatus describes whether the upstream sample provides the stable relevance labels required for Cortex retrieval release evidence.
type TemporalLocator ¶
TemporalLocator retains temporal metadata present on an upstream chat turn. SessionID and Timestamp are locators, not Cortex memory IDs or relevance labels.
Source Files