longmemeval

package
v1.1.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Apr 12, 2026 License: MIT Imports: 7 Imported by: 0

Documentation

Overview

Package longmemeval implements the LongMemEval benchmark runner for Cortex.

LongMemEval tests 500 questions across 5 memory abilities: Information Extraction, Multi-Session Reasoning, Temporal Reasoning, Knowledge Updates, and Abstention.

Reference: arXiv:2410.10813

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func Run

func Run(cfg Config) (*common.BenchmarkResult, error)

Run executes the LongMemEval benchmark against Cortex.

Types

type ChatTurn

type ChatTurn struct {
	Role      string `json:"role"`
	Content   string `json:"content"`
	SessionID int    `json:"session_id"`
	Timestamp string `json:"timestamp"`
}

ChatTurn represents a single turn in the chat history.

type Config

type Config struct {
	DataPath   string
	Limit      int
	JudgeCfg   *common.JudgeConfig
	GraphBoost bool
}

Config controls the benchmark run.

type Dataset

type Dataset struct {
	Questions []Question `json:"questions"`
}

Dataset represents the LongMemEval dataset structure.

type Question

type Question struct {
	ID          string     `json:"id"`
	Question    string     `json:"question"`
	Answer      string     `json:"answer"`
	Category    string     `json:"category"` // IE, MR, TR, KU, ABS
	ChatHistory []ChatTurn `json:"chat_history"`
}

Question represents a single evaluation question.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL