Documentation
¶
Overview ¶
Package dmr implements the Deep Memory Retrieval benchmark runner for Cortex.
DMR uses the MSC-Self-Instruct dataset (500 conversations). Each conversation has multiple dialog sessions and self-instruct Q&A pairs that test retrieval from earlier sessions.
Reference: arXiv:2310.08560 (MemGPT)
Index ¶
Constants ¶
const ( // DMRBaselineAdapterSchemaVersion identifies the adapter metadata contract. DMRBaselineAdapterSchemaVersion = "1.0" // ReportKindExternalBenchmarkAdapter distinguishes adapted external benchmark // evidence from a versioned Cortex reproduction. ReportKindExternalBenchmarkAdapter = "external-benchmark-adapter" )
Variables ¶
This section is empty.
Functions ¶
Types ¶
type BaselineEvidence ¶
type BaselineEvidence struct {
SchemaVersion string `json:"schema_version"`
Benchmark string `json:"benchmark"`
Dataset DatasetAttribution `json:"dataset"`
QueryID string `json:"query_id"`
QueryClass string `json:"query_class"`
QuerySourceField string `json:"query_source_field"`
EvaluationMode EvaluationMode `json:"evaluation_mode"`
ScoreSemantics string `json:"score_semantics"`
LegacyResult common.QuestionResult `json:"legacy_result"`
ReportKind string `json:"report_kind"`
CortexReproduction bool `json:"cortex_reproduction"`
RetrievalLabelsComplete bool `json:"retrieval_labels_complete"`
ReleaseRetrievalEvidenceEligible bool `json:"release_retrieval_evidence_eligible"`
Limitations []string `json:"limitations"`
}
BaselineEvidence adapts one legacy DMR result into explicit evidence metadata without changing its score or inventing retrieval relevance labels.
func AdaptBaselineEvidence ¶
func AdaptBaselineEvidence(conversationIndex int, result common.QuestionResult, evaluation EvaluationMode) (BaselineEvidence, error)
AdaptBaselineEvidence maps a legacy conversation result to deterministic query metadata. It deliberately leaves retrieval-label fields incomplete: MSC-Self-Instruct supplies answers, not stable Cortex relevance IDs/spans.
type Config ¶
type Config struct {
DataPath string
Limit int
JudgeCfg *common.JudgeConfig
GraphBoost bool
}
Config controls the benchmark run.
type DatasetAttribution ¶
type DatasetAttribution struct {
Name string `json:"name"`
Source string `json:"source"`
License string `json:"license"`
Attribution string `json:"attribution"`
}
DatasetAttribution preserves the upstream dataset source and licence.
type DialogTurn ¶
DialogTurn represents a single turn.
type EvaluationMode ¶
type EvaluationMode string
EvaluationMode records the legacy DMR evaluator used for a result. Neither mode is equivalent to labelled retrieval relevance.
const ( EvaluationCombinedF1Rouge EvaluationMode = "combined-f1-rouge" EvaluationCombinedF1RougeWithJudge EvaluationMode = "combined-f1-rouge-with-judge" )
type MSCConversation ¶
type MSCConversation struct {
Personas [][]string `json:"personas"`
Dialog []DialogTurn `json:"dialog"`
PreviousDialogs []PreviousDialog `json:"previous_dialogs"`
SelfInstruct map[string]string `json:"self_instruct"` // {"B": question, "A": answer}
Summary1 json.RawMessage `json:"summary_speaker_1"`
Summary2 json.RawMessage `json:"summary_speaker_2"`
}
MSCConversation represents a single MSC-Self-Instruct conversation.
type PreviousDialog ¶
type PreviousDialog struct {
Personas [][]string `json:"personas"`
Dialog []DialogTurn `json:"dialog"`
}
PreviousDialog represents a previous conversation session.
Source Files