scenarios

package
v1.0.0-alpha.21 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Mar 9, 2026 License: MIT Imports: 20 Imported by: 0

Documentation

Overview

Package scenarios provides E2E test scenarios for SemStreams

Package scenarios provides E2E test scenarios for SemStreams

Package scenarios provides E2E test scenarios for SemStreams

Package scenarios provides E2E test scenarios for SemStreams semantic processing

Package scenarios provides common result types for E2E tests

Package scenarios provides tier capability result types for E2E tests

Package scenarios defines E2E test scenarios for SemStreams

Package scenarios provides E2E test scenarios for SemStreams semantic processing

Package scenarios provides E2E test scenarios for SemStreams semantic processing

Package scenarios provides E2E test scenarios for SemStreams semantic processing

Package scenarios provides E2E test scenarios for SemStreams semantic processing

Package scenarios provides E2E test scenarios for SemStreams semantic processing

Package scenarios provides E2E test scenarios for SemStreams semantic processing

Package scenarios provides E2E test scenarios for SemStreams semantic processing

Package scenarios provides E2E test scenarios for SemStreams semantic processing

Index

Constants

View Source
const (
	// StructuralMinEntities is the minimum entities expected for structural tier
	// All tiers now load testdata/semantic/*.jsonl with 74 unique entities
	StructuralMinEntities = 50

	// StatisticalMinEntities is the minimum entities expected for statistical tier
	// Statistical tier loads testdata/semantic/*.jsonl with 74 unique entities
	StatisticalMinEntities = 50

	// SemanticMinEntities is the minimum entities expected for semantic tier
	// Same testdata as statistical tier
	SemanticMinEntities = 50
)

Variant-specific entity count expectations

Variables

This section is empty.

Functions

func SaveMetricsDump

func SaveMetricsDump(metricsData string, variant string, outputDir string) (string, error)

SaveMetricsDump saves the raw Prometheus metrics to a text file. The filename follows the pattern: {variant}-{timestamp}-metrics.txt

func SaveStructuredResults

func SaveStructuredResults(tr *TieredResults, outputDir string) (string, error)

SaveStructuredResults writes the structured results to a JSON file. The filename format is: {variant}-{timestamp}.json

Types

type AnomalyDetail

type AnomalyDetail struct {
	ID         string  `json:"id"`
	Type       string  `json:"type"`
	EntityA    string  `json:"entity_a"`
	EntityB    string  `json:"entity_b,omitempty"`
	Confidence float64 `json:"confidence"`
	Status     string  `json:"status"`
}

AnomalyDetail contains the details of a single anomaly for auditability.

type AnomalyGroundTruthResults

type AnomalyGroundTruthResults struct {
	ExpectedTotal      int              `json:"expected_total"`
	ExpectedFound      int              `json:"expected_found"`
	FalsePositiveTotal int              `json:"false_positive_total"`
	DetectedTotal      int              `json:"detected_total"`
	FalsePositiveRate  float64          `json:"false_positive_rate"`
	Passed             bool             `json:"passed"`
	Violations         []map[string]any `json:"violations,omitempty"`
}

AnomalyGroundTruthResults contains the results of ground truth validation.

type AnomalyResults

type AnomalyResults struct {
	// Total is the total number of anomalies detected
	Total int `json:"total"`

	// SemanticGap is count of semantic-structural gap anomalies (entities semantically
	// similar but structurally distant - detected via pivot distance)
	SemanticGap int `json:"semantic_gap"`

	// CoreIsolation is count of hub isolation anomalies (high k-core entities with
	// few connections to other high-core entities)
	CoreIsolation int `json:"core_isolation"`

	// CoreDemotion is count of core demotion anomalies (entities that dropped
	// significantly in k-core value between analysis runs)
	CoreDemotion int `json:"core_demotion"`

	// Transitivity is count of transitivity gap anomalies (missing expected
	// transitive relationships)
	Transitivity int `json:"transitivity"`

	// ByStatus contains counts by review status
	ByStatus AnomalyStatusCounts `json:"by_status"`

	// VirtualEdges contains counts of auto-applied semantic edges
	VirtualEdges *VirtualEdgeResults `json:"virtual_edges,omitempty"`

	// List contains the actual anomaly details for auditability
	List []AnomalyDetail `json:"list,omitempty"`

	// GroundTruth contains the results of ground truth validation
	GroundTruth *AnomalyGroundTruthResults `json:"ground_truth,omitempty"`
}

AnomalyResults contains structural anomaly detection results.

type AnomalyStatusCounts

type AnomalyStatusCounts struct {
	Pending   int `json:"pending"`
	Confirmed int `json:"confirmed"`
	Dismissed int `json:"dismissed"`
}

AnomalyStatusCounts contains anomaly counts by review status.

type CommunityComparison

type CommunityComparison struct {
	CommunityID        string   `json:"community_id"`
	Level              int      `json:"level"`
	MemberCount        int      `json:"member_count"`
	StatisticalSummary string   `json:"statistical_summary"`
	LLMSummary         string   `json:"llm_summary,omitempty"`
	SummaryStatus      string   `json:"summary_status"`
	Keywords           []string `json:"keywords"`
	SummaryLengthRatio float64  `json:"summary_length_ratio,omitempty"`
	WordOverlap        float64  `json:"word_overlap,omitempty"`
}

CommunityComparison represents a comparison of statistical vs LLM summaries for a community

type CommunityResults

type CommunityResults struct {
	// TotalCommunities is the number of communities detected
	TotalCommunities int `json:"total_communities"`

	// NonSingletonCount is communities with more than one member
	NonSingletonCount int `json:"non_singleton_count"`

	// LargestSize is the size of the largest community
	LargestSize int `json:"largest_size"`

	// AverageSize is the average community size
	AverageSize float64 `json:"average_size"`

	// WithKeywords is count of communities that have keywords
	WithKeywords int `json:"with_keywords"`

	// LLMEnhanced is count of communities with LLM summaries (semantic only)
	LLMEnhanced int `json:"llm_enhanced,omitempty"`
}

CommunityResults contains community detection results.

type CommunitySummaryReport

type CommunitySummaryReport struct {
	Variant               string                `json:"variant"`
	Timestamp             time.Time             `json:"timestamp"`
	CommunitiesTotal      int                   `json:"communities_total"`
	LLMEnhancedCount      int                   `json:"llm_enhanced_count"`
	StatisticalOnlyCount  int                   `json:"statistical_only_count"`
	LLMFailedCount        int                   `json:"llm_failed_count,omitempty"`
	LLMPendingCount       int                   `json:"llm_pending_count,omitempty"`
	LLMWaitDurationMs     int64                 `json:"llm_wait_duration_ms,omitempty"`
	AvgSummaryLengthRatio float64               `json:"avg_summary_length_ratio"`
	AvgWordOverlap        float64               `json:"avg_word_overlap"`
	NonSingletonCount     int                   `json:"non_singleton_count"`
	LargestCommunitySize  int                   `json:"largest_community_size"`
	AvgNonSingletonSize   float64               `json:"avg_non_singleton_size"`
	Communities           []CommunityComparison `json:"communities"`
}

CommunitySummaryReport contains aggregated community summary metrics

type ComponentResults

type ComponentResults struct {
	// ExpectedCount is the number of expected components
	ExpectedCount int `json:"expected_count"`

	// FoundCount is the number of components found healthy
	FoundCount int `json:"found_count"`

	// Components lists individual component status
	Components map[string]bool `json:"components,omitempty"`
}

ComponentResults contains component health results.

type ContextIndexResults

type ContextIndexResults struct {
	// TotalKeys is the number of unique context values in the index
	TotalKeys int `json:"total_keys"`

	// HierarchyContextFound indicates if "inference.hierarchy" key exists
	HierarchyContextFound bool `json:"hierarchy_context_found"`

	// HierarchyEntryCount is the number of entity+predicate pairs for hierarchy inference
	HierarchyEntryCount int `json:"hierarchy_entry_count,omitempty"`

	// SampleContexts contains sample context keys for debugging
	SampleContexts []string `json:"sample_contexts,omitempty"`
}

ContextIndexResults contains ContextIndex verification results. Phase 5: Added to verify triple provenance tracking via context values.

type CoreDataflowConfig

type CoreDataflowConfig struct {
	// Test data configuration
	MessageCount    int           `json:"message_count"`
	MessageInterval time.Duration `json:"message_interval"`

	// Validation configuration
	ValidationDelay time.Duration `json:"validation_delay"`
	MinProcessed    int           `json:"min_processed"`

	// WebSocket configuration
	WebSocketTimeout time.Duration `json:"websocket_timeout"`
	TestFlowID       string        `json:"test_flow_id"`
}

CoreDataflowConfig contains configuration for dataflow test

func DefaultCoreDataflowConfig

func DefaultCoreDataflowConfig() *CoreDataflowConfig

DefaultCoreDataflowConfig returns default configuration

type CoreDataflowScenario

type CoreDataflowScenario struct {
	// contains filtered or unexported fields
}

CoreDataflowScenario validates complete core data pipeline

func NewCoreDataflowScenario

func NewCoreDataflowScenario(
	obsClient *client.ObservabilityClient,
	wsClient *client.WebSocketClient,
	udpAddr string,
	config *CoreDataflowConfig,
) *CoreDataflowScenario

NewCoreDataflowScenario creates a new core dataflow test scenario

func (*CoreDataflowScenario) Description

func (s *CoreDataflowScenario) Description() string

Description returns the scenario description

func (*CoreDataflowScenario) Execute

func (s *CoreDataflowScenario) Execute(ctx context.Context) (*Result, error)

Execute runs the dataflow test scenario

func (*CoreDataflowScenario) Name

func (s *CoreDataflowScenario) Name() string

Name returns the scenario name

func (*CoreDataflowScenario) Setup

Setup prepares the scenario

func (*CoreDataflowScenario) Teardown

func (s *CoreDataflowScenario) Teardown(_ context.Context) error

Teardown cleans up after the scenario

type CoreFederationConfig

type CoreFederationConfig struct {
	// Test data configuration
	MessageCount    int           `json:"message_count"`
	MessageInterval time.Duration `json:"message_interval"`

	// Validation configuration
	ValidationDelay    time.Duration `json:"validation_delay"`
	MinMessagesOnCloud int           `json:"min_messages_on_cloud"`
	AckVerification    bool          `json:"ack_verification"`

	// Container configuration for file output verification
	CloudContainerName string `json:"cloud_container_name"`
	CloudOutputPattern string `json:"cloud_output_pattern"`
}

CoreFederationConfig contains configuration for federation test

func DefaultCoreFederationConfig

func DefaultCoreFederationConfig() *CoreFederationConfig

DefaultCoreFederationConfig returns default configuration

type CoreFederationScenario

type CoreFederationScenario struct {
	// contains filtered or unexported fields
}

CoreFederationScenario validates federation data flow with ack/nack protocol

func NewCoreFederationScenario

func NewCoreFederationScenario(
	edgeClient *client.ObservabilityClient,
	cloudClient *client.ObservabilityClient,
	udpAddr string,
	wsURL string,
	config *CoreFederationConfig,
) *CoreFederationScenario

NewCoreFederationScenario creates a new federation test scenario

func (*CoreFederationScenario) Description

func (s *CoreFederationScenario) Description() string

Description returns the scenario description

func (*CoreFederationScenario) Execute

func (s *CoreFederationScenario) Execute(ctx context.Context) (*Result, error)

Execute runs the federation test scenario

func (*CoreFederationScenario) Name

func (s *CoreFederationScenario) Name() string

Name returns the scenario name

func (*CoreFederationScenario) Setup

Setup prepares the scenario

func (*CoreFederationScenario) Teardown

Teardown cleans up after the scenario

type CoreHealthConfig

type CoreHealthConfig struct {
	// Validation thresholds
	RequireAllHealthy    bool `json:"require_all_healthy"`
	MinHealthyComponents int  `json:"min_healthy_components"`

	// Required core components
	RequiredComponents []string `json:"required_components"`
}

CoreHealthConfig contains configuration for core health check

func DefaultCoreHealthConfig

func DefaultCoreHealthConfig() *CoreHealthConfig

DefaultCoreHealthConfig returns default configuration for core components

type CoreHealthScenario

type CoreHealthScenario struct {
	// contains filtered or unexported fields
}

CoreHealthScenario validates core component health

func NewCoreHealthScenario

func NewCoreHealthScenario(obsClient *client.ObservabilityClient, config *CoreHealthConfig) *CoreHealthScenario

NewCoreHealthScenario creates a new core health check scenario

func (*CoreHealthScenario) Description

func (s *CoreHealthScenario) Description() string

Description returns the scenario description

func (*CoreHealthScenario) Execute

func (s *CoreHealthScenario) Execute(ctx context.Context) (*Result, error)

Execute runs the core health check scenario

func (*CoreHealthScenario) Name

func (s *CoreHealthScenario) Name() string

Name returns the scenario name

func (*CoreHealthScenario) Setup

Setup prepares the scenario (no-op for health check)

func (*CoreHealthScenario) Teardown

func (s *CoreHealthScenario) Teardown(ctx context.Context) error

Teardown cleans up after the scenario (no-op for health check)

type EmbeddingMetrics

type EmbeddingMetrics struct {
	// QueuedTotal is the total number of embeddings sent to the queue
	QueuedTotal int64 `json:"queued_total"`

	// GeneratedTotal is the total number of embeddings successfully generated
	GeneratedTotal int64 `json:"generated_total"`

	// DedupHits is the count of embeddings deduplicated (reused from cache)
	DedupHits int64 `json:"dedup_hits"`

	// FailedTotal is the count of failed embedding generations
	FailedTotal int64 `json:"failed_total"`

	// PendingCount is the current queue depth (should be 0 at test end)
	PendingCount int64 `json:"pending_count"`

	// DedupRate is the deduplication efficiency (dedupHits / queuedTotal)
	DedupRate float64 `json:"dedup_rate,omitempty"`

	// QueueDrained indicates if the queue was empty at validation time
	QueueDrained bool `json:"queue_drained"`

	// NoFailures indicates if there were zero failures
	NoFailures bool `json:"no_failures"`
}

EmbeddingMetrics contains embedding queue health metrics (Phase 4). These metrics provide visibility into the embedding pipeline flow.

type EntityResults

type EntityResults struct {
	// ExpectedCount is the expected number of entities from testdata
	ExpectedCount int `json:"expected_count"`

	// ActualCount is the number of entities found
	ActualCount int `json:"actual_count"`

	// MissingCount is expected - actual
	MissingCount int `json:"missing_count"`

	// DataLossPercent is the percentage of expected entities not found
	DataLossPercent float64 `json:"data_loss_percent"`

	// SampledCount is number of entities sampled for structure validation
	SampledCount int `json:"sampled_count"`

	// ValidatedCount is number passing structure validation
	ValidatedCount int `json:"validated_count"`

	// RetrievedCount is number successfully retrieved by ID
	RetrievedCount int `json:"retrieved_count"`
}

EntityResults contains entity validation results.

type GraphRAGQueryResult

type GraphRAGQueryResult struct {
	// Query is the search query used
	Query string `json:"query"`

	// Response is the generated response
	Response string `json:"response,omitempty"`

	// EntitiesUsed is the number of entities in context
	EntitiesUsed int `json:"entities_used"`

	// CommunitiesUsed is the number of communities in context
	CommunitiesUsed int `json:"communities_used"`

	// LatencyMs is the query execution time
	LatencyMs int64 `json:"latency_ms"`

	// Success indicates if the query completed successfully
	Success bool `json:"success"`
}

GraphRAGQueryResult contains results from a single GraphRAG query.

type GraphRAGResults

type GraphRAGResults struct {
	// LocalQuery contains local search results
	LocalQuery *GraphRAGQueryResult `json:"local_query,omitempty"`

	// GlobalQuery contains global search results
	GlobalQuery *GraphRAGQueryResult `json:"global_query,omitempty"`
}

GraphRAGResults contains GraphRAG query test results. GraphRAG is a Tier 2 (semantic) capability - runs on semantic tier only.

type HierarchyResults

type HierarchyResults struct {
	// ContainerCount is the number of hierarchy container entities detected
	ContainerCount int `json:"container_count"`

	// SourceEntityCount is the number of non-container entities (original testdata entities,
	// not auto-created by hierarchy inference). Previously called "content entities" but
	// renamed to avoid confusion with ContentStorable entities that have text for embeddings.
	SourceEntityCount int `json:"source_entity_count"`

	// ExpectedMinContainers is the minimum expected containers for the entity count
	ExpectedMinContainers int `json:"expected_min_containers"`

	// InferenceWorking indicates if hierarchy inference is creating containers
	InferenceWorking bool `json:"inference_working"`

	// ContainerTypes contains counts by container type suffix
	ContainerTypes map[string]int `json:"container_types,omitempty"`
}

HierarchyResults contains hierarchy inference validation results (Phase 4). These validate that the KV watcher pattern is creating container entities.

type IncomingIndexResults

type IncomingIndexResults struct {
	// EntriesWithPredicates is the count of entries that have predicate info
	EntriesWithPredicates int `json:"entries_with_predicates"`

	// HierarchyMemberCount is the count of "hierarchy.type.member" predicates
	HierarchyMemberCount int `json:"hierarchy_member_count"`

	// PredicateValidation indicates if predicates are stored correctly
	PredicateValidation bool `json:"predicate_validation"`

	// SampleContainerID is the container entity used for validation
	SampleContainerID string `json:"sample_container_id,omitempty"`
}

IncomingIndexResults contains IncomingIndex verification results. Phase 5: Added to verify bidirectional graph traversal preserves predicates.

type IndexDetail

type IndexDetail struct {
	// Name of the index/bucket
	Name string `json:"name"`

	// Populated indicates if the index has data
	Populated bool `json:"populated"`

	// KeyCount is the number of keys (if available)
	KeyCount int `json:"key_count,omitempty"`
}

IndexDetail contains details about a single index.

type IndexResults

type IndexResults struct {
	// ExpectedIndexes is the total number of expected indexes
	ExpectedIndexes int `json:"expected_indexes"`

	// PopulatedIndexes is the number with data
	PopulatedIndexes int `json:"populated_indexes"`

	// IndexDetails contains per-index status
	IndexDetails map[string]IndexDetail `json:"index_details,omitempty"`
}

IndexResults contains index population results.

type KCoreResults

type KCoreResults struct {
	// MaxCore is the highest core number in the graph
	MaxCore int `json:"max_core"`

	// EntityCount is the number of entities in the index
	EntityCount int `json:"entity_count"`

	// CoreBucketCounts maps core level to entity count
	CoreBucketCounts map[int]int `json:"core_bucket_counts,omitempty"`

	// Verified indicates if the index was verified
	Verified bool `json:"verified"`
}

KCoreResults contains k-core verification results.

type OutputResults

type OutputResults struct {
	// ExpectedCount is the number of expected outputs
	ExpectedCount int `json:"expected_count"`

	// FoundCount is the number of outputs found
	FoundCount int `json:"found_count"`

	// Outputs lists individual output status
	Outputs map[string]bool `json:"outputs,omitempty"`
}

OutputResults contains output processor verification results.

type PathRAGBoundaryResults

type PathRAGBoundaryResults struct {
	// MaxNodesLimit is the configured maxNodes parameter
	MaxNodesLimit int `json:"max_nodes_limit"`

	// EntitiesReturned is the actual number of entities returned
	EntitiesReturned int `json:"entities_returned"`

	// RespectedLimit indicates if the limit was properly enforced
	RespectedLimit bool `json:"respected_limit"`
}

PathRAGBoundaryResults contains results from PathRAG boundary testing.

type PathRAGEntity

type PathRAGEntity struct {
	// ID is the entity identifier
	ID string `json:"id"`

	// Score is the decay-weighted relevance score (1.0 = start, decreases with hops)
	Score float64 `json:"score"`
}

PathRAGEntity represents a single entity from PathRAG results.

type PathRAGResults

type PathRAGResults struct {
	// StartEntity is the entity ID used as traversal starting point
	StartEntity string `json:"start_entity"`

	// EntitiesFound is the total number of entities discovered
	EntitiesFound int `json:"entities_found"`

	// PathsFound is the number of unique paths discovered
	PathsFound int `json:"paths_found"`

	// Entities contains the discovered entities with their scores
	Entities []PathRAGEntity `json:"entities,omitempty"`

	// ScoresValid indicates if decay scoring was validated correctly
	ScoresValid bool `json:"scores_valid"`

	// Truncated indicates if results were truncated due to maxNodes limit
	Truncated bool `json:"truncated"`

	// LatencyMs is the query execution time in milliseconds
	LatencyMs int64 `json:"latency_ms"`

	// BoundaryTest contains results from maxNodes boundary testing
	BoundaryTest *PathRAGBoundaryResults `json:"boundary_test,omitempty"`
}

PathRAGResults contains PathRAG graph traversal test results. PathRAG is a Tier 0 (structural) capability - runs on all tiers.

type PivotResults

type PivotResults struct {
	// PivotCount is the number of pivot nodes
	PivotCount int `json:"pivot_count"`

	// EntityCount is the number of entities with distance vectors
	EntityCount int `json:"entity_count"`

	// TriangleInequalityValid indicates if distance bounds are valid
	TriangleInequalityValid bool `json:"triangle_inequality_valid"`

	// Verified indicates if the index was verified
	Verified bool `json:"verified"`
}

PivotResults contains pivot index verification results.

type Result

type Result struct {
	// Scenario identification
	ScenarioName string        `json:"scenario_name"`
	StartTime    time.Time     `json:"start_time"`
	EndTime      time.Time     `json:"end_time"`
	Duration     time.Duration `json:"duration"`

	// Overall status
	Success bool   `json:"success"`
	Error   string `json:"error,omitempty"`

	// Detailed results (legacy flat format)
	Metrics  map[string]any `json:"metrics,omitempty"`
	Details  map[string]any `json:"details,omitempty"`
	Errors   []string       `json:"errors,omitempty"`
	Warnings []string       `json:"warnings,omitempty"`

	// Structured results (new typed format for tiered scenarios)
	// This is populated alongside Metrics/Details for backward compatibility
	Structured *TieredResults `json:"structured,omitempty"`
}

Result contains the outcome of a scenario execution

type RuleResults

type RuleResults struct {
	// EvaluatedCount is total rules evaluated
	EvaluatedCount int `json:"evaluated_count"`

	// FiringsCount is rules that fired (conditions met)
	FiringsCount int `json:"firings_count"`

	// ValidationPassed indicates if rule validation passed
	ValidationPassed bool `json:"validation_passed"`

	// ActionsDispatched is count of actions dispatched by reactive workflow
	ActionsDispatched int `json:"actions_dispatched,omitempty"`

	// ExecutionsCreated is count of workflow executions created
	ExecutionsCreated int `json:"executions_created,omitempty"`
}

RuleResults contains rule evaluation results.

type Scenario

type Scenario interface {
	// Name returns the scenario name for identification and reporting
	Name() string

	// Description provides a human-readable description of what the scenario tests
	Description() string

	// Setup prepares the scenario environment before execution
	// This may include creating test data, configuring components, etc.
	Setup(ctx context.Context) error

	// Execute runs the actual test scenario
	// Returns detailed results including pass/fail status and diagnostics
	Execute(ctx context.Context) (*Result, error)

	// Teardown cleans up after the scenario execution
	// This should restore the system to its original state
	Teardown(ctx context.Context) error
}

Scenario defines the interface that all E2E test scenarios must implement

type SearchResults

type SearchResults struct {
	// Stats from the search executor
	Stats *search.Stats `json:"stats"`

	// WeakResultsThreshold is the minimum acceptable average score
	WeakResultsThreshold float64 `json:"weak_results_threshold"`

	// IsWeak indicates if overall results are below threshold
	IsWeak bool `json:"is_weak"`
}

SearchResults contains search quality test results.

type StructuralIndexResults

type StructuralIndexResults struct {
	// KCore contains k-core decomposition results
	KCore *KCoreResults `json:"kcore,omitempty"`

	// Pivot contains pivot distance index results
	Pivot *PivotResults `json:"pivot,omitempty"`
}

StructuralIndexResults contains k-core and pivot index verification results. This is used by Phase 7 structural index verification.

type TestMetadata

type TestMetadata struct {
	// Variant that was tested
	Variant string `json:"variant"`

	// StartedAt is when the test started
	StartedAt time.Time `json:"started_at"`

	// CompletedAt is when the test completed
	CompletedAt time.Time `json:"completed_at"`

	// Success indicates if the test passed
	Success bool `json:"success"`

	// ErrorCount is the number of errors
	ErrorCount int `json:"error_count"`

	// WarningCount is the number of warnings
	WarningCount int `json:"warning_count"`

	// Errors contains the actual error messages
	Errors []string `json:"errors,omitempty"`

	// Warnings contains the actual warning messages
	Warnings []string `json:"warnings,omitempty"`

	// Version information
	Version string `json:"version,omitempty"`
}

TestMetadata contains test run metadata.

type TieredConfig

type TieredConfig struct {
	// Variant configuration
	Variant string `json:"variant"` // "structural", "statistical", "semantic"

	// Test data configuration
	MessageCount    int           `json:"message_count"`
	MessageInterval time.Duration `json:"message_interval"`

	// Validation configuration (event-driven, matching structural tier patterns)
	ValidationTimeout time.Duration `json:"validation_timeout"` // Timeout for metric waits (30s for semantic)
	PollInterval      time.Duration `json:"poll_interval"`      // Poll interval for metric waits (100ms)
	MinProcessed      int           `json:"min_processed"`

	// Entity verification (from test data files)
	MinExpectedEntities int    `json:"min_expected_entities"`
	NatsURL             string `json:"nats_url"`
	MetricsURL          string `json:"metrics_url"`
	ServiceManagerURL   string `json:"service_manager_url"`
	GatewayURL          string `json:"gateway_url"`
	GraphQLURL          string `json:"graphql_url"` // GraphQL endpoint (port varies by profile)

	// Comparison output configuration
	OutputDir string `json:"output_dir"`

	// Baseline comparison (matching structural tier patterns)
	BaselineFile         string  `json:"baseline_file,omitempty"` // Path to baseline JSON (optional)
	MaxRegressionPercent float64 `json:"max_regression_percent"`  // Default 20%

	// Structural tier config (rules-only, no ML dependencies)
	ExpectedEmbeddings   int `json:"expected_embeddings"`    // 0 for structural variant
	ExpectedClusters     int `json:"expected_clusters"`      // 0 for structural variant
	MinRulesEvaluated    int `json:"min_rules_evaluated"`    // Min rules evaluated for structural
	MinRuleFirings       int `json:"min_rule_firings"`       // Min rule firings (conditions met)
	MinActionsDispatched int `json:"min_actions_dispatched"` // Min actions dispatched
}

TieredConfig contains configuration for tiered E2E tests

func DefaultTieredConfig

func DefaultTieredConfig() *TieredConfig

DefaultTieredConfig returns default configuration

type TieredResults

type TieredResults struct {
	// Variant information
	Variant VariantResults `json:"variant"`

	// Entity validation results
	Entities EntityResults `json:"entities"`

	// Index population results
	Indexes IndexResults `json:"indexes"`

	// Search quality results
	Search SearchResults `json:"search"`

	// Rule evaluation results
	Rules RuleResults `json:"rules"`

	// Community detection results (statistical/semantic only)
	Communities *CommunityResults `json:"communities,omitempty"`

	// Anomaly detection results (semantic only - uses k-core and pivot indexes)
	Anomalies *AnomalyResults `json:"anomalies,omitempty"`

	// PathRAG graph traversal results (Tier 0 - runs on all tiers)
	// Two PathRAG tests: sensor (structured IoT) and document (text-rich)
	PathRAGSensor   *PathRAGResults `json:"pathrag_sensor,omitempty"`
	PathRAGDocument *PathRAGResults `json:"pathrag_document,omitempty"`

	// Structural index results (k-core, pivot)
	StructuralIndexes *StructuralIndexResults `json:"structural_indexes,omitempty"`

	// GraphRAG query results (Tier 2 - semantic only)
	GraphRAG *GraphRAGResults `json:"graphrag,omitempty"`

	// Component health results
	Components ComponentResults `json:"components"`

	// Output verification results
	Outputs OutputResults `json:"outputs"`

	// Embedding queue metrics (Phase 4 - statistical/semantic only)
	Embeddings *EmbeddingMetrics `json:"embeddings,omitempty"`

	// Hierarchy inference results (Phase 4 - statistical/semantic only)
	Hierarchy *HierarchyResults `json:"hierarchy,omitempty"`

	// Timing information
	Timing TimingResults `json:"timing"`

	// Test metadata
	Metadata TestMetadata `json:"metadata"`
}

TieredResults contains structured results from a tiered e2e test run. This replaces the flat result.Details and result.Metrics maps with typed structures.

func BuildTieredResults

func BuildTieredResults(result *Result, searchStats *search.Stats) *TieredResults

BuildTieredResults creates a TieredResults from legacy Result data and search stats. This provides the bridge between the old flat format and new structured format.

func LoadStructuredResults

func LoadStructuredResults(path string) (*TieredResults, error)

LoadStructuredResults reads structured results from a JSON file.

type TieredScenario

type TieredScenario struct {
	// contains filtered or unexported fields
}

TieredScenario validates comprehensive semantic processing

func NewTieredScenario

func NewTieredScenario(
	obsClient *client.ObservabilityClient,
	udpAddr string,
	cfg *TieredConfig,
) *TieredScenario

NewTieredScenario creates a new tiered semantic test scenario

func (*TieredScenario) Description

func (s *TieredScenario) Description() string

Description returns the scenario description

func (*TieredScenario) Execute

func (s *TieredScenario) Execute(ctx context.Context) (*Result, error)

Execute runs the tiered semantic test scenario

func (*TieredScenario) Name

func (s *TieredScenario) Name() string

Name returns the scenario name

func (*TieredScenario) Setup

func (s *TieredScenario) Setup(ctx context.Context) error

Setup prepares the scenario

func (*TieredScenario) Teardown

func (s *TieredScenario) Teardown(ctx context.Context) error

Teardown cleans up after the scenario

type TimingResults

type TimingResults struct {
	// TotalDurationMs is the total test duration
	TotalDurationMs int64 `json:"total_duration_ms"`

	// StageDurations maps stage name to duration in ms
	StageDurations map[string]int64 `json:"stage_durations"`
}

TimingResults contains timing information for each stage.

type VariantResults

type VariantResults struct {
	// Name is the variant: "structural", "statistical", "semantic"
	Name string `json:"name"`

	// EmbeddingProvider: "disabled", "bm25", "http"
	EmbeddingProvider string `json:"embedding_provider"`

	// SemembedAvailable indicates if external embedding service is available
	SemembedAvailable bool `json:"semembed_available"`
}

VariantResults contains variant detection information.

type VirtualEdgeResults

type VirtualEdgeResults struct {
	// Total is the total number of virtual edges created
	Total int `json:"total"`

	// High is count of edges with similarity >= 0.9
	High int `json:"high"`

	// Medium is count of edges with similarity >= 0.85
	Medium int `json:"medium"`

	// Related is count of edges with similarity >= 0.8
	Related int `json:"related"`

	// AutoApplied is count of anomalies with auto_applied status
	AutoApplied int `json:"auto_applied"`
}

VirtualEdgeResults contains virtual edge creation metrics from semantic inference.

Directories

Path Synopsis
Package agentic provides the agentic E2E test scenario.
Package agentic provides the agentic E2E test scenario.
Package anomaly provides ground truth validation for anomaly detection.
Package anomaly provides ground truth validation for anomaly detection.
Package boids provides the Boids coordination A/B test scenario.
Package boids provides the Boids coordination A/B test scenario.
Package community provides ground truth validation for community detection.
Package community provides ground truth validation for community detection.
Package search provides unified search query execution for e2e tests.
Package search provides unified search query execution for e2e tests.
Package stages contains extracted stage implementations for tiered E2E tests
Package stages contains extracted stage implementations for tiered E2E tests
Package throughput provides a high-throughput E2E scenario for performance profiling.
Package throughput provides a high-throughput E2E scenario for performance profiling.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL