Documentation
¶
Overview ¶
Package scenarios provides E2E test scenarios for SemStreams
Package scenarios provides E2E test scenarios for SemStreams ¶
Package scenarios provides E2E test scenarios for SemStreams ¶
Package scenarios provides E2E test scenarios for SemStreams semantic processing ¶
Package scenarios provides common result types for E2E tests ¶
Package scenarios provides tier capability result types for E2E tests ¶
Package scenarios defines E2E test scenarios for SemStreams ¶
Package scenarios provides E2E test scenarios for SemStreams semantic processing ¶
Package scenarios provides E2E test scenarios for SemStreams semantic processing ¶
Package scenarios provides E2E test scenarios for SemStreams semantic processing ¶
Package scenarios provides E2E test scenarios for SemStreams semantic processing ¶
Package scenarios provides E2E test scenarios for SemStreams semantic processing ¶
Package scenarios provides E2E test scenarios for SemStreams semantic processing ¶
Package scenarios provides E2E test scenarios for SemStreams semantic processing ¶
Package scenarios provides E2E test scenarios for SemStreams semantic processing
Index ¶
- Constants
- func SaveMetricsDump(metricsData string, variant string, outputDir string) (string, error)
- func SaveStructuredResults(tr *TieredResults, outputDir string) (string, error)
- type AnomalyDetail
- type AnomalyGroundTruthResults
- type AnomalyResults
- type AnomalyStatusCounts
- type CommunityComparison
- type CommunityResults
- type CommunitySummaryReport
- type ComponentResults
- type ContextIndexResults
- type CoreDataflowConfig
- type CoreDataflowScenario
- func (s *CoreDataflowScenario) Description() string
- func (s *CoreDataflowScenario) Execute(ctx context.Context) (*Result, error)
- func (s *CoreDataflowScenario) Name() string
- func (s *CoreDataflowScenario) Setup(_ context.Context) error
- func (s *CoreDataflowScenario) Teardown(_ context.Context) error
- type CoreFederationConfig
- type CoreFederationScenario
- func (s *CoreFederationScenario) Description() string
- func (s *CoreFederationScenario) Execute(ctx context.Context) (*Result, error)
- func (s *CoreFederationScenario) Name() string
- func (s *CoreFederationScenario) Setup(_ context.Context) error
- func (s *CoreFederationScenario) Teardown(_ context.Context) error
- type CoreHealthConfig
- type CoreHealthScenario
- type EmbeddingMetrics
- type EntityResults
- type GraphRAGQueryResult
- type GraphRAGResults
- type HierarchyResults
- type IncomingIndexResults
- type IndexDetail
- type IndexResults
- type KCoreResults
- type OutputResults
- type PathRAGBoundaryResults
- type PathRAGEntity
- type PathRAGResults
- type PivotResults
- type Result
- type RuleResults
- type Scenario
- type SearchResults
- type StructuralIndexResults
- type TestMetadata
- type TieredConfig
- type TieredResults
- type TieredScenario
- type TimingResults
- type VariantResults
- type VirtualEdgeResults
Constants ¶
const ( // StructuralMinEntities is the minimum entities expected for structural tier // All tiers now load testdata/semantic/*.jsonl with 74 unique entities StructuralMinEntities = 50 // StatisticalMinEntities is the minimum entities expected for statistical tier // Statistical tier loads testdata/semantic/*.jsonl with 74 unique entities StatisticalMinEntities = 50 // SemanticMinEntities is the minimum entities expected for semantic tier // Same testdata as statistical tier SemanticMinEntities = 50 )
Variant-specific entity count expectations
Variables ¶
This section is empty.
Functions ¶
func SaveMetricsDump ¶
SaveMetricsDump saves the raw Prometheus metrics to a text file. The filename follows the pattern: {variant}-{timestamp}-metrics.txt
func SaveStructuredResults ¶
func SaveStructuredResults(tr *TieredResults, outputDir string) (string, error)
SaveStructuredResults writes the structured results to a JSON file. The filename format is: {variant}-{timestamp}.json
Types ¶
type AnomalyDetail ¶
type AnomalyDetail struct {
ID string `json:"id"`
Type string `json:"type"`
EntityA string `json:"entity_a"`
EntityB string `json:"entity_b,omitempty"`
Confidence float64 `json:"confidence"`
Status string `json:"status"`
}
AnomalyDetail contains the details of a single anomaly for auditability.
type AnomalyGroundTruthResults ¶
type AnomalyGroundTruthResults struct {
ExpectedTotal int `json:"expected_total"`
ExpectedFound int `json:"expected_found"`
FalsePositiveTotal int `json:"false_positive_total"`
DetectedTotal int `json:"detected_total"`
FalsePositiveRate float64 `json:"false_positive_rate"`
Passed bool `json:"passed"`
Violations []map[string]any `json:"violations,omitempty"`
}
AnomalyGroundTruthResults contains the results of ground truth validation.
type AnomalyResults ¶
type AnomalyResults struct {
// Total is the total number of anomalies detected
Total int `json:"total"`
// SemanticGap is count of semantic-structural gap anomalies (entities semantically
// similar but structurally distant - detected via pivot distance)
SemanticGap int `json:"semantic_gap"`
// CoreIsolation is count of hub isolation anomalies (high k-core entities with
// few connections to other high-core entities)
CoreIsolation int `json:"core_isolation"`
// CoreDemotion is count of core demotion anomalies (entities that dropped
// significantly in k-core value between analysis runs)
CoreDemotion int `json:"core_demotion"`
// Transitivity is count of transitivity gap anomalies (missing expected
// transitive relationships)
Transitivity int `json:"transitivity"`
// ByStatus contains counts by review status
ByStatus AnomalyStatusCounts `json:"by_status"`
// VirtualEdges contains counts of auto-applied semantic edges
VirtualEdges *VirtualEdgeResults `json:"virtual_edges,omitempty"`
// List contains the actual anomaly details for auditability
List []AnomalyDetail `json:"list,omitempty"`
// GroundTruth contains the results of ground truth validation
GroundTruth *AnomalyGroundTruthResults `json:"ground_truth,omitempty"`
}
AnomalyResults contains structural anomaly detection results.
type AnomalyStatusCounts ¶
type AnomalyStatusCounts struct {
Pending int `json:"pending"`
Confirmed int `json:"confirmed"`
Dismissed int `json:"dismissed"`
}
AnomalyStatusCounts contains anomaly counts by review status.
type CommunityComparison ¶
type CommunityComparison struct {
CommunityID string `json:"community_id"`
Level int `json:"level"`
MemberCount int `json:"member_count"`
StatisticalSummary string `json:"statistical_summary"`
LLMSummary string `json:"llm_summary,omitempty"`
SummaryStatus string `json:"summary_status"`
Keywords []string `json:"keywords"`
SummaryLengthRatio float64 `json:"summary_length_ratio,omitempty"`
WordOverlap float64 `json:"word_overlap,omitempty"`
}
CommunityComparison represents a comparison of statistical vs LLM summaries for a community
type CommunityResults ¶
type CommunityResults struct {
// TotalCommunities is the number of communities detected
TotalCommunities int `json:"total_communities"`
// NonSingletonCount is communities with more than one member
NonSingletonCount int `json:"non_singleton_count"`
// LargestSize is the size of the largest community
LargestSize int `json:"largest_size"`
// AverageSize is the average community size
AverageSize float64 `json:"average_size"`
// WithKeywords is count of communities that have keywords
WithKeywords int `json:"with_keywords"`
// LLMEnhanced is count of communities with LLM summaries (semantic only)
LLMEnhanced int `json:"llm_enhanced,omitempty"`
}
CommunityResults contains community detection results.
type CommunitySummaryReport ¶
type CommunitySummaryReport struct {
Variant string `json:"variant"`
Timestamp time.Time `json:"timestamp"`
CommunitiesTotal int `json:"communities_total"`
LLMEnhancedCount int `json:"llm_enhanced_count"`
StatisticalOnlyCount int `json:"statistical_only_count"`
LLMFailedCount int `json:"llm_failed_count,omitempty"`
LLMPendingCount int `json:"llm_pending_count,omitempty"`
LLMWaitDurationMs int64 `json:"llm_wait_duration_ms,omitempty"`
AvgSummaryLengthRatio float64 `json:"avg_summary_length_ratio"`
AvgWordOverlap float64 `json:"avg_word_overlap"`
NonSingletonCount int `json:"non_singleton_count"`
LargestCommunitySize int `json:"largest_community_size"`
AvgNonSingletonSize float64 `json:"avg_non_singleton_size"`
Communities []CommunityComparison `json:"communities"`
}
CommunitySummaryReport contains aggregated community summary metrics
type ComponentResults ¶
type ComponentResults struct {
// ExpectedCount is the number of expected components
ExpectedCount int `json:"expected_count"`
// FoundCount is the number of components found healthy
FoundCount int `json:"found_count"`
// Components lists individual component status
Components map[string]bool `json:"components,omitempty"`
}
ComponentResults contains component health results.
type ContextIndexResults ¶
type ContextIndexResults struct {
// TotalKeys is the number of unique context values in the index
TotalKeys int `json:"total_keys"`
// HierarchyContextFound indicates if "inference.hierarchy" key exists
HierarchyContextFound bool `json:"hierarchy_context_found"`
// HierarchyEntryCount is the number of entity+predicate pairs for hierarchy inference
HierarchyEntryCount int `json:"hierarchy_entry_count,omitempty"`
// SampleContexts contains sample context keys for debugging
SampleContexts []string `json:"sample_contexts,omitempty"`
}
ContextIndexResults contains ContextIndex verification results. Phase 5: Added to verify triple provenance tracking via context values.
type CoreDataflowConfig ¶
type CoreDataflowConfig struct {
// Test data configuration
MessageCount int `json:"message_count"`
MessageInterval time.Duration `json:"message_interval"`
// Validation configuration
ValidationDelay time.Duration `json:"validation_delay"`
MinProcessed int `json:"min_processed"`
// WebSocket configuration
WebSocketTimeout time.Duration `json:"websocket_timeout"`
TestFlowID string `json:"test_flow_id"`
}
CoreDataflowConfig contains configuration for dataflow test
func DefaultCoreDataflowConfig ¶
func DefaultCoreDataflowConfig() *CoreDataflowConfig
DefaultCoreDataflowConfig returns default configuration
type CoreDataflowScenario ¶
type CoreDataflowScenario struct {
// contains filtered or unexported fields
}
CoreDataflowScenario validates complete core data pipeline
func NewCoreDataflowScenario ¶
func NewCoreDataflowScenario( obsClient *client.ObservabilityClient, wsClient *client.WebSocketClient, udpAddr string, config *CoreDataflowConfig, ) *CoreDataflowScenario
NewCoreDataflowScenario creates a new core dataflow test scenario
func (*CoreDataflowScenario) Description ¶
func (s *CoreDataflowScenario) Description() string
Description returns the scenario description
func (*CoreDataflowScenario) Execute ¶
func (s *CoreDataflowScenario) Execute(ctx context.Context) (*Result, error)
Execute runs the dataflow test scenario
func (*CoreDataflowScenario) Name ¶
func (s *CoreDataflowScenario) Name() string
Name returns the scenario name
type CoreFederationConfig ¶
type CoreFederationConfig struct {
// Test data configuration
MessageCount int `json:"message_count"`
MessageInterval time.Duration `json:"message_interval"`
// Validation configuration
ValidationDelay time.Duration `json:"validation_delay"`
MinMessagesOnCloud int `json:"min_messages_on_cloud"`
AckVerification bool `json:"ack_verification"`
// Container configuration for file output verification
CloudContainerName string `json:"cloud_container_name"`
CloudOutputPattern string `json:"cloud_output_pattern"`
}
CoreFederationConfig contains configuration for federation test
func DefaultCoreFederationConfig ¶
func DefaultCoreFederationConfig() *CoreFederationConfig
DefaultCoreFederationConfig returns default configuration
type CoreFederationScenario ¶
type CoreFederationScenario struct {
// contains filtered or unexported fields
}
CoreFederationScenario validates federation data flow with ack/nack protocol
func NewCoreFederationScenario ¶
func NewCoreFederationScenario( edgeClient *client.ObservabilityClient, cloudClient *client.ObservabilityClient, udpAddr string, wsURL string, config *CoreFederationConfig, ) *CoreFederationScenario
NewCoreFederationScenario creates a new federation test scenario
func (*CoreFederationScenario) Description ¶
func (s *CoreFederationScenario) Description() string
Description returns the scenario description
func (*CoreFederationScenario) Execute ¶
func (s *CoreFederationScenario) Execute(ctx context.Context) (*Result, error)
Execute runs the federation test scenario
func (*CoreFederationScenario) Name ¶
func (s *CoreFederationScenario) Name() string
Name returns the scenario name
type CoreHealthConfig ¶
type CoreHealthConfig struct {
// Validation thresholds
RequireAllHealthy bool `json:"require_all_healthy"`
MinHealthyComponents int `json:"min_healthy_components"`
// Required core components
RequiredComponents []string `json:"required_components"`
}
CoreHealthConfig contains configuration for core health check
func DefaultCoreHealthConfig ¶
func DefaultCoreHealthConfig() *CoreHealthConfig
DefaultCoreHealthConfig returns default configuration for core components
type CoreHealthScenario ¶
type CoreHealthScenario struct {
// contains filtered or unexported fields
}
CoreHealthScenario validates core component health
func NewCoreHealthScenario ¶
func NewCoreHealthScenario(obsClient *client.ObservabilityClient, config *CoreHealthConfig) *CoreHealthScenario
NewCoreHealthScenario creates a new core health check scenario
func (*CoreHealthScenario) Description ¶
func (s *CoreHealthScenario) Description() string
Description returns the scenario description
func (*CoreHealthScenario) Execute ¶
func (s *CoreHealthScenario) Execute(ctx context.Context) (*Result, error)
Execute runs the core health check scenario
func (*CoreHealthScenario) Name ¶
func (s *CoreHealthScenario) Name() string
Name returns the scenario name
type EmbeddingMetrics ¶
type EmbeddingMetrics struct {
// QueuedTotal is the total number of embeddings sent to the queue
QueuedTotal int64 `json:"queued_total"`
// GeneratedTotal is the total number of embeddings successfully generated
GeneratedTotal int64 `json:"generated_total"`
// DedupHits is the count of embeddings deduplicated (reused from cache)
DedupHits int64 `json:"dedup_hits"`
// FailedTotal is the count of failed embedding generations
FailedTotal int64 `json:"failed_total"`
// PendingCount is the current queue depth (should be 0 at test end)
PendingCount int64 `json:"pending_count"`
// DedupRate is the deduplication efficiency (dedupHits / queuedTotal)
DedupRate float64 `json:"dedup_rate,omitempty"`
// QueueDrained indicates if the queue was empty at validation time
QueueDrained bool `json:"queue_drained"`
// NoFailures indicates if there were zero failures
NoFailures bool `json:"no_failures"`
}
EmbeddingMetrics contains embedding queue health metrics (Phase 4). These metrics provide visibility into the embedding pipeline flow.
type EntityResults ¶
type EntityResults struct {
// ExpectedCount is the expected number of entities from testdata
ExpectedCount int `json:"expected_count"`
// ActualCount is the number of entities found
ActualCount int `json:"actual_count"`
// MissingCount is expected - actual
MissingCount int `json:"missing_count"`
// DataLossPercent is the percentage of expected entities not found
DataLossPercent float64 `json:"data_loss_percent"`
// SampledCount is number of entities sampled for structure validation
SampledCount int `json:"sampled_count"`
// ValidatedCount is number passing structure validation
ValidatedCount int `json:"validated_count"`
// RetrievedCount is number successfully retrieved by ID
RetrievedCount int `json:"retrieved_count"`
}
EntityResults contains entity validation results.
type GraphRAGQueryResult ¶
type GraphRAGQueryResult struct {
// Query is the search query used
Query string `json:"query"`
// Response is the generated response
Response string `json:"response,omitempty"`
// EntitiesUsed is the number of entities in context
EntitiesUsed int `json:"entities_used"`
// CommunitiesUsed is the number of communities in context
CommunitiesUsed int `json:"communities_used"`
// LatencyMs is the query execution time
LatencyMs int64 `json:"latency_ms"`
// Success indicates if the query completed successfully
Success bool `json:"success"`
}
GraphRAGQueryResult contains results from a single GraphRAG query.
type GraphRAGResults ¶
type GraphRAGResults struct {
// LocalQuery contains local search results
LocalQuery *GraphRAGQueryResult `json:"local_query,omitempty"`
// GlobalQuery contains global search results
GlobalQuery *GraphRAGQueryResult `json:"global_query,omitempty"`
}
GraphRAGResults contains GraphRAG query test results. GraphRAG is a Tier 2 (semantic) capability - runs on semantic tier only.
type HierarchyResults ¶
type HierarchyResults struct {
// ContainerCount is the number of hierarchy container entities detected
ContainerCount int `json:"container_count"`
// SourceEntityCount is the number of non-container entities (original testdata entities,
// not auto-created by hierarchy inference). Previously called "content entities" but
// renamed to avoid confusion with ContentStorable entities that have text for embeddings.
SourceEntityCount int `json:"source_entity_count"`
// ExpectedMinContainers is the minimum expected containers for the entity count
ExpectedMinContainers int `json:"expected_min_containers"`
// InferenceWorking indicates if hierarchy inference is creating containers
InferenceWorking bool `json:"inference_working"`
// ContainerTypes contains counts by container type suffix
ContainerTypes map[string]int `json:"container_types,omitempty"`
}
HierarchyResults contains hierarchy inference validation results (Phase 4). These validate that the KV watcher pattern is creating container entities.
type IncomingIndexResults ¶
type IncomingIndexResults struct {
// EntriesWithPredicates is the count of entries that have predicate info
EntriesWithPredicates int `json:"entries_with_predicates"`
// HierarchyMemberCount is the count of "hierarchy.type.member" predicates
HierarchyMemberCount int `json:"hierarchy_member_count"`
// PredicateValidation indicates if predicates are stored correctly
PredicateValidation bool `json:"predicate_validation"`
// SampleContainerID is the container entity used for validation
SampleContainerID string `json:"sample_container_id,omitempty"`
}
IncomingIndexResults contains IncomingIndex verification results. Phase 5: Added to verify bidirectional graph traversal preserves predicates.
type IndexDetail ¶
type IndexDetail struct {
// Name of the index/bucket
Name string `json:"name"`
// Populated indicates if the index has data
Populated bool `json:"populated"`
// KeyCount is the number of keys (if available)
KeyCount int `json:"key_count,omitempty"`
}
IndexDetail contains details about a single index.
type IndexResults ¶
type IndexResults struct {
// ExpectedIndexes is the total number of expected indexes
ExpectedIndexes int `json:"expected_indexes"`
// PopulatedIndexes is the number with data
PopulatedIndexes int `json:"populated_indexes"`
// IndexDetails contains per-index status
IndexDetails map[string]IndexDetail `json:"index_details,omitempty"`
}
IndexResults contains index population results.
type KCoreResults ¶
type KCoreResults struct {
// MaxCore is the highest core number in the graph
MaxCore int `json:"max_core"`
// EntityCount is the number of entities in the index
EntityCount int `json:"entity_count"`
// CoreBucketCounts maps core level to entity count
CoreBucketCounts map[int]int `json:"core_bucket_counts,omitempty"`
// Verified indicates if the index was verified
Verified bool `json:"verified"`
}
KCoreResults contains k-core verification results.
type OutputResults ¶
type OutputResults struct {
// ExpectedCount is the number of expected outputs
ExpectedCount int `json:"expected_count"`
// FoundCount is the number of outputs found
FoundCount int `json:"found_count"`
// Outputs lists individual output status
Outputs map[string]bool `json:"outputs,omitempty"`
}
OutputResults contains output processor verification results.
type PathRAGBoundaryResults ¶
type PathRAGBoundaryResults struct {
// MaxNodesLimit is the configured maxNodes parameter
MaxNodesLimit int `json:"max_nodes_limit"`
// EntitiesReturned is the actual number of entities returned
EntitiesReturned int `json:"entities_returned"`
// RespectedLimit indicates if the limit was properly enforced
RespectedLimit bool `json:"respected_limit"`
}
PathRAGBoundaryResults contains results from PathRAG boundary testing.
type PathRAGEntity ¶
type PathRAGEntity struct {
// ID is the entity identifier
ID string `json:"id"`
// Score is the decay-weighted relevance score (1.0 = start, decreases with hops)
Score float64 `json:"score"`
}
PathRAGEntity represents a single entity from PathRAG results.
type PathRAGResults ¶
type PathRAGResults struct {
// StartEntity is the entity ID used as traversal starting point
StartEntity string `json:"start_entity"`
// EntitiesFound is the total number of entities discovered
EntitiesFound int `json:"entities_found"`
// PathsFound is the number of unique paths discovered
PathsFound int `json:"paths_found"`
// Entities contains the discovered entities with their scores
Entities []PathRAGEntity `json:"entities,omitempty"`
// ScoresValid indicates if decay scoring was validated correctly
ScoresValid bool `json:"scores_valid"`
// Truncated indicates if results were truncated due to maxNodes limit
Truncated bool `json:"truncated"`
// LatencyMs is the query execution time in milliseconds
LatencyMs int64 `json:"latency_ms"`
// BoundaryTest contains results from maxNodes boundary testing
BoundaryTest *PathRAGBoundaryResults `json:"boundary_test,omitempty"`
}
PathRAGResults contains PathRAG graph traversal test results. PathRAG is a Tier 0 (structural) capability - runs on all tiers.
type PivotResults ¶
type PivotResults struct {
// PivotCount is the number of pivot nodes
PivotCount int `json:"pivot_count"`
// EntityCount is the number of entities with distance vectors
EntityCount int `json:"entity_count"`
// TriangleInequalityValid indicates if distance bounds are valid
TriangleInequalityValid bool `json:"triangle_inequality_valid"`
// Verified indicates if the index was verified
Verified bool `json:"verified"`
}
PivotResults contains pivot index verification results.
type Result ¶
type Result struct {
// Scenario identification
ScenarioName string `json:"scenario_name"`
StartTime time.Time `json:"start_time"`
EndTime time.Time `json:"end_time"`
Duration time.Duration `json:"duration"`
// Overall status
Success bool `json:"success"`
Error string `json:"error,omitempty"`
// Detailed results (legacy flat format)
Metrics map[string]any `json:"metrics,omitempty"`
Details map[string]any `json:"details,omitempty"`
Errors []string `json:"errors,omitempty"`
Warnings []string `json:"warnings,omitempty"`
// Structured results (new typed format for tiered scenarios)
// This is populated alongside Metrics/Details for backward compatibility
Structured *TieredResults `json:"structured,omitempty"`
}
Result contains the outcome of a scenario execution
type RuleResults ¶
type RuleResults struct {
// EvaluatedCount is total rules evaluated
EvaluatedCount int `json:"evaluated_count"`
// FiringsCount is rules that fired (conditions met)
FiringsCount int `json:"firings_count"`
// ValidationPassed indicates if rule validation passed
ValidationPassed bool `json:"validation_passed"`
// ActionsDispatched is count of actions dispatched by reactive workflow
ActionsDispatched int `json:"actions_dispatched,omitempty"`
// ExecutionsCreated is count of workflow executions created
ExecutionsCreated int `json:"executions_created,omitempty"`
}
RuleResults contains rule evaluation results.
type Scenario ¶
type Scenario interface {
// Name returns the scenario name for identification and reporting
Name() string
// Description provides a human-readable description of what the scenario tests
Description() string
// Setup prepares the scenario environment before execution
// This may include creating test data, configuring components, etc.
Setup(ctx context.Context) error
// Execute runs the actual test scenario
// Returns detailed results including pass/fail status and diagnostics
Execute(ctx context.Context) (*Result, error)
// Teardown cleans up after the scenario execution
// This should restore the system to its original state
Teardown(ctx context.Context) error
}
Scenario defines the interface that all E2E test scenarios must implement
type SearchResults ¶
type SearchResults struct {
// Stats from the search executor
Stats *search.Stats `json:"stats"`
// WeakResultsThreshold is the minimum acceptable average score
WeakResultsThreshold float64 `json:"weak_results_threshold"`
// IsWeak indicates if overall results are below threshold
IsWeak bool `json:"is_weak"`
}
SearchResults contains search quality test results.
type StructuralIndexResults ¶
type StructuralIndexResults struct {
// KCore contains k-core decomposition results
KCore *KCoreResults `json:"kcore,omitempty"`
// Pivot contains pivot distance index results
Pivot *PivotResults `json:"pivot,omitempty"`
}
StructuralIndexResults contains k-core and pivot index verification results. This is used by Phase 7 structural index verification.
type TestMetadata ¶
type TestMetadata struct {
// Variant that was tested
Variant string `json:"variant"`
// StartedAt is when the test started
StartedAt time.Time `json:"started_at"`
// CompletedAt is when the test completed
CompletedAt time.Time `json:"completed_at"`
// Success indicates if the test passed
Success bool `json:"success"`
// ErrorCount is the number of errors
ErrorCount int `json:"error_count"`
// WarningCount is the number of warnings
WarningCount int `json:"warning_count"`
// Errors contains the actual error messages
Errors []string `json:"errors,omitempty"`
// Warnings contains the actual warning messages
Warnings []string `json:"warnings,omitempty"`
// Version information
Version string `json:"version,omitempty"`
}
TestMetadata contains test run metadata.
type TieredConfig ¶
type TieredConfig struct {
// Variant configuration
Variant string `json:"variant"` // "structural", "statistical", "semantic"
// Test data configuration
MessageCount int `json:"message_count"`
MessageInterval time.Duration `json:"message_interval"`
// Validation configuration (event-driven, matching structural tier patterns)
ValidationTimeout time.Duration `json:"validation_timeout"` // Timeout for metric waits (30s for semantic)
PollInterval time.Duration `json:"poll_interval"` // Poll interval for metric waits (100ms)
MinProcessed int `json:"min_processed"`
// Entity verification (from test data files)
MinExpectedEntities int `json:"min_expected_entities"`
NatsURL string `json:"nats_url"`
MetricsURL string `json:"metrics_url"`
ServiceManagerURL string `json:"service_manager_url"`
GatewayURL string `json:"gateway_url"`
GraphQLURL string `json:"graphql_url"` // GraphQL endpoint (port varies by profile)
// Comparison output configuration
OutputDir string `json:"output_dir"`
// Baseline comparison (matching structural tier patterns)
BaselineFile string `json:"baseline_file,omitempty"` // Path to baseline JSON (optional)
MaxRegressionPercent float64 `json:"max_regression_percent"` // Default 20%
// Structural tier config (rules-only, no ML dependencies)
ExpectedEmbeddings int `json:"expected_embeddings"` // 0 for structural variant
ExpectedClusters int `json:"expected_clusters"` // 0 for structural variant
MinRulesEvaluated int `json:"min_rules_evaluated"` // Min rules evaluated for structural
MinRuleFirings int `json:"min_rule_firings"` // Min rule firings (conditions met)
MinActionsDispatched int `json:"min_actions_dispatched"` // Min actions dispatched
}
TieredConfig contains configuration for tiered E2E tests
func DefaultTieredConfig ¶
func DefaultTieredConfig() *TieredConfig
DefaultTieredConfig returns default configuration
type TieredResults ¶
type TieredResults struct {
// Variant information
Variant VariantResults `json:"variant"`
// Entity validation results
Entities EntityResults `json:"entities"`
// Index population results
Indexes IndexResults `json:"indexes"`
// Search quality results
Search SearchResults `json:"search"`
// Rule evaluation results
Rules RuleResults `json:"rules"`
// Community detection results (statistical/semantic only)
Communities *CommunityResults `json:"communities,omitempty"`
// Anomaly detection results (semantic only - uses k-core and pivot indexes)
Anomalies *AnomalyResults `json:"anomalies,omitempty"`
// PathRAG graph traversal results (Tier 0 - runs on all tiers)
// Two PathRAG tests: sensor (structured IoT) and document (text-rich)
PathRAGSensor *PathRAGResults `json:"pathrag_sensor,omitempty"`
PathRAGDocument *PathRAGResults `json:"pathrag_document,omitempty"`
// Structural index results (k-core, pivot)
StructuralIndexes *StructuralIndexResults `json:"structural_indexes,omitempty"`
// GraphRAG query results (Tier 2 - semantic only)
GraphRAG *GraphRAGResults `json:"graphrag,omitempty"`
// Component health results
Components ComponentResults `json:"components"`
// Output verification results
Outputs OutputResults `json:"outputs"`
// Embedding queue metrics (Phase 4 - statistical/semantic only)
Embeddings *EmbeddingMetrics `json:"embeddings,omitempty"`
// Hierarchy inference results (Phase 4 - statistical/semantic only)
Hierarchy *HierarchyResults `json:"hierarchy,omitempty"`
// Timing information
Timing TimingResults `json:"timing"`
// Test metadata
Metadata TestMetadata `json:"metadata"`
}
TieredResults contains structured results from a tiered e2e test run. This replaces the flat result.Details and result.Metrics maps with typed structures.
func BuildTieredResults ¶
func BuildTieredResults(result *Result, searchStats *search.Stats) *TieredResults
BuildTieredResults creates a TieredResults from legacy Result data and search stats. This provides the bridge between the old flat format and new structured format.
func LoadStructuredResults ¶
func LoadStructuredResults(path string) (*TieredResults, error)
LoadStructuredResults reads structured results from a JSON file.
type TieredScenario ¶
type TieredScenario struct {
// contains filtered or unexported fields
}
TieredScenario validates comprehensive semantic processing
func NewTieredScenario ¶
func NewTieredScenario( obsClient *client.ObservabilityClient, udpAddr string, cfg *TieredConfig, ) *TieredScenario
NewTieredScenario creates a new tiered semantic test scenario
func (*TieredScenario) Description ¶
func (s *TieredScenario) Description() string
Description returns the scenario description
func (*TieredScenario) Execute ¶
func (s *TieredScenario) Execute(ctx context.Context) (*Result, error)
Execute runs the tiered semantic test scenario
type TimingResults ¶
type TimingResults struct {
// TotalDurationMs is the total test duration
TotalDurationMs int64 `json:"total_duration_ms"`
// StageDurations maps stage name to duration in ms
StageDurations map[string]int64 `json:"stage_durations"`
}
TimingResults contains timing information for each stage.
type VariantResults ¶
type VariantResults struct {
// Name is the variant: "structural", "statistical", "semantic"
Name string `json:"name"`
// EmbeddingProvider: "disabled", "bm25", "http"
EmbeddingProvider string `json:"embedding_provider"`
// SemembedAvailable indicates if external embedding service is available
SemembedAvailable bool `json:"semembed_available"`
}
VariantResults contains variant detection information.
type VirtualEdgeResults ¶
type VirtualEdgeResults struct {
// Total is the total number of virtual edges created
Total int `json:"total"`
// High is count of edges with similarity >= 0.9
High int `json:"high"`
// Medium is count of edges with similarity >= 0.85
Medium int `json:"medium"`
// Related is count of edges with similarity >= 0.8
Related int `json:"related"`
// AutoApplied is count of anomalies with auto_applied status
AutoApplied int `json:"auto_applied"`
}
VirtualEdgeResults contains virtual edge creation metrics from semantic inference.
Source Files
¶
Directories
¶
| Path | Synopsis |
|---|---|
|
Package agentic provides the agentic E2E test scenario.
|
Package agentic provides the agentic E2E test scenario. |
|
Package anomaly provides ground truth validation for anomaly detection.
|
Package anomaly provides ground truth validation for anomaly detection. |
|
Package boids provides the Boids coordination A/B test scenario.
|
Package boids provides the Boids coordination A/B test scenario. |
|
Package community provides ground truth validation for community detection.
|
Package community provides ground truth validation for community detection. |
|
Package search provides unified search query execution for e2e tests.
|
Package search provides unified search query execution for e2e tests. |
|
Package stages contains extracted stage implementations for tiered E2E tests
|
Package stages contains extracted stage implementations for tiered E2E tests |
|
Package throughput provides a high-throughput E2E scenario for performance profiling.
|
Package throughput provides a high-throughput E2E scenario for performance profiling. |