Documentation
¶
Overview ¶
Package reporting provides a comprehensive reporting system for LLM test results.
Index ¶
- Constants
- Variables
- func CreateFormatter(format common.ReportFormat, options map[string]interface{}) (common.ReportFormatter, error)
- type BatchReportingService
- type CSVRenderer
- type CategoryCount
- type ComplianceFramework
- type ComplianceMapping
- type ComplianceMappingProvider
- type ComplianceStatus
- type CustomComplianceProvider
- func (p *CustomComplianceProvider) AddMapping(id string, name string, description string, url string)
- func (p *CustomComplianceProvider) GetAllMappings() map[string]ComplianceMapping
- func (p *CustomComplianceProvider) GetFrameworks() []ComplianceFramework
- func (p *CustomComplianceProvider) GetMapping(id string) (ComplianceMapping, error)
- func (p *CustomComplianceProvider) GetMappings(ctx context.Context, testResult *TestResult) ([]ComplianceMapping, error)
- func (p *CustomComplianceProvider) RemoveMapping(id string)
- type DateRange
- type DefaultReportGenerator
- func (g *DefaultReportGenerator) GenerateReport(ctx context.Context, testSuites []*TestSuite, options *ReportOptions) (*Report, error)
- func (g *DefaultReportGenerator) GetFormatter(format common.ReportFormat) (common.ReportFormatter, bool)
- func (g *DefaultReportGenerator) ListFormats() []common.ReportFormat
- func (g *DefaultReportGenerator) RegisterComplianceProvider(provider ComplianceMappingProvider)
- func (g *DefaultReportGenerator) RegisterFormatter(formatter common.ReportFormatter)
- type Evidence
- type FilterFunc
- type Finding
- type HTMLRenderer
- type ISOComplianceProvider
- type JSONRenderer
- type MarkdownRenderer
- type MultiFormatRenderer
- type OWASPComplianceProvider
- type Reference
- type Remediation
- type RenderOptions
- type Renderer
- type Report
- type ReportBuilder
- type ReportFilter
- type ReportFormat
- type ReportGenerator
- type ReportMetadata
- type ReportOptions
- type ReportStatistics
- type ReportSummary
- type SecurityReport
- type SecurityReportSummary
- type SeverityLevel
- type SortFunc
- type StreamingRenderer
- type TemplateReportingService
- type TemplateResultConverter
- type TestResult
- type TestStatus
- type TestSuite
Constants ¶
const ( TextFormat = common.TextFormat MarkdownFormat = common.MarkdownFormat JSONFormat = common.JSONFormat JSONLFormat = common.JSONLFormat CSVFormat = common.CSVFormat ExcelFormat = common.ExcelFormat PDFFormat = common.PDFFormat HTMLFormat = common.HTMLFormat )
Supported report formats
const ( CriticalSeverity = common.Critical HighSeverity = common.High MediumSeverity = common.Medium LowSeverity = common.Low InfoSeverity = common.Info )
Supported severity levels
Variables ¶
var SeverityLevelMapping = map[string]common.SeverityLevel{ "critical": common.Critical, "high": common.High, "medium": common.Medium, "low": common.Low, "info": common.Info, }
SeverityLevelMapping maps string representations to SeverityLevel constants
Functions ¶
func CreateFormatter ¶
func CreateFormatter(format common.ReportFormat, options map[string]interface{}) (common.ReportFormatter, error)
CreateFormatter creates a formatter for the specified format
Types ¶
type BatchReportingService ¶
type BatchReportingService struct {
// contains filtered or unexported fields
}
BatchReportingService provides reporting services for multiple test suites
func NewBatchReportingService ¶
func NewBatchReportingService(generator common.ReportGenerator) *BatchReportingService
NewBatchReportingService creates a new batch reporting service
func (*BatchReportingService) GenerateReport ¶
func (s *BatchReportingService) GenerateReport(ctx context.Context, suites []*TestSuite, options *ReportOptions) ([]byte, error)
GenerateReport generates a report from multiple test suites
type CSVRenderer ¶
type CSVRenderer struct{}
CSVRenderer renders reports as CSV
func (*CSVRenderer) GetContentType ¶
func (r *CSVRenderer) GetContentType() string
GetContentType returns the content type for CSV
func (*CSVRenderer) GetFileExtension ¶
func (r *CSVRenderer) GetFileExtension() string
GetFileExtension returns the file extension for CSV
func (*CSVRenderer) Render ¶
func (r *CSVRenderer) Render(report *SecurityReport, options RenderOptions) ([]byte, error)
Render renders a report as CSV
type CategoryCount ¶
type ComplianceFramework ¶
type ComplianceFramework string
ComplianceFramework represents a compliance framework
const ( OWASPFramework ComplianceFramework = "owasp-top-10-llm" ISOFramework ComplianceFramework = "iso-iec-42001" NISTFramework ComplianceFramework = "nist-ai-risk" CustomFramework ComplianceFramework = "custom" )
Supported compliance frameworks
type ComplianceMapping ¶
type ComplianceMapping struct {
// Framework is the compliance framework
Framework ComplianceFramework `json:"framework"`
// ID is the ID of the item within the framework
ID string `json:"id"`
// Name is the name of the item within the framework
Name string `json:"name"`
// Description is the description of the item within the framework
Description string `json:"description,omitempty"`
// URL is the URL to the item within the framework
URL string `json:"url,omitempty"`
}
ComplianceMapping represents a mapping to a compliance framework
type ComplianceMappingProvider ¶
type ComplianceMappingProvider interface {
// GetMappings returns compliance mappings for a test result
GetMappings(ctx context.Context, testResult *TestResult) ([]ComplianceMapping, error)
// GetFrameworks returns a list of supported compliance frameworks
GetFrameworks() []ComplianceFramework
}
ComplianceMappingProvider is the interface for compliance mapping providers
type ComplianceStatus ¶
type CustomComplianceProvider ¶
type CustomComplianceProvider struct {
// contains filtered or unexported fields
}
CustomComplianceProvider provides custom compliance mappings
func NewCustomComplianceProvider ¶
func NewCustomComplianceProvider(mappings map[string]ComplianceMapping) *CustomComplianceProvider
NewCustomComplianceProvider creates a new custom compliance provider
func (*CustomComplianceProvider) AddMapping ¶
func (p *CustomComplianceProvider) AddMapping(id string, name string, description string, url string)
AddMapping adds a mapping to the provider
func (*CustomComplianceProvider) GetAllMappings ¶
func (p *CustomComplianceProvider) GetAllMappings() map[string]ComplianceMapping
GetAllMappings gets all mappings from the provider
func (*CustomComplianceProvider) GetFrameworks ¶
func (p *CustomComplianceProvider) GetFrameworks() []ComplianceFramework
GetFrameworks returns a list of supported compliance frameworks
func (*CustomComplianceProvider) GetMapping ¶
func (p *CustomComplianceProvider) GetMapping(id string) (ComplianceMapping, error)
GetMapping gets a mapping from the provider
func (*CustomComplianceProvider) GetMappings ¶
func (p *CustomComplianceProvider) GetMappings(ctx context.Context, testResult *TestResult) ([]ComplianceMapping, error)
GetMappings returns compliance mappings for a test result
func (*CustomComplianceProvider) RemoveMapping ¶
func (p *CustomComplianceProvider) RemoveMapping(id string)
RemoveMapping removes a mapping from the provider
type DefaultReportGenerator ¶
type DefaultReportGenerator struct {
// contains filtered or unexported fields
}
DefaultReportGenerator is the default implementation of ReportGenerator
func CreateDefaultReportGenerator ¶
func CreateDefaultReportGenerator() (*DefaultReportGenerator, error)
CreateDefaultReportGenerator creates a default report generator with all formatters registered
func NewReportGenerator ¶
func NewReportGenerator() *DefaultReportGenerator
NewReportGenerator creates a new report generator
func (*DefaultReportGenerator) GenerateReport ¶
func (g *DefaultReportGenerator) GenerateReport(ctx context.Context, testSuites []*TestSuite, options *ReportOptions) (*Report, error)
GenerateReport generates a report from test results
func (*DefaultReportGenerator) GetFormatter ¶
func (g *DefaultReportGenerator) GetFormatter(format common.ReportFormat) (common.ReportFormatter, bool)
GetFormatter returns a formatter for a specific format
func (*DefaultReportGenerator) ListFormats ¶
func (g *DefaultReportGenerator) ListFormats() []common.ReportFormat
ListFormats returns a list of supported formats
func (*DefaultReportGenerator) RegisterComplianceProvider ¶
func (g *DefaultReportGenerator) RegisterComplianceProvider(provider ComplianceMappingProvider)
RegisterComplianceProvider registers a compliance mapping provider
func (*DefaultReportGenerator) RegisterFormatter ¶
func (g *DefaultReportGenerator) RegisterFormatter(formatter common.ReportFormatter)
RegisterFormatter registers a formatter for a specific format
type FilterFunc ¶
type FilterFunc func(result *TestResult) bool
FilterFunc is a function that filters test results
type Finding ¶
type Finding struct {
ID string `json:"id"`
Timestamp time.Time `json:"timestamp"`
Category string `json:"category"`
Subcategory string `json:"subcategory"`
Severity string `json:"severity"`
Confidence float64 `json:"confidence"`
Title string `json:"title"`
Description string `json:"description"`
Remediation Remediation `json:"remediation"`
OWASPMapping []string `json:"owasp_mapping"`
References []Reference `json:"references"`
Evidence Evidence `json:"evidence"`
Tags []string `json:"tags"`
Metadata map[string]interface{} `json:"metadata,omitempty"`
}
type HTMLRenderer ¶
type HTMLRenderer struct {
// contains filtered or unexported fields
}
HTMLRenderer renders reports as HTML
func (*HTMLRenderer) GetContentType ¶
func (r *HTMLRenderer) GetContentType() string
GetContentType returns the content type for HTML
func (*HTMLRenderer) GetFileExtension ¶
func (r *HTMLRenderer) GetFileExtension() string
GetFileExtension returns the file extension for HTML
func (*HTMLRenderer) Render ¶
func (r *HTMLRenderer) Render(report *SecurityReport, options RenderOptions) ([]byte, error)
Render renders a report as HTML
type ISOComplianceProvider ¶
type ISOComplianceProvider struct {
// contains filtered or unexported fields
}
ISOComplianceProvider provides compliance mappings for ISO/IEC 42001
func NewISOComplianceProvider ¶
func NewISOComplianceProvider() *ISOComplianceProvider
NewISOComplianceProvider creates a new ISO compliance provider
func (*ISOComplianceProvider) GetFrameworks ¶
func (p *ISOComplianceProvider) GetFrameworks() []ComplianceFramework
GetFrameworks returns a list of supported compliance frameworks
func (*ISOComplianceProvider) GetMappings ¶
func (p *ISOComplianceProvider) GetMappings(ctx context.Context, testResult *TestResult) ([]ComplianceMapping, error)
GetMappings returns compliance mappings for a test result
type JSONRenderer ¶
type JSONRenderer struct{}
JSONRenderer renders reports as JSON
func (*JSONRenderer) GetContentType ¶
func (r *JSONRenderer) GetContentType() string
GetContentType returns the content type for JSON
func (*JSONRenderer) GetFileExtension ¶
func (r *JSONRenderer) GetFileExtension() string
GetFileExtension returns the file extension for JSON
func (*JSONRenderer) Render ¶
func (r *JSONRenderer) Render(report *SecurityReport, options RenderOptions) ([]byte, error)
Render renders a report as JSON
type MarkdownRenderer ¶
type MarkdownRenderer struct{}
MarkdownRenderer renders reports as Markdown
func (*MarkdownRenderer) GetContentType ¶
func (r *MarkdownRenderer) GetContentType() string
GetContentType returns the content type for Markdown
func (*MarkdownRenderer) GetFileExtension ¶
func (r *MarkdownRenderer) GetFileExtension() string
GetFileExtension returns the file extension for Markdown
func (*MarkdownRenderer) Render ¶
func (r *MarkdownRenderer) Render(report *SecurityReport, options RenderOptions) ([]byte, error)
Render renders a report as Markdown
type MultiFormatRenderer ¶
type MultiFormatRenderer struct {
// contains filtered or unexported fields
}
MultiFormatRenderer handles rendering reports in multiple formats
func NewMultiFormatRenderer ¶
func NewMultiFormatRenderer() *MultiFormatRenderer
NewMultiFormatRenderer creates a new multi-format renderer
func (*MultiFormatRenderer) RegisterRenderer ¶
func (r *MultiFormatRenderer) RegisterRenderer(format common.ReportFormat, renderer Renderer)
RegisterRenderer registers a renderer for a format
func (*MultiFormatRenderer) Render ¶
func (r *MultiFormatRenderer) Render(report *SecurityReport, format common.ReportFormat, options RenderOptions) ([]byte, error)
Render renders a report in the specified format
type OWASPComplianceProvider ¶
type OWASPComplianceProvider struct {
// contains filtered or unexported fields
}
OWASPComplianceProvider provides compliance mappings for OWASP Top 10 for LLMs
func NewOWASPComplianceProvider ¶
func NewOWASPComplianceProvider() *OWASPComplianceProvider
NewOWASPComplianceProvider creates a new OWASP compliance provider
func (*OWASPComplianceProvider) GetFrameworks ¶
func (p *OWASPComplianceProvider) GetFrameworks() []ComplianceFramework
GetFrameworks returns a list of supported compliance frameworks
func (*OWASPComplianceProvider) GetMappings ¶
func (p *OWASPComplianceProvider) GetMappings(ctx context.Context, testResult *TestResult) ([]ComplianceMapping, error)
GetMappings returns compliance mappings for a test result
type Remediation ¶
type RenderOptions ¶
type RenderOptions struct {
Template string `json:"template"`
Locale string `json:"locale"`
TimeZone string `json:"timezone"`
IncludeRawData bool `json:"includeRawData"`
Filters ReportFilter `json:"filters"`
CustomFields map[string]interface{} `json:"customFields"`
}
RenderOptions contains options for rendering
type Renderer ¶
type Renderer interface {
Render(report *SecurityReport, options RenderOptions) ([]byte, error)
GetContentType() string
GetFileExtension() string
}
Renderer interface for format-specific renderers
type Report ¶
type Report struct {
// ID is the unique identifier for the report
ID string `json:"id"`
// Title is the title of the report
Title string `json:"title"`
// Description is the description of the report
Description string `json:"description,omitempty"`
// GeneratedAt is the time the report was generated
GeneratedAt time.Time `json:"generated_at"`
// Summary is the summary of the report
Summary ReportSummary `json:"summary"`
// TestSuites is the list of test suites
TestSuites []*TestSuite `json:"test_suites"`
// Metadata is additional metadata for the report
Metadata map[string]interface{} `json:"metadata,omitempty"`
}
Report represents a complete test report
type ReportBuilder ¶
type ReportBuilder struct {
// contains filtered or unexported fields
}
ReportBuilder builds reports with various options
func NewReportBuilder ¶
func NewReportBuilder() *ReportBuilder
NewReportBuilder creates a new report builder
func (*ReportBuilder) AddFilter ¶
func (rb *ReportBuilder) AddFilter(filter func(Finding) bool) *ReportBuilder
AddFilter adds a filter function
func (*ReportBuilder) AddFinding ¶
func (rb *ReportBuilder) AddFinding(finding Finding) *ReportBuilder
AddFinding adds a finding to the report
func (*ReportBuilder) Build ¶
func (rb *ReportBuilder) Build() *SecurityReport
Build builds the final report
func (*ReportBuilder) WithMetadata ¶
func (rb *ReportBuilder) WithMetadata(metadata ReportMetadata) *ReportBuilder
WithMetadata sets report metadata
type ReportFilter ¶
type ReportFilter struct {
Severity []string `json:"severity"`
Categories []string `json:"categories"`
DateRange *DateRange `json:"dateRange"`
Tags []string `json:"tags"`
Custom map[string]interface{} `json:"custom"`
}
ReportFilter defines filtering options
type ReportFormat ¶
type ReportFormat = common.ReportFormat
ReportFormat represents the format of a report
type ReportGenerator ¶
type ReportGenerator interface {
// GenerateReport generates a report from test results
GenerateReport(ctx context.Context, testSuites []*TestSuite, options *ReportOptions) (*Report, error)
// RegisterFormatter registers a formatter for a specific format
RegisterFormatter(formatter common.ReportFormatter)
// GetFormatter returns a formatter for a specific format
GetFormatter(format common.ReportFormat) (common.ReportFormatter, bool)
// ListFormats returns a list of supported formats
ListFormats() []common.ReportFormat
}
ReportGenerator interface defines methods for report generation
type ReportMetadata ¶
type ReportOptions ¶
type ReportOptions struct {
// Format is the format of the report
Format common.ReportFormat `json:"format"`
// Title is the title of the report
Title string `json:"title"`
// Description is the description of the report
Description string `json:"description,omitempty"`
// IncludePassedTests indicates whether to include passed tests
IncludePassedTests bool `json:"include_passed_tests"`
// IncludeSkippedTests indicates whether to include skipped tests
IncludeSkippedTests bool `json:"include_skipped_tests"`
// IncludePendingTests indicates whether to include pending tests
IncludePendingTests bool `json:"include_pending_tests"`
// MinimumSeverity is the minimum severity level to include
MinimumSeverity common.SeverityLevel `json:"minimum_severity"`
// IncludeTags is the list of tags to include
IncludeTags []string `json:"include_tags,omitempty"`
// ExcludeTags is the list of tags to exclude
ExcludeTags []string `json:"exclude_tags,omitempty"`
// SortBy is the field to sort by
SortBy string `json:"sort_by,omitempty"`
// SortOrder is the sort order (asc or desc)
SortOrder string `json:"sort_order,omitempty"`
// TemplatePath is the path to a custom template
TemplatePath string `json:"template_path,omitempty"`
// OutputPath is the path to write the report to
OutputPath string `json:"output_path,omitempty"`
// Metadata is additional metadata for the report
Metadata map[string]interface{} `json:"metadata,omitempty"`
}
ReportOptions represents options for generating a report
type ReportStatistics ¶
type ReportSummary ¶
type ReportSummary struct {
// TotalTests is the total number of tests
TotalTests int `json:"total_tests"`
// PassedTests is the number of passed tests
PassedTests int `json:"passed_tests"`
// FailedTests is the number of failed tests
FailedTests int `json:"failed_tests"`
// ErrorTests is the number of tests with errors
ErrorTests int `json:"error_tests"`
// SkippedTests is the number of skipped tests
SkippedTests int `json:"skipped_tests"`
// PendingTests is the number of pending tests
PendingTests int `json:"pending_tests"`
// PassRate is the pass rate as a percentage
PassRate float64 `json:"pass_rate"`
// AverageScore is the average score of all tests
AverageScore float64 `json:"average_score"`
// AverageDuration is the average duration of all tests
AverageDuration time.Duration `json:"average_duration"`
// TotalDuration is the total duration of all tests
TotalDuration time.Duration `json:"total_duration"`
// SeverityCounts is the count of tests by severity
SeverityCounts map[common.SeverityLevel]int `json:"severity_counts"`
// TagCounts is the count of tests by tag
TagCounts map[string]int `json:"tag_counts"`
}
ReportSummary represents a summary of a report
type SecurityReport ¶
type SecurityReport struct {
Metadata ReportMetadata `json:"metadata"`
Summary SecurityReportSummary `json:"summary"`
Findings []Finding `json:"findings"`
Statistics ReportStatistics `json:"statistics"`
Compliance map[string]*ComplianceStatus `json:"compliance,omitempty"`
}
Add type definitions that are missing
type SecurityReportSummary ¶
type SeverityLevel ¶
type SeverityLevel = common.SeverityLevel
SeverityLevel represents the severity level of a test
type SortFunc ¶
type SortFunc func(results []*TestResult) []*TestResult
SortFunc is a function that sorts test results
type StreamingRenderer ¶
type StreamingRenderer interface {
StartRender(w io.Writer, metadata ReportMetadata, options RenderOptions) error
RenderFinding(w io.Writer, finding Finding) error
FinishRender(w io.Writer, summary SecurityReportSummary) error
}
StreamingRenderer supports streaming large reports
type TemplateReportingService ¶
type TemplateReportingService struct {
// contains filtered or unexported fields
}
TemplateReportingService provides reporting services for template results
func NewTemplateReportingService ¶
func NewTemplateReportingService(converter *TemplateResultConverter, generator common.ReportGenerator) *TemplateReportingService
NewTemplateReportingService creates a new template reporting service
func (*TemplateReportingService) GenerateReport ¶
func (s *TemplateReportingService) GenerateReport(ctx context.Context, results []*types.TemplateResult, options *ReportOptions) ([]byte, error)
GenerateReport generates a report from template results
type TemplateResultConverter ¶
type TemplateResultConverter struct {
// contains filtered or unexported fields
}
TemplateResultConverter converts template results to test results
func NewTemplateResultConverter ¶
func NewTemplateResultConverter(complianceProviders []ComplianceMappingProvider) *TemplateResultConverter
NewTemplateResultConverter creates a new template result converter
func (*TemplateResultConverter) ConvertToTestSuite ¶
func (c *TemplateResultConverter) ConvertToTestSuite(ctx context.Context, results []*types.TemplateResult, suiteID string, suiteName string) (*TestSuite, error)
ConvertToTestSuite converts template results to a test suite
type TestResult ¶
type TestResult struct {
// ID is the unique identifier for the test result
ID string `json:"id"`
// Name is the name of the test
Name string `json:"name"`
// Description is the description of the test
Description string `json:"description,omitempty"`
// Status is the status of the test
Status TestStatus `json:"status"`
// Severity is the severity level of the test
Severity common.SeverityLevel `json:"severity"`
// Score is the score of the test (0-100)
Score int `json:"score"`
// StartTime is the time the test started
StartTime time.Time `json:"start_time"`
// EndTime is the time the test ended
EndTime time.Time `json:"end_time"`
// Duration is the duration of the test
Duration time.Duration `json:"duration"`
// Error is any error that occurred during the test
Error string `json:"error,omitempty"`
// Input is the input to the test
Input string `json:"input,omitempty"`
// ExpectedOutput is the expected output of the test
ExpectedOutput string `json:"expected_output,omitempty"`
// ActualOutput is the actual output of the test
ActualOutput string `json:"actual_output,omitempty"`
// ComplianceMappings is the list of compliance mappings for the test
ComplianceMappings []ComplianceMapping `json:"compliance_mappings,omitempty"`
// Tags is the list of tags for the test
Tags []string `json:"tags,omitempty"`
// Metadata is additional metadata for the test
Metadata map[string]interface{} `json:"metadata,omitempty"`
}
TestResult represents a single test result
type TestStatus ¶
type TestStatus string
TestStatus represents the status of a test
const ( PassStatus TestStatus = "pass" FailStatus TestStatus = "fail" ErrorStatus TestStatus = "error" SkippedStatus TestStatus = "skipped" PendingStatus TestStatus = "pending" )
Supported test statuses
type TestSuite ¶
type TestSuite struct {
// ID is the unique identifier for the test suite
ID string `json:"id"`
// Name is the name of the test suite
Name string `json:"name"`
// Description is the description of the test suite
Description string `json:"description,omitempty"`
// StartTime is the time the test suite started
StartTime time.Time `json:"start_time"`
// EndTime is the time the test suite ended
EndTime time.Time `json:"end_time"`
// Duration is the duration of the test suite
Duration time.Duration `json:"duration"`
// Results is the list of test results
Results []*TestResult `json:"results"`
// Metadata is additional metadata for the test suite
Metadata map[string]interface{} `json:"metadata,omitempty"`
}
TestSuite represents a collection of test results
Source Files
¶
Directories
¶
| Path | Synopsis |
|---|---|
|
Package api provides common types and interfaces for the reporting system
|
Package api provides common types and interfaces for the reporting system |
|
Package common provides common types and interfaces for the reporting system
|
Package common provides common types and interfaces for the reporting system |
|
backup
Package common provides common types and interfaces for the reporting system
|
Package common provides common types and interfaces for the reporting system |
|
consolidated
Package common provides common types and interfaces for the reporting system
|
Package common provides common types and interfaces for the reporting system |
|
types
Package types provides common types for the reporting formatters
|
Package types provides common types for the reporting formatters |
|
Package interfaces provides common interfaces for the reporting system
|
Package interfaces provides common interfaces for the reporting system |
|
Package types provides common types for the reporting system
|
Package types provides common types for the reporting system |