types

Layer 8 Types (L8Types)

A foundational library for Layer 8 distributed networking systems, providing Protocol Buffer-based type definitions, Go interfaces, and utilities for building distributed applications with service discovery, health monitoring, and secure communication.

🚀 Recent Updates

Latest Features (v1.5.0)

• Enhanced Security Framework: Comprehensive security enhancements
  • Two-Factor Authentication (TFA) with TFASetup() and TFAVerify() methods
  • Captcha support for bot protection via Captcha() method
  • User registration with Register() method
  • Credential fetching with Credential() method for secure credential management
• Advanced SLA Decorators: New Service Level Agreement capabilities
  • Non-Unique Keys support with SetNonUniqueKeys() for flexible indexing
  • Always Overwrite decorator with SetAlwaysOverwrite() for conflict resolution
  • Metadata functions for custom service metadata handling
• Local Communication: Same-VNic messaging optimization
  • Local() and LocalRequest() methods for intra-VNic communication
  • Reduced latency for co-located services
• Service Callbacks: Enhanced callback system
  • Continuation indication in IServiceCallback.Before() and After() methods
  • Fine-grained control over service request/response lifecycle
• Web Service Refactoring: Improved web service architecture
  • Cleaner separation of concerns with refactored interfaces
  • Enhanced plugin system with IPlugin interface
  • Bearer token validation with BearerValidator interface
• Apache 2.0 Licensing: All source files now include proper copyright headers

Previous Release (v1.4.0)

• Map-Reduce Framework: Distributed Map-Reduce capabilities for parallel data processing
  • New IMapReduceService interface for distributed computation
  • MapReduce() flag in query system for distributed aggregation
  • Collect() method for data collection and filtering across services
• Leader Election: Leader-based communication patterns
  • Leader() and LeaderRequest() methods in VNic for leader-only messaging
  • Automatic leader election and failover support
• Remote VNet Support: Enhanced multi-network connectivity
  • Added Vnet() field to web services for cross-network operations
  • Improved service discovery across network boundaries

Previous Release (v1.3.0)

• Enhanced Authentication System: Token-based authentication with validation and activation mechanisms
• Improved Security: Added hash-based security functions and error handling for auth operations
• Logging Enhancements: Comprehensive logging system for debugging and monitoring
• Sorting Capabilities: Added flexible sorting mechanisms for query results
• Token Management: Secure token generation, validation, and lifecycle management

Overview

Layer 8 Types serves as the core type system and interface library for Layer 8 networking applications, offering:

• Protocol Buffer Schemas: Comprehensive type definitions for distributed system components
• Virtual Network Interface (VNic): Advanced networking abstractions with leader election and cross-network support
• Service Discovery & Management: Built-in service registration, discovery, and area-based routing
• Distributed Computing: Map-Reduce framework for parallel data processing and aggregation
• Health Monitoring: Real-time system health tracking with Unix top-style output formatting
• Transaction Management: Distributed transaction support with state tracking
• Notification System: Property change notifications and event propagation
• Security Framework: Enhanced authentication with token validation, authorization, and encryption
• Multi-Language Support: Go implementation with Zig bindings

Key Features

Virtual Networking (VNic)

• Multiple Communication Patterns: Unicast, Multicast, Round-robin, Proximity-based, Leader selection, Local
• Local Communication: Same-VNic messaging for co-located services (v1.5.0)
• Leader Election: Automatic leader election with failover support (v1.4.0)
• Remote VNet Support: Cross-network service discovery and communication (v1.4.0)
• Network Mode Support: Native, Docker, and Kubernetes networking modes
• Service API: RESTful service interfaces (POST, PUT, PATCH, DELETE, GET)
• Message Priorities: 8-level priority system (P1-P8) for message handling
• Transaction Support: Distributed transaction state management

Service Discovery & Management

• Service Areas: Logical service grouping and area-based routing
• Health States: Up, Down, Unreachable status tracking with statistics
• Replication: Service replication with endpoint scoring and key-based routing
• Dynamic Service Registration: Runtime service addition and removal
• Map-Reduce Framework: Distributed computation and aggregation across services (v1.4.0)

Type System & Reflection

• Dynamic Type Registry: Runtime type registration and introspection
• RNode Reflection: Advanced reflection system with decorators and table views
• Serialization Framework: Pluggable serialization with multiple format support
• Query Engine: Expression-based querying with comparators and conditions

Security & Configuration

• Token Authentication: Secure token-based authentication with validation and activation
• Two-Factor Authentication: TFA setup and verification for enhanced security (v1.5.0)
• Captcha Protection: Bot protection with captcha challenge/response (v1.5.0)
• User Registration: Secure user registration workflow (v1.5.0)
• Credential Management: Secure credential fetching and handling (v1.5.0)
• AES Encryption: Built-in symmetric encryption for secure communication
• Hash Functions: Cryptographic hash support for data integrity and password security
• System Configuration: Comprehensive configuration management with VNet settings
• Authentication Framework: Enhanced AAA (Authentication, Authorization, Accounting) support
• Access Control: Resource-based security with permission management and scope views
• Bearer Token Validation: HTTP bearer token validation for web services (v1.5.0)

Protocol Buffer Schemas

Core Types

• api.proto: Query expressions, conditions, comparators, and routing tables
• services.proto: Service discovery, areas, replication indices, and transactions
• health.proto: Health monitoring, statistics, and system status tracking
• config.proto: System configuration, VNet settings, and connection parameters
• notification.proto: Property change notifications and event propagation
• web.proto: Web service definitions and plugin system
• reflect.proto: Type reflection, nodes, decorators, and table views

Message Types

• Query: Advanced search with criteria, sorting, pagination, and schema filtering
• Health: Process health with statistics (CPU, memory, message counts)
• Services: Service-to-area mappings and replication management
• SysConfig: Network configuration including VNet ports, UUIDs, and keep-alive settings
• NotificationSet: Batched property change notifications with sequencing

Project Structure

l8types/
├── proto/                      # Protocol Buffer definitions
│   ├── api.proto              # Query system and routing
│   ├── services.proto         # Service discovery and management
│   ├── health.proto           # Health monitoring and statistics
│   ├── config.proto           # System configuration
│   ├── notification.proto     # Event notification system
│   ├── web.proto              # Web service interfaces
│   ├── reflect.proto          # Type reflection system
│   └── make-bindings.sh       # Code generation script
├── go/                        # Go implementation
│   ├── ifs/                   # Core interfaces
│   │   ├── API.go             # Elements, Query, Property interfaces
│   │   ├── Registry.go        # Type registration and reflection
│   │   ├── Resources.go       # Resource management
│   │   ├── VNic.go            # Virtual network interface
│   │   ├── Message.go         # Message structure and operations
│   │   ├── MessageEnums.go    # Priority, Action, Transaction enums
│   │   ├── Security.go        # Security and authentication
│   │   ├── Services.go        # Service management
│   │   ├── Web.go             # Web service interfaces
│   │   └── ...                # Additional interfaces
│   ├── types/                 # Generated Protocol Buffer types
│   ├── nets/                  # Network protocol implementation
│   ├── aes/                   # AES encryption utilities
│   ├── tests/                 # Test suite
│   └── testtypes/             # Test-specific generated types
└── zig/                       # Zig implementation (experimental)
    ├── src/                   # Zig source files
    └── build.zig              # Build configuration

Getting Started

Prerequisites

• Go 1.23.8+: Core implementation language
• Docker: Required for Protocol Buffer code generation
• Protocol Buffers: For schema compilation (handled via Docker)
• Zig (optional): For Zig language bindings

Installation

# Clone the repository
git clone https://github.com/saichler/l8types.git
cd l8types

# Install Go dependencies
cd go && go mod download

Generate Protocol Buffer Bindings

cd proto
./make-bindings.sh

This script uses Docker to generate Go bindings from all Protocol Buffer schemas and organizes them into the appropriate packages.

Running Tests

cd go
go test ./...

API Migration Guide

Breaking Changes in v1.3.0

1. Service Activation Refactor

The Activate() method has been significantly simplified:

Before:

handler, err := service.Activate(serviceName, serviceArea, priority,
    resources, listener, args...)

After:

sla := &ServiceLevelAgreement{}
sla.SetServiceName("my-service")
sla.SetServiceArea(1)
sla.SetArgs(args...)
handler, err := service.Activate(sla, vnic)

2. Primary Keys API Change

Before:

sla.SetPrimaryKeys([]string{"id", "name"})

After:

sla.SetPrimaryKeys("id", "name")  // Variadic parameters

3. Registry Enhancement

New Method:

// Create a new instance of a registered type dynamically
newInstance := registry.NewOf(existingInstance)

4. Web Service Definition Rename

Before:

webServiceDef := sla.WebServiceDef()
sla.SetWebServiceDef(webService)

After:

webService := sla.WebService()
sla.SetWebService(webService)

Usage Examples

Service Registration and VNic Setup

import (
    "github.com/saichler/l8types/go/ifs"
    "github.com/saichler/l8types/go/types"
)

// Create system configuration
config := &types.SysConfig{
    LocalUuid: "node-123",
    VnetPort: 8080,
    MaxDataSize: 1024*1024,
    TxQueueSize: 1000,
    RxQueueSize: 1000,
}

// Add services to configuration
ifs.AddService(config, "user-service", 1)
ifs.AddService(config, "auth-service", 2)

// Create VNic for networking
vnic := NewVNic(config, resources)
vnic.Start()

Service Level Agreement Configuration (v1.3.0+)

// Create Service Level Agreement
sla := &ifs.ServiceLevelAgreement{}
sla.SetServiceName("user-service")
sla.SetServiceArea(1)
sla.SetStateful(true)
sla.SetTransactional(true)
sla.SetReplication(true)
sla.SetReplicationCount(3)
sla.SetPrimaryKeys("userId", "email")  // Variadic parameters
sla.SetArgs(dbConnection, cacheLayer)  // Variadic parameters

// Activate service with SLA
handler, err := services.Activate(sla, vnic)
if err != nil {
    log.Fatal("Failed to activate service:", err)
}

// Dynamic type creation using registry
registry := ifs.NewRegistry()
userType := &User{}
registry.Register(userType)

// Create new instances dynamically
newUser := registry.NewOf(userType).(*User)

Message Communication Patterns

// Unicast to specific destination
err := vnic.Unicast("node-456", "user-service", 1, ifs.POST, userData)

// Round-robin to service providers
destination, err := vnic.RoundRobin("auth-service", 2, ifs.GET, request)

// Request with response
response := vnic.Request("node-789", "data-service", 1, ifs.GET, query)

// Multicast to all service providers
err := vnic.Multicast("notification-service", 1, ifs.Notify, event)

// Leader-based communication (v1.4.0+)
err := vnic.Leader("consensus-service", 1, ifs.POST, proposal)
leaderResponse := vnic.LeaderRequest("state-service", 1, ifs.GET, stateQuery, timeout)

// Local communication - same VNic (v1.5.0+)
err := vnic.Local("local-service", 1, ifs.POST, localData)
localResponse := vnic.LocalRequest("local-service", 1, ifs.GET, query, timeout)

Query System

// Build complex query with expressions
query := &types.Query{
    RootType: "User",
    Properties: []string{"name", "email", "status"},
    Criteria: &types.Expression{
        Condition: &types.Condition{
            Comparator: &types.Comparator{
                Left: "status",
                Oper: "==",
                Right: "active",
            },
        },
        AndOr: "AND",
        Next: &types.Expression{
            Condition: &types.Condition{
                Comparator: &types.Comparator{
                    Left: "age",
                    Oper: ">",
                    Right: "18",
                },
            },
        },
    },
    SortBy: "name",
    Descending: false,
    Limit: 100,
    Page: 1,
    MapReduce: true, // Enable distributed Map-Reduce processing (v1.4.0+)
}

Map-Reduce Operations (v1.4.0+)

// Implement IMapReduceService for distributed computation
type MyMapReduceService struct {
    cache IDistributedCache
}

func (s *MyMapReduceService) Merge(results map[string]IElements) IElements {
    // Merge results from multiple nodes
    merged := NewElements()
    for nodeId, elements := range results {
        // Custom merging logic
        merged.Combine(elements)
    }
    return merged
}

// Use Collect for distributed data filtering and aggregation
results := cache.Collect(func(item interface{}) (bool, interface{}) {
    user := item.(*User)
    if user.Age > 18 && user.Status == "active" {
        return true, user.Summary() // Include in results with transformation
    }
    return false, nil // Exclude from results
})

Health Monitoring

// Create health information
health := &types.Health{
    AUuid: "node-123",
    Alias: "web-server-1",
    Status: types.HealthState_Up,
    Stats: &types.HealthStats{
        CpuUsage: 45.2,
        MemoryUsage: 512*1024*1024, // 512MB
        TxMsgCount: 1500,
        RxMsgCount: 1200,
    },
    StartTime: time.Now().Unix(),
}

// Format for display (Unix top-style output)
topData := &types.Top{
    Healths: map[string]*types.Health{
        "node-123": health,
    },
}
formatted := ifs.FormatTop(topData)
fmt.Println(formatted)

Type Registration and Reflection

// Register custom types
registry := NewRegistry()
registry.Register(&MyCustomType{})

// Create instances dynamically
info, err := registry.Info("MyCustomType")
if err == nil {
    instance, err := info.NewInstance()
}

// Use serialization
serializer := info.Serializer(ifs.PROTOBUF)
data, err := serializer.Serialize(instance)

Two-Factor Authentication (v1.5.0+)

// Setup TFA for a user
qrCode, secret, err := securityProvider.TFASetup(userId, vnic)
if err != nil {
    log.Fatal("TFA setup failed:", err)
}
// Display QR code to user for authenticator app

// Verify TFA code during login
err = securityProvider.TFAVerify(userId, totpCode, sessionId, vnic)
if err != nil {
    log.Fatal("TFA verification failed:", err)
}

// Get captcha for bot protection
captchaImage := securityProvider.Captcha()

// Register new user
err = securityProvider.Register(email, password, captchaResponse, vnic)

Service Callbacks (v1.5.0+)

// Implement IServiceCallback for lifecycle hooks
type MyCallback struct{}

func (c *MyCallback) Before(data interface{}, action ifs.Action, isLocal bool, vnic ifs.IVNic) (interface{}, bool, error) {
    // Pre-process data before service handles it
    // Return modified data, continue flag, and error
    processedData := preProcess(data)
    shouldContinue := true  // Set to false to stop processing
    return processedData, shouldContinue, nil
}

func (c *MyCallback) After(data interface{}, action ifs.Action, isLocal bool, vnic ifs.IVNic) (interface{}, bool, error) {
    // Post-process data after service handles it
    return postProcess(data), true, nil
}

// Use in SLA
sla := ifs.NewServiceLevelAgreement(handler, "my-service", 1, true, &MyCallback{})

Architecture

Network Layers

1. Physical Network: TCP/UDP transport layer
2. VNet Protocol: Custom protocol for message routing and service discovery
3. Service Layer: Business logic with service areas and discovery
4. Application Layer: User applications and services

Message Flow

1. Message Creation: Applications create messages with service targets
2. Routing: VNic determines optimal destination based on service topology
3. Serialization: Messages are serialized using configured serializers
4. Transport: Messages are sent via network protocols
5. Processing: Receiving VNic deserializes and routes to appropriate handlers

Service Discovery

• Service Areas: Logical groupings for service organization
• Health Monitoring: Continuous health checks with statistics
• Load Balancing: Multiple communication patterns for optimal distribution
• Replication: Service replication with intelligent endpoint selection

Security

Encryption

• AES Symmetric Encryption: Built-in encryption for sensitive data
• Secure Message Transport: Optional message-level encryption
• Key Management: Secure key distribution and rotation support

Authentication & Authorization

• AAA Framework: Authentication, Authorization, and Accounting interfaces
• Resource-Based Security: Fine-grained access control
• Service Authentication: Mutual authentication between services

Development

Adding New Protocol Types

1. Define your schema in the appropriate .proto file
2. Run ./make-bindings.sh to generate Go types
3. Implement required interfaces in go/ifs/
4. Add comprehensive tests

Extending VNic Functionality

1. Implement new communication patterns in VNic interface
2. Add corresponding message routing logic
3. Update service discovery mechanisms as needed

Custom Serialization

1. Implement ISerializer interface
2. Register serializer with type registry
3. Configure serialization mode in resources

Testing

The project includes comprehensive test coverage for all major components:

# Quick test with automated coverage report
cd go
./test.sh  # Runs tests and opens coverage report automatically

# Or run tests manually
go test -v -cover ./...

# Generate coverage report
go test -coverprofile=coverage.out ./...
go tool cover -html=coverage.out -o coverage.html

# View coverage report in browser
open coverage.html  # or cover.html

Test Coverage

• Current Coverage: 95% overall test coverage
• Coverage Reports Available:
  • go/coverage.html - Detailed HTML coverage report
  • go/cover.html - Alternative coverage visualization
  • go/coverage.out - Raw coverage data

Test Coverage Areas

• Message Operations: Marshal/unmarshal, cloning, edge cases
• Encryption: AES encryption/decryption with various key sizes
• Network Protocol: Read/write operations, protocol handling
• Type Conversion: String conversion functions and type safety
• Authentication: Token validation, activation, and error handling
• Service Level Agreement: SLA configuration and activation flows
• Registry Operations: Dynamic type creation and registration

Documentation

Web Documentation

The project includes an interactive web-based documentation site (web.html) featuring:

• Interactive architecture visualizations
• Live code examples
• Feature showcase with animations
• Quick start guide
• API documentation links

To view the documentation locally:

# Open the documentation website
open web.html
# Or serve it with a local web server
python3 -m http.server 8000
# Then navigate to http://localhost:8000/web.html

Dependencies

Go Modules

• github.com/google/uuid v1.6.0: UUID generation for node identification
• google.golang.org/protobuf v1.36.10: Protocol Buffer runtime and code generation

Build Tools

• Docker: Protocol Buffer code generation via containerized protoc
• Go 1.23.8+: Core language runtime and build tools

Changelog

Version 1.5.0 (Current)

• Enhanced Security Framework - TFA, Captcha, Registration, and Credential management
• Advanced SLA Decorators - Non-Unique Keys and Always Overwrite support
• Local Communication - Same-VNic messaging with Local() and LocalRequest()
• Service Callbacks - Continuation indication for fine-grained control
• Web Service Refactoring - Improved architecture with plugin and bearer validation
• Apache 2.0 Licensing - Proper copyright headers on all source files

Version 1.4.0

• Map-Reduce Framework - Distributed computation and data aggregation
• Leader Election - Leader-based communication patterns and automatic failover
• Remote VNet Support - Cross-network connectivity and service discovery
• Protocol Enhancements - Improved serialization utilities for services
• Query System Enhancement - Added MapReduce flag for distributed queries

Version 1.3.0

• Service Level Agreement (SLA) Framework - Complete refactoring of service activation
• Dynamic Type Creation - Added NewOf() method to IRegistry
• API Improvements - Variadic parameters for better ergonomics
• Code Quality - 95% test coverage with comprehensive reporting
• Documentation - Interactive web documentation with visualizations

Version 1.2.0

• Enhanced authentication system with token validation
• Improved security with hash-based functions
• Comprehensive logging system
• Flexible sorting mechanisms for queries
• Secure token lifecycle management

Version 1.1.0

• Initial release with core networking features
• Protocol Buffer schemas
• Virtual Network Interface (VNic)
• Service discovery and health monitoring

Contributing

We welcome contributions! Please:

1. Fork the repository
2. Create a feature branch
3. Run tests with ./test.sh to ensure 95%+ coverage
4. Submit a pull request with clear description of changes

License

This project is licensed under the terms specified in the LICENSE file.

---

Note: L8Types is designed for building distributed systems with sophisticated networking requirements. The "Layer 8" concept extends beyond traditional OSI model layers to provide application-level distributed system primitives.