production

Production OAuth Example

This example demonstrates a production-ready OAuth 2.1 setup with all security features enabled.

Features

• ✅ Token encryption at rest (AES-256-GCM)
• ✅ Refresh token rotation with reuse detection (OAuth 2.1)
• ✅ Comprehensive audit logging
• ✅ Multi-layered rate limiting (IP, user, client registration)
• ✅ Provider-side token revocation (Google/GitHub/etc)
• ✅ Secure client registration with access token
• ✅ HTTPS/TLS support
• ✅ Structured JSON logging
• ✅ Health and readiness endpoints
• ✅ Security headers
• ✅ Request logging
• ✅ Configurable timeouts and retention periods

Prerequisites

1. Google OAuth Credentials (see basic example)
2. Encryption Key: Generate and store securely
3. TLS Certificates (for HTTPS)

Setup

1. Secret Management (REQUIRED for Production)

CRITICAL: NEVER use environment variables for production secrets!

Production deployments MUST use a secret management solution. Choose one:

Option 1: HashiCorp Vault (Recommended)

# 1. Generate and store encryption key
vault kv put secret/oauth/encryption-key \
  value="$(openssl rand -base64 32)"

# 2. Store OAuth credentials
vault kv put secret/oauth/google \
  client_id="your-client-id.apps.googleusercontent.com" \
  client_secret="your-client-secret"

# 3. Store registration token
vault kv put secret/oauth/registration \
  token="$(openssl rand -base64 32)"

# 4. Retrieve secrets in your application
export VAULT_ADDR="https://vault.example.com"
export VAULT_TOKEN="your-vault-token"

# Read secrets from Vault
OAUTH_ENCRYPTION_KEY=$(vault kv get -field=value secret/oauth/encryption-key)
GOOGLE_CLIENT_ID=$(vault kv get -field=client_id secret/oauth/google)
GOOGLE_CLIENT_SECRET=$(vault kv get -field=client_secret secret/oauth/google)

See Vault documentation for production setup with AppRole or Kubernetes auth.

Option 2: AWS Secrets Manager

# 1. Generate and store encryption key
aws secretsmanager create-secret \
  --name oauth-encryption-key \
  --secret-string "$(openssl rand -base64 32)"

# 2. Store OAuth credentials
aws secretsmanager create-secret \
  --name oauth-google-credentials \
  --secret-string '{
    "client_id": "your-client-id.apps.googleusercontent.com",
    "client_secret": "your-client-secret"
  }'

# 3. Retrieve in application (use AWS SDK in production)
OAUTH_ENCRYPTION_KEY=$(aws secretsmanager get-secret-value \
  --secret-id oauth-encryption-key \
  --query SecretString --output text)

# Or use IAM roles for EC2/ECS/EKS (recommended)

Example Go code (using AWS SDK v2):

import (
    "context"
    "fmt"
    "time"

    "github.com/aws/aws-sdk-go-v2/aws"
    "github.com/aws/aws-sdk-go-v2/config"
    "github.com/aws/aws-sdk-go-v2/service/secretsmanager"
)

func getEncryptionKey(ctx context.Context) (string, error) {
    // Load AWS configuration (uses IAM roles, environment variables, or shared config)
    cfg, err := config.LoadDefaultConfig(ctx)
    if err != nil {
        return "", fmt.Errorf("failed to load AWS config: %w", err)
    }
    
    svc := secretsmanager.NewFromConfig(cfg)
    
    // Add timeout for secret retrieval
    ctx, cancel := context.WithTimeout(ctx, 10*time.Second)
    defer cancel()
    
    input := &secretsmanager.GetSecretValueInput{
        SecretId: aws.String("oauth-encryption-key"),
        VersionStage: aws.String("AWSCURRENT"), // Explicit version for production
    }
    
    result, err := svc.GetSecretValue(ctx, input)
    if err != nil {
        return "", fmt.Errorf("failed to retrieve secret: %w", err)
    }
    
    // Nil safety check
    if result.SecretString == nil {
        return "", fmt.Errorf("secret value is nil")
    }
    
    return *result.SecretString, nil
}

Option 3: Google Cloud Secret Manager

# 1. Generate and store encryption key
echo -n "$(openssl rand -base64 32)" | \
  gcloud secrets create oauth-encryption-key --data-file=-

# 2. Store OAuth credentials
gcloud secrets create oauth-google-client-id \
  --data-file=- <<< "your-client-id.apps.googleusercontent.com"

gcloud secrets create oauth-google-client-secret \
  --data-file=- <<< "your-client-secret"

# 3. Retrieve in application
OAUTH_ENCRYPTION_KEY=$(gcloud secrets versions access latest \
  --secret=oauth-encryption-key)

# Or use Workload Identity for GKE (recommended)

Option 4: Kubernetes with External Secrets Operator

Best for Kubernetes deployments - syncs secrets from external secret managers:

# Install External Secrets Operator first:
# helm install external-secrets external-secrets/external-secrets

# 1. Configure SecretStore (for Vault)
apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
  name: vault-backend
  namespace: mcp-oauth
spec:
  provider:
    vault:
      server: "https://vault.example.com"
      path: "secret"
      version: "v2"
      auth:
        kubernetes:
          mountPath: "kubernetes"
          role: "mcp-oauth"

---
# 2. Create ExternalSecret
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: oauth-secrets
  namespace: mcp-oauth
spec:
  refreshInterval: 1h  # Trade-off: shorter interval = more API calls but faster sync
                       # Use 5-15m for high-security secrets requiring rapid rotation
  secretStoreRef:
    name: vault-backend
    kind: SecretStore
  target:
    name: oauth-secrets
    creationPolicy: Owner
  data:
  - secretKey: encryption-key
    remoteRef:
      key: oauth/encryption-key
      property: value
  - secretKey: google-client-id
    remoteRef:
      key: oauth/google
      property: client_id
  - secretKey: google-client-secret
    remoteRef:
      key: oauth/google
      property: client_secret

---
# 3. Reference in Deployment
apiVersion: apps/v1
kind: Deployment
metadata:
  name: mcp-server
spec:
  template:
    spec:
      containers:
      - name: mcp-server
        env:
        - name: OAUTH_ENCRYPTION_KEY
          valueFrom:
            secretKeyRef:
              name: oauth-secrets
              key: encryption-key
        - name: GOOGLE_CLIENT_ID
          valueFrom:
            secretKeyRef:
              name: oauth-secrets
              key: google-client-id
        - name: GOOGLE_CLIENT_SECRET
          valueFrom:
            secretKeyRef:
              name: oauth-secrets
              key: google-client-secret

External Secrets Operator Security Best Practices:

• Set refreshInterval based on secret rotation SLA (5-15 minutes for high-security secrets)
• Monitor External Secrets controller logs for sync failures
• Set up alerts for ExternalSecret resource status changes to SecretSyncedError
• Use RBAC to restrict who can create/modify ExternalSecret resources
• Enable audit logging for all secret access in your secret manager
• Test secret rotation procedures regularly to ensure zero-downtime updates

2. Development Only: Environment Variables

⚠️ WARNING: For local development ONLY! NEVER use in production!

For local testing, you can use environment variables, but this is INSECURE for production.

Create a .env file (DO NOT commit this):

# Required
GOOGLE_CLIENT_ID=your-client-id.apps.googleusercontent.com
GOOGLE_CLIENT_SECRET=your-client-secret

# Security (DO NOT use command substitution - generate separately)
# Generate with: openssl rand -base64 32 > /tmp/encryption_key.txt
# Then manually copy the value here and securely delete the temp file
OAUTH_ENCRYPTION_KEY=paste-generated-key-here
OAUTH_REGISTRATION_TOKEN=paste-generated-token-here

# Server
MCP_RESOURCE=http://localhost:8080
LISTEN_ADDR=:8080

# Logging
LOG_LEVEL=debug
LOG_JSON=false

Secure secret generation for .env file:

# Generate secrets securely (avoids shell history exposure)
openssl rand -base64 32 > /tmp/encryption_key.txt
openssl rand -base64 32 > /tmp/registration_token.txt

# Copy values into .env file manually, then securely delete temp files
shred -u /tmp/encryption_key.txt /tmp/registration_token.txt 2>/dev/null || rm -f /tmp/encryption_key.txt /tmp/registration_token.txt

Why environment variables are UNSAFE for production:

• Visible in process listings (ps aux, docker inspect)
• Leaked in error messages, stack traces, and logs
• Exposed in container orchestration metadata (Docker, Kubernetes)
• Not rotatable without restarting the application
• No audit trail of secret access
• No encryption at rest
• Vulnerable to memory dumps and side-channel attacks

Migration path from development to production:

1. Set up a secret manager (Vault, AWS Secrets Manager, etc.)
2. Store all secrets in the secret manager
3. Update application code to read from secret manager
4. Remove .env file and environment variables
5. Rotate all secrets that were previously in environment variables
6. Add secret access monitoring and alerting

Load environment variables (development only):

# WARNING: This exposes secrets in shell history and process listings
# Only use for local development on trusted systems
set +o history  # Disable shell history temporarily
export $(cat .env | xargs)
set -o history  # Re-enable shell history

Running

First, generate go.mod files and build all examples from the repository root:

make build-examples

Development (HTTP)

From examples/production:

# Without TLS (development only)
unset TLS_CERT_FILE TLS_KEY_FILE
go run main.go

Production (HTTPS)

# Generate self-signed cert for testing
openssl req -x509 -newkey rsa:4096 -keyout key.pem -out cert.pem -days 365 -nodes

# Run with TLS
export TLS_CERT_FILE=cert.pem
export TLS_KEY_FILE=key.pem
go run main.go

Usage

1. Register a Client

curl -X POST https://localhost:8443/oauth/register \
  -H "Authorization: Bearer $OAUTH_REGISTRATION_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "client_name": "Production MCP Client",
    "client_type": "confidential",
    "redirect_uris": ["https://client.example.com/callback"],
    "token_endpoint_auth_method": "client_secret_basic",
    "grant_types": ["authorization_code", "refresh_token"],
    "scope": "https://www.googleapis.com/auth/gmail.readonly"
  }'

Response:

{
  "client_id": "generated-client-id",
  "client_secret": "generated-client-secret",
  "client_name": "Production MCP Client",
  ...
}

IMPORTANT: Save the client_secret - it's only shown once!

2. Check Server Health

# Health check
curl https://localhost:8443/health

# Readiness check
curl https://localhost:8443/ready

3. View Metadata

# OAuth Protected Resource Metadata
curl https://localhost:8443/.well-known/oauth-protected-resource

# Authorization Server Metadata
curl https://localhost:8443/.well-known/oauth-authorization-server

4. Authorization Flow

See the basic example for the complete OAuth flow.

Production Security Checklist

Before deploying to production, verify all items:

Secret Management:

• All secrets stored in a secret manager (NOT environment variables)
• Secret rotation policy configured and documented
• Secrets encrypted at rest in the secret manager
• Secrets encrypted in transit (TLS for all connections)
• Secret access restricted by IAM/RBAC with least privilege
• Audit logging enabled for all secret access
• No secrets in version control (verify with git log -p | grep -i 'secret\|password\|key')
• No secrets in container images (verify with docker history)
• No secrets in CI/CD logs or build artifacts

Network Security:

• HTTPS/TLS enabled for all endpoints (no HTTP in production)
• Valid TLS certificates from a trusted CA (not self-signed)
• TLS 1.2 or higher enforced
• Strong cipher suites configured
• HSTS headers enabled
• Firewall rules restrict access to authorized networks

Application Security:

• All security features enabled (rate limiting, audit logging, encryption)
• Rate limits tuned for production traffic patterns
• PKCE enforced for all OAuth flows
• Refresh token rotation enabled
• Token encryption at rest enabled
• Secure session timeouts configured
• Input validation on all endpoints

Monitoring & Alerting:

• Audit logs centralized and monitored (SIEM integration)
• Alerts configured for critical security events:
  • Authorization code reuse detection
  • Refresh token reuse detection
  • Rate limit violations
  • Failed authentication attempts
  • Unusual client registration activity
  • Provider token revocation failures
• Metrics collection enabled (Prometheus/OpenTelemetry)
• Dashboard created for security metrics
• On-call rotation defined for security alerts

Operational Security:

• Principle of least privilege applied to all service accounts
• Regular security updates and patching schedule
• Disaster recovery and backup procedures documented
• Incident response plan created and tested
• Security audit completed
• Penetration testing performed
• Compliance requirements verified (GDPR, SOC2, etc.)

Supply Chain Security:

• Dependency scanning enabled (Dependabot, Renovate, Snyk)
• Container image scanning enabled (Trivy, Grype, Docker Scout)
• SBOM (Software Bill of Materials) generated for releases
• All dependencies from trusted sources only
• Dependency versions pinned with checksums (go.sum verified)
• Regular security updates for all dependencies
• Vulnerability monitoring and alerting configured
• Pre-commit hooks prevent committing secrets (gitleaks, detect-secrets)
• GitHub secret scanning enabled
• CI/CD pipeline includes security scanning steps

Documentation:

• Architecture diagram updated
• Runbook created for common operations
• Security documentation complete
• Secret rotation procedures documented
• Incident response procedures documented

Security Considerations

Encryption Key Management

Encryption keys MUST be managed through a secret manager (see Setup section above).

Key requirements:

• 32 bytes (256 bits) of cryptographically secure random data
• Base64 encoded for storage and transmission
• Never hardcoded in code, config files, or environment variables
• Rotated regularly (recommend every 90 days)
• Backed up securely with the secret manager's backup features

Encryption Key Rotation Procedure

Regular key rotation is critical for security. Follow this zero-downtime procedure:

Step 1: Generate New Key (v2)

# In your secret manager (example: Vault)
vault kv put secret/oauth/encryption-key-v2 \
  value="$(openssl rand -base64 32)"

Step 2: Update Application Configuration Configure the application to support both keys:

• Use v2 for encrypting new tokens
• Keep v1 for decrypting existing tokens

Step 3: Wait for Token Expiry

• Wait for all tokens encrypted with v1 to expire (based on your token TTL)
• Or implement re-encryption: decrypt with v1, re-encrypt with v2
• Monitor logs to verify no v1 decryption attempts

Step 4: Remove Old Key

# After confirming all tokens use v2
vault kv delete secret/oauth/encryption-key-v1

Step 5: Audit and Verify

• Review audit logs for any decryption failures
• Verify all active tokens can be decrypted
• Document rotation in your security log

Rotation Schedule:

• Normal operations: Every 90 days
• After security incident: Immediately
• Staff changes: Within 24 hours if key access was involved
• Compliance requirements: Follow your organization's policy

Registration Token

The registration token protects the client registration endpoint:

# Generate a secure token
openssl rand -base64 32

# Use it in registration requests
curl -H "Authorization: Bearer $OAUTH_REGISTRATION_TOKEN" ...

Share this token ONLY with trusted client developers.

Incident Response: Compromised Secrets

If secrets are leaked or compromised, follow this immediate response procedure:

Immediate Actions (Within 1 Hour)

If Secrets Are Exposed in Logs/Monitoring:

1. Rotate the affected secret immediately in your secret manager
2. Restart all application instances to load the new secret
3. Review audit logs to identify scope of exposure
4. Revoke any OAuth tokens that may have been issued
5. Notify security team and stakeholders

If Secrets Are Committed to Version Control:

1. DO NOT just delete the file - Git history persists forever!
2. Rotate the secret immediately in your secret manager (assume it's public)
3. Use git-filter-repo or BFG Repo-Cleaner to rewrite Git history:

   # Install git-filter-repo
   pip install git-filter-repo
   
   # Remove secret from entire history
   git filter-repo --path .env --invert-paths
   
   # Or use BFG Repo-Cleaner for specific strings
   bfg --replace-text secrets.txt  # File with secret=REMOVED mappings
   

4. Force-push to all branches (coordinate with entire team first)
5. Notify all developers to delete local clones and re-clone
6. Consider the secret permanently compromised - never reuse it

If Secrets Are in Container Images:

1. Delete all affected image tags from registry immediately
2. Rotate all secrets in the image
3. Rebuild images without secrets
4. Scan all layers with docker history to verify secrets removed
5. Update all deployments with new images
6. Revoke any tokens that may have been issued

If Secrets Are in CI/CD Logs:

1. Rotate secrets immediately
2. Delete/redact the CI/CD run logs if possible
3. Configure secret masking in CI/CD system
4. Add pre-commit hooks to prevent future leaks

Prevention Measures

Pre-Commit Hooks:

# Install git-secrets or gitleaks
brew install gitleaks

# Add to .git/hooks/pre-commit
gitleaks protect --staged

GitHub Secret Scanning:

• Enable GitHub Advanced Security if available
• Configure custom secret patterns for your organization
• Set up notifications for secret push protection

Development Best Practices:

• Never use real secrets in development (use dummy values)
• Use .env.example with placeholder values in version control
• Add .env to .gitignore (verify: git check-ignore .env)
• Use secret scanning in CI/CD pipelines

Monitoring and Alerting:

• Monitor secret access patterns in your secret manager
• Alert on unusual access patterns (time, location, volume)
• Log all secret retrievals for audit trail
• Review logs regularly for suspicious activity

Rate Limiting

The production setup uses multiple layers of rate limiting for defense in depth:

1. IP-based Rate Limiting: Prevents DoS attacks from external sources

   rateLimiter := security.NewRateLimiter(10, 20, logger)
   defer rateLimiter.Stop()
   // Pass to server.New as an option:
   //   server.New(..., server.WithRateLimiter(rateLimiter))
   

2. User-based Rate Limiting: Prevents abuse from authenticated users

   userRateLimiter := security.NewRateLimiter(100, 200, logger)
   defer userRateLimiter.Stop()
   // server.New(..., server.WithUserRateLimiter(userRateLimiter))
   

3. Client Registration Rate Limiting: Prevents registration DoS

   clientRegRateLimiter := security.NewClientRegistrationRateLimiter(logger)
   defer clientRegRateLimiter.Stop()
   // server.New(..., server.WithClientRegistrationRateLimiter(clientRegRateLimiter))
   

Why multiple layers?

• IP limiting stops attacks before authentication
• User limiting prevents authenticated abuse
• Client registration limiting prevents registration/deletion cycle DoS

Adjust based on your threat model and traffic:

// More permissive (high-traffic production)
rateLimiter := security.NewRateLimiter(100, 200, logger)

// More restrictive (sensitive environments)
rateLimiter := security.NewRateLimiter(5, 10, logger)

Monitor rate limit violations in logs:

{
  "level": "warn",
  "msg": "Rate limit exceeded",
  "ip": "203.0.113.42",
  "user": "user@example.com"
}

Audit Logging

All security events are logged:

{
  "level": "info",
  "msg": "Token issued",
  "client_id": "abc123",
  "user_email_hash": "sha256:...",
  "scopes": "gmail.readonly",
  "timestamp": "2025-11-23T10:30:00Z"
}

Sensitive data (tokens, emails) is hashed before logging.

Monitoring

Structured Logs

Use JSON logs for easy parsing:

# Filter by level
cat app.log | jq 'select(.level == "error")'

# Find rate limit violations
cat app.log | jq 'select(.msg == "Rate limit exceeded")'

# Track user activity
cat app.log | jq 'select(.user_email_hash != null)'

Metrics

Enable Prometheus metrics:

export ENABLE_METRICS=true
curl https://localhost:8443/metrics

Alerts

Set up alerts for:

• High error rates
• Rate limit violations (IP, user, or client registration)
• Token reuse detection (CRITICAL - indicates attack!)
• Authorization code reuse (CRITICAL - indicates attack!)
• Provider revocation failures (tokens remain valid at provider)
• Failed authentication attempts
• Unusual client registration activity

Critical Security Alerts:

{
  "level": "error",
  "event_type": "authorization_code_reuse_detected",
  "severity": "critical",
  "action": "all_tokens_revoked",
  "user_id": "user@example.com",
  "client_id": "abc123"
}

These indicate potential token theft attacks and should trigger immediate investigation.

Deployment

Docker

CRITICAL: Never bake secrets into Docker images!

Secure multi-stage build example:

# Build stage
FROM golang:1.24-alpine AS builder
WORKDIR /app
COPY go.mod go.sum ./
RUN go mod download
COPY . .
RUN CGO_ENABLED=0 GOOS=linux go build -a -installsuffix cgo -o mcp-server .

# Runtime stage
FROM alpine:latest

# Security: Run as non-root user
RUN addgroup -g 1000 mcp && \
    adduser -D -u 1000 -G mcp mcp && \
    apk --no-cache add ca-certificates

WORKDIR /app
COPY --from=builder /app/mcp-server /usr/local/bin/mcp-server

# Security: Use non-root user
USER mcp

EXPOSE 8443

# Health check (Note: adjust to http://localhost:8080 if not using TLS in container)
HEALTHCHECK --interval=30s --timeout=3s --start-period=5s --retries=3 \
  CMD wget --no-verbose --tries=1 --spider http://localhost:8080/health || exit 1

CMD ["mcp-server"]

Docker Security Practices:

# Build image
docker build -t your-registry/mcp-server:latest .

# Verify no secrets in image layers
docker history your-registry/mcp-server:latest

# Run with secrets from secret manager (Docker Swarm)
docker service create \
  --name mcp-server \
  --secret oauth-encryption-key \
  --secret google-client-secret \
  your-registry/mcp-server:latest

# Or use Docker secrets mount (Docker Compose)
# See docker-compose.yml with secrets from files

NEVER:

• Use ENV or ARG for secrets in Dockerfile
• Copy .env files into the image
• Hardcode secrets in the image
• Use docker commit with running containers that have secrets

Docker Scout will flag these as critical vulnerabilities!

Kubernetes

Production Kubernetes deployments should use External Secrets Operator (see Setup section above).

Example deployment referencing synced secrets:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: mcp-server
  namespace: mcp-oauth
spec:
  replicas: 3
  selector:
    matchLabels:
      app: mcp-server
  template:
    metadata:
      labels:
        app: mcp-server
    spec:
      # Use Workload Identity or IRSA for secret access
      serviceAccountName: mcp-oauth
      containers:
      - name: mcp-server
        image: your-registry/mcp-server:latest
        env:
        # Secrets synced from External Secrets Operator
        - name: GOOGLE_CLIENT_ID
          valueFrom:
            secretKeyRef:
              name: oauth-secrets  # Created by ExternalSecret
              key: google-client-id
        - name: GOOGLE_CLIENT_SECRET
          valueFrom:
            secretKeyRef:
              name: oauth-secrets
              key: google-client-secret
        - name: OAUTH_ENCRYPTION_KEY
          valueFrom:
            secretKeyRef:
              name: oauth-secrets
              key: encryption-key
        # Non-secret configuration
        - name: MCP_RESOURCE
          value: "https://mcp.example.com"
        - name: LOG_JSON
          value: "true"
        ports:
        - containerPort: 8443
          name: https
        livenessProbe:
          httpGet:
            path: /health
            port: 8443
            scheme: HTTPS
          initialDelaySeconds: 10
          periodSeconds: 30
        readinessProbe:
          httpGet:
            path: /ready
            port: 8443
            scheme: HTTPS
          initialDelaySeconds: 5
          periodSeconds: 10
        resources:
          requests:
            memory: "128Mi"
            cpu: "100m"
          limits:
            memory: "512Mi"
            cpu: "500m"
        securityContext:
          runAsNonRoot: true
          runAsUser: 1000
          allowPrivilegeEscalation: false
          readOnlyRootFilesystem: true
          capabilities:
            drop:
            - ALL

Important Kubernetes Security Notes:

• Use External Secrets Operator to sync from Vault/AWS/GCP secret managers
• Never create Kubernetes Secrets manually with kubectl create secret
• Use Workload Identity (GKE) or IRSA (EKS) for secret manager authentication
• Enable Pod Security Standards (restricted profile)
• Use NetworkPolicies to limit pod-to-pod communication
• Enable audit logging for secret access

Troubleshooting

Tokens Not Persisting

If tokens don't persist across restarts, ensure:

• OAUTH_ENCRYPTION_KEY is set and consistent
• Key is exactly 32 bytes (base64 encoded)
• Using persistent storage (not in-memory only)

Rate Limiting Too Strict

Adjust limits in configuration:

Rate: 100,    // Increase limits
Burst: 200,

TLS Certificate Errors

For development, accept self-signed certs:

curl -k https://localhost:8443/health

For production, use valid certificates from Let's Encrypt or your CA.

Next Steps

• Set up log aggregation (ELK, Splunk, CloudWatch)
• Configure metrics and monitoring (Prometheus, Grafana)
• Implement backup and disaster recovery
• Set up CI/CD pipeline
• Load testing and performance tuning
• Security audit and penetration testing