terraform-provider-quant

Terraform Provider Quant

The QuantCDN Terraform provider allows you to manage resources in your Quant CDN environment with built-in API rate limiting and intelligent retry mechanisms.

Quickstarts

• Getting started with QuantCDN and terraform

Rate Limiting and Retry Configuration

The QuantCDN Terraform provider includes built-in API rate limiting and intelligent retry mechanisms with exponential backoff to ensure reliable operation and compliance with API limits.

Features

• Token Bucket Rate Limiting: Controls the number of requests per second to the QuantCDN API
• Exponential Backoff: Automatically retries failed requests with increasing delays
• Smart Retry Logic: Handles rate limiting (429), server errors (5xx), and network errors
• Safe for creates: a POST or PATCH is re-sent only after a 429, never after a timeout or 5xx, because the server may already have applied it and a second send would create a duplicate that fails with 409. Idempotent methods retry on everything above.
• Rules lock budget: the rules API answers 409 while another change holds the project lock. That is retried on its own budget (10 attempts, at least 2 s apart) instead of the general 3, so many rules changing at once do not exhaust it.
• Jitter Support: Adds randomisation to retry delays to prevent thundering herd effects
• Retry-After Support: Respects API-provided retry timing headers
• Context-Aware: Properly handles request cancellation and timeouts

Configuration Options

All rate limiting parameters can be configured via the provider block:

provider "quant" {
  bearer              = var.quantcdn_api_token
  organization        = var.quantcdn_organization
  
  # Rate limiting configuration (all optional)
  requests_per_second = 10.0    # Max requests per second (default: 10.0)
  max_retries        = 3        # Max retry attempts (default: 3)
  base_delay_ms      = 500      # Initial retry delay in ms (default: 500)
  max_delay_ms       = 30000    # Maximum retry delay in ms (default: 30000)
  enable_jitter      = true     # Add jitter to delays (default: true)
  timeout_seconds    = 120      # HTTP request timeout in seconds (default: 120)
}

Environment Variables

Configuration can also be set via environment variables:

export QUANTCDN_API_TOKEN="your-token"
export QUANTCDN_ORGANIZATION="your-org"
export QUANTCDN_BASE_URL="https://dashboard.example.com"  # Optional custom base URL
export QUANTCDN_REQUESTS_PER_SECOND="15.0"
export QUANTCDN_MAX_RETRIES="5"
export QUANTCDN_BASE_DELAY_MS="1000"
export QUANTCDN_MAX_DELAY_MS="60000"
export QUANTCDN_ENABLE_JITTER="true"
export QUANTCDN_TIMEOUT_SECONDS="120"

Usage Examples

Default Configuration

Most users can simply use the default configuration:

provider "quant" {
  bearer       = var.quantcdn_api_token
  organization = var.quantcdn_organization
}

High-Throughput Configuration

For environments requiring higher API throughput:

provider "quant" {
  bearer              = var.quantcdn_api_token
  organization        = var.quantcdn_organization
  requests_per_second = 20.0
  max_retries        = 5
  base_delay_ms      = 1000
  max_delay_ms       = 60000
  timeout_seconds    = 180      # 3 minutes for large operations
}

Conservative Configuration

For sensitive environments requiring maximum reliability:

provider "quant" {
  bearer              = var.quantcdn_api_token
  organization        = var.quantcdn_organization
  requests_per_second = 2.0
  max_retries        = 10
  base_delay_ms      = 2000
  max_delay_ms       = 120000
  timeout_seconds    = 300      # 5 minutes for maximum reliability
}

Multiple Provider Configurations

Use aliases for different rate limiting profiles:

# Default provider for most resources
provider "quant" {
  bearer       = var.quant_bearer_token
  organization = var.quant_organization
}

# High-throughput provider for bulk operations
provider "quant" {
  alias               = "bulk"
  bearer              = var.quant_bearer_token
  organization        = var.quant_organization
  requests_per_second = 25.0
  max_retries        = 2
}

# Conservative provider for critical resources
provider "quant" {
  alias               = "critical"
  bearer              = var.quant_bearer_token
  organization        = var.quant_organization
  requests_per_second = 1.0
  max_retries        = 15
  max_delay_ms       = 300000  # 5 minutes
}

# AU Government platform
provider "quant" {
  alias        = "quantgov"
  bearer       = var.quantgov_bearer_token
  organization = var.quantgov_organization
  base_url     = "https://dash.quantgov.cloud"
}

# Use different providers for different resources
resource "quant_domain" "example" {
  domain  = "example.com"
  project = "default"
}

resource "quant_project" "bulk_import" {
  provider = quant.bulk
  name     = "Bulk Import Project"
}

resource "quant_crawler" "critical" {
  provider = quant.critical
  name     = "Mission Critical Crawler"
  domain   = "https://critical.example.com"
}

resource "quant_project" "quantgov_project" {
  provider = quant.quantgov
  name     = "QuantGov Project"
}

Default Values

Best Practices

1. Start with defaults: The default configuration works well for most use cases
2. Monitor API usage: Adjust requests_per_second based on your API quota and usage patterns
3. Increase retries for reliability: Use higher max_retries for critical infrastructure
4. Use provider aliases: Configure different rate limiting profiles for different resource types
5. Test in development: Verify your rate limiting configuration in non-production environments
6. Respect API limits: Don't set requests_per_second higher than your API plan allows

Troubleshooting

• Frequent 429 errors: Reduce requests_per_second or increase retry delays
• Slow deployments: Increase requests_per_second if within your API limits
• Timeout errors: Increase max_delay_ms for better resilience in unstable networks
• Thundering herd: Ensure enable_jitter is set to true when running multiple providers

Advanced Configuration with Rate Limiting

provider "quant" {
  bearer              = "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
  organization        = "quant"
  base_url            = "https://dashboard.example.com"  # Optional custom base URL
  
  # Rate limiting configuration
  requests_per_second = 15.0    # Allow 15 requests per second
  max_retries        = 5        # Retry failed requests up to 5 times
  base_delay_ms      = 1000     # Start with 1 second delay for retries
  max_delay_ms       = 60000    # Maximum 60 second delay between retries
  enable_jitter      = true     # Add randomisation to prevent thundering herd
  timeout_seconds    = 120      # HTTP client timeout in seconds
}

Resources and Data Sources

Resources

• quant_project - Manage Quant projects with basic auth, query params, and revision settings
• quant_domain - Manage domains within projects
• quant_crawler - Manage web crawlers for content discovery and ingestion
  • Configure crawl depth, delays, browser mode, asset harvesting
  • Set up authentication, headers, and URL patterns (include/exclude)
  • Support for multi-domain crawling and sitemap discovery
• quant_crawler_schedule - Manage automated crawler schedules with cron expressions
• quant_header - Manage custom HTTP headers for domains
• quant_rule_proxy - Manage proxy rules for URL routing
  • Origin servers, caching, WAF configuration
  • Application proxy support for containerized apps
  • Header manipulation, IP whitelisting/blacklisting
• quant_rule_redirect - Manage HTTP redirects (301/302)
• quant_rule_custom_response - Manage custom HTTP responses with custom status codes
• quant_rule_content_filter - Manage content filtering rules with methods and IP restrictions

For detailed documentation on each resource, see the docs/resources directory.

Data Sources

• quant_projects - List all projects in an organisation
• quant_project - Fetch details for a specific project, including write tokens

Using Project Write Tokens in Other Providers

The quant_project data source allows you to fetch project write tokens for use in other providers, such as setting GitHub Actions secrets or AWS SSM parameters.

Example: GitHub Actions Integration

# Fetch project details including write token
data "quant_project" "main" {
  machine_name = "my-project"
  with_token   = true  # Default is true for data sources
}

# Set GitHub Actions secret with the write token
resource "github_actions_secret" "quant_token" {
  repository      = "my-org/my-repo"
  secret_name     = "QUANT_TOKEN"
  plaintext_value = data.quant_project.main.write_token
}

Example: Module Pattern

Create a reusable module for fetching Quant credentials:

# Module: quant-credentials/main.tf
data "quant_project" "project" {
  machine_name = var.project
  with_token   = true
}

output "token" {
  value     = data.quant_project.project.write_token
  sensitive = true
}

# Usage in main configuration
module "quant_credentials" {
  source = "./quant-credentials"
  count  = local.include_quant ? 1 : 0

  project = var.quant_project
}

resource "github_actions_secret" "quant_token" {
  count = local.include_quant ? 1 : 0

  repository      = var.repository
  secret_name     = "QUANT_TOKEN"
  plaintext_value = module.quant_credentials[0].token
}

Available Project Data

The quant_project data source provides the following attributes:

• id - Numeric project ID
• uuid - Project UUID
• name - Project display name
• machine_name - Project machine name
• write_token - Project write token (when with_token = true)

Requirements

• Terraform >= 1.0
• Go >= 1.23

Contributing

We welcome contributions! Please see our Contributing Guide for details.

Quick Start for Contributors

1. Fork the repository
2. Create a feature branch: git checkout -b feature/amazing-feature
3. Make your changes and add tests
4. Run tests: make testacc
5. Commit your changes: git commit -m 'Add amazing feature'
6. Push to the branch: git push origin feature/amazing-feature
7. Open a Pull Request

Testing

# Run unit tests
make test

# Run acceptance tests (requires QuantCDN API access)
make testacc

# Run acceptance tests with shorter timeout
make testacc-short

# Run specific test
TF_ACC=1 go test ./internal/provider/ -v -run TestAccProjectResource

Code Coverage

We maintain high test coverage. To view coverage locally:

make coverage

Or manually:

go test -coverprofile=coverage.out ./...
go tool cover -html=coverage.out

Building The Provider

1. Clone the repository
2. Enter the repository directory
3. Build the provider using the Makefile:

make build

Or manually:

go install

Adding Dependencies

This provider uses Go modules. Please see the Go documentation for the most up to date information about using Go modules.

To add a new dependency github.com/author/dependency to your Terraform provider:

go get github.com/author/dependency
go mod tidy

Then commit the changes to go.mod and go.sum.

Developing the Provider

With terraform v0.14 and later, development overrides for provider development can be used.

provider_installation {

  # Use /home/developer/tmp/terraform-null as an overridden package directory
  # for the hashicorp/null provider. This disables the version and checksum
  # verifications for this provider and forces Terraform to look for the
  # null provider plugin in the given directory.
  dev_overrides {
    "registry.terraform.io/quantcdn/quant" = "/path/to/quant-provider"
  }

  # For all other providers, install them directly from their origin provider
  # registries as normal. If you omit this, Terraform will _only_ use
  # the dev_overrides block, and so no other providers will be available.
  direct {}
}

If you wish to work on the provider, you'll first need Go installed on your machine (see Requirements above).

To compile the provider, run go install. This will build the provider and put the provider binary in the $GOPATH/bin directory.

In order to run the full suite of Acceptance tests, run make testacc.

Note: Acceptance tests create real resources, and often cost money to run.

make testacc

Or with shorter timeout:

make testacc-short