lfx-v2-query-service

lfx-v2-query-service

HTTP service for LFX API consumers to perform access-controlled queries for LFX resources, including typeahead and full-text search.

Agents working in this repo should start with CLAUDE.md. API contract truth lives under docs/.

Architecture Overview

The implementation follows the clean architecture principles where:

• Domain Layer: Contains business logic and interfaces
• Service Layer: Orchestrates business operations
• Infrastructure Layer: Implements external dependencies
• Presentation Layer: Handles HTTP/API concerns (generated by Goa)

Directory Structure

├── .github/                        # Github files
│   └── workflows/                  # Github Action workflow files
├── charts/                         # Helm charts
├── design/                         # GOA design specification files
├── gen/                            # GOA generated code (HTTP server, client, OpenAPI)
├── cmd/                            # Services (main packages)
│   └── service/                    # Service implementation
├── internal/                       # Internal service packages
│   ├── domain/                     # Domain logic layer
│   │   ├── model/                  # Domain models and entities
│   │   └── port/                   # Domain interfaces/ports
│   ├── service/                    # Business logic/use cases layer
│   ├── infrastructure/             # Infrastructure layer
│   └── middleware/                 # HTTP middleware components
└── pkg/                            # Shared packages for internal and external services

Key Components

Domain Layer (internal/domain/)

• ResourceService: Contains business logic and validation

Model (internal/domain/model/)

• Domain Models: Core business entities and data structures
• Value Objects: Immutable objects that represent domain concepts

Ports (internal/domain/port/)

• ResourceSearcher Interface: Defines the contract for resource search operations
• OrganizationSearcher Interface: Defines the contract for organization search operations
• AccessControlChecker Interface: Defines the contract for access control operations
• Authenticator Interface: Defines the contract for authentication operations

Service Layer (internal/service/)

• Business Logic: Application-specific business rules and operations
• Service Orchestration: Coordinates between domain models and infrastructure

Infrastructure Layer (internal/infrastructure/)

Authentication Implementation

The authentication system provides JWT-based authentication with support for Heimdall tokens:

• JWT Authentication (internal/infrastructure/auth/jwt.go):

  • Validates JWT tokens using JWKS (JSON Web Key Set)
  • Extracts custom claims including principal and email
  • Supports PS256 signature algorithm (default for Heimdall)
  • Configurable JWKS URL and audience

• Mock Authentication (internal/infrastructure/mock/auth.go):

  • Uses environment variable for mock principal
  • Bypasses JWT validation for local development

Authentication Configuration:

• AUTH_SOURCE: Choose between "mock" or "jwt" (default: "jwt")
• JWKS_URL: JSON Web Key Set endpoint URL
• JWT_AUDIENCE: Intended audience for JWT tokens
• JWT_AUTH_DISABLED_MOCK_LOCAL_PRINCIPAL: Mock principal for development

OpenSearch Implementation

The OpenSearch implementation includes query templates, a searcher, and a client for interacting with the OpenSearch cluster.

NATS Implementation

The NATS implementation consists of a client, access control logic, and request/response models for messaging and access control.

Clearbit Implementation

The Clearbit implementation provides organization search capabilities using the Clearbit Company API. It includes a client for API communication, searcher for organization queries, and configuration management for API credentials and settings.

Dependency Injection

Dependency injection starts in cmd/main.go, with concrete implementations selected in cmd/service/providers.go based on environment configuration and then injected into the service constructor.

Benefits of This Architecture

1. Testability: Easy to swap implementations for testing
2. Flexibility: Can easily switch between different search backends and access control systems
3. Maintainability: Clear separation of concerns
4. Scalability: Easy to add new search and access control implementations
5. Independence: Layers don't depend on external frameworks

Docker

Building the Docker Image

Example: Build the Docker image with Make

make docker-build

Running with Docker

Basic Docker Run

make docker-run

Usage

Queryable Resource Types

All resource types searchable via this service are listed in docs/resource-catalog.md, organized by the service that indexes them. Each entry links to that service's indexer contract, the authoritative reference for its data schemas, tags, access control config, and parent references.

For agent-oriented contract docs:

• docs/query-service-contract.md: query API, access filtering, OpenSearch fields, tags, filters, CEL, page size, date ranges, and clause limits.
• docs/indexed-data-types.md: active queryable resource types and where their service-owned contracts live.

Running Locally

With Mock Implementation (Default for Development)

# Using mock implementations
SEARCH_SOURCE=mock ACCESS_CONTROL_SOURCE=mock ORG_SEARCH_SOURCE=mock AUTH_SOURCE=mock \
JWT_AUTH_DISABLED_MOCK_LOCAL_PRINCIPAL=dev_user \
PAGE_TOKEN_SECRET=12345678901234567890123456789012 go run ./cmd

# With custom port
SEARCH_SOURCE=mock ACCESS_CONTROL_SOURCE=mock ORG_SEARCH_SOURCE=mock AUTH_SOURCE=mock \
JWT_AUTH_DISABLED_MOCK_LOCAL_PRINCIPAL=dev_user \
PAGE_TOKEN_SECRET=12345678901234567890123456789012 go run ./cmd -p 3000

With Production Services

# production-like setup
SEARCH_SOURCE=opensearch \
ORG_SEARCH_SOURCE=clearbit \
ACCESS_CONTROL_SOURCE=nats \
OPENSEARCH_URL={{placeholder}} \
OPENSEARCH_INDEX=resources \
NATS_URL={{placeholder}} \
PAGE_TOKEN_SECRET=12345678901234567890123456789012 \
CLEARBIT_CREDENTIAL=your_clearbit_api_key \
CLEARBIT_BASE_URL=https://company.clearbit.com \
CLEARBIT_TIMEOUT=30s \
CLEARBIT_MAX_RETRIES=5 \
CLEARBIT_RETRY_DELAY=2s \
go run ./cmd

Available Environment Variables

Search Implementation:

• SEARCH_SOURCE: Choose between "mock" or "opensearch" (default: "opensearch")

Organization Search Implementation:

• ORG_SEARCH_SOURCE: Choose between "mock" or "clearbit" (default: "clearbit")

OpenSearch Configuration:

• OPENSEARCH_URL: OpenSearch URL (default: http://localhost:9200)
• OPENSEARCH_INDEX: OpenSearch index name (default: "resources")
• PAGE_TOKEN_SECRET: 32-character secret used to encode/decode opaque page tokens. Required for any deployment: the service calls global.PageTokenSecret(ctx) (which fatals if the var is unset) whenever it generates a page_token for a full page, so even an ordinary first-page query can crash the process if this is missing.

Access Control Implementation:

• ACCESS_CONTROL_SOURCE: Choose between "mock" or "nats" (default: "nats")

NATS Configuration:

• NATS_URL: NATS server URL (default: nats://localhost:4222)
• NATS_TIMEOUT: Request timeout duration (default: "10s")
• NATS_MAX_RECONNECT: Maximum reconnection attempts (default: "3")
• NATS_RECONNECT_WAIT: Time between reconnection attempts (default: "2s")

Clearbit Configuration:

• CLEARBIT_CREDENTIAL: Clearbit API key (required for organization search)
• CLEARBIT_BASE_URL: Clearbit Company API base URL (default: https://company.clearbit.com)
• CLEARBIT_AUTOCOMPLETE_BASE_URL: Clearbit Autocomplete API base URL (default: https://autocomplete.clearbit.com)
• CLEARBIT_TIMEOUT: HTTP client timeout for API requests (default: "10s")
• CLEARBIT_MAX_RETRIES: Maximum number of retry attempts for failed requests (default: "3")
• CLEARBIT_RETRY_DELAY: Delay between retry attempts (default: "1s")

Authentication Configuration:

• AUTH_SOURCE: Choose between "mock" or "jwt"
• JWKS_URL: JSON Web Key Set endpoint URL
• JWT_AUDIENCE: Intended audience for JWT tokens
• JWT_AUTH_DISABLED_MOCK_LOCAL_PRINCIPAL: Mock principal for development (required when AUTH_SOURCE=mock)

Server Configuration:

• -p: HTTP port (default: "8080")
• -bind: Interface to bind on (default: "*")
• -d: Enable debug logging

API Usage

The service exposes a RESTful API through the Goa framework with JWT authentication:

Resource Search API

GET /query/resources?name=committee&type=committee&v=1
Authorization: Bearer <jwt_token>

Parameters:

• name: Resource name or alias (supports typeahead search)
• type: Resource type to filter by
• parent: Parent resource for hierarchical queries
• tags: Array of tags to filter by (OR logic - matches resources with any of these tags)
• tags_all: Array of tags to filter by (AND logic - matches resources that have all of these tags)
• filters, filters_all, filters_or: field:value filters against data.*
• cel_filter: CEL expression for advanced post-query filtering
• filter_grants: direct to pre-filter to resources with direct FGA grants (requires type)
• date_field: Date field to filter on (within data object) - used with date_from and/or date_to
• date_from: Start date (inclusive). Format: ISO 8601 datetime or date-only (YYYY-MM-DD). Date-only uses start of day UTC
• date_to: End date (inclusive). Format: ISO 8601 datetime or date-only (YYYY-MM-DD). Date-only uses end of day UTC
• sort: Sort order (name_asc, name_desc, updated_asc, updated_desc, best_match)
• page_size: Number of results per page (1-1000, default 50)
• page_token: Pagination token
• v: API version (required)

Response:

{
  "resources": [
    {
      "type": "committee",
      "id": "123",
      "data": {
        "name": "Technical Advisory Committee",
        "description": "Main technical governance body",
        "status": "active"
      }
    }
  ],
  "page_token": "opaque-token-or-omitted",
  "cache_control": "public, max-age=300"
}

At least one of name, parent, type, tags, or filter_grants must be provided; a request with only cel_filter, date_*, filters*, sort, or pagination parameters is rejected with 400 Bad Request.

Resource Count API

GET /query/resources/count?type=committee&v=1
Authorization: Bearer <jwt_token>

Accepts the same filter parameters as GET /query/resources except cel_filter, filter_grants, sort, page_size, and page_token, and returns a count rather than the resources themselves:

{ "count": 42, "has_more": false }

has_more is true when the count is not guaranteed to be exhaustive and the client should request a narrower query.

For API contract details (page size, date ranges, CEL filter, clause limits, anonymous vs authenticated semantics, access-control flow), see docs/query-service-contract.md.

Organization Search API

Query Organizations:

GET /query/orgs?name=Linux Foundation&domain=linuxfoundation.org&v=1
Authorization: Bearer <jwt_token>

Parameters:

• name: Organization name (optional)
• domain: Organization domain or website URL (optional)
• v: API version (required)

Response:

{
  "name": "Linux Foundation",
  "domain": "linuxfoundation.org",
  "industry": "Non-Profit",
  "sector": "Technology",
  "employees": "100-499"
}

Organization Suggestions API:

GET /query/orgs/suggest?query=linux&v=1
Authorization: Bearer <jwt_token>

Parameters:

• query: Search query for organization suggestions (required, minimum 1 character)
• v: API version (required)

Response:

{
  "suggestions": [
    {
      "name": "Linux Foundation",
      "domain": "linuxfoundation.org",
      "logo": "https://example.com/logo.png"
    }
  ]
}

Clearbit API Integration

The service integrates with Clearbit's Company API to provide enriched organization data for search operations. This integration allows the service to fetch detailed company information including industry classification, employee count, and domain information.

Clearbit API Setup

1. Obtain API Credentials

To use Clearbit integration, you need to obtain an API key from Clearbit:

1. Sign up for a Clearbit account at https://clearbit.com
2. Navigate to your API settings to generate an API key
3. Copy the API key for use in your environment configuration

2. Configure Environment Variables

Set the required environment variables for Clearbit integration:

# Required: Clearbit API key
export CLEARBIT_CREDENTIAL=your_clearbit_api_key_here

# Optional: Custom configuration (defaults shown)
export CLEARBIT_BASE_URL=https://company.clearbit.com
export CLEARBIT_TIMEOUT=10s
export CLEARBIT_MAX_RETRIES=3
export CLEARBIT_RETRY_DELAY=1s

# Set organization search source to use Clearbit
export ORG_SEARCH_SOURCE=clearbit

3. API Usage and Features

The Clearbit integration supports the following search operations:

Search by Company Name:

• Searches for companies using their registered business name
• Falls back to domain-based search for additional data enrichment

Search by Domain:

• More accurate search method using company domain names
• Provides comprehensive company information

4. Error Handling

The Clearbit integration includes robust error handling:

• 404 Not Found: Returns appropriate "organization not found" errors
• Rate Limiting: Automatic retry with exponential backoff
• Network Issues: Configurable retry attempts with delays
• API Errors: Proper error propagation with context

Testing

The clean architecture makes testing straightforward:

// Use mock implementations for unit tests
searcher := mock.NewMockResourceSearcher()
searcher.AddResource(testResource)

accessChecker := mock.NewMockAccessControlChecker()
// CheckAccess returns a map keyed by the per-resource access-check line
accessChecker.SetCheckAccessResponse(map[string]string{
    "project:abc#viewer@user:123": "true",
})

resourceFilter := mock.NewMockResourceFilter()

service := service.NewResourceSearch(searcher, accessChecker, resourceFilter)
result, err := service.QueryResources(ctx, criteria)

Extending the Architecture

To add a new search implementation:

1. Create a new package in internal/infrastructure/
2. Implement the domain.ResourceSearcher interface
3. Add configuration options to main.go
4. Update the dependency injection switch statement

To add a new access control implementation:

1. Create a new package in internal/infrastructure/
2. Implement the domain.AccessControlChecker interface
3. Add configuration options to main.go
4. Update the dependency injection switch statement

Development

Prerequisites

This project uses the GOA Framework for API generation. You'll need to install GOA before building the project.

Installing GOA Framework

Follow the GOA installation guide to install GOA:

go install goa.design/goa/v3/cmd/goa@latest

Verify the installation:

goa version

Building and Development

1. Generate Code

The project uses GOA to generate API code from the design specification. Run the following command to generate all necessary code:

goa gen github.com/linuxfoundation/lfx-v2-query-service/design

This command generates:

• HTTP server and client code
• OpenAPI specification
• Service interfaces and types
• Transport layer implementations

2. Initial Project Structure

Note: The initial cmd structure was generated using GOA's example generator:

goa example github.com/linuxfoundation/lfx-v2-query-service/design

This command generated the basic server structure, which was then customized and adjusted to fit our project's clean architecture principles.

3. Development Workflow

1. Make design changes: Edit files in the design/ directory

2. Regenerate code: Run goa gen github.com/linuxfoundation/lfx-v2-query-service/design after design changes

3. Build the project:

   go build ./cmd
   

4. Run with mock data (for development):

   SEARCH_SOURCE=mock ACCESS_CONTROL_SOURCE=mock ORG_SEARCH_SOURCE=mock AUTH_SOURCE=mock \
   JWT_AUTH_DISABLED_MOCK_LOCAL_PRINCIPAL=dev_user \
   PAGE_TOKEN_SECRET=12345678901234567890123456789012 go run ./cmd
   

5. Run tests:

   make test
   
   # or run go test to set custom flags
   go test ./... -v
   

6. Lint the code

   # From the root of the directory, run megalinter (https://megalinter.io/latest/mega-linter-runner/) to ensure the code passes the linter checks. The CI/CD has a check that uses megalinter.
   npx mega-linter-runner .
   

7. Docker build + K8

   # Build the dockerfile (from the root of the repo)
   docker build -t lfx-v2-query-service:<release_number> .
   
   # Install the helm chart for the service into the lfx namespace (from the root of the repo)
   helm install lfx-v2-query-service ./charts/lfx-v2-query-service/ -n lfx
   
   # Once you have already installed the helm chart and need to just update it, use the following command (from the root of the repo):
   helm upgrade lfx-v2-query-service ./charts/lfx-v2-query-service/ -n lfx
   

Contributing

To contribute to this repository:

1. Fork the repository
2. Make your changes
3. Submit a pull request

License

Copyright The Linux Foundation and each contributor to LFX.

This project’s source code is licensed under the MIT License. A copy of the license is available in LICENSE.

This project’s documentation is licensed under the Creative Commons Attribution 4.0 International License (CC-BY-4.0). A copy of the license is available in LICENSE-docs.