README
¶
Remote implementation of Catalog datastore APIs
This implementation calls a gRPC service which is backed by the Catalog postgres database. The service offers a streaming gRPC API, which allows for transaction state to be maintained through that stream connection. If the stream closes, normal cleanup will rollback the transaction. The Catalog service APIs include message to begin, commit, and roll-back a transaction.
Running Tests
The tests in this package use testcontainers to automatically start the required services:
- PostgreSQL database
- Catalog service (from ECR or local image)
- Database migration
Important: Testcontainers uses images from your local Docker daemon. You must either:
- Build the image locally from the
op-catalogdirectory:cd op-catalog docker build -t op-catalog-service:latest . - Pull from ECR with a specific tag:
docker pull 123123123123.dkr.ecr.us-east-1.amazonaws.com/op-catalog-service:TAG- Use
aws ecr describe-imagesto find available tags - Common patterns: version tags (v0.0.1)
- Use
Quick Start (Local Build)
# Build the catalog service image first
cd op-catalog
docker build -t op-catalog-service:latest .
# Run the tests (must be in the remote directory)
cd ../chainlink-deployments-framework/datastore/catalog/remote
go test -v
Configuration Options
Environment Variables
-
CATALOG_SERVICE_IMAGE: The Docker image to use for the catalog service- Default:
op-catalog-service:latest - CI: Set to the ECR image URL
- Example:
export CATALOG_SERVICE_IMAGE="123456789.dkr.ecr.us-east-1.amazonaws.com/op-catalog-service:latest"
- Default:
-
CATALOG_GRPC_ADDRESS: Connect to an existing catalog service instead of starting containers- Example:
export CATALOG_GRPC_ADDRESS="localhost:8080"
- Example:
Running Without Testcontainers
If you prefer to manage the services manually:
Option 1: Docker Compose
cd op-catalog
docker-compose up -d
# Wait for the service to be ready
sleep 5
# Run tests pointing to the local service
cd ../chainlink-deployments-framework/datastore/catalog/remote
export CATALOG_GRPC_ADDRESS="localhost:8080"
go test -v
CI/CD Integration
The CI workflow automatically:
- Downloads the latest Catalog service image from ECR
- Runs the tests with testcontainers using the downloaded image
- Generates coverage reports
See .github/workflows/pull-request-main.yml for the full configuration.
Test Workflow
The test setup follows this flow:
-
TestMain (
main_test.go) - Entry point that:- Checks for
CATALOG_GRPC_ADDRESSenvironment variable - If not set, starts testcontainers automatically
- Sets up PostgreSQL and Catalog service
- Runs all tests
- Cleans up containers
- Checks for
-
TestContainerSetup (
testcontainer_setup.go) - Manages:- Docker network creation
- PostgreSQL container startup
- Database migration via catalog service
- Catalog service container startup
- Port mapping and connection details
Documentation
¶
Index ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func NewCatalogDataStore ¶
func NewCatalogDataStore(config CatalogDataStoreConfig) *catalogDataStore
func ThrowAndCatch ¶
func ThrowAndCatch( catalog *catalogDataStore, request *pb.DataAccessRequest, ) (*pb.DataAccessResponse, error)
Types ¶
type CatalogClient ¶
type CatalogClient struct {
// contains filtered or unexported fields
}
func NewCatalogClient ¶
func NewCatalogClient(ctx context.Context, cfg CatalogConfig) (*CatalogClient, error)
NewCatalogClient creates a new CatalogClient with the provided configuration.
Example usage:
cfg := CatalogConfig{
GRPC: "op-catalog.example.com:443",
Creds: credentials.NewTLS(&tls.Config{}),
HMACConfig: &HMACAuthConfig{
KeyID: "kms-key-id",
KeyRegion: "us-west-2",
Authority: "op-catalog.example.com",
},
}
client, err := NewCatalogClient(ctx, cfg)
func (*CatalogClient) Close ¶ added in v0.67.0
func (c *CatalogClient) Close() error
Close closes the underlying gRPC connection.
func (*CatalogClient) CloseStream ¶
func (c *CatalogClient) CloseStream() error
CloseStream closes the current stream.
func (*CatalogClient) DataAccess ¶
func (c *CatalogClient) DataAccess(req proto.Message) (grpc.BidiStreamingClient[pb.DataAccessRequest, pb.DataAccessResponse], error)
type CatalogConfig ¶
type CatalogConfig struct {
GRPC string
Creds credentials.TransportCredentials
HMACConfig *HMACAuthConfig
}
type CatalogDataStoreConfig ¶
type CatalogDataStoreConfig struct {
Domain string
Environment string
Client *CatalogClient
}
type HMACAuthConfig ¶ added in v0.65.0
type HMACAuthConfig struct {
KeyID string
KeyRegion string
Authority string // The gRPC authority (hostname without port) used for HMAC signing
}
HMACAuthConfig holds HMAC authentication configuration.
type TestContainerSetup ¶ added in v0.60.0
type TestContainerSetup struct {
PostgresContainer *postgres.PostgresContainer
CatalogContainer testcontainers.Container
PostgresDSN string
CatalogGRPCAddr string
Network *testcontainers.DockerNetwork
}
TestContainerSetup holds all testcontainer resources for catalog service testing
func SetupCatalogTestContainers ¶ added in v0.60.0
func SetupCatalogTestContainers(ctx context.Context) (*TestContainerSetup, error)
SetupCatalogTestContainers initializes all required containers for testing: 1. PostgreSQL database 2. Catalog service (with db migration)
func (*TestContainerSetup) GetCatalogGRPCAddress ¶ added in v0.60.0
func (s *TestContainerSetup) GetCatalogGRPCAddress() string
GetCatalogGRPCAddress returns the gRPC address for catalog service
Source Files