btcachegeneration

package
v1.1.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Aug 13, 2020 License: Apache-2.0 Imports: 8 Imported by: 0

Documentation

Overview

Package btcachegeneration runs a GCF function that triggers in 2 scenarios:

  1. completion of prophet-flume job in borg. The trigger is based on GCS file prophet-cache/latest_base_cache_run.txt.
  2. On completion of BT cache ingestion via an airflow job. This trigger is based on GCS file prophet-cache/[success|failure].txt

In the first case, on triggering it sets up new cloud BT table, scales up BT cluster to 300 nodes and starts an airflow job by writing to prophet-cache/airflow.txt

In the second case it scales BT cluster to 20 nodes.

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func GCSTrigger 

func GCSTrigger(ctx context.Context, e GCSEvent) error

GCSTrigger consumes a GCS event.

Types

type GCSEvent 

type GCSEvent struct {
	Name   string `json:"name"`
	Bucket string `json:"bucket"`
}

GCSEvent is the payload of a GCS event.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.