e2e

e2e

Go Module providing robust framework for running complex workload scenarios in isolation, using Go and Docker. For integration, e2e tests, benchmarks and more! 💪

What are the goals?

• Ability to schedule isolated processes programmatically from single process on single machine.
• Focus on cluster workloads, cloud native services and microservices.
• Developer scenarios in mind e.g preserving scenario readability, Go unit test integration.
• Metric monitoring as the first citizen. Assert on Prometheus metric values during test scenarios or check overall performance characteristics.

Usage Models

There are three main use cases envisioned for this Go module:

• Unit test use (see example). Use e2e in unit tests to quickly run complex test scenarios involving many container services. This was the main reason we created this module. You can check usage of it in Cortex and Thanos projects.
• Standalone use (see example). Use e2e to run setups in interactive mode where you spin up workloads as you want programmatically and poke with it on your own using your browser or other tools. No longer need to deploy full Kubernetes or external machines.
• Benchmark use (see example). Use e2e in local Go benchmarks when your code depends on external services with ease.

Getting Started

Let's go through an example leveraging go test flow:

1. Implement the workload by embeddinge2e.Runnable or *e2e.InstrumentedRunnable. Or you can use existing ones in e2edb package. For example implementing Thanos Querier with our desired configuration could look like this:

   func newThanosSidecar(env e2e.Environment, name string, prom e2e.Linkable) *e2e.InstrumentedRunnable {
   	ports := map[string]int{
   		"http": 9090,
   		"grpc": 9091,
   	}
   	return e2e.NewInstrumentedRunnable(env, name, ports, "http", e2e.StartOptions{
   		Image: "quay.io/thanos/thanos:v0.21.1",
   		Command: e2e.NewCommand("sidecar", e2e.BuildArgs(map[string]string{
   			"--debug.name":     name,
   			"--grpc-address":   fmt.Sprintf(":%d", ports["grpc"]),
   			"--http-address":   fmt.Sprintf(":%d", ports["http"]),
   			"--prometheus.url": "http://" + prom.InternalEndpoint(e2edb.AccessPortName),
   			"--log.level":      "info",
   		})...),
   		Readiness: e2e.NewHTTPReadinessProbe("http", "/-/ready", 200, 200),
   		User:      strconv.Itoa(os.Getuid()),
   	})
   }
   

2. Implement test. Start by creating environment. Currently e2e supports Docker environment only. Use unique name for all your tests. It's recommended to keep it stable so resources are consistently cleaned.

   	// Start isolated environment with given ref.
   	e, err := e2e.NewDockerEnvironment("e2e_example")
   	testutil.Ok(t, err)
   	// Make sure resources (e.g docker containers, network, dir) are cleaned.
   	t.Cleanup(e.Close)
   

3. Program your scenario as you want. You can start, wait for their readiness, stop, check their metrics and use their network endpoints from both unit test (Endpoint) as well as within each workload (InternalEndpoint). You can also access workload directory. There is a shared directory across all workloads. Check Dir and InternalDir runnable methods.

   	// Create structs for Prometheus containers scraping itself.
   	p1, err := e2edb.NewPrometheus(e, "prometheus-1")
   	testutil.Ok(t, err)
   	s1 := newThanosSidecar(e, "sidecar-1", p1)
   
   	p2, err := e2edb.NewPrometheus(e, "prometheus-2")
   	testutil.Ok(t, err)
   	s2 := newThanosSidecar(e, "sidecar-2", p2)
   
   	// Create Thanos Query container. We can point the peer network addresses of both Prometheus instance
   	// using InternalEndpoint methods, even before they started.
   	t1 := newThanosQuerier(e, "query-1", s1.InternalEndpoint("grpc"), s2.InternalEndpoint("grpc"))
   
   	// Start them.
   	testutil.Ok(t, e2e.StartAndWaitReady(p1, s1, p2, s2, t1))
   
   	// To ensure query should have access we can check its Prometheus metric using WaitSumMetrics method. Since the metric we are looking for
   	// only appears after init, we add option to wait for it.
   	testutil.Ok(t, t1.WaitSumMetricsWithOptions(e2e.Equals(2), []string{"thanos_store_nodes_grpc_connections"}, e2e.WaitMissingMetrics()))
   
   	// To ensure Prometheus scraped already something ensure number of scrapes.
   	testutil.Ok(t, p1.WaitSumMetrics(e2e.Greater(50), "prometheus_tsdb_head_samples_appended_total"))
   	testutil.Ok(t, p2.WaitSumMetrics(e2e.Greater(50), "prometheus_tsdb_head_samples_appended_total"))
   
   	// We can now query Thanos Querier directly from here, using it's host address thanks to Endpoint method.
   	a, err := api.NewClient(api.Config{Address: "http://" + t1.Endpoint("http")})
   	testutil.Ok(t, err)
   
   	{
        now := model.Now()
        v, w, err := v1.NewAPI(a).Query(context.Background(), "up{}", now.Time())
        testutil.Ok(t, err)
        testutil.Equals(t, 0, len(w))
        testutil.Equals(
            t,
            fmt.Sprintf(`up{instance="%v", job="myself", prometheus="prometheus-1"} => 1 @[%v]
   up{instance="%v", job="myself", prometheus="prometheus-2"} => 1 @[%v]`, p1.InternalEndpoint(e2edb.AccessPortName), now, p2.InternalEndpoint(e2edb.AccessPortName), now),
            v.String(),
        )
   	}
   
   	// Stop first Prometheus and sidecar.
   	testutil.Ok(t, s1.Stop())
   	testutil.Ok(t, p1.Stop())
   
   	// Wait a bit until Thanos drops connection to stopped Prometheus.
   	testutil.Ok(t, t1.WaitSumMetricsWithOptions(e2e.Equals(1), []string{"thanos_store_nodes_grpc_connections"}, e2e.WaitMissingMetrics()))
   
   	{
        now := model.Now()
        v, w, err := v1.NewAPI(a).Query(context.Background(), "up{}", now.Time())
        testutil.Ok(t, err)
        testutil.Equals(t, 0, len(w))
        testutil.Equals(
            t,
            fmt.Sprintf(`up{instance="%v", job="myself", prometheus="prometheus-2"} => 1 @[%v]`, p2.InternalEndpoint(e2edb.AccessPortName), now),
            v.String(),
        )
   	}
   }
   

Credits

• Initial Authors: @pracucci, @bwplotka, @pstibrany
• Cortex Team hosting previous form of this module initially.