pipeline-backend

pipeline-backend

pipeline-backend manages all pipeline resources within Versatile Data Pipeline (VDP) to streamline data from a source connector to models, and to a destination connector at the end.

Local dev

On the local machine, clone vdp repository in your workspace, move to the repository folder, and launch all dependent microservices:

$ cd <your-workspace>
$ git clone https://github.com/instill-ai/vdp.git
$ cd vdp
$ make dev PROFILE=pipeline

Clone pipeline-backend repository in your workspace and move to the repository folder:

$ cd <your-workspace>
$ git clone https://github.com/instill-ai/pipeline-backend.git
$ cd pipeline-backend

Build the dev image

$ make build

Run the dev container

$ make dev

Now, you have the Go project set up in the container, in which you can compile and run the binaries together with the integration test in each container shell.

Run the server

$ docker exec -it pipeline-backend /bin/bash
$ go run ./cmd/migration
$ go run ./cmd/main

Run the integration test

$ docker exec -it pipeline-backend /bin/bash
$ make integration-test

Stop the dev container

$ make stop

CI/CD

• pull_request to the main branch will trigger the Integration Test workflow running the integration test using the image built on the PR head branch.
• push to the main branch will trigger
  • the Integration Test workflow building and pushing the :latest image on the main branch, following by running the integration test, and
  • the Release Please workflow, which will create and update a PR with respect to the up-to-date main branch using release-please-action.

Once the release PR is merged to the main branch, the release-please-action will tag and release a version correspondingly.

The images are pushed to Docker Hub repository.

License

See the LICENSE file for licensing information.