cf-networking-release

netman-release

A garden-runc add-on that provides container networking.

Overview

netman provides a batteries included container to container system and several APIs for swapping in third party components.

• IPAM and connectivity are provided by a swappable CNI plugin (flannel in the batteries included case).
• A swappable policy agent polls garden and the policy server for polices to enforce on the cell. In the provided solution, the VXLAN policy agent writes iptables rules to filter packets based on VXLAN gbp tags.
• Inbound traffic from the gorouter is port forwarded from the cell to the container via a NetIn rule. NetIn calls are made by garden to the external networker which then writes the iptables NAT rule.
• Application security groups are enforced by NetOut calls from garden. The external networker also writes iptables rules to enforce ASGs.

Project links

• Design doc for Container Networking Policy
• Engineering backlog
• Chat with us at the #container-networking channel on CloudFoundry Slack
• CI dashboard and config

Install the CF CLI Plugin

1. Get the binary

• Option 1: Download a precompiled binary of the network-policy-plugin for your operating system from our GitHub Releases

• Option 2: Build from source

  go build -o /tmp/network-policy-plugin ./src/cli-plugin
  

1. Install it

chmod +x ~/Downloads/network-policy-plugin
cf install-plugin ~/Downloads/network-policy-plugin

Deploy to bosh-lite

Follow the instructions here to install bosh-lite on your machine.

Ensure that br_netfilter is enabled on your vagrant box:

pushd ~/workspace/bosh-lite
  vagrant ssh -c 'sudo modprobe br_netfilter'
popd

Upload the latest bosh-lite stemcell

bosh upload stemcell https://bosh.io/d/stemcells/bosh-warden-boshlite-ubuntu-trusty-go_agent

Or download the stemcell and manually upload it to bosh-lite (potentially faster)

curl -L -o bosh-lite-stemcell-latest.tgz https://bosh.io/d/stemcells/bosh-warden-boshlite-ubuntu-trusty-go_agent
bosh upload stemcell bosh-lite-stemcell-latest.tgz

Then grab the required releases

pushd ~/workspace
  git clone https://github.com/cloudfoundry/diego-release
  git clone https://github.com/cloudfoundry/cf-release
  git clone https://github.com/cloudfoundry-incubator/netman-release
popd

Then deploy

pushd ~/workspace/netman-release
  ./scripts/deploy-to-bosh-lite
popd

Then follow the instructions for testing with the cats & dogs example.

NOTE: There is a known issue where VMs on bosh-lite can start failing, particularly if the host machine goes to sleep.

If you run bosh vms and see any failing VMs, then you can either recreate the individual failing vm(s) with bosh recreate <vm_name> or you can run bosh deploy --recreate to recreate all VMs.

Deploy to AWS

1. Upload stemcell with Linux kernel 4.4 to bosh director. Versions >= 3263.2 should work.
2. Create netman stubs

• Add under properties: uaa in stubs/cf/properties.yml:

  scim:
    users:
    - name: admin
      password: <admin-password>
      groups:
        - scim.write
        - scim.read
        - openid
        - cloud_controller.admin
        - clients.read
        - clients.write
        - doppler.firehose
        - routing.router_groups.read
        - routing.router_groups.write
        - network.admin
  clients:
    cf:
      scope: cloud_controller.read,cloud_controller.write,openid,password.write,cloud_controller.admin,scim.read,scim.write,doppler.firehose,uaa.user,routing.router_groups.read,network.admin
    network-policy:
      authorities: uaa.resource
      secret: <network-policy-secret>
  

• Create a netman stub stubs/netman/stub.yml:

  • The policy-agent communicates with the policy-server using mutual TLS. Generate PEM encoded certs and keys for vxlan-policy-agent and policy-server and update the associated properties.
    • See the generate-certs script for an example
  • All other fields with REPLACE_* values must be provided

  ---
  netman_overrides:
    releases:
    - name: netman
      version: latest
    driver_templates:
    - name: garden-cni
      release: netman
    - name: cni-flannel
      release: netman
    - name: netmon
      release: netman
    - name: vxlan-policy-agent
      release: netman
    properties:
      vxlan-policy-agent:
        policy_server_url: https://policy-server.service.cf.internal:4003
        ca_cert: REPLACE_WITH_SERVER_CA_CERT
        client_cert: REPLACE_WITH_CLIENT_CERT
        client_key: REPLACE_WITH_CLIENT_KEY
      policy-server:
        uaa_client_secret: REPLACE_WITH_UAA_CLIENT_SECRET
        uaa_url: (( "https://uaa." config_from_cf.system_domain ))
        skip_ssl_validation: true
        database:
          type: REPLACE_WITH_DB_TYPE # mysql or postgres
          connection_string: REPLACE_WITH_DB_CONNECTION_STRING
        ca_cert: REPLACE_WITH_CLIENT_CA_CERT
        server_cert: REPLACE_WITH_SERVER_CERT
        server_key: REPLACE_WITH_SERVER_KEY
      garden-cni:
        cni_plugin_dir: /var/vcap/packages/flannel/bin
        cni_config_dir: /var/vcap/jobs/cni-flannel/config/cni
      cni-flannel:
        flannel:
          etcd:
            require_ssl: (( config_from_cf.etcd.require_ssl))
        etcd_endpoints:
          - (( config_from_cf.etcd.advertise_urls_dns_suffix ))
        etcd_client_cert: (( config_from_cf.etcd.client_cert ))
        etcd_client_key: (( config_from_cf.etcd.client_key ))
        etcd_ca_cert: (( config_from_cf.etcd.ca_cert ))
    garden_properties:
      network_plugin: /var/vcap/packages/runc-cni/bin/garden-external-networker
      network_plugin_extra_args:
      - --configFile=/var/vcap/jobs/garden-cni/config/adapter.json
    jobs:
    - name: policy-server
      instances: 1
      persistent_disk: 256
      templates:
      - name: policy-server
        release: netman
      - name: route_registrar
        release: cf
      - name: consul_agent
        release: cf
      - name: metron_agent
        release: cf
      resource_pool: database_z1
      networks:
        - name: diego1
      properties:
        nats:
          machines: (( config_from_cf.nats.machines ))
          user: (( config_from_cf.nats.user ))
          password: (( config_from_cf.nats.password ))
          port: (( config_from_cf.nats.port ))
        metron_agent:
          zone: z1
        route_registrar:
          routes:
          - name: policy-server
            port: 4002
            registration_interval: 20s
            uris:
            - (( "api." config_from_cf.system_domain "/networking" ))
        consul:
          agent:
            services:
              policy-server:
                name: policy-server
                check:
                  interval: 5s
                  script: /bin/true
  
  config_from_cf: (( merge ))
  

1. Generate diego with netman manifest:

• Run the following bash script. Set environment_path to the directory containing your stubs for cf, diego, and netman. Set output_path to the directory you want your manifest to be created in. Set diego_release_path to your local copy of the diego-release repository.

set -e -x -u

environment_path=
output_path=
diego_release_path=

pushd cf-release
  ./scripts/generate_deployment_manifest aws \
    ${environment_path}/stubs/director-uuid.yml \
    ${diego_release_path}/examples/aws/stubs/cf/diego.yml \
    ${environment_path}/stubs/cf/properties.yml \
    ${environment_path}/stubs/cf/instance-count-overrides.yml \
    ${environment_path}/stubs/cf/stub.yml \
    > ${output_path}/cf.yml
popd

pushd diego-release
  ./scripts/generate-deployment-manifest \
    -g \
    -c ${output_path}/cf.yml \
    -i ${environment_path}/stubs/diego/iaas-settings.yml \
    -p ${environment_path}/stubs/diego/property-overrides.yml \
    -n ${environment_path}/stubs/diego/instance-count-overrides.yml \
    -N ${environment_path}/stubs/netman/stub.yml \
    -v ${environment_path}/stubs/diego/release-versions.yml \
    > ${output_path}/diego.yml
popd

1. Deploy

• Target your bosh director.

bosh target <your-director>

• Set the deployment

bosh deployment ${output_path}/diego.yml

• Deploy

bosh deploy

Other infrastructures

We do not currently test this software on infrastructures other than BOSH-lite and AWS. With recent stemcells and the appropriate manifest changes, it should work. Let us know if you find issues.

To replace flannel with your own CNI plugin

1. Remove the following BOSH jobs:

• cni-flannel
• vxlan-policy-agent

1. Remove the following BOSH packages:

• flannel
• flannel-watchdog

1. Add in all packages and jobs required by your CNI plugin. At a minimum, you must provide a CNI binary program and a CNI config file.

• For more info on bosh packaging scripts read this.
• For more info on bosh jobs read this.

1. Update the deployment manifest properties

garden-cni:
  adapter:
    cni_plugin_dir: /var/vcap/packages/YOUR_PACKAGE/bin # your CNI binary goes in this directory
    cni_config_dir: /var/vcap/jobs/YOUR_JOB/config/cni  # your CNI config file goes in this directory

Remove any lingering references to flannel or cni-flannel in the deployment manifest.

To deploy a local-only (no-op) CNI plugin

As a baseline, you can deploy using only the basic bridge CNI plugin.

This plugin will provide connectivity between containers on the same Garden host (Diego cell) but will not provide a cross-host network. However, it can be a useful baseline configuration for testing and development.

cd bosh-lite
bosh target lite
bosh update cloud-config cloud-config.yml
bosh deployment local-only.yml
bosh deploy

To deploy diego with CNI but without cross-host container networking

For generating a cloudfoundry-diego deployment without container to container connectivity, but using the CNI bridge plugin for NAT'ed connectivity.

CNI_BRIDGE=true ./scripts/generate-bosh-lite-manifests
bosh deploy

Development

Running low-level tests

~/workspace/netman-release/scripts/docker-test

Running the full acceptance test on bosh-lite

WARNING: This test is taxing and has an aggressive timeout. It may fail on a laptop or other underpowered bosh-lite.

cd src/netman-cf-acceptance
./run-locally.sh

Referencing a new library from existing BOSH package

1. Add any new libraries into the submodule from the root of the repo

cd $GOPATH
git submodule add https://github.com/foo/bar src/github.com/foo/bar
./scripts/sync-package-specs

Adding a new BOSH package

1. Add any new libraries into the submodules from the root of the repo

cd $GOPATH
git submodule add https://github.com/foo/bar src/github.com/foo/bar

1. Update the package sync script:

vim $GOPATH/scripts/sync-package-specs

Find or create the sync_package line for baz

1. Run the sync script:

./scripts/sync-package-specs

Syslog forwarding

To forward all logs from your bosh-lite to a syslog destination (like Papertrail), add the following block to manifest-generation/stubs/bosh-lite-cf.yml:

  syslog_daemon_config:
    address: some-syslog-host.example.com
    port: 12345
    transport: udp