On-premise Layer-2 Service type LoadBalancer with Antrea

While kubernetes clusters running with cloud providers benefit from external load balancers which can be easily interacted with through the CPI, on-premise installations don’t get that by default. Sure if you have an external load balancer you can always install the kubernetes operator if there is one but it is not such a straightforward operation. For instance, if you take clusters deployed to vSphere through CAPV of TCE, your LoadBalancer service resources will remain in pending state out of the box.

❯ k get svc whoami-lb
NAME        TYPE           CLUSTER-IP      EXTERNAL-IP   PORT(S)          AGE
whoami-lb   LoadBalancer   10.102.27.166   <pending>     8080:30001/TCP   17s

There are a few options out there to get Load Balancer capabilities such as metalLB but I liked the idea of having it included in the CNI with Antrea since v1.5.0. Antrea is a CNI project maintained by VMware and backed by a really cool and helpful community. In fact they have a weekly podcast on Youtube called Antrea Live. You can also find the #Antrea Slack channel in the Kubernetes Slack.

Anyways, Antrea can allocate external IPs to load balancer service types from a pool that you set yourself. When a service is created, the “virtual IP” is allocated to one of the nodes with a “nodeSelector” and the traffic is routed ky kube-proxy (or antrea proxy) to the service endpoints.

Note that the range of external IPs you allocate to it will need to be reserved so they can’t be distributed by DHCP in order to avoid duplicates. The documentation about all this is in the Antrea repo and you will find everything that is missing in my blog there.

Install Antrea v1.5.0

Let’s take things one by one and start with the installation of Antrea. You can check out my blog on CAPV (cluster api provier vSphere) if you want to provision a kubernetes cluster to vSphere. You can also use Tanzu Community Edition which already comes with Antrea by default I believe.

Anyway, if you deploy your cluster with CAPV like in my other blog, by default the nodes will be in “NotReady” state until you install a CNI.

Let’s then install the Antrea CNI by applying the default manifest from upstream.

kubectl apply -f https://github.com/antrea-io/antrea/releases/download/v1.5.0/antrea.yml

This will deploy CRDs, service accounts, and other antrea components into the cluster which should then be in Ready state.

Configure Antrea Load Balancer

Now we need to enable the ServiceExternalIP features of Antrea. To do that we will edit the Antrea configmap that holds (you guessed it) the configuration of Antrea. It is named “antrea-config-xxx” and is in the “kube-system” namespace.

k edit cm antrea-config-2k82d6f6t4 -n kube-system

Now you want to find the “ServiceExternalIP” records, uncomment them and set them to true:

data:
  antrea-agent.conf: |
    featureGates:
      ServiceExternalIP: true
  antrea-controller.conf: |
    featureGates:
      ServiceExternalIP: true

You won’t need to do this if you deploy everything correctly from the get go but if you change values on the fly like we just did, you’ll need to destroy the antrea pod so they can pick up the new values (that’s what I had to do at least), I believe it’s only true for the agent pods but you might as well go crazy.

kubectl delete pod -l app=antrea -n kube-system

Next prepare a manifest to deploy an ExternalIPPool, I named mine antrea-externalIPpool.yaml. This is an Antrea CRD that the loadbalancer service can pick from to assign. In my example I only allowed 3 IP addresses from my node network. Don’t forget the “ “ if your label isn’t a string (mine is a boolean), otherwise the Antrea Controller will log errors and it won’t work.

Note that I manually applied the antrea-lb=true to all my worker nodes beforehand which is what will make them participate.

apiVersion: crd.antrea.io/v1alpha2
kind: ExternalIPPool
metadata:
  name: service-external-ip-pool
spec:
  ipRanges:
  - start: 10.10.222.240
    end: 10.10.222.242
  - cidr: 10.10.222.0/24
  nodeSelector:
    matchLabels:
      antrea-lb: "true"

Now apply the manifest:

k apply -f antrea-externalIPpool.yaml

Create load balancer service types

We can now start with load balancer service types. In order for them to work with Antrea however, you need to apply the service.antrea.io/external-ip-pool: “service-external-ip-pool” annotation to them.

Let’s first create a small deployment with an image that displays information about the pod.

k create deployment whoami --image=containous/whoami --replicas=3

Then expose the deployment with a service type load balancer.

kubectl expose deployment whoami --port=8080 --target-port=80 --name=whoami-lb --type=LoadBalancer

You will find that it is still in Pending state because we haven’t set the annotation yet.

NAME         TYPE           CLUSTER-IP     EXTERNAL-IP   PORT(S)          AGE
whoami-lb    LoadBalancer   10.96.43.248   <pending>     8080:31882/TCP   6m16s

Let’s do that by editing the service with “kubectl edit svc whoami-lb” and add the service.antrea.io/external-ip-pool: “whoami-lb” annotation in the metadata section.

apiVersion: v1
kind: Service
metadata:
  name: whoami-lb
  annotations:
    service.antrea.io/external-ip-pool: "whoami-lb"
 ...

The service should now get an IP address like so:

NAME        TYPE           CLUSTER-IP     EXTERNAL-IP     PORT(S)          AGE
whoami-lb   LoadBalancer   10.97.177.72   10.10.222.240   8080:30402/TCP   23m

Now you can try to curl the external IP address several times to witness the load balancing between the pods. As you can see below I get a response from all three pods. Note that this is not round robin so I had to run it a bunch of times to get a different response three times in a row.

Wrap up

I very much enjoyed playing with the load balancer capability of Antrea as I think it is great to have an alternative to external load balancers, especially in lab environments or for learning purpose if you want to get up and running quickly.

I should point out that we did a bunch of stuff imperatively and edited configs on the fly which is not best practice in an actual environment. To do correctly you want to get all your manifests in order prior to deploying them.

If you are going to play with this, you might be interested in my blog about ClusterResourceSets with Calico. Those are CRDs that let you install a resource in a cluster from the management cluster using a configMap. You can also check out my capi repo (which should be renamed capv) where I have a Helm chart for CAPV to install workload clusters with Antrea in a clusterResourceSet. Although the load balancing stuff isn’t set up and I make quite a few changes in there so this might become obsolete as I tinker with it.