Workload Reachability 2: Service Types

in Kubernetes on Kubernetes, Basics, Service, Clusterip, Nodeport, Loadbalancer Last modified at:
Workload Reachability 2: Service Types

The Service object is the foundation for DNS and load-balancing in Kubernetes, and there are different Service types available to fit various use cases, including allowing for external traffic to reach cluster-internal workloads or making an external workload available from within the cluster.

The previous blog post introduced you to the basics of the Service object and what problem it solves in Kubernetes, so let’s now spice things up a little and introduce the various different Service types to the party!

Before we get started: There will be a couple of examples in the upcoming sections, and to follow along, you’ll need your own Kubernetes cluster. In case you don’t have access to one yet, please refer to this section in one of my previous blog posts to help you get started.

The Most Common Service Types

The three types of Services you’ll probably encounter most often are ClusterIP, NodePort, and LoadBalancer, and they build on each other to provide their specific behavior. You’ll learn about this in the upcoming sections, but here’s a minimal overview to get you prepared:

  • ClusterIP is the default Service type and it works with virtual cluster IPs, which are only reachable from within the cluster (without using tricks, anyway)
  • To address use cases involving traffic from outside the cluster, there are two more types of Service Kubernetes has in store for you, namely, NodePort and LoadBalancer
  • NodePort and LoadBalancer allow directing external traffic into the cluster (without the need for tricks), and they have their own pros and cons, which make them suitable for different situations

We’ll take a look at each of these Service types in the upcoming sections from the perspective of enabling external access. Let’s get started with the classic: the ClusterIP type.

ClusterIP: The Classic

Introduction

We’re going to take a look at the ClusterIP Service type first because it’s the default type of Service – if the spec.type field is left unspecified in the Service definition, then Kubernetes will automatically create a ClusterIP-type Service for you and assign a cluster-internal, stable, virtual IP from its pool of IP addresses, which is visible in the spec.clusterIP field after creation (you could also specify this IP yourself in the Service definition, but that’s not a very common use case). The emphasis here is on cluster-internal since the ClusterIP type does not allow external traffic to reach the Service (unless you employ what I called “tricks” above, namely, port forwarding or proxying, which we’ll take a look at a bit further down the line).

Sample Manifest

A typical Service specification for the ClusterIP type might look like this (the following is an excerpt from a sample manifest I’ve created to go along with this blog post):

apiVersion: v1
kind: Service
metadata:
  name: hello-service-clusterip
  namespace: service-types-example
spec:
  # Could also be omitted since 'ClusterIP' is the default
  type: ClusterIP
  selector:
    app: hello-app
  ports:
    - name: http
      port: 8081

You can apply the entire manifest like so:

$ alias k=kubectl
$ k apply -f https://raw.githubusercontent.com/AntsInMyEy3sJohnson/blog-examples/master/kubernetes/workload-reachability/deployment-with-various-service-types.yaml
namespace/service-types-example created
deployment.apps/hello-service created
service/hello-service-clusterip created
service/hello-service-nodeport created
service/hello-service-loadbalancer created
service/hello-service-headless-with-selector created

You’ll notice this command creates four Services, and that’s because the given manifest contains all we need for the entire blog post. For now, we’ll focus on the ClusterIP Service, which is the Service aptly called hello-service-clusterip:

$ k -n service-types-example get svc --selector type=clusterip
NAME                      TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)    AGE
hello-service-clusterip   ClusterIP   10.43.195.63   <none>        8081/TCP   10m

By now, it probably won’t hit you as a surprise that no external IP has been allocated, but Kubernetes has assigned a cluster-internal IP, the cluster IP, to that Service, which clients can refer to using the Service name, and all traffic received on that IP address will be load-balanced across all Pods identified by the Service’s selector.

ClusterIP And External Traffic

The statement there is no external access to a ClusterIP Service is not entirely precise – it is possible to direct external traffic to a ClusterIP service by means of port-forwarding or proxying, but as you’ll see, these are not good choices for production workloads and should really only be used for testing and debugging purposes.

With that being said, let’s take a look at port-forwarding. The way of doing this is by means of the kubectl port-forward command, which is able to direct traffic either directly to a Pod or to a Service. In one terminal, type the following command:

$ k -n service-types-example port-forward svc/hello-service-clusterip 8081:8081
Forwarding from 127.0.0.1:8081 -> 8081
Forwarding from [::1]:8081 -> 8081

If your output is akin to the above, the port-forwarding will be active. Let’s invoke our Service from another terminal:

$ curl localhost:8081
Hello from port 8081

There it is, a response from one of the Pods! This is an incredibly quick and easy way to invoke a workload running inside a cluster.

Another option is to use the kubectl proxy command. Stop the port-forward command and re-use the terminal to issue the proxy command:

$ k proxy --port 8081

This will essentially proxy the entire Kubernetes API server to localhost on the given port, and we can use it to invoke all Services present in the cluster. With the proxy command active in the first terminal, issue the following command in another terminal:

$ curl localhost:8081/api/v1/namespaces/service-types-example/services/hello-service-clusterip:http/proxy/
Hello from port 8081

Downsides Of Port-Forwarding And Proxying For External Access

Both kubectl port-forward and kubectl proxy are great for testing and debugging a Service, but they both have major caveats:

  • The connection is only opened on the machine running the kubectl port-forward or kubectl proxy command
  • For the commands to work, kubectl needs to authenticate with the cluster’s API server, so there will be a kubeconfig file containing authentication details floating around somewhere on the host’s filesystem, which is something you probably want to avoid unless that host really needs access to the API server
  • In the case of kubectl proxy, manually constructing the Service URL is cumbersome and unnecessary work in view of better alternatives (which will be introduced in the following sections)

It is clear, then, that neither of these two commands should be used to expose a Service in a production setting.

NodePort: The Portable

A better way to open up your Services to the external world is by means of the NodePort Service type, which builds on ClusterIP. NodePort is very portable since it does not depend on load balancer integration or other things to function (as the LoadBalancer type does, for example).

Introduction

Specifying the NodePort Service type will make Kubernetes expose a static port on each node in its cluster and then forward all traffic received on that port to the NodePort Service. The port to expose on each node can either be defined by the user in the Service specification, or Kubernetes will assign one randomly out of a range from 30.000 to 32.767 (that’s the default range, anyway, which the cluster admin can change by means of the --service-node-port-range flag). This means if a cluster-external client application knows the port opened on the nodes, it can send requests to that IP-port combination, and they will be forwarded to the target workload. The client application does not need to be concerned with which nodes the workload Pods are running on, nor how many Pods there currently are – those details are nicely abstracted by the NodePort Service.

Sample Manifest

Let’s again refer to the sample manifest. Among other things, you’ll find a typical NodePort Service there:

apiVersion: v1
kind: Service
metadata:
  name: hello-service-nodeport
  namespace: service-types-example
spec:
  type: NodePort
  selector:
    app: hello-app
  ports:
    - name: http8080
      port: 8080
      # Will assume the value of 'port' if unspecified
      targetPort: http8080
      nodePort: 30080
    - name: http8081
      port: 8081
      # Will assume the value of 'port' if unspecified
      targetPort: http8081
      nodePort: 30081

As you’ve probably expected, the type is now NodePort rather than ClusterIP. Another thing you might have noticed is there are three different port-related properties for each port definition in there! That may be a bit confusing and we’ll get to that in just a minute, but let’s examine the NodePort Service previously created for us:

$ k -n service-types-example get svc --selector type=nodeport
NAME                     TYPE       CLUSTER-IP      EXTERNAL-IP   PORT(S)                         AGE
hello-service-nodeport   NodePort   10.43.108.188   <none>        8080:30080/TCP,8081:30081/TCP   6m3s

This particular Services exposes two ports, namely, port 30080 with target port 8080, and port 30081 with target port 8081. There is still no external IP, though, but that makes sense since the external IP for us to invoke is the IP of one the cluster’s worker nodes. In my case, one of the worker’s IPs is 10.211.55.6:

$ curl 10.211.55.6:30080
Hello from port 8080

$ curl 10.211.55.6:30081
Hello from port 8081

Hooray! We can see in the Deployment specification (see sample manifest) the workload actually responds like this, so the external communication works just fine.

port vs. targetPort vs. nodePort

I’ve promised you we’d get back to the three port-related properties you’ve seen above, and this paragraph will provide some additional explanations.

Imagine you had an application that exposes a small web server on port 8080, and since you’re awesome and want to run it on Kubernetes, you’ve containerized it and set up a Pod manifest specifying the containerPort to be 8080. Now to expose this workload using a NodePort Service, that Service needs to know three things:

  1. Which port to direct traffic to, i.e., which port is actually available on the target Pods to serve incoming requests
  2. Which port should be used for cluster-internal communication by clients wishing to invoke the workload Pods
  3. And, finally, which port to expose on the cluster nodes to allow external traffic to reach the target Pods

The description of the first kind of port indicates this must be the targetPort. You won’t see this on a Service spec very often, though, because for convenience and simplicity, it is often omitted, and will assume the value given for port behind the curtains (as mentioned elsewhere). The port property defines cluster-internal communication and thus covers the second point above. Finally, if you think about cluster-internal communication versus traffic from outside the cluster, it seems to make sense to separate the handling of both in the Service definition by introducing a dedicated property for each type of communication. Recall that the nodePort is opened on each of the cluster’s nodes and it becomes obvious it would be unelegant to open frequently used ports on those nodes (such as 8080), but similarly, it would be inconvenient to employ more unusual port numbers (such as 30080) for cluster-internal communication. Hence, the nodePort port is taken from a dedicated range, but that’s only possible due to the separation of concerns introduced by employing two distinct port properties.

By the way: The hello-service-clusterip Service you can find in the sample manifest is an example for a Service specification omitting the targetPort property, and once you’ve applied the manifest, you can see the targetPort property has been added for us, carrying the value given for port.

Downsides Of NodePort For External Access

Exposing workloads to the external world by means of NodePort-type Services is hugely advantageous compared to the port-forwarding or proxying we had to do with ClusterIP Services, but there are a couple of disadvantages, too:

  • A NodePort can only be allocated within a certain range (30.000 to 32.767, by default)
  • The change of one or multiple node IPs constitutes a breaking change for all external clients making use of the NodePort Service
  • External clients still have to invoke the workloads by means of IP addresses (unless they employ some DNS-ing on their own or you put a load balancer in front of the node IPs)
  • For those opened ports on the nodes to be useful in terms of allowing external traffic to reach the Service, the node IPs have to be publicly reachable (at least one of them, anyway) – not a good setup for security reasons
  • Each node port can only serve one Service port – if a NodePort-type Service exposes two ports (like the one you’ve worked with above), then that’s two node ports allocated
  • With an increasing number of Services and node ports exposed, it becomes more and more difficult to keep track of which workload is reachable under which port

For these reasons, although the NodePort Service is a fantastic solution for both testing and debugging and even for temporary scenarios like prototypes or demo settings, it’s still not ideal for production use cases.

LoadBalancer: The Expensive

Finally, let’s examine the solution that’s the most powerful, but also the most expensive in this threesome: the LoadBalancer type. You’ve already had a bit of exposure to the LoadBalancer type because in the last blog post, I’ve used it to make calling the sample Service shown there more convenient for you.

Introduction

Assuming that the environment your Kubernetes cluster runs in – most often, this will be with a cloud provider like GKE, AWS or Azure – supports it, applying a LoadBalancer Service will provision or configure an actual load balancer for the Service and assign a static IP or hostname to it, which your client applications can then refer to to invoke all Pods identified by the Service’s label selector.

Just like NodePort builds on ClusterIP, LoadBalancer extends and thus builds on NodePort, and the load balancer provisioned for your Service will direct all traffic received on the combination of external IP address (or hostname) and port to whatever node ports have been opened on the cluster’s nodes. Of course, that means a node port is automatically allocated for you once you’ve applied the Service.

Sample Manifest

You can find an example for a LoadBalancer Service in the manifest provided over on my GitHub. This is the Service specification:

apiVersion: v1
kind: Service
metadata:
  name: hello-service-loadbalancer
  namespace: service-types-example
  labels:
    type: loadbalancer
spec: 
  type: LoadBalancer
  selector: 
    app: hello-app
  ports:
  - name: http8080
    port: 8080
  - name: http8081
    port: 8081

You may be surprised to see the only relevant difference in this manifest compared to its ClusterIP sibling is the Service type. I think the LoadBalancer Service type is one of the examples of just how powerful Kubernetes can be: All it needs to make a workload reachable from anywhere on the planet is just a couple of lines of Yaml like the ones above (and a decent cloud provider)! Sure, there is a lot more going on behind the scenes, but if everything is configured correctly, all that complexity is abstracted from you, and you only need to submit a Service like the one above to your cluster.

After you’ve applied the manifest (k apply -f https://raw.githubusercontent.com/AntsInMyEy3sJohnson/blog-examples/master/kubernetes/workload-reachability/deployment-with-various-service-types.yaml), give the cloud provider of your choice some time, and then query the Service:

$ k -n service-types-example get svc --selector type=loadbalancer
NAME                         TYPE           CLUSTER-IP     EXTERNAL-IP   PORT(S)                         AGE
hello-service-loadbalancer   LoadBalancer   10.43.53.239   10.211.55.6   8080:30733/TCP,8081:30973/TCP   27m

Hooray, an external IP to invoke the workload! Meanwhile, the PORT(S) column hints us at the fact there is a node port behind the scenes for each Service port. Indeed, if you query the Service’s manifest after it has been applied, you’ll see a nodePort property has automatically been added to each port the Service defines, and, of course, since NodePort builds on ClusterIP, there is also a cluster IP:

# ...
spec:
  clusterIP: 10.43.53.239
  clusterIPs:
  - 10.43.53.239
  # ...
  ports:
  - name: http8080
    nodePort: 30733
    port: 8080
    protocol: TCP
    targetPort: 8080
  - name: http8081
    nodePort: 30973
    port: 8081
    protocol: TCP
    targetPort: 8081
  # ...
# ...

What we can do now is simply invoke the Pods this Service targets by means of its label selector using the external IP. For example:

$ curl 10.211.55.6:8080
Hello from port 8080

$ curl 10.211.55.6:8081
Hello from port 8081

That’s awesome! That is, until the cloud provider sends you the bill for all the fun… which brings us to the downside of working with the LoadBalancer Service type.

Downsides Of LoadBalancer For External Access

The LoadBalancer Service type is expensive, literally. That’s because cloud providers typically charge for each LoadBalancer Service you have in your Kubernetes cluster. Consider a large cluster with hundreds of Services and you can probably guess using the LoadBalancer type for each of them will make the cloud provider very happy, but your management significantly less so (or you, if you’re unlucky enough to have to pay the bill yourself).

Plus, let’s not forget LoadBalancer is simply NodePort on steroids, so the drawbacks of the latter still apply (albeit much less so since the nodes don’t have to be exposed publicly anymore and because clients now invoke the load balancer’s IP or hostname, they don’t care about the node’s IP addresses and the individual node ports).

So, there has to be a better solution for providing external access to your awesome workloads – and there is, namely, in the form of the Ingress resource, which you’ll learn more about in the next blog post. For now, though, let’s take a look at the more unusual candidates in the world of Services.

More Unusual Suspects

What you’ve seen so far is Service types designed to load-balance traffic directed at cluster-internal workloads with varying degrees of convenience for allowing external traffic to hit the workload. Load-balancing traffic to cluster-internal workloads is the most common use case, and thus, logically, Service types enabling this become the most common types. There are use cases, though, the Service types introduced so far are not a good fit for:

  • The workload you would like to reach sits somewhere outside the cluster
  • For cluster-internal traffic, you actually don’t need load balancing across the Pod IPs

Kubernetes addresses those use cases through ExternalName and headless Services. Let’s start with the former.

ExternalName

The ExternalName Service type simply maps a Service to an existing DNS name. To define that name, you specify it using the spec.externalName property, which expects a lower-case RFC-1123 hostname. (You can also specify a single IPv4 address here as a string, but in this case, you’d be better off using a headless Service instead, see below.) The ExternalName type is the only type what works in combination with the spec.externalName field.

It follows from the use case the ExternalName type addresses that defining a selector to identify target Pods does not make sense since the target workload always lies outside the cluster. Thus, although a selector can be specified on an ExternalName Service, Kubernetes will ignore it.

Headless

Before we get started with what a headless Service is, let’s go ahead and ask the built-in docs about the various Service types using the kubectl explain command:

$ k explain service.spec.type
# ...
type determines how the Service is exposed. Defaults to ClusterIP. Valid
options are ExternalName, ClusterIP, NodePort, and LoadBalancer.
# ...

You’ll notice there is no Headless type available, and indeed, a Service is made headless by explicitly setting its spec.clusterIP field to None rather than by providing a specific type. The effect of None for the cluster IP is that Kubernetes will not allocate a virtual IP, and no load balancing or proxying will be done for that Service.

Use Cases For Headless Services

Headless Services are a great solution if you face one of the following two challenges:

Backend Pods have state: The assumption Kubernetes makes about Pods is that they are ephemeral and stateless and thus don’t have an identity (as in: the client does not care which Pod replica it connects to). This assumption does not work well for stateful workloads such as databases, because for this kind workload, the Pods have state and identity, and talking to the correct Pod matters. Therefore, clients may have to connect to one specific pod, and this cannot be done with a “standard” Service as it will load-balance across its endpoint IPs. This scenario is addressed with the help of headless Services: If a client calls a headless Service by its name, it will receive the IPs of all pods identified by this Service. Consequently, the client can make an informed decision about which Pod or Pods to connect to.

Backend is outside of the cluster: Sometimes, you may have to connect to a legacy application running somewhere outside the cluster only reachable with an IP rather than a hostname. As you’ve learned above, the ExternalName type helps in this case, but it works best if the “external name” is a canonical DNS name, and the Kubernetes docs recommend using a headless Service if the goal is to hard-code an IP. To make this work, though, a correct Endpoints object has to be created and mapped to the Service manually, see below.

Headless Services And The Endpoints Object

You’ve learned in scope of the previous blog post that Kubernetes will automatically create an Endpoints object for each Service unless the Service does not specify a selector – in this case, because there is no way to identify the correct set of target Pods, there are no IPs to populate the Endpoints object with, and so it would be pointless to allocate one. Specifying a Service without a selector is rather unusual for the three most common Service types, but things are different for headless Services. Basically, whether you have to specify a selector for a headless Service depends on which of the aforementioned use cases you want to employ the Service for:

  • If you use the headless Service because your backend Pods have state and you want the Service to return all Pod IPs rather than perform load-balancing, you’ll have to specify a selector. In this case, as usual, Kubernetes will create a buddy Endpoints object.
  • On the other hand, if your use case is to connect to an application running outside the cluster via a single IP, omit the selector on the headless Service since there are no Pods inside the cluster this selector could or should identify. However, because the way a Service “knows” which IP or IPs to forward traffic to is through an Endpoints object, you’ll have to create one manually in this case and map it to the Service by giving it the same name as the Service (one of the rare cases in Kubernetes where the relationship between two objects is not defined by labels).

Sample Manifest

As you’ve probably guessed, there is a sample Service definition available in the manifest over here on my GitHub. Let’s take a look:

apiVersion: v1
kind: Service
metadata:
  name: hello-service-headless-with-selector
  namespace: service-types-example
  labels: 
    type: headless
spec:
  clusterIP: None
  selector:
    app: hello-app
  ports:
  - name: http8080
    port: 8080

There are two little details worth noting:

  • As mentioned previously, there is no spec.type: Headless, rather, a Service carrying one of the other available types (here, I’ve simply gone for “the classic”) is made headless by…
  • … defining the spec.clusterIP field to be None.

According to the clusterIP: None definition, you probably won’t be surprised to see Kubernetes, in fact, did not assign a cluster IP for this Service:

$ k -n service-types-example get svc --selector type=headless
NAME                                   TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)    AGE
hello-service-headless-with-selector   ClusterIP   None         <none>        8080/TCP   6m2s

Since this headless Service defines a selector, Kubernetes has created a buddy Endpoints object as usual, but because it has not allocated a cluster IP, we can use that headless Service to find out the individual IPs of the Service’s Pods. For this, we’ll launch a little helper Pod coming with nslookup in the same namespace:

$ k -n service-types-example run -it --generator=run-pod/v1 --image=registry.access.redhat.com/rhel7/rhel-tools -- bash

Once you’re inside the Pod, query the ClusterIP Service first:

[root@bash /]# nslookup hello-service-clusterip
Server:		10.43.0.10
Address:	10.43.0.10#53

Name:	hello-service-clusterip.service-types-example.svc.cluster.local
Address: 10.43.81.68

The query returns the virtual cluster IP of the Service. Now, how about the headless Service?

[root@bash /]# nslookup hello-service-headless-with-selector
Server:		10.43.0.10
Address:	10.43.0.10#53

Name:	hello-service-headless-with-selector.service-types-example.svc.cluster.local
Address: 10.42.0.180
Name:	hello-service-headless-with-selector.service-types-example.svc.cluster.local
Address: 10.42.0.182
Name:	hello-service-headless-with-selector.service-types-example.svc.cluster.local
Address: 10.42.0.181

Indeed, this query returns the individual name and address of each Pod the Service’s label selector identifies. Thus, if we were talking to a stateful application running in the backend made up of those Pods, we could now make an informed decision about which Pod to talk to.

Cleaning Up

You can delete everything created in scope of this blog post using the following command:

$ k delete -f https://raw.githubusercontent.com/AntsInMyEy3sJohnson/blog-examples/master/kubernetes/workload-reachability/deployment-with-various-service-types.yaml

Summary

Phew, that’s a lot of content to digest. Let’s do a short recap:

The three most common Service types in Kubernetes are ClusterIP, NodePort, and LoadBalancer, which build on each other to offer their distinct behavior. The ClusterIP type allocates a virtual IP called the “cluster IP” (how aptly named, isn’t it?) that allows cluster-internal load-balancing across the Pods identified by the Service’s selector. External access can be provided through the kubectl port-forward and kubectl proxy commands, which work well for testing and debugging, but are not appropriate for anything beyond that. NodePort builds on ClusterIP and enables external access to cluster-internal workloads by opening the given node port on each of the cluster’s nodes. LoadBalancer, in turn, leverages NodePort by provisioning an actual load balancer with a public IP or hostname which directs all traffic to the exposed node ports, but this only works if the environment the Kubernetes cluster runs in supports it (as most cloud providers do).

Other Service types are ExternalName and headless. The use cases they address occur more rarely, hence those Service types are less common. The ExternalName type is used to connect to cluster-external workloads using a DNS name, and headless Services are employed if (a) clients need to know the individual Pod IPs or (b) an external resource is to be connected using a single IPv4 address. The headless Service is not a type specified by means of the spec.type property; rather, a Service is made headless by explicitly configuring its spec.clusterIP field to None (hence the different text styling for the word headless versus ExternalName and the other Service types).

Although the LoadBalancer Service type is fantastically easy to use to expose your workloads externally (thanks to the amazing work of the cloud provider’s engineers to do all sorts of configuration work behind the curtains), its usage can become very expensive if many Services of that type are used, thus for exposing many different workloads to the external world, the LoadBalancer type is still not ideal. But, of course, Kubernetes has you covered: In the form of the Ingress resource, it provides a means to expose a high number of workloads in a simple, reliable, and efficient way, and you’ll learn more about the Ingress resource in the next blog post.