Workload Reachability 2: Service Types
Service object is the foundation for DNS and load-balancing in Kubernetes, and there are different Service types available to fit various use cases, including allowing for external traffic to reach cluster-internal workloads or making an external workload available from within the cluster.
The previous blog post introduced you to the basics of the Service object and what problem it solves in Kubernetes, so let’s now spice things up a little and introduce the various different Service types to the party!
- The Most Common Service Types
- ClusterIP: The Classic
- NodePort: The Portable
- LoadBalancer: The Expensive
- More Unusual Suspects
- Cleaning Up
Before we get started: There will be a couple of examples in the upcoming sections, and to follow along, you’ll need your own Kubernetes cluster. In case you don’t have access to one yet, please refer to this section in one of my previous blog posts to help you get started.
The Most Common Service Types
The three types of Services you’ll probably encounter most often are
LoadBalancer, and they build on each other to provide their specific behavior. You’ll learn about this in the upcoming sections, but here’s a minimal overview to get you prepared:
ClusterIPis the default Service type and it works with virtual cluster IPs, which are only reachable from within the cluster (without using tricks, anyway)
- To address use cases involving traffic from outside the cluster, there are two more types of Service Kubernetes has in store for you, namely,
LoadBalancerallow directing external traffic into the cluster (without the need for tricks), and they have their own pros and cons, which make them suitable for different situations
We’ll take a look at each of these Service types in the upcoming sections from the perspective of enabling external access. Let’s get started with the classic: the
ClusterIP: The Classic
We’re going to take a look at the
ClusterIP Service type first because it’s the default type of Service – if the
spec.type field is left unspecified in the Service definition, then Kubernetes will automatically create a
ClusterIP-type Service for you and assign a cluster-internal, stable, virtual IP from its pool of IP addresses, which is visible in the
spec.clusterIP field after creation (you could also specify this IP yourself in the Service definition, but that’s not a very common use case). The emphasis here is on cluster-internal since the
ClusterIP type does not allow external traffic to reach the Service (unless you employ what I called “tricks” above, namely, port forwarding or proxying, which we’ll take a look at a bit further down the line).
A typical Service specification for the
ClusterIP type might look like this (the following is an excerpt from a sample manifest I’ve created to go along with this blog post):
apiVersion: v1 kind: Service metadata: name: hello-service-clusterip namespace: service-types-example spec: # Could also be omitted since 'ClusterIP' is the default type: ClusterIP selector: app: hello-app ports: - name: http port: 8081
You can apply the entire manifest like so:
$ alias k=kubectl $ k apply -f https://raw.githubusercontent.com/AntsInMyEy3sJohnson/blog-examples/master/kubernetes/workload-reachability/deployment-with-various-service-types.yaml namespace/service-types-example created deployment.apps/hello-service created service/hello-service-clusterip created service/hello-service-nodeport created service/hello-service-loadbalancer created service/hello-service-headless-with-selector created
You’ll notice this command creates four Services, and that’s because the given manifest contains all we need for the entire blog post. For now, we’ll focus on the
ClusterIP Service, which is the Service aptly called
$ k -n service-types-example get svc --selector type=clusterip NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE hello-service-clusterip ClusterIP 10.43.195.63 <none> 8081/TCP 10m
By now, it probably won’t hit you as a surprise that no external IP has been allocated, but Kubernetes has assigned a cluster-internal IP, the cluster IP, to that Service, which clients can refer to using the Service name, and all traffic received on that IP address will be load-balanced across all Pods identified by the Service’s selector.
ClusterIP And External Traffic
The statement there is no external access to a
ClusterIP Service is not entirely precise – it is possible to direct external traffic to a
ClusterIP service by means of port-forwarding or proxying, but as you’ll see, these are not good choices for production workloads and should really only be used for testing and debugging purposes.
With that being said, let’s take a look at port-forwarding. The way of doing this is by means of the
kubectl port-forward command, which is able to direct traffic either directly to a Pod or to a Service. In one terminal, type the following command:
$ k -n service-types-example port-forward svc/hello-service-clusterip 8081:8081 Forwarding from 127.0.0.1:8081 -> 8081 Forwarding from [::1]:8081 -> 8081
If your output is akin to the above, the port-forwarding will be active. Let’s invoke our Service from another terminal:
$ curl localhost:8081 Hello from port 8081
There it is, a response from one of the Pods! This is an incredibly quick and easy way to invoke a workload running inside a cluster.
Another option is to use the
kubectl proxy command. Stop the
port-forward command and re-use the terminal to issue the
$ k proxy --port 8081
This will essentially proxy the entire Kubernetes API server to localhost on the given port, and we can use it to invoke all Services present in the cluster. With the
proxy command active in the first terminal, issue the following command in another terminal:
$ curl localhost:8081/api/v1/namespaces/service-types-example/services/hello-service-clusterip:http/proxy/ Hello from port 8081
Downsides Of Port-Forwarding And Proxying For External Access
kubectl port-forward and
kubectl proxy are great for testing and debugging a Service, but they both have major caveats:
- The connection is only opened on the machine running the
- For the commands to work,
kubectlneeds to authenticate with the cluster’s API server, so there will be a kubeconfig file containing authentication details floating around somewhere on the host’s filesystem, which is something you probably want to avoid unless that host really needs access to the API server
- In the case of
kubectl proxy, manually constructing the Service URL is cumbersome and unnecessary work in view of better alternatives (which will be introduced in the following sections)
It is clear, then, that neither of these two commands should be used to expose a Service in a production setting.
NodePort: The Portable
A better way to open up your Services to the external world is by means of the
NodePort Service type, which builds on
NodePort is very portable since it does not depend on load balancer integration or other things to function (as the
LoadBalancer type does, for example).
NodePort Service type will make Kubernetes expose a static port on each node in its cluster and then forward all traffic received on that port to the
NodePort Service. The port to expose on each node can either be defined by the user in the Service specification, or Kubernetes will assign one randomly out of a range from 30.000 to 32.767 (that’s the default range, anyway, which the cluster admin can change by means of the
--service-node-port-range flag). This means if a cluster-external client application knows the port opened on the nodes, it can send requests to that IP-port combination, and they will be forwarded to the target workload. The client application does not need to be concerned with which nodes the workload Pods are running on, nor how many Pods there currently are – those details are nicely abstracted by the
Let’s again refer to the sample manifest. Among other things, you’ll find a typical
NodePort Service there:
apiVersion: v1 kind: Service metadata: name: hello-service-nodeport namespace: service-types-example spec: type: NodePort selector: app: hello-app ports: - name: http8080 port: 8080 # Will assume the value of 'port' if unspecified targetPort: http8080 nodePort: 30080 - name: http8081 port: 8081 # Will assume the value of 'port' if unspecified targetPort: http8081 nodePort: 30081
As you’ve probably expected, the type is now
NodePort rather than
ClusterIP. Another thing you might have noticed is there are three different port-related properties for each port definition in there! That may be a bit confusing and we’ll get to that in just a minute, but let’s examine the
NodePort Service previously created for us:
$ k -n service-types-example get svc --selector type=nodeport NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE hello-service-nodeport NodePort 10.43.108.188 <none> 8080:30080/TCP,8081:30081/TCP 6m3s
This particular Services exposes two ports, namely, port
30080 with target port
8080, and port
30081 with target port
8081. There is still no external IP, though, but that makes sense since the external IP for us to invoke is the IP of one the cluster’s worker nodes. In my case, one of the worker’s IPs is
$ curl 10.211.55.6:30080 Hello from port 8080 $ curl 10.211.55.6:30081 Hello from port 8081
Hooray! We can see in the Deployment specification (see sample manifest) the workload actually responds like this, so the external communication works just fine.
I’ve promised you we’d get back to the three port-related properties you’ve seen above, and this paragraph will provide some additional explanations.
Imagine you had an application that exposes a small web server on port 8080, and since you’re awesome and want to run it on Kubernetes, you’ve containerized it and set up a Pod manifest specifying the
containerPort to be
8080. Now to expose this workload using a
NodePort Service, that Service needs to know three things:
- Which port to direct traffic to, i.e., which port is actually available on the target Pods to serve incoming requests
- Which port should be used for cluster-internal communication by clients wishing to invoke the workload Pods
- And, finally, which port to expose on the cluster nodes to allow external traffic to reach the target Pods
The description of the first kind of port indicates this must be the
targetPort. You won’t see this on a Service spec very often, though, because for convenience and simplicity, it is often omitted, and will assume the value given for
port behind the curtains (as mentioned elsewhere). The
port property defines cluster-internal communication and thus covers the second point above. Finally, if you think about cluster-internal communication versus traffic from outside the cluster, it seems to make sense to separate the handling of both in the Service definition by introducing a dedicated property for each type of communication. Recall that the
nodePort is opened on each of the cluster’s nodes and it becomes obvious it would be unelegant to open frequently used ports on those nodes (such as
8080), but similarly, it would be inconvenient to employ more unusual port numbers (such as
30080) for cluster-internal communication. Hence, the
nodePort port is taken from a dedicated range, but that’s only possible due to the separation of concerns introduced by employing two distinct port properties.
By the way: The
hello-service-clusterip Service you can find in the sample manifest is an example for a Service specification omitting the
targetPort property, and once you’ve applied the manifest, you can see the
targetPort property has been added for us, carrying the value given for
NodePort For External Access
Exposing workloads to the external world by means of
NodePort-type Services is hugely advantageous compared to the port-forwarding or proxying we had to do with
ClusterIP Services, but there are a couple of disadvantages, too:
NodePortcan only be allocated within a certain range (30.000 to 32.767, by default)
- The change of one or multiple node IPs constitutes a breaking change for all external clients making use of the
- External clients still have to invoke the workloads by means of IP addresses (unless they employ some DNS-ing on their own or you put a load balancer in front of the node IPs)
- For those opened ports on the nodes to be useful in terms of allowing external traffic to reach the Service, the node IPs have to be publicly reachable (at least one of them, anyway) – not a good setup for security reasons
- Each node port can only serve one Service port – if a
NodePort-type Service exposes two ports (like the one you’ve worked with above), then that’s two node ports allocated
- With an increasing number of Services and node ports exposed, it becomes more and more difficult to keep track of which workload is reachable under which port
For these reasons, although the
NodePort Service is a fantastic solution for both testing and debugging and even for temporary scenarios like prototypes or demo settings, it’s still not ideal for production use cases.
LoadBalancer: The Expensive
Finally, let’s examine the solution that’s the most powerful, but also the most expensive in this threesome: the
LoadBalancer type. You’ve already had a bit of exposure to the
LoadBalancer type because in the last blog post, I’ve used it to make calling the sample Service shown there more convenient for you.
Assuming that the environment your Kubernetes cluster runs in – most often, this will be with a cloud provider like GKE, AWS or Azure – supports it, applying a
LoadBalancer Service will provision or configure an actual load balancer for the Service and assign a static IP or hostname to it, which your client applications can then refer to to invoke all Pods identified by the Service’s label selector.
NodePort builds on
LoadBalancer extends and thus builds on
NodePort, and the load balancer provisioned for your Service will direct all traffic received on the combination of external IP address (or hostname) and port to whatever node ports have been opened on the cluster’s nodes. Of course, that means a node port is automatically allocated for you once you’ve applied the Service.
You can find an example for a
LoadBalancer Service in the manifest provided over on my GitHub. This is the Service specification:
apiVersion: v1 kind: Service metadata: name: hello-service-loadbalancer namespace: service-types-example labels: type: loadbalancer spec: type: LoadBalancer selector: app: hello-app ports: - name: http8080 port: 8080 - name: http8081 port: 8081
You may be surprised to see the only relevant difference in this manifest compared to its
ClusterIP sibling is the Service type. I think the
LoadBalancer Service type is one of the examples of just how powerful Kubernetes can be: All it needs to make a workload reachable from anywhere on the planet is just a couple of lines of Yaml like the ones above (and a decent cloud provider)! Sure, there is a lot more going on behind the scenes, but if everything is configured correctly, all that complexity is abstracted from you, and you only need to submit a Service like the one above to your cluster.
After you’ve applied the manifest (
k apply -f https://raw.githubusercontent.com/AntsInMyEy3sJohnson/blog-examples/master/kubernetes/workload-reachability/deployment-with-various-service-types.yaml), give the cloud provider of your choice some time, and then query the Service:
$ k -n service-types-example get svc --selector type=loadbalancer NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE hello-service-loadbalancer LoadBalancer 10.43.53.239 10.211.55.6 8080:30733/TCP,8081:30973/TCP 27m
Hooray, an external IP to invoke the workload! Meanwhile, the
PORT(S) column hints us at the fact there is a node port behind the scenes for each Service port. Indeed, if you query the Service’s manifest after it has been applied, you’ll see a
nodePort property has automatically been added to each port the Service defines, and, of course, since
NodePort builds on
ClusterIP, there is also a cluster IP:
# ... spec: clusterIP: 10.43.53.239 clusterIPs: - 10.43.53.239 # ... ports: - name: http8080 nodePort: 30733 port: 8080 protocol: TCP targetPort: 8080 - name: http8081 nodePort: 30973 port: 8081 protocol: TCP targetPort: 8081 # ... # ...
What we can do now is simply invoke the Pods this Service targets by means of its label selector using the external IP. For example:
$ curl 10.211.55.6:8080 Hello from port 8080 $ curl 10.211.55.6:8081 Hello from port 8081
That’s awesome! That is, until the cloud provider sends you the bill for all the fun… which brings us to the downside of working with the
LoadBalancer Service type.
LoadBalancer For External Access
LoadBalancer Service type is expensive, literally. That’s because cloud providers typically charge for each
LoadBalancer Service you have in your Kubernetes cluster. Consider a large cluster with hundreds of Services and you can probably guess using the
LoadBalancer type for each of them will make the cloud provider very happy, but your management significantly less so (or you, if you’re unlucky enough to have to pay the bill yourself).
Plus, let’s not forget
LoadBalancer is simply
NodePort on steroids, so the drawbacks of the latter still apply (albeit much less so since the nodes don’t have to be exposed publicly anymore and because clients now invoke the load balancer’s IP or hostname, they don’t care about the node’s IP addresses and the individual node ports).
So, there has to be a better solution for providing external access to your awesome workloads – and there is, namely, in the form of the
Ingress resource, which you’ll learn more about in the next blog post. For now, though, let’s take a look at the more unusual candidates in the world of Services.
More Unusual Suspects
What you’ve seen so far is Service types designed to load-balance traffic directed at cluster-internal workloads with varying degrees of convenience for allowing external traffic to hit the workload. Load-balancing traffic to cluster-internal workloads is the most common use case, and thus, logically, Service types enabling this become the most common types. There are use cases, though, the Service types introduced so far are not a good fit for:
- The workload you would like to reach sits somewhere outside the cluster
- For cluster-internal traffic, you actually don’t need load balancing across the Pod IPs
Kubernetes addresses those use cases through
ExternalName and headless Services. Let’s start with the former.
ExternalName Service type simply maps a Service to an existing DNS name. To define that name, you specify it using the
spec.externalName property, which expects a lower-case RFC-1123 hostname. (You can also specify a single IPv4 address here as a string, but in this case, you’d be better off using a headless Service instead, see below.) The
ExternalName type is the only type what works in combination with the
It follows from the use case the
ExternalName type addresses that defining a selector to identify target Pods does not make sense since the target workload always lies outside the cluster. Thus, although a selector can be specified on an
ExternalName Service, Kubernetes will ignore it.
Before we get started with what a headless Service is, let’s go ahead and ask the built-in docs about the various Service types using the
kubectl explain command:
$ k explain service.spec.type # ... type determines how the Service is exposed. Defaults to ClusterIP. Valid options are ExternalName, ClusterIP, NodePort, and LoadBalancer. # ...
You’ll notice there is no
Headless type available, and indeed, a Service is made headless by explicitly setting its
spec.clusterIP field to
None rather than by providing a specific type. The effect of
None for the cluster IP is that Kubernetes will not allocate a virtual IP, and no load balancing or proxying will be done for that Service.
Use Cases For Headless Services
Headless Services are a great solution if you face one of the following two challenges:
Backend Pods have state: The assumption Kubernetes makes about Pods is that they are ephemeral and stateless and thus don’t have an identity (as in: the client does not care which Pod replica it connects to). This assumption does not work well for stateful workloads such as databases, because for this kind workload, the Pods have state and identity, and talking to the correct Pod matters. Therefore, clients may have to connect to one specific pod, and this cannot be done with a “standard” Service as it will load-balance across its endpoint IPs. This scenario is addressed with the help of headless Services: If a client calls a headless Service by its name, it will receive the IPs of all pods identified by this Service. Consequently, the client can make an informed decision about which Pod or Pods to connect to.
Backend is outside of the cluster: Sometimes, you may have to connect to a legacy application running somewhere outside the cluster only reachable with an IP rather than a hostname. As you’ve learned above, the
ExternalName type helps in this case, but it works best if the “external name” is a canonical DNS name, and the Kubernetes docs recommend using a headless Service if the goal is to hard-code an IP. To make this work, though, a correct Endpoints object has to be created and mapped to the Service manually, see below.
Headless Services And The Endpoints Object
You’ve learned in scope of the previous blog post that Kubernetes will automatically create an Endpoints object for each Service unless the Service does not specify a selector – in this case, because there is no way to identify the correct set of target Pods, there are no IPs to populate the Endpoints object with, and so it would be pointless to allocate one. Specifying a Service without a selector is rather unusual for the three most common Service types, but things are different for headless Services. Basically, whether you have to specify a selector for a headless Service depends on which of the aforementioned use cases you want to employ the Service for:
- If you use the headless Service because your backend Pods have state and you want the Service to return all Pod IPs rather than perform load-balancing, you’ll have to specify a selector. In this case, as usual, Kubernetes will create a buddy Endpoints object.
- On the other hand, if your use case is to connect to an application running outside the cluster via a single IP, omit the selector on the headless Service since there are no Pods inside the cluster this selector could or should identify. However, because the way a Service “knows” which IP or IPs to forward traffic to is through an Endpoints object, you’ll have to create one manually in this case and map it to the Service by giving it the same name as the Service (one of the rare cases in Kubernetes where the relationship between two objects is not defined by labels).
As you’ve probably guessed, there is a sample Service definition available in the manifest over here on my GitHub. Let’s take a look:
apiVersion: v1 kind: Service metadata: name: hello-service-headless-with-selector namespace: service-types-example labels: type: headless spec: clusterIP: None selector: app: hello-app ports: - name: http8080 port: 8080
There are two little details worth noting:
- As mentioned previously, there is no
spec.type: Headless, rather, a Service carrying one of the other available types (here, I’ve simply gone for “the classic”) is made headless by…
- … defining the
spec.clusterIPfield to be
According to the
clusterIP: None definition, you probably won’t be surprised to see Kubernetes, in fact, did not assign a cluster IP for this Service:
$ k -n service-types-example get svc --selector type=headless NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE hello-service-headless-with-selector ClusterIP None <none> 8080/TCP 6m2s
Since this headless Service defines a selector, Kubernetes has created a buddy Endpoints object as usual, but because it has not allocated a cluster IP, we can use that headless Service to find out the individual IPs of the Service’s Pods. For this, we’ll launch a little helper Pod coming with nslookup in the same namespace:
$ k -n service-types-example run -it --generator=run-pod/v1 --image=registry.access.redhat.com/rhel7/rhel-tools -- bash
Once you’re inside the Pod, query the
ClusterIP Service first:
[root@bash /]# nslookup hello-service-clusterip Server: 10.43.0.10 Address: 10.43.0.10#53 Name: hello-service-clusterip.service-types-example.svc.cluster.local Address: 10.43.81.68
The query returns the virtual cluster IP of the Service. Now, how about the headless Service?
[root@bash /]# nslookup hello-service-headless-with-selector Server: 10.43.0.10 Address: 10.43.0.10#53 Name: hello-service-headless-with-selector.service-types-example.svc.cluster.local Address: 10.42.0.180 Name: hello-service-headless-with-selector.service-types-example.svc.cluster.local Address: 10.42.0.182 Name: hello-service-headless-with-selector.service-types-example.svc.cluster.local Address: 10.42.0.181
Indeed, this query returns the individual name and address of each Pod the Service’s label selector identifies. Thus, if we were talking to a stateful application running in the backend made up of those Pods, we could now make an informed decision about which Pod to talk to.
You can delete everything created in scope of this blog post using the following command:
$ k delete -f https://raw.githubusercontent.com/AntsInMyEy3sJohnson/blog-examples/master/kubernetes/workload-reachability/deployment-with-various-service-types.yaml
Phew, that’s a lot of content to digest. Let’s do a short recap:
The three most common Service types in Kubernetes are
LoadBalancer, which build on each other to offer their distinct behavior. The
ClusterIP type allocates a virtual IP called the “cluster IP” (how aptly named, isn’t it?) that allows cluster-internal load-balancing across the Pods identified by the Service’s selector. External access can be provided through the
kubectl port-forward and
kubectl proxy commands, which work well for testing and debugging, but are not appropriate for anything beyond that.
NodePort builds on
ClusterIP and enables external access to cluster-internal workloads by opening the given node port on each of the cluster’s nodes.
LoadBalancer, in turn, leverages
NodePort by provisioning an actual load balancer with a public IP or hostname which directs all traffic to the exposed node ports, but this only works if the environment the Kubernetes cluster runs in supports it (as most cloud providers do).
Other Service types are
ExternalName and headless. The use cases they address occur more rarely, hence those Service types are less common. The
ExternalName type is used to connect to cluster-external workloads using a DNS name, and headless Services are employed if (a) clients need to know the individual Pod IPs or (b) an external resource is to be connected using a single IPv4 address. The headless Service is not a type specified by means of the
spec.type property; rather, a Service is made headless by explicitly configuring its
spec.clusterIP field to
None (hence the different text styling for the word headless versus
ExternalName and the other Service types).
LoadBalancer Service type is fantastically easy to use to expose your workloads externally (thanks to the amazing work of the cloud provider’s engineers to do all sorts of configuration work behind the curtains), its usage can become very expensive if many Services of that type are used, thus for exposing many different workloads to the external world, the
LoadBalancer type is still not ideal. But, of course, Kubernetes has you covered: In the form of the
Ingress resource, it provides a means to expose a high number of workloads in a simple, reliable, and efficient way, and you’ll learn more about the
Ingress resource in the next blog post.