Challenges of observing Kubernetes: Understanding a complex and dynamic system

When you look at traditional IT operations monitoring, tools are built to monitor static entities and infrastructure for known signals commonly referred to as the three pillars (logs, metrics, and traces), collecting these signals regularly from a number of manageable hosts deploying monolithic applications. In today’s cloud-native world, the equivalent applications have been atomized into containerized microservices and distributed across platforms like Kubernetes for very real reasons. With the ability to scale and self-managing resiliency, Kubernetes is a key technology and architectural choice for today’s modern apps.

In Kubernetes, infrastructure is defined and immutable. If there is an issue with a pod wrongly configured or something needs to be upgraded, there is no straightforward way to upgrade the existing pod; it is recommended to just deploy a new pod with a new name. The same goes for containers inside the pod. These get restarted as needed, making the performance of the infrastructure supporting your applications challenging to track and even harder to troubleshoot.

Kubernetes is a distributed system composed of different components running across multiple nodes (physical or virtual machines). While many of them are accessible through the Kubernetes api-server, there is no correlation of all the signals that are exposed and this task is left down to the observer.

The metrics and signals provided by the Kubernetes API is raw and does not easily give you insights. Another layer of analysis is needed to transform the data into visualizations with Kubernetes-centric dashboards and log analysis tools that can find patterns and understand trends. One potential solution would be to leverage machine learning for meaningful correlation and easier troubleshooting.

One of the biggest challenges organizations have around Kubernetes is the lack of expertise on how it really works under the hood. Acquiring and showing signals is not enough, and while understanding Kubernetes in detail is an option, one could argue that Kubernetes monitoring solutions need to bear the brunt of correlating seemingly disparate signals, providing users with answers and not just raw data. Kubernetes’ ability to run and resolve many performance issues (self-managing) means that when manual intervention and investigation is needed for more advanced problems, it can be a challenge for administrators.

The complexity of operating and maintaining the components that make Kubernetes appealing has paved the way for the rise of managed Kubernetes offerings across the most popular cloud providers (AWS, Azure, GCP) or other on-premises alternatives for self-management (Openshift, Tanzu).

And while at the core, the Kubernetes code used in most of these instances is what you could get directly from the Kubernetes project, the implementation varies across cloud providers — some allow more access to components and configuration of your Kubernetes clusters, while others limit your access to provide a more managed experience. 

AWS — most specifically EKS — allows you more access and custom configuration of your clusters, which requires more seasoned Kubernetes teams. Azure and GKE provide a more managed experience with more automation, reducing the need for Kubernetes expertise. This means that where and how you deploy Kubernetes will directly impact how you manage and what you monitor on your deployments for observability.

The recent rise of tools like ChatGPT have shown us that knowledge can be used in ways that are more aligned with a user’s end goal than with the instructions on how to get there.

Tools built to monitor your Kubernetes ecosystem should not expect you to be someone who knows the following things about a CrashLoopBackOff:

• It is a pod status error, meaning that a container keeps crashing when restarted
• Requesting the pod logs via kubectl log POD can help you understand what happened before
• Checking the exit code for crashed container kubectl describe POD
  1. If the exit code is 1, the container crashed because the application crashed.
  2. If the exit code is 0, verify for how long your app was running.

Even if you did know the above, just having to perform the actions yourself over having the solution serve that context is probably a better way. Solutions that are built on a repository of knowledge of Kubernetes infrastructure common issues, offer Kubernetes expertise within the UI, and support root cause analysis would be extremely valuable for less experienced Kubernetes administrators. Any issues that required human attention could present users with potential solutions (known context we have for their situation) to reduce MTTD and MTTR.

As you can see, Kubernetes observability poses some interesting challenges to users and operators. Let’s take a look at how Elastic Observability can help you with your Kubernetes monitoring in real life. 
In the examples below, we ran a variant of the popular Hipster Shop demo app on GKE and deployed Elastic Agents with Kubernetes integration as a DaemonSet on the same GKE cluster. Upon deployment of Elastic agent, Elastic starts ingesting metrics from the Kubernetes cluster (specifically from kube-state-metrics) and additionally, Elastic will pull container and audit logs from the cluster.

Kubernetes logs can be searched and analyzed in Elastic’s Discover capability. Here we see how Kubernetes logs can be searched, reviewed, and analyzed. You can run more specific searches with filters if you are familiar with the specific data stream and fields that are being ingested into Elasticsearch from your Kubernetes cluster. Or you can just do a plain-text search.

For observability practitioners whose goal is to keep Kubernetes up and running at all times, such as an SRE or a CloudOps professional, being stuck to a dashboard with the hope of looking at the right thing at the right time is not a feasible option. Many common use cases can be covered by implementing rules that will trigger informative alerts.

For example, set up an automated rule that will send a Slack message whenever a container is restarted or a pod goes into CrashLoopBackOff mode.