Kubernetes Logging Simplified – Pt 2: Kubernetes Events

Kubernetes Logging Simplified – Pt 2: Kubernetes Events

by Joe Howell on April 5, 2021

Overview

In my first post in the Kubernetes Logging Simplified blog series, I touched on some of the ‘need to know’ concepts and architectures to effectively manage your application logs in Kubernetes – providing steps on how to implement a Cluster-level logging solution to debug and analyze your application workloads.

In my second post, I’m going to touch on another signal to keep an eye on: Kubernetes events. Kubernetes events are important objects that can provide visibility into your cluster resources, and can be useful to correlate with your application and system logs.

What is a Kubernetes Event?

Kubernetes events are JSON objects made accessible via the Kubernetes API that signify a state change of a Kubernetes resource. These changes are reported to the API by their related component.  For example, if a pod is evicted or created, a container fails to start, or a node restarts – all of these state-changes would generate a Kubernetes Event, made accessible via API via kubectl commands.

Unlike container logs, Kubernetes events don’t ultimately get logged to a file somewhere; Kubernetes lacks a built-in mechanism to ship these events to an external backend. As a result, attempting to utilize a typical node-level log agent architecture to grab these events may not work. These events can be captured with a custom application, a number of OSS tools, or with an observIQ Event Collector, which I’ll walkthrough below.

What information does a Kubernetes Event Contain?

In addition to useful environment metadata, a Kubernetes Event contains the following key bits of information

  • When the event occurred
  • Severity of the event (info, warning, error)
  • Reason the event occurred (abbreviated description of the event)
  • Kind of Kubernetes resource (node, pod, container)
  • Description of the event
  • Component that reported the event (kubelet, kube-proxy, kube-api, etc)

Why is a Kubernetes Event useful to capture?

Postmortems

Tracking Kubernetes Events can be useful to understand what’s happening in your cluster over time, which can be particularly helpful to review during a postmortem. Digging into the ‘when’ and ‘why’ over time can reveal useful trends and a good discussion point when an application or service fails.

Custom Kubernetes Events dashboard in observIQ
Custom Kubernetes Events dashboard in observIQ

Real-Time Environment Awareness

If you’re using a complete Cluster-Level Logging solution like observIQ, KubernetesEvents can be used as a basis to create informational or error-level alerts that notify Slack for example, that provide real-time notifications that can keep your entire time in the loop as the state of your cluster resources change.

Kubernetes Events

Container Log Correlation

Having visibility into the state of your resources can help provide useful hints as to what’s happening with your applications. Kubernetes Events gathered by observIQ are automatically enriched with Kubernetes Metadata like namespace, deployment, and pod names – all of which allow you to correlate an application log directly to a resource event with a single filter.

Correlating container logs and kubernetes events with a resource filter in observIQ
Correlating container logs and kubernetes events with a resource filter in observIQ

How do I get Kubernetes Events?

By default, events are stored in etcd for a limited amount of time, typically ~60 minutes, and are made accessible by kubectl commands. Though the commands are useful to learn and employ in certain situations, utilizing a custom application or implementing a complete Cluster-level logging solution that captures, ships, and stores events for long-term analysis is highly recommended.

Accessing Kubernetes Events with Kubectl

Here are a few commands that will allow you to see your events:

kubectl describe pods <podname>

Describing a pod will provide you with related Kubernetes event information, if available:

Kubernetes event information

kubectl get events

Provides a list of current Kubernetes events for all resources:

Kubernetes events for all resources

kubectl get events -o json

Same as above, but each Kubernetes Event is presented as the raw JSON object:

raw JSON object

Accessing Kubernetes Events with OSS Tools:

Both kube-events and kubernetes-event-exporter are really nifty, highly customizable tools that have the functionality to capture and forward Kubernetes events to a preferred output or sink (e.g. S3, Kafka, etc)

Kube Events

https://github.com/kubesphere/kube-events

Kubernetes Event Explorer

https://github.com/opsgenie/kubernetes-event-exporter

Accessing Kubernetes Events with observIQ

With observIQ, you can easily enable Kubernetes Event collection by deploying the observIQ log agent as an Event Collector. Just select the option on your Kubernetes Template. See steps below:

observIQ and OpenTelemtry

A quick aside, our log agent Stanza is now the official log agent of the OpenTelemetry project check out our blog here for more details. 

Deploying the observIQ Agent as a Kubernetes Event Collector

In my first post in the series, I walked through how to create a Kubernetes Template and enable container log collection. With observIQ, you can easily enable or disable logging options from your template, even after you’ve deployed agents to your cluster. In this example, we’ll enable event collection in our existing template, re-apply the observ-agent.yaml, and add an observIQ Event Collector to our existing deployment. This will leave us with both 1) an observIQ agent daemonset that will gather the application’s logs and  2) a single Event Collector deployment that will gather the Kubernetes Events, running side by side. 

Update your Kubernetes Template in observIQ

Navigate to the Fleet > Templates page and choose your previously created Kubernetes template. Select the ‘Enable Cluster Events’ option, then click ‘Update’.

Enabling Kubernetes Events in observIQ Kubernetes Template
Enabling Kubernetes Events in observIQ Kubernetes Template

Next, click ‘Add Agents’.

Add Agents

On the Install Kubernetes Agents page, download and copy the newly-generated observiq-agent.yaml to your cluster, and apply by running kubectl apply -f observiq-agent.yaml command. After 15-30 seconds, you’ll see the new Event Collector in the discovery panel below.

Install Kubernetes Agents
observIQ daemonset agents + event collector
observIQ daemonset agents + event collector

View your Kubernetes Events in observIQ

After a few minutes, you’ll start to see Kubernetes Events on the observIQ Explore page. The messages will be typed as k8s.events.

Opening up one of the k8s.events, you can see parsed JSON object, as well as useful labels and metadata that have been automatically added to the event to help correlate to a specific application.

Kubernetes event in with Kubernetes Labels and Metadata

Wrapping up

Gathering your Kubernetes events is important if you want a complete understanding of what’s going in your cluster. Kubernetes Events are easily accessible with kubectl commands but are short-lived. Container logs and Kubernetes events can be correlated together – but it can be challenging without the right tool.

For a complete log management solution that will capture your Kuberentes Events, Container logs and more, sign-up for a free 14 day trial here for observIQ Cloud here:

https://app.observiq.com/sign_up/

Sign Up for the observIQ Cloud Beta

Download the Splunk Solution Brief

Sign Up to receive updates on our products

observIQ Support

For support on observIQ Cloud, please contact:

support@observIQ.com

For the Open Source Log Agent, community-based support is available on our:

GitHub Repository

Sign Up for Our Newsletter