Complimentary Gartner® Report! 'A CTO's Guide to Open-Source Software: Answering the Top 10 FAQs.'Read more

Kubernetes Container Logs

Read container logs from Kubernetes nodes

Supported Platforms

PlatformMetricsLogsTraces
Kubernetes DaemonSet✓
OpenShift 4 DaemonSet✓

Configuration Table

FieldDefaultDescription
Cluster Name*The cluster name that will be added as the k8s.cluster.name resource attribute.
Log SourceFileWhere to read logs from. Generally, this is file. file source supports Docker json-file and Containerd cri-o log formats. Options include file and journald.
File Path(s)/var/log/containers/\*.logWhen log_source is file. File or directory paths to tail for logs. Defaults to all container logs.
Journald Path*When log_source is journald. The directory containing Journald's log files.
Exclude File Path/var/log/containers/bindplane-*-agent-*File or directory paths to exclude. Generally, the collector's own log should be excluded.
Start AtendStart reading the logs from the 'beginning' or 'end'.
Recombine LogsOptions for configuring multi line logging. See Multi Line Logging.
*required field

Example Configuration

The Kubernetes Container logs source has one required parameter:

  • Cluster Name: The name of the cluster, which will be inserted as the k8s.cluster.name resource attribute
observIQ docs - Kubernetes Container Logs - image 1

Once running on an agent, logs will look like this:

observIQ docs - Kubernetes Container Logs - image 2

Multi Line Logging

Multi line logging is useful for re-assembling multi line logs into a single log entry. Multi-line log re-assembly requires that all logs emitted by the application are consistent in structure. For example, the logs must start or end in a consistent way, in order to reliably match on the beginning or end of the log. If your application has inconsistent logging, multi-line log re-assembly will behave in irregular ways, such as combining two unique logs into one.

Multi line logging is supported by configuring a selector, selector match expression, and a recombine match expression.

FieldDescription
SelectorThe OTTL path to match on.
Selector Match ExpressionA regular expression used to match the selector.
Recombine TypeWhether or not to recombine logs by matching the first or last line in the log.
Recombine WithThe delimiter used to recombine logs. Defaults a single space or newline character.
Recombine Match ExpressionThe regular expression used to recombine the multi-line log.

Example

In this example, there are two Deployments. One is logging XML while the other is logging JSON.

bash
1NAME                    READY   STATUS    RESTARTS      AGE
2json-7d4fd9fd99-f29sl   1/1     Running   1 (38m ago)   3h49m
3xml-b44858b84-hdbqp     1/1     Running   1 (38m ago)   3h49m

The XML logs are a combination of multi-line and single line logs. Each log has a timestamp prefix indicating the start of the log.

xml
12024-07-01 18:49:15 <person>
2                      <name>John Doe</name>
3  <age>30</age>
4</person>
52024-07-01 18:49:15 <person><name>John Doe</name><age>30</age></person>

The JSON logs are a combination of multi-line and single line logs. Each log has a trailing } without a comma, indicating the end of the log.

json
1{
2  "timestamp": "2024-07-01 18:51:56",
3  "name": "John Doe",
4  "user": {
5    "name": "John Doe",
6    "age": 25
7  },
8  "location": "New York",
9  "type": "Checking",
10  "severity": "warn"
11}
12{"timestamp":"2024-07-01 18:51:56","name":"John Doe","user":{"name":"John Doe","age":25},"location":"New York","type":"Checking","severity":"warn"}

Multi-line logging can be configured by matching on the First Entry of the XML logs and the Last Entry of the JSON logs. The k8s.pod.name resource attribute is used to select the XML and JSON pods.

observIQ docs - Kubernetes Container Multi Line Logs - image

Once configured, logs will be re-assembled into a single line.

observIQ docs - Kubernetes Container Multi Line Logs - image 2