observIQ Cloud and the OpenTelemetry Collector

An upgrade to observIQ’s log agent – incorporating OpenTelemetry

Our log agent is powerful, efficient, and highly adaptable. Now, with OpenTelemetry setting new standards in the observability space, we wanted to incorporate that collaboration into our log agent and offer our users the ability to take advantage of the OpenTelemetry ecosystem. Starting today, you can upgrade the log agents in your observIQ account to the new Open Telemetry-based observIQ log agent with a single click.

OpenTelemetry’s logging USP adheres to the “textbook” definition of a log. By aligning our agent to the OpenTelemtry collector, we aim to attain textbook perfection for our log management capabilities. To understand why this is a game-changer, let’s dive into the basic architecture of the OpenTelemetry log collector.

Log Collector’s Architecture

The open telemetry collector is designed to support logs from legacy systems, logging libraries, and cloud-native applications. The problem the open telemetry collector addresses is the incohesive logging solutions and libraries that have an incomplete correlation between the aspects of observability data, namely, metrics, logs, and traces. By implementing a standardization in how observability data is parsed, ingested, distributed, and consumed, OpenTelemetry has made the telemetry data very relevant and highly informative.

Open Telemetry has standardized data models, for all logs, metrics, and traces that they parse. Once parsed, the OpenTelemetry collector enriches the data further to create more correlation between the data. The most notable factor here is that the enrichment across logs, traces, and metrics has the same attribute names and values, maintaining uniformity across all observability data. OpenTelemetry’s log collector follows a defined log data model that dictates the information that should or should not be recorded in the log data. This log data model is created with the intent of having log data transmitted, saved, and analyzed in a standardized manner. It is expected that the existing log libraries would align themselves to this log data model in their future versions.

The components of the log collector’s architecture are Receivers, Processors, and Exporters. The traces and metrics data’s trajectory is defined by a pipeline in the collector but not the log data. The receiver is essentially the entry point for the log data, where the data is collected, assimilated, and forwarded to the processor, which enriches and correlates the data. Once enriched the data is transmitted to the destination path/ applications via the exporters.

How does correlation work?

The primary aspect of correlation according to OpenTelemetry standards is the time-based correlation, where logs, traces, and metrics are mapped to each other based on the time or time period of execution. The next level of correlation is based on the execution. Logs and traces in the same execution context are associated with each other using TraceID and SpanID. Another significant correlation factor is the resource name that is included in the traces and metrics data.

What difference does this upgrade make to your log management?

• It leverages the benefits of the evolving OpenTelemetry log collector’s components
• Now restarting an agent would be a unique event not necessitating restarting the application or the other components of your infrastructure.
• There are more refined log level standardizations, mirroring the log levels/ service text in OpenTelemetry such as trace, debug, error, warn, info, and fatal.
• observIQ’s log agent now adapts a new log rotation and checkpoint resuming capability using OpenTelemetry’s filelog receiver.

Steps to upgrade to the log agent

Technically the upgrade to the new log agent works at the click of a button. This is in line with our mantra – keep everything simple.

The gif above is unedited to show how simple the upgrade works. It takes under 15 seconds to navigate, initiate and complete the upgrade. Try this out in your observIQ account or sign up for an account.

The upgraded log agent combined with the vast set of source plugins that observIQ offers makes your log management a breeze. As always, we are around to assist you with any log management questions or requests you may have. Please reach out to our awesome support team. Stay observant with observIQ.