What Is the OpenTelemetry Project and Why Is It Important?

At observIQ, we are big believers and contributors to the OpenTelemetry project. In 2023, we noticed project awareness reached an all-time high as we attended trade shows like KubeCon and Monitorama. The project’s benefits of flexibility, performance, and vendor agnosticism have been making their rounds; we’ve seen a groundswell of customer interest.

What is OpenTelemetry?

OpenTelemetry (“OTel”) aims to standardize how telemetry is generated, transmitted, and processed while being flexible enough to work with the telemetry streams you already have. It’s an open-source project governed by the Cloud Native Computing Foundation (CNCF).

At the core of OpenTelemetry is OTLP, a protocol for modeling and transmitting all common types of telemetry. On top of this protocol is an ecosystem of standardized instrumentation libraries that make capturing and transmitting logs, metrics, traces, and (soon) profiles easy. Additionally, OTel provides a standalone collector that can receive telemetry from various sources or actively pull it (e.g. from a Prometheus exporter or log file). The collector can then process the telemetry in a wide variety of useful ways, and finally forward it to almost any popular telemetry backend.

The best part is that it’s highly interoperable with other telemetry tools. If you’re using FluentBit or Prometheus, you can redirect those data streams through the OTel collector. It will translate the data into OTel’s data format and can be forwarded using OTLP or another common protocol.

The project has 2 guiding principles (quoted from OpenTelemetry.io)

You own the data that you generate. There’s no vendor lock-in.
You only have to learn a single set of APIs and convention.

As we wade deeper into the project, you see these principles are pervasive and drive decisions that help steer the project to where it is today.

Related Content: OpenTelemetry in Production: A Primer

What isn’t OpenTelemtry?

The project does not include tools to analyze your data directly (other than providing KPIs around collection and transmission). Instead, the project squarely focuses on data collection to delivery - leaving actionable insights to the observability platforms like Datadog, Google Cloud, New Relic, and Splunk.

Benefits of OpenTelemety

The OpenTelemetry provides several key benefits:

Standardization

Opentelelemtry provides a single set of compatible, open-source monitoring tools that collect telemetry data to understand the state of your distributed systems or applications. For those building and maintaining applications and distributed systems: proprietary tools, agent fatigue, non-standard configuration, and performance issues are familiar challenges the project addresses directly.

Vendor Neutral, Vendor Agnostic

The project provides the telemetry tools to implement a vendor-neutral framework. This means you can safely de-risk instrumenting your applications and infrastructure, as OTel allows you to reroute your data with minimal configuration and standardization.

Buy-in & Adoption

Splunk, Datadog, New Relic, Google, Honeycomb, Grafana, observIQ (and many other organizations) have all promoted and contributed, rapidly accelerating the project since 2020—further solidifying ‘neutrality’ as a core principle. We also see native OpenTelemetry Protocol support within both applications and backends. You can find a nice breakdown of all the contribution activity here.

Extensible

OpenTelemetry is extensible, providing flexibility through its collector SDKs, Integrations, and Distributions, which can be extended to observe any application and use case.

Similarly, Golang is inherently portable and can run on Linux, MacOS, Windows, and more. Platforms like Kubernetes and Cloud Foundry are at the forefront of rapid development and iteration in the project.

Disparate Pipes vs. an Observability Pipeline

Logically, each OpenTelemetry Collector constructs data pipelines within its configuration. More broadly, it enables the consolidation of disparate telemetry pipes into an Observation Pipeline. This enables actionability by making it possible to standardize and centralize configurations in a way that can be easily updated as an organization's needs change.

Building your Observability Pipeline with BindPlane OP

observIQ’s Unified Telemetry Platform, BindPlane OP, simplifies the creation and management of observability pipelines by

Remotely deploying and managing a fleet of OpenTelemetry Collectors
Streamlining OTel configuration with a guided configuration builder
Providing a single-pane summary of the size and cost of your observability pipeline
Simple controls to refine, reduce, and route your data

You can find out more about BindPlane OP here: https://observiq.com/solutions

Next, let’s break down some of the critical components of the project:

Key Components of OpenTelemetry

Overview

The project is broken down into several components, guided in detail by the OpenTelemetry Specification. Here’s a quick breakdown:

OpenTelemetry Specification
- OpenTelemetry Semantic Conventions
- OpenTelemetry Protocol (OTLP)
- Open Agent Management Protocol (OpAMP)
- OpenTelemetry APIs/SDKs
- OpenTelemetry Collector

OpenTelemetry Semantic conventions

OpenTelemetry Semantic Conventions are the guidelines and standards that ensure consistent and meaningful telemetry data by providing a common understanding of how to label and structure telemetry data. Some of the items semantic conventions cover:

Data Formats: define the standard formats for telemetry data, such as distributed tracing and metric formats, to ensure interoperability between different systems and tools.
Attribute Naming Convention: standardize attribute names for telemetry data to ensure consistency across different services and components within an application.
Semantic Context: Describe the contextual information that should be included with telemetry to provide meaningful insights into the behavior and performance of applications.

OpenTelemetry Protocol (OTLP)

The OpenTelemetry Protocol (OTLP) transmits telemetry data (metrics, logs, and traces) between OpenTelemetry clients and backends. In many ways, it’s the project's backbone - its implementation catalyzes the rest of the project. Here’s a quick breakdown of its feature set

Supports transmission of all signal types
Efficient in terms of CPU and network usage
Vendor-Neutral: the protocol itself isn’t biased towards a specific application or backend

Open Agent Management Protocol (OpAMP)

OpAMP is a network protocol that enables remote management of a fleet of agents or collectors. It can gather collector statistics, push configs, and handle other agent lifecycle tasks. It recently entered a beta phase in the OpenTelemetry Collector contrib repository.

At observIQ, we’ve spent much time considering and contributing to OpAMP’s development. We use it as a core component of BindPlane OP, deploying and managing OpenTelemetry collectors remotely.

Andy Keller, Principal Engineer at observIQ, and Jacob Aronoff, Staff Software Engineer from Lightstep, recently provided an excellent overview at Kubecon. I recommend watching it if you’ve got 30 minutes to spare. You can watch it here.

OpenTelemetry APIs and SDKs

OpenTelemetry APIs/SDKs allow you to instrument your code to expose OTLP-compatible telemetry data. Applications can be instrumented to ship this data directly to a backend or an OpenTelemetry Collector. Here’s the current list of available SDKs -- and the maturity of each signal contained in each (pulled from Opentelemetry.io):

Language	Metrics	Logs	Traces
C++	Stable	Stable	Stable
C#/.NET	Stable	Stable	Stable
Erlang/Elixir	Experimental	Experimental	Stable
Go	Stable	In Development	Stable
Java	Stable	Stable	Stable
JavaScript	Stable	In Development	Stable
PHP	Stable	Stable	Stable
Python	Stable	Experimental	Stable
Ruby	Stable	In Development	Stable
Rust	In Development	In Development	Stable
Swift	Experimental	In Development	Stable

OpenTelemetry Collector

The collector is the component responsible for receiving, processing, and exporting telemetry. It’s a critical piece of the OpenTelemetry framework, often (but not always) acting as the middleware between an application and the observability backend.

Collector Components

Receivers

Receivers are responsible for ingesting telemetry data, translating it into OTLP-compatible data, and passing it into the OpenTelemetry pipeline for processing and analysis. Receivers collect data: metrics, events, logs, traces (MELT). The collector’s contrib repository currently includes ~80 receivers, which non-scientifically cover the ‘95%’ use case for our customers.

If a technology is missing from the list, please file an issue in the project or contact us at (info@observiq.com) if you need a hand.

Processors

Processors are responsible for processing the data that moves through your OTel pipeline. They enable data enrichment, filtering, routing rules, and much more. You can find a full list of processors that are available here:

Exporters

Exporters are responsible for exporting the data from an OTel pipeline and collector. Exporters translate OTLP-compatible data in a format that’s compatible with the destination it’s headed to. Over time, the need for vendor-specific exporters will decrease as native OTLP.

Connectors

Connectors enable the transmission of telemetry data between different types of telemetry pipelines. For example, translating logs into metrics by processing log data and exporting it to a metrics pipeline.

Dan Jaglowski, a Principal Engineer at observIQ and maintainer on the OpenTelemetry Collector, recently gave a great talk on Connectors at Kubecon. You can watch it here.

Extensions

Connectors enable the transmission of telemetry data between different types of telemetry pipelines. For example, translating logs into metrics by processing log data and exporting it to a metrics pipeline. You’ll see more posts from me on Connectors at a later date.

OpenTelemetry Transform Language (OTTL)

OTTL is a transformative language used in conjunction with several processors that enables users to filter, reduce, and route their data in conjunction with several processors.

Related Content: What is the OpenTelemetry Transform Language (OTTL)?

Familiarizing yourself with OpenTelemetry: kicking the tires

If you’re interested in trying it out for yourself, here are a few resources I’d recommend:

The Official OpenTelemetry Demo provides an expansive environment demonstrating instrumentation and usage in a typical Kubernetes-backed microservices environment.
OpenTelemetry Registry provides a searchable list of OpenTelemetry components, simplifying the OTel mapping process.
Lastly, to get an OpenTelemetry collector up and running quickly, refer to my recent post, How to Install and Configure an OpenTelemetry Collector

OpenTelemetry in 2024: A preview

Here are the recorded goals for the project in 2024, broken down by Github issue:

OTel Project Goals:

And that’s a wrap. If you’re interested in OpenTelemetry or BindPlane OP or have any general questions, contact us at info@observiq.com or join us on BindPlane OP Slack to take the next step.

What is OpenTelemetry?

What is OpenTelemetry?

What isn’t OpenTelemtry?