What is the OpenTelemetry Transform Language (OTTL)?

The OpenTelemetry Transformation Language, or OTTL for short, offers a powerful way to manipulate telemetry data within the OpenTelemetry Collector. It can be leveraged in conjunction with OpenTelemetry processors (such as filter, routing, and transform), core components of the OpenTelemetry Collector.

It caters to a range of tasks from simple alterations to complex changes. Whether you're dealing with metrics, spans, or logs, OTTL equips you with the flexibility to refine and shape your data before it's dispatched to its final destination for monitoring and analysis.

The language is built on the principles of clarity and efficiency, allowing developers and DevOps to write expressive statements that perform transformations effectively.

In this post, I’ll step through the following:

• Key Benefits of Leveraging OTTL
• Core Concepts of OTTL
• Developing with OTTL
• Common use cases with configuration samples

Related Content: What is OpenTelemtry?

Keys Benefits

The key benefit that OTTL provides is giving the user granular controls to sculpt and refine their observability data -- broken down into some supporting benefits:

• Reduced verbosity - OTTL enables powerful transformations with minimal configuration
• Data Routing - enables the routing of observability data to one or many destinations
• Noise Reduction - trimming excessive data to streamline root-cause analysis
• Cost Reduction - trimming excessive data to keep observability costs within your budget
• Data Enrichment - tagging, identifying, and elevating key data streams for deeper analysis in your observability backend
  

Core Concepts

Under the hood of OTTL, there are a few core concepts to familiarize yourself with:

• Statements: These define the transformations performed on your telemetry data. They are structured commands following the OTTL's grammar and are defined in the collector’s configuration file (config.yaml)
• Contexts: Specifies the domain (like traces or metrics) in which the statements will apply. Here’s the list of available contexts:
  • Resource
  • Instrumentation Scope
  • Span
  • SpanEvent
  • Metric
  • DataPoint
  • Log
• Functions: Invokable operations within statements that determine the nature of the transformation, such as renaming attributes and filtering data. There are 2 primary types of functions: converters (provide the utilities to transform telemetry) and editors (used to transform telemetry). You can find a full list of available functions here.

Developing with OTTL

Best Practices

A few things to keep in mind, as you’re building your OTTL statements

• Simplicity: Keep your transformation rules as simple as possible. Complex rules take more work to maintain and understand.
• Modularity: Write modular statements that can be easily understood and replaced.
• Testing: Regularly test your rules to make sure they work as intended.
  Remember that while OTTL is powerful, the simpler and clearer your transformation rules are, the more maintainable your configuration will be. This straightforward approach to developing with OTTL can save you time and effort in the long run.

Statements and Operators

Statements in OTTL are combinations of items like variables, literals, and operators that evaluate a value.
The structure of statements is intuitive if you're familiar with programming. For example, you could use an expression to increase a metric's value or to choose a specific span attribute.

Meanwhile, common operators in OTTL include:

• == (equality)
• != (inequality)
• > (greater than)
• < (less than)
• >= (greater than or equal to)
• <= (less than or equal to)

These operators allow you to perform logical comparisons with the data fields in your telemetry.

Writing a simple OTTL statement

To begin writing rules with OTTL, you'll need to understand its syntax and structure.

A typical OTTL rule consists of an editor (set being the most common) and a list of assignments or transformations to apply. For example:

1set(attributes["http.status_code"], 200) where attributes["http.method"] == "POST"

This rule sets the attribute to 200 for all telemetry data where the http.method attribute is "POST".

Common Use Cases

Below are several common use cases we see where OTTL comes in handy when working with OpenTelemetry in the field:

Parsing ‘body’ from the contents of a JSON log:

1transform:
2    error_mode: ignore
3    log_statements:
4        - context: log
5          statements:
6            - set(
7              attributes,
8              ParseJSON(
9                attributes["body"]
10              )

Parsing ‘Severity’ and mapping severity within a log event:

1transform:
2    error_mode: ignore
3    log_statements:
4        - context: log
5          statements:
6            - set(severity_number, 5) where([body["level"]], "") == "debug"
7            - set(severity_number, 17) where([body["level"]], "") == "error"
8            - set(severity_number, 21) where([body["level"]], "") == "fatal"
9            - set(severity_number, 9) where([body["level"]], "") == "info"
10            - set(severity_number, 1) where([body["level"]], "") == "trace"
11            - set(severity_number, 13) where([body["level"]], "") == "warn")

Parsing ‘Timestamp’ from a log event:

1transforms:
2    error_mode: ignore
3    log_statements:
4        - context: log
5          statements:
6            - set(time, Time(attributes["timestamp"], "%Y-%m-%dT%H:%M:%S.%f%z"))

Renaming a field within a log event:

1transform:
2    error_mode: ignore
3    log_statements:
4        - context: resource
5          statements:
6            - set(attributes["host.name"], attributes["hostname"])
7            - delete_key(attributes, "hostname")

OTTL Setup and Configuration

Configuring and testing OTTL consists of a few simple steps:

Install the OpenTelemetry Collector:

You can check out my recent post, “How to Install and Configure an OpenTelemetry Collector” -- this guides you through the detailed installation and configuration steps.

Access the Collector’s Configuration File:

The first step is accessing the OpenTelemetry Collector's configuration file where you'll specify your OTTL configurations.

/etc/otelcol-contrib/config.yaml

Define Transformation Rules:

Within the configuration file, under the transform processor section, you will define your transformation rules using the OTTL syntax.

1transform:
2    error_mode: ignore
3    log_statements:
4        - context: log
5          statements:
6            - set(
7              attributes,
8              ParseJSON(
9                attributes["body"]
10              )

Associate with Pipelines:

Decide which pipelines (metrics, traces, logs) will use the transformation rules by linking them to the corresponding processors in the configuration file.

1pipelines:
2    logs:
3        receivers:
4            - syslog
5        processors:
6            - transform
7        exporters:
8            - otlp

Test and Validate:

After setting your rules, it's important to test and validate them to ensure they work as expected.

Performance Considerations

Lastly, while you’re working with OTTL, it’s important to be aware of the performance ramifications:

• Overhead: Introducing any transformation operations can add extra processing overhead. It's important to balance the need for data transformation against the potential impact on the OpenTelemetry Collector's performance.
• Complexity: The more complex your transformations, the more load is put on your host. Try to keep your OTTL statements as simple as possible to minimize performance degradation.
• Sampling Considerations: Implementing transformations may interfere with the telemetry data's fidelity. If transformations occur before sampling decisions, this could affect the resulting sample and, your performance measurements as indicated in the Performance Benchmark of OpenTelemetry API.

Hopefully, this overview will help you get started as you begin to use OTTL. If you have questions or feedback, or want to chat about OpenTelemetry or OTTL, feel free to reach out on the CNCF Slack (jhowell), or e-mail me at joseph.howell@observiq.com.