The Observability Blog

  • Uncategorized

Configuring OpenTelemetry Agents to Enrich Data and Reduce Observability Costs

Paul Stefanski headshot
by Paul Stefanski on
December 20, 2022

BindPlane OP is a powerful, open source tool that makes it easy to build and manage telemetry pipelines to ship data from IT environments of any kind and size to any analysis tool or storage destination. BindPlane OP installs and configures OpenTelemetry agents, which support a wide variety of sources and can be configured to ship data to multiple destinations while enriching or reducing data simultaneously. The vendor-agnostic toolset is excellent for reducing data costs and getting the most out of your data. OpenTelemetry agent configurations are formatted as yaml files that are always editable without the need to uninstall and reinstall the agent. BindPlane OP makes creating and editing configurations even easier with a codeless visual interface and step by step options for collecting and managing your data. This blog covers how to create and edit a configuration in BindPlane OP, add sources and destinations, add processors to reduce or enrich data, and apply that configuration to one or many agents with a few clicks. 

What are Configs in BindPlane OP?

Configurations in BindPlane OP are instructions for directing OpenTelemetry agents. The Configs tab is where you create, save, and apply agent configurations in BindPlane OP. Existing configurations appear in a sortable, searchable list, and new configurations can be created by clicking the “New Configuration” button at the top. Clicking on a configuration will open a page where you can see details for that configuration, including the sources, destinations, and a topographical map representing the real time data flow and processing of agents using that configuration. 

You can edit the sources and destinations by clicking them, and add new ones by clicking the respective buttons. New configurations created on the Configs page exist as templates that do not affect your agents unless explicitly applied, so don’t worry about affecting your environment while creating or editing new configurations, but if you edit an existing configuration that is already applied to agents in your environment, those changes will be pushed to the appropriate agents when you click “Save” after an edit is made.

At the bottom of any configuration’s page, you can see a searchable list of agents using that configuration, their operating system, and their connection status. You can also apply the configuration to new agents by clicking the “+” button at the top of this section. 

Creating an OpenTelemetry Agent Configuration in BindPlane OP

Creating a configuration is a three step process.

  • Give your configuration a name, select your intended operating system, and write a brief description.
    • It’s good practice to give enough detail in the description so that other users can quickly identify the intended use of the configuration. 
  • Add your intended source(s)
    • Sources appear in a searchable list with” log”, “metrics”, and “traces” labels.
    • You can adjust your basic settings for the source, like the hostname and port. You can also adjust advanced settings, such as the collection interval. Advanced settings vary by source.
  • Add your destination(s)
    • Destinations appear in a searchable list. Click a destination to select it.
    • Give your destination a name for use in BindPlane OP (the name does not have to match anything outside of BindPlane OP, but that can make organization easier)
    • Add the necessary routing and authentication information for the destination. The specific requirements vary by destination, but typically include a region and an access key or password. Tooltips in BindPlane OP can help you find your authentication information if you don’t have it.
    • You can add as many destinations as you wish by repeating the previous steps. OpenTelemetry agents can ship data to any number of destinations.
    • Once you add a destination to a configuration, it is saved as a component, so you can skip the routing and authentication when you use it in another configuration.

Adding Processors to Enrich Data and Reduce Data Costs

Processors are components in configurations that manipulate data at the source before it’s sent to your destinations. There are many ways data can be manipulated, but the two general categories of data manipulation are enrichment and reduction. Enrichment means you add additional information to data at the source that helps you use the data more effectively at your destination. Reduction means you filter or eliminate data to reduce data flow and save on data costs. Below are the steps for adding a processor, and list of processors and their uses.

  1. On the Configs page, click on the configuration you want to add a processor to, then click on the source you want the processor to affect data from.
  2. Click “Add Processor” at the bottom of the source details.
  3. Select the processor you want from the list, or create a custom processor.
    1. Input the details for how you want the processor to manipulate your data, then click “Save”. The processor will be applied to any agents using the configuration automatically.


Add Attribute

The Add Log Record Attribute processor can be used to enrich logs by adding attributes to all log records in the pipeline.
Add Resource

The Add Resource Attribute processor can be used to enrich telemetry by adding resources to all metrics, traces, and logs in the pipeline.

Custom Processor

The Custom processor can be used to inject a custom processor configuration into a pipeline.

The Custom processor is useful for solving use cases not covered by BindPlane OP's other processor types.
Delete Attribute

The Delete Log Record Attribute processor can be used to remove attributes from all
log records in the pipeline.
Delete Resource

The Delete Resource processor can be used to remove resources from metrics, traces, and logs.

Filter Severity

The Severity Filter processor can be used to filter out logs that do not meet a given severity threshold.

Filter Log Record Attribute

The Log Record Attribute Filter processor can be used to include or exclude logs based on matched attributes.

Filter Resource Attribute

The Resource Attribute Filter processor can be used to include or exclude logs based on matched resources.

Filter Metric Name

The Metric Name Filter processor can be used to include or exclude metrics based on their name.

Log Sampling

The Log Sampling processor can be used to filter out logs with a configured "drop ratio".

Extract Metric

The Extract Metric Processor can look at all logs matching a filter, extract a numerical value from a field, and then create a metric with that value.
Both the name and units of the created metric can be configured. Additionally, fields from matching logs can be preserved as metric attributes.
Count Logs

The Count Logs Processor can count the number of logs matching some filter, and create a metric with that value.
Both the name and units of the created metric can be configured. Additionally, fields from matching logs can be preserved as metric attributes.