Managing a custom distribution of the OTel collector with BindPlane

Exciting news: it’s now possible to build a custom distribution of the OpenTelemetry Collector and remotely manage it with BindPlane. Though not all of BindPlane’s capabilities are available when managing a custom distribution (yet), it’s #prettycool, as it cracks open the door for teams looking to BYOF (bring your fleet), and manage them with our OTel-native telemetry pipeline.

This advancement is made possible because of the significant contributions and progress made in the development OpAMP. OpAMP is the not-secret-at-all sauce that enables remote management of an OpenTelemetry Collector; it’s one of the critical components powering BindPlane’s scaled fleet management capabilities.

Though we lightly touched on this in our Summer Announcement (also a fun read if you’re a fan of product updates and 16-bit era video game memes), I wanted to do a short walkthrough for users to do this themselves to familiarize themselves with the process as OpAMP’s refinement continues into 2024/25.

But first, a quick refresher.

What is OpAMP?

OpAMP is a protocol designed to manage and configure telemetry agents at scale. It allows centralized control over agent configurations, health monitoring, remote updates, and lifecycle management, making managing agents in distributed environments easier. OpAMP improves the scalability and automation of agent management in modern observability systems.

Andy Keller, a Principal Engineer at observIQ, contributed significantly to the design and implementation and gave a great talk with our friend Jacob Aranoff at ServiceNow at Kubecon 2024. Check it out if you’d like a quick deep dive by some of OpAMP’s subject matter experts.

How is OpAMP implemented in the OpenTelemetry Collector?

OpAMP is implemented in 2 ways in OpenTelemetry Collector:

1. As a Collector extension that provides access to the health and configuration of the collector via OpAMP.
2. As a Collector supervisor that relays the health and configuration of the collector and enables remote management of the collector configuration via OpAMP.

Both of these implementations can be used together, providing a solution that includes information about the health of an OTel collector while also opening the door to remote configuration and management.

What is the OpAMP Extension?

The opampextension is an extension available in the OpenTelemetry Collector that can be configured to contact an OpAMP server, relaying read-only information such as the collector’s description, health, and configuration. It’s self-described as having limited functionality, a subset of OpAMP’s overall capabilities.

What is the OpAMP Supervisor?

When I mentioned progress previously, I primarily had Supervisor in-mind. The supervisor is a separate process that can run an OTel collector, opening the door to more of OpAMP’s remote management capabilities, such as:

• Receiving and pushing configuration from an OpAMP backend to an OTel collector
• Stopping and starting an OTel collector process
• Restarting the OTel collector in the event of a crash or failure
• Accepting connection and OTel collector details from the opampextension
• Updating the OTel collector packages/collector updates

The supervisor is still very much in development (you can see the state of each of the key bits of functionality here), which means some significant functionality still needs to be introduced, and existing functionality may change. But it’s certainly ready for tinkering and consumption and ripe for additional feedback from the community.

Connecting a Custom Distribution of the OpenTelemetry Collector to BindPlane

Pre-reqs

• A running BindPlane instance and access to your SECRET_KEY and OpAMP server endpoint within your BindPlane config. You can use either of the following links to setup an instance:
  • BindPlane On Prem https://observiq.com/download
  • BindPlane Cloud: https://app.bindplane.com/signup
• Host/VM to run the custom OTel collector (using the same host as your BindPlane VM for testing purposes also works).

Building a Custom Collector Distribution with the OpenTelemetry Collector Builder:

First, build your custom collector with the OCB using the following steps:

• Step 1: Install the collector builder. You can follow the steps provided here: https://opentelemetry.io/docs/collector/custom-collector/#step-1---install-the-builde
• Step 2: Create a builder manifest with the steps provided here: https://opentelemetry.io/docs/collector/custom-collector/#step-2---create-a-builder-manifest-file or use the sample manifest I’ve created below:

1dist:
2  module: my-first-distro
3  name: my-first-distro
4  description: Minimal distro of OpenTelemetry Collector for Bindplane OP
5  version: "v0.0.1"
6  output_path: ./dist
7  otelcol_version: 0.108.0
8
9extensions:
10  - gomod: github.com/open-telemetry/opentelemetry-collector-contrib/extension/healthcheckextension v0.108.0
11  - gomod: github.com/open-telemetry/opentelemetry-collector-contrib/extension/opampextension v0.108.0
12
13exporters:
14  - gomod: go.opentelemetry.io/collector/exporter/nopexporter v0.108.0
15
16processors:
17  # snapshotprocessor is required for snapshots in BindPlane OP
18  - gomod: github.com/observiq/bindplane-agent/processor/snapshotprocessor v1.59.1
19  # resourcedetectionprocessor is required for the host source type in BindPlane OP
20  - gomod: github.com/open-telemetry/opentelemetry-collector-contrib/processor/resourcedetectionprocessor v0.105.0
21
22receivers:
23  - gomod: go.opentelemetry.io/collector/receiver/nopreceiver v0.108.0
24  # hostmetricsreceiver is required for the host source type in BindPlane OP
25  - gomod: github.com/open-telemetry/opentelemetry-collector-contrib/receiver/hostmetricsreceiver v0.108.0
26
27connectors: []

A couple notes about this manifest:

• The opampextension, healthcheckextension, snapshotprocessor are required for remote management with BindPlane. You can read more about Snapshots in BindPlane here.
• In this example, we’ve included the hostreceiver and the noop exporter for our minimal configuration. Respectively, these map to the Host Source and Dev Null destination in BindPlane. You can swap or add additional config

Recently, Michelle Artreche published a great guide on building a custom distribution of the collector using the OCB, which takes you through the end-to-end process in detail--and provides information about some of the other collector distribution options.

Installing the Supervisor

Next, Install the supervisor on your collector host using the following commands provided below:

• Run the the following cmd/opampsupervisor directory of the opentelemetry-collector-contrib repository.

1go install .

• Then run:

1mkdir -p ./local/storage && cp supervisor-config.example.yaml ./local/supervisor-config.yaml.

• Fill in ./local/supervisor-config.yaml with your BindPlane secret key and opamp endpoint.

To help expedite, I’ve included a sample of the supervisor-config.yaml below:

1server:
2  # BindPlane OP URL for OpAMP. This points to SaaS. The `wss` protocol uses TLS.
3  # If you are not using TLS, you may use `ws` for the protocol instead.
4  endpoint: wss://app.bindplane.com/v1/opamp
5
6  # headers is a key-value mapping of header keys to values.
7  # Here, we are configuring the "Authorization" header to be our secret key
8  # so that we can connect to BindPlane OP. Replace <your-secret-key-here>
9  # with your secret key from BindPlane OP.
10  headers:
11    Authorization: "Secret-Key <your-secret-key-here>"
12
13  # TLS settings. Uncomment if connecting to an instance that does not use TLS.
14  # tls:
15  #   insecure: true
16  #   insecure_skip_verify: true
17
18capabilities:
19  ## accepts_remote_config allows the supervisor to take configs from BindPlane
20  accepts_remote_config: true
21  # reports_remote_config allows the supervisor to report the status of the remote config
22  # (e.g. whether it was applied or failed to apply)
23  reports_remote_config: true
24
25agent:
26  # This is the path to the binary you built with the OpenTelemetry Collector Builder (OCB)
27  executable: ./dist/my-first-distro
28
29storage:
30  # This is the path to a storage directory that the supervisor will use
31  # for persistent state. It will be created if it does not exist when the supervisor starts.
32  directory: "./local/storage"

Running the Supervisor

Lastly, start the supervisor using the following command.

1opampsupervisor --config supervisor-config.yaml

Since the supervisor's job is to run the collector, starting the supervisor will also launch your collector and connect the collector via the supervisor to your BindPlane instance.

Viewing your Managed OpenTelemetry Collector in BindPlane

Head over to your BindPlane instance, and you’ll now see your Collector on the ‘Agents’ page of BindPlane.

And that’s it! Your custom collector should now appear in BindPlane with a ‘connected’ status, and the collector’s config (seen here as a no-code visualization) is now fully readable, editable, copyable, and deployable to other managed collectors in your fleet.

Custom Distributions of the OTel Collector and BindPlane: Stayed Tune

As OpAMP progresses, so will the connection between custom distributions of the OTel collector and BindPlane. Though some extra configuration is required to kick the tires today, the end state will be drastically simplified and seamless, something you’d expect from an OTel-native Telemetry Pipeline.

If you’d like additional information about OpenTelemetry, OpAMP, or BindPlane, contact our team at info@observiq.com.