How to Build a Custom OpenTelemetry Collector

Telemetry data collection and analysis are important for businesses. We're diving right in to explain the ins and outs of the OpenTelemetry Collector, including its core components, distribution selection, and customization tips for optimal data collection and integration. Whether you're new to OpenTelemetry or expanding your capabilities, this will help you effectively use the OpenTelemetry Collector in your observability strategy.

Understanding the OpenTelemetry Collector

The OpenTelemetry Collector is made up of several components: receivers, processors, exporters, connectors, and extensions. Each component serves a unique function in the data pipeline, facilitating the ingestion, processing, and export of telemetry data from various sources. Customizing these components allows organizations to fine-tune data collection strategies, optimize performance, and seamlessly integrate with existing infrastructures.

Choosing a Collector Distribution

Assess Your Needs

When choosing an OpenTelemetry Collector distribution, there are a few factors to consider. These include your specific telemetry data requirements, the complexity of your environment, the level of support you need, and the extent of customization or scalability you require.

Research Distributions

To fully understand different distribution options, explore their features, built-in components, and available support services. Make sure to verify that the distribution supports all the platforms and languages your systems use.

Documentation & Community

A well-documented and easy-to-use distribution can save time and effort in setup and maintenance. Don’t forget about the valuable input and reviews from the community; research to see what your peers think about the distribution options.

Options for OpenTelemetery Collector Distributions

We've put together a list of common distributions of OpenTelemetry Collector for your consideration:

1. OpenTelemetry Collector (Core/Contrib):
   • This is the primary version provided by the OpenTelemetry community. It includes a basic set of components in the Core version and an expanded set in the Contrib version, which provides for extra receivers, processors, and exporters.
2. AWS Distro for OpenTelemetry (ADOT):
   • This Amazon Web Services distribution has been optimized for use in AWS environments. It includes specific enhancements and setups tailored for AWS and integrates with various AWS services and systems.
3. Splunk Distribution of OpenTelemetry Collector:
   • Optimized to work seamlessly with Splunk Observability Cloud and includes various improvements and extra features designed specifically for Splunk environments.
4. Grafana Agent:
   • The Grafana Agent, primarily designed for Prometheus, can send metrics and traces to Grafana Cloud and other backends using OpenTelemetry.
5. Lightstep Distro for OpenTelemetry:
   • Optimized for the Lightstep Observability platform and includes additions and optimizations for better integration and performance with the Lightstep suite.
6. BindPlane Agent:
   • The BindPlane Agent is a tool that can act as an agent, a gateway, or both. When used as an agent, the collector runs on the same host and collects telemetry. As a gateway, it collects telemetry from other agents and sends the data to its final destination.

Building Your Collector Distribution

The hypothetical scenario detailed within the webinar illustrates the process of building a custom OTel Collector using the OpenTelemetry Collector builder. The steps include downloading the OpenTelemetry Collector builder binary, refining the manifest file to remove unnecessary parts, and adding special connectors or processors for hotel data analytics. This method improves data collection accuracy and helps organizations get important insights to run operations better and satisfy customers.

Critical Steps in Building a Custom Collector Distribution

Install the builder

Download the OTel Collector builder binary (ocb).

Note: On Linux and OSX you will likely need to make the binary executable with chmod u+x ocb

To check if the ocb is ready to be used, open your terminal and type ./ocb help. After pressing enter, you should see the help command output in your console.

Step-by-step example of downloading and making executable on Linux; while also putting it in its own dedicated folder:

1$ mkdir ocb
2$ wget https://github.com/open-telemetry/opentelemetry-collector/releases/download/cmd%2Fbuilder%2Fv0.104.0/ocb_0.104.0_linux_amd64 -O ocb/ocb
3ocb/ocb    	100%
4$ chmod u+x ocb/ocb
5$ ocb/ocb version
6ocb version 0.104.0

You can also add this to your path, or create a symbolic link to it on the path if desired.

Configure the initial manifest file

The builder's manifest file is a yaml, and it acts as a blueprint to modify and compile all of the components you want to add to your Collector’s distribution — details like distribution name, description, and version. This step allows for precise integration of specialized functionalities needed for unique data collection scenarios.

The dist map at the beginning of manifest contains tags to help you configure the code generation and compile process. These tags for dist are the same as the ocb command line flags.

Here are the tags for the dist map:

This is a friendly reminder that you can add custom values for distribution tags based on whether you want your custom Collector distribution to be available for others to use or if you're simply using the ocb for your component development and testing environment. All dist tags are optional and meant for customization.

Testing and validation

Before you deploy it, make sure to thoroughly test the custom collector in different situations to confirm that it works well. Testing will ensure that the collector meets performance benchmarks and captures telemetry data as intended.

We will create a distribution for the Collector to help develop and test custom components. To get started, create a manifest file named builder-config.yaml with the following content:

1dist:
2name: otelcol-dev
3description: Basic OTel Collector distribution for Developers
4output_path: ./otelcol-dev

To customize the Collector distribution, start by adding modules, which are the specific components you want to include.

For our development and testing collector distribution, we will add the following components:

• Exporters: OTLP and Debug
• Receivers: OTLP
• Processors: Batch

Once you’ve added these components, the builder-config.yaml manifest file will reflect these changes:

1dist:
2 name: otelcol-dev
3 description: Basic OTel Collector distribution for Developers
4 output_path: ./otelcol-dev
5 otelcol_version: 0.104.0
6
7
8exporters:
9   # NOTE: Prior to v0.86.0 use the `loggingexporter` instead of `debugexporter`.
10 - gomod: go.opentelemetry.io/collector/exporter/debugexporter v0.104.0
11 - gomod: go.opentelemetry.io/collector/exporter/otlpexporter v0.104.0
12
13
14processors:
15 - gomod: go.opentelemetry.io/collector/processor/batchprocessor v0.104.0
16
17
18receivers:
19 - gomod: go.opentelemetry.io/collector/receiver/otlpreceiver v0.104.0

Create the Code and Establish your Collector’s distribution

Just let the ocb do its job to ensure everything goes smoothly. Open your terminal and type the following command: ./ocb --config builder-config.yaml

If the command runs successfully, the output should look like this:

12022-06-13T14:25:03.037-0500	INFO	internal/command.go:85	OpenTelemetry Collector distribution builder	{"version": "0.104.0", "date": "2023-01-03T15:05:37Z"}
22022-06-13T14:25:03.039-0500	INFO	internal/command.go:108	Using config file	{"path": "builder-config.yaml"}
32022-06-13T14:25:03.040-0500	INFO	builder/config.go:99	Using go	{"go-executable": "/usr/local/go/bin/go"}
42022-06-13T14:25:03.041-0500	INFO	builder/main.go:76	Sources created	{"path": "./otelcol-dev"}
52022-06-13T14:25:03.445-0500	INFO	builder/main.go:108	Getting go modules
62022-06-13T14:25:04.675-0500	INFO	builder/main.go:87	Compiling
72022-06-13T14:25:17.259-0500	INFO	builder/main.go:94	Compiled	{"binary": "./otelcol-dev/otelcol-dev"}

The folder otelcol-dev has been created in the dist section of your config file. It includes all the source code and the binary for your Collector’s distribution. The folder structure should look like this:

1.
2├── builder-config.yaml
3├── ocb
4└── otelcol-dev
5   ├── components.go
6   ├── components_test.go
7   ├── go.mod
8   ├── go.sum
9   ├── main.go
10   ├── main_others.go
11   ├── main_windows.go
12   └── otelcol-dev

You can use the code you generated to start your component development projects. This makes it easier for you to create and share your own collector distribution with your components.

Automate Builds for Efficient Deployment

Making builds and sending them out is important for making sure everything works well. Developers can streamline their build processes and ensure smooth and effective deployments by using tools like OpenTelemetry Collector Contrib, Manifest, and GoReleaser in a structured approach.

The OpenTelemetry Collector Contrib offers a solid framework for building customized telemetry collectors. Manifest files serve as templates for defining crucial components. GoReleaser streamlines this process by automating the release of Go projects, minimizing the manual effort required for building and deploying applications.

Staying Up-to-Date

OpenTelemetry Releases

Developers can access pre-configured workflows and templates by following the OpenTelemetry release process and using the OpenTelemetry releases repository. These resources make automating builds for different architectures easier, saving time and ensuring a standardized and reliable deployment process.

Continuous Integration and Continuous Deployment (CI/CD)

Deployed systems need to stay healthy and perform well. This requires using continuous integration and continuous deployment practices. These practices allow for regular updates, ensuring that builds always include the latest changes and enhancements. This helps to keep deployments secure and optimized. Tools like GitHub Actions automate builds and releases, improving overall efficiency.

Why You Need Vulnerability Checks

It is important to check for potential security risks before deploying software components to production. These checks are typically performed within the OpenTelemetry framework to ensure that custom components such as receivers and processors are secure and don't introduce vulnerabilities into the system. By addressing security concerns proactively, developers can prevent potential breaches and maintain a strong security posture.

Integrating Vulnerability Checks into the Workflow

To effectively check for vulnerabilities, adding a validation step to the development workflow is important. This step includes:

• Building the component: Make sure the custom component is correctly built and works.
• Checking for vulnerabilities: Scan the component for security issues.
• Meeting security standards: Verify that the component follows security guidelines before deployment.

Automating Vulnerability Checks

Developers can automate the scanning and validation processes by leveraging tools like GitHub Actions. This helps improve security by seamlessly integrating these processes into the deployment pipeline. Here’s how automation enhances security.

• Consistency: Automated processes make sure that every part undergoes the same strict security checks, reducing human errors and oversights.
• Efficiency: Simplified workflows speed up the scanning and validation processes, making deployments faster and more reliable.
• Proactive Security: Regular automated checks help find vulnerabilities early in the development cycle, allowing prompt fixes.

Implementing Automated Vulnerability Checks

1. Set up automated actions: Configure your workflow to build components and perform vulnerability scans automatically.

2. Continuous monitoring: Ensure that your automation tools continuously monitor for vulnerabilities, even after deployment.

3. Regular updates: Keep the vulnerability scanning tools up to date with the latest security standards and threat intelligence to detect new vulnerabilities.

Mastering the OpenTelemetry Collector empowers your organization to manage and optimize telemetry data efficiently. You can ensure a seamless and robust data collection process by customizing components, leveraging the OTel contrib distribution, and following best practices for setup, deployment, and security. Ready to take control of your telemetry data? Start building your custom OpenTelemetry Collector today and transform how you manage and analyze data.