OpenTelemetry is at the core of standardizing telemetry solutions and observIQ is focused on building the very best in open source telemetry software. Our relationship with OpenTelemetry began in 2021, with observIQ contributing our logging agent, Stanza, to the OpenTelemetry community. Now, we are shifting our focus to simplifying OpenTelemetry solutions to its large base of users. On that note, we launched a collector that combines the best of both worlds, with OpenTelemetry at its core, combined with observIQ’s functionalities to simplify its usage.
In this post, we are taking you through the installation of the observIQ distribution of the OpenTelemetry collector and the steps to configure the collector to fetch metrics from a MySQL server, eventually forwarding those metrics to Google Cloud Operations. Before we delve into the installation, here’s a quick look at the architecture of an OpenTelemetry Collector:
- observIQ-Otel Collector: It collects the metrics fetched by the receiver, runs the processor, and forwards it to the exporter for distribution to the destination.
- Receiver: Scrapes the metrics from the source based on push or pull-based requests. The receiver is capable of fetching metrics from one or more data sources.
- Processor: Enrich the metrics data using processors. Metrics data are enriched after being received from the receiver and before being forwarded to the exporter. Although processors are optional, in some scenarios using processors is highly recommended.
- Exporter: Sends the metrics to the configured destination using push or pull-based requests. As in the case of receivers, exporters support having one or more data destinations for the metrics data.
Getting Started: Installation
- There are two ways to do the installation, a simple installation script or via deb/ rpm packages.
Prerequisites for Installation:
- Root level permissions for manual installations
- Download the latest deb and rpm packages
1. Visit the Releases page in the repo to download the latest version of the installation package or use the following installation script:
sudo sh -c "$(curl -fsSlL https://github.com/observiq/observiq-otel-collector/releases/latest/download/install_unix.sh)" install_unix.sh
2. After running the installation script, you can verify the service is running with the following command:
systemctl status observiq-otel-collector
3. View the logs from the collector that are logged to journalD.
journalctl -u observiq-otel-collector.service
- The collector is installed using the latest version of MSI available at the releases page in our repo. You may choose to install using the MSI or use the powershell command to install in quiet mode. This helps when replicating and mirroring the installation/ configuration using Ansible, Terraform etc.
- Post installation, you may start or stop the collector from Windows Service Manager
Prerequisites for Installation:
- Administrator level permissions for manual installations
- Download the MSI to the local drive
1. Visit the releases page in the repo to download the latest version of the MSI. Windows may ask you to change some security settings for package downloads.
2. Run the installation
3. At the end of the installation, the following is installed under C:\Program Files\observiq-otel-collector:
4. use the command
Alternatively, Open services to check if the collector is running.
- Plugins folder: This folder contains the configuration files for a host of logging configurations.
- Config.yaml: The default configuration file with default settings. Visit our repo to see a sample configuration.
- Observiq-otel-collector: The actual collector binary file.
- Opentelemetry-java-contrib-jmx-metrics: The jar file that must be used with the JMX receiver for scraping JMX metrics.
Getting Started: Configuring the collector
Before we begin the configuration, here are some important points to note about the default configuration that is downloaded as part of the installation:
- For fetching metrics specific to certain sources, we use the configuration specific to those receivers. A list of available receivers is documented in OpenTelemetry’s repo.
- The advantage of using obersvIQ’s distribution of OpenTelemetry is in how the collector seamlessly ties in the receivers, processors, and exporters, all done using a single line installation.
- In the default config.yaml, the scraped metrics are not exported to any destination. After we configure the receiver, we must configure an exporter to specify the destination for the metrics export.
In this sample configuration, the steps to use the mysqlreceiver to fetch metrics from a MySQL database and export them to Google Cloud Operations are detailed. This is how it works:
- The collector scrapes metrics and logs from MySQL and exports them to a destination assigned in the configuration file.
- To export the metrics to Google Cloud Operations, use the configurations outlined in the googlecloudexporter
- Have a MySQL database setup
- Set up an OTel user in MySQL. This user needs to have select and process permissions enabled for all databases to be monitored. The configuration detailed below monitors all databases.
- A Google cloud service account and JSON key file downloaded and saved to /opt/observiq-otel-collector
- Move the Google service account key to the observiq-collector folder, and change permissions.
sudo cp ./otel-agent-dev.json /opt/observiq-otel-collector/otel-agent-dev.json sudo chgrp observiq-otel-collector /opt/observiq-otel-collector/otel-agent-dev.json sudo chown observiq-otel-collector /opt/observiq-otel-collector/otel-agent-dev.json sudo chmod 0640 /opt/observiq-otel-collector/otel-agent-dev.json
2. Modify the service using systemctl to include the service account information in environment
sudo systemctl edit observiq-otel-collector
Paste the following, to include the environment variable to the service.
3. Change the config.yaml with the configuration to use the mysqlreceiver and the googlecloudexporter
receivers: mysql: username: otel password: password endpoint: localhost:3306 processors: batch: send_batch_max_size: 200 send_batch_size: 200 exporters: googlecloud: retry_on_failure: enabled: false project: otel-agent-dev service: pipelines: metrics: receivers: [mysql] processors: [batch] exporters: [googlecloud]
4. Restart the collector
systemctl restart observiq-otel-collector
Getting Started: Viewing the metrics in Google Cloud Operations
You should now be able to view the MySQL metrics in your Metrics explorer. Nice work! This is how simple it is to work with receivers and exporters in OpenTelemetry.
Here’s a dashboard created using the metrics gathered:
|mysql.buffer_pool_pages||The number of pages in the InnoDB buffer pool.||custom.googleapis.com/opencensus/mysql.buffer_pool_pages|
|mysql.buffer_pool_operations||The number of operations on the InnoDB buffer pool.||custom.googleapis.com/opencensus/mysql.buffer_pool_operations|
|mysql.buffer_pool_size||The number of operations on the InnoDB buffer pool.||custom.googleapis.com/opencensus/mysql.buffer_pool_operations|
|mysql.buffer_pool_size||The number of bytes in the InnoDB buffer pool.||custom.googleapis.com/opencensus/mysql.buffer_pool_size|
|mysql.commands||The number of times each type of command has been executed.||custom.googleapis.com/opencensus/mysql.commands|
|mysql.handlers||The number of requests to various MySQL handlers.||custom.googleapis.com/opencensus/mysql.handlers|
|mysql.double_writes||The number of writes to the InnoDB doublewrite buffer.||custom.googleapis.com/opencensus/mysql.double_writes|
|mysql.log_operations||The number of InndoDB log operations.||custom.googleapis.com/opencensus/mysql.log_operations|
|mysql.operations||The number of InndoDB operations.||custom.googleapis.com/opencensus/mysql.operations
|mysql.page_operations||The number of InndoDB page operations.||custom.googleapis.com/opencensus/mysql.page_operations
|mysql.row_locks||The number of InndoDB row locks.||custom.googleapis.com/opencensus/mysql.row_locks
|mysql.row_operations||The number of InndoDB row operations.||custom.googleapis.com/opencensus/mysql.row_operations
|mysql.locks||The number of MySQL locks.||custom.googleapis.com/opencensus/mysql.locks
|mysql.sorts||The number of MySQL sorts.||custom.googleapis.com/opencensus/mysql.sorts
|mysql.threads||The state of MySQL threads.||custom.googleapis.com/opencensus/mysql.threads
observIQ’s distribution is a game-changer for companies looking to implement the OpenTelemetry standards. The single line installer, seamlessly integrated receivers, exporter, and processor pool make working with this collector simple. Follow this space to keep up with all our future posts and simplified configurations for various sources. For questions, requests, and suggestions, reach out to our support team at support@observIQ.com.