How to Monitor Riak Metrics with OpenTelemetry

observIQ’s OpenTelemetry members contributed Riak metric monitoring support to OpenTelemetry! You can now monitor your Riak agent performance with OpenTelemetry and deploy it simply with the oIQ OpenTelemetry Collector.

You can add the Riak metric receiver to any OpenTelemetry collector. This post demonstrates a configuration for shipping metrics to Google Cloud Operations with OpenTelemetry components. This configuration and many other observIQ OpenTelemetry configurations are available in the oIQ Opentelemetry Collector.

Installation and configuration are simple, but you can refine your configuration once the metric receiver is up and running. The configuration is easily editable as a yaml file. You can find more documentation, example configurations for other receivers, and observability tools on GitHub and our blog.

What Matters for Riak Metrics

Riak deployments can get complicated and tedious. Large environments stress throughput, and monitoring metrics offer insight into resource usage, stability, and overall health.

Step 1: Installing the Collector

The oIQ OpenTelemetry Collector can be installed on Windows, MacOS, and Linux using single-line install commands that can be copied directly from GitHub. Please make sure you have administrator privileges on your device or VM when running the installation.

Windows:

1msiexec /i "https://github.com/observIQ/observiq-otel-collector/releases/latest/download/observiq-otel-collector.msi" /quiet

MacOS/Linux:

1sudo sh -c "$(curl -fsSlL https://github.com/observiq/observiq-otel-collector/releases/latest/download/install_unix.sh)" install_unix.sh

Step 2: Prerequisites and Authentication Credentials

In the following example, we are using Google Cloud Operations as the destination. However, OpenTelemtry offers exporters for many destinations. Check out the list of exporters here.

Setting up Google Cloud exporter prerequisites:

If running outside of Google Cloud (On-prem, AWS, etc) or without the Cloud Monitoring scope, the Google Exporter requires a service account.

Create a service account with the following roles:

• Metrics: roles/monitoring.metricWriter
• Logs: roles/logging.logWriter

Create a service account JSON key and place it in the system running the collector.

Related Content: Rapid telemetry for Windows with OpenTelemetry and BindPlane OP

MacOS/Linux

In this example, the key is placed at /opt/observiq-otel-collector/sa.json, and its permissions are restricted to the user running the collector process.

1sudo cp sa.json /opt/observiq-otel-collector/sa.json
2sudo chown observiq-otel-collector: /opt/observiq-otel-collector/sa.json
3sudo chmod 0400 /opt/observiq-otel-collector/sa.json

Set the GOOGLE_APPLICATION_CREDENTIALS environment variable by creating a systemd override. A systemd override allows users to modify the systemd service configuration without changing the service directly. This allows package upgrades to happen seamlessly. You can learn more about systemd units and overrides here.

Run the following command.

1sudo systemctl edit observiq-otel-collector

If this is the first time an override is being created, paste the following contents into the file:

1[Service]
2Environment=GOOGLE_APPLICATION_CREDENTIALS=/opt/observiq-otel-collector/sa.json

If an override is already in place, insert the Environment parameter into the existing Service section.

Restart the collector

1sudo systemctl restart observiq-otel-collector

Windows

In this example, the key is placed at C:/observiq/collector/sa.json.

Set the GOOGLE_APPLICATION_CREDENTIALS with the command prompt setx command.

Run the following command.

1setx GOOGLE_APPLICATION_CREDENTIALS "C:/observiq/collector/sa.json" /m

Restart the service using the services application.

Related Content: How to Monitor MySQL with OpenTelemetry

Step 3: Configure the Riak Receiver

After installation, the config file for the collector can be found at:

Windows: C:\Program Files\observIQ OpenTelemetry Collector\config.yaml

MacOS/Linux: /opt/observiq-otel-collector/config.yaml

Edit the config file with the following configuration:

1receivers:
2  riak:
3    endpoint: "localhost:8098"    
4    username: username    
5    password: password
6    collection_interval: 60s
7
8processors:
9  # Resourcedetection is used to add a unique (host.name)
10  # to the metric resource(s), allowing users to filter
11  # between multiple agent systems.
12  resourcedetection:
13    detectors: ["system"]
14    system:
15      hostname_sources: ["os"]
16
17  # Used for Google generic_node mapping.
18  resource:
19    attributes:
20    - key: namespace
21      value: "riak"
22      action: upsert
23    - key: location
24      value: "global"
25      action: upsert
26
27  normalizesums:
28
29  batch:
30
31exporters:
32  googlecloud:
33    retry_on_failure:
34      enabled: false
35    metric:
36      prefix: workload.googleapis.com
37    resource_mappings:
38    - source_type: ""
39      target_type: generic_node
40      label_mappings:
41      - source_key: host.name
42        target_key: node_id
43      - source_key: location
44        target_key: location
45      - source_key: namespace
46        target_key: namespace
47
48service:
49  pipelines:
50    metrics:
51      receivers:
52      - riak
53      processors:
54      - resourcedetection
55      - resource
56      - normalizesums
57      - batch
58      exporters:
59      - googlecloud

The configuration is set to receive metrics from the Riak system to Google Cloud. You can specify your destination and insert any necessary credentials near the top of the “receivers” section of the config file.

The following notes apply to Google Cloud:

1. The interval for fetching metrics is 60 seconds by default.

• In the Google Cloud exporter, do the following mapping:
  • Set the target type to a generic node to simplify filtering metrics from the collector in cloud monitoring.
  • Set node_id, location, and namespace for the metrics. The resource processor sets the location and namespace.

1. The project ID is not set in the configuration. Google automatically detects the project ID.
2. Add the normalizesums processor to exclude the first metric with a zero value when the configuration is done and the collector is restarted.
3. Add the batch processor to bundle the metrics from multiple receivers. We highly recommend using this processor in the configuration, especially for the benefit of the collector's logging component.

Related Content: Turning Logs into Metrics with OpenTelemetry and BindPlane OP

Step 4: View Your Metrics

Below is a list of metrics that are collected by the OpenTelemetry Riak receiver. The metrics are sent to Google Cloud, or the destination designated during setup, for analysis.

Go to the Google Cloud Console—Head to Metrics Explorer to view the metrics. Select the resource as a generic node. You can filter by namespace to view specific metrics.

Broken image

observIQ’s OpenTelemetry distribution is an easy way for anyone looking to implement OpenTelemetry observability standards in their IT environments. The single-line installer, seamlessly integrated receivers, exporter, and processor pool make working with this collector simple. Follow this space to keep up with all our future posts and simplified configurations for various sources. For questions, requests, and suggestions, reach out to our support team at support@observIQ.com.