The Observability Blog

Categories:
  • Metrics
  • OpenTelemetry

How to Monitor Redis with OpenTelemetry

Deepa Ramachandra headshot
by Deepa Ramachandra on
April 29, 2022

We’re excited to announce that we’ve recently contributed Redis monitoring support to the OpenTelemetry collector. You can check it out here!

You can utilize this receiver in conjunction with any OTel collector: including the contrib collector, the observIQ’s distribution of the collector, as well as Google’s Ops Agent, as a few examples.

Below are steps to get up and running quickly with observIQ’s distribution, and shipping Redis metrics to a popular backend: Google Cloud Ops. You can find out more on observIQ’s GitHub page: https://github.com/observIQ/observiq-otel-collector

What signals matter?

Unlike other databases, monitoring the performance of Redis is relatively simple, focusing on the following categories of KPIs:

  • Memory Utilization
  • Database Throughput
  • Cache hit ratio and evicted cache data
  • Number of connections
  • Replication

All of the above categories can be gathered with the Redis receiver – so let’s get started.

Step 1: Installing the collector

The simplest way to get started is with one of the single-line installation commands shown below. For more advanced options, you’ll find a variety of installation options for Linux, Windows, and macOS on GitHub.

Use the following single-line installation script to install the observIQ OpenTelemetry collector. Please note that the collector must be installed on the Redis system. 

Windows:

msiexec /i "https://github.com/observIQ/observiq-otel-collector/releases/latest/download/observiq-otel-collector.msi" /quiet

MacOS/Linux:

sudo sh -c "$(curl -fsSlL https://github.com/observiq/observiq-otel-collector/releases/latest/download/install_unix.sh)" install_unix.sh

Step 2: Setting up pre-requisites and authentication credentials

In the following example, we are using Google Cloud Operations as the destination. However, OpenTelemtry offers exporters for many destinations. Check out the list of exporters here

Setting up Google Cloud exporter prerequisites:

If running outside of Google Cloud (On prem, AWS, etc) or without the Cloud Monitoring scope, the Google Exporter requires a service account.

Create a service account with the following roles:

  • Metrics: roles/monitoring.metricWriter
  • Logs: roles/logging.logWriter

Create a service account JSON key and place it on the system that is running the collector.

MacOS/Linux

In this example, the key is placed at /opt/observiq-otel-collector/sa.json and its permissions are restricted to the user running the collector process.

sudo cp sa.json /opt/observiq-otel-collector/sa.json
sudo chown observiq-otel-collector: /opt/observiq-otel-collector/sa.json
sudo chmod 0400 /opt/observiq-otel-collector/sa.json

Set the GOOGLE_APPLICATION_CREDENTIALS environment variable by creating a systemd override. A systemd override allows users to modify the systemd service configuration without modifying the service directly. This allows package upgrades to happen seamlessly. You can learn more about systemd units and overrides here.

Run the following command

sudo systemctl edit observiq-otel-collector

If this is the first time an override is being created, paste the following contents into the file:

[Service]
Environment=GOOGLE_APPLICATION_CREDENTIALS=/opt/observiq-otel-collector/sa.json

If an override is already in place, simply insert the Environment parameter into the existing Service section.

Restart the collector

sudo systemctl restart observiq-otel-collector

Windows

In this example, the key is placed at C:/observiq/collector/sa.json.

Set the GOOGLE_APPLICATION_CREDENTIALS with the command prompt setx command.

Run the following command

setx GOOGLE_APPLICATION_CREDENTIALS "C:/observiq/collector/sa.json" /m

Restart the service using the services application.

Step 3: Configuring the Redis receiver

After the installation, the config file for the collector can be found at 

  • C:\Program Files\observIQ OpenTelemetry Collector\config.yaml (Windows)
  • /opt/observiq-otel-collector/config.yaml(Linux)

Edit the configuration file and use the following configuration. 

receivers:
  redis:
    endpoint: "localhost:6379"
    collection_interval: 60s

processors:
  # Resourcedetection is used to add a unique (host.name)
  # to the metric resource(s), allowing users to filter
  # between multiple agent systems.
  resourcedetection:
    detectors: ["system"]
    system:
      hostname_sources: ["os"]

  # Used for Google generic_node mapping.
  resource:
    attributes:
    - key: namespace
      value: "redis"
      action: upsert
    - key: location
      value: "global"
      action: upsert

  normalizesums:

  batch:

exporters:
  googlecloud:
    retry_on_failure:
      enabled: false
    metric:
      prefix: workload.googleapis.com
    resource_mappings:
    - source_type: ""
      target_type: generic_node
      label_mappings:
      - source_key: host.name
        target_key: node_id
      - source_key: location
        target_key: location
      - source_key: namespace
        target_key: namespace

service:
  pipelines:
    metrics:
      receivers:
      - redis
      processors:
      - resourcedetection
      - resource
      - normalizesums
      - batch
      exporters:
      - googlecloud

In the example above, the Redis receiver configuration is set to:

  1. Receive metrics from the Redis system at the specified endpoint. 
  2. Set the time interval for fetching the metrics. The default value for this parameter is 10s. However, if exporting metrics to Google Cloud operations, this value is set to 60s by default. 
  3. The resource detection processor is used to create a distinction between metrics received from multiple Redis systems. This helps with filtering metrics from specific Redis hosts in the monitoring tool, in this case, Google Cloud operations.
  4. In the Google Cloud exporter here, do the following mapping:
  • Set the target type to a generic node, to simplify filtering metrics from the collector in cloud monitoring.
  • Set node_id, location, and namespace for the metrics. Location and namespace are set from the resource processor. 
  1. It is important to note that the project ID is not set in the googlecloud exporter configuration. Google automatically detects the project ID
  2. Add the normalizesums processor to exclude the first metric that has a zero value when the configuration is done and the collector is restarted. To know more about this processor, check the OpenTelemetry documentation.
  3. Add the batch processor to bundle the metrics from multiple receivers. We highly recommend using this processor in the configuration, especially for the benefit of the logging component of the collector. To learn more about this processor check the documentation.
  4. It is recommended to set the retry_on_failure to false. If this is not set, the retry attempts fall into a loop for five attempts. 

Step 4: Viewing the metrics collected in Google cloud operations

If you followed the steps detailed above, you should see the following metrics exported to Metrics Explorer. 

 MetricDescriptionNamespace
1
redis.uptimeNumber of seconds since Redis server startcustom.googleapis.com/opencensus/redis.uptime
2
redis.cpu.timeSystem CPU consumed by the Redis server in seconds since server startcustom.googleapis.com/opencensus/redis.cpu.time
3
redis.clients.connectedNumber of client connections (excluding connections from replicas)custom.googleapis.com/opencensus/redis.clients.connected
4
redis.clients.max_input_bufferBiggest input buffer among current client connectionscustom.googleapis.com/opencensus/redis.clients.max_input_buffer
5
redis.clients.max_output_bufferLongest output list among current client connectionscustom.googleapis.com/opencensus/redis.clients.max_output_buffer
6
redis.clients.blockedNumber of clients pending on a blocking callcustom.googleapis.com/opencensus/redis.clients.blocked
7
redis.keys.expiredTotal number of key expiration eventscustom.googleapis.com/opencensus/redis.keys.expired
8
redis.keys.evictedNumber of evicted keys due to maxmemory limitcustom.googleapis.com/opencensus/redis.keys.evicted
9
redis.connections.receivedTotal number of connections accepted by the servercustom.googleapis.com/opencensus/redis.connections.received
10
redis.connections.rejected:Number of connections rejected because of maxclients limitcustom.googleapis.com/opencensus/redis.connections.rejected:
11
redis.memory.usedTotal number of bytes allocated by Redis using its allocatorcustom.googleapis.com/opencensus/redis.memory.used
12
redis.memory.peakPeak memory consumed by Redis (in bytes)custom.googleapis.com/opencensus/redis.memory.peak
13
redis.memory.rssNumber of bytes that Redis allocated as seen by the operating systemcustom.googleapis.com/opencensus/redis.memory.rss
14
redis.memory.luaNumber of bytes used by the Lua enginecustom.googleapis.com/opencensus/redis.memory.lua
15
redis.memory.fragmentation_ratioRatio between used_memory_rss and used_memorycustom.googleapis.com/opencensus/redis.memory.fragmentation_ratio
16
redis.rdb.changes_since_last_saveNumber of changes since the last dumpcustom.googleapis.com/opencensus/redis.rdb.changes_since_last_save
17
redis.commandsNumber of commands processed per secondcustom.googleapis.com/opencensus/redis.commands
18
redis.commands.processedTotal number of commands processed by the servercustom.googleapis.com/opencensus/redis.commands.processed
19
redis.net.inputThe total number of bytes read from the networkcustom.googleapis.com/opencensus/redis.net.input
20
redis.net.outputThe total number of bytes written to the networkcustom.googleapis.com/opencensus/redis.net.output
21
redis.keyspace.hitsNumber of successful lookup of keys in the main dictionarycustom.googleapis.com/opencensus/redis.keyspace.hits
22
redis.keyspace.missesNumber of failed lookup of keys in the main dictionarycustom.googleapis.com/opencensus/redis.keyspace.misses
23
redis.latest_forkDuration of the latest fork operation in microsecondscustom.googleapis.com/opencensus/redis.latest_fork
24
redis.slaves.connectedNumber of connected replicascustom.googleapis.com/opencensus/redis.slaves.connected
25
redis.replication.backlog_first_byte_offsetThe master offset of the replication backlog buffercustom.googleapis.com/opencensus/redis.replication.backlog_first_byte_offset
26
redis.replication.offsetThe server's current replication offsetcustom.googleapis.com/opencensus/redis.replication.offset
27
redis.db.keysNumber of keyspace keyscustom.googleapis.com/opencensus/redis.db.keys
28
redis.db.expiresNumber of keyspace keys with an expirationcustom.googleapis.com/opencensus/ redis.db.expires
29
redis.db.avg_ttlAverage keyspace keys TTLcustom.googleapis.com/opencensus/redis.db.avg_ttl

To view the metrics follow the steps outlined below:

  1. In the Google Cloud Console, head to metrics explorer 
  2. Select the resource as a generic node. 
  3. Follow the namespace equivalent in the table above and filter the metric to view the chart. 

observIQ’s distribution is a game-changer for companies looking to implement the OpenTelemetry standards. The single line installer, seamlessly integrated receivers, exporter, and processor pool make working with this collector simple. Follow this space to keep up with all our future posts and simplified configurations for various sources. For questions, requests, and suggestions, reach out to our support team at support@observIQ.com.