Complimentary Gartner® Report! 'A CTO's Guide to Open-Source Software: Answering the Top 10 FAQs.'Read more

Prometheus Configuration

When operating a self-managed Prometheus instance, BindPlane's server configuration must be updated to connect to the remote Prometheus instance.

BindPlane Configuration

After installing BindPlane OP, update the configuration file at /etc/bindplane/config.yaml using the editor of your choice.

  • Set prometheus.enableRemote to true
  • Set prometheus.host to the IP address or Hostname of your Prometheus server.
yaml
1prometheus:
2  enableRemote: true
3  localFolder: /var/lib/bindplane/prometheus
4  host: prometheus.c.project.internal
5  port: '9090'
6  remoteWrite:
7    endpoint: /api/v1/write
8  auth:
9    type: none

Once enableRemote and host are configured, restart the BindPlane server process.

bash
1sudo systemctl restart bindplane

At this point, BindPlane OP is installed and configured to use the remote Prometheus instance.

Security

Prometheus supports several options for security. Basic authentication (Basic auth), Transport Layer Security (TLS), and Mutual TLS (mTLS).

Basic Authentication

Follow the Prometheus Basic Auth Password Hashing documentation to generate a password hash.

Once you have your hash, update /etc/prometheus/web.yml with your basic auth username and password hash.

yaml
1# Example use only: admin:password
2basic_auth_users:
3  admin: $2b$12$maOicLymWgsIQleRCm604ePbaaavp9cKj3bJUg0IrcVXCHB3terLa

Restart the Prometheus service.

bash
1sudo systemctl restart prometheus

Test by making a curl request, without basic auth. You should expect a "401 Unauthorized" response.

bash
1curl -v -s localhost:9090/metrics > /dev/null

Test by making a curl request with your username and password.

bash
1curl -v -s -u 'admin:password' localhost:9090/metrics > /dev/null

You should expect a "200 OK" response. This will indicate that basic auth is working correctly.

Next, we need to update BindPlane with the new credentials. Edit /etc/bindplane/config.yaml on all of your BindPlane servers.

yaml
1prometheus:
2  enableRemote: true
3  localFolder: /var/lib/bindplane/prometheus
4  host: prometheus.c.bpcli-dev.internal
5  port: '9090'
6  remoteWrite:
7    endpoint: /api/v1/write
8  auth:
9    type: basic
10    username: admin
11    password: password

Restart the BindPlane service.

bash
1sudo systemctl restart bindplane

Transport Layer Security (TLS)

Copy the certificate keypair to /etc/prometheus/tls. The example commands assume that you have a certificate key pair in your working directory named prometheus.crt and prometheus.key

bash
1sudo mkdir /etc/prometheus/tls
2
3sudo mv prometheus.crt prometheus.key /etc/prometheus/tls
4
5sudo chown -R prometheus:prometheus /etc/prometheus/tls
6sudo chmod 0600 \
7  /etc/prometheus/tls/prometheus.crt \
8  /etc/prometheus/tls/prometheus.key

Server side TLS can be configured by editing the web configuration file at /etc/prometheus/web.yml and configuring the certificate file and private key file paths.

yaml
1tls_server_config:
2  cert_file: /etc/prometheus/tls/prometheus.crt
3  key_file: /etc/prometheus/tls/prometheus.key

Restart the Prometheus service.

bash
1sudo systemctl restart prometheus

You can test if Prometheus is using TLS by using curl.

bash
1curl -kvs https://localhost:9090/metrics > /dev/null

You should expect a "200 OK" response. This will indicate that server side TLS is working correctly.

Next, we need to update BindPlane to use TLS when communicating with Prometheus. On all of your servers, perform the following steps.

Copy the certificate authority to /etc/bindplane/tls. The example commands assume that you have a certificate authority public key named ca.crt in your working directory.

bash
1sudo mkdir /etc/bindplane/tls
2sudo mv ca.crt /etc/bindplane/tls
3
4sudo chown -R bindplane:bindplane /etc/bindplane/tls
5sudo chmod 0600 /etc/bindplane/tls/ca.crt

Edit /etc/bindplane/config.yaml on all of your BindPlane servers and add the tls.tlsCa parameter.

yaml
1prometheus:
2  enableRemote: true
3  localFolder: /var/lib/bindplane/prometheus
4  host: prometheus.c.bpcli-dev.internal
5  port: '9090'
6  remoteWrite:
7    endpoint: /api/v1/write
8  auth:
9    type: none
10  enableTLS: true
11  tls:
12    tlsSkipVerify: false
13    tlsCa:
14      - /etc/bindplane/tls/ca.crt

note

Make sure prometheus.host matches the hostname of the Prometheus server's certificate. If the hostname does not match, you can set prometheus.tls.tlsSkipVerify to true to skip TLS verification. Skipping TLS verification is not recommended in a production environment.

Restart the BindPlane service.

bash
1sudo systemctl restart bindplane

Mutual TLS

Copy the certificate keypair and certificate authority to /etc/prometheus/tls. The example commands assume that you have a certificate key pair in your working directory named prometheus.crt and prometheus.key and a certificate authority named ca.crt.

bash
1sudo mkdir /etc/prometheus/tls
2
3sudo mv prometheus.crt prometheus.key ca.crt /etc/prometheus/tls
4
5sudo chown -R prometheus:prometheus /etc/prometheus/tls
6sudo chmod 0600 \
7  /etc/prometheus/tls/prometheus.crt \
8  /etc/prometheus/tls/prometheus.key \
9  /etc/prometheus/tls/ca.crt

Mutual TLS can be configured by editing the web configuration file at /etc/prometheus/web.yml and configuring the certificate file, private key file paths and certificate authority paths.

yaml
1tls_server_config:
2  client_auth_type: RequireAndVerifyClientCert
3  client_ca_file: /etc/prometheus/tls/ca.crt
4  cert_file: /etc/prometheus/tls/prometheus.crt
5  key_file: /etc/prometheus/tls/prometheus.key

Restart the Prometheus service.

bash
1sudo systemctl restart prometheus

You can test if Prometheus is using TLS by using curl on the Prometheus system.

bash
1# Sudo is required to read the TLS certificate files
2# in /etc/prometheus/tls.
3# Replace $(hostname -f) with the hostname that matches
4# the prometheus server and certificate.
5sudo curl -vs \
6  --cacert /etc/prometheus/tls/ca.crt \
7  --cert /etc/prometheus/tls/prometheus.crt \
8  --key /etc/prometheus/tls/prometheus.key \
9  "https://$(hostname -f):9090/metrics" > /dev/null

You should expect a "200 OK" response. This will indicate that mutual TLS is working correctly.

Next, we need to update BindPlane to use mutual TLS when communicating with Prometheus. On all of your servers, perform the following steps.

Copy the certificate authority and client keypair to /etc/bindplane/tls. The example commands assume that you have a certificate key pair in your working directory named bindplane.crt and bindplane.key and a certificate authority named ca.crt.

bash
1sudo mkdir /etc/bindplane/tls
2sudo mv bindplane.crt bindplane.key ca.crt /etc/bindplane/tls
3
4sudo chown -R bindplane:bindplane /etc/bindplane/tls
5sudo chmod 0600 \
6  /etc/bindplane/tls/bindplane.crt \
7  /etc/bindplane/tls/bindplane.key \
8  /etc/bindplane/tls/ca.crt

Edit /etc/bindplane/config.yaml on all of your BindPlane servers and add the tls parameters.

yaml
1prometheus:
2  enableRemote: true
3  localFolder: /var/lib/bindplane/prometheus
4  host: prometheus.c.bpcli-dev.internal
5  port: '9090'
6  remoteWrite:
7    endpoint: /api/v1/write
8  auth:
9    type: none
10  enableTLS: true
11  tls:
12    tlsSkipVerify: false
13    tlsCa:
14      - /etc/bindplane/tls/ca.crt
15    tlsCert: /etc/bindplane/tls/bindplane.crt
16    tlsKey: /etc/bindplane/tls/bindplane.key

note

Make sure prometheus.host matches the hostname of the Prometheus server's certificate. If the hostname does not match, you can set prometheus.tls.tlsSkipVerify to true to skip TLS verification. Skipping TLS verification is not recommended in a production environment.

Restart the BindPlane service.

bash
1sudo systemctl restart bindplane